2023-10-11T14:12:08.738 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-11T14:12:08.739 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167 --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} --verbose -- /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/orig.config.yaml 2023-10-11T14:12:08.780 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T14:12:08.863 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167 branch: wip-yuri-testing-2023-10-10-0840-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7421167' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-10-10-0840-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-qa-63004-reef sha1: 4305a5835ec2627a352020742c8983c32bac9341 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 1482 sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-qa-63004-reef suite_path: /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 4305a5835ec2627a352020742c8983c32bac9341 targets: smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdZOUR/JCj79beNQ0wybs7M7i43qMxJmr70rOt4AEhBgiCaM4r0c1U41R5LwI8ihJ/8urmC0YXaW4daE5rbCfs= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBaymyTQnulDVyWSaS/bdKw8zsfFDB9kBVESWAo2vuV0yWDK8SWyG3kxrnaNRny+vL4X8wX5KsD37JgctSCCvU= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-11_13:59:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-11T14:12:08.863 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa; will attempt to use it 2023-10-11T14:12:08.864 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks 2023-10-11T14:12:08.864 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-11T14:12:08.867 INFO:teuthology.task.internal:Checking packages... 2023-10-11T14:12:08.892 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22' 2023-10-11T14:12:08.893 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-11T14:12:08.893 INFO:teuthology.packaging:ref: None 2023-10-11T14:12:08.893 INFO:teuthology.packaging:tag: None 2023-10-11T14:12:08.894 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-10-0840-reef 2023-10-11T14:12:08.894 INFO:teuthology.packaging:sha1: aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-11T14:12:08.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-10-0840-reef 2023-10-11T14:12:09.188 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-666.gaa3250ce 2023-10-11T14:12:09.189 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-11T14:12:09.196 INFO:teuthology.task.internal:no buildpackages task found 2023-10-11T14:12:09.196 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-11T14:12:09.240 INFO:teuthology.task.internal:Saving configuration 2023-10-11T14:12:09.259 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-11T14:12:09.265 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T14:12:09.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 14:00:48.411449', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdZOUR/JCj79beNQ0wybs7M7i43qMxJmr70rOt4AEhBgiCaM4r0c1U41R5LwI8ihJ/8urmC0YXaW4daE5rbCfs='} 2023-10-11T14:12:09.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 14:00:48.409643', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBaymyTQnulDVyWSaS/bdKw8zsfFDB9kBVESWAo2vuV0yWDK8SWyG3kxrnaNRny+vL4X8wX5KsD37JgctSCCvU='} 2023-10-11T14:12:09.315 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-11T14:12:09.322 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-10-11T14:12:09.322 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-10-11T14:12:09.323 INFO:teuthology.run_tasks:Running task console_log... 2023-10-11T14:12:09.393 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9a0a927a60>, signals=[15]) 2023-10-11T14:12:09.394 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-11T14:12:09.403 INFO:teuthology.task.internal:Opening connections... 2023-10-11T14:12:09.404 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:12:09.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:12:09.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:12:09.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:12:09.544 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-11T14:12:09.551 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2023-10-11T14:12:09.574 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2023-10-11T14:12:09.574 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2023-10-11T14:12:09.635 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2023-10-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8" 2023-10-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2023-10-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2023-10-11T14:12:09.636 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8" 2023-10-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2023-10-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2023-10-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-11T14:12:09.637 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2023-10-11T14:12:09.638 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T14:12:09.638 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T14:12:09.638 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-11T14:12:09.639 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2023-10-11T14:12:09.663 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-10-11T14:12:09.685 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-10-11T14:12:09.685 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-10-11T14:12:09.747 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2023-10-11T14:12:09.747 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8" 2023-10-11T14:12:09.747 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2023-10-11T14:12:09.747 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2023-10-11T14:12:09.748 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2023-10-11T14:12:09.748 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-10-11T14:12:09.748 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-11T14:12:09.748 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-10-11T14:12:09.748 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-11T14:12:09.749 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2023-10-11T14:12:09.749 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T14:12:09.749 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T14:12:09.749 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-11T14:12:09.750 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-10-11T14:12:09.805 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-11T14:12:09.814 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-11T14:12:09.820 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-11T14:12:09.821 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2023-10-11T14:12:09.823 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-10-11T14:12:09.843 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-11T14:12:09.850 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-11T14:12:09.851 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2023-10-11T14:12:09.884 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-10-11T14:12:09.932 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-11T14:12:09.986 INFO:teuthology.run_tasks:Running task kernel... 2023-10-11T14:12:10.001 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-11T14:12:10.002 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-11T14:12:10.003 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T14:12:10.004 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T14:12:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T14:12:10.024 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-10-11T14:12:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T14:12:10.026 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-11T14:12:10.087 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-489.el8.x86_64 2023-10-11T14:12:10.087 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-489.el8.x86_64 2023-10-11T14:12:10.088 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-489.el8.x86_64 2023-10-11T14:12:10.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2023-10-11T14:12:10.089 INFO:teuthology.task.kernel:Running kernel on smithi158: 4.18.0-489.el8.x86_64 2023-10-11T14:12:10.089 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2023-10-11T14:12:14.641 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - AppStream 16 MB/s | 33 MB 00:02 2023-10-11T14:12:14.947 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 15 MB/s | 33 MB 00:02 2023-10-11T14:12:22.691 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 49 MB 00:01 2023-10-11T14:12:22.855 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 49 MB 00:01 2023-10-11T14:12:30.215 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2023-10-11T14:12:30.474 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2023-10-11T14:12:30.819 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-10-11T14:12:31.074 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-10-11T14:12:31.167 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-10-11T14:12:31.484 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-10-11T14:12:32.512 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-11T14:12:32.979 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-11T14:12:36.301 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 548 kB/s | 24 kB 00:00 2023-10-11T14:12:36.885 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 409 kB/s | 24 kB 00:00 2023-10-11T14:12:39.759 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-11T14:12:39.759 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-11T14:12:40.016 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:12:40.017 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:12:40.017 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-10-11T14:12:40.017 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:12:40.017 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:12:40.018 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-11T14:12:40.018 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:12:40.018 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-11T14:12:40.018 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-11T14:12:40.018 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:12:40.019 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:12:40.019 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:12:40.019 INFO:teuthology.orchestra.run.smithi172.stdout:Install 3 Packages 2023-10-11T14:12:40.019 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:12:40.020 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 89 M 2023-10-11T14:12:40.020 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 96 M 2023-10-11T14:12:40.021 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:12:40.359 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-11T14:12:40.360 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-11T14:12:40.638 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:12:40.639 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:12:40.639 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2023-10-11T14:12:40.640 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:12:40.640 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:12:40.640 INFO:teuthology.orchestra.run.smithi158.stdout: kernel x86_64 4.18.0-516.el8 baseos 10 M 2023-10-11T14:12:40.640 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:12:40.640 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core x86_64 4.18.0-516.el8 baseos 43 M 2023-10-11T14:12:40.641 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules x86_64 4.18.0-516.el8 baseos 36 M 2023-10-11T14:12:40.641 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:12:40.641 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:12:40.641 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:12:40.641 INFO:teuthology.orchestra.run.smithi158.stdout:Install 3 Packages 2023-10-11T14:12:40.642 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:12:40.642 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 89 M 2023-10-11T14:12:40.643 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 96 M 2023-10-11T14:12:40.643 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:12:41.036 INFO:teuthology.orchestra.run.smithi172.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2023-10-11T14:12:41.595 INFO:teuthology.orchestra.run.smithi158.stdout:(1/3): kernel-4.18.0-516.el8.x86_64.rpm 11 MB/s | 10 MB 00:00 2023-10-11T14:12:41.862 INFO:teuthology.orchestra.run.smithi172.stdout:(2/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-10-11T14:12:42.395 INFO:teuthology.orchestra.run.smithi172.stdout:(3/3): kernel-core-4.18.0-516.el8.x86_64.rpm 20 MB/s | 43 MB 00:02 2023-10-11T14:12:42.395 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:12:42.395 INFO:teuthology.orchestra.run.smithi172.stdout:Total 37 MB/s | 89 MB 00:02 2023-10-11T14:12:42.470 INFO:teuthology.orchestra.run.smithi158.stdout:(2/3): kernel-modules-4.18.0-516.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2023-10-11T14:12:42.956 INFO:teuthology.orchestra.run.smithi158.stdout:(3/3): kernel-core-4.18.0-516.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-10-11T14:12:42.956 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:12:42.956 INFO:teuthology.orchestra.run.smithi158.stdout:Total 38 MB/s | 89 MB 00:02 2023-10-11T14:12:43.073 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:12:43.485 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:12:43.486 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:12:43.625 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:12:44.051 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:12:44.051 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:12:44.712 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:12:44.742 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:12:45.268 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:12:45.295 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:12:47.500 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:12:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:12:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:12:48.635 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:12:51.674 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:12:52.214 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:12:52.292 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:12:52.592 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:12:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:12:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:12:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:12:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.030 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.055 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.115 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.140 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.411 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:13:18.412 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:13:18.557 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:18.557 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-4.18.0-516.el8.x86_64 1/3 2023-10-11T14:13:18.558 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-core-4.18.0-516.el8.x86_64 2/3 2023-10-11T14:13:19.172 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:19.172 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:13:19.172 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:13:19.173 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.173 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.173 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:13:19.173 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:13:19.304 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-4.18.0-516.el8.x86_64 3/3 2023-10-11T14:13:19.305 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:13:19.305 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:13:19.305 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-4.18.0-516.el8.x86_64 kernel-core-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.305 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.305 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:13:19.306 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:13:19.452 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T14:13:19.668 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.669 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-11T14:13:19.669 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-11T14:13:19.669 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-11T14:13:19.669 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-11T14:13:19.670 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-10-11T14:13:19.670 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2023-10-11T14:13:19.670 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T14:13:19.878 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-11T14:13:19.879 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-11T14:13:19.879 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2023-10-11T14:13:19.913 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2023-10-11T14:13:19.913 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2023-10-11T14:13:19.914 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.02 2023-10-11T14:13:19.914 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 129.el8 2023-10-11T14:13:19.914 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2023-10-11T14:13:19.914 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-11T14:13:19.914 INFO:teuthology.orchestra.run.smithi158.stdout:Group : System Environment/Base 2023-10-11T14:13:19.915 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 9265586 2023-10-11T14:13:19.915 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2023-10-11T14:13:19.915 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-11T14:13:19.915 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-11T14:13:19.915 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-11T14:13:19.916 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-11T14:13:19.916 INFO:teuthology.orchestra.run.smithi158.stdout:Relocations : (not relocatable) 2023-10-11T14:13:19.916 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : CentOS BuildSystem 2023-10-11T14:13:19.916 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : CentOS 2023-10-11T14:13:19.916 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-11T14:13:19.917 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2023-10-11T14:13:19.917 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2023-10-11T14:13:19.917 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:13:19.917 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-11T14:13:19.918 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-11T14:13:19.918 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-11T14:13:19.918 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2023-10-11T14:13:19.918 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:13:19.918 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2023-10-11T14:13:19.919 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-11T14:13:19.920 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 4.18.0-516.el8.x86_64 2023-10-11T14:13:19.920 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-11T14:13:20.220 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T14:13:20.440 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-11T14:13:20.441 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-11T14:13:20.441 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-516.el8.x86_64 2023-10-11T14:13:20.441 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-11T14:13:20.441 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-11T14:13:20.442 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-10-11T14:13:20.442 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2023-10-11T14:13:20.442 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T14:13:20.650 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-516.el8.x86_64 2023-10-11T14:13:20.650 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-516.el8.x86_64 2023-10-11T14:13:20.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2023-10-11T14:13:20.684 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2023-10-11T14:13:20.685 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2023-10-11T14:13:20.685 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.02 2023-10-11T14:13:20.686 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 129.el8 2023-10-11T14:13:20.686 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2023-10-11T14:13:20.686 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-11T14:13:20.687 INFO:teuthology.orchestra.run.smithi172.stdout:Group : System Environment/Base 2023-10-11T14:13:20.687 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 9265586 2023-10-11T14:13:20.687 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2023-10-11T14:13:20.688 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-11T14:13:20.688 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-11T14:13:20.688 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-11T14:13:20.689 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-11T14:13:20.689 INFO:teuthology.orchestra.run.smithi172.stdout:Relocations : (not relocatable) 2023-10-11T14:13:20.689 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : CentOS BuildSystem 2023-10-11T14:13:20.690 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2023-10-11T14:13:20.690 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-11T14:13:20.690 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2023-10-11T14:13:20.691 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2023-10-11T14:13:20.691 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:13:20.693 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-11T14:13:20.693 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-11T14:13:20.694 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-11T14:13:20.694 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2023-10-11T14:13:20.695 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:13:20.695 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2023-10-11T14:13:20.697 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-11T14:13:20.697 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 4.18.0-516.el8.x86_64 2023-10-11T14:13:20.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-11T14:13:21.459 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2023-10-11T14:13:22.059 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2023-10-11T14:13:23.731 INFO:teuthology.orchestra.run.smithi158.stderr:done 2023-10-11T14:13:23.733 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2023-10-11T14:13:23.767 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.zmh5o3GwJl 2023-10-11T14:13:23.768 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zmh5o3GwJl 2023-10-11T14:13:23.851 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.zmh5o3GwJl 2023-10-11T14:13:23.993 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.zmh5o3GwJl is 6KB 2023-10-11T14:13:24.015 INFO:teuthology.orchestra.run.smithi172.stderr:done 2023-10-11T14:13:24.016 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-11T14:13:24.043 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.zmh5o3GwJl 2023-10-11T14:13:24.055 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.Y1mAzFdsoO 2023-10-11T14:13:24.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y1mAzFdsoO 2023-10-11T14:13:24.063 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2023-10-11T14:13:24.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.Y1mAzFdsoO 2023-10-11T14:13:24.141 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-11T14:13:24.141 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-11T14:13:24.141 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-11T14:13:24.142 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-11T14:13:24.143 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-11T14:13:24.296 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.Y1mAzFdsoO is 6KB 2023-10-11T14:13:24.316 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2023-10-11T14:13:24.345 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.Y1mAzFdsoO 2023-10-11T14:13:24.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2023-10-11T14:13:24.447 INFO:teuthology.orchestra.run.smithi172.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-11T14:13:24.447 INFO:teuthology.orchestra.run.smithi172.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-11T14:13:24.448 INFO:teuthology.orchestra.run.smithi172.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-11T14:13:24.448 INFO:teuthology.orchestra.run.smithi172.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64.conf 2023-10-11T14:13:24.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-516.el8.x86_64 2023-10-11T14:13:24.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2023-10-11T14:13:54.346 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-10-11T14:13:54.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:13:54.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:13:54.642 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-10-11T14:13:54.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:13:54.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:14:28.987 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-11T14:14:29.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-11T14:14:37.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:14:37.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:14:38.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:14:38.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:14:41.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-11T14:14:53.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:14:53.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:14:56.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-11T14:15:08.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:15:08.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:15:10.106 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-11T14:15:10.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:15:10.529 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-516.el8.x86_64"... 2023-10-11T14:15:10.529 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-11T14:15:10.547 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-516.el8.x86_64 2023-10-11T14:15:10.547 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-11T14:15:10.547 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-11T14:15:10.548 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-11T14:15:11.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-11T14:15:11.549 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-10-11T14:15:11.549 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-11T14:15:11.695 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-10-11T14:15:11.704 DEBUG:teuthology.parallel:result is None 2023-10-11T14:15:26.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:15:26.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:15:26.853 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-10-11T14:15:27.112 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:15:27.112 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-516.el8.x86_64"... 2023-10-11T14:15:27.112 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-10-11T14:15:27.130 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-516.el8.x86_64 2023-10-11T14:15:27.131 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-516.el8.x86_64 vs 4.18.0-516.el8.x86_64 2023-10-11T14:15:27.131 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-11T14:15:27.131 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-11T14:15:28.133 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-10-11T14:15:28.133 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-11T14:15:28.222 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2023-10-11T14:15:28.239 DEBUG:teuthology.parallel:result is None 2023-10-11T14:15:28.239 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-11T14:15:28.246 INFO:teuthology.task.internal:Creating test directory... 2023-10-11T14:15:28.247 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T14:15:28.249 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T14:15:28.275 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-11T14:15:28.290 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-11T14:15:28.360 INFO:teuthology.task.internal:Creating archive directory... 2023-10-11T14:15:28.360 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T14:15:28.363 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T14:15:28.402 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-11T14:15:28.409 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-11T14:15:28.410 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T14:15:28.433 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T14:15:28.471 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T14:15:28.478 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T14:15:28.483 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T14:15:28.492 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T14:15:28.493 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-11T14:15:28.500 INFO:teuthology.task.internal:Configuring sudo... 2023-10-11T14:15:28.500 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T14:15:28.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T14:15:28.564 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-11T14:15:28.578 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-11T14:15:28.579 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T14:15:28.590 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T14:15:28.624 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T14:15:28.691 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T14:15:28.756 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T14:15:28.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T14:15:28.841 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:15:28.842 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T14:15:28.907 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T14:15:28.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T14:15:29.011 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T14:15:29.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T14:15:29.141 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:15:29.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T14:15:29.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2023-10-11T14:15:29.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-10-11T14:15:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T14:15:29.370 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T14:15:29.817 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-11T14:15:29.829 INFO:teuthology.task.internal:Starting timer... 2023-10-11T14:15:29.829 INFO:teuthology.run_tasks:Running task pcp... 2023-10-11T14:15:29.841 INFO:teuthology.run_tasks:Running task selinux... 2023-10-11T14:15:29.861 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-11T14:15:29.862 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2023-10-11T14:15:30.646 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2023-10-11T14:15:30.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-10-11T14:15:30.765 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2023-10-11T14:15:30.770 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-11T14:15:30.770 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2023-10-11T14:15:30.927 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2023-10-11T14:15:30.927 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-10-11T14:15:30.954 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-10-11T14:15:30.955 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2023-10-11T14:15:30.955 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T14:15:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T14:15:30.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T14:15:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T14:15:31.032 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-11T14:15:31.032 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2023-10-11T14:15:31.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-10-11T14:15:31.100 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-11T14:15:31.117 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-11T14:15:31.118 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-11T14:15:31.147 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-11T14:15:31.152 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi158.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-11T14:24:32.521 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi158.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-10-11T14:24:32.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:24:32.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:24:32.592 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-10-11T14:24:32.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-11T14:24:32.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:24:32.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T14:24:32.732 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-11T14:24:32.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-11T14:24:32.803 INFO:teuthology.run_tasks:Running task clock... 2023-10-11T14:24:32.816 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-11T14:24:32.817 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T14:24:32.818 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T14:24:32.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T14:24:32.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T14:24:32.849 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T14:24:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T14:24:32.875 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T14:24:32.890 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T14:24:32.923 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2023-10-11T14:24:32.934 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2023-10-11T14:24:32.948 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T14:24:32.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2023-10-11T14:24:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T14:24:32.972 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2023-10-11T14:24:32.987 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T14:24:33.000 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T14:24:33.033 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-10-11T14:24:33.038 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T14:24:33.038 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-10-11T14:24:33.038 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.039 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.039 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.039 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.067 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-11T14:24:33.071 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T14:24:33.071 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-11T14:24:33.072 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.072 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.072 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.072 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T14:24:33.073 INFO:teuthology.run_tasks:Running task install... 2023-10-11T14:24:33.082 DEBUG:teuthology.task.install:project ceph 2023-10-11T14:24:33.082 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}} 2023-10-11T14:24:33.083 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'} 2023-10-11T14:24:33.083 INFO:teuthology.task.install:Using flavor: default 2023-10-11T14:24:33.089 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-11T14:24:33.089 INFO:teuthology.task.install:extra packages: [] 2023-10-11T14:24:33.090 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'tag': None, 'wait_for_package': False} 2023-10-11T14:24:33.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-11T14:24:33.092 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'tag': None, 'wait_for_package': False} 2023-10-11T14:24:33.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-11T14:24:33.325 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/ 2023-10-11T14:24:33.325 INFO:teuthology.task.install.rpm:Package version is 18.2.0-666.gaa3250ce 2023-10-11T14:24:33.349 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/ 2023-10-11T14:24:33.349 INFO:teuthology.task.install.rpm:Package version is 18.2.0-666.gaa3250ce 2023-10-11T14:24:33.634 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T14:24:33.634 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:24:33.635 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T14:24:33.666 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-11T14:24:33.666 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T14:24:33.737 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T14:24:33.744 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-10-0840-reef/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T14:24:33.745 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:24:33.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T14:24:33.779 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-11T14:24:33.779 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/aa3250ced88a675b95b6b3e9bb6e205f0186ed22/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T14:24:33.818 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T14:24:33.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T14:24:33.886 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2023-10-11T14:24:33.888 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2023-10-11T14:24:33.933 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T14:24:34.003 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2023-10-11T14:24:34.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2023-10-11T14:24:34.157 INFO:teuthology.orchestra.run.smithi158.stdout:76 files removed 2023-10-11T14:24:34.184 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw 2023-10-11T14:24:34.289 INFO:teuthology.orchestra.run.smithi172.stdout:76 files removed 2023-10-11T14:24:34.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2023-10-11T14:24:35.375 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 103 kB/s | 80 kB 00:00 2023-10-11T14:24:35.651 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 87 kB/s | 80 kB 00:00 2023-10-11T14:24:36.117 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-10-11T14:24:36.426 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-10-11T14:24:36.859 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 2.1 kB/s | 1.5 kB 00:00 2023-10-11T14:24:37.227 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2023-10-11T14:24:40.593 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 9.0 MB/s | 33 MB 00:03 2023-10-11T14:24:40.827 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - AppStream 9.4 MB/s | 33 MB 00:03 2023-10-11T14:24:48.404 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 49 MB 00:01 2023-10-11T14:24:48.619 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - BaseOS 30 MB/s | 49 MB 00:01 2023-10-11T14:24:55.854 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-10-11T14:24:56.253 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2023-10-11T14:24:56.563 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-10-11T14:24:56.870 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-10-11T14:24:58.031 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - PowerTools 5.0 MB/s | 6.5 MB 00:01 2023-10-11T14:24:58.396 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 6.5 MB 00:01 2023-10-11T14:24:59.490 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-10-11T14:24:59.797 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-10-11T14:25:00.808 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-11T14:25:03.926 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 16 MB 00:03 2023-10-11T14:25:04.651 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 325 kB/s | 24 kB 00:00 2023-10-11T14:25:07.913 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 292 kB/s | 24 kB 00:00 2023-10-11T14:25:09.076 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:25:09.079 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:25:09.079 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:25:09.079 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 13 M 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.7 M 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.6 M 2023-10-11T14:25:09.080 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:25:09.081 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 5.4 M 2023-10-11T14:25:09.081 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 22 M 2023-10-11T14:25:09.081 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 24 k 2023-10-11T14:25:09.081 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T14:25:09.081 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 770 k 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 474 k 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.5 M 2023-10-11T14:25:09.082 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 45 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 123 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 190 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-11T14:25:09.083 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 371 k 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 364 k 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 107 k 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:25:09.084 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:25:09.085 INFO:teuthology.orchestra.run.smithi158.stdout:Install 21 Packages 2023-10-11T14:25:09.085 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 2 Packages 2023-10-11T14:25:09.085 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:25:09.085 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 59 M 2023-10-11T14:25:09.086 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:25:12.308 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:25:12.311 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:25:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:25:12.311 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:25:12.311 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:25:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 13 M 2023-10-11T14:25:12.312 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2023-10-11T14:25:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.7 M 2023-10-11T14:25:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.6 M 2023-10-11T14:25:12.312 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:25:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 5.4 M 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 22 M 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 24 k 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 770 k 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T14:25:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-11T14:25:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 474 k 2023-10-11T14:25:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-11T14:25:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.5 M 2023-10-11T14:25:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T14:25:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-11T14:25:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 45 k 2023-10-11T14:25:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 123 k 2023-10-11T14:25:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 190 k 2023-10-11T14:25:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 371 k 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 364 k 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 107 k 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:25:12.316 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:25:12.317 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:25:12.317 INFO:teuthology.orchestra.run.smithi172.stdout:Install 21 Packages 2023-10-11T14:25:12.317 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2023-10-11T14:25:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:25:12.318 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 59 M 2023-10-11T14:25:12.318 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:25:15.374 INFO:teuthology.orchestra.run.smithi172.stdout:(1/23): ceph-base-18.2.0-666.gaa3250ce.el8.x86_ 2.0 MB/s | 5.4 MB 00:02 2023-10-11T14:25:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:(2/23): ceph-selinux-18.2.0-666.gaa3250ce.el8.x 266 kB/s | 24 kB 00:00 2023-10-11T14:25:15.715 INFO:teuthology.orchestra.run.smithi172.stdout:(3/23): libcephfs2-18.2.0-666.gaa3250ce.el8.x86 3.0 MB/s | 770 kB 00:00 2023-10-11T14:25:15.883 INFO:teuthology.orchestra.run.smithi172.stdout:(4/23): libradosstriper1-18.2.0-666.gaa3250ce.e 2.8 MB/s | 474 kB 00:00 2023-10-11T14:25:17.458 INFO:teuthology.orchestra.run.smithi172.stdout:(5/23): librgw2-18.2.0-666.gaa3250ce.el8.x86_64 4.2 MB/s | 6.5 MB 00:01 2023-10-11T14:25:17.533 INFO:teuthology.orchestra.run.smithi172.stdout:(6/23): python3-ceph-argparse-18.2.0-666.gaa325 596 kB/s | 45 kB 00:00 2023-10-11T14:25:17.616 INFO:teuthology.orchestra.run.smithi172.stdout:(7/23): python3-ceph-common-18.2.0-666.gaa3250c 1.4 MB/s | 123 kB 00:00 2023-10-11T14:25:17.716 INFO:teuthology.orchestra.run.smithi172.stdout:(8/23): python3-cephfs-18.2.0-666.gaa3250ce.el8 1.9 MB/s | 190 kB 00:00 2023-10-11T14:25:17.892 INFO:teuthology.orchestra.run.smithi172.stdout:(9/23): python3-rados-18.2.0-666.gaa3250ce.el8. 2.1 MB/s | 371 kB 00:00 2023-10-11T14:25:18.025 INFO:teuthology.orchestra.run.smithi158.stdout:(1/23): ceph-base-18.2.0-666.gaa3250ce.el8.x86_ 1.3 MB/s | 5.4 MB 00:04 2023-10-11T14:25:18.050 INFO:teuthology.orchestra.run.smithi172.stdout:(10/23): python3-rbd-18.2.0-666.gaa3250ce.el8.x 2.2 MB/s | 364 kB 00:00 2023-10-11T14:25:18.142 INFO:teuthology.orchestra.run.smithi172.stdout:(11/23): python3-rgw-18.2.0-666.gaa3250ce.el8.x 1.1 MB/s | 107 kB 00:00 2023-10-11T14:25:18.151 INFO:teuthology.orchestra.run.smithi158.stdout:(2/23): ceph-selinux-18.2.0-666.gaa3250ce.el8.x 194 kB/s | 24 kB 00:00 2023-10-11T14:25:18.455 INFO:teuthology.orchestra.run.smithi158.stdout:(3/23): libcephfs2-18.2.0-666.gaa3250ce.el8.x86 2.5 MB/s | 770 kB 00:00 2023-10-11T14:25:18.651 INFO:teuthology.orchestra.run.smithi172.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 696 kB/s | 354 kB 00:00 2023-10-11T14:25:18.701 INFO:teuthology.orchestra.run.smithi158.stdout:(4/23): ceph-radosgw-18.2.0-666.gaa3250ce.el8.x 2.7 MB/s | 13 MB 00:04 2023-10-11T14:25:18.760 INFO:teuthology.orchestra.run.smithi158.stdout:(5/23): libradosstriper1-18.2.0-666.gaa3250ce.e 1.5 MB/s | 474 kB 00:00 2023-10-11T14:25:18.768 INFO:teuthology.orchestra.run.smithi172.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2023-10-11T14:25:18.852 INFO:teuthology.orchestra.run.smithi158.stdout:(6/23): python3-ceph-argparse-18.2.0-666.gaa325 490 kB/s | 45 kB 00:00 2023-10-11T14:25:18.868 INFO:teuthology.orchestra.run.smithi172.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2023-10-11T14:25:18.960 INFO:teuthology.orchestra.run.smithi158.stdout:(7/23): python3-ceph-common-18.2.0-666.gaa3250c 1.1 MB/s | 123 kB 00:00 2023-10-11T14:25:19.077 INFO:teuthology.orchestra.run.smithi158.stdout:(8/23): python3-cephfs-18.2.0-666.gaa3250ce.el8 1.6 MB/s | 190 kB 00:00 2023-10-11T14:25:19.102 INFO:teuthology.orchestra.run.smithi172.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2023-10-11T14:25:19.185 INFO:teuthology.orchestra.run.smithi158.stdout:(9/23): python3-rados-18.2.0-666.gaa3250ce.el8. 3.3 MB/s | 371 kB 00:00 2023-10-11T14:25:19.302 INFO:teuthology.orchestra.run.smithi158.stdout:(10/23): python3-rbd-18.2.0-666.gaa3250ce.el8.x 3.0 MB/s | 364 kB 00:00 2023-10-11T14:25:19.310 INFO:teuthology.orchestra.run.smithi172.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-10-11T14:25:19.353 INFO:teuthology.orchestra.run.smithi172.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-10-11T14:25:19.395 INFO:teuthology.orchestra.run.smithi172.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-10-11T14:25:19.419 INFO:teuthology.orchestra.run.smithi158.stdout:(11/23): python3-rgw-18.2.0-666.gaa3250ce.el8.x 916 kB/s | 107 kB 00:00 2023-10-11T14:25:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:(19/23): ceph-radosgw-18.2.0-666.gaa3250ce.el8. 1.8 MB/s | 13 MB 00:07 2023-10-11T14:25:19.956 INFO:teuthology.orchestra.run.smithi158.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 660 kB/s | 354 kB 00:00 2023-10-11T14:25:20.021 INFO:teuthology.orchestra.run.smithi172.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2023-10-11T14:25:20.252 INFO:teuthology.orchestra.run.smithi158.stdout:(13/23): librgw2-18.2.0-666.gaa3250ce.el8.x86_6 4.2 MB/s | 6.5 MB 00:01 2023-10-11T14:25:20.329 INFO:teuthology.orchestra.run.smithi158.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 697 kB/s | 259 kB 00:00 2023-10-11T14:25:20.554 INFO:teuthology.orchestra.run.smithi158.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 209 kB/s | 47 kB 00:00 2023-10-11T14:25:20.613 INFO:teuthology.orchestra.run.smithi158.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 122 kB/s | 44 kB 00:00 2023-10-11T14:25:20.721 INFO:teuthology.orchestra.run.smithi158.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 543 kB/s | 59 kB 00:00 2023-10-11T14:25:20.772 INFO:teuthology.orchestra.run.smithi172.stdout:(21/23): librados2-18.2.0-666.gaa3250ce.el8.x86 3.6 MB/s | 3.7 MB 00:01 2023-10-11T14:25:20.773 INFO:teuthology.orchestra.run.smithi158.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-10-11T14:25:20.797 INFO:teuthology.orchestra.run.smithi158.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 996 kB/s | 75 kB 00:00 2023-10-11T14:25:20.947 INFO:teuthology.orchestra.run.smithi158.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 9.9 MB/s | 1.7 MB 00:00 2023-10-11T14:25:22.041 INFO:teuthology.orchestra.run.smithi172.stdout:(22/23): librbd1-18.2.0-666.gaa3250ce.el8.x86_6 1.8 MB/s | 3.6 MB 00:02 2023-10-11T14:25:22.114 INFO:teuthology.orchestra.run.smithi158.stdout:(21/23): librados2-18.2.0-666.gaa3250ce.el8.x86 2.8 MB/s | 3.7 MB 00:01 2023-10-11T14:25:22.173 INFO:teuthology.orchestra.run.smithi158.stdout:(22/23): librbd1-18.2.0-666.gaa3250ce.el8.x86_6 2.9 MB/s | 3.6 MB 00:01 2023-10-11T14:25:24.487 INFO:teuthology.orchestra.run.smithi172.stdout:(23/23): ceph-common-18.2.0-666.gaa3250ce.el8.x 1.8 MB/s | 22 MB 00:11 2023-10-11T14:25:24.489 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:25:24.490 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.9 MB/s | 59 MB 00:12 2023-10-11T14:25:24.532 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:25:24.577 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:25:24.577 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:25:24.932 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:25:24.935 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:25:26.235 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:25:26.453 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-11T14:25:26.696 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-11T14:25:26.897 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-11T14:25:27.294 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-11T14:25:27.572 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:25:27.671 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:25:27.953 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-11T14:25:28.172 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:25:28.549 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:25:28.838 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:25:28.935 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:25:28.964 INFO:teuthology.orchestra.run.smithi158.stdout:(23/23): ceph-common-18.2.0-666.gaa3250ce.el8.x 1.4 MB/s | 22 MB 00:15 2023-10-11T14:25:28.966 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:25:28.967 INFO:teuthology.orchestra.run.smithi158.stdout:Total 3.0 MB/s | 59 MB 00:19 2023-10-11T14:25:29.011 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:25:29.057 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:25:29.057 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:25:29.132 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-11T14:25:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-11T14:25:29.347 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-11T14:25:29.406 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:25:29.409 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:25:29.549 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 10/25 2023-10-11T14:25:29.775 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-11T14:25:30.015 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-11T14:25:30.197 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:25:30.266 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:25:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-11T14:25:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:25:30.926 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-11T14:25:31.156 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-10-11T14:25:31.276 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-11T14:25:31.471 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-11T14:25:31.494 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-11T14:25:31.531 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-11T14:25:31.721 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 17/25 2023-10-11T14:25:31.894 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-10-11T14:25:31.897 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-11T14:25:32.080 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 19/25 2023-10-11T14:25:32.127 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:25:32.208 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:25:32.516 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-11T14:25:32.702 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:25:33.099 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:25:33.294 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:25:33.362 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:25:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-11T14:25:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-11T14:25:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-11T14:25:34.004 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 10/25 2023-10-11T14:25:34.246 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-11T14:25:34.478 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-11T14:25:34.619 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:25:34.662 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:25:34.896 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:35.037 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-10-11T14:25:35.521 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-10-11T14:25:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-11T14:25:35.894 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-11T14:25:35.960 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-11T14:25:36.034 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 16/25 2023-10-11T14:25:36.104 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:25:36.243 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 17/25 2023-10-11T14:25:36.709 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-11T14:25:36.902 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 19/25 2023-10-11T14:25:39.509 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:39.760 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 20/25 2023-10-11T14:25:40.652 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-11T14:25:40.763 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 21/25 2023-10-11T14:25:40.976 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:25:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:25:54.775 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2023-10-11T14:25:54.775 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2023-10-11T14:25:54.775 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-11T14:25:54.775 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2023-10-11T14:25:54.775 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2023-10-11T14:25:54.776 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2023-10-11T14:25:54.776 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2023-10-11T14:25:54.776 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2023-10-11T14:25:54.776 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-11T14:25:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:25:56.177 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-11T14:25:56.198 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-11T14:25:56.198 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T14:25:56.444 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T14:25:56.444 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:25:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:25:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 1/25 2023-10-11T14:25:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2/25 2023-10-11T14:25:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 3/25 2023-10-11T14:25:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:25:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-11T14:25:57.147 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:25:57.147 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:25:57.147 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 8/25 2023-10-11T14:25:57.147 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 9/25 2023-10-11T14:25:57.147 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 10/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-11T14:25:57.148 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-11T14:25:57.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:25:57.150 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-11T14:25:57.150 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 24/25 2023-10-11T14:25:58.361 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:25:58.362 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:25:58.362 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2023-10-11T14:25:58.362 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.362 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.362 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T14:25:58.363 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.364 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T14:25:58.364 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-11T14:25:58.364 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.364 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T14:25:58.364 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.365 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T14:25:58.365 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T14:25:58.365 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.365 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.366 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.366 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T14:25:58.367 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.367 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.367 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:25:58.368 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T14:25:58.368 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:25:58.368 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:25:58.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2023-10-11T14:25:59.541 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:52 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:25:59.684 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:25:59.684 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2023-10-11T14:25:59.684 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2023-10-11T14:25:59.685 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-10-11T14:25:59.685 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /run 2023-10-11T14:25:59.685 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2023-10-11T14:25:59.685 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2023-10-11T14:25:59.685 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2023-10-11T14:25:59.686 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2023-10-11T14:25:59.686 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2023-10-11T14:25:59.686 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-11T14:26:01.054 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 23/25 2023-10-11T14:26:01.054 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T14:26:01.140 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:01.142 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:01.142 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:26:01.142 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:01.142 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:26:01.143 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 47 M 2023-10-11T14:26:01.143 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:26:01.143 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-11T14:26:01.143 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 137 k 2023-10-11T14:26:01.143 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-11T14:26:01.144 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-11T14:26:01.144 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-11T14:26:01.144 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:01.144 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:26:01.144 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:01.145 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-10-11T14:26:01.145 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:01.145 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 48 M 2023-10-11T14:26:01.146 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 211 M 2023-10-11T14:26:01.146 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:26:01.258 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-11T14:26:01.259 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:26:01.597 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): libcephsqlite-18.2.0-666.gaa3250ce.el8.x 372 kB/s | 137 kB 00:00 2023-10-11T14:26:01.656 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): jq-1.6-7.el8.x86_64.rpm 474 kB/s | 202 kB 00:00 2023-10-11T14:26:01.848 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2023-10-11T14:26:01.981 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 530 kB/s | 71 kB 00:00 2023-10-11T14:26:02.023 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 440 kB/s | 187 kB 00:00 2023-10-11T14:26:02.030 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:26:02.030 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 1/25 2023-10-11T14:26:02.030 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2/25 2023-10-11T14:26:02.030 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 3/25 2023-10-11T14:26:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 4/25 2023-10-11T14:26:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 5/25 2023-10-11T14:26:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 6/25 2023-10-11T14:26:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 7/25 2023-10-11T14:26:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x 8/25 2023-10-11T14:26:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86 9/25 2023-10-11T14:26:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 10/25 2023-10-11T14:26:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 11/25 2023-10-11T14:26:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 12/25 2023-10-11T14:26:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 13/25 2023-10-11T14:26:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-11T14:26:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-11T14:26:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-11T14:26:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-11T14:26:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-11T14:26:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-11T14:26:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-11T14:26:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-11T14:26:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 22/25 2023-10-11T14:26:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-11T14:26:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 24/25 2023-10-11T14:26:03.276 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-11T14:26:03.276 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:03.276 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2023-10-11T14:26:03.277 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.277 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.277 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:26:03.277 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.277 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T14:26:03.278 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-11T14:26:03.279 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.279 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T14:26:03.279 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.279 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T14:26:03.279 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.280 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.281 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:03.281 INFO:teuthology.orchestra.run.smithi158.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T14:26:03.281 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:03.281 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:03.597 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-test 2023-10-11T14:26:04.457 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:06.063 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:06.064 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:06.064 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:26:06.065 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:06.065 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:26:06.065 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 47 M 2023-10-11T14:26:06.065 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:26:06.065 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-11T14:26:06.066 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 137 k 2023-10-11T14:26:06.066 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-11T14:26:06.066 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-11T14:26:06.066 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-11T14:26:06.066 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:06.067 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:26:06.067 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:06.067 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2023-10-11T14:26:06.067 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:06.068 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 48 M 2023-10-11T14:26:06.068 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 211 M 2023-10-11T14:26:06.068 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:26:06.726 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 478 kB/s | 202 kB 00:00 2023-10-11T14:26:06.843 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-10-11T14:26:06.952 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2023-10-11T14:26:07.077 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 564 kB/s | 71 kB 00:00 2023-10-11T14:26:07.169 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): libcephsqlite-18.2.0-666.gaa3250ce.el8.x 158 kB/s | 137 kB 00:00 2023-10-11T14:26:11.002 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): ceph-test-18.2.0-666.gaa3250ce.el8.x86_6 9.9 MB/s | 47 MB 00:04 2023-10-11T14:26:11.004 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:11.005 INFO:teuthology.orchestra.run.smithi158.stdout:Total 9.6 MB/s | 48 MB 00:04 2023-10-11T14:26:11.049 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:26:11.070 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:26:11.070 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:26:11.414 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:26:11.417 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:26:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:26:12.145 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-11T14:26:12.197 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-test-18.2.0-666.gaa3250ce.el8.x86_6 4.3 MB/s | 47 MB 00:10 2023-10-11T14:26:12.199 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:12.200 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.3 MB/s | 48 MB 00:11 2023-10-11T14:26:12.243 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:26:12.263 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:26:12.264 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:26:12.398 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-11T14:26:12.582 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:26:12.585 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:26:12.587 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-11T14:26:12.666 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-11T14:26:12.875 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-11T14:26:13.029 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-11T14:26:13.066 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:26:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-11T14:26:13.550 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-11T14:26:13.730 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-11T14:26:13.794 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-11T14:26:14.018 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-11T14:26:14.197 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-11T14:26:17.571 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-11T14:26:17.833 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-11T14:26:18.135 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-11T14:26:18.136 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 1/6 2023-10-11T14:26:18.136 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2/6 2023-10-11T14:26:18.136 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-11T14:26:18.136 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-11T14:26:18.137 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-11T14:26:18.727 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 5/6 2023-10-11T14:26:18.834 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-11T14:26:18.834 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:18.834 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:26:18.834 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: jq-1.6-7.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T14:26:18.835 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:18.836 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:19.003 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-11T14:26:19.058 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph 2023-10-11T14:26:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 6/6 2023-10-11T14:26:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 1/6 2023-10-11T14:26:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2/6 2023-10-11T14:26:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-11T14:26:19.289 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-11T14:26:19.290 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-11T14:26:19.930 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:19.965 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-11T14:26:19.965 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:19.965 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:26:19.965 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:19.965 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-7.el8.x86_64 2023-10-11T14:26:19.966 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:19.966 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T14:26:19.966 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T14:26:19.966 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T14:26:19.967 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:19.967 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:20.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2023-10-11T14:26:21.046 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:14 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:21.537 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:21.540 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:21.540 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:26:21.540 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:21.540 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:26:21.540 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.4 k 2023-10-11T14:26:21.541 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:26:21.541 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 2.3 M 2023-10-11T14:26:21.541 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 1.5 M 2023-10-11T14:26:21.541 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 242 k 2023-10-11T14:26:21.541 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 4.6 M 2023-10-11T14:26:21.542 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 18 M 2023-10-11T14:26:21.542 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T14:26:21.542 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T14:26:21.542 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-11T14:26:21.542 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T14:26:21.543 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-11T14:26:21.544 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T14:26:21.545 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T14:26:21.546 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T14:26:21.546 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T14:26:21.546 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-11T14:26:21.546 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T14:26:21.546 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:21.547 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:26:21.547 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:21.547 INFO:teuthology.orchestra.run.smithi158.stdout:Install 32 Packages 2023-10-11T14:26:21.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:21.548 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 30 M 2023-10-11T14:26:21.548 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 119 M 2023-10-11T14:26:21.548 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:26:22.079 INFO:teuthology.orchestra.run.smithi158.stdout:(1/32): ceph-18.2.0-666.gaa3250ce.el8.x86_64.rp 44 kB/s | 6.4 kB 00:00 2023-10-11T14:26:22.387 INFO:teuthology.orchestra.run.smithi158.stdout:(2/32): ceph-mds-18.2.0-666.gaa3250ce.el8.x86_6 5.0 MB/s | 2.3 MB 00:00 2023-10-11T14:26:22.651 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:22.654 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:22.655 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:26:22.655 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:22.655 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:26:22.655 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 6.4 k 2023-10-11T14:26:22.656 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:26:22.656 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 2.3 M 2023-10-11T14:26:22.656 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 1.5 M 2023-10-11T14:26:22.656 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 242 k 2023-10-11T14:26:22.656 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 4.6 M 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 18 M 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T14:26:22.657 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-11T14:26:22.658 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-11T14:26:22.659 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T14:26:22.660 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:26:22.661 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:22.662 INFO:teuthology.orchestra.run.smithi172.stdout:Install 32 Packages 2023-10-11T14:26:22.662 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:22.662 INFO:teuthology.orchestra.run.smithi158.stdout:(3/32): ceph-mon-18.2.0-666.gaa3250ce.el8.x86_6 7.9 MB/s | 4.6 MB 00:00 2023-10-11T14:26:22.663 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 30 M 2023-10-11T14:26:22.663 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 119 M 2023-10-11T14:26:22.663 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:26:22.729 INFO:teuthology.orchestra.run.smithi158.stdout:(4/32): ceph-mgr-modules-core-18.2.0-666.gaa325 3.4 MB/s | 242 kB 00:00 2023-10-11T14:26:23.130 INFO:teuthology.orchestra.run.smithi158.stdout:(5/32): ceph-mgr-18.2.0-666.gaa3250ce.el8.x86_6 1.3 MB/s | 1.5 MB 00:01 2023-10-11T14:26:23.155 INFO:teuthology.orchestra.run.smithi158.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 369 kB/s | 157 kB 00:00 2023-10-11T14:26:23.288 INFO:teuthology.orchestra.run.smithi158.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 775 kB/s | 103 kB 00:00 2023-10-11T14:26:23.481 INFO:teuthology.orchestra.run.smithi158.stdout:(8/32): ceph-osd-18.2.0-666.gaa3250ce.el8.x86_6 16 MB/s | 18 MB 00:01 2023-10-11T14:26:23.506 INFO:teuthology.orchestra.run.smithi158.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 103 kB/s | 39 kB 00:00 2023-10-11T14:26:23.554 INFO:teuthology.orchestra.run.smithi158.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 215 kB/s | 57 kB 00:00 2023-10-11T14:26:23.656 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-18.2.0-666.gaa3250ce.el8.x86_64.rp 23 kB/s | 6.4 kB 00:00 2023-10-11T14:26:23.790 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-mds-18.2.0-666.gaa3250ce.el8.x86_6 5.6 MB/s | 2.3 MB 00:00 2023-10-11T14:26:23.898 INFO:teuthology.orchestra.run.smithi158.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 634 kB/s | 249 kB 00:00 2023-10-11T14:26:23.965 INFO:teuthology.orchestra.run.smithi158.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 1.5 MB/s | 641 kB 00:00 2023-10-11T14:26:23.991 INFO:teuthology.orchestra.run.smithi158.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-10-11T14:26:24.107 INFO:teuthology.orchestra.run.smithi158.stdout:(14/32): python3-werkzeug-0.12.2-4.el8.noarch.r 731 kB/s | 457 kB 00:00 2023-10-11T14:26:24.132 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): ceph-mgr-18.2.0-666.gaa3250ce.el8.x86_6 2.1 MB/s | 1.5 MB 00:00 2023-10-11T14:26:24.152 INFO:teuthology.orchestra.run.smithi158.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 233 kB/s | 44 kB 00:00 2023-10-11T14:26:24.183 INFO:teuthology.orchestra.run.smithi158.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 964 kB/s | 185 kB 00:00 2023-10-11T14:26:24.208 INFO:teuthology.orchestra.run.smithi158.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2023-10-11T14:26:24.233 INFO:teuthology.orchestra.run.smithi158.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-10-11T14:26:24.237 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): ceph-mgr-modules-core-18.2.0-666.gaa325 2.3 MB/s | 242 kB 00:00 2023-10-11T14:26:24.259 INFO:teuthology.orchestra.run.smithi158.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-10-11T14:26:24.284 INFO:teuthology.orchestra.run.smithi158.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 970 kB/s | 49 kB 00:00 2023-10-11T14:26:24.309 INFO:teuthology.orchestra.run.smithi158.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-10-11T14:26:24.334 INFO:teuthology.orchestra.run.smithi158.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2023-10-11T14:26:24.360 INFO:teuthology.orchestra.run.smithi158.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-10-11T14:26:24.385 INFO:teuthology.orchestra.run.smithi158.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-10-11T14:26:24.410 INFO:teuthology.orchestra.run.smithi158.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-10-11T14:26:24.456 INFO:teuthology.orchestra.run.smithi158.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-10-11T14:26:24.486 INFO:teuthology.orchestra.run.smithi158.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-10-11T14:26:24.511 INFO:teuthology.orchestra.run.smithi158.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-10-11T14:26:24.536 INFO:teuthology.orchestra.run.smithi158.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.1 MB/s | 255 kB 00:00 2023-10-11T14:26:24.562 INFO:teuthology.orchestra.run.smithi158.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2023-10-11T14:26:24.587 INFO:teuthology.orchestra.run.smithi158.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-10-11T14:26:24.612 INFO:teuthology.orchestra.run.smithi158.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 849 kB/s | 85 kB 00:00 2023-10-11T14:26:24.614 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:24.614 INFO:teuthology.orchestra.run.smithi158.stdout:Total 9.8 MB/s | 30 MB 00:03 2023-10-11T14:26:24.665 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 367 kB/s | 157 kB 00:00 2023-10-11T14:26:24.697 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:26:24.736 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:26:24.736 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:26:24.782 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): ceph-mon-18.2.0-666.gaa3250ce.el8.x86_6 4.1 MB/s | 4.6 MB 00:01 2023-10-11T14:26:24.799 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 290 kB/s | 39 kB 00:00 2023-10-11T14:26:24.966 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): ceph-osd-18.2.0-666.gaa3250ce.el8.x86_6 15 MB/s | 18 MB 00:01 2023-10-11T14:26:25.023 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 256 kB/s | 57 kB 00:00 2023-10-11T14:26:25.043 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:26:25.046 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:26:25.317 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 194 kB/s | 103 kB 00:00 2023-10-11T14:26:25.359 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 740 kB/s | 249 kB 00:00 2023-10-11T14:26:25.451 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-10-11T14:26:25.568 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): python3-werkzeug-0.12.2-4.el8.noarch.r 761 kB/s | 457 kB 00:00 2023-10-11T14:26:25.659 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 209 kB/s | 44 kB 00:00 2023-10-11T14:26:25.735 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 1.5 MB/s | 641 kB 00:00 2023-10-11T14:26:25.793 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 822 kB/s | 185 kB 00:00 2023-10-11T14:26:25.902 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 712 kB/s | 173 kB 00:00 2023-10-11T14:26:25.928 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-10-11T14:26:25.961 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-10-11T14:26:26.053 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 117 kB/s | 18 kB 00:00 2023-10-11T14:26:26.078 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 324 kB/s | 49 kB 00:00 2023-10-11T14:26:26.104 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 413 kB/s | 59 kB 00:00 2023-10-11T14:26:26.225 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:26:26.237 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 141 kB/s | 19 kB 00:00 2023-10-11T14:26:26.296 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 81 kB/s | 16 kB 00:00 2023-10-11T14:26:26.321 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2023-10-11T14:26:26.363 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 190 kB/s | 24 kB 00:00 2023-10-11T14:26:26.422 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-10-11T14:26:26.447 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2023-10-11T14:26:26.466 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-11T14:26:26.580 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-10-11T14:26:26.606 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.0 MB/s | 255 kB 00:00 2023-10-11T14:26:26.631 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 464 kB/s | 85 kB 00:00 2023-10-11T14:26:26.676 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-11T14:26:26.689 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2023-10-11T14:26:26.692 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:26.692 INFO:teuthology.orchestra.run.smithi172.stdout:Total 7.4 MB/s | 30 MB 00:04 2023-10-11T14:26:26.779 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:26:26.817 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:26:26.817 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:26:26.885 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-11T14:26:27.083 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-11T14:26:27.144 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:26:27.147 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:26:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-11T14:26:27.561 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-11T14:26:27.738 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-11T14:26:27.982 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-11T14:26:28.233 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-11T14:26:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:26:28.494 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-11T14:26:28.633 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-11T14:26:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-11T14:26:28.869 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-11T14:26:28.991 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-11T14:26:29.060 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-11T14:26:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-11T14:26:29.417 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-11T14:26:29.727 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-11T14:26:29.934 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-11T14:26:30.016 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-11T14:26:30.158 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-11T14:26:30.264 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-11T14:26:30.402 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-11T14:26:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-11T14:26:30.734 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-11T14:26:30.762 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-11T14:26:30.960 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-11T14:26:30.998 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-11T14:26:31.188 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-11T14:26:31.237 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-11T14:26:31.427 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-11T14:26:31.464 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-11T14:26:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-11T14:26:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-11T14:26:31.954 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-11T14:26:32.068 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-11T14:26:32.203 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-11T14:26:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-11T14:26:32.569 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-11T14:26:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-11T14:26:32.910 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-11T14:26:32.940 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-11T14:26:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-11T14:26:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 27/32 2023-10-11T14:26:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-11T14:26:33.539 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-11T14:26:33.766 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-11T14:26:34.004 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-11T14:26:34.215 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-11T14:26:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-11T14:26:34.739 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-11T14:26:35.012 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-11T14:26:35.062 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-11T14:26:35.227 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-11T14:26:35.346 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-11T14:26:35.682 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-11T14:26:35.687 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-11T14:26:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-11T14:26:36.074 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-11T14:26:36.082 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 27/32 2023-10-11T14:26:36.250 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-11T14:26:36.269 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-11T14:26:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-11T14:26:36.375 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-11T14:26:36.769 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-11T14:26:36.770 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 1/32 2023-10-11T14:26:36.770 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2/32 2023-10-11T14:26:36.770 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 3/32 2023-10-11T14:26:36.770 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 4/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 5/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 6/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-11T14:26:36.771 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-11T14:26:36.772 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-11T14:26:36.773 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-11T14:26:36.773 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-11T14:26:36.773 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-11T14:26:36.773 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-11T14:26:36.773 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-11T14:26:36.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-11T14:26:36.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-11T14:26:36.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-11T14:26:36.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-11T14:26:36.774 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-11T14:26:36.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-11T14:26:36.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-11T14:26:36.775 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-11T14:26:36.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-11T14:26:36.776 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-11T14:26:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 28/32 2023-10-11T14:26:37.912 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-11T14:26:37.913 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:37.913 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:26:37.913 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:37.913 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:37.913 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:37.914 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:26:37.914 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:37.914 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:37.914 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T14:26:37.914 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T14:26:37.915 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-11T14:26:37.915 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T14:26:37.915 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T14:26:37.915 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-11T14:26:37.915 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T14:26:37.916 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T14:26:37.916 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T14:26:37.916 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-11T14:26:37.916 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T14:26:37.916 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-11T14:26:37.917 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T14:26:37.918 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T14:26:37.919 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T14:26:37.919 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T14:26:37.919 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:37.919 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:38.061 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-11T14:26:38.176 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-base 2023-10-11T14:26:38.520 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 29/32 2023-10-11T14:26:38.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-11T14:26:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 30/32 2023-10-11T14:26:39.033 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:39.101 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-11T14:26:39.119 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 31/32 2023-10-11T14:26:39.259 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-11T14:26:39.666 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 32/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 1/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 3/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 4/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 5/32 2023-10-11T14:26:39.667 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.n 6/32 2023-10-11T14:26:39.668 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-11T14:26:39.668 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-11T14:26:39.668 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-11T14:26:39.668 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-11T14:26:39.668 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-11T14:26:39.669 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-11T14:26:39.670 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-11T14:26:39.670 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-11T14:26:39.670 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-11T14:26:39.670 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-11T14:26:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-11T14:26:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-11T14:26:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-11T14:26:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-11T14:26:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-11T14:26:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-11T14:26:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-11T14:26:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-11T14:26:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-11T14:26:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-11T14:26:40.442 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:26:40.641 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:40.642 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:26:40.643 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:40.745 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-11T14:26:40.746 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:40.746 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:26:40.746 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:40.746 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:40.746 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:40.747 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:26:40.747 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:40.747 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:40.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T14:26:40.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T14:26:40.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-11T14:26:40.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T14:26:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T14:26:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-11T14:26:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T14:26:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T14:26:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T14:26:40.750 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-11T14:26:40.750 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T14:26:40.750 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T14:26:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T14:26:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T14:26:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T14:26:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T14:26:40.751 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-11T14:26:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T14:26:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T14:26:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T14:26:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T14:26:40.752 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:40.753 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:40.761 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install cephadm 2023-10-11T14:26:41.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2023-10-11T14:26:41.615 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:41.949 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:43.243 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:43.243 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:43.243 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:26:43.244 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:43.244 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:26:43.244 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 169 k 2023-10-11T14:26:43.244 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:43.245 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:26:43.245 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:43.245 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:26:43.245 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:43.246 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 169 k 2023-10-11T14:26:43.246 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 671 k 2023-10-11T14:26:43.246 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:26:43.362 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:26:43.460 INFO:teuthology.orchestra.run.smithi158.stdout:cephadm-18.2.0-666.gaa3250ce.el8.noarch.rpm 783 kB/s | 169 kB 00:00 2023-10-11T14:26:43.460 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:43.461 INFO:teuthology.orchestra.run.smithi158.stdout:Total 779 kB/s | 169 kB 00:00 2023-10-11T14:26:43.461 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:26:43.469 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:26:43.469 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:26:43.471 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:26:43.474 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:26:43.519 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:26:43.555 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:43.556 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:26:43.557 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:43.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2023-10-11T14:26:44.069 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:44.204 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:44.432 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:44.516 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:45.050 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:45.051 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:45.051 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:26:45.051 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:26:45.051 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:45.051 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:46.112 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:46.113 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:46.113 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:26:46.113 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:46.114 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:26:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 169 k 2023-10-11T14:26:46.114 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:46.114 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:26:46.115 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:46.115 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:26:46.115 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:46.115 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 169 k 2023-10-11T14:26:46.115 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 671 k 2023-10-11T14:26:46.116 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:26:46.413 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-18.2.0-666.gaa3250ce.el8.noarch.rpm 564 kB/s | 169 kB 00:00 2023-10-11T14:26:46.414 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:46.415 INFO:teuthology.orchestra.run.smithi172.stdout:Total 561 kB/s | 169 kB 00:00 2023-10-11T14:26:46.415 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:26:46.422 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:26:46.422 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:26:46.424 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:26:46.427 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:26:46.475 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:26:47.050 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:47.169 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-immutable-object-cache 2023-10-11T14:26:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:47.448 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:48.045 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:26:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:48.069 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:26:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:26:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:48.070 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:48.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2023-10-11T14:26:49.139 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:49.645 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:49.646 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:49.647 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2023-10-11T14:26:49.647 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:49.647 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:26:49.647 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 165 k 2023-10-11T14:26:49.647 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:49.648 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:26:49.648 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:49.648 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:26:49.648 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:49.648 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 165 k 2023-10-11T14:26:49.649 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 503 k 2023-10-11T14:26:49.649 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:26:49.949 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-immutable-object-cache-18.2.0-666.gaa3250c 549 kB/s | 165 kB 00:00 2023-10-11T14:26:49.949 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:49.949 INFO:teuthology.orchestra.run.smithi158.stdout:Total 547 kB/s | 165 kB 00:00 2023-10-11T14:26:49.950 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:26:49.961 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:26:49.961 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:26:50.007 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:26:50.010 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:26:50.144 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:26:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:50.723 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:50.740 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:50.741 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:50.741 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-11T14:26:50.741 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:50.741 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:26:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 165 k 2023-10-11T14:26:50.742 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:26:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:26:50.743 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 165 k 2023-10-11T14:26:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 503 k 2023-10-11T14:26:50.744 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:26:51.185 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-18.2.0-666.gaa3250c 373 kB/s | 165 kB 00:00 2023-10-11T14:26:51.186 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:51.187 INFO:teuthology.orchestra.run.smithi172.stdout:Total 372 kB/s | 165 kB 00:00 2023-10-11T14:26:51.187 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:26:51.199 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:26:51.199 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:26:51.251 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:26:51.254 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:26:51.405 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:51.405 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:51.405 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:26:51.406 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:51.406 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:51.406 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:51.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr 2023-10-11T14:26:52.125 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:26:52.520 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:52.599 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:53.057 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:53.699 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.e 1/1 2023-10-11T14:26:53.699 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:53.699 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:26:53.699 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:26:53.700 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:53.700 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:53.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2023-10-11T14:26:53.938 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:26:54.132 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:54.133 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:26:54.134 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:26:54.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-dashboard 2023-10-11T14:26:54.789 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:55.105 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:26:56.204 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:26:56.401 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:56.403 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:26:56.403 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:26:56.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2023-10-11T14:26:56.707 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:26:56.708 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:56.709 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:26:56.709 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:56.709 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:26:56.709 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 1.8 M 2023-10-11T14:26:56.710 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:26:56.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 23 k 2023-10-11T14:26:56.710 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 15 k 2023-10-11T14:26:56.710 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-11T14:26:56.710 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2023-10-11T14:26:56.711 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:26:56.712 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 2.1 M 2023-10-11T14:26:56.712 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 8.5 M 2023-10-11T14:26:56.713 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:26:57.367 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:26:57.451 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): ceph-grafana-dashboards-18.2.0-666.gaa32 107 kB/s | 23 kB 00:00 2023-10-11T14:26:57.476 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): ceph-prometheus-alerts-18.2.0-666.gaa325 60 kB/s | 15 kB 00:00 2023-10-11T14:26:57.519 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): ceph-mgr-dashboard-18.2.0-666.gaa3250ce. 6.2 MB/s | 1.8 MB 00:00 2023-10-11T14:26:57.612 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 249 kB/s | 34 kB 00:00 2023-10-11T14:26:57.661 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-10-11T14:26:57.695 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 175 kB/s | 43 kB 00:00 2023-10-11T14:26:57.697 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:26:57.697 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2023-10-11T14:26:57.709 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:26:57.719 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:26:57.719 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:26:57.803 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:26:57.806 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:26:58.151 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:26:58.352 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-11T14:26:58.605 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-11T14:26:58.807 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-11T14:26:58.921 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 4/6 2023-10-11T14:26:58.981 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:26:58.982 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:58.982 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 1.8 M 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 23 k 2023-10-11T14:26:58.983 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 15 k 2023-10-11T14:26:58.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-11T14:26:58.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T14:26:58.984 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T14:26:58.984 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:58.984 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:26:58.985 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:26:58.985 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-10-11T14:26:58.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:26:58.986 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.1 M 2023-10-11T14:26:58.986 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 8.5 M 2023-10-11T14:26:58.986 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:26:59.239 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 5/6 2023-10-11T14:26:59.463 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-11T14:26:59.508 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): ceph-grafana-dashboards-18.2.0-666.gaa32 140 kB/s | 23 kB 00:00 2023-10-11T14:26:59.528 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): ceph-prometheus-alerts-18.2.0-666.gaa325 79 kB/s | 15 kB 00:00 2023-10-11T14:26:59.755 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-11T14:26:59.755 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 1/6 2023-10-11T14:26:59.755 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2/6 2023-10-11T14:26:59.755 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 3/6 2023-10-11T14:26:59.756 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-11T14:26:59.756 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-11T14:26:59.770 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 163 kB/s | 43 kB 00:00 2023-10-11T14:26:59.879 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): ceph-mgr-dashboard-18.2.0-666.gaa3250ce. 3.3 MB/s | 1.8 MB 00:00 2023-10-11T14:26:59.904 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 89 kB/s | 34 kB 00:00 2023-10-11T14:27:00.346 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 341 kB/s | 196 kB 00:00 2023-10-11T14:27:00.349 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:00.349 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.5 MB/s | 2.1 MB 00:01 2023-10-11T14:27:00.359 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:00.368 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:00.368 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:00.388 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-11T14:27:00.389 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:00.389 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:00.389 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:00.389 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:00.390 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:00.390 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T14:27:00.390 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T14:27:00.390 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T14:27:00.390 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:00.391 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:00.443 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:00.447 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:00.520 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-11T14:27:00.737 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:00.953 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-11T14:27:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-11T14:27:01.377 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:01.433 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-11T14:27:01.587 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 4/6 2023-10-11T14:27:01.926 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 5/6 2023-10-11T14:27:02.181 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-11T14:27:02.453 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 6/6 2023-10-11T14:27:02.454 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.n 1/6 2023-10-11T14:27:02.454 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2/6 2023-10-11T14:27:02.454 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.no 3/6 2023-10-11T14:27:02.454 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-11T14:27:02.454 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-11T14:27:02.972 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:02.974 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================================ 2023-10-11T14:27:02.974 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:27:02.974 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================================ 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 7.4 M 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-11T14:27:02.975 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-11T14:27:02.976 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-11T14:27:02.976 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-11T14:27:02.976 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-11T14:27:02.976 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-11T14:27:02.976 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:02.977 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:02.977 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================================ 2023-10-11T14:27:02.977 INFO:teuthology.orchestra.run.smithi158.stdout:Install 8 Packages 2023-10-11T14:27:02.977 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:02.978 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 35 M 2023-10-11T14:27:02.978 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 210 M 2023-10-11T14:27:02.978 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:03.260 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-11T14:27:03.260 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:03.260 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:03.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:03.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:03.261 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:03.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-11T14:27:04.242 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:04.811 INFO:teuthology.orchestra.run.smithi158.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.0-666 12 MB/s | 7.4 MB 00:00 2023-10-11T14:27:05.286 INFO:teuthology.orchestra.run.smithi158.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2023-10-11T14:27:05.361 INFO:teuthology.orchestra.run.smithi158.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2023-10-11T14:27:05.420 INFO:teuthology.orchestra.run.smithi158.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-10-11T14:27:05.812 INFO:teuthology.orchestra.run.smithi158.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.7 MB/s | 3.7 MB 00:00 2023-10-11T14:27:05.837 INFO:teuthology.orchestra.run.smithi158.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2023-10-11T14:27:05.844 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:05.845 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================================ 2023-10-11T14:27:05.845 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:27:05.846 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================================ 2023-10-11T14:27:05.846 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:05.846 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 7.4 M 2023-10-11T14:27:05.846 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:27:05.846 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-10-11T14:27:05.847 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-10-11T14:27:05.847 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-11T14:27:05.847 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-11T14:27:05.847 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-11T14:27:05.847 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-11T14:27:05.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-11T14:27:05.848 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:05.848 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:05.848 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================================ 2023-10-11T14:27:05.848 INFO:teuthology.orchestra.run.smithi172.stdout:Install 8 Packages 2023-10-11T14:27:05.849 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:05.849 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 35 M 2023-10-11T14:27:05.849 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 210 M 2023-10-11T14:27:05.850 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:06.129 INFO:teuthology.orchestra.run.smithi158.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 545 kB/s | 172 kB 00:00 2023-10-11T14:27:06.529 INFO:teuthology.orchestra.run.smithi158.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 12 MB/s | 14 MB 00:01 2023-10-11T14:27:06.530 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:06.530 INFO:teuthology.orchestra.run.smithi158.stdout:Total 10 MB/s | 35 MB 00:03 2023-10-11T14:27:06.768 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:06.787 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:06.787 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:06.835 INFO:teuthology.orchestra.run.smithi172.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.0-666 14 MB/s | 7.4 MB 00:00 2023-10-11T14:27:07.177 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:07.180 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:07.336 INFO:teuthology.orchestra.run.smithi172.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2023-10-11T14:27:07.486 INFO:teuthology.orchestra.run.smithi172.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.1 MB/s | 4.9 MB 00:01 2023-10-11T14:27:07.511 INFO:teuthology.orchestra.run.smithi172.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2023-10-11T14:27:07.742 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-11T14:27:08.071 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-11T14:27:08.194 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-11T14:27:08.422 INFO:teuthology.orchestra.run.smithi172.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 709 kB/s | 645 kB 00:00 2023-10-11T14:27:08.763 INFO:teuthology.orchestra.run.smithi172.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 505 kB/s | 172 kB 00:00 2023-10-11T14:27:08.780 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-11T14:27:08.964 INFO:teuthology.orchestra.run.smithi172.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.8 MB/s | 3.7 MB 00:02 2023-10-11T14:27:09.156 INFO:teuthology.orchestra.run.smithi172.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 8.2 MB/s | 14 MB 00:01 2023-10-11T14:27:09.156 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:09.157 INFO:teuthology.orchestra.run.smithi172.stdout:Total 11 MB/s | 35 MB 00:03 2023-10-11T14:27:09.393 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:09.411 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:09.411 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:09.686 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:09.795 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:09.798 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:10.322 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:10.373 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:10.486 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-11T14:27:10.554 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-11T14:27:10.734 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-10-11T14:27:10.904 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-11T14:27:11.173 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-11T14:27:11.507 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-10-11T14:27:11.539 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T14:27:11.704 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:11.794 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T14:27:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:12.504 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-11T14:27:13.189 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-11T14:27:13.532 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T14:27:13.553 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T14:27:13.811 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T14:27:14.803 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-11T14:27:15.060 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-11T14:27:15.419 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 1/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T14:27:15.420 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-11T14:27:15.421 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-11T14:27:15.576 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-11T14:27:16.114 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-11T14:27:16.115 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:16.115 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:16.115 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:16.115 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-11T14:27:16.116 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-11T14:27:16.116 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-11T14:27:16.116 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-11T14:27:16.116 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T14:27:16.116 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T14:27:16.117 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-11T14:27:16.117 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:16.117 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:16.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-rook 2023-10-11T14:27:16.804 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-11T14:27:17.036 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-11T14:27:17.115 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:17.356 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 8/8 2023-10-11T14:27:17.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce 1/8 2023-10-11T14:27:17.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-11T14:27:17.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-11T14:27:17.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-11T14:27:17.357 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-11T14:27:17.358 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-11T14:27:17.358 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-10-11T14:27:18.107 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-10-11T14:27:18.107 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:18.108 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:18.108 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:18.108 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-10-11T14:27:18.108 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-10-11T14:27:18.108 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:18.109 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:18.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2023-10-11T14:27:18.709 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:18.710 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:18.710 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:27:18.711 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:18.711 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:18.711 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 47 k 2023-10-11T14:27:18.711 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:27:18.711 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T14:27:18.712 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T14:27:18.712 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T14:27:18.712 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-11T14:27:18.712 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-11T14:27:18.712 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T14:27:18.713 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T14:27:18.714 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:18.714 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:18.714 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:18.714 INFO:teuthology.orchestra.run.smithi158.stdout:Install 13 Packages 2023-10-11T14:27:18.714 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:18.715 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 1.9 M 2023-10-11T14:27:18.715 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 25 M 2023-10-11T14:27:18.715 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:19.118 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:19.586 INFO:teuthology.orchestra.run.smithi158.stdout:(1/13): ceph-mgr-rook-18.2.0-666.gaa3250ce.el8. 291 kB/s | 47 kB 00:00 2023-10-11T14:27:19.694 INFO:teuthology.orchestra.run.smithi158.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 98 kB/s | 27 kB 00:00 2023-10-11T14:27:19.719 INFO:teuthology.orchestra.run.smithi158.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 67 kB/s | 20 kB 00:00 2023-10-11T14:27:19.878 INFO:teuthology.orchestra.run.smithi158.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 599 kB/s | 110 kB 00:00 2023-10-11T14:27:20.037 INFO:teuthology.orchestra.run.smithi158.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 489 kB/s | 155 kB 00:00 2023-10-11T14:27:20.062 INFO:teuthology.orchestra.run.smithi158.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 265 kB/s | 126 kB 00:00 2023-10-11T14:27:20.095 INFO:teuthology.orchestra.run.smithi158.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-10-11T14:27:20.137 INFO:teuthology.orchestra.run.smithi158.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 329 kB/s | 33 kB 00:00 2023-10-11T14:27:20.225 INFO:teuthology.orchestra.run.smithi158.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 627 kB/s | 81 kB 00:00 2023-10-11T14:27:20.255 INFO:teuthology.orchestra.run.smithi158.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 85 kB/s | 16 kB 00:00 2023-10-11T14:27:20.363 INFO:teuthology.orchestra.run.smithi158.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 566 kB/s | 61 kB 00:00 2023-10-11T14:27:20.405 INFO:teuthology.orchestra.run.smithi158.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 358 kB/s | 64 kB 00:00 2023-10-11T14:27:20.489 INFO:teuthology.orchestra.run.smithi158.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.3 MB/s | 1.2 MB 00:00 2023-10-11T14:27:20.489 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:20.490 INFO:teuthology.orchestra.run.smithi158.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-10-11T14:27:20.550 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:20.566 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:20.566 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:20.652 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:20.656 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:20.731 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:20.733 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:20.733 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:27:20.733 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:20.733 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:20.734 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 47 k 2023-10-11T14:27:20.734 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:27:20.734 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T14:27:20.734 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T14:27:20.734 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T14:27:20.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-11T14:27:20.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-11T14:27:20.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T14:27:20.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-11T14:27:20.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-11T14:27:20.736 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-11T14:27:20.736 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-11T14:27:20.736 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T14:27:20.736 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T14:27:20.736 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:20.737 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:20.737 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:20.737 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2023-10-11T14:27:20.737 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.9 M 2023-10-11T14:27:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 25 M 2023-10-11T14:27:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:21.159 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:21.352 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): ceph-mgr-rook-18.2.0-666.gaa3250ce.el8. 268 kB/s | 47 kB 00:00 2023-10-11T14:27:21.436 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 102 kB/s | 27 kB 00:00 2023-10-11T14:27:21.469 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-11T14:27:21.499 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 62 kB/s | 20 kB 00:00 2023-10-11T14:27:21.675 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-11T14:27:21.703 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 412 kB/s | 110 kB 00:00 2023-10-11T14:27:21.787 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 291 kB/s | 126 kB 00:00 2023-10-11T14:27:21.812 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 495 kB/s | 155 kB 00:00 2023-10-11T14:27:21.921 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-10-11T14:27:21.946 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2023-10-11T14:27:21.953 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-11T14:27:21.971 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2023-10-11T14:27:22.122 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 428 kB/s | 64 kB 00:00 2023-10-11T14:27:22.155 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 347 kB/s | 81 kB 00:00 2023-10-11T14:27:22.205 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 737 kB/s | 61 kB 00:00 2023-10-11T14:27:22.232 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-11T14:27:22.267 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.6 MB/s | 1.2 MB 00:00 2023-10-11T14:27:22.268 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:22.268 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-10-11T14:27:22.329 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:22.345 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:22.345 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:22.431 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:22.434 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:22.474 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-11T14:27:22.757 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-11T14:27:22.992 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:23.076 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-11T14:27:23.271 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-11T14:27:23.331 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-11T14:27:23.443 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-11T14:27:23.687 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-11T14:27:23.950 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-11T14:27:24.224 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-11T14:27:24.373 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-11T14:27:24.509 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-11T14:27:24.682 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-11T14:27:24.924 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-11T14:27:24.947 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-11T14:27:25.185 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-11T14:27:25.207 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-11T14:27:25.428 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-11T14:27:25.718 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-11T14:27:25.719 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 1/13 2023-10-11T14:27:25.719 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-11T14:27:25.719 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-11T14:27:25.719 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-11T14:27:25.719 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-11T14:27:25.720 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-11T14:27:25.720 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-11T14:27:25.720 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-11T14:27:25.720 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-11T14:27:25.720 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-11T14:27:25.721 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-11T14:27:25.721 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-11T14:27:26.282 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-11T14:27:26.425 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-11T14:27:26.425 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:26.425 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:26.425 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:26.425 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T14:27:26.426 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T14:27:26.426 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T14:27:26.426 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T14:27:26.426 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T14:27:26.426 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T14:27:26.427 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T14:27:26.427 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T14:27:26.427 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T14:27:26.427 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T14:27:26.427 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T14:27:26.428 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T14:27:26.428 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:26.428 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:26.574 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-cephadm 2023-10-11T14:27:26.591 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-11T14:27:26.833 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-11T14:27:27.085 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-11T14:27:27.320 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-11T14:27:27.436 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:23 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:27.645 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 13/13 2023-10-11T14:27:27.645 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 1/13 2023-10-11T14:27:27.646 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-11T14:27:27.646 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-11T14:27:27.646 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-11T14:27:27.646 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-11T14:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-11T14:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-11T14:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-11T14:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-11T14:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-11T14:27:27.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-11T14:27:27.648 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-11T14:27:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-11T14:27:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:28.386 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T14:27:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T14:27:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T14:27:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T14:27:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T14:27:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T14:27:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T14:27:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:28.388 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:28.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2023-10-11T14:27:29.023 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:29.024 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:29.024 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:27:29.024 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:29.025 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:29.025 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 132 k 2023-10-11T14:27:29.025 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:27:29.025 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T14:27:29.025 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-11T14:27:29.026 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-11T14:27:29.026 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T14:27:29.026 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:29.026 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:29.026 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:29.027 INFO:teuthology.orchestra.run.smithi158.stdout:Install 5 Packages 2023-10-11T14:27:29.027 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:29.027 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 6.0 M 2023-10-11T14:27:29.028 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 26 M 2023-10-11T14:27:29.028 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:29.280 INFO:teuthology.orchestra.run.smithi158.stdout:(1/5): ceph-mgr-cephadm-18.2.0-666.gaa3250ce.el 728 kB/s | 132 kB 00:00 2023-10-11T14:27:29.398 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:22 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:29.473 INFO:teuthology.orchestra.run.smithi158.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.4 MB/s | 476 kB 00:00 2023-10-11T14:27:29.539 INFO:teuthology.orchestra.run.smithi158.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 904 kB/s | 60 kB 00:00 2023-10-11T14:27:29.689 INFO:teuthology.orchestra.run.smithi158.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 912 kB/s | 538 kB 00:00 2023-10-11T14:27:30.249 INFO:teuthology.orchestra.run.smithi158.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.2 MB/s | 4.8 MB 00:01 2023-10-11T14:27:30.250 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:30.251 INFO:teuthology.orchestra.run.smithi158.stdout:Total 4.9 MB/s | 6.0 MB 00:01 2023-10-11T14:27:30.304 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:30.316 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:30.317 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:30.426 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:30.429 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:30.762 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:31.008 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:31.009 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:31.009 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:27:31.009 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:31.009 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:31.010 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 132 k 2023-10-11T14:27:31.010 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:27:31.010 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T14:27:31.010 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-11T14:27:31.010 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-11T14:27:31.011 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T14:27:31.011 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:31.011 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:31.011 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:31.011 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2023-10-11T14:27:31.012 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:31.012 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.0 M 2023-10-11T14:27:31.012 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2023-10-11T14:27:31.013 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:31.129 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-11T14:27:31.572 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): ceph-mgr-cephadm-18.2.0-666.gaa3250ce.el 421 kB/s | 132 kB 00:00 2023-10-11T14:27:31.856 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 902 kB/s | 538 kB 00:00 2023-10-11T14:27:32.048 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2023-10-11T14:27:32.115 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 233 kB/s | 60 kB 00:00 2023-10-11T14:27:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-11T14:27:32.273 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.7 MB/s | 4.8 MB 00:01 2023-10-11T14:27:32.274 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:32.274 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.7 MB/s | 6.0 MB 00:01 2023-10-11T14:27:32.329 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:32.341 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:32.341 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:32.445 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:32.448 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:32.493 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-11T14:27:32.761 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:32.763 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-11T14:27:32.965 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-11T14:27:33.099 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-11T14:27:33.241 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-11T14:27:33.241 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/5 2023-10-11T14:27:33.241 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-11T14:27:33.242 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-11T14:27:33.242 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-11T14:27:33.879 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-11T14:27:33.879 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T14:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T14:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T14:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T14:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:34.002 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-fuse 2023-10-11T14:27:34.128 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-11T14:27:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-11T14:27:34.863 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:30 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:34.865 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-11T14:27:35.083 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-11T14:27:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 5/5 2023-10-11T14:27:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 1/5 2023-10-11T14:27:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-11T14:27:35.373 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-11T14:27:35.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-11T14:27:36.019 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-11T14:27:36.020 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:36.020 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:36.020 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:36.020 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T14:27:36.021 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T14:27:36.021 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T14:27:36.021 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T14:27:36.021 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:36.021 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:36.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2023-10-11T14:27:36.458 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:36.459 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:36.459 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:27:36.459 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:36.460 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:36.460 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 847 k 2023-10-11T14:27:36.460 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2023-10-11T14:27:36.460 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-11T14:27:36.460 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:36.461 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:36.461 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:36.461 INFO:teuthology.orchestra.run.smithi158.stdout:Install 2 Packages 2023-10-11T14:27:36.461 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:36.462 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 930 k 2023-10-11T14:27:36.462 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 3.0 M 2023-10-11T14:27:36.462 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:36.790 INFO:teuthology.orchestra.run.smithi158.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 304 kB/s | 83 kB 00:00 2023-10-11T14:27:36.992 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:29 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:37.600 INFO:teuthology.orchestra.run.smithi158.stdout:(2/2): ceph-fuse-18.2.0-666.gaa3250ce.el8.x86_6 782 kB/s | 847 kB 00:01 2023-10-11T14:27:37.601 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:37.601 INFO:teuthology.orchestra.run.smithi158.stdout:Total 815 kB/s | 930 kB 00:01 2023-10-11T14:27:37.614 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:37.637 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:37.638 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:37.730 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:37.734 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:38.208 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-11T14:27:38.423 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-11T14:27:38.591 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:38.592 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:38.592 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:27:38.592 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:38.592 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 847 k 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:38.593 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:38.594 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2023-10-11T14:27:38.594 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:38.594 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 930 k 2023-10-11T14:27:38.595 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 3.0 M 2023-10-11T14:27:38.595 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:38.857 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-11T14:27:38.857 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/2 2023-10-11T14:27:38.914 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 298 kB/s | 83 kB 00:00 2023-10-11T14:27:39.373 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-18.2.0-666.gaa3250ce.el8.x86_6 1.1 MB/s | 847 kB 00:00 2023-10-11T14:27:39.373 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:39.374 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.2 MB/s | 930 kB 00:00 2023-10-11T14:27:39.384 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:39.406 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:39.407 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:39.493 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:39.496 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:39.507 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-11T14:27:39.507 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:39.508 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:39.508 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-11T14:27:39.508 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:39.508 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:39.676 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:39.713 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-volume 2023-10-11T14:27:40.010 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-11T14:27:40.223 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-11T14:27:40.581 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:36 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:40.639 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2/2 2023-10-11T14:27:40.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/2 2023-10-11T14:27:41.300 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-11T14:27:41.300 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:41.300 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:41.300 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-11T14:27:41.301 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:41.301 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:41.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2023-10-11T14:27:42.201 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:42.201 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:42.202 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2023-10-11T14:27:42.202 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:42.202 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:42.202 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 261 k 2023-10-11T14:27:42.202 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 261 k 2023-10-11T14:27:42.203 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 1.3 M 2023-10-11T14:27:42.204 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:42.376 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:35 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:42.580 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-volume-18.2.0-666.gaa3250ce.el8.noarch.rpm 691 kB/s | 261 kB 00:00 2023-10-11T14:27:42.581 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:42.581 INFO:teuthology.orchestra.run.smithi158.stdout:Total 689 kB/s | 261 kB 00:00 2023-10-11T14:27:42.582 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:42.591 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:42.592 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:42.619 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:42.622 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:42.798 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:43.064 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:43.490 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:43.984 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:43.985 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:43.985 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-11T14:27:43.985 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:43.986 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:43.986 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:18.2.0-666.gaa3250ce.el8 ceph-noarch 261 k 2023-10-11T14:27:43.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:43.986 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:43.986 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:43.987 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:27:43.987 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:43.987 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 261 k 2023-10-11T14:27:43.987 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.3 M 2023-10-11T14:27:43.988 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:44.109 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:44.110 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:44.110 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:44.110 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:44.110 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:44.111 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:44.343 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-volume-18.2.0-666.gaa3250ce.el8.noarch.rpm 732 kB/s | 261 kB 00:00 2023-10-11T14:27:44.344 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:44.344 INFO:teuthology.orchestra.run.smithi172.stdout:Total 730 kB/s | 261 kB 00:00 2023-10-11T14:27:44.344 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:44.347 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados-devel 2023-10-11T14:27:44.354 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:44.354 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:44.382 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:44.385 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:44.562 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:45.216 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:41 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:46.188 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:46.796 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:46.797 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:46.797 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2023-10-11T14:27:46.797 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:46.797 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 139 k 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:27:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:46.799 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 139 k 2023-10-11T14:27:46.799 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 513 k 2023-10-11T14:27:46.799 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:47.430 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 1/1 2023-10-11T14:27:47.430 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:47.430 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:47.431 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:18.2.0-666.gaa3250ce.el8.noarch 2023-10-11T14:27:47.431 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:47.431 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:47.573 INFO:teuthology.orchestra.run.smithi158.stdout:librados-devel-18.2.0-666.gaa3250ce.el8.x86_64. 179 kB/s | 139 kB 00:00 2023-10-11T14:27:47.574 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:47.574 INFO:teuthology.orchestra.run.smithi158.stdout:Total 179 kB/s | 139 kB 00:00 2023-10-11T14:27:47.575 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:47.581 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:47.581 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:47.636 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:47.639 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:47.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2023-10-11T14:27:47.793 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:48.062 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:48.357 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:48.532 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:41 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:48.989 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:48.989 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:48.990 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:48.990 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:27:48.990 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:48.990 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:49.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs2 2023-10-11T14:27:50.112 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:46 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:50.150 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:50.151 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:50.151 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-11T14:27:50.151 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:50.151 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:50.151 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 139 k 2023-10-11T14:27:50.152 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:50.152 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:50.152 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:50.152 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:27:50.152 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:50.153 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 139 k 2023-10-11T14:27:50.153 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 513 k 2023-10-11T14:27:50.153 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:50.475 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-18.2.0-666.gaa3250ce.el8.x86_64. 430 kB/s | 139 kB 00:00 2023-10-11T14:27:50.475 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:50.476 INFO:teuthology.orchestra.run.smithi172.stdout:Total 428 kB/s | 139 kB 00:00 2023-10-11T14:27:50.476 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:50.481 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:50.481 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:50.531 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:50.534 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:51.217 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:51.495 INFO:teuthology.orchestra.run.smithi158.stdout:Package libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:27:51.690 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:51.692 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:27:51.692 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:51.794 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs-devel 2023-10-11T14:27:51.842 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:51.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:51.855 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:27:51.856 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:51.856 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:52.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2023-10-11T14:27:52.652 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:48 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:52.934 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:45 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:54.251 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:54.252 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:54.252 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2023-10-11T14:27:54.252 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:54.252 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 30 k 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:27:54.253 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:54.254 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 30 k 2023-10-11T14:27:54.254 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 137 k 2023-10-11T14:27:54.255 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:27:54.343 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:27:54.403 INFO:teuthology.orchestra.run.smithi158.stdout:libcephfs-devel-18.2.0-666.gaa3250ce.el8.x86_64 203 kB/s | 30 kB 00:00 2023-10-11T14:27:54.404 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:54.404 INFO:teuthology.orchestra.run.smithi158.stdout:Total 201 kB/s | 30 kB 00:00 2023-10-11T14:27:54.404 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:27:54.407 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:27:54.408 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:27:54.432 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:27:54.435 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:27:54.526 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:27:54.537 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:54.539 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:27:54.539 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:54.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2023-10-11T14:27:54.727 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:55.024 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:55.487 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:48 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:27:55.609 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:55.610 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:55.610 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:27:55.610 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:27:55.610 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:27:55.610 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:55.761 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados2 2023-10-11T14:27:56.616 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:52 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:27:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:57.083 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-11T14:27:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 30 k 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:27:57.084 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:27:57.085 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:57.085 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 30 k 2023-10-11T14:27:57.085 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 137 k 2023-10-11T14:27:57.086 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:27:57.236 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-18.2.0-666.gaa3250ce.el8.x86_64 200 kB/s | 30 kB 00:00 2023-10-11T14:27:57.237 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:27:57.237 INFO:teuthology.orchestra.run.smithi172.stdout:Total 199 kB/s | 30 kB 00:00 2023-10-11T14:27:57.238 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:27:57.241 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:27:57.241 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:27:57.267 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:27:57.270 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:27:57.333 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:27:58.024 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:27:58.106 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:58.216 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:27:58.218 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:27:58.218 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:27:58.386 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:58.396 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librbd1 2023-10-11T14:27:58.979 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:27:58.979 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:58.979 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:27:58.979 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:27:58.979 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:27:58.980 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:27:59.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2023-10-11T14:27:59.255 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:55 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:27:59.953 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:52 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:00.682 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:00.881 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:00.882 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:28:00.882 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:00.980 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rados 2023-10-11T14:28:01.352 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:01.545 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:01.546 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:01.546 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:01.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2023-10-11T14:28:01.850 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:57 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:02.496 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:55 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:03.261 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:03.475 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:03.476 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:28:03.477 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:03.580 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rgw 2023-10-11T14:28:03.906 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:04.105 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:04.106 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:04.106 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:04.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2023-10-11T14:28:04.438 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:00 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:05.057 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:58 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:05.826 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:06.027 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:06.028 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:28:06.028 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:06.132 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-cephfs 2023-10-11T14:28:06.457 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:06.649 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:06.650 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:06.650 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:06.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2023-10-11T14:28:06.990 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:02 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:07.616 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:00 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:08.407 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:08.604 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:08.606 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:28:08.606 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:08.712 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rbd 2023-10-11T14:28:09.035 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:09.231 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:09.232 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:09.232 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:09.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2023-10-11T14:28:09.583 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:05 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:10.199 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:03 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:10.976 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:11.176 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:11.178 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-11T14:28:11.178 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:11.288 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-fuse 2023-10-11T14:28:11.609 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:11.805 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:11.806 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:11.806 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:11.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2023-10-11T14:28:12.151 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:08 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:12.772 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:05 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:13.759 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:13.760 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:13.760 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2023-10-11T14:28:13.761 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:13.761 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:28:13.761 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 95 k 2023-10-11T14:28:13.761 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:13.761 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:28:13.762 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:13.762 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:28:13.762 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:13.762 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 95 k 2023-10-11T14:28:13.763 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 280 k 2023-10-11T14:28:13.763 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:28:14.196 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:18.2.0-666.gaa3250ce.el8.x86_64 is already installed. 2023-10-11T14:28:14.201 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-fuse-18.2.0-666.gaa3250ce.el8.x86_64.rpm 215 kB/s | 95 kB 00:00 2023-10-11T14:28:14.202 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:14.202 INFO:teuthology.orchestra.run.smithi158.stdout:Total 214 kB/s | 95 kB 00:00 2023-10-11T14:28:14.202 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:28:14.213 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:28:14.213 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:28:14.263 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:28:14.266 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:28:14.376 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:28:14.400 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:14.402 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-11T14:28:14.402 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:14.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2023-10-11T14:28:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:14.958 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:15.376 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:08 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:15.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:15.599 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:15.599 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:28:15.599 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:15.599 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:15.599 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:15.831 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-mirror 2023-10-11T14:28:16.697 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:12 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:16.986 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:16.987 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:16.987 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-10-11T14:28:16.987 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:16.987 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 95 k 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:28:16.988 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:16.989 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 95 k 2023-10-11T14:28:16.989 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 280 k 2023-10-11T14:28:16.989 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:28:17.309 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-18.2.0-666.gaa3250ce.el8.x86_64.rpm 294 kB/s | 95 kB 00:00 2023-10-11T14:28:17.309 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:17.310 INFO:teuthology.orchestra.run.smithi172.stdout:Total 293 kB/s | 95 kB 00:00 2023-10-11T14:28:17.310 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:28:17.321 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:28:17.321 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:28:17.371 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:28:17.374 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:28:17.482 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:28:17.725 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:18.010 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:18.306 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:18.307 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:18.307 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2023-10-11T14:28:18.307 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:18.307 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:28:18.308 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.4 M 2023-10-11T14:28:18.308 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:18.308 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:28:18.308 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:18.308 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:28:18.309 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:18.309 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.4 M 2023-10-11T14:28:18.309 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 16 M 2023-10-11T14:28:18.310 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:28:18.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:18.682 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:18.682 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:28:18.682 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:18.682 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:18.682 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:18.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2023-10-11T14:28:19.447 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-mirror-18.2.0-666.gaa3250ce.el8.x86_64.rpm 3.0 MB/s | 3.4 MB 00:01 2023-10-11T14:28:19.448 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:19.449 INFO:teuthology.orchestra.run.smithi158.stdout:Total 3.0 MB/s | 3.4 MB 00:01 2023-10-11T14:28:19.449 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:28:19.461 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:28:19.462 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:28:19.524 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:28:19.526 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:28:19.798 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:12 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:19.963 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:28:20.158 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:20.581 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:21.212 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:21.397 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:21.398 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:21.398 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-11T14:28:21.398 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:21.398 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:28:21.399 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 3.4 M 2023-10-11T14:28:21.399 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:21.399 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:28:21.399 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:21.399 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:28:21.400 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:21.400 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.4 M 2023-10-11T14:28:21.400 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 16 M 2023-10-11T14:28:21.400 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:28:21.452 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-nbd 2023-10-11T14:28:22.312 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:03:18 ago on Wed 11 Oct 2023 02:25:04 PM UTC. 2023-10-11T14:28:22.864 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-18.2.0-666.gaa3250ce.el8.x86_64.rpm 2.4 MB/s | 3.4 MB 00:01 2023-10-11T14:28:22.864 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:22.865 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.4 MB/s | 3.4 MB 00:01 2023-10-11T14:28:22.865 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:28:22.878 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:28:22.879 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:28:22.945 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:28:22.949 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:28:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:28:23.566 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:23.925 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-11T14:28:23.926 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:23.926 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2023-10-11T14:28:23.926 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:23.926 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 179 k 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2023-10-11T14:28:23.927 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:23.928 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 179 k 2023-10-11T14:28:23.928 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 568 k 2023-10-11T14:28:23.928 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2023-10-11T14:28:23.999 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:24.369 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-nbd-18.2.0-666.gaa3250ce.el8.x86_64.rpm 406 kB/s | 179 kB 00:00 2023-10-11T14:28:24.370 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:24.370 INFO:teuthology.orchestra.run.smithi158.stdout:Total 405 kB/s | 179 kB 00:00 2023-10-11T14:28:24.370 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2023-10-11T14:28:24.382 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2023-10-11T14:28:24.382 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2023-10-11T14:28:24.434 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2023-10-11T14:28:24.437 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2023-10-11T14:28:24.550 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2023-10-11T14:28:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:24.639 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:28:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:24.639 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:24.755 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:24.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2023-10-11T14:28:25.048 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:25.697 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:25.698 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:25.698 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2023-10-11T14:28:25.698 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:25.698 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:25.699 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-11T14:28:25.723 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:18 ago on Wed 11 Oct 2023 02:25:07 PM UTC. 2023-10-11T14:28:25.936 DEBUG:teuthology.parallel:result is None 2023-10-11T14:28:27.331 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-11T14:28:27.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:27.332 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-10-11T14:28:27.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:27.332 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-11T14:28:27.333 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:18.2.0-666.gaa3250ce.el8 ceph 179 k 2023-10-11T14:28:27.333 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:27.333 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-11T14:28:27.333 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-11T14:28:27.333 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-11T14:28:27.334 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:27.334 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 179 k 2023-10-11T14:28:27.334 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 568 k 2023-10-11T14:28:27.334 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-11T14:28:27.530 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-18.2.0-666.gaa3250ce.el8.x86_64.rpm 906 kB/s | 179 kB 00:00 2023-10-11T14:28:27.531 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-11T14:28:27.531 INFO:teuthology.orchestra.run.smithi172.stdout:Total 900 kB/s | 179 kB 00:00 2023-10-11T14:28:27.532 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-11T14:28:27.543 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-11T14:28:27.543 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-11T14:28:27.591 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-11T14:28:27.594 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-11T14:28:27.740 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-11T14:28:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:28.267 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:28.915 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 1/1 2023-10-11T14:28:28.915 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:28.915 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-11T14:28:28.915 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:18.2.0-666.gaa3250ce.el8.x86_64 2023-10-11T14:28:28.916 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:28.916 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-11T14:28:29.127 DEBUG:teuthology.parallel:result is None 2023-10-11T14:28:29.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-11T14:28:29.342 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T14:28:29.370 INFO:teuthology.orchestra.run.smithi158.stdout:18.2.0-666.gaa3250ce.el8 2023-10-11T14:28:29.370 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-666.gaa3250ce.el8 2023-10-11T14:28:29.371 INFO:teuthology.task.install:The correct ceph version 18.2.0-666.gaa3250ce is installed. 2023-10-11T14:28:29.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=aa3250ced88a675b95b6b3e9bb6e205f0186ed22 2023-10-11T14:28:29.519 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T14:28:29.542 INFO:teuthology.orchestra.run.smithi172.stdout:18.2.0-666.gaa3250ce.el8 2023-10-11T14:28:29.542 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-666.gaa3250ce.el8 2023-10-11T14:28:29.543 INFO:teuthology.task.install:The correct ceph version 18.2.0-666.gaa3250ce is installed. 2023-10-11T14:28:29.543 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-11T14:28:29.544 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:29.544 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T14:28:29.579 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:29.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T14:28:29.611 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-11T14:28:29.611 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:29.612 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T14:28:29.651 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T14:28:29.720 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:29.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T14:28:29.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T14:28:29.815 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-11T14:28:29.816 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:29.816 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T14:28:29.846 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T14:28:29.915 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:29.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T14:28:29.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T14:28:30.009 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-11T14:28:30.009 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:30.010 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T14:28:30.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T14:28:30.107 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:30.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T14:28:30.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T14:28:30.201 INFO:teuthology.run_tasks:Running task ceph... 2023-10-11T14:28:30.259 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-11T14:28:30.260 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /var/log/ceph 2023-10-11T14:28:30.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-10-11T14:28:30.289 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-11T14:28:30.289 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-11T14:28:30.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-11T14:28:30.357 INFO:tasks.ceph:Creating extra log directories... 2023-10-11T14:28:30.357 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-11T14:28:30.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-11T14:28:30.434 INFO:tasks.ceph:Creating ceph cluster c1... 2023-10-11T14:28:30.434 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-11T14:28:30.435 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'branch': 'wip-yuri-testing-2023-10-10-0840-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7421167', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-10-0840-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-reef', 'sha1': '4305a5835ec2627a352020742c8983c32bac9341'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1482, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-reef', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '4305a5835ec2627a352020742c8983c32bac9341', 'targets': {'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdZOUR/JCj79beNQ0wybs7M7i43qMxJmr70rOt4AEhBgiCaM4r0c1U41R5LwI8ihJ/8urmC0YXaW4daE5rbCfs=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBaymyTQnulDVyWSaS/bdKw8zsfFDB9kBVESWAo2vuV0yWDK8SWyG3kxrnaNRny+vL4X8wX5KsD37JgctSCCvU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-11_13:59:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-11T14:28:30.435 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-11T14:28:30.465 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-10-11T14:28:30.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T14:28:30.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T14:28:30.550 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:30.550 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T14:28:30.607 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T14:28:30.608 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_1 2023-10-11T14:28:30.665 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T14:28:30.665 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:30.665 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 389681 Links: 1 2023-10-11T14:28:30.665 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:30.666 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:30.666 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-11 14:28:20.334532313 +0000 2023-10-11T14:28:30.666 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-11 14:23:43.002671980 +0000 2023-10-11T14:28:30.666 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-11 14:23:43.002671980 +0000 2023-10-11T14:28:30.667 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-11T14:28:30.667 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T14:28:30.731 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-11T14:28:30.731 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-11T14:28:30.731 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000102377 s, 5.0 MB/s 2023-10-11T14:28:30.732 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T14:28:30.791 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_2 2023-10-11T14:28:30.848 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T14:28:30.849 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:30.849 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 394232 Links: 1 2023-10-11T14:28:30.849 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:30.849 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:30.849 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-11 14:28:20.334532313 +0000 2023-10-11T14:28:30.850 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-11 14:23:43.522666489 +0000 2023-10-11T14:28:30.850 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-11 14:23:43.522666489 +0000 2023-10-11T14:28:30.850 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-11T14:28:30.850 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T14:28:30.914 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-11T14:28:30.915 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-11T14:28:30.915 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000108495 s, 4.7 MB/s 2023-10-11T14:28:30.916 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T14:28:30.974 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_3 2023-10-11T14:28:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T14:28:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:31.033 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 399412 Links: 1 2023-10-11T14:28:31.033 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:31.034 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:31.034 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-11 14:28:20.334532313 +0000 2023-10-11T14:28:31.034 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-11 14:23:44.031661115 +0000 2023-10-11T14:28:31.034 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-11 14:23:44.031661115 +0000 2023-10-11T14:28:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-11T14:28:31.035 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T14:28:31.098 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-11T14:28:31.099 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-11T14:28:31.099 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000110924 s, 4.6 MB/s 2023-10-11T14:28:31.100 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T14:28:31.158 DEBUG:teuthology.orchestra.run.smithi158:> stat /dev/vg_nvme/lv_4 2023-10-11T14:28:31.217 INFO:teuthology.orchestra.run.smithi158.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T14:28:31.217 INFO:teuthology.orchestra.run.smithi158.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:31.217 INFO:teuthology.orchestra.run.smithi158.stdout:Device: 6h/6d Inode: 399517 Links: 1 2023-10-11T14:28:31.218 INFO:teuthology.orchestra.run.smithi158.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:31.218 INFO:teuthology.orchestra.run.smithi158.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:31.218 INFO:teuthology.orchestra.run.smithi158.stdout:Access: 2023-10-11 14:28:20.334532313 +0000 2023-10-11T14:28:31.218 INFO:teuthology.orchestra.run.smithi158.stdout:Modify: 2023-10-11 14:23:44.525655898 +0000 2023-10-11T14:28:31.219 INFO:teuthology.orchestra.run.smithi158.stdout:Change: 2023-10-11 14:23:44.525655898 +0000 2023-10-11T14:28:31.219 INFO:teuthology.orchestra.run.smithi158.stdout: Birth: - 2023-10-11T14:28:31.219 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T14:28:31.283 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records in 2023-10-11T14:28:31.283 INFO:teuthology.orchestra.run.smithi158.stderr:1+0 records out 2023-10-11T14:28:31.283 INFO:teuthology.orchestra.run.smithi158.stderr:512 bytes copied, 0.000101864 s, 5.0 MB/s 2023-10-11T14:28:31.284 DEBUG:teuthology.orchestra.run.smithi158:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T14:28:31.342 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:31.343 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T14:28:31.343 INFO:tasks.ceph:Generating config... 2023-10-11T14:28:31.344 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-11T14:28:31.345 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-11T14:28:31.345 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-11T14:28:31.345 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-11T14:28:31.345 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:31.346 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-11T14:28:31.346 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-11T14:28:31.346 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-11T14:28:31.347 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-11T14:28:31.347 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-11T14:28:31.347 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-11T14:28:31.347 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-11T14:28:31.348 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-11T14:28:31.348 INFO:tasks.ceph:[client] setuser = ceph 2023-10-11T14:28:31.348 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-11T14:28:31.348 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-11T14:28:31.349 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-11T14:28:31.349 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-11T14:28:31.349 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-11T14:28:31.350 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-11T14:28:31.350 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-11T14:28:31.350 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-11T14:28:31.350 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-11T14:28:31.351 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-11T14:28:31.351 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:31.351 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:31.352 INFO:tasks.ceph:Setting up c1.mon.a... 2023-10-11T14:28:31.352 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-10-11T14:28:31.451 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/c1.keyring 2023-10-11T14:28:31.454 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-10-11T14:28:31.499 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-11T14:28:31.526 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.158')] 2023-10-11T14:28:31.526 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.158'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-11T14:28:31.527 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:31.528 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T14:28:31.584 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.158 --print /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:28:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:28:31.667 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:31.667 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: generated fsid abd2bec7-d5d1-4367-b60e-e053f138e4e8 2023-10-11T14:28:31.667 INFO:teuthology.orchestra.run.smithi158.stdout:setting min_mon_release = pacific 2023-10-11T14:28:31.668 INFO:teuthology.orchestra.run.smithi158.stdout:epoch 0 2023-10-11T14:28:31.668 INFO:teuthology.orchestra.run.smithi158.stdout:fsid abd2bec7-d5d1-4367-b60e-e053f138e4e8 2023-10-11T14:28:31.668 INFO:teuthology.orchestra.run.smithi158.stdout:last_changed 2023-10-11T14:28:31.665694+0000 2023-10-11T14:28:31.668 INFO:teuthology.orchestra.run.smithi158.stdout:created 2023-10-11T14:28:31.665694+0000 2023-10-11T14:28:31.668 INFO:teuthology.orchestra.run.smithi158.stdout:min_mon_release 16 (pacific) 2023-10-11T14:28:31.669 INFO:teuthology.orchestra.run.smithi158.stdout:election_strategy: 1 2023-10-11T14:28:31.669 INFO:teuthology.orchestra.run.smithi158.stdout:0: [v2:172.21.15.158:3300/0,v1:172.21.15.158:6789/0] mon.a 2023-10-11T14:28:31.669 INFO:teuthology.orchestra.run.smithi158.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-10-11T14:28:31.670 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T14:28:31.728 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID abd2bec7-d5d1-4367-b60e-e053f138e4e8... 2023-10-11T14:28:31.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-11T14:28:31.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-10-11T14:28:31.813 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2023-10-11T14:28:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2023-10-11T14:28:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T14:28:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2023-10-11T14:28:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2023-10-11T14:28:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2023-10-11T14:28:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2023-10-11T14:28:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2023-10-11T14:28:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2023-10-11T14:28:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2023-10-11T14:28:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T14:28:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2023-10-11T14:28:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T14:28:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T14:28:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2023-10-11T14:28:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2023-10-11T14:28:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2023-10-11T14:28:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2023-10-11T14:28:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2023-10-11T14:28:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2023-10-11T14:28:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2023-10-11T14:28:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2023-10-11T14:28:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T14:28:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T14:28:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2023-10-11T14:28:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T14:28:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2023-10-11T14:28:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2023-10-11T14:28:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2023-10-11T14:28:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2023-10-11T14:28:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2023-10-11T14:28:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = 172.21.15.158 2023-10-11T14:28:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = abd2bec7-d5d1-4367-b60e-e053f138e4e8 2023-10-11T14:28:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.823 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2023-10-11T14:28:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2023-10-11T14:28:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2023-10-11T14:28:31.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2023-10-11T14:28:31.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2023-10-11T14:28:31.824 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2023-10-11T14:28:31.824 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2023-10-11T14:28:31.825 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.825 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2023-10-11T14:28:31.825 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2023-10-11T14:28:31.825 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2023-10-11T14:28:31.826 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.827 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2023-10-11T14:28:31.827 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.827 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2023-10-11T14:28:31.827 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.828 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2023-10-11T14:28:31.828 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2023-10-11T14:28:31.828 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.828 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2023-10-11T14:28:31.828 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2023-10-11T14:28:31.829 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:31.829 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2023-10-11T14:28:31.829 INFO:teuthology.orchestra.run.smithi158.stdout: rocksdb delete range threshold = 0 2023-10-11T14:28:31.829 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.829 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2023-10-11T14:28:31.830 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:31.830 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2023-10-11T14:28:31.830 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-11T14:28:31.830 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-11T14:28:31.830 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:31.831 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:31.831 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2023-10-11T14:28:31.831 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.831 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2023-10-11T14:28:31.831 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2023-10-11T14:28:31.832 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:31.833 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2023-10-11T14:28:31.833 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2023-10-11T14:28:31.833 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:31.833 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.833 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T14:28:31.834 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T14:28:31.834 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T14:28:31.834 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.834 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T14:28:31.834 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T14:28:31.835 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T14:28:31.835 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.835 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2023-10-11T14:28:31.835 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2023-10-11T14:28:31.835 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2023-10-11T14:28:31.836 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2023-10-11T14:28:31.837 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T14:28:31.837 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T14:28:31.837 INFO:teuthology.orchestra.run.smithi158.stdout: debug rgw = 20 2023-10-11T14:28:31.837 INFO:teuthology.orchestra.run.smithi158.stdout: debug rgw sync = 20 2023-10-11T14:28:31.838 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt require ssl = False 2023-10-11T14:28:31.838 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt s3 kms backend = testing 2023-10-11T14:28:31.838 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:31.838 INFO:teuthology.orchestra.run.smithi158.stdout: rgw curl low speed time = 300 2023-10-11T14:28:31.838 INFO:teuthology.orchestra.run.smithi158.stdout: rgw data log num shards = 4 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: rgw md log max shards = 4 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync data inject err probability = 0.1 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync log trim interval = 0 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync meta inject err probability = 0.1 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync obj etag verify = True 2023-10-11T14:28:31.839 INFO:teuthology.orchestra.run.smithi158.stdout: setgroup = ceph 2023-10-11T14:28:31.840 INFO:teuthology.orchestra.run.smithi158.stdout: setuser = ceph 2023-10-11T14:28:31.840 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2023-10-11T14:28:31.840 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-11T14:28:31.840 INFO:teuthology.orchestra.run.smithi158.stdout: rgw data notify interval msec = 0 2023-10-11T14:28:31.840 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2023-10-11T14:28:31.841 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 10 2023-10-11T14:28:31.841 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2023-10-11T14:28:31.841 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:31.841 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2023-10-11T14:28:31.841 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:31.843 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-10-11T14:28:31.843 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-10-11T14:28:31.843 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T14:28:31.843 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-10-11T14:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-10-11T14:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-10-11T14:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-10-11T14:28:31.845 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.846 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T14:28:31.846 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-10-11T14:28:31.846 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.846 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T14:28:31.847 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T14:28:31.847 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.847 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-10-11T14:28:31.847 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.847 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-10-11T14:28:31.848 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-10-11T14:28:31.848 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-10-11T14:28:31.848 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-10-11T14:28:31.848 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-10-11T14:28:31.848 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.849 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T14:28:31.850 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.850 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-10-11T14:28:31.850 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.850 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-10-11T14:28:31.850 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.851 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-10-11T14:28:31.851 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-10-11T14:28:31.851 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-10-11T14:28:31.851 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.158 2023-10-11T14:28:31.851 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = abd2bec7-d5d1-4367-b60e-e053f138e4e8 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-10-11T14:28:31.852 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-10-11T14:28:31.853 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-10-11T14:28:31.853 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.853 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-10-11T14:28:31.853 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-10-11T14:28:31.853 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.854 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-10-11T14:28:31.854 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-10-11T14:28:31.854 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-10-11T14:28:31.854 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.854 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-10-11T14:28:31.855 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.856 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-10-11T14:28:31.856 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.856 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-10-11T14:28:31.856 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-10-11T14:28:31.856 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.857 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-10-11T14:28:31.857 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-10-11T14:28:31.857 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:31.857 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-10-11T14:28:31.857 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2023-10-11T14:28:31.858 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.858 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-10-11T14:28:31.858 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:31.858 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-10-11T14:28:31.858 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.859 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-10-11T14:28:31.860 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.861 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T14:28:31.862 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T14:28:31.862 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T14:28:31.862 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.862 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T14:28:31.862 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.863 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-10-11T14:28:31.864 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:31.864 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-10-11T14:28:31.864 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-10-11T14:28:31.864 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-10-11T14:28:31.864 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-10-11T14:28:31.865 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T14:28:31.865 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T14:28:31.865 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2023-10-11T14:28:31.865 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw sync = 20 2023-10-11T14:28:31.865 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2023-10-11T14:28:31.866 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2023-10-11T14:28:31.866 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:31.866 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2023-10-11T14:28:31.866 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2023-10-11T14:28:31.866 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2023-10-11T14:28:31.867 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync data inject err probability = 0.1 2023-10-11T14:28:31.867 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2023-10-11T14:28:31.867 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync meta inject err probability = 0.1 2023-10-11T14:28:31.867 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2023-10-11T14:28:31.867 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2023-10-11T14:28:31.868 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2023-10-11T14:28:31.868 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-10-11T14:28:31.868 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-11T14:28:31.868 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data notify interval msec = 0 2023-10-11T14:28:31.868 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-11T14:28:31.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2023-10-11T14:28:31.869 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-11T14:28:31.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:31.869 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-11T14:28:31.869 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:31.870 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-10-11T14:28:31.870 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-10-11T14:28:31.941 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-11T14:28:31.941 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:31.941 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-10-11T14:28:31.960 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:31.960 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-10-11T14:28:32.018 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:28:32.018 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:32.019 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/c1.keyring 2023-10-11T14:28:32.019 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-11T14:28:32.094 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:32.094 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:32.152 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:28:32.152 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:32.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c1.keyring 2023-10-11T14:28:32.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-10-11T14:28:32.196 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:32.197 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:32.256 INFO:tasks.ceph:Setting up mon nodes... 2023-10-11T14:28:32.257 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-11T14:28:32.257 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-10-11T14:28:32.310 INFO:teuthology.orchestra.run.smithi158.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-10-11T14:28:32.313 INFO:tasks.ceph:Setting up mds nodes... 2023-10-11T14:28:32.313 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-11T14:28:32.314 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-10-11T14:28:32.357 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/c1.client.0.keyring 2023-10-11T14:28:32.369 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-10-11T14:28:32.413 INFO:teuthology.orchestra.run.smithi158.stdout:creating /etc/ceph/c1.client.1.keyring 2023-10-11T14:28:32.425 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-11T14:28:32.425 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T14:28:32.425 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-10-11T14:28:32.452 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:32.452 INFO:tasks.ceph:role: c1.osd.0 2023-10-11T14:28:32.453 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:28:32.453 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-11T14:28:32.518 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:32.519 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:32.519 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:32.519 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:32.519 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:32.519 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:32.520 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:32.520 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:32.520 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:32.520 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:32.541 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-11T14:28:32.543 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-11T14:28:32.543 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-10-11T14:28:32.624 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-10-11T14:28:32.695 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-10-11T14:28:32.763 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:32.763 INFO:tasks.ceph:role: c1.osd.1 2023-10-11T14:28:32.763 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:28:32.764 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-11T14:28:32.829 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:32.829 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:32.829 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:32.830 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:32.830 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:32.830 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:32.830 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:32.830 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:32.831 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:32.831 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:32.852 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-11T14:28:32.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-11T14:28:32.853 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-10-11T14:28:32.933 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-10-11T14:28:33.003 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-10-11T14:28:33.071 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:33.071 INFO:tasks.ceph:role: c1.osd.2 2023-10-11T14:28:33.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:28:33.072 DEBUG:teuthology.orchestra.run.smithi158:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-11T14:28:33.137 INFO:teuthology.orchestra.run.smithi158.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:33.138 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:33.138 INFO:teuthology.orchestra.run.smithi158.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:33.138 INFO:teuthology.orchestra.run.smithi158.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:33.138 INFO:teuthology.orchestra.run.smithi158.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:33.138 INFO:teuthology.orchestra.run.smithi158.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:33.139 INFO:teuthology.orchestra.run.smithi158.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:33.139 INFO:teuthology.orchestra.run.smithi158.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:33.139 INFO:teuthology.orchestra.run.smithi158.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:33.139 INFO:teuthology.orchestra.run.smithi158.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:33.160 INFO:teuthology.orchestra.run.smithi158.stdout:Discarding blocks...Done. 2023-10-11T14:28:33.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi158.front.sepia.ceph.com -o noatime 2023-10-11T14:28:33.162 DEBUG:teuthology.orchestra.run.smithi158:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-10-11T14:28:33.241 DEBUG:teuthology.orchestra.run.smithi158:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-10-11T14:28:33.312 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:33.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.414+0000 7f2c0c74f640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-10-11T14:28:33.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.414+0000 7f2c0c74f640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-10-11T14:28:33.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.415+0000 7f2c0c74f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-11T14:28:33.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.415+0000 7f2c0c74f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-11T14:28:33.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.415+0000 7f2c0c74f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-10-11T14:28:33.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:33.415+0000 7f2c0c74f640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-10-11T14:28:35.491 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-10-11T14:28:35.525 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:35.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-10-11T14:28:35.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-10-11T14:28:35.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-11T14:28:35.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-11T14:28:35.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-10-11T14:28:35.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:35.617+0000 7f5a20e0d640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-10-11T14:28:37.690 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-10-11T14:28:37.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.814+0000 7f8735123640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-10-11T14:28:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.815+0000 7f8735123640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-10-11T14:28:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.815+0000 7f8735123640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-11T14:28:37.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.815+0000 7f8735123640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-11T14:28:37.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.815+0000 7f8735123640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-10-11T14:28:37.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:28:37.815+0000 7f8735123640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-10-11T14:28:39.896 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-10-11T14:28:39.930 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-11T14:28:39.930 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:39.930 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-10-11T14:28:40.000 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:40.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-10-11T14:28:40.069 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:40.069 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-10-11T14:28:40.136 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:40.137 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-10-11T14:28:40.204 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:40.205 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-10-11T14:28:40.263 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:40.263 DEBUG:teuthology.orchestra.run.smithi158:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-10-11T14:28:40.321 INFO:tasks.ceph:Adding keys to all mons... 2023-10-11T14:28:40.321 DEBUG:teuthology.orchestra.run.smithi158:> sudo tee -a /etc/ceph/c1.keyring 2023-10-11T14:28:40.386 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr.x] 2023-10-11T14:28:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQAQsSZlIuJuEhAAf+eCIfYzLhO9EaRshQdatA== 2023-10-11T14:28:40.387 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2023-10-11T14:28:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQARsSZlzoDCGBAAg50H7dPc+B4aBgRLsrJ6KA== 2023-10-11T14:28:40.387 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQATsSZl9VTQJBAAYHgWFFgSIF4rEHGU31jvKg== 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQAVsSZlFXyaMBAAHr751uNMPJedpLB8M6flJA== 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQAQsSZlJuc6FRAAFl33vjNpItj8HlyNmH2AGA== 2023-10-11T14:28:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:[client.1] 2023-10-11T14:28:40.389 INFO:teuthology.orchestra.run.smithi158.stdout: key = AQAQsSZlNt+IGBAAQUxjjL8v9z5ZlSaDFJlC6A== 2023-10-11T14:28:40.389 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-11T14:28:40.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:28:40.536 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:28:40.590 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:28:40.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T14:28:40.699 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T14:28:40.756 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-11T14:28:40.756 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-10-11T14:28:40.783 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-10-11T14:28:40.881 DEBUG:teuthology.orchestra.run.smithi158:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-10-11T14:28:40.908 DEBUG:teuthology.orchestra.run.smithi158:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-10-11T14:28:40.966 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-10-11T14:28:40.967 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-10-11T14:28:40.968 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-10-11T14:28:41.009 INFO:tasks.ceph.c1.mon.a:Started 2023-10-11T14:28:41.009 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-10-11T14:28:41.010 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-10-11T14:28:41.010 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-10-11T14:28:41.012 INFO:tasks.ceph.c1.mgr.x:Started 2023-10-11T14:28:41.012 INFO:tasks.ceph:Setting crush tunables to default 2023-10-11T14:28:41.013 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd crush tunables default 2023-10-11T14:28:41.230 INFO:teuthology.orchestra.run.smithi158.stderr:adjusted tunables profile to default 2023-10-11T14:28:41.241 INFO:tasks.ceph:check_enable_crimson: False 2023-10-11T14:28:41.241 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-10-11T14:28:41.241 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:41.242 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-10-11T14:28:41.269 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:41.269 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-10-11T14:28:41.336 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:41.336 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-10-11T14:28:41.402 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd new 4f66fc55-af91-43de-b054-1ee35ad5d78f 0 2023-10-11T14:28:41.420 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:41.418+0000 7f7dae702200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T14:28:41.618 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:41.616+0000 7f7dae702200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T14:28:41.660 INFO:teuthology.orchestra.run.smithi158.stdout:0 2023-10-11T14:28:41.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd new 30ad98e4-bf09-4dd2-94ee-1d02f3424c38 1 2023-10-11T14:28:41.885 INFO:teuthology.orchestra.run.smithi158.stdout:1 2023-10-11T14:28:41.898 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd new 56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e 2 2023-10-11T14:28:41.975 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:41.973+0000 7f7dae702200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T14:28:42.114 INFO:teuthology.orchestra.run.smithi158.stdout:2 2023-10-11T14:28:42.127 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-10-11T14:28:42.127 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-10-11T14:28:42.129 INFO:tasks.ceph.c1.osd.0:Started 2023-10-11T14:28:42.130 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-10-11T14:28:42.130 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-10-11T14:28:42.132 INFO:tasks.ceph.c1.osd.1:Started 2023-10-11T14:28:42.132 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-10-11T14:28:42.133 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-10-11T14:28:42.135 INFO:tasks.ceph.c1.osd.2:Started 2023-10-11T14:28:42.135 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-11T14:28:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:42.355 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":5,"fsid":"abd2bec7-d5d1-4367-b60e-e053f138e4e8","created":"2023-10-11T14:28:41.141146+0000","modified":"2023-10-11T14:28:42.111636+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T14:28:42.111636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f66fc55-af91-43de-b054-1ee35ad5d78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30ad98e4-bf09-4dd2-94ee-1d02f3424c38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:28:42.368 INFO:tasks.ceph.ceph_manager.c1:[] 2023-10-11T14:28:42.368 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-11T14:28:42.518 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:42.516+0000 7f7dae702200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T14:28:42.670 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-11T14:28:42.716 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:42.713+0000 7f7dae702200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T14:28:42.746 INFO:tasks.ceph.c1.osd.1.smithi158.stderr:2023-10-11T14:28:42.743+0000 7f07b80ea640 -1 Falling back to public interface 2023-10-11T14:28:42.754 INFO:tasks.ceph.c1.osd.2.smithi158.stderr:2023-10-11T14:28:42.752+0000 7fcc75637640 -1 Falling back to public interface 2023-10-11T14:28:42.758 INFO:tasks.ceph.c1.osd.0.smithi158.stderr:2023-10-11T14:28:42.756+0000 7f08146d4640 -1 Falling back to public interface 2023-10-11T14:28:42.863 INFO:teuthology.misc.health.smithi158.stdout: 2023-10-11T14:28:42.864 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":5,"fsid":"abd2bec7-d5d1-4367-b60e-e053f138e4e8","created":"2023-10-11T14:28:41.141146+0000","modified":"2023-10-11T14:28:42.111636+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T14:28:42.111636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f66fc55-af91-43de-b054-1ee35ad5d78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30ad98e4-bf09-4dd2-94ee-1d02f3424c38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:28:42.875 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T14:28:43.088 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:43.085+0000 7f7dae702200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T14:28:43.771 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:43.769+0000 7f7dae702200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T14:28:43.844 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:43.842+0000 7f7dae702200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T14:28:43.979 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:43.977+0000 7f7dae702200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T14:28:44.048 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.046+0000 7f7dae702200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T14:28:44.233 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.231+0000 7f7dae702200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T14:28:44.311 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.309+0000 7f7dae702200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T14:28:44.389 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.387+0000 7f7dae702200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T14:28:44.482 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.480+0000 7f7dae702200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T14:28:44.665 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.663+0000 7f7dae702200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T14:28:44.748 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:44.746+0000 7f7dae702200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T14:28:45.059 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:45.057+0000 7f7dae702200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T14:28:45.131 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:45.129+0000 7f7dae702200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T14:28:45.459 INFO:tasks.ceph.c1.osd.1.smithi158.stderr:2023-10-11T14:28:45.457+0000 7f07b80ea640 -1 osd.1 0 log_to_monitors true 2023-10-11T14:28:45.502 INFO:tasks.ceph.c1.osd.2.smithi158.stderr:2023-10-11T14:28:45.500+0000 7fcc75637640 -1 osd.2 0 log_to_monitors true 2023-10-11T14:28:45.527 INFO:tasks.ceph.c1.osd.0.smithi158.stderr:2023-10-11T14:28:45.524+0000 7f08146d4640 -1 osd.0 0 log_to_monitors true 2023-10-11T14:28:45.663 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:45.661+0000 7f7dae702200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T14:28:45.984 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:45.982+0000 7f7dae702200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T14:28:46.128 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:46.126+0000 7f7dae702200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T14:28:46.532 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:46.530+0000 7f7dae702200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T14:28:46.724 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:46.722+0000 7f7dae702200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T14:28:46.920 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:46.918+0000 7f7dae702200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T14:28:47.010 INFO:tasks.ceph.c1.osd.2.smithi158.stderr:2023-10-11T14:28:47.008+0000 7fcc699d8700 -1 osd.2 0 waiting for initial osdmap 2023-10-11T14:28:47.022 INFO:tasks.ceph.c1.osd.2.smithi158.stderr:2023-10-11T14:28:47.020+0000 7fcc615b2700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:28:47.032 INFO:tasks.ceph.c1.osd.1.smithi158.stderr:2023-10-11T14:28:47.030+0000 7f07ac48b700 -1 osd.1 0 waiting for initial osdmap 2023-10-11T14:28:47.043 INFO:tasks.ceph.c1.osd.1.smithi158.stderr:2023-10-11T14:28:47.041+0000 7f07a4065700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:28:47.252 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:47.249+0000 7f7dae702200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T14:28:47.837 INFO:tasks.ceph.c1.osd.0.smithi158.stderr:2023-10-11T14:28:47.834+0000 7f0808a75700 -1 osd.0 0 waiting for initial osdmap 2023-10-11T14:28:47.846 INFO:tasks.ceph.c1.osd.0.smithi158.stderr:2023-10-11T14:28:47.844+0000 7f080064f700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:28:48.260 INFO:tasks.ceph.c1.mgr.x.smithi158.stderr:2023-10-11T14:28:48.258+0000 7f7d6fb1b700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-11T14:28:49.178 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-10-11T14:28:49.502 INFO:teuthology.misc.health.smithi158.stdout: 2023-10-11T14:28:49.502 INFO:teuthology.misc.health.smithi158.stdout:{"epoch":9,"fsid":"abd2bec7-d5d1-4367-b60e-e053f138e4e8","created":"2023-10-11T14:28:41.141146+0000","modified":"2023-10-11T14:28:48.835712+0000","last_up_change":"2023-10-11T14:28:48.835712+0000","last_in_change":"2023-10-11T14:28:42.111636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f66fc55-af91-43de-b054-1ee35ad5d78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":114249},{"type":"v1","addr":"172.21.15.158:6817","nonce":114249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":114249},{"type":"v1","addr":"172.21.15.158:6819","nonce":114249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":114249},{"type":"v1","addr":"172.21.15.158:6823","nonce":114249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":114249},{"type":"v1","addr":"172.21.15.158:6821","nonce":114249}]},"public_addr":"172.21.15.158:6817/114249","cluster_addr":"172.21.15.158:6819/114249","heartbeat_back_addr":"172.21.15.158:6823/114249","heartbeat_front_addr":"172.21.15.158:6821/114249","state":["exists","up"]},{"osd":1,"uuid":"30ad98e4-bf09-4dd2-94ee-1d02f3424c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":114248},{"type":"v1","addr":"172.21.15.158:6801","nonce":114248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":114248},{"type":"v1","addr":"172.21.15.158:6803","nonce":114248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":114248},{"type":"v1","addr":"172.21.15.158:6807","nonce":114248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":114248},{"type":"v1","addr":"172.21.15.158:6805","nonce":114248}]},"public_addr":"172.21.15.158:6801/114248","cluster_addr":"172.21.15.158:6803/114248","heartbeat_back_addr":"172.21.15.158:6807/114248","heartbeat_front_addr":"172.21.15.158:6805/114248","state":["exists","up"]},{"osd":2,"uuid":"56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":114258},{"type":"v1","addr":"172.21.15.158:6809","nonce":114258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":114258},{"type":"v1","addr":"172.21.15.158:6811","nonce":114258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":114258},{"type":"v1","addr":"172.21.15.158:6815","nonce":114258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":114258},{"type":"v1","addr":"172.21.15.158:6813","nonce":114258}]},"public_addr":"172.21.15.158:6809/114258","cluster_addr":"172.21.15.158:6811/114258","heartbeat_back_addr":"172.21.15.158:6815/114258","heartbeat_front_addr":"172.21.15.158:6813/114258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:46.483578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:47.009116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:28:49.515 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-11T14:28:49.515 INFO:tasks.ceph:Creating RBD pool 2023-10-11T14:28:49.515 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd pool create rbd 8 2023-10-11T14:28:49.855 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'rbd' created 2023-10-11T14:28:49.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-11T14:28:50.859 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-11T14:28:50.875 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-10-11T14:28:50.875 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-10-11T14:28:50.876 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-10-11T14:28:50.876 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-10-11T14:28:50.877 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-11T14:28:51.294 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:51.307 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6824","nonce":113836},{"type":"v1","addr":"172.21.15.158:6825","nonce":113836}]},"active_addr":"172.21.15.158:6825/113836","active_change":"2023-10-11T14:28:47.251454+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":8593047}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":2278397966}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.158:0","nonce":1283846810}]}]} 2023-10-11T14:28:51.309 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-10-11T14:28:51.310 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-10-11T14:28:51.310 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-11T14:28:51.632 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:51.632 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":11,"fsid":"abd2bec7-d5d1-4367-b60e-e053f138e4e8","created":"2023-10-11T14:28:41.141146+0000","modified":"2023-10-11T14:28:50.847025+0000","last_up_change":"2023-10-11T14:28:48.835712+0000","last_in_change":"2023-10-11T14:28:42.111636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T14:28:49.296650+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T14:28:49.844149+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f66fc55-af91-43de-b054-1ee35ad5d78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":114249},{"type":"v1","addr":"172.21.15.158:6817","nonce":114249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":114249},{"type":"v1","addr":"172.21.15.158:6819","nonce":114249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":114249},{"type":"v1","addr":"172.21.15.158:6823","nonce":114249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":114249},{"type":"v1","addr":"172.21.15.158:6821","nonce":114249}]},"public_addr":"172.21.15.158:6817/114249","cluster_addr":"172.21.15.158:6819/114249","heartbeat_back_addr":"172.21.15.158:6823/114249","heartbeat_front_addr":"172.21.15.158:6821/114249","state":["exists","up"]},{"osd":1,"uuid":"30ad98e4-bf09-4dd2-94ee-1d02f3424c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":114248},{"type":"v1","addr":"172.21.15.158:6801","nonce":114248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":114248},{"type":"v1","addr":"172.21.15.158:6803","nonce":114248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":114248},{"type":"v1","addr":"172.21.15.158:6807","nonce":114248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":114248},{"type":"v1","addr":"172.21.15.158:6805","nonce":114248}]},"public_addr":"172.21.15.158:6801/114248","cluster_addr":"172.21.15.158:6803/114248","heartbeat_back_addr":"172.21.15.158:6807/114248","heartbeat_front_addr":"172.21.15.158:6805/114248","state":["exists","up"]},{"osd":2,"uuid":"56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":114258},{"type":"v1","addr":"172.21.15.158:6809","nonce":114258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":114258},{"type":"v1","addr":"172.21.15.158:6811","nonce":114258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":114258},{"type":"v1","addr":"172.21.15.158:6815","nonce":114258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":114258},{"type":"v1","addr":"172.21.15.158:6813","nonce":114258}]},"public_addr":"172.21.15.158:6809/114258","cluster_addr":"172.21.15.158:6811/114258","heartbeat_back_addr":"172.21.15.158:6815/114258","heartbeat_front_addr":"172.21.15.158:6813/114258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:46.481522+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:46.483578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:47.009116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:28:51.645 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-10-11T14:28:51.646 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-10-11T14:28:51.964 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:51.965 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":12,"fsid":"abd2bec7-d5d1-4367-b60e-e053f138e4e8","created":"2023-10-11T14:28:41.141146+0000","modified":"2023-10-11T14:28:51.848003+0000","last_up_change":"2023-10-11T14:28:48.835712+0000","last_in_change":"2023-10-11T14:28:42.111636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T14:28:49.296650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T14:28:49.844149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4f66fc55-af91-43de-b054-1ee35ad5d78f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6816","nonce":114249},{"type":"v1","addr":"172.21.15.158:6817","nonce":114249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6818","nonce":114249},{"type":"v1","addr":"172.21.15.158:6819","nonce":114249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6822","nonce":114249},{"type":"v1","addr":"172.21.15.158:6823","nonce":114249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6820","nonce":114249},{"type":"v1","addr":"172.21.15.158:6821","nonce":114249}]},"public_addr":"172.21.15.158:6817/114249","cluster_addr":"172.21.15.158:6819/114249","heartbeat_back_addr":"172.21.15.158:6823/114249","heartbeat_front_addr":"172.21.15.158:6821/114249","state":["exists","up"]},{"osd":1,"uuid":"30ad98e4-bf09-4dd2-94ee-1d02f3424c38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6800","nonce":114248},{"type":"v1","addr":"172.21.15.158:6801","nonce":114248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6802","nonce":114248},{"type":"v1","addr":"172.21.15.158:6803","nonce":114248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6806","nonce":114248},{"type":"v1","addr":"172.21.15.158:6807","nonce":114248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6804","nonce":114248},{"type":"v1","addr":"172.21.15.158:6805","nonce":114248}]},"public_addr":"172.21.15.158:6801/114248","cluster_addr":"172.21.15.158:6803/114248","heartbeat_back_addr":"172.21.15.158:6807/114248","heartbeat_front_addr":"172.21.15.158:6805/114248","state":["exists","up"]},{"osd":2,"uuid":"56e0bbd7-3b22-4ce2-a2e5-eaa141eab80e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6808","nonce":114258},{"type":"v1","addr":"172.21.15.158:6809","nonce":114258}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6810","nonce":114258},{"type":"v1","addr":"172.21.15.158:6811","nonce":114258}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6814","nonce":114258},{"type":"v1","addr":"172.21.15.158:6815","nonce":114258}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.158:6812","nonce":114258},{"type":"v1","addr":"172.21.15.158:6813","nonce":114258}]},"public_addr":"172.21.15.158:6809/114258","cluster_addr":"172.21.15.158:6811/114258","heartbeat_back_addr":"172.21.15.158:6815/114258","heartbeat_front_addr":"172.21.15.158:6813/114258","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:46.481522+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:46.483578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:28:47.009116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:28:51.979 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-10-11T14:28:51.979 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-10-11T14:28:51.980 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-10-11T14:28:52.146 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-10-11T14:28:52.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-11T14:28:52.160 INFO:teuthology.orchestra.run.smithi158.stdout:38654705667 2023-10-11T14:28:52.161 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-11T14:28:52.162 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-10-11T14:28:52.162 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-11T14:28:52.494 INFO:teuthology.orchestra.run.smithi158.stdout:34359738369 2023-10-11T14:28:52.508 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2023-10-11T14:28:52.540 INFO:teuthology.orchestra.run.smithi158.stdout:34359738369 2023-10-11T14:28:52.546 INFO:teuthology.orchestra.run.smithi158.stdout:38654705666 2023-10-11T14:28:52.555 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-10-11T14:28:52.560 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2023-10-11T14:28:53.510 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-10-11T14:28:53.556 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-10-11T14:28:53.561 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-10-11T14:28:53.903 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-10-11T14:28:53.904 INFO:teuthology.orchestra.run.smithi158.stdout:34359738370 2023-10-11T14:28:53.923 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2023-10-11T14:28:53.924 DEBUG:teuthology.parallel:result is None 2023-10-11T14:28:53.924 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-10-11T14:28:53.925 DEBUG:teuthology.parallel:result is None 2023-10-11T14:28:53.974 INFO:teuthology.orchestra.run.smithi158.stdout:38654705667 2023-10-11T14:28:53.988 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.0 2023-10-11T14:28:53.988 DEBUG:teuthology.parallel:result is None 2023-10-11T14:28:53.989 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-10-11T14:28:53.989 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-11T14:28:54.303 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:54.305 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-11T14:28:54.316 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-11T14:28:53.255950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82620,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490180,"statfs":{"total":322122547200,"available":322037944320,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856924+0000","last_change":"2023-10-11T14:28:50.854856+0000","last_active":"2023-10-11T14:28:50.856924+0000","last_peered":"2023-10-11T14:28:50.856924+0000","last_clean":"2023-10-11T14:28:50.856924+0000","last_became_active":"2023-10-11T14:28:50.854024+0000","last_became_peered":"2023-10-11T14:28:50.854024+0000","last_unstale":"2023-10-11T14:28:50.856924+0000","last_undegraded":"2023-10-11T14:28:50.856924+0000","last_fullsized":"2023-10-11T14:28:50.856924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:08:22.597994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856891+0000","last_change":"2023-10-11T14:28:50.854857+0000","last_active":"2023-10-11T14:28:50.856891+0000","last_peered":"2023-10-11T14:28:50.856891+0000","last_clean":"2023-10-11T14:28:50.856891+0000","last_became_active":"2023-10-11T14:28:50.853942+0000","last_became_peered":"2023-10-11T14:28:50.853942+0000","last_unstale":"2023-10-11T14:28:50.856891+0000","last_undegraded":"2023-10-11T14:28:50.856891+0000","last_fullsized":"2023-10-11T14:28:50.856891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:02.301792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856856+0000","last_change":"2023-10-11T14:28:50.855394+0000","last_active":"2023-10-11T14:28:50.856856+0000","last_peered":"2023-10-11T14:28:50.856856+0000","last_clean":"2023-10-11T14:28:50.856856+0000","last_became_active":"2023-10-11T14:28:50.854579+0000","last_became_peered":"2023-10-11T14:28:50.854579+0000","last_unstale":"2023-10-11T14:28:50.856856+0000","last_undegraded":"2023-10-11T14:28:50.856856+0000","last_fullsized":"2023-10-11T14:28:50.856856+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:42:39.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.852110+0000","last_change":"2023-10-11T14:28:50.854585+0000","last_active":"2023-10-11T14:28:51.852110+0000","last_peered":"2023-10-11T14:28:51.852110+0000","last_clean":"2023-10-11T14:28:51.852110+0000","last_became_active":"2023-10-11T14:28:50.854348+0000","last_became_peered":"2023-10-11T14:28:50.854348+0000","last_unstale":"2023-10-11T14:28:51.852110+0000","last_undegraded":"2023-10-11T14:28:51.852110+0000","last_fullsized":"2023-10-11T14:28:51.852110+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:19:46.233669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.239697+0000","last_change":"2023-10-11T14:28:50.855224+0000","last_active":"2023-10-11T14:28:51.239697+0000","last_peered":"2023-10-11T14:28:51.239697+0000","last_clean":"2023-10-11T14:28:51.239697+0000","last_became_active":"2023-10-11T14:28:50.854945+0000","last_became_peered":"2023-10-11T14:28:50.854945+0000","last_unstale":"2023-10-11T14:28:51.239697+0000","last_undegraded":"2023-10-11T14:28:51.239697+0000","last_fullsized":"2023-10-11T14:28:51.239697+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:17:56.689811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856817+0000","last_change":"2023-10-11T14:28:50.853800+0000","last_active":"2023-10-11T14:28:50.856817+0000","last_peered":"2023-10-11T14:28:50.856817+0000","last_clean":"2023-10-11T14:28:50.856817+0000","last_became_active":"2023-10-11T14:28:50.853530+0000","last_became_peered":"2023-10-11T14:28:50.853530+0000","last_unstale":"2023-10-11T14:28:50.856817+0000","last_undegraded":"2023-10-11T14:28:50.856817+0000","last_fullsized":"2023-10-11T14:28:50.856817+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:09:33.364917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.852176+0000","last_change":"2023-10-11T14:28:50.854450+0000","last_active":"2023-10-11T14:28:51.852176+0000","last_peered":"2023-10-11T14:28:51.852176+0000","last_clean":"2023-10-11T14:28:51.852176+0000","last_became_active":"2023-10-11T14:28:50.853856+0000","last_became_peered":"2023-10-11T14:28:50.853856+0000","last_unstale":"2023-10-11T14:28:51.852176+0000","last_undegraded":"2023-10-11T14:28:51.852176+0000","last_fullsized":"2023-10-11T14:28:51.852176+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:59:58.684504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.857516+0000","last_change":"2023-10-11T14:28:50.856960+0000","last_active":"2023-10-11T14:28:50.857516+0000","last_peered":"2023-10-11T14:28:50.857516+0000","last_clean":"2023-10-11T14:28:50.857516+0000","last_became_active":"2023-10-11T14:28:50.856699+0000","last_became_peered":"2023-10-11T14:28:50.856699+0000","last_unstale":"2023-10-11T14:28:50.857516+0000","last_undegraded":"2023-10-11T14:28:50.857516+0000","last_fullsized":"2023-10-11T14:28:50.857516+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:22:46.376709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856869+0000","last_change":"2023-10-11T14:28:50.855122+0000","last_active":"2023-10-11T14:28:50.856869+0000","last_peered":"2023-10-11T14:28:50.856869+0000","last_clean":"2023-10-11T14:28:50.856869+0000","last_became_active":"2023-10-11T14:28:50.854034+0000","last_became_peered":"2023-10-11T14:28:50.854034+0000","last_unstale":"2023-10-11T14:28:50.856869+0000","last_undegraded":"2023-10-11T14:28:50.856869+0000","last_fullsized":"2023-10-11T14:28:50.856869+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:32.348371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27816,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829784,"statfs":{"total":107374182400,"available":107345698816,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27756,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829844,"statfs":{"total":107374182400,"available":107345760256,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T14:28:54.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-10-11T14:28:54.631 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:54.634 INFO:teuthology.orchestra.run.smithi158.stderr:dumped all 2023-10-11T14:28:54.645 INFO:teuthology.orchestra.run.smithi158.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-10-11T14:28:53.255950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82620,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490180,"statfs":{"total":322122547200,"available":322037944320,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856924+0000","last_change":"2023-10-11T14:28:50.854856+0000","last_active":"2023-10-11T14:28:50.856924+0000","last_peered":"2023-10-11T14:28:50.856924+0000","last_clean":"2023-10-11T14:28:50.856924+0000","last_became_active":"2023-10-11T14:28:50.854024+0000","last_became_peered":"2023-10-11T14:28:50.854024+0000","last_unstale":"2023-10-11T14:28:50.856924+0000","last_undegraded":"2023-10-11T14:28:50.856924+0000","last_fullsized":"2023-10-11T14:28:50.856924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:08:22.597994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856891+0000","last_change":"2023-10-11T14:28:50.854857+0000","last_active":"2023-10-11T14:28:50.856891+0000","last_peered":"2023-10-11T14:28:50.856891+0000","last_clean":"2023-10-11T14:28:50.856891+0000","last_became_active":"2023-10-11T14:28:50.853942+0000","last_became_peered":"2023-10-11T14:28:50.853942+0000","last_unstale":"2023-10-11T14:28:50.856891+0000","last_undegraded":"2023-10-11T14:28:50.856891+0000","last_fullsized":"2023-10-11T14:28:50.856891+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:02.301792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856856+0000","last_change":"2023-10-11T14:28:50.855394+0000","last_active":"2023-10-11T14:28:50.856856+0000","last_peered":"2023-10-11T14:28:50.856856+0000","last_clean":"2023-10-11T14:28:50.856856+0000","last_became_active":"2023-10-11T14:28:50.854579+0000","last_became_peered":"2023-10-11T14:28:50.854579+0000","last_unstale":"2023-10-11T14:28:50.856856+0000","last_undegraded":"2023-10-11T14:28:50.856856+0000","last_fullsized":"2023-10-11T14:28:50.856856+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:42:39.734780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.852110+0000","last_change":"2023-10-11T14:28:50.854585+0000","last_active":"2023-10-11T14:28:51.852110+0000","last_peered":"2023-10-11T14:28:51.852110+0000","last_clean":"2023-10-11T14:28:51.852110+0000","last_became_active":"2023-10-11T14:28:50.854348+0000","last_became_peered":"2023-10-11T14:28:50.854348+0000","last_unstale":"2023-10-11T14:28:51.852110+0000","last_undegraded":"2023-10-11T14:28:51.852110+0000","last_fullsized":"2023-10-11T14:28:51.852110+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:19:46.233669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.239697+0000","last_change":"2023-10-11T14:28:50.855224+0000","last_active":"2023-10-11T14:28:51.239697+0000","last_peered":"2023-10-11T14:28:51.239697+0000","last_clean":"2023-10-11T14:28:51.239697+0000","last_became_active":"2023-10-11T14:28:50.854945+0000","last_became_peered":"2023-10-11T14:28:50.854945+0000","last_unstale":"2023-10-11T14:28:51.239697+0000","last_undegraded":"2023-10-11T14:28:51.239697+0000","last_fullsized":"2023-10-11T14:28:51.239697+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:17:56.689811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856817+0000","last_change":"2023-10-11T14:28:50.853800+0000","last_active":"2023-10-11T14:28:50.856817+0000","last_peered":"2023-10-11T14:28:50.856817+0000","last_clean":"2023-10-11T14:28:50.856817+0000","last_became_active":"2023-10-11T14:28:50.853530+0000","last_became_peered":"2023-10-11T14:28:50.853530+0000","last_unstale":"2023-10-11T14:28:50.856817+0000","last_undegraded":"2023-10-11T14:28:50.856817+0000","last_fullsized":"2023-10-11T14:28:50.856817+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:09:33.364917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T14:28:51.852176+0000","last_change":"2023-10-11T14:28:50.854450+0000","last_active":"2023-10-11T14:28:51.852176+0000","last_peered":"2023-10-11T14:28:51.852176+0000","last_clean":"2023-10-11T14:28:51.852176+0000","last_became_active":"2023-10-11T14:28:50.853856+0000","last_became_peered":"2023-10-11T14:28:50.853856+0000","last_unstale":"2023-10-11T14:28:51.852176+0000","last_undegraded":"2023-10-11T14:28:51.852176+0000","last_fullsized":"2023-10-11T14:28:51.852176+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:59:58.684504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.857516+0000","last_change":"2023-10-11T14:28:50.856960+0000","last_active":"2023-10-11T14:28:50.857516+0000","last_peered":"2023-10-11T14:28:50.857516+0000","last_clean":"2023-10-11T14:28:50.857516+0000","last_became_active":"2023-10-11T14:28:50.856699+0000","last_became_peered":"2023-10-11T14:28:50.856699+0000","last_unstale":"2023-10-11T14:28:50.857516+0000","last_undegraded":"2023-10-11T14:28:50.857516+0000","last_fullsized":"2023-10-11T14:28:50.857516+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T15:22:46.376709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:28:50.856869+0000","last_change":"2023-10-11T14:28:50.855122+0000","last_active":"2023-10-11T14:28:50.856869+0000","last_peered":"2023-10-11T14:28:50.856869+0000","last_clean":"2023-10-11T14:28:50.856869+0000","last_became_active":"2023-10-11T14:28:50.854034+0000","last_became_peered":"2023-10-11T14:28:50.854034+0000","last_unstale":"2023-10-11T14:28:50.856869+0000","last_undegraded":"2023-10-11T14:28:50.856869+0000","last_fullsized":"2023-10-11T14:28:50.856869+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:28:49.844635+0000","last_clean_scrub_stamp":"2023-10-11T14:28:49.844635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:32.348371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27816,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829784,"statfs":{"total":107374182400,"available":107345698816,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27756,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829844,"statfs":{"total":107374182400,"available":107345760256,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]},{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T14:28:54.646 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-10-11T14:28:54.646 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-10-11T14:28:54.646 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-10-11T14:28:54.647 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-10-11T14:28:55.006 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:55.006 INFO:teuthology.orchestra.run.smithi158.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T14:28:55.019 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-10-11T14:28:55.020 INFO:teuthology.run_tasks:Running task ceph... 2023-10-11T14:28:55.029 INFO:tasks.ceph:Creating ceph cluster c2... 2023-10-11T14:28:55.029 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-11T14:28:55.029 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'branch': 'wip-yuri-testing-2023-10-10-0840-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7421167', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-10-0840-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-qa-63004-reef', 'sha1': '4305a5835ec2627a352020742c8983c32bac9341'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 1482, 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-qa-63004-reef', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': '4305a5835ec2627a352020742c8983c32bac9341', 'targets': {'smithi158.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdZOUR/JCj79beNQ0wybs7M7i43qMxJmr70rOt4AEhBgiCaM4r0c1U41R5LwI8ihJ/8urmC0YXaW4daE5rbCfs=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBaymyTQnulDVyWSaS/bdKw8zsfFDB9kBVESWAo2vuV0yWDK8SWyG3kxrnaNRny+vL4X8wX5KsD37JgctSCCvU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'aa3250ced88a675b95b6b3e9bb6e205f0186ed22', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-11_13:59:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-11T14:28:55.030 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-11T14:28:55.032 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-10-11T14:28:55.055 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T14:28:55.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T14:28:55.126 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:55.127 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T14:28:55.186 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T14:28:55.187 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-10-11T14:28:55.246 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T14:28:55.246 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:55.246 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 396550 Links: 1 2023-10-11T14:28:55.247 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:55.247 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:55.247 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-11 14:28:23.745934716 +0000 2023-10-11T14:28:55.247 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-11 14:23:52.951983457 +0000 2023-10-11T14:28:55.247 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-11 14:23:52.951983457 +0000 2023-10-11T14:28:55.248 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-11T14:28:55.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T14:28:55.314 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-11T14:28:55.314 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-11T14:28:55.315 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000244306 s, 2.1 MB/s 2023-10-11T14:28:55.316 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T14:28:55.376 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-10-11T14:28:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T14:28:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 388027 Links: 1 2023-10-11T14:28:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-11 14:28:23.746934692 +0000 2023-10-11T14:28:55.437 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-11 14:23:53.485970806 +0000 2023-10-11T14:28:55.437 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-11 14:23:53.485970806 +0000 2023-10-11T14:28:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-11T14:28:55.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T14:28:55.502 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-11T14:28:55.502 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-11T14:28:55.503 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000207097 s, 2.5 MB/s 2023-10-11T14:28:55.504 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T14:28:55.563 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-10-11T14:28:55.622 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T14:28:55.622 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:55.622 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 396102 Links: 1 2023-10-11T14:28:55.622 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:55.622 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:55.623 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-11 14:28:23.746934692 +0000 2023-10-11T14:28:55.623 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-11 14:23:54.038957705 +0000 2023-10-11T14:28:55.623 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-11 14:23:54.038957705 +0000 2023-10-11T14:28:55.623 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-11T14:28:55.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T14:28:55.689 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-11T14:28:55.689 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-11T14:28:55.689 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00022247 s, 2.3 MB/s 2023-10-11T14:28:55.690 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T14:28:55.749 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-10-11T14:28:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T14:28:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 391805 Links: 1 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-11 14:28:23.746934692 +0000 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-11 14:23:54.573945033 +0000 2023-10-11T14:28:55.808 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-11 14:23:54.573945033 +0000 2023-10-11T14:28:55.809 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-11T14:28:55.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T14:28:55.873 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-11T14:28:55.873 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-11T14:28:55.874 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000205421 s, 2.5 MB/s 2023-10-11T14:28:55.875 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T14:28:55.933 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:55.933 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T14:28:55.933 INFO:tasks.ceph:Generating config... 2023-10-11T14:28:55.935 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-11T14:28:55.935 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-10-11T14:28:55.935 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-11T14:28:55.936 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-11T14:28:55.936 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:55.936 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-10-11T14:28:55.936 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-10-11T14:28:55.937 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-10-11T14:28:55.937 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-10-11T14:28:55.937 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-10-11T14:28:55.938 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-10-11T14:28:55.938 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-10-11T14:28:55.938 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-11T14:28:55.938 INFO:tasks.ceph:[client] setuser = ceph 2023-10-11T14:28:55.939 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-10-11T14:28:55.939 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-11T14:28:55.939 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-11T14:28:55.939 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-11T14:28:55.940 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-11T14:28:55.940 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-11T14:28:55.940 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-11T14:28:55.940 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-11T14:28:55.941 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-10-11T14:28:55.941 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-10-11T14:28:55.941 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:55.941 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:55.942 INFO:tasks.ceph:Setting up c2.mon.a... 2023-10-11T14:28:55.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-10-11T14:28:56.034 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.keyring 2023-10-11T14:28:56.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-10-11T14:28:56.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-11T14:28:56.108 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.172')] 2023-10-11T14:28:56.108 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-10-11T14:28:56.109 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:56.110 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T14:28:56.166 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:28:56.248 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:28:56.248 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:28:56.249 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:28:56.249 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 6feb2062-e670-4c2d-bd4c-87b41529d0a8 2023-10-11T14:28:56.249 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = pacific 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 6feb2062-e670-4c2d-bd4c-87b41529d0a8 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2023-10-11T14:28:56.247301+0000 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:created 2023-10-11T14:28:56.247301+0000 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 16 (pacific) 2023-10-11T14:28:56.250 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2023-10-11T14:28:56.251 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2023-10-11T14:28:56.251 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-10-11T14:28:56.252 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T14:28:56.310 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 6feb2062-e670-4c2d-bd4c-87b41529d0a8... 2023-10-11T14:28:56.311 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-11T14:28:56.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-10-11T14:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout:[global] 2023-10-11T14:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: chdir = "" 2023-10-11T14:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T14:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: auth supported = cephx 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: filestore xattr use omap = true 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: mon clock drift allowed = 1.000 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: osd crush chooseleaf type = 0 2023-10-11T14:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: auth debug = true 2023-10-11T14:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on old message = true 2023-10-11T14:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: ms die on bug = true 2023-10-11T14:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T14:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: mon pg warn max object skew = 0 2023-10-11T14:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T14:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T14:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default size = 2 2023-10-11T14:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow primary affinity = true 2023-10-11T14:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd allow pg remap = true 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on legacy crush tunables = false 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on crush straw calc version zero = false 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on no sortbitwise = false 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on osd down out interval zero = false 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon warn on too few osds = false 2023-10-11T14:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T14:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T14:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: mon_allow_pool_size_one = true 2023-10-11T14:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T14:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: osd default data pool replay window = 5 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: mon allow pool delete = true 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: mon cluster log file level = debug 2023-10-11T14:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: debug asserts on shutdown = true 2023-10-11T14:28:56.374 INFO:teuthology.orchestra.run.smithi158.stdout: mon health detail to clog = false 2023-10-11T14:28:56.374 INFO:teuthology.orchestra.run.smithi158.stdout: mon host = 172.21.15.172 2023-10-11T14:28:56.374 INFO:teuthology.orchestra.run.smithi158.stdout: fsid = 6feb2062-e670-4c2d-bd4c-87b41529d0a8 2023-10-11T14:28:56.374 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.374 INFO:teuthology.orchestra.run.smithi158.stdout:[osd] 2023-10-11T14:28:56.375 INFO:teuthology.orchestra.run.smithi158.stdout: osd journal size = 100 2023-10-11T14:28:56.375 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.375 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub load threshold = 5.0 2023-10-11T14:28:56.375 INFO:teuthology.orchestra.run.smithi158.stdout: osd scrub max interval = 600 2023-10-11T14:28:56.375 INFO:teuthology.orchestra.run.smithi158.stdout: osd mclock profile = high_recovery_ops 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: osd recover clone overlap = true 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: osd recovery max chunk = 1048576 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug shutdown = true 2023-10-11T14:28:56.376 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug op order = true 2023-10-11T14:28:56.377 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug verify stray on activate = true 2023-10-11T14:28:56.377 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.377 INFO:teuthology.orchestra.run.smithi158.stdout: osd open classes on start = true 2023-10-11T14:28:56.377 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug pg log writeout = true 2023-10-11T14:28:56.377 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.378 INFO:teuthology.orchestra.run.smithi158.stdout: osd deep scrub update digest min age = 30 2023-10-11T14:28:56.378 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.378 INFO:teuthology.orchestra.run.smithi158.stdout: osd map max advance = 10 2023-10-11T14:28:56.378 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.378 INFO:teuthology.orchestra.run.smithi158.stdout: journal zero on create = true 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: filestore ondisk finisher threads = 3 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: filestore apply finisher threads = 3 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: bdev debug aio = true 2023-10-11T14:28:56.379 INFO:teuthology.orchestra.run.smithi158.stdout: osd debug misdirected ops = true 2023-10-11T14:28:56.380 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:56.380 INFO:teuthology.orchestra.run.smithi158.stdout: debug osd = 20 2023-10-11T14:28:56.380 INFO:teuthology.orchestra.run.smithi158.stdout: rocksdb delete range threshold = 0 2023-10-11T14:28:56.380 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.380 INFO:teuthology.orchestra.run.smithi158.stdout:[mgr] 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: debug mgr = 20 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:56.381 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout: mgr/telemetry/nag = false 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout:[mon] 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout: debug ms = 1 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout: debug mon = 20 2023-10-11T14:28:56.382 INFO:teuthology.orchestra.run.smithi158.stdout: debug paxos = 20 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: debug auth = 20 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: mon data avail warn = 5 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: mon mgr mkfs grace = 240 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd reporter subtree level = osd 2023-10-11T14:28:56.383 INFO:teuthology.orchestra.run.smithi158.stdout: mon osd prime pg temp = true 2023-10-11T14:28:56.384 INFO:teuthology.orchestra.run.smithi158.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:56.384 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.384 INFO:teuthology.orchestra.run.smithi158.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T14:28:56.384 INFO:teuthology.orchestra.run.smithi158.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T14:28:56.384 INFO:teuthology.orchestra.run.smithi158.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T14:28:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T14:28:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T14:28:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T14:28:56.385 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout: # 1m isn't quite enough 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout: mon_down_mkfs_grace = 2m 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout: mon_warn_on_filestore_osds = false 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:28:56.386 INFO:teuthology.orchestra.run.smithi158.stdout:[client] 2023-10-11T14:28:56.387 INFO:teuthology.orchestra.run.smithi158.stdout: rgw cache enabled = true 2023-10-11T14:28:56.387 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable ops log = true 2023-10-11T14:28:56.387 INFO:teuthology.orchestra.run.smithi158.stdout: rgw enable usage log = true 2023-10-11T14:28:56.387 INFO:teuthology.orchestra.run.smithi158.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T14:28:56.387 INFO:teuthology.orchestra.run.smithi158.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: debug rgw = 20 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: debug rgw sync = 20 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt require ssl = False 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt s3 kms backend = testing 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:56.388 INFO:teuthology.orchestra.run.smithi158.stdout: rgw curl low speed time = 300 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw data log num shards = 4 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw md log max shards = 4 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync data inject err probability = 0.1 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync log trim interval = 0 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync meta inject err probability = 0.1 2023-10-11T14:28:56.389 INFO:teuthology.orchestra.run.smithi158.stdout: rgw sync obj etag verify = True 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout: setgroup = ceph 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout: setuser = ceph 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout:[mon.a] 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout:[client.0] 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout: rgw data notify interval msec = 0 2023-10-11T14:28:56.390 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.0] 2023-10-11T14:28:56.391 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 10 2023-10-11T14:28:56.391 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.1] 2023-10-11T14:28:56.391 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:56.391 INFO:teuthology.orchestra.run.smithi158.stdout:[osd.2] 2023-10-11T14:28:56.391 INFO:teuthology.orchestra.run.smithi158.stdout: osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:56.395 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-10-11T14:28:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-10-11T14:28:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T14:28:56.395 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-10-11T14:28:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-10-11T14:28:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-10-11T14:28:56.396 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-10-11T14:28:56.397 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T14:28:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-10-11T14:28:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T14:28:56.398 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-10-11T14:28:56.399 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T14:28:56.400 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.401 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-10-11T14:28:56.402 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.402 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-10-11T14:28:56.402 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-10-11T14:28:56.402 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-10-11T14:28:56.402 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 6feb2062-e670-4c2d-bd4c-87b41529d0a8 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.403 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-10-11T14:28:56.404 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-10-11T14:28:56.404 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-10-11T14:28:56.404 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.404 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-10-11T14:28:56.404 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-10-11T14:28:56.405 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.405 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-10-11T14:28:56.405 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-10-11T14:28:56.405 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-10-11T14:28:56.405 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.406 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-10-11T14:28:56.407 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2023-10-11T14:28:56.408 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-11T14:28:56.409 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-11T14:28:56.410 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-11T14:28:56.411 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-10-11T14:28:56.411 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-11T14:28:56.411 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-10-11T14:28:56.411 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-10-11T14:28:56.411 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-11T14:28:56.412 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-10-11T14:28:56.412 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-10-11T14:28:56.412 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-11T14:28:56.412 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.412 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T14:28:56.413 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T14:28:56.413 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T14:28:56.413 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.413 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T14:28:56.413 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T14:28:56.414 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T14:28:56.414 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.414 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-10-11T14:28:56.414 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-10-11T14:28:56.414 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-10-11T14:28:56.415 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-10-11T14:28:56.416 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T14:28:56.416 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T14:28:56.416 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2023-10-11T14:28:56.416 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw sync = 20 2023-10-11T14:28:56.416 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2023-10-11T14:28:56.417 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync data inject err probability = 0.1 2023-10-11T14:28:56.418 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2023-10-11T14:28:56.418 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync meta inject err probability = 0.1 2023-10-11T14:28:56.418 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2023-10-11T14:28:56.418 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2023-10-11T14:28:56.418 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data notify interval msec = 0 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2023-10-11T14:28:56.419 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-11T14:28:56.420 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2023-10-11T14:28:56.420 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-11T14:28:56.420 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2023-10-11T14:28:56.421 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-10-11T14:28:56.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-10-11T14:28:56.519 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-11T14:28:56.519 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:56.519 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-10-11T14:28:56.537 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:56.537 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-10-11T14:28:56.595 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T14:28:56.595 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:56.595 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/ceph/c2.keyring 2023-10-11T14:28:56.595 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-11T14:28:56.640 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-11T14:28:56.640 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-11T14:28:56.701 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:28:56.701 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:56.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c2.keyring 2023-10-11T14:28:56.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-10-11T14:28:56.738 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:28:56.738 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-10-11T14:28:56.796 INFO:tasks.ceph:Setting up mon nodes... 2023-10-11T14:28:56.796 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-11T14:28:56.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-10-11T14:28:56.890 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-10-11T14:28:56.892 INFO:tasks.ceph:Setting up mds nodes... 2023-10-11T14:28:56.893 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-11T14:28:56.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-10-11T14:28:56.937 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.0.keyring 2023-10-11T14:28:56.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-10-11T14:28:56.993 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.1.keyring 2023-10-11T14:28:57.005 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-11T14:28:57.005 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi158.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T14:28:57.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-10-11T14:28:57.032 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:57.032 INFO:tasks.ceph:role: c2.osd.0 2023-10-11T14:28:57.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:28:57.033 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-11T14:28:57.106 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:57.107 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:57.107 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:57.107 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:57.107 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:57.108 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:57.108 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:57.108 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:57.108 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:57.108 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:57.542 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-11T14:28:57.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-11T14:28:57.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-10-11T14:28:57.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-10-11T14:28:57.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-10-11T14:28:57.826 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:57.826 INFO:tasks.ceph:role: c2.osd.1 2023-10-11T14:28:57.827 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:28:57.827 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-11T14:28:57.901 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:57.901 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:57.901 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:57.901 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:57.902 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:57.902 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:57.902 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:57.902 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:57.902 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:57.903 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:58.326 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-11T14:28:58.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-11T14:28:58.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-10-11T14:28:58.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-10-11T14:28:58.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-10-11T14:28:58.537 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T14:28:58.537 INFO:tasks.ceph:role: c2.osd.2 2023-10-11T14:28:58.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T14:28:58.538 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-11T14:28:58.611 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T14:28:58.612 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T14:28:58.612 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T14:28:58.612 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-11T14:28:58.612 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T14:28:58.612 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-11T14:28:58.613 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T14:28:58.613 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-11T14:28:58.613 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T14:28:58.613 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T14:28:59.053 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-11T14:28:59.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-11T14:28:59.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-10-11T14:28:59.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-10-11T14:28:59.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:28:59.339 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.337+0000 7fd3c06d3640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-10-11T14:28:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.341+0000 7fd3c06d3640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-10-11T14:28:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.341+0000 7fd3c06d3640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-11T14:28:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.341+0000 7fd3c06d3640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-11T14:28:59.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.341+0000 7fd3c06d3640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-10-11T14:28:59.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:28:59.349+0000 7fd3c06d3640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-10-11T14:29:01.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-10-11T14:29:01.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:29:01.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-10-11T14:29:01.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-10-11T14:29:01.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-11T14:29:01.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-11T14:29:01.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-10-11T14:29:01.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:01.556+0000 7fd8a18c8640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-10-11T14:29:03.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-10-11T14:29:03.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:29:03.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.760+0000 7efec9734640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-10-11T14:29:03.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.761+0000 7efec9734640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-10-11T14:29:03.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.761+0000 7efec9734640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-11T14:29:03.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.761+0000 7efec9734640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-11T14:29:03.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.761+0000 7efec9734640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-10-11T14:29:03.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:03.761+0000 7efec9734640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-10-11T14:29:05.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-10-11T14:29:05.874 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-11T14:29:05.874 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:05.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-10-11T14:29:05.944 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:05.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-10-11T14:29:06.014 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:06.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-10-11T14:29:06.083 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:06.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-10-11T14:29:06.152 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:06.152 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-10-11T14:29:06.211 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:06.211 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-10-11T14:29:06.270 INFO:tasks.ceph:Adding keys to all mons... 2023-10-11T14:29:06.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/c2.keyring 2023-10-11T14:29:06.336 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-10-11T14:29:06.336 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAosSZlXdf3NBAAQjw6BrqBsLqj15sgUo64TQ== 2023-10-11T14:29:06.336 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-11T14:29:06.336 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQArsSZlbt8pFBAAEzHT1ugh9bcpqsWZxdf55Q== 2023-10-11T14:29:06.337 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-11T14:29:06.337 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAtsSZlhg80IRAAT49GO0nj2NUVamyto8pxsg== 2023-10-11T14:29:06.337 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-11T14:29:06.337 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAvsSZlqNl1LRAAf5OhfM0OKrnYM1kJStdZ6w== 2023-10-11T14:29:06.337 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-11T14:29:06.338 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAosSZlMWjGNxAAK2XSN32fmzjb6+ZSRf/3dA== 2023-10-11T14:29:06.338 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2023-10-11T14:29:06.338 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAosSZl8cAbOxAAFon4YDsa0GHIGVMTiaPzpQ== 2023-10-11T14:29:06.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-11T14:29:06.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:29:06.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:29:06.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T14:29:06.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T14:29:06.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T14:29:06.708 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-11T14:29:06.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-10-11T14:29:06.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-10-11T14:29:06.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-10-11T14:29:06.862 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-10-11T14:29:06.920 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-10-11T14:29:06.920 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-10-11T14:29:06.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-10-11T14:29:06.962 INFO:tasks.ceph.c2.mon.a:Started 2023-10-11T14:29:06.963 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-10-11T14:29:06.963 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-10-11T14:29:06.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-10-11T14:29:06.965 INFO:tasks.ceph.c2.mgr.x:Started 2023-10-11T14:29:06.965 INFO:tasks.ceph:Setting crush tunables to default 2023-10-11T14:29:06.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd crush tunables default 2023-10-11T14:29:07.175 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2023-10-11T14:29:07.186 INFO:tasks.ceph:check_enable_crimson: False 2023-10-11T14:29:07.186 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-10-11T14:29:07.186 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:07.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-10-11T14:29:07.214 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:07.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-10-11T14:29:07.281 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-11T14:29:07.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-10-11T14:29:07.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new 8499ee7e-19e1-4611-9e70-546bca9d63c7 0 2023-10-11T14:29:07.364 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:07.362+0000 7fcbf066a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T14:29:07.546 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:07.544+0000 7fcbf066a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T14:29:07.605 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-10-11T14:29:07.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new a129c76b-0054-486f-ab0d-1338939b9e2d 1 2023-10-11T14:29:07.830 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-11T14:29:07.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new cebfe5f8-c713-4fbd-adef-4480102eda20 2 2023-10-11T14:29:07.914 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:07.911+0000 7fcbf066a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T14:29:08.060 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-10-11T14:29:08.072 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-10-11T14:29:08.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-10-11T14:29:08.074 INFO:tasks.ceph.c2.osd.0:Started 2023-10-11T14:29:08.075 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-10-11T14:29:08.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-10-11T14:29:08.077 INFO:tasks.ceph.c2.osd.1:Started 2023-10-11T14:29:08.077 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-10-11T14:29:08.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-10-11T14:29:08.080 INFO:tasks.ceph.c2.osd.2:Started 2023-10-11T14:29:08.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-11T14:29:08.323 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:08.323 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"fsid":"6feb2062-e670-4c2d-bd4c-87b41529d0a8","created":"2023-10-11T14:29:07.086082+0000","modified":"2023-10-11T14:29:08.056610+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T14:29:08.056610+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8499ee7e-19e1-4611-9e70-546bca9d63c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a129c76b-0054-486f-ab0d-1338939b9e2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cebfe5f8-c713-4fbd-adef-4480102eda20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:29:08.335 INFO:tasks.ceph.ceph_manager.c2:[] 2023-10-11T14:29:08.336 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-11T14:29:08.479 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:08.477+0000 7fcbf066a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T14:29:08.638 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-11T14:29:08.680 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2023-10-11T14:29:08.677+0000 7fcb182ee640 -1 Falling back to public interface 2023-10-11T14:29:08.681 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:08.678+0000 7fcbf066a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T14:29:08.689 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2023-10-11T14:29:08.686+0000 7fe62cf9d640 -1 Falling back to public interface 2023-10-11T14:29:08.707 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2023-10-11T14:29:08.704+0000 7fcb2cabd640 -1 Falling back to public interface 2023-10-11T14:29:08.846 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-11T14:29:08.846 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"6feb2062-e670-4c2d-bd4c-87b41529d0a8","created":"2023-10-11T14:29:07.086082+0000","modified":"2023-10-11T14:29:08.056610+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T14:29:08.056610+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8499ee7e-19e1-4611-9e70-546bca9d63c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a129c76b-0054-486f-ab0d-1338939b9e2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cebfe5f8-c713-4fbd-adef-4480102eda20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:29:08.857 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T14:29:09.063 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:09.060+0000 7fcbf066a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T14:29:09.732 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:09.730+0000 7fcbf066a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T14:29:09.805 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:09.803+0000 7fcbf066a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T14:29:09.942 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:09.940+0000 7fcbf066a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T14:29:10.012 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.009+0000 7fcbf066a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T14:29:10.197 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.195+0000 7fcbf066a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T14:29:10.276 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.273+0000 7fcbf066a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T14:29:10.353 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.351+0000 7fcbf066a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T14:29:10.452 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.450+0000 7fcbf066a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T14:29:10.635 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.633+0000 7fcbf066a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T14:29:10.718 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:10.716+0000 7fcbf066a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T14:29:11.012 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:11.009+0000 7fcbf066a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T14:29:11.083 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:11.080+0000 7fcbf066a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T14:29:11.489 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2023-10-11T14:29:11.486+0000 7fcb182ee640 -1 osd.0 0 log_to_monitors true 2023-10-11T14:29:11.510 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2023-10-11T14:29:11.508+0000 7fcb2cabd640 -1 osd.1 0 log_to_monitors true 2023-10-11T14:29:11.521 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2023-10-11T14:29:11.518+0000 7fe62cf9d640 -1 osd.2 0 log_to_monitors true 2023-10-11T14:29:11.628 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:11.626+0000 7fcbf066a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T14:29:11.948 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:11.946+0000 7fcbf066a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T14:29:12.092 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:12.090+0000 7fcbf066a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T14:29:12.487 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:12.484+0000 7fcbf066a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T14:29:12.661 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:12.658+0000 7fcbf066a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T14:29:12.828 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:12.826+0000 7fcbf066a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T14:29:13.175 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:13.173+0000 7fcbf066a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T14:29:13.772 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2023-10-11T14:29:13.770+0000 7fcb0c68f700 -1 osd.0 0 waiting for initial osdmap 2023-10-11T14:29:13.785 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2023-10-11T14:29:13.783+0000 7fcb20e5e700 -1 osd.1 0 waiting for initial osdmap 2023-10-11T14:29:13.858 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2023-10-11T14:29:13.856+0000 7fcb04269700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:29:13.859 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2023-10-11T14:29:13.856+0000 7fcb18a38700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:29:14.067 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2023-10-11T14:29:14.065+0000 7fe62133e700 -1 osd.2 0 waiting for initial osdmap 2023-10-11T14:29:14.083 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2023-10-11T14:29:14.081+0000 7fe618f18700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T14:29:14.229 INFO:tasks.ceph.c2.mgr.x.smithi172.stderr:2023-10-11T14:29:14.227+0000 7fcbb1a83700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-11T14:29:15.160 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-10-11T14:29:15.490 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-11T14:29:15.490 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":8,"fsid":"6feb2062-e670-4c2d-bd4c-87b41529d0a8","created":"2023-10-11T14:29:07.086082+0000","modified":"2023-10-11T14:29:14.784814+0000","last_up_change":"2023-10-11T14:29:14.784814+0000","last_in_change":"2023-10-11T14:29:08.056610+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8499ee7e-19e1-4611-9e70-546bca9d63c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114355},{"type":"v1","addr":"172.21.15.172:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":114355},{"type":"v1","addr":"172.21.15.172:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":114355},{"type":"v1","addr":"172.21.15.172:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114355},{"type":"v1","addr":"172.21.15.172:6805","nonce":114355}]},"public_addr":"172.21.15.172:6801/114355","cluster_addr":"172.21.15.172:6803/114355","heartbeat_back_addr":"172.21.15.172:6807/114355","heartbeat_front_addr":"172.21.15.172:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"a129c76b-0054-486f-ab0d-1338939b9e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114357},{"type":"v1","addr":"172.21.15.172:6817","nonce":114357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114357},{"type":"v1","addr":"172.21.15.172:6819","nonce":114357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114357},{"type":"v1","addr":"172.21.15.172:6823","nonce":114357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114357},{"type":"v1","addr":"172.21.15.172:6821","nonce":114357}]},"public_addr":"172.21.15.172:6817/114357","cluster_addr":"172.21.15.172:6819/114357","heartbeat_back_addr":"172.21.15.172:6823/114357","heartbeat_front_addr":"172.21.15.172:6821/114357","state":["exists","up"]},{"osd":2,"uuid":"cebfe5f8-c713-4fbd-adef-4480102eda20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114356},{"type":"v1","addr":"172.21.15.172:6809","nonce":114356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":114356},{"type":"v1","addr":"172.21.15.172:6811","nonce":114356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":114356},{"type":"v1","addr":"172.21.15.172:6815","nonce":114356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114356},{"type":"v1","addr":"172.21.15.172:6813","nonce":114356}]},"public_addr":"172.21.15.172:6809/114356","cluster_addr":"172.21.15.172:6811/114356","heartbeat_back_addr":"172.21.15.172:6815/114356","heartbeat_front_addr":"172.21.15.172:6813/114356","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:29:15.503 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-11T14:29:15.503 INFO:tasks.ceph:Creating RBD pool 2023-10-11T14:29:15.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool create rbd 8 2023-10-11T14:29:16.241 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2023-10-11T14:29:16.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-11T14:29:17.242 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-11T14:29:17.256 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-10-11T14:29:17.258 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-10-11T14:29:17.258 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-10-11T14:29:17.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-10-11T14:29:17.259 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-11T14:29:17.675 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:17.687 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":113943},{"type":"v1","addr":"172.21.15.172:6825","nonce":113943}]},"active_addr":"172.21.15.172:6825/113943","active_change":"2023-10-11T14:29:13.186765+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2708402826}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":4049707268}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1833520047}]}]} 2023-10-11T14:29:17.691 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-10-11T14:29:17.691 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-10-11T14:29:17.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-11T14:29:18.010 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:18.011 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":10,"fsid":"6feb2062-e670-4c2d-bd4c-87b41529d0a8","created":"2023-10-11T14:29:07.086082+0000","modified":"2023-10-11T14:29:17.231114+0000","last_up_change":"2023-10-11T14:29:14.784814+0000","last_in_change":"2023-10-11T14:29:08.056610+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T14:29:15.268847+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T14:29:15.837506+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8499ee7e-19e1-4611-9e70-546bca9d63c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114355},{"type":"v1","addr":"172.21.15.172:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":114355},{"type":"v1","addr":"172.21.15.172:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":114355},{"type":"v1","addr":"172.21.15.172:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114355},{"type":"v1","addr":"172.21.15.172:6805","nonce":114355}]},"public_addr":"172.21.15.172:6801/114355","cluster_addr":"172.21.15.172:6803/114355","heartbeat_back_addr":"172.21.15.172:6807/114355","heartbeat_front_addr":"172.21.15.172:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"a129c76b-0054-486f-ab0d-1338939b9e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114357},{"type":"v1","addr":"172.21.15.172:6817","nonce":114357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114357},{"type":"v1","addr":"172.21.15.172:6819","nonce":114357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114357},{"type":"v1","addr":"172.21.15.172:6823","nonce":114357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114357},{"type":"v1","addr":"172.21.15.172:6821","nonce":114357}]},"public_addr":"172.21.15.172:6817/114357","cluster_addr":"172.21.15.172:6819/114357","heartbeat_back_addr":"172.21.15.172:6823/114357","heartbeat_front_addr":"172.21.15.172:6821/114357","state":["exists","up"]},{"osd":2,"uuid":"cebfe5f8-c713-4fbd-adef-4480102eda20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114356},{"type":"v1","addr":"172.21.15.172:6809","nonce":114356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":114356},{"type":"v1","addr":"172.21.15.172:6811","nonce":114356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":114356},{"type":"v1","addr":"172.21.15.172:6815","nonce":114356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114356},{"type":"v1","addr":"172.21.15.172:6813","nonce":114356}]},"public_addr":"172.21.15.172:6809/114356","cluster_addr":"172.21.15.172:6811/114356","heartbeat_back_addr":"172.21.15.172:6815/114356","heartbeat_front_addr":"172.21.15.172:6813/114356","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.447192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.531471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.477017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:29:18.024 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-10-11T14:29:18.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-10-11T14:29:18.347 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:18.347 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":11,"fsid":"6feb2062-e670-4c2d-bd4c-87b41529d0a8","created":"2023-10-11T14:29:07.086082+0000","modified":"2023-10-11T14:29:18.232187+0000","last_up_change":"2023-10-11T14:29:14.784814+0000","last_in_change":"2023-10-11T14:29:08.056610+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T14:29:15.268847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T14:29:15.837506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8499ee7e-19e1-4611-9e70-546bca9d63c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":114355},{"type":"v1","addr":"172.21.15.172:6801","nonce":114355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":114355},{"type":"v1","addr":"172.21.15.172:6803","nonce":114355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":114355},{"type":"v1","addr":"172.21.15.172:6807","nonce":114355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":114355},{"type":"v1","addr":"172.21.15.172:6805","nonce":114355}]},"public_addr":"172.21.15.172:6801/114355","cluster_addr":"172.21.15.172:6803/114355","heartbeat_back_addr":"172.21.15.172:6807/114355","heartbeat_front_addr":"172.21.15.172:6805/114355","state":["exists","up"]},{"osd":1,"uuid":"a129c76b-0054-486f-ab0d-1338939b9e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":114357},{"type":"v1","addr":"172.21.15.172:6817","nonce":114357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":114357},{"type":"v1","addr":"172.21.15.172:6819","nonce":114357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":114357},{"type":"v1","addr":"172.21.15.172:6823","nonce":114357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":114357},{"type":"v1","addr":"172.21.15.172:6821","nonce":114357}]},"public_addr":"172.21.15.172:6817/114357","cluster_addr":"172.21.15.172:6819/114357","heartbeat_back_addr":"172.21.15.172:6823/114357","heartbeat_front_addr":"172.21.15.172:6821/114357","state":["exists","up"]},{"osd":2,"uuid":"cebfe5f8-c713-4fbd-adef-4480102eda20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":114356},{"type":"v1","addr":"172.21.15.172:6809","nonce":114356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":114356},{"type":"v1","addr":"172.21.15.172:6811","nonce":114356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":114356},{"type":"v1","addr":"172.21.15.172:6815","nonce":114356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":114356},{"type":"v1","addr":"172.21.15.172:6813","nonce":114356}]},"public_addr":"172.21.15.172:6809/114356","cluster_addr":"172.21.15.172:6811/114356","heartbeat_back_addr":"172.21.15.172:6815/114356","heartbeat_front_addr":"172.21.15.172:6813/114356","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.447192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.531471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T14:29:12.477017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T14:29:18.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-10-11T14:29:18.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-10-11T14:29:18.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-10-11T14:29:18.525 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:18.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-11T14:29:18.544 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:18.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-11T14:29:18.546 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:18.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-11T14:29:18.881 INFO:teuthology.orchestra.run.smithi172.stdout:34359738369 2023-10-11T14:29:18.895 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2023-10-11T14:29:18.931 INFO:teuthology.orchestra.run.smithi172.stdout:34359738369 2023-10-11T14:29:18.946 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2023-10-11T14:29:18.948 INFO:teuthology.orchestra.run.smithi172.stdout:34359738369 2023-10-11T14:29:18.962 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2023-10-11T14:29:19.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-10-11T14:29:19.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-10-11T14:29:19.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-10-11T14:29:20.256 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:20.270 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2023-10-11T14:29:20.270 DEBUG:teuthology.parallel:result is None 2023-10-11T14:29:20.309 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:20.323 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2023-10-11T14:29:20.323 DEBUG:teuthology.parallel:result is None 2023-10-11T14:29:20.335 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-11T14:29:20.349 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.2 2023-10-11T14:29:20.349 DEBUG:teuthology.parallel:result is None 2023-10-11T14:29:20.349 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-10-11T14:29:20.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-11T14:29:20.702 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:20.704 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-11T14:29:20.715 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-11T14:29:19.225579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82716,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490084,"statfs":{"total":322122547200,"available":322037846016,"internally_reserved":0,"allocated":1449984,"data_stored":1246254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236248+0000","last_change":"2023-10-11T14:29:17.239256+0000","last_active":"2023-10-11T14:29:18.236248+0000","last_peered":"2023-10-11T14:29:18.236248+0000","last_clean":"2023-10-11T14:29:18.236248+0000","last_became_active":"2023-10-11T14:29:17.238766+0000","last_became_peered":"2023-10-11T14:29:17.238766+0000","last_unstale":"2023-10-11T14:29:18.236248+0000","last_undegraded":"2023-10-11T14:29:18.236248+0000","last_fullsized":"2023-10-11T14:29:18.236248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:58.237332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236440+0000","last_change":"2023-10-11T14:29:17.239406+0000","last_active":"2023-10-11T14:29:18.236440+0000","last_peered":"2023-10-11T14:29:18.236440+0000","last_clean":"2023-10-11T14:29:18.236440+0000","last_became_active":"2023-10-11T14:29:17.239203+0000","last_became_peered":"2023-10-11T14:29:17.239203+0000","last_unstale":"2023-10-11T14:29:18.236440+0000","last_undegraded":"2023-10-11T14:29:18.236440+0000","last_fullsized":"2023-10-11T14:29:18.236440+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:46:21.233939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236219+0000","last_change":"2023-10-11T14:29:17.241502+0000","last_active":"2023-10-11T14:29:18.236219+0000","last_peered":"2023-10-11T14:29:18.236219+0000","last_clean":"2023-10-11T14:29:18.236219+0000","last_became_active":"2023-10-11T14:29:17.241232+0000","last_became_peered":"2023-10-11T14:29:17.241232+0000","last_unstale":"2023-10-11T14:29:18.236219+0000","last_undegraded":"2023-10-11T14:29:18.236219+0000","last_fullsized":"2023-10-11T14:29:18.236219+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:35:35.207344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.522743+0000","last_change":"2023-10-11T14:29:17.239468+0000","last_active":"2023-10-11T14:29:18.522743+0000","last_peered":"2023-10-11T14:29:18.522743+0000","last_clean":"2023-10-11T14:29:18.522743+0000","last_became_active":"2023-10-11T14:29:17.239229+0000","last_became_peered":"2023-10-11T14:29:17.239229+0000","last_unstale":"2023-10-11T14:29:18.522743+0000","last_undegraded":"2023-10-11T14:29:18.522743+0000","last_fullsized":"2023-10-11T14:29:18.522743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:22:23.617824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236214+0000","last_change":"2023-10-11T14:29:17.241841+0000","last_active":"2023-10-11T14:29:18.236214+0000","last_peered":"2023-10-11T14:29:18.236214+0000","last_clean":"2023-10-11T14:29:18.236214+0000","last_became_active":"2023-10-11T14:29:17.241000+0000","last_became_peered":"2023-10-11T14:29:17.241000+0000","last_unstale":"2023-10-11T14:29:18.236214+0000","last_undegraded":"2023-10-11T14:29:18.236214+0000","last_fullsized":"2023-10-11T14:29:18.236214+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:02:56.716702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236168+0000","last_change":"2023-10-11T14:29:17.240392+0000","last_active":"2023-10-11T14:29:18.236168+0000","last_peered":"2023-10-11T14:29:18.236168+0000","last_clean":"2023-10-11T14:29:18.236168+0000","last_became_active":"2023-10-11T14:29:17.240161+0000","last_became_peered":"2023-10-11T14:29:17.240161+0000","last_unstale":"2023-10-11T14:29:18.236168+0000","last_undegraded":"2023-10-11T14:29:18.236168+0000","last_fullsized":"2023-10-11T14:29:18.236168+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:09:05.761066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.522743+0000","last_change":"2023-10-11T14:29:17.239042+0000","last_active":"2023-10-11T14:29:18.522743+0000","last_peered":"2023-10-11T14:29:18.522743+0000","last_clean":"2023-10-11T14:29:18.522743+0000","last_became_active":"2023-10-11T14:29:17.238745+0000","last_became_peered":"2023-10-11T14:29:17.238745+0000","last_unstale":"2023-10-11T14:29:18.522743+0000","last_undegraded":"2023-10-11T14:29:18.522743+0000","last_fullsized":"2023-10-11T14:29:18.522743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:49:06.219548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.490899+0000","last_change":"2023-10-11T14:29:17.240224+0000","last_active":"2023-10-11T14:29:18.490899+0000","last_peered":"2023-10-11T14:29:18.490899+0000","last_clean":"2023-10-11T14:29:18.490899+0000","last_became_active":"2023-10-11T14:29:17.239992+0000","last_became_peered":"2023-10-11T14:29:17.239992+0000","last_unstale":"2023-10-11T14:29:18.490899+0000","last_undegraded":"2023-10-11T14:29:18.490899+0000","last_fullsized":"2023-10-11T14:29:18.490899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:41:59.566516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236205+0000","last_change":"2023-10-11T14:29:17.238802+0000","last_active":"2023-10-11T14:29:18.236205+0000","last_peered":"2023-10-11T14:29:18.236205+0000","last_clean":"2023-10-11T14:29:18.236205+0000","last_became_active":"2023-10-11T14:29:17.238566+0000","last_became_peered":"2023-10-11T14:29:17.238566+0000","last_unstale":"2023-10-11T14:29:18.236205+0000","last_undegraded":"2023-10-11T14:29:18.236205+0000","last_fullsized":"2023-10-11T14:29:18.236205+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:25:56.949748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27744,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829856,"statfs":{"total":107374182400,"available":107345772544,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27936,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829664,"statfs":{"total":107374182400,"available":107345575936,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T14:29:20.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-10-11T14:29:21.031 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:21.033 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-11T14:29:21.045 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-11T14:29:19.225579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82716,"kb_used_data":1416,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490084,"statfs":{"total":322122547200,"available":322037846016,"internally_reserved":0,"allocated":1449984,"data_stored":1246254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236248+0000","last_change":"2023-10-11T14:29:17.239256+0000","last_active":"2023-10-11T14:29:18.236248+0000","last_peered":"2023-10-11T14:29:18.236248+0000","last_clean":"2023-10-11T14:29:18.236248+0000","last_became_active":"2023-10-11T14:29:17.238766+0000","last_became_peered":"2023-10-11T14:29:17.238766+0000","last_unstale":"2023-10-11T14:29:18.236248+0000","last_undegraded":"2023-10-11T14:29:18.236248+0000","last_fullsized":"2023-10-11T14:29:18.236248+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:58.237332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236440+0000","last_change":"2023-10-11T14:29:17.239406+0000","last_active":"2023-10-11T14:29:18.236440+0000","last_peered":"2023-10-11T14:29:18.236440+0000","last_clean":"2023-10-11T14:29:18.236440+0000","last_became_active":"2023-10-11T14:29:17.239203+0000","last_became_peered":"2023-10-11T14:29:17.239203+0000","last_unstale":"2023-10-11T14:29:18.236440+0000","last_undegraded":"2023-10-11T14:29:18.236440+0000","last_fullsized":"2023-10-11T14:29:18.236440+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:46:21.233939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236219+0000","last_change":"2023-10-11T14:29:17.241502+0000","last_active":"2023-10-11T14:29:18.236219+0000","last_peered":"2023-10-11T14:29:18.236219+0000","last_clean":"2023-10-11T14:29:18.236219+0000","last_became_active":"2023-10-11T14:29:17.241232+0000","last_became_peered":"2023-10-11T14:29:17.241232+0000","last_unstale":"2023-10-11T14:29:18.236219+0000","last_undegraded":"2023-10-11T14:29:18.236219+0000","last_fullsized":"2023-10-11T14:29:18.236219+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:35:35.207344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.522743+0000","last_change":"2023-10-11T14:29:17.239468+0000","last_active":"2023-10-11T14:29:18.522743+0000","last_peered":"2023-10-11T14:29:18.522743+0000","last_clean":"2023-10-11T14:29:18.522743+0000","last_became_active":"2023-10-11T14:29:17.239229+0000","last_became_peered":"2023-10-11T14:29:17.239229+0000","last_unstale":"2023-10-11T14:29:18.522743+0000","last_undegraded":"2023-10-11T14:29:18.522743+0000","last_fullsized":"2023-10-11T14:29:18.522743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:22:23.617824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236214+0000","last_change":"2023-10-11T14:29:17.241841+0000","last_active":"2023-10-11T14:29:18.236214+0000","last_peered":"2023-10-11T14:29:18.236214+0000","last_clean":"2023-10-11T14:29:18.236214+0000","last_became_active":"2023-10-11T14:29:17.241000+0000","last_became_peered":"2023-10-11T14:29:17.241000+0000","last_unstale":"2023-10-11T14:29:18.236214+0000","last_undegraded":"2023-10-11T14:29:18.236214+0000","last_fullsized":"2023-10-11T14:29:18.236214+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:02:56.716702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236168+0000","last_change":"2023-10-11T14:29:17.240392+0000","last_active":"2023-10-11T14:29:18.236168+0000","last_peered":"2023-10-11T14:29:18.236168+0000","last_clean":"2023-10-11T14:29:18.236168+0000","last_became_active":"2023-10-11T14:29:17.240161+0000","last_became_peered":"2023-10-11T14:29:17.240161+0000","last_unstale":"2023-10-11T14:29:18.236168+0000","last_undegraded":"2023-10-11T14:29:18.236168+0000","last_fullsized":"2023-10-11T14:29:18.236168+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:09:05.761066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.522743+0000","last_change":"2023-10-11T14:29:17.239042+0000","last_active":"2023-10-11T14:29:18.522743+0000","last_peered":"2023-10-11T14:29:18.522743+0000","last_clean":"2023-10-11T14:29:18.522743+0000","last_became_active":"2023-10-11T14:29:17.238745+0000","last_became_peered":"2023-10-11T14:29:17.238745+0000","last_unstale":"2023-10-11T14:29:18.522743+0000","last_undegraded":"2023-10-11T14:29:18.522743+0000","last_fullsized":"2023-10-11T14:29:18.522743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:49:06.219548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.490899+0000","last_change":"2023-10-11T14:29:17.240224+0000","last_active":"2023-10-11T14:29:18.490899+0000","last_peered":"2023-10-11T14:29:18.490899+0000","last_clean":"2023-10-11T14:29:18.490899+0000","last_became_active":"2023-10-11T14:29:17.239992+0000","last_became_peered":"2023-10-11T14:29:17.239992+0000","last_unstale":"2023-10-11T14:29:18.490899+0000","last_undegraded":"2023-10-11T14:29:18.490899+0000","last_fullsized":"2023-10-11T14:29:18.490899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:41:59.566516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-11T14:29:18.236205+0000","last_change":"2023-10-11T14:29:17.238802+0000","last_active":"2023-10-11T14:29:18.236205+0000","last_peered":"2023-10-11T14:29:18.236205+0000","last_clean":"2023-10-11T14:29:18.236205+0000","last_became_active":"2023-10-11T14:29:17.238566+0000","last_became_peered":"2023-10-11T14:29:17.238566+0000","last_unstale":"2023-10-11T14:29:18.236205+0000","last_undegraded":"2023-10-11T14:29:18.236205+0000","last_fullsized":"2023-10-11T14:29:18.236205+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T14:29:16.227714+0000","last_clean_scrub_stamp":"2023-10-11T14:29:16.227714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:25:56.949748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27036,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830564,"statfs":{"total":107374182400,"available":107346497536,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27744,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829856,"statfs":{"total":107374182400,"available":107345772544,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27936,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829664,"statfs":{"total":107374182400,"available":107345575936,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T14:29:21.046 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-10-11T14:29:21.046 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-10-11T14:29:21.046 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-10-11T14:29:21.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-10-11T14:29:21.409 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-11T14:29:21.409 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T14:29:21.423 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-10-11T14:29:21.423 INFO:teuthology.run_tasks:Running task rgw... 2023-10-11T14:29:21.436 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-10-11T14:29:21.436 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-10-11T14:29:21.437 INFO:tasks.rgw:Creating data pools 2023-10-11T14:29:21.437 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-10-11T14:29:21.437 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-11T14:29:22.127 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'default.rgw.buckets.data' created 2023-10-11T14:29:22.137 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-11T14:29:23.144 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-11T14:29:23.156 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-11T14:29:24.132 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'default.rgw.buckets.index' created 2023-10-11T14:29:24.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-11T14:29:25.141 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-11T14:29:25.159 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-10-11T14:29:25.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-10-11T14:29:25.484 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-11T14:29:25.495 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-10-11T14:29:26.128 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-11T14:29:26.140 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-10-11T14:29:26.466 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-11T14:29:26.478 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-10-11T14:29:27.128 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-11T14:29:27.139 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-10-11T14:29:27.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-11T14:29:28.076 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' created 2023-10-11T14:29:28.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-11T14:29:29.089 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-11T14:29:29.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-11T14:29:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' created 2023-10-11T14:29:30.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-11T14:29:31.093 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-11T14:29:31.113 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-10-11T14:29:31.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-10-11T14:29:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' already exists 2023-10-11T14:29:31.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-10-11T14:29:32.076 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-11T14:29:32.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-10-11T14:29:32.412 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' already exists 2023-10-11T14:29:32.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-10-11T14:29:33.078 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-11T14:29:33.090 DEBUG:tasks.rgw:Pools created 2023-10-11T14:29:33.090 INFO:tasks.rgw:Configuring compression type = random 2023-10-11T14:29:33.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-10-11T14:29:33.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-10-11T14:29:33.092 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-10-11T14:29:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:33.215+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.306 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.303+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.303+0000 7f409958c780 20 realm 2023-10-11T14:29:34.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.303+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.307 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.304+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.304+0000 7f409958c780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-11T14:29:34.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.304+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.308 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.305+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.309 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.305+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.308+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.308+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.309+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.309+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.309+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.309+0000 7f409958c780 10 Creating default zonegroup 2023-10-11T14:29:34.313 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.309+0000 7f409958c780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-11T14:29:34.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.318+0000 7f409958c780 10 WARNING: driver->list_zones() returned r=-2 2023-10-11T14:29:34.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.318+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.319+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.324 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 10 could not read realm id: (2) No such file or directory 2023-10-11T14:29:34.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 10 WARNING: failed to set zone as default, r=-22 2023-10-11T14:29:34.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.321+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.323+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados_obj.operate() r=0 bl.length=874 2023-10-11T14:29:34.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados_obj.operate() r=0 bl.length=420 2023-10-11T14:29:34.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 10 Using default name default 2023-10-11T14:29:34.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:34.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados_obj.operate() r=0 bl.length=874 2023-10-11T14:29:34.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 zone default found 2023-10-11T14:29:34.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 4 Realm: () 2023-10-11T14:29:34.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 4 ZoneGroup: default (d014fdba-8299-4ddf-8e77-afc00e3c5d76) 2023-10-11T14:29:34.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 4 Zone: default (a418518f-092c-45b7-b2fd-9b8b80bb533a) 2023-10-11T14:29:34.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-11T14:29:34.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 zonegroup default 2023-10-11T14:29:34.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.324+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:34.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.325+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:34.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:34.325+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:36.318 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:36.315+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:36.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:36.315+0000 7f409958c780 20 rados->read ofs=0 len=0 2023-10-11T14:29:36.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:36.316+0000 7f409958c780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:36.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:36.316+0000 7f409958c780 20 started sync module instance, tier type = 2023-10-11T14:29:36.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:36.316+0000 7f409958c780 20 started zone id=a418518f-092c-45b7-b2fd-9b8b80bb533a (name=default) with tier type = 2023-10-11T14:29:38.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=0 2023-10-11T14:29:38.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=1 2023-10-11T14:29:38.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=2 2023-10-11T14:29:38.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=3 2023-10-11T14:29:38.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=4 2023-10-11T14:29:38.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=5 2023-10-11T14:29:38.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.334+0000 7f409958c780 20 add_watcher() i=6 2023-10-11T14:29:38.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.335+0000 7f409958c780 20 add_watcher() i=7 2023-10-11T14:29:38.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.335+0000 7f409958c780 2 all 8 watchers are set, enabling cache 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.335+0000 7f409958c780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.335+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.335+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-11T14:29:38.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:38.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-11T14:29:38.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-11T14:29:38.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:38.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-11T14:29:38.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-11T14:29:38.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:38.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-11T14:29:38.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.336+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-11T14:29:38.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.337+0000 7f409958c780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:38.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.340+0000 7f407e508700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:38.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.340+0000 7f409958c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:38.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:38.340+0000 7f409958c780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:41.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.315+0000 7f409958c780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-11T14:29:41.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.315+0000 7f409958c780 5 note: GC not initialized 2023-10-11T14:29:41.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.317+0000 7f409958c780 20 init_complete bucket index max shards: 11 2023-10-11T14:29:41.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.317+0000 7f409958c780 10 rgw notify: Started notification manager with: 1 workers 2023-10-11T14:29:41.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.317+0000 7f4077cfb700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 11 14:30:11 2023 2023-10-11T14:29:41.322 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-11T14:29:41.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.317+0000 7f40774d9700 20 reqs_thread_entry: start 2023-10-11T14:29:41.323 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.318+0000 7f4079cff700 20 reqs_thread_entry: start 2023-10-11T14:29:41.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.337+0000 7f407e508700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.339+0000 7f409958c780 20 remove_watcher() i=0 2023-10-11T14:29:41.342 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.339+0000 7f409958c780 2 removed watcher, disabling cache 2023-10-11T14:29:41.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.340+0000 7f409958c780 20 remove_watcher() i=1 2023-10-11T14:29:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.341+0000 7f409958c780 20 remove_watcher() i=2 2023-10-11T14:29:41.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.342+0000 7f409958c780 20 remove_watcher() i=3 2023-10-11T14:29:41.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.343+0000 7f409958c780 20 remove_watcher() i=4 2023-10-11T14:29:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.344+0000 7f409958c780 20 remove_watcher() i=5 2023-10-11T14:29:41.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.345+0000 7f409958c780 20 remove_watcher() i=6 2023-10-11T14:29:41.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.346+0000 7f409958c780 20 remove_watcher() i=7 2023-10-11T14:29:41.351 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:29:41.353 DEBUG:tasks.util.rgw: json result: [] 2023-10-11T14:29:41.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:41.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:41.354 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-11T14:29:41.407 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:41.407 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:41.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.413+0000 7f37e77a2780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:41.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.413+0000 7f37e77a2780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:41.420 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.417+0000 7f37c8f17700 20 reqs_thread_entry: start 2023-10-11T14:29:41.426 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:41.426 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a418518f-092c-45b7-b2fd-9b8b80bb533a", 2023-10-11T14:29:41.427 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default", 2023-10-11T14:29:41.427 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "default.rgw.meta:root", 2023-10-11T14:29:41.427 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "default.rgw.control", 2023-10-11T14:29:41.427 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-11T14:29:41.427 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-11T14:29:41.428 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "default.rgw.log", 2023-10-11T14:29:41.428 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-11T14:29:41.428 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-11T14:29:41.428 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-11T14:29:41.428 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "default.rgw.otp", 2023-10-11T14:29:41.429 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:29:41.430 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "", 2023-10-11T14:29:41.430 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "" 2023-10-11T14:29:41.430 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:41.430 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:29:41.430 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:29:41.431 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:29:41.431 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:29:41.431 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-11T14:29:41.431 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:29:41.431 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:29:41.432 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-11T14:29:41.432 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:29:41.432 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:41.432 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:41.433 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "", 2023-10-11T14:29:41.434 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-11T14:29:41.434 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:41.435 DEBUG:tasks.util.rgw: json result: {'id': 'a418518f-092c-45b7-b2fd-9b8b80bb533a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-11T14:29:41.435 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-10-11T14:29:41.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-10-11T14:29:41.436 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-10-11T14:29:41.479 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:41.479 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:41.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.484+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 realm 2023-10-11T14:29:41.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.485+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.486+0000 7fe5a056e780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:41.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.486+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.486+0000 7fe5a056e780 20 rados_obj.operate() r=0 bl.length=890 2023-10-11T14:29:41.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.486+0000 7fe5a056e780 20 searching for the correct realm 2023-10-11T14:29:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 RGWRados::pool_iterate: got zonegroup_info.d014fdba-8299-4ddf-8e77-afc00e3c5d76 2023-10-11T14:29:41.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-11T14:29:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-11T14:29:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 RGWRados::pool_iterate: got zone_info.a418518f-092c-45b7-b2fd-9b8b80bb533a 2023-10-11T14:29:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:41.498 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.493+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados_obj.operate() r=0 bl.length=420 2023-10-11T14:29:41.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 zone default found 2023-10-11T14:29:41.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 4 Realm: () 2023-10-11T14:29:41.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 4 ZoneGroup: default (d014fdba-8299-4ddf-8e77-afc00e3c5d76) 2023-10-11T14:29:41.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 4 Zone: default (a418518f-092c-45b7-b2fd-9b8b80bb533a) 2023-10-11T14:29:41.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-11T14:29:41.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 zonegroup default 2023-10-11T14:29:41.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados->read ofs=0 len=0 2023-10-11T14:29:41.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:41.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 started sync module instance, tier type = 2023-10-11T14:29:41.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.494+0000 7fe5a056e780 20 started zone id=a418518f-092c-45b7-b2fd-9b8b80bb533a (name=default) with tier type = 2023-10-11T14:29:41.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=0 2023-10-11T14:29:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=1 2023-10-11T14:29:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=2 2023-10-11T14:29:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=3 2023-10-11T14:29:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=4 2023-10-11T14:29:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=5 2023-10-11T14:29:41.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.501+0000 7fe5a056e780 20 add_watcher() i=6 2023-10-11T14:29:41.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.502+0000 7fe5a056e780 20 add_watcher() i=7 2023-10-11T14:29:41.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.502+0000 7fe5a056e780 2 all 8 watchers are set, enabling cache 2023-10-11T14:29:41.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.503+0000 7fe5854ea700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:41.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.503+0000 7fe5a056e780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:41.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.503+0000 7fe5a056e780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:41.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.507+0000 7fe5a056e780 5 note: GC not initialized 2023-10-11T14:29:41.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.508+0000 7fe580ce1700 20 reqs_thread_entry: start 2023-10-11T14:29:41.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.508+0000 7fe5a056e780 20 init_complete bucket index max shards: 11 2023-10-11T14:29:41.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.508+0000 7fe5a056e780 10 rgw notify: Started notification manager with: 1 workers 2023-10-11T14:29:41.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.508+0000 7fe57ecdd700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 11 14:30:11 2023 2023-10-11T14:29:41.512 INFO:teuthology.orchestra.run.smithi158.stderr: 2023-10-11T14:29:41.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.508+0000 7fe57e4bb700 20 reqs_thread_entry: start 2023-10-11T14:29:41.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.515+0000 7fe5854ea700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:41.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.517+0000 7fe5a056e780 20 remove_watcher() i=0 2023-10-11T14:29:41.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.517+0000 7fe5a056e780 2 removed watcher, disabling cache 2023-10-11T14:29:41.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.518+0000 7fe5a056e780 20 remove_watcher() i=1 2023-10-11T14:29:41.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.519+0000 7fe5a056e780 20 remove_watcher() i=2 2023-10-11T14:29:41.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.520+0000 7fe5a056e780 20 remove_watcher() i=3 2023-10-11T14:29:41.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.520+0000 7fe5a056e780 20 remove_watcher() i=4 2023-10-11T14:29:41.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.521+0000 7fe5a056e780 20 remove_watcher() i=5 2023-10-11T14:29:41.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.522+0000 7fe5a056e780 20 remove_watcher() i=6 2023-10-11T14:29:41.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.523+0000 7fe5a056e780 20 remove_watcher() i=7 2023-10-11T14:29:41.528 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:29:41.530 DEBUG:tasks.util.rgw: json result: [] 2023-10-11T14:29:41.530 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:41.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:41.531 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-11T14:29:41.620 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:41.620 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:41.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.625+0000 7f6f48aa7780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:41.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.625+0000 7f6f48aa7780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:41.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:41.628+0000 7f6f2a21c700 20 reqs_thread_entry: start 2023-10-11T14:29:41.637 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:41.637 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a418518f-092c-45b7-b2fd-9b8b80bb533a", 2023-10-11T14:29:41.637 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default", 2023-10-11T14:29:41.638 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "default.rgw.meta:root", 2023-10-11T14:29:41.638 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "default.rgw.control", 2023-10-11T14:29:41.638 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-11T14:29:41.638 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-11T14:29:41.638 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "default.rgw.log", 2023-10-11T14:29:41.639 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-11T14:29:41.639 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-11T14:29:41.639 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-11T14:29:41.639 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-11T14:29:41.639 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "default.rgw.otp", 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:29:41.640 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "", 2023-10-11T14:29:41.641 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "" 2023-10-11T14:29:41.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:41.641 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:29:41.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:29:41.641 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:29:41.642 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:29:41.642 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-11T14:29:41.642 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:29:41.642 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:29:41.642 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-11T14:29:41.643 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:29:41.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:41.643 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-11T14:29:41.643 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "", 2023-10-11T14:29:41.644 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-11T14:29:41.645 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:41.646 DEBUG:tasks.util.rgw: json result: {'id': 'a418518f-092c-45b7-b2fd-9b8b80bb533a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-11T14:29:41.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-10-11T14:29:41.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-10-11T14:29:41.647 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-10-11T14:29:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:41.781 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:41.778+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.321+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.321+0000 7f765bb69780 20 realm 2023-10-11T14:29:43.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.321+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.322+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.322+0000 7f765bb69780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-11T14:29:43.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.322+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.322+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.322+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.330+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.330+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.331+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.331+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.332+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.332+0000 7f765bb69780 10 Creating default zonegroup 2023-10-11T14:29:43.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.332+0000 7f765bb69780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-10-11T14:29:43.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.342+0000 7f765bb69780 10 WARNING: driver->list_zones() returned r=-2 2023-10-11T14:29:43.344 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.342+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.345 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.342+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.347 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.347 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.347 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 10 could not read realm id: (2) No such file or directory 2023-10-11T14:29:43.347 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 10 WARNING: failed to set zone as default, r=-22 2023-10-11T14:29:43.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.344+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.349 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.347+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.349 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.347+0000 7f765bb69780 20 rados_obj.operate() r=0 bl.length=874 2023-10-11T14:29:43.350 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.347+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.351 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados_obj.operate() r=0 bl.length=420 2023-10-11T14:29:43.351 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 10 Using default name default 2023-10-11T14:29:43.351 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados_obj.operate() r=0 bl.length=874 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 zone default found 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 4 Realm: () 2023-10-11T14:29:43.352 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 4 ZoneGroup: default (b7373068-cb03-49c6-b37e-36f1b1d901a7) 2023-10-11T14:29:43.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 4 Zone: default (38ee82c0-0023-4c65-aa65-321b2c47b3db) 2023-10-11T14:29:43.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-11T14:29:43.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 zonegroup default 2023-10-11T14:29:43.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.348+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:43.353 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.349+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:43.354 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:43.349+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:45.340 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:45.336+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:45.340 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:45.336+0000 7f765bb69780 20 rados->read ofs=0 len=0 2023-10-11T14:29:45.341 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:45.337+0000 7f765bb69780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:45.341 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:45.337+0000 7f765bb69780 20 started sync module instance, tier type = 2023-10-11T14:29:45.342 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:45.337+0000 7f765bb69780 20 started zone id=38ee82c0-0023-4c65-aa65-321b2c47b3db (name=default) with tier type = 2023-10-11T14:29:47.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=0 2023-10-11T14:29:47.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=1 2023-10-11T14:29:47.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=2 2023-10-11T14:29:47.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=3 2023-10-11T14:29:47.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=4 2023-10-11T14:29:47.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=5 2023-10-11T14:29:47.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.376+0000 7f765bb69780 20 add_watcher() i=6 2023-10-11T14:29:47.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.377+0000 7f765bb69780 20 add_watcher() i=7 2023-10-11T14:29:47.383 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.377+0000 7f765bb69780 2 all 8 watchers are set, enabling cache 2023-10-11T14:29:47.383 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.377+0000 7f765bb69780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-10-11T14:29:47.383 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.377+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-10-11T14:29:47.383 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.377+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:47.384 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-10-11T14:29:47.384 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-10-11T14:29:47.384 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:47.384 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-10-11T14:29:47.384 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-10-11T14:29:47.385 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:47.385 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-10-11T14:29:47.385 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-10-11T14:29:47.385 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:47.386 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-10-11T14:29:47.386 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.378+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-10-11T14:29:47.387 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.379+0000 7f765bb69780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-10-11T14:29:47.387 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.382+0000 7f7640ae5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:47.387 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.382+0000 7f765bb69780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:47.387 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:47.382+0000 7f765bb69780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:50.346 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.343+0000 7f765bb69780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-10-11T14:29:50.346 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.343+0000 7f765bb69780 5 note: GC not initialized 2023-10-11T14:29:50.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.345+0000 7f765bb69780 20 init_complete bucket index max shards: 11 2023-10-11T14:29:50.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.345+0000 7f765bb69780 10 rgw notify: Started notification manager with: 1 workers 2023-10-11T14:29:50.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.345+0000 7f7639ad7700 20 reqs_thread_entry: start 2023-10-11T14:29:50.348 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.345+0000 7f763c2dc700 20 reqs_thread_entry: start 2023-10-11T14:29:50.349 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.346+0000 7f763a2d8700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 11 14:30:20 2023 2023-10-11T14:29:50.349 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-11T14:29:50.368 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.366+0000 7f7640ae5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:50.372 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.370+0000 7f765bb69780 20 remove_watcher() i=0 2023-10-11T14:29:50.372 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.370+0000 7f765bb69780 2 removed watcher, disabling cache 2023-10-11T14:29:50.375 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.372+0000 7f765bb69780 20 remove_watcher() i=1 2023-10-11T14:29:50.376 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.373+0000 7f765bb69780 20 remove_watcher() i=2 2023-10-11T14:29:50.377 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.374+0000 7f765bb69780 20 remove_watcher() i=3 2023-10-11T14:29:50.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.375+0000 7f765bb69780 20 remove_watcher() i=4 2023-10-11T14:29:50.379 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.376+0000 7f765bb69780 20 remove_watcher() i=5 2023-10-11T14:29:50.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.377+0000 7f765bb69780 20 remove_watcher() i=6 2023-10-11T14:29:50.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.378+0000 7f765bb69780 20 remove_watcher() i=7 2023-10-11T14:29:50.383 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2023-10-11T14:29:50.385 DEBUG:tasks.util.rgw: json result: [] 2023-10-11T14:29:50.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:50.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:50.387 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-11T14:29:50.440 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:50.440 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:50.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.447+0000 7fd52902d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:50.449 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.447+0000 7fd52902d780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:50.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.450+0000 7fd50a7a2700 20 reqs_thread_entry: start 2023-10-11T14:29:50.459 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:29:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "38ee82c0-0023-4c65-aa65-321b2c47b3db", 2023-10-11T14:29:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2023-10-11T14:29:50.459 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "default.rgw.meta:root", 2023-10-11T14:29:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "default.rgw.control", 2023-10-11T14:29:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-11T14:29:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-11T14:29:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "default.rgw.log", 2023-10-11T14:29:50.460 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-11T14:29:50.461 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "default.rgw.otp", 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "", 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "" 2023-10-11T14:29:50.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-11T14:29:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:29:50.464 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:29:50.465 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2023-10-11T14:29:50.466 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-11T14:29:50.466 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:29:50.467 DEBUG:tasks.util.rgw: json result: {'id': '38ee82c0-0023-4c65-aa65-321b2c47b3db', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-11T14:29:50.467 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-10-11T14:29:50.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-10-11T14:29:50.468 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-10-11T14:29:50.513 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:50.513 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:50.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.518+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.519+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.519+0000 7f5e05fcb780 20 realm 2023-10-11T14:29:50.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.519+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-11T14:29:50.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.523 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:50.524 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 rados_obj.operate() r=0 bl.length=890 2023-10-11T14:29:50.525 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.520+0000 7f5e05fcb780 20 searching for the correct realm 2023-10-11T14:29:50.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 RGWRados::pool_iterate: got zonegroup_info.b7373068-cb03-49c6-b37e-36f1b1d901a7 2023-10-11T14:29:50.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 RGWRados::pool_iterate: got zone_info.38ee82c0-0023-4c65-aa65-321b2c47b3db 2023-10-11T14:29:50.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 RGWRados::pool_iterate: got zone_names.default 2023-10-11T14:29:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-11T14:29:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.527+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados_obj.operate() r=0 bl.length=46 2023-10-11T14:29:50.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados_obj.operate() r=0 bl.length=420 2023-10-11T14:29:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 zone default found 2023-10-11T14:29:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 4 Realm: () 2023-10-11T14:29:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 4 ZoneGroup: default (b7373068-cb03-49c6-b37e-36f1b1d901a7) 2023-10-11T14:29:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 4 Zone: default (38ee82c0-0023-4c65-aa65-321b2c47b3db) 2023-10-11T14:29:50.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-11T14:29:50.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 zonegroup default 2023-10-11T14:29:50.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.534 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.528+0000 7f5e05fcb780 20 rados->read ofs=0 len=0 2023-10-11T14:29:50.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.529+0000 7f5e05fcb780 20 rados_obj.operate() r=-2 bl.length=0 2023-10-11T14:29:50.535 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.529+0000 7f5e05fcb780 20 started sync module instance, tier type = 2023-10-11T14:29:50.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.529+0000 7f5e05fcb780 20 started zone id=38ee82c0-0023-4c65-aa65-321b2c47b3db (name=default) with tier type = 2023-10-11T14:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=0 2023-10-11T14:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=1 2023-10-11T14:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=2 2023-10-11T14:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=3 2023-10-11T14:29:50.542 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=4 2023-10-11T14:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=5 2023-10-11T14:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.539+0000 7f5e05fcb780 20 add_watcher() i=6 2023-10-11T14:29:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.540+0000 7f5e05fcb780 20 add_watcher() i=7 2023-10-11T14:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.540+0000 7f5e05fcb780 2 all 8 watchers are set, enabling cache 2023-10-11T14:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.541+0000 7f5deaf47700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.541+0000 7f5e05fcb780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:50.544 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.541+0000 7f5e05fcb780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:50.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.545+0000 7f5e05fcb780 5 note: GC not initialized 2023-10-11T14:29:50.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.546+0000 7f5de673e700 20 reqs_thread_entry: start 2023-10-11T14:29:50.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.546+0000 7f5e05fcb780 20 init_complete bucket index max shards: 11 2023-10-11T14:29:50.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.546+0000 7f5e05fcb780 10 rgw notify: Started notification manager with: 1 workers 2023-10-11T14:29:50.549 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.546+0000 7f5de473a700 20 rgw notify: INFO: next queues processing will happen at: Wed Oct 11 14:30:20 2023 2023-10-11T14:29:50.550 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-11T14:29:50.550 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.546+0000 7f5de3f18700 20 reqs_thread_entry: start 2023-10-11T14:29:50.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.554+0000 7f5deaf47700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-11T14:29:50.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.555+0000 7f5e05fcb780 20 remove_watcher() i=0 2023-10-11T14:29:50.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.555+0000 7f5e05fcb780 2 removed watcher, disabling cache 2023-10-11T14:29:50.558 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.556+0000 7f5e05fcb780 20 remove_watcher() i=1 2023-10-11T14:29:50.559 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.557+0000 7f5e05fcb780 20 remove_watcher() i=2 2023-10-11T14:29:50.560 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.558+0000 7f5e05fcb780 20 remove_watcher() i=3 2023-10-11T14:29:50.561 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.559+0000 7f5e05fcb780 20 remove_watcher() i=4 2023-10-11T14:29:50.562 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.559+0000 7f5e05fcb780 20 remove_watcher() i=5 2023-10-11T14:29:50.563 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.560+0000 7f5e05fcb780 20 remove_watcher() i=6 2023-10-11T14:29:50.564 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.561+0000 7f5e05fcb780 20 remove_watcher() i=7 2023-10-11T14:29:50.566 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2023-10-11T14:29:50.568 DEBUG:tasks.util.rgw: json result: [] 2023-10-11T14:29:50.568 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:50.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-10-11T14:29:50.569 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-10-11T14:29:50.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:50.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:50.626 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.624+0000 7f0981a8b780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-11T14:29:50.626 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.624+0000 7f0981a8b780 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-11T14:29:50.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:29:50.627+0000 7f0963200700 20 reqs_thread_entry: start 2023-10-11T14:29:50.635 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:29:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "38ee82c0-0023-4c65-aa65-321b2c47b3db", 2023-10-11T14:29:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2023-10-11T14:29:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "default.rgw.meta:root", 2023-10-11T14:29:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "default.rgw.control", 2023-10-11T14:29:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-11T14:29:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-11T14:29:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "default.rgw.log", 2023-10-11T14:29:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-11T14:29:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-11T14:29:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-11T14:29:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-11T14:29:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-11T14:29:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-11T14:29:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-11T14:29:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-11T14:29:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "default.rgw.otp", 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "", 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "" 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:29:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:29:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:29:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:29:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-11T14:29:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:29:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "default.rgw.buckets.data", 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-11T14:29:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:29:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:29:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:29:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:29:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "", 2023-10-11T14:29:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-11T14:29:50.643 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:29:50.644 DEBUG:tasks.util.rgw: json result: {'id': '38ee82c0-0023-4c65-aa65-321b2c47b3db', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-11T14:29:50.644 INFO:tasks.rgw:Starting rgw... 2023-10-11T14:29:50.644 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-10-11T14:29:50.645 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-11T14:29:50.645 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-11T14:29:50.645 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-11T14:29:50.648 INFO:tasks.rgw.c1.client.0:Started 2023-10-11T14:29:50.649 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-10-11T14:29:50.649 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-11T14:29:50.649 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T14:29:50.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T14:29:50.652 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T14:29:50.652 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-10-11T14:29:50.652 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-11T14:29:50.653 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-11T14:29:50.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-11T14:29:50.656 INFO:tasks.rgw.c2.client.0:Started 2023-10-11T14:29:50.656 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-10-11T14:29:50.656 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-11T14:29:50.657 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-11T14:29:50.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-11T14:29:50.659 INFO:tasks.rgw.c2.client.1:Started 2023-10-11T14:29:50.659 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi158.front.sepia.ceph.com:8000/ 2023-10-11T14:29:50.660 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8000/ 2023-10-11T14:29:50.715 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:29:50.715 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:29:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T14:29:50.716 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8000: Connection refused 2023-10-11T14:29:51.718 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8000/ 2023-10-11T14:29:51.740 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:29:51.741 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:29:51.741 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:29:51.742 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T14:29:51.743 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi158.front.sepia.ceph.com:8001/ 2023-10-11T14:29:51.743 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001/ 2023-10-11T14:29:51.805 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:29:51.805 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:29:51.806 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:29:51.807 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T14:29:51.807 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8000/ 2023-10-11T14:29:51.807 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000/ 2023-10-11T14:29:51.842 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:29:51.843 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:29:51.843 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:29:51.844 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2023-10-11T14:29:51.844 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8001/ 2023-10-11T14:29:51.845 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001/ 2023-10-11T14:29:51.906 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:29:51.906 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:29:51.907 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:29:51.908 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2023-10-11T14:29:51.908 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-10-11T14:29:51.962 INFO:tasks.rgw_multisite:creating realm.. 2023-10-11T14:29:51.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:51.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:51.963 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:29:52.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:52.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:52.035 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:29:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "test-realm", 2023-10-11T14:29:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "current_period": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:29:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T14:29:52.036 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:52.038 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-10-11T14:29:52.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:52.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:52.039 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:29:52.086 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:52.086 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:52.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:52.097+0000 7f6e97c22780 -1 old format 2023-10-11T14:29:52.107 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:52.107 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:29:52.107 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:29:52.107 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:29:52.107 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:29:52.108 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:29:52.108 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:29:52.108 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:52.108 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:29:52.108 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "", 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [], 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:29:52.109 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:29:52.110 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [] 2023-10-11T14:29:52.110 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:52.110 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:52.110 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:29:52.110 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:29:52.111 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:29:52.111 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:29:52.111 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:52.111 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:29:52.111 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:29:52.112 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:29:52.112 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:52.113 INFO:tasks.rgw_multisite:creating master zone.. 2023-10-11T14:29:52.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi158.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:52.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi158.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:52.114 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --endpoints http://smithi158.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:29:52.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:52.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:52.222 INFO:teuthology.orchestra.run.smithi158.stderr:NOTICE: set zone's realm_id=17c10fd9-3252-456e-81e5-d40bac926339 2023-10-11T14:29:52.236 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:29:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:29:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-11T14:29:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a1.rgw.control", 2023-10-11T14:29:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-11T14:29:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-11T14:29:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a1.rgw.log", 2023-10-11T14:29:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-11T14:29:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-11T14:29:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a1.rgw.otp", 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:29:52.241 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:29:52.241 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:29:52.241 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:29:52.241 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-11T14:29:52.241 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:29:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:29:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a1.rgw.buckets.data" 2023-10-11T14:29:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:52.242 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:52.243 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:29:52.244 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-11T14:29:52.244 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:52.245 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-10-11T14:29:53.131 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'a1.rgw.buckets.data' created 2023-10-11T14:29:53.146 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-10-11T14:29:54.150 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-10-11T14:29:54.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:54.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:54.171 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:29:54.225 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:54.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:54.244 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a1.rgw.meta:root", 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a1.rgw.control", 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a1.rgw.log:gc", 2023-10-11T14:29:54.245 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a1.rgw.log:lc", 2023-10-11T14:29:54.246 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a1.rgw.log", 2023-10-11T14:29:54.246 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-10-11T14:29:54.246 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-10-11T14:29:54.246 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-10-11T14:29:54.246 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-10-11T14:29:54.247 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-10-11T14:29:54.247 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-10-11T14:29:54.247 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-10-11T14:29:54.247 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-10-11T14:29:54.247 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a1.rgw.otp", 2023-10-11T14:29:54.248 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:29:54.248 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:29:54.248 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:29:54.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a1.rgw.buckets.index", 2023-10-11T14:29:54.249 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:29:54.250 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:29:54.250 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a1.rgw.buckets.data", 2023-10-11T14:29:54.250 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:29:54.250 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:54.250 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:29:54.251 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:29:54.252 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:29:54.252 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a1.rgw.log:notif" 2023-10-11T14:29:54.252 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:29:54.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:54.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:29:54.254 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:29:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:29:54.302 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:29:54.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:29:54.313+0000 7fcabe302780 0 period (3707f754-0ea4-480f-82c9-e0395881372c does not have zone daa316e9-869c-4074-a756-a8adfb4a3573 configured 2023-10-11T14:30:01.202 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:01.202 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:01.202 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1, 2023-10-11T14:30:01.202 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:30:01.203 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:30:01.203 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:30:01.203 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:01.203 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:30:01.203 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:01.204 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:30:01.204 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:30:01.204 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:30:01.204 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:30:01.205 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:30:01.205 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:01.205 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:01.205 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:30:01.206 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:30:01.206 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:01.206 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:30:01.206 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:01.207 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:01.207 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:30:01.207 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:30:01.207 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:01.207 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": false, 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:30:01.208 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:30:01.209 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:30:01.209 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:30:01.209 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:30:01.209 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:30:01.210 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:30:01.210 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:01.210 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:01.210 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:30:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:30:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:30:01.211 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:30:01.212 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:30:01.212 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:01.212 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:01.212 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:01.212 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:30:01.213 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:30:01.214 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:01.214 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:01.214 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:01.214 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:30:01.215 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:01.215 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:01.215 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:30:01.215 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:01.215 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:01.216 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.216 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:30:01.216 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:01.216 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:30:01.216 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:01.217 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.218 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:30:01.218 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:01.218 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:01.218 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:01.218 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:01.219 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:01.220 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:30:01.221 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:01.222 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:01.222 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:01.222 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:01.222 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:01.222 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:01.223 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:01.223 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:01.223 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:30:01.223 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:01.223 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:01.225 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-11T14:30:01.225 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T14:30:01.226 INFO:tasks.rgw.c1.client.0.smithi158.stdout:2023-10-11T14:30:01.223+0000 7f9728749700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 119149) UID: 0 2023-10-11T14:30:01.226 INFO:tasks.rgw.c1.client.0.smithi158.stdout:2023-10-11T14:30:01.223+0000 7f972c9b27c0 -1 shutting down 2023-10-11T14:30:03.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-11T14:30:07.326 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-11T14:30:07.327 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-10-11T14:30:07.327 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-10-11T14:30:07.330 INFO:tasks.rgw.c1.client.0:Started 2023-10-11T14:30:07.330 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8000 2023-10-11T14:30:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T14:30:07.357 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:07.357 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:07.357 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8000: Connection refused 2023-10-11T14:30:08.358 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8000 2023-10-11T14:30:08.384 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:08.384 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:08.386 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:30:08.386 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T14:30:08.387 INFO:tasks.rgw_multisite:creating admin user.. 2023-10-11T14:30:08.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:08.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:08.388 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:08.489 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:08.489 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:08.523 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:08.523 INFO:teuthology.orchestra.run.smithi158.stdout: "user_id": "realm-admin", 2023-10-11T14:30:08.523 INFO:teuthology.orchestra.run.smithi158.stdout: "display_name": "Realm Admin", 2023-10-11T14:30:08.524 INFO:teuthology.orchestra.run.smithi158.stdout: "email": "", 2023-10-11T14:30:08.524 INFO:teuthology.orchestra.run.smithi158.stdout: "suspended": 0, 2023-10-11T14:30:08.524 INFO:teuthology.orchestra.run.smithi158.stdout: "max_buckets": 1000, 2023-10-11T14:30:08.524 INFO:teuthology.orchestra.run.smithi158.stdout: "subusers": [], 2023-10-11T14:30:08.524 INFO:teuthology.orchestra.run.smithi158.stdout: "keys": [ 2023-10-11T14:30:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "realm-admin", 2023-10-11T14:30:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:08.525 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:08.526 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:08.526 INFO:teuthology.orchestra.run.smithi158.stdout: "swift_keys": [], 2023-10-11T14:30:08.526 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": [], 2023-10-11T14:30:08.526 INFO:teuthology.orchestra.run.smithi158.stdout: "op_mask": "read, write, delete", 2023-10-11T14:30:08.526 INFO:teuthology.orchestra.run.smithi158.stdout: "system": true, 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "", 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "default_storage_class": "", 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_tags": [], 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:08.527 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:08.528 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:08.528 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:08.528 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:08.528 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:08.528 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:30:08.529 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:08.529 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:08.529 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:08.529 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:08.529 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout: "temp_url_keys": [], 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "rgw", 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout: "mfa_ids": [] 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:08.530 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:30:08.535 INFO:tasks.rgw_multisite:creating zone.. 2023-10-11T14:30:08.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi158.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:08.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi158.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:08.536 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --endpoints http://smithi158.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:08.590 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:08.590 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:08.605 INFO:teuthology.orchestra.run.smithi158.stderr:NOTICE: set zone's realm_id=17c10fd9-3252-456e-81e5-d40bac926339 2023-10-11T14:30:08.618 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:08.619 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:08.619 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:30:08.619 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:30:08.619 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:30:08.620 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:30:08.621 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:30:08.621 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:30:08.621 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:30:08.621 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:30:08.621 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:30:08.622 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:30:08.622 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:30:08.622 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:08.622 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:08.622 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:08.623 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:30:08.623 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:08.623 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:30:08.623 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:08.623 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:30:08.624 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:30:08.624 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:30:08.624 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data" 2023-10-11T14:30:08.624 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:08.624 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:08.625 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:08.626 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:08.626 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:30:08.626 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:08.627 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-10-11T14:30:09.029 INFO:teuthology.orchestra.run.smithi158.stderr:pool 'a2.rgw.buckets.data' created 2023-10-11T14:30:09.045 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-10-11T14:30:09.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-11T14:30:10.047 INFO:teuthology.orchestra.run.smithi158.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-10-11T14:30:10.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:10.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:10.065 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:10.119 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:10.119 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:10.138 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:10.138 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:30:10.139 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:30:10.140 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:30:10.140 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:30:10.140 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:30:10.140 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:30:10.140 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:30:10.141 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:30:10.141 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:30:10.141 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:30:10.141 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:30:10.141 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:30:10.142 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:10.142 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:10.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:10.142 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:30:10.142 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:30:10.143 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-11T14:30:10.144 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:30:10.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:10.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:10.144 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:30:10.144 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:30:10.145 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:30:10.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:10.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:10.145 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:10.145 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:10.146 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:30:10.146 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:10.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:10.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:10.147 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:10.197 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:10.197 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:10.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:30:10.206+0000 7f9112d23780 0 period (064106cf-4523-427d-9a69-628884ec70af does not have zone 37b32c30-76c0-479d-b458-e4824d081f13 configured 2023-10-11T14:30:17.096 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:30:17.114 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:17.114 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:17.115 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2, 2023-10-11T14:30:17.115 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:30:17.115 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:30:17.115 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:30:17.115 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:17.116 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:30:17.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.116 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:30:17.116 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.117 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:30:17.118 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:30:17.118 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:17.118 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:30:17.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.119 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:17.119 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:30:17.119 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:30:17.119 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:30:17.120 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.120 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:30:17.120 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:30:17.120 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:17.120 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:30:17.121 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:30:17.121 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:30:17.121 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:30:17.121 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:30:17.121 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.122 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:17.123 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:30:17.123 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:30:17.123 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:17.123 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.124 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:30:17.124 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:30:17.124 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:17.124 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:30:17.125 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:30:17.125 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:30:17.125 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:30:17.125 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:30:17.126 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:30:17.126 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:30:17.126 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:30:17.126 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:17.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:17.127 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.127 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:30:17.127 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.127 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:30:17.128 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:30:17.128 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:30:17.128 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:30:17.128 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:17.128 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:17.129 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.129 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:30:17.129 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:17.129 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:30:17.130 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:30:17.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.130 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:30:17.130 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:30:17.130 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:17.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:17.131 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:17.131 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:30:17.131 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.131 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:17.132 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:30:17.132 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:17.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:30:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:30:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:30:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:30:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:30:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:30:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:30:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:30:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:30:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:30:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:30:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:30:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:30:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:17.142 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:17.145 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T14:30:17.145 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T14:30:17.146 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T14:30:17.144+0000 7ff42a694700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 119158) UID: 0 2023-10-11T14:30:17.146 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T14:30:17.144+0000 7ff42e8fd7c0 -1 shutting down 2023-10-11T14:30:20.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T14:30:23.247 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T14:30:23.248 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T14:30:23.249 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T14:30:23.252 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T14:30:23.252 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T14:30:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T14:30:23.279 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:23.279 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:23.279 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T14:30:24.280 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T14:30:24.303 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:24.303 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:24.304 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:30:24.305 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T14:30:24.305 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-11T14:30:24.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.306 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi158.front.sepia.ceph.com:8000 --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:24.368 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:24.368 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:24.418 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-realm", 2023-10-11T14:30:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "current_period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 2 2023-10-11T14:30:24.420 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:24.421 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-10-11T14:30:24.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.422 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:24.470 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:24.471 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:24.484 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:30:24.482+0000 7feab9bb0780 -1 old format 2023-10-11T14:30:24.494 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:24.494 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "b", 2023-10-11T14:30:24.495 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b", 2023-10-11T14:30:24.495 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2023-10-11T14:30:24.495 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": false, 2023-10-11T14:30:24.495 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:24.495 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:30:24.496 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:24.496 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2023-10-11T14:30:24.496 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2023-10-11T14:30:24.496 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "", 2023-10-11T14:30:24.496 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [], 2023-10-11T14:30:24.497 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2023-10-11T14:30:24.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:24.497 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2023-10-11T14:30:24.497 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2023-10-11T14:30:24.497 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [] 2023-10-11T14:30:24.498 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:24.498 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:24.498 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2023-10-11T14:30:24.498 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:24.498 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled_features": [ 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:24.499 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:24.500 INFO:tasks.rgw_multisite:creating zone.. 2023-10-11T14:30:24.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:24.501 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --endpoints http://smithi172.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:24.596 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:24.596 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:24.610 INFO:teuthology.orchestra.run.smithi172.stderr:NOTICE: set zone's realm_id=17c10fd9-3252-456e-81e5-d40bac926339 2023-10-11T14:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b1", 2023-10-11T14:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-11T14:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "b1.rgw.control", 2023-10-11T14:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-11T14:30:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "b1.rgw.log", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-11T14:30:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "b1.rgw.otp", 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:30:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:30:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:30:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-11T14:30:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:30:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:30:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "b1.rgw.buckets.data" 2023-10-11T14:30:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-11T14:30:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:30:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:30:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-11T14:30:24.637 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:24.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-10-11T14:30:25.034 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'b1.rgw.buckets.data' created 2023-10-11T14:30:25.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-10-11T14:30:26.053 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-10-11T14:30:26.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:26.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:26.072 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:26.127 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:26.127 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:26.145 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:26.145 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:26.145 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b1", 2023-10-11T14:30:26.145 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "b1.rgw.meta:root", 2023-10-11T14:30:26.146 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "b1.rgw.control", 2023-10-11T14:30:26.146 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "b1.rgw.log:gc", 2023-10-11T14:30:26.146 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "b1.rgw.log:lc", 2023-10-11T14:30:26.146 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "b1.rgw.log", 2023-10-11T14:30:26.146 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-10-11T14:30:26.147 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-10-11T14:30:26.147 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-10-11T14:30:26.147 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-10-11T14:30:26.147 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-10-11T14:30:26.147 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-10-11T14:30:26.148 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-10-11T14:30:26.148 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-10-11T14:30:26.148 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "b1.rgw.otp", 2023-10-11T14:30:26.148 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:30:26.148 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:30:26.149 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:30:26.150 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "b1.rgw.buckets.index", 2023-10-11T14:30:26.150 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:30:26.150 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:30:26.150 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "b1.rgw.buckets.data", 2023-10-11T14:30:26.150 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2023-10-11T14:30:26.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:26.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:26.151 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-10-11T14:30:26.151 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:30:26.151 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:30:26.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:26.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:26.152 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:26.152 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:26.152 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "b1.rgw.log:notif" 2023-10-11T14:30:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:26.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:26.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:26.154 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:26.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T14:30:26.202 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:26.202 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:26.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:30:26.211+0000 7f56a7bfe780 0 period (064106cf-4523-427d-9a69-628884ec70af does not have zone 02f9727d-0de9-432b-942a-b3bdfbfa6b9c configured 2023-10-11T14:30:33.115 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:30:33.148 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:33.148 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 3, 2023-10-11T14:30:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:30:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2023-10-11T14:30:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2023-10-11T14:30:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:33.150 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2023-10-11T14:30:33.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.150 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "a", 2023-10-11T14:30:33.150 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a", 2023-10-11T14:30:33.150 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": true, 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2023-10-11T14:30:33.151 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2023-10-11T14:30:33.152 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:33.152 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2023-10-11T14:30:33.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.152 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:33.152 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a2", 2023-10-11T14:30:33.153 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:33.153 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:30:33.153 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.153 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:33.153 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:33.154 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a1", 2023-10-11T14:30:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:33.156 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:33.157 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:33.158 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.159 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2023-10-11T14:30:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2023-10-11T14:30:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2023-10-11T14:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2023-10-11T14:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled_features": [ 2023-10-11T14:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "b", 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b", 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2023-10-11T14:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": false, 2023-10-11T14:30:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:30:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2023-10-11T14:30:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2023-10-11T14:30:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2023-10-11T14:30:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b1", 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": false, 2023-10-11T14:30:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2023-10-11T14:30:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2023-10-11T14:30:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2023-10-11T14:30:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2023-10-11T14:30:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2023-10-11T14:30:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2023-10-11T14:30:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2023-10-11T14:30:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled_features": [ 2023-10-11T14:30:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2023-10-11T14:30:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2025758522 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 3270403816 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2663201345 2023-10-11T14:30:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "a", 2023-10-11T14:30:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2023-10-11T14:30:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2023-10-11T14:30:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-11T14:30:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-11T14:30:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-11T14:30:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-11T14:30:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-11T14:30:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2023-10-11T14:30:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-11T14:30:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-11T14:30:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-11T14:30:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "user_ratelimit": { 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_ratelimit": { 2023-10-11T14:30:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "anonymous_ratelimit": { 2023-10-11T14:30:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "test-realm", 2023-10-11T14:30:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:30:33.184 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:33.185 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-11T14:30:33.185 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T14:30:33.186 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2023-10-11T14:30:33.183+0000 7f408e574700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 119938) UID: 0 2023-10-11T14:30:33.186 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2023-10-11T14:30:33.183+0000 7f40927dd7c0 -1 shutting down 2023-10-11T14:30:36.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-11T14:30:39.287 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-11T14:30:39.287 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-10-11T14:30:39.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-10-11T14:30:39.291 INFO:tasks.rgw.c2.client.0:Started 2023-10-11T14:30:39.291 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2023-10-11T14:30:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T14:30:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8000: Connection refused 2023-10-11T14:30:40.319 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2023-10-11T14:30:40.342 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:40.342 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:40.343 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2023-10-11T14:30:40.344 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-10-11T14:30:40.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:40.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--url', 'http://smithi158.front.sepia.ceph.com:8000', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:40.345 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi158.front.sepia.ceph.com:8000 --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --url http://smithi158.front.sepia.ceph.com:8000 --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:40.444 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:40.444 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:40.486 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:40.486 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:40.487 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "test-realm", 2023-10-11T14:30:40.487 INFO:teuthology.orchestra.run.smithi172.stdout: "current_period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:40.487 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 2 2023-10-11T14:30:40.487 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:40.488 INFO:tasks.rgw_multisite:creating zone.. 2023-10-11T14:30:40.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:40.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:40.489 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --endpoints http://smithi172.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:40.542 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:40.543 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:40.557 INFO:teuthology.orchestra.run.smithi172.stderr:NOTICE: set zone's realm_id=17c10fd9-3252-456e-81e5-d40bac926339 2023-10-11T14:30:40.570 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:40.571 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:30:40.571 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b2", 2023-10-11T14:30:40.571 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-11T14:30:40.571 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "b2.rgw.control", 2023-10-11T14:30:40.571 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-11T14:30:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-11T14:30:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "b2.rgw.log", 2023-10-11T14:30:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-11T14:30:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-11T14:30:40.572 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-11T14:30:40.573 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "b2.rgw.otp", 2023-10-11T14:30:40.574 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:30:40.574 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:40.574 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:40.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:40.575 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:30:40.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:40.575 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:30:40.575 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:30:40.575 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-11T14:30:40.576 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:30:40.576 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:30:40.576 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "b2.rgw.buckets.data" 2023-10-11T14:30:40.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:40.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:40.577 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-11T14:30:40.577 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:30:40.577 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:30:40.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:40.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:40.578 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:40.578 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:40.578 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-11T14:30:40.578 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:40.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-10-11T14:30:40.973 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'b2.rgw.buckets.data' created 2023-10-11T14:30:40.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-10-11T14:30:41.998 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-10-11T14:30:42.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:42.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:42.021 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:42.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-11T14:30:42.079 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:42.079 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:42.097 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:30:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b2", 2023-10-11T14:30:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "b2.rgw.meta:root", 2023-10-11T14:30:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "b2.rgw.control", 2023-10-11T14:30:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "b2.rgw.log:gc", 2023-10-11T14:30:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "b2.rgw.log:lc", 2023-10-11T14:30:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "b2.rgw.log", 2023-10-11T14:30:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-10-11T14:30:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-10-11T14:30:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-10-11T14:30:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-10-11T14:30:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-10-11T14:30:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-10-11T14:30:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-10-11T14:30:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-10-11T14:30:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "b2.rgw.otp", 2023-10-11T14:30:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2023-10-11T14:30:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:30:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:30:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2023-10-11T14:30:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2023-10-11T14:30:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:30:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "b2.rgw.buckets.index", 2023-10-11T14:30:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2023-10-11T14:30:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2023-10-11T14:30:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "b2.rgw.buckets.data", 2023-10-11T14:30:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "compression_type": "random" 2023-10-11T14:30:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0, 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "inline_data": true 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "b2.rgw.log:notif" 2023-10-11T14:30:42.105 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:42.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:42.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:42.107 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:42.155 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:42.155 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:42.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:30:42.165+0000 7f76296e7780 0 period (064106cf-4523-427d-9a69-628884ec70af does not have zone d3375114-0016-4301-a85b-ef71f68a1016 configured 2023-10-11T14:30:49.034 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:30:49.064 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:30:49.065 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:49.065 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 4, 2023-10-11T14:30:49.065 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:30:49.065 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2023-10-11T14:30:49.065 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2023-10-11T14:30:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2023-10-11T14:30:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "a", 2023-10-11T14:30:49.067 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a", 2023-10-11T14:30:49.067 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2023-10-11T14:30:49.067 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": true, 2023-10-11T14:30:49.067 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2023-10-11T14:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2023-10-11T14:30:49.068 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:49.069 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2023-10-11T14:30:49.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.069 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:49.069 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a2", 2023-10-11T14:30:49.069 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.070 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:30:49.070 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.070 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:49.070 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:49.070 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:49.071 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:49.071 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:49.071 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:49.073 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:49.073 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:49.073 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:49.073 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:49.074 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.075 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:49.075 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "a1", 2023-10-11T14:30:49.075 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.075 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:30:49.075 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.076 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:49.076 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:49.076 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:49.076 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:49.076 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:49.077 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:49.077 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:49.077 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:49.077 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:49.077 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:49.078 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.078 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.078 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.079 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2023-10-11T14:30:49.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.079 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2023-10-11T14:30:49.079 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2023-10-11T14:30:49.079 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2023-10-11T14:30:49.080 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2023-10-11T14:30:49.080 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.080 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.080 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.081 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled_features": [ 2023-10-11T14:30:49.082 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.082 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.083 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "b", 2023-10-11T14:30:49.083 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b", 2023-10-11T14:30:49.083 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "", 2023-10-11T14:30:49.083 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": false, 2023-10-11T14:30:49.083 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.084 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:30:49.084 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.084 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2023-10-11T14:30:49.085 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2023-10-11T14:30:49.085 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:49.085 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2023-10-11T14:30:49.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.086 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:49.086 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b1", 2023-10-11T14:30:49.086 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.086 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "b2", 2023-10-11T14:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [ 2023-10-11T14:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": false, 2023-10-11T14:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": true, 2023-10-11T14:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": false, 2023-10-11T14:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2023-10-11T14:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": true, 2023-10-11T14:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2023-10-11T14:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "", 2023-10-11T14:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "supported_features": [ 2023-10-11T14:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "compress-encrypted", 2023-10-11T14:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2023-10-11T14:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2023-10-11T14:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2023-10-11T14:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2023-10-11T14:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2023-10-11T14:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2023-10-11T14:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2023-10-11T14:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2023-10-11T14:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled_features": [ 2023-10-11T14:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "resharding" 2023-10-11T14:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2025758522 2023-10-11T14:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 3270403816 2023-10-11T14:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 178712359 2023-10-11T14:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 2663201345 2023-10-11T14:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "a", 2023-10-11T14:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2023-10-11T14:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2023-10-11T14:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-11T14:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-11T14:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-11T14:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-11T14:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-11T14:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2023-10-11T14:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-11T14:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-11T14:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-11T14:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-11T14:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-11T14:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "user_ratelimit": { 2023-10-11T14:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_ratelimit": { 2023-10-11T14:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "anonymous_ratelimit": { 2023-10-11T14:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2023-10-11T14:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2023-10-11T14:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2023-10-11T14:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2023-10-11T14:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2023-10-11T14:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "test-realm", 2023-10-11T14:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:30:49.112 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-11T14:30:49.113 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T14:30:49.114 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2023-10-11T14:30:49.111+0000 7f04c1291700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 119942) UID: 0 2023-10-11T14:30:49.114 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2023-10-11T14:30:49.111+0000 7f04c54fa7c0 -1 shutting down 2023-10-11T14:30:53.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-11T14:30:55.214 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-11T14:30:55.215 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-10-11T14:30:55.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-10-11T14:30:55.218 INFO:tasks.rgw.c2.client.1:Started 2023-10-11T14:30:55.219 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2023-10-11T14:30:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T14:30:55.244 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T14:30:56.246 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2023-10-11T14:30:56.268 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T14:30:56.268 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-11T14:30:56.269 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T14:30:56.270 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2023-10-11T14:30:56.270 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-10-11T14:30:56.271 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-10-11T14:30:56.283 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi 2023-10-11T14:30:56.311 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-10-11T14:30:56.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:56.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:30:56.312 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:30:56.379 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:56.379 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:56.414 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:56.414 INFO:teuthology.orchestra.run.smithi158.stdout: "user_id": "rgw-multisite-test-user", 2023-10-11T14:30:56.415 INFO:teuthology.orchestra.run.smithi158.stdout: "display_name": "Multisite Test User", 2023-10-11T14:30:56.415 INFO:teuthology.orchestra.run.smithi158.stdout: "email": "", 2023-10-11T14:30:56.415 INFO:teuthology.orchestra.run.smithi158.stdout: "suspended": 0, 2023-10-11T14:30:56.415 INFO:teuthology.orchestra.run.smithi158.stdout: "max_buckets": 1000, 2023-10-11T14:30:56.415 INFO:teuthology.orchestra.run.smithi158.stdout: "subusers": [], 2023-10-11T14:30:56.416 INFO:teuthology.orchestra.run.smithi158.stdout: "keys": [ 2023-10-11T14:30:56.416 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.416 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "rgw-multisite-test-user", 2023-10-11T14:30:56.416 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "MWUFUL61ESHN8BNC6ISZ", 2023-10-11T14:30:56.416 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "atVyFeHFI1js1KTYQmBZ9YXskWmkr5giCZDdHoLM" 2023-10-11T14:30:56.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.417 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:56.417 INFO:teuthology.orchestra.run.smithi158.stdout: "swift_keys": [], 2023-10-11T14:30:56.417 INFO:teuthology.orchestra.run.smithi158.stdout: "caps": [ 2023-10-11T14:30:56.417 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.418 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "roles", 2023-10-11T14:30:56.418 INFO:teuthology.orchestra.run.smithi158.stdout: "perm": "*" 2023-10-11T14:30:56.418 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.418 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:30:56.418 INFO:teuthology.orchestra.run.smithi158.stdout: "op_mask": "read, write, delete", 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "", 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "default_storage_class": "", 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_tags": [], 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:56.419 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:56.420 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:56.420 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:56.420 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:56.420 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.420 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:30:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.422 INFO:teuthology.orchestra.run.smithi158.stdout: "temp_url_keys": [], 2023-10-11T14:30:56.422 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "rgw", 2023-10-11T14:30:56.422 INFO:teuthology.orchestra.run.smithi158.stdout: "mfa_ids": [] 2023-10-11T14:30:56.422 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:56.422 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:30:56.427 INFO:rgw_multi.tests:meta checkpoint 2023-10-11T14:30:56.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:30:56.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:30:56.428 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:30:56.477 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:56.477 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:56.504 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:30:56.515 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:30:56.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:30:56.516 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:30:56.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.516 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.517 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:30:56.517 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.517 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.517 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.517 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:30:56.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.518 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.518 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.518 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:30:56.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.519 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:30:56.520 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:30:56.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:30:56.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:30:56.521 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:30:56.569 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:30:56.570 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:30:56.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:30:56.592+0000 7f63c7dbc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:30:56.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:30:56.593+0000 7f63c7dbc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:30:56.606 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:30:56.607 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:30:56.607 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:30:56.607 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:30:56.607 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:30:56.608 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:30:56.608 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:56.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.608 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:30:56.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:30:56.609 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:56.609 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:30:56.609 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.609 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:30:56.610 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:30:56.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:30:56.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:30:56.610 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:56.610 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.611 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.611 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.611 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:30:56.611 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:56.611 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:30:56.612 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.612 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:30:56.612 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:30:56.612 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:30:56.612 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:30:56.613 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:56.613 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.613 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.613 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:30:56.614 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:56.614 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:30:56.614 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.614 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:30:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:30:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:30:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:30:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:56.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:30:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:30:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:30:56.616 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:30:56.617 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:30:56.618 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:30:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:30:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:30:56.619 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:30:56Z" 2023-10-11T14:30:56.619 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:30:56.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:30:59.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-11T14:31:01.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:01.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:01.622 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:01.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:01.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:01.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:01.706+0000 7fc4b72e9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:01.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:01.707+0000 7fc4b72e9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:01.721 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:01.722 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:01.722 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:01.722 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:01.722 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:01.723 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:01.723 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:01.723 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.723 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:01.723 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:01.724 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:01.725 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:01.725 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:01.725 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:01.725 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:01.725 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:01.726 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:01.727 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:01.727 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:01.727 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:01.727 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:01.727 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:01.728 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:01.728 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:01.728 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:01.728 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:01.729 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:01.729 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:01.729 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:01.729 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:01.729 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:01.730 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:01.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:01.730 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.730 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:01.730 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:01.731 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:01.732 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:01.732 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:01.732 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:01.732 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:01.732 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:01.733 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:01Z" 2023-10-11T14:31:01.734 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:01.735 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:06.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:06.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:06.737 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:06.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:06.821+0000 7fbf2ad22780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:06.822+0000 7fbf2ad22780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:06.837 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:06.837 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:06.837 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:06.837 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:06.837 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:06.838 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:06.838 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:06.838 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.838 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:06.839 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:06.840 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:06.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.841 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:06.841 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:06.841 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:06.841 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:06.842 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:06.843 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:06.843 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.843 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:06.843 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:06.843 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:06.844 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:06.845 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:06.845 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:06.845 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.845 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:06.845 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:06.846 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:06.847 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:06.847 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:06.847 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:06.847 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:06.847 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:06.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.848 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:06.848 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:06.848 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:06.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:06.849 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:06Z" 2023-10-11T14:31:06.849 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:06.850 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:11.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:11.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:11.852 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:11.909 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:11.909 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:11.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:11.935+0000 7fad003d1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:11.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:11.936+0000 7fad003d1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:11.950 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:11.951 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:11.951 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:11.951 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:11.951 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:11.951 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:11.952 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:11.952 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.952 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:11.952 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:11.952 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:11.953 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:11.954 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:11.954 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:11.954 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:11.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.954 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:11.955 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:11.955 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:11.955 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:11.955 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:11.955 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:11.956 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.957 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:11.957 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:11.957 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:11.957 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:11.957 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:11.958 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:11.958 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:11.958 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:11.958 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:11.958 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:11.959 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:11.960 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:11.960 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:11.960 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:11.960 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:11.960 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:11.961 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:11.961 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:11.961 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:11.961 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:11.961 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.962 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:11.962 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:11.962 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:11.962 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:11.962 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:11Z" 2023-10-11T14:31:11.963 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:11.963 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:16.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:16.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:16.965 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:17.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:17.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:17.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:17.049+0000 7fac4c735780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:17.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:17.051+0000 7fac4c735780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:17.065 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:17.066 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:17.066 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:17.066 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:17.066 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:17.067 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:17.068 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:17.068 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:17.068 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:17.068 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:17.068 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:17.069 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:17.069 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:17.069 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:17.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:17.069 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.070 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:17.070 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:17.070 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:17.070 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:17.070 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:17.071 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:17.072 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:17.073 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:17.073 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:17.073 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:17.073 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:17.073 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:17.074 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:17.074 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.074 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:17.074 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:17.075 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:17.075 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:17.075 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:17.075 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:17.075 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:17.076 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:17.076 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:17.076 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:17.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:17.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:17.077 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:17.078 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:17Z" 2023-10-11T14:31:17.078 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:17.079 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:22.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:22.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:22.081 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:22.138 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:22.138 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:22.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:22.170+0000 7f6d9be45780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:22.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:22.171+0000 7f6d9be45780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:22.185 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:22.185 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:22.185 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:22.186 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:22.186 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:22.186 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:22.186 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:22.187 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:22.188 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:22.188 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:22.188 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:22.188 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:22.188 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:22.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:22.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:22.191 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:22.192 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:22.193 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:22.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:22.193 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.193 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:22.193 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:22.194 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:22.194 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:22.194 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:22.194 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:22.194 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:22.195 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:22.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.196 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:22.196 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:22.196 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:22.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:22.197 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:22Z" 2023-10-11T14:31:22.197 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:22.198 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:27.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:27.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:27.200 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:27.257 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:27.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:27.284+0000 7fcee7d66780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:27.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:27.286+0000 7fcee7d66780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:27.299 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:27.299 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:27Z" 2023-10-11T14:31:27.312 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:27.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:32.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:32.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:32.315 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:32.373 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:32.374 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:32.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:32.400+0000 7fb6e072b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:32.401+0000 7fb6e072b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:32.415 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:32.415 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:32.415 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:32.415 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:32.416 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:32.416 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:32.416 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:32.416 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.417 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:32.417 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:32.417 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:32.417 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:32.417 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:32.418 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:32.418 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:32.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:32.418 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:32.418 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:32.419 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:32.420 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:32.420 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:32.420 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:32.420 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:32.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:32.421 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:32.421 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:32.421 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:32.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.421 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:32.422 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:32.422 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:32.422 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:32.422 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:32.422 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:32.423 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:32.423 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:32.423 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:32.423 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:32.423 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:32.424 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:32.424 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:32.424 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:32.424 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:32.425 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:32.425 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:32.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:32.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:32.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:32.426 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:32.426 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:32.426 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:32.426 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:32.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:32Z" 2023-10-11T14:31:32.427 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:32.428 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:37.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:37.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:37.430 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:37.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:37.513+0000 7fda9895c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:37.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:37.514+0000 7fda9895c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:37.528 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:37.528 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:37.529 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:37.529 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:37.529 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:37.530 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:37.530 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:37.531 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:37.532 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:37.532 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:37.532 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:37.532 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:37.532 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:37.533 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:37.533 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:37.533 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:37.533 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.533 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:37.534 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:37.534 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:37.534 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:37.534 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:37.534 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:37.535 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:37.535 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:37.535 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:37.535 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:37.535 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:37.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.536 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:37.536 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:37.536 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:37.536 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:37.537 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:37.538 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:37.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.538 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:37.538 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:37.538 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:37.539 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:37.539 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:37.539 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:37.539 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:37.539 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:37.540 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:37.540 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:37.540 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:37.540 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:37.540 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:37.541 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.541 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:37.541 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:37.541 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:37.541 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:37.542 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:37Z" 2023-10-11T14:31:37.542 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:37.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:42.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:42.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:42.545 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:42.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:42.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:42.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:42.630+0000 7fc7a32d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:42.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:42.631+0000 7fc7a32d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:42.645 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:42.645 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:42.645 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:42.645 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:42.646 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:42.646 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:42.646 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:42.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.646 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:42.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:42.647 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:42.647 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:42.647 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:42.647 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:42.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:42.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:42.648 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:42.648 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:42.648 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:42.649 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:42.650 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:42.650 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:42.650 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:42.650 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:42.650 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:42.651 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:42.651 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:42.651 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.651 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:42.651 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:42.652 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:42.653 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:42.653 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:42.653 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:42.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.653 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:42.654 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:42.654 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:42.654 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:42.654 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:42.654 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:42.655 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:42.656 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:42.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.656 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:42.656 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:42.656 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:42.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:42.657 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:42Z" 2023-10-11T14:31:42.657 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:42.658 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:47.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:47.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:47.660 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:47.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:47.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:47.746+0000 7f8aa6731780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:47.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:47.747+0000 7f8aa6731780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:47.761 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:47.761 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:47.761 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:47.762 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:47.762 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:47.762 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:47.763 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:47.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.763 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:47.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:47.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:47.764 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:47.765 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:47.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:47.765 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:47.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:47.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:47.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:47.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:47.767 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:47.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:47.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:47.768 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:47.768 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:47.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:47.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:47.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:47.770 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:47.770 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:47.770 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:47.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:47.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:47.771 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:47.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:47.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:47.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:47.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:47.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:47.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:47.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:47.772 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:47.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:47.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:47.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:47.773 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:47.773 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.774 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:47.775 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:47.775 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:47.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:47.775 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:47Z" 2023-10-11T14:31:47.775 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:47.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:52.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:52.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:52.779 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:52.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:52.837 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:52.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:52.863+0000 7f0eee0c1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:52.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:52.864+0000 7f0eee0c1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:52.878 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:52.878 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:52.879 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:52.879 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:52.879 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:52.879 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:52.879 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:52.880 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.880 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:52.880 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:52.880 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:52.881 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:52.881 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:52.881 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:52.881 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:52.881 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:52.882 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:52.882 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:52.882 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:52.882 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:52.882 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.883 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:52.883 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:52.883 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:52.883 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:52.883 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:52.884 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:52.884 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:52.884 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:52.884 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:52.884 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:52.885 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:52.886 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:52.886 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:52.886 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:52.886 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:52.887 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:52.887 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:52.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:52.887 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.887 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:52.888 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:52.889 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:52.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.890 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:52.890 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:52.890 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:52.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:52.891 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:52Z" 2023-10-11T14:31:52.891 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:52.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:31:57.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:57.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:31:57.894 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:31:57.951 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:31:57.951 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:31:57.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:57.979+0000 7fea09cea780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:57.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:31:57.980+0000 7fea09cea780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:31:57.993 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:31:57.994 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:31:57.994 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:31:57.994 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:31:57.994 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:31:57.995 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:31:57.995 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:57.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:57.995 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:31:57.995 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:57.996 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:31:57.996 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:57.996 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:57.996 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:57.996 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:57.997 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:31:57.997 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:57.997 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:57.997 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:57.997 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:57.998 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:57.998 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:57.998 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:31:57.998 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:57.998 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:57.999 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:58.000 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:58.001 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:58.001 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:58.001 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:58.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:58.001 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:58.002 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:58.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:58.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:58.002 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:31:58.002 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:31:58.003 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:31:58.003 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:31:58.003 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:31:58.003 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:31:58.003 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:31:58.004 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:31:58.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:58.005 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:31:58.005 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:31:58.005 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:31:58.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:31:58.006 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:31:57Z" 2023-10-11T14:31:58.006 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:31:58.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:03.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:03.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:03.009 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:03.067 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:03.068 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:03.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:03.095+0000 7fed34ce4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:03.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:03.096+0000 7fed34ce4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:03.110 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:03.119 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:03.119 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:03.119 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:03.119 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:03.119 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:03.120 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:03.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:03.121 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:03Z" 2023-10-11T14:32:03.121 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:03.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:08.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:08.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:08.125 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:08.183 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:08.211+0000 7fc339304780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:08.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:08.212+0000 7fc339304780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:08.225 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:08.229 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:08.229 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:08.229 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:08.229 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:08.229 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.230 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:08.230 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:08.230 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:08.230 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:08.230 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:08.231 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:08.231 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:08.231 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:08.231 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:08.231 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:08.232 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:08.232 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.232 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:08.232 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:08.232 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:08.233 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:08.234 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:08.234 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:08.234 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.235 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:08.235 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:08.235 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:08.235 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:08.235 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:08.236 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:08.236 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:08.236 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:08.236 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:08.236 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:08.237 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:08.237 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:08.237 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:08.237 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.238 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:08.238 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:08.238 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:08.238 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:08.238 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:08Z" 2023-10-11T14:32:08.239 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:08.240 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:13.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:13.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:13.242 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:13.298 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:13.299 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:13.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:13.326+0000 7f8062929780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:13.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:13.327+0000 7f8062929780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:13.342 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:13.343 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:13.343 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:13.343 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:13.343 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:13.343 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:13Z" 2023-10-11T14:32:13.354 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:13.355 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:18.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:18.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:18.357 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:18.413 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:18.413 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:18.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:18.439+0000 7fa2c653b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:18.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:18.440+0000 7fa2c653b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:18.455 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:18Z" 2023-10-11T14:32:18.470 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:18.471 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:23.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:23.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:23.473 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:23.531 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:23.531 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:23.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:23.557+0000 7f5f80ffe780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:23.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:23.558+0000 7f5f80ffe780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:23.573 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:23.573 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:23.573 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:23.574 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:23.574 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:23.574 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:23.574 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:23.575 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.575 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:23.575 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:23.575 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:23.575 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:23.576 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:23.576 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:23.576 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:23.576 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:23.576 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:23.577 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:23.577 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:23.577 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:23.577 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.577 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:23.578 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:23.579 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:23.579 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:23.579 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:23.579 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:23.579 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.580 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:23.580 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:23.580 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:23.580 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:23.580 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:23.581 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:23.582 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.582 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:23.582 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:23.582 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:23.583 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:23.584 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:23.584 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:23.584 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:23.584 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:23.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.585 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:23.585 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:23.585 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:23.585 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:23.585 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:23Z" 2023-10-11T14:32:23.586 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:23.587 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:28.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:28.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:28.589 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:28.647 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:28.647 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:28.677 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:28.674+0000 7f2467ec9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:28.678 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:28.675+0000 7f2467ec9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:28.689 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:28.690 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:28.690 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:28.690 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:28.690 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:28.691 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:28.691 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:28.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.691 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:28.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:28.692 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:28.692 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:28.692 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:28.692 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:28.693 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:28.693 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:28.693 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:28.693 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:28.693 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:28.694 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:28.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.694 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:28.694 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:28.694 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:28.695 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:28.695 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:28.695 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:28.695 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:28.695 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:28.696 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:28.696 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:28.696 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:28.696 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.696 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:28.697 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:28.697 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:28.697 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:28.697 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:28.697 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:28.698 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:28.698 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:28.698 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:28.698 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:28.698 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:28.699 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:28.700 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:28.700 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:28.700 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:28.700 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:28.700 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:28.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:28.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:28.701 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:28.701 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.701 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:28.702 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:28.702 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:28.702 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:28.702 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:28Z" 2023-10-11T14:32:28.702 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:28.704 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:33.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:33.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:33.705 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:33.761 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:33.762 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:33.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:33.788+0000 7ff7c8069780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:33.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:33.789+0000 7ff7c8069780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:33.804 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:33Z" 2023-10-11T14:32:33.817 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:33.817 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:38.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:38.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:38.820 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:38.876 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:38.877 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:38.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:38.906+0000 7efe4489a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:38.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:38.907+0000 7efe4489a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:38.922 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:38.922 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:38.922 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:38.922 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:38.923 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:38.924 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:38.925 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:38.925 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:38.925 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:38.925 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:38.925 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.926 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:38.926 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:38.926 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:38.926 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:38.926 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:38.927 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:38.928 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:38.929 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:38.929 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:38.929 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:38.929 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:38.929 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:38.930 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:38.931 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:38.932 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:38.933 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:38.933 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:38.933 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:38Z" 2023-10-11T14:32:38.933 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:38.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:43.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:43.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:43.936 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:43.993 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:43.994 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:44.024 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:44.020+0000 7f4783d4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:44.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:44.022+0000 7f4783d4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:44.037 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:44.037 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:44.037 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:44.038 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:44.038 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:44.038 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:44.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:44.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.039 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:44.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:44.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:44.039 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:44.040 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:44.040 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:44.040 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:44.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:44.040 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:44.041 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:44.041 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:44.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:44.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.041 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:44.042 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:44.042 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:44.042 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:44.042 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:44.042 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:44.043 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:44.043 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:44.043 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:44.043 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:44.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:44.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.044 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:44.044 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:44.044 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:44.045 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:44.045 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:44.045 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:44.045 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:44.045 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:44.046 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:44.046 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:44.046 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:44.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.046 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:44.047 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:44.047 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:44.047 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:44.047 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:44.047 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:44.048 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:44.048 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:44.048 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:44.048 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:44.048 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:44.049 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:44.050 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:44.050 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:44Z" 2023-10-11T14:32:44.050 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:44.051 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:49.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:49.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:49.054 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:49.111 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:49.111 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:49.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:49.138+0000 7f40af6a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:49.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:49.139+0000 7f40af6a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:49.154 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:49.154 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:49.154 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:49.155 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:49.155 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:49.155 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:49.155 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:49.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.156 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:49.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:49.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:49.156 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:49.156 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:49.157 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:49.158 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:49.159 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:49.159 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:49.159 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:49.159 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:49.159 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:49.160 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:49.160 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:49.160 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.160 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:49.160 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:49.161 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:49.162 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:49.162 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:49.162 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:49.162 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.162 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:49.163 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:49.164 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:49.164 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:49.164 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:49.164 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:49.164 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:49.165 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:49.166 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:49Z" 2023-10-11T14:32:49.166 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:49.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1697034656.410656_41.1 target= 2023-10-11T14:32:54.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.169 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.224 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.224 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:54.251+0000 7f3d7559d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:54.252+0000 7f3d7559d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.266 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:54.266 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:54.267 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:54.267 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:54.267 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:54.267 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.267 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.268 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:54.268 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.268 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:54.268 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.269 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.270 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.270 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.270 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.270 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.270 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:54.271 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.271 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.271 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.271 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.271 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.272 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.273 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:54.273 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.273 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.273 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.273 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.274 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.274 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.274 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.274 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.274 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.275 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.275 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:54.275 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.275 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.276 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.276 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.276 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.277 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:54.278 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:54.278 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:54.278 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.278 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.278 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:54.279 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:32:54.280 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-11T14:32:54.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.281 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.339 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.339 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.375 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.372+0000 7f67d4fef780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.376 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.373+0000 7f67d4fef780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.388 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:32:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2023-10-11T14:32:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2023-10-11T14:32:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2023-10-11T14:32:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2023-10-11T14:32:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2023-10-11T14:32:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2023-10-11T14:32:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2023-10-11T14:32:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2023-10-11T14:32:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2023-10-11T14:32:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2023-10-11T14:32:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2023-10-11T14:32:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2023-10-11T14:32:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.400 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:32:54.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-11T14:32:54.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-11T14:32:54.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.403 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.454 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.454 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.484 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.482+0000 7f79dcf59780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.485 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.483+0000 7f79dcf59780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.496 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:32:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2023-10-11T14:32:54.496 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2023-10-11T14:32:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2023-10-11T14:32:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2023-10-11T14:32:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2023-10-11T14:32:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2023-10-11T14:32:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.498 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2023-10-11T14:32:54.499 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2023-10-11T14:32:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.501 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.502 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2023-10-11T14:32:54.503 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.504 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.505 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.506 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.507 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.508 INFO:teuthology.orchestra.run.smithi172.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.509 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:32:54.510 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-11T14:32:54.510 INFO:rgw_multi.tests:meta checkpoint 2023-10-11T14:32:54.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.511 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.562 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:32:54.601 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:32:54.602 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.602 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.602 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:32:54.602 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.602 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.603 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:32:54.604 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.604 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.604 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.604 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:32:54.604 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.605 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:32:54.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:32:54.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.607 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.655 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.655 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:54.678+0000 7f125bfbe780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:32:54.679+0000 7f125bfbe780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.693 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:32:54.693 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:32:54.693 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:32:54.694 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:32:54.694 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:32:54.694 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.694 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.695 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:32:54.695 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.695 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:32:54.695 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.696 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.696 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.696 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.696 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:32:54.696 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.697 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.698 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.699 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.699 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.699 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.699 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.699 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.700 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.701 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:32:54.702 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:32:54.702 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:32:54.702 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:32:54.702 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:32:54.702 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:32:54.703 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:32:54.703 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:32:54.703 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.703 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:32:54.703 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.704 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:32:54.704 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:32:54.704 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.704 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:32:54.704 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:32:54.705 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:32:54.705 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:32:54.705 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.705 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:32:54.706 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:32:54.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-10-11T14:32:54.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.708 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.758 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.758 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.782+0000 7f6654e4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.783+0000 7f6654e4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.796 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:32:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2023-10-11T14:32:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2023-10-11T14:32:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2023-10-11T14:32:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2023-10-11T14:32:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2023-10-11T14:32:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2023-10-11T14:32:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2023-10-11T14:32:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:32:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2023-10-11T14:32:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2023-10-11T14:32:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2023-10-11T14:32:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.809 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:32:54.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-10-11T14:32:54.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-10-11T14:32:54.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:32:54.811 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:32:54.860 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:32:54.861 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:32:54.887 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.885+0000 7f6ae26b1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.888 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-11T14:32:54.886+0000 7f6ae26b1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:32:54.898 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-11T14:32:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": { 2023-10-11T14:32:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "info": { 2023-10-11T14:32:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "status": "sync", 2023-10-11T14:32:54.899 INFO:teuthology.orchestra.run.smithi172.stdout: "num_shards": 4, 2023-10-11T14:32:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:32:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: "markers": [ 2023-10-11T14:32:54.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 0, 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.901 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 1, 2023-10-11T14:32:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:32:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 1, 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.903 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 2, 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.905 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "key": 3, 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "state": 1, 2023-10-11T14:32:54.907 INFO:teuthology.orchestra.run.smithi172.stdout: "marker": "", 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "next_step_marker": "", 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "total_entries": 0, 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "pos": 0, 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "timestamp": "0.000000", 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2023-10-11T14:32:54.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "full_sync": { 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "total": 1, 2023-10-11T14:32:54.909 INFO:teuthology.orchestra.run.smithi172.stdout: "complete": 1 2023-10-11T14:32:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-11T14:32:54.910 INFO:teuthology.orchestra.run.smithi172.stdout: "current_time": "2023-10-11T14:32:54Z" 2023-10-11T14:32:54.910 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-11T14:32:54.911 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-10-11T14:32:54.911 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-10-11T14:32:54.928 DEBUG:boto:Using access key provided by client. 2023-10-11T14:32:54.930 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:32:54.931 DEBUG:boto:Using access key provided by client. 2023-10-11T14:32:54.932 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:32:54.933 DEBUG:boto:Using access key provided by client. 2023-10-11T14:32:54.933 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:32:54.934 DEBUG:boto:Using access key provided by client. 2023-10-11T14:32:54.936 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:32:54.936 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-1 2023-10-11T14:32:54.937 DEBUG:boto:path=/dghqvd-1/ 2023-10-11T14:32:54.937 DEBUG:boto:auth_path=/dghqvd-1/ 2023-10-11T14:32:54.937 DEBUG:boto:Method: PUT 2023-10-11T14:32:54.938 DEBUG:boto:Path: /dghqvd-1/ 2023-10-11T14:32:54.938 DEBUG:boto:Data: 2023-10-11T14:32:54.938 DEBUG:boto:Headers: {} 2023-10-11T14:32:54.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:32:54.939 DEBUG:boto:Port: 8000 2023-10-11T14:32:54.941 DEBUG:boto:Params: {} 2023-10-11T14:32:54.942 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:32:54.943 DEBUG:boto:Token: None 2023-10-11T14:32:54.943 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:32:54 GMT /dghqvd-1/ 2023-10-11T14:32:54.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EMAwW2zKliMNkuz+1HXuKvi+ryw= 2023-10-11T14:32:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:32:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EMAwW2zKliMNkuz+1HXuKvi+ryw=', 'Content-Length': '0'} 2023-10-11T14:32:57.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075275e2c02937d21-006526b216-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:32:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:32:57.223 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-2 2023-10-11T14:32:57.223 DEBUG:boto:path=/dghqvd-2/ 2023-10-11T14:32:57.224 DEBUG:boto:auth_path=/dghqvd-2/ 2023-10-11T14:32:57.224 DEBUG:boto:Method: PUT 2023-10-11T14:32:57.225 DEBUG:boto:Path: /dghqvd-2/ 2023-10-11T14:32:57.225 DEBUG:boto:Data: 2023-10-11T14:32:57.225 DEBUG:boto:Headers: {} 2023-10-11T14:32:57.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:32:57.226 DEBUG:boto:Port: 8001 2023-10-11T14:32:57.227 DEBUG:boto:Params: {} 2023-10-11T14:32:57.227 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:32:57.228 DEBUG:boto:Token: None 2023-10-11T14:32:57.228 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:32:57 GMT /dghqvd-2/ 2023-10-11T14:32:57.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqJebnh/VQVAAjAzx8EWk4bJhdQ= 2023-10-11T14:32:57.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:32:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqJebnh/VQVAAjAzx8EWk4bJhdQ=', 'Content-Length': '0'} 2023-10-11T14:33:00.236 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9845c0bbb5f38e4-006526b219-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:00.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:00.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:00.238 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:00.289 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:00.289 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:00.319 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:00.330 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:00.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.331 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:00.331 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:00.331 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.332 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034777.244019_76.1", 2023-10-11T14:33:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:32:57.244019Z" 2023-10-11T14:33:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.333 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:00.334 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:00.334 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.334 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:00.335 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034777.244019_76.1', 3: ''} 2023-10-11T14:33:00.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:00.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:00.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:00.337 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:00.386 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:00.387 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:00.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:00.411+0000 7fc95ff0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:00.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:00.412+0000 7fc95ff0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:00.425 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:00.426 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:00.426 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:00.426 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:00.426 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:00.426 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:00.427 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:00.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.427 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:00.427 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.428 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:00.428 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:00.428 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:00.428 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:00.428 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:00.429 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:00.429 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:00.429 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:00.429 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:00.429 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.430 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.430 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.430 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:00.430 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:00.430 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:00.431 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:00.431 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:00.431 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:00.431 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.432 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:00.433 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:00.433 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:00.433 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034777.211667_69.1", 2023-10-11T14:33:00.433 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:00.433 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:00.434 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:00.434 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:32:57.211667Z", 2023-10-11T14:33:00.434 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:00.434 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.435 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.435 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:00.435 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:00.435 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:00.435 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:00.436 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:00.436 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:00.436 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:00.436 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:00.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:00.437 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:00.437 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.437 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:00.437 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:00.438 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.438 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:00.438 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:00.438 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:00.438 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:00.439 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:00Z" 2023-10-11T14:33:00.439 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:00.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034777.211667_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:32:57.211667Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:00Z" } 2023-10-11T14:33:00.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034777.211667_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:32:57.211667Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:00.441 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034777.244019_76.1', 3: ''} 2023-10-11T14:33:00.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034777.211667_69.1', 3: ''} 2023-10-11T14:33:00.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697034777.244019_76.1 target=1_1697034777.211667_69.1 2023-10-11T14:33:05.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.445 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:05.502 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:05.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:05.531+0000 7fbeaf00e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:05.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:05.532+0000 7fbeaf00e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:05.545 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:05.546 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:05.546 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:05.546 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:05.546 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:05.546 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:05.547 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.547 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.547 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:05.547 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.547 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.548 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:05.549 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.549 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:05.549 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.549 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.550 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.550 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:05.550 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.550 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.551 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.551 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.551 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.551 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.551 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:05.552 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.552 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.552 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.552 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.552 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034777.244019_76.1", 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.553 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.554 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:32:57.244019Z", 2023-10-11T14:33:05.554 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.554 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.554 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.554 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.555 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:05.555 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.555 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.555 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.555 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.556 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.556 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.556 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:05.556 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.556 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.557 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.557 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:05.557 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.557 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:05.557 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:05.558 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:05.558 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.558 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:05Z" 2023-10-11T14:33:05.558 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:05.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034777.244019_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:32:57.244019Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:05Z" } 2023-10-11T14:33:05.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034777.244019_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:32:57.244019Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:05.560 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034777.244019_76.1', 3: ''} 2023-10-11T14:33:05.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034777.244019_76.1', 3: ''} 2023-10-11T14:33:05.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:05.561 DEBUG:boto:path=/dghqvd-1/ 2023-10-11T14:33:05.562 DEBUG:boto:auth_path=/dghqvd-1/ 2023-10-11T14:33:05.562 DEBUG:boto:Method: HEAD 2023-10-11T14:33:05.563 DEBUG:boto:Path: /dghqvd-1/ 2023-10-11T14:33:05.563 DEBUG:boto:Data: 2023-10-11T14:33:05.563 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:05.564 DEBUG:boto:Port: 8000 2023-10-11T14:33:05.564 DEBUG:boto:Params: {} 2023-10-11T14:33:05.565 DEBUG:boto:Token: None 2023-10-11T14:33:05.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-1/ 2023-10-11T14:33:05.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zb7hS14dbC3TUt1mHpfTUdiiK/8= 2023-10-11T14:33:05.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zb7hS14dbC3TUt1mHpfTUdiiK/8=', 'Content-Length': '0'} 2023-10-11T14:33:05.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c3ba8e590ac9109-006526b221-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.569 DEBUG:boto:path=/dghqvd-2/ 2023-10-11T14:33:05.569 DEBUG:boto:auth_path=/dghqvd-2/ 2023-10-11T14:33:05.570 DEBUG:boto:Method: HEAD 2023-10-11T14:33:05.570 DEBUG:boto:Path: /dghqvd-2/ 2023-10-11T14:33:05.571 DEBUG:boto:Data: 2023-10-11T14:33:05.571 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:05.572 DEBUG:boto:Port: 8000 2023-10-11T14:33:05.572 DEBUG:boto:Params: {} 2023-10-11T14:33:05.572 DEBUG:boto:Token: None 2023-10-11T14:33:05.573 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-2/ 2023-10-11T14:33:05.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kMrV2YxNLID06rVx345ekLuIxO4= 2023-10-11T14:33:05.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kMrV2YxNLID06rVx345ekLuIxO4=', 'Content-Length': '0'} 2023-10-11T14:33:05.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055c985b96baff2b6-006526b221-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.576 DEBUG:boto:path=/dghqvd-1/ 2023-10-11T14:33:05.577 DEBUG:boto:auth_path=/dghqvd-1/ 2023-10-11T14:33:05.577 DEBUG:boto:Method: HEAD 2023-10-11T14:33:05.578 DEBUG:boto:Path: /dghqvd-1/ 2023-10-11T14:33:05.578 DEBUG:boto:Data: 2023-10-11T14:33:05.578 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:05.579 DEBUG:boto:Port: 8001 2023-10-11T14:33:05.580 DEBUG:boto:Params: {} 2023-10-11T14:33:05.580 DEBUG:boto:Token: None 2023-10-11T14:33:05.580 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-1/ 2023-10-11T14:33:05.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zb7hS14dbC3TUt1mHpfTUdiiK/8= 2023-10-11T14:33:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zb7hS14dbC3TUt1mHpfTUdiiK/8=', 'Content-Length': '0'} 2023-10-11T14:33:05.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a140a742b24a1350-006526b221-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.584 DEBUG:boto:path=/dghqvd-2/ 2023-10-11T14:33:05.585 DEBUG:boto:auth_path=/dghqvd-2/ 2023-10-11T14:33:05.585 DEBUG:boto:Method: HEAD 2023-10-11T14:33:05.586 DEBUG:boto:Path: /dghqvd-2/ 2023-10-11T14:33:05.586 DEBUG:boto:Data: 2023-10-11T14:33:05.586 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:05.587 DEBUG:boto:Port: 8001 2023-10-11T14:33:05.588 DEBUG:boto:Params: {} 2023-10-11T14:33:05.588 DEBUG:boto:Token: None 2023-10-11T14:33:05.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-2/ 2023-10-11T14:33:05.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kMrV2YxNLID06rVx345ekLuIxO4= 2023-10-11T14:33:05.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kMrV2YxNLID06rVx345ekLuIxO4=', 'Content-Length': '0'} 2023-10-11T14:33:05.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077dee476b558b299-006526b221-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-10-11T14:33:05.593 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-3 2023-10-11T14:33:05.594 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:05.594 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:05.595 DEBUG:boto:Method: PUT 2023-10-11T14:33:05.595 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:05.595 DEBUG:boto:Data: 2023-10-11T14:33:05.596 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:05.597 DEBUG:boto:Port: 8000 2023-10-11T14:33:05.597 DEBUG:boto:Params: {} 2023-10-11T14:33:05.597 DEBUG:boto:Token: None 2023-10-11T14:33:05.598 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-3/ 2023-10-11T14:33:05.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LNALfZQl/B+tep7bMDKxlR2rhuU= 2023-10-11T14:33:05.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LNALfZQl/B+tep7bMDKxlR2rhuU=', 'Content-Length': '0'} 2023-10-11T14:33:05.611 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ee637173cbd54a3-006526b221-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.611 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-4 2023-10-11T14:33:05.611 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:05.612 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:05.612 DEBUG:boto:Method: PUT 2023-10-11T14:33:05.612 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:05.613 DEBUG:boto:Data: 2023-10-11T14:33:05.613 DEBUG:boto:Headers: {} 2023-10-11T14:33:05.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:05.614 DEBUG:boto:Port: 8001 2023-10-11T14:33:05.614 DEBUG:boto:Params: {} 2023-10-11T14:33:05.615 DEBUG:boto:Token: None 2023-10-11T14:33:05.615 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:05 GMT /dghqvd-4/ 2023-10-11T14:33:05.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ws5O8S+Pm2v00E9uKEWd2Z0rDOo= 2023-10-11T14:33:05.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ws5O8S+Pm2v00E9uKEWd2Z0rDOo=', 'Content-Length': '0'} 2023-10-11T14:33:05.637 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003a41395437318efa-006526b221-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:05.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.638 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:05.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:05.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:05.715 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:05.726 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:05.726 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.726 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:05.727 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:05.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.727 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.727 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.727 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034785.626266_84.1", 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:33:05.626266Z" 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.729 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.729 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:05.729 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.729 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:05.730 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:05.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:05.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:05.732 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:05.782 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:05.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:05.814+0000 7f9e3c66b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:05.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:05.815+0000 7f9e3c66b780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:05.828 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:05.829 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:05.829 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:05.829 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:05.829 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:05.830 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:05.830 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.830 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.830 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:05.830 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.831 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:05.831 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.831 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.831 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:05.831 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.832 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:05.832 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.832 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:05.832 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.832 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.833 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.834 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.834 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.834 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.834 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.834 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:05.835 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.835 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.835 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.835 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.835 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:05.836 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.836 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.836 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034785.602088_77.1", 2023-10-11T14:33:05.836 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.836 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.837 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.837 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:05.602088Z", 2023-10-11T14:33:05.837 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.837 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.838 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.838 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:05.838 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:05.838 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:05.838 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:05.839 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:05.839 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:05.839 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:05.839 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:05.839 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:05.840 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:05.840 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.840 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:05.840 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:05.840 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.841 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:05.841 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:05.841 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:05.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:05.841 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:05Z" 2023-10-11T14:33:05.842 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:05.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034785.602088_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:05.602088Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:05Z" } 2023-10-11T14:33:05.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034785.602088_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:05.602088Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:05.843 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:05.844 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.602088_77.1', 3: ''} 2023-10-11T14:33:05.844 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697034785.626266_84.1 target=1_1697034785.602088_77.1 2023-10-11T14:33:10.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:10.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:10.847 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:10.906 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:10.936 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:10.933+0000 7f420e077780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:10.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:10.934+0000 7f420e077780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:10.948 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:10.949 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:10.949 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:10.949 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:10.949 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:10.950 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:10.950 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:10.950 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.950 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:10.951 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:10.951 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:10.951 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:10.951 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:10.951 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:10.952 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:10.952 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:10.952 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:10.952 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:10.953 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:10.953 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:10.953 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.953 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:10.953 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:10.954 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:10.954 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:10.954 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:10.954 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:10.955 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:10.955 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:10.955 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:10.955 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:10.956 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:10.956 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.956 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:10.956 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:10.956 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:10.957 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:10.957 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034785.626266_84.1", 2023-10-11T14:33:10.957 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:10.957 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:10.957 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:10.958 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:05.626266Z", 2023-10-11T14:33:10.958 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:10.958 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:10.958 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.959 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:10.959 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:10.959 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:10.959 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:10.959 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:10.960 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:10.960 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:10.960 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:10.960 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:10.960 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:10.961 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:10.961 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:10.961 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:10.961 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.962 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:10.963 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:10.963 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:10.963 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:10.963 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:10Z" 2023-10-11T14:33:10.964 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:10.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034785.626266_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:05.626266Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:10Z" } 2023-10-11T14:33:10.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034785.626266_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:05.626266Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:10.966 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:10.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:10.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:10.967 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:10.967 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:10.968 DEBUG:boto:Method: HEAD 2023-10-11T14:33:10.968 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:10.969 DEBUG:boto:Data: 2023-10-11T14:33:10.969 DEBUG:boto:Headers: {} 2023-10-11T14:33:10.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:10.970 DEBUG:boto:Port: 8000 2023-10-11T14:33:10.970 DEBUG:boto:Params: {} 2023-10-11T14:33:10.971 DEBUG:boto:Token: None 2023-10-11T14:33:10.971 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:10 GMT /dghqvd-3/ 2023-10-11T14:33:10.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur7VG9LNwFW+NmWp/oESdNbuZL8= 2023-10-11T14:33:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur7VG9LNwFW+NmWp/oESdNbuZL8=', 'Content-Length': '0'} 2023-10-11T14:33:10.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee04affc336c991f-006526b226-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:10.975 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:10.976 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:10.976 DEBUG:boto:Method: HEAD 2023-10-11T14:33:10.976 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:10.977 DEBUG:boto:Data: 2023-10-11T14:33:10.977 DEBUG:boto:Headers: {} 2023-10-11T14:33:10.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:10.978 DEBUG:boto:Port: 8000 2023-10-11T14:33:10.978 DEBUG:boto:Params: {} 2023-10-11T14:33:10.979 DEBUG:boto:Token: None 2023-10-11T14:33:10.979 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:10 GMT /dghqvd-4/ 2023-10-11T14:33:10.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vVt1xf+ocroU9TqNWDM0z1XBqh4= 2023-10-11T14:33:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vVt1xf+ocroU9TqNWDM0z1XBqh4=', 'Content-Length': '0'} 2023-10-11T14:33:10.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085278e763b57e43b-006526b226-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:10.983 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:10.983 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:10.984 DEBUG:boto:Method: HEAD 2023-10-11T14:33:10.984 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:10.985 DEBUG:boto:Data: 2023-10-11T14:33:10.985 DEBUG:boto:Headers: {} 2023-10-11T14:33:10.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:10.986 DEBUG:boto:Port: 8001 2023-10-11T14:33:10.986 DEBUG:boto:Params: {} 2023-10-11T14:33:10.987 DEBUG:boto:Token: None 2023-10-11T14:33:10.987 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:10 GMT /dghqvd-3/ 2023-10-11T14:33:10.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur7VG9LNwFW+NmWp/oESdNbuZL8= 2023-10-11T14:33:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur7VG9LNwFW+NmWp/oESdNbuZL8=', 'Content-Length': '0'} 2023-10-11T14:33:10.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006721aab20a161f7b-006526b226-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:10.991 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:10.991 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:10.992 DEBUG:boto:Method: HEAD 2023-10-11T14:33:10.992 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:10.993 DEBUG:boto:Data: 2023-10-11T14:33:10.993 DEBUG:boto:Headers: {} 2023-10-11T14:33:10.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:10.994 DEBUG:boto:Port: 8001 2023-10-11T14:33:10.994 DEBUG:boto:Params: {} 2023-10-11T14:33:10.995 DEBUG:boto:Token: None 2023-10-11T14:33:10.995 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:10 GMT /dghqvd-4/ 2023-10-11T14:33:10.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vVt1xf+ocroU9TqNWDM0z1XBqh4= 2023-10-11T14:33:10.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vVt1xf+ocroU9TqNWDM0z1XBqh4=', 'Content-Length': '0'} 2023-10-11T14:33:10.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3b2bf2ce983161d-006526b226-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:10.999 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:10.999 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.000 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.000 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.001 DEBUG:boto:Data: 2023-10-11T14:33:11.001 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.002 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.002 DEBUG:boto:Params: {} 2023-10-11T14:33:11.003 DEBUG:boto:Token: None 2023-10-11T14:33:11.003 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MP8DcK7w0Uzjt+SaIjXXQs1dLBU= 2023-10-11T14:33:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MP8DcK7w0Uzjt+SaIjXXQs1dLBU=', 'Content-Length': '0'} 2023-10-11T14:33:11.007 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002941911bfa35f3b5-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.008 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.008 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.008 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.009 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.009 DEBUG:boto:Data: 2023-10-11T14:33:11.010 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.010 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.011 DEBUG:boto:Params: {} 2023-10-11T14:33:11.011 DEBUG:boto:Token: None 2023-10-11T14:33:11.012 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u1W7IFMKdFFx8ptfW2l7bCxDDjk= 2023-10-11T14:33:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u1W7IFMKdFFx8ptfW2l7bCxDDjk=', 'Content-Length': '0'} 2023-10-11T14:33:11.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004065b8b3aaec8709-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.015 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:11.016 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.016 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.017 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.017 DEBUG:boto:Data: 2023-10-11T14:33:11.017 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.018 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.019 DEBUG:boto:Params: {} 2023-10-11T14:33:11.019 DEBUG:boto:Token: None 2023-10-11T14:33:11.019 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MP8DcK7w0Uzjt+SaIjXXQs1dLBU= 2023-10-11T14:33:11.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MP8DcK7w0Uzjt+SaIjXXQs1dLBU=', 'Content-Length': '0'} 2023-10-11T14:33:11.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079f53c7f3eda80b2-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.025 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.026 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.026 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.027 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.027 DEBUG:boto:Data: 2023-10-11T14:33:11.027 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.028 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.029 DEBUG:boto:Params: {} 2023-10-11T14:33:11.029 DEBUG:boto:Token: None 2023-10-11T14:33:11.030 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u1W7IFMKdFFx8ptfW2l7bCxDDjk= 2023-10-11T14:33:11.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u1W7IFMKdFFx8ptfW2l7bCxDDjk=', 'Content-Length': '0'} 2023-10-11T14:33:11.076 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd2e47637debf036-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.077 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:11.077 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.077 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.078 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.078 DEBUG:boto:Data: 2023-10-11T14:33:11.079 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.079 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.079 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.080 DEBUG:boto:Params: {} 2023-10-11T14:33:11.080 DEBUG:boto:Token: None 2023-10-11T14:33:11.080 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.081 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI= 2023-10-11T14:33:11.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI=', 'Content-Length': '0'} 2023-10-11T14:33:11.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aa2fd397ab2f246-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.084 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.084 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.085 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.085 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.086 DEBUG:boto:Data: 2023-10-11T14:33:11.086 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.087 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.087 DEBUG:boto:Params: {} 2023-10-11T14:33:11.088 DEBUG:boto:Token: None 2023-10-11T14:33:11.088 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0= 2023-10-11T14:33:11.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0=', 'Content-Length': '0'} 2023-10-11T14:33:11.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9639b988f6cceee-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.092 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:11.092 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.093 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.093 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.094 DEBUG:boto:Data: 2023-10-11T14:33:11.094 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.095 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.095 DEBUG:boto:Params: {} 2023-10-11T14:33:11.096 DEBUG:boto:Token: None 2023-10-11T14:33:11.096 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.097 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI= 2023-10-11T14:33:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI=', 'Content-Length': '0'} 2023-10-11T14:33:11.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c1e9b18ea3750c7-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.100 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.100 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.101 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.101 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.101 DEBUG:boto:Data: 2023-10-11T14:33:11.102 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.103 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.103 DEBUG:boto:Params: {} 2023-10-11T14:33:11.104 DEBUG:boto:Token: None 2023-10-11T14:33:11.104 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0= 2023-10-11T14:33:11.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0=', 'Content-Length': '0'} 2023-10-11T14:33:11.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e70a5bcd15ff018-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.109 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.158 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.159 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.186 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:11.197 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:11.198 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.198 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.198 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:11.198 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.199 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.199 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.199 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.199 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.200 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.200 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034785.626266_84.1", 2023-10-11T14:33:11.200 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:33:05.626266Z" 2023-10-11T14:33:11.200 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.200 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.201 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.201 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.201 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.201 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:11.202 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:11.203 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:11.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.204 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.253 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.253 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.277+0000 7fa43a6c7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.278+0000 7fa43a6c7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.290 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.295 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.295 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.295 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.295 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034785.626266_84.1", 2023-10-11T14:33:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:05.626266Z", 2023-10-11T14:33:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:11Z" 2023-10-11T14:33:11.304 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:11.305 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034785.626266_84.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:05.626266Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:11Z" } 2023-10-11T14:33:11.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034785.626266_84.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:05.626266Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:11.306 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:11.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034785.626266_84.1', 3: ''} 2023-10-11T14:33:11.307 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:11.308 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:11.308 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.308 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.309 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.309 DEBUG:boto:Data: 2023-10-11T14:33:11.310 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.311 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.311 DEBUG:boto:Params: {} 2023-10-11T14:33:11.311 DEBUG:boto:Token: None 2023-10-11T14:33:11.312 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI= 2023-10-11T14:33:11.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI=', 'Content-Length': '0'} 2023-10-11T14:33:11.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007e326e884928272-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.316 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.317 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.317 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.317 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.318 DEBUG:boto:Data: 2023-10-11T14:33:11.318 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.319 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.319 DEBUG:boto:Params: {} 2023-10-11T14:33:11.320 DEBUG:boto:Token: None 2023-10-11T14:33:11.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.321 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0= 2023-10-11T14:33:11.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0=', 'Content-Length': '0'} 2023-10-11T14:33:11.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de3130c1663a9e4-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.324 DEBUG:boto:path=/dghqvd-3/ 2023-10-11T14:33:11.324 DEBUG:boto:auth_path=/dghqvd-3/ 2023-10-11T14:33:11.325 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.325 DEBUG:boto:Path: /dghqvd-3/ 2023-10-11T14:33:11.325 DEBUG:boto:Data: 2023-10-11T14:33:11.326 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.327 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.327 DEBUG:boto:Params: {} 2023-10-11T14:33:11.328 DEBUG:boto:Token: None 2023-10-11T14:33:11.328 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-3/ 2023-10-11T14:33:11.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI= 2023-10-11T14:33:11.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Qt7e2Jn1MTklvonbEME01HRfTI=', 'Content-Length': '0'} 2023-10-11T14:33:11.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004701976cd6f39a4a-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.331 DEBUG:boto:path=/dghqvd-4/ 2023-10-11T14:33:11.332 DEBUG:boto:auth_path=/dghqvd-4/ 2023-10-11T14:33:11.332 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.333 DEBUG:boto:Path: /dghqvd-4/ 2023-10-11T14:33:11.333 DEBUG:boto:Data: 2023-10-11T14:33:11.333 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.334 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.335 DEBUG:boto:Params: {} 2023-10-11T14:33:11.335 DEBUG:boto:Token: None 2023-10-11T14:33:11.337 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-4/ 2023-10-11T14:33:11.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0= 2023-10-11T14:33:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChUiWPdE5wyW3g4RQZvxnF73/l0=', 'Content-Length': '0'} 2023-10-11T14:33:11.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000174cfbaa210916b0-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-10-11T14:33:11.341 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-5 2023-10-11T14:33:11.342 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.342 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.343 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.343 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.343 DEBUG:boto:Data: 2023-10-11T14:33:11.344 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.345 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.345 DEBUG:boto:Params: {} 2023-10-11T14:33:11.345 DEBUG:boto:Token: None 2023-10-11T14:33:11.346 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Opv/NYCGxDHAvsujEnflu24IT2A= 2023-10-11T14:33:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Opv/NYCGxDHAvsujEnflu24IT2A=', 'Content-Length': '0'} 2023-10-11T14:33:11.359 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eced9ef4150a93b1-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.359 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-6 2023-10-11T14:33:11.359 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.360 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.360 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.361 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.361 DEBUG:boto:Data: 2023-10-11T14:33:11.361 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.362 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.363 DEBUG:boto:Params: {} 2023-10-11T14:33:11.363 DEBUG:boto:Token: None 2023-10-11T14:33:11.364 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NEMqm2H6zgxrF5D7FtUrDfPZ5yI= 2023-10-11T14:33:11.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NEMqm2H6zgxrF5D7FtUrDfPZ5yI=', 'Content-Length': '0'} 2023-10-11T14:33:11.385 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000633af987dcbe25b1-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.386 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.437 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.438 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.470 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:11.481 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.375148_92.1", 2023-10-11T14:33:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:33:11.375148Z" 2023-10-11T14:33:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.485 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:11.486 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.375148_92.1', 3: ''} 2023-10-11T14:33:11.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:11.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.488 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.537 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.537 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.561+0000 7fdd2a3a6780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.562+0000 7fdd2a3a6780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.575 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:11.575 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:11.576 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:11.576 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:11.576 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:11.576 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:11.577 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.577 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.577 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:11.577 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.577 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:11.578 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.578 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.578 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.578 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.578 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:11.579 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.579 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:11.579 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.579 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.579 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.580 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.581 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.581 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.581 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.581 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.581 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.582 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.583 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.375148_92.1", 2023-10-11T14:33:11.583 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.583 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.583 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.583 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:11.375148Z", 2023-10-11T14:33:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.584 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:11.585 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.585 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.585 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.585 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.586 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.586 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.586 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.586 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.586 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.587 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.587 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:11.587 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.587 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:11.587 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:11.588 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:11.588 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.588 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:11Z" 2023-10-11T14:33:11.588 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:11.589 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034791.375148_92.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:11.375148Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:11Z" } 2023-10-11T14:33:11.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034791.375148_92.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:11.375148Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:11.590 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.375148_92.1', 3: ''} 2023-10-11T14:33:11.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.375148_92.1', 3: ''} 2023-10-11T14:33:11.591 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:11.591 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.592 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.592 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.592 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.593 DEBUG:boto:Data: 2023-10-11T14:33:11.593 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.594 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.594 DEBUG:boto:Params: {} 2023-10-11T14:33:11.595 DEBUG:boto:Token: None 2023-10-11T14:33:11.595 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo= 2023-10-11T14:33:11.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo=', 'Content-Length': '0'} 2023-10-11T14:33:11.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7f3be0286b391cf-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.599 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.599 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.600 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.600 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.600 DEBUG:boto:Data: 2023-10-11T14:33:11.601 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.602 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.602 DEBUG:boto:Params: {} 2023-10-11T14:33:11.603 DEBUG:boto:Token: None 2023-10-11T14:33:11.603 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64= 2023-10-11T14:33:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64=', 'Content-Length': '0'} 2023-10-11T14:33:11.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf548d02b29c6f60-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.607 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.607 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.607 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.608 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.608 DEBUG:boto:Data: 2023-10-11T14:33:11.609 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.609 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.610 DEBUG:boto:Params: {} 2023-10-11T14:33:11.610 DEBUG:boto:Token: None 2023-10-11T14:33:11.611 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo= 2023-10-11T14:33:11.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo=', 'Content-Length': '0'} 2023-10-11T14:33:11.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011efe6e4727fff99-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.614 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.615 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.615 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.616 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.616 DEBUG:boto:Data: 2023-10-11T14:33:11.616 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.617 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.617 DEBUG:boto:Params: {} 2023-10-11T14:33:11.618 DEBUG:boto:Token: None 2023-10-11T14:33:11.618 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.619 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64= 2023-10-11T14:33:11.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64=', 'Content-Length': '0'} 2023-10-11T14:33:11.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000223b89385c6df898-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.622 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.622 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.623 DEBUG:boto:Method: DELETE 2023-10-11T14:33:11.623 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.623 DEBUG:boto:Data: 2023-10-11T14:33:11.624 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.625 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.625 DEBUG:boto:Params: {} 2023-10-11T14:33:11.625 DEBUG:boto:Token: None 2023-10-11T14:33:11.626 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.626 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:guiZTmPeqRS6mg15kUOY4M8unbI= 2023-10-11T14:33:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:guiZTmPeqRS6mg15kUOY4M8unbI=', 'Content-Length': '0'} 2023-10-11T14:33:11.652 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001e90ffba88c6fe0f-006526b227-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.653 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.653 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.654 DEBUG:boto:Method: DELETE 2023-10-11T14:33:11.654 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.655 DEBUG:boto:Data: 2023-10-11T14:33:11.655 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.656 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.656 DEBUG:boto:Params: {} 2023-10-11T14:33:11.657 DEBUG:boto:Token: None 2023-10-11T14:33:11.657 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zTYyUBmobzFE3GpMK/mxInpnI2s= 2023-10-11T14:33:11.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zTYyUBmobzFE3GpMK/mxInpnI2s=', 'Content-Length': '0'} 2023-10-11T14:33:11.709 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077b5e35e2deed7da-006526b227-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.710 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.758 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.759 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.787 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:11.798 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:11.799 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.799 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.799 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:11.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.800 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.800 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.800 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.800 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.684337_100.1", 2023-10-11T14:33:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:33:11.684337Z" 2023-10-11T14:33:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.802 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:11.802 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.802 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:11.803 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.684337_100.1', 3: ''} 2023-10-11T14:33:11.804 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:11.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:11.805 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:11.856 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:11.856 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:11.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.882+0000 7f52e3247780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.885 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:11.883+0000 7f52e3247780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:11.896 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:11.896 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:11.896 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:11.897 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:11.897 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:11.897 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:11.897 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.898 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.898 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:11.898 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.898 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:11.898 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.899 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.899 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:11.899 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.899 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:11.900 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.900 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:11.900 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.901 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.901 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:11.901 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.901 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.902 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.902 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.902 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.903 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.903 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.904 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:11.904 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.904 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.904 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.684337_100.1", 2023-10-11T14:33:11.904 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.905 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.905 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:11.684337Z", 2023-10-11T14:33:11.905 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.906 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:11.906 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:11.906 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:11.907 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:11.908 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:11.908 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:11.908 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.908 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:11.908 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:11.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.909 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:11.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:11.909 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:11.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:11.910 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:11Z" 2023-10-11T14:33:11.910 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:11.911 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034791.684337_100.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:11.684337Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:11Z" } 2023-10-11T14:33:11.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034791.684337_100.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:11.684337Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:11.912 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.684337_100.1', 3: ''} 2023-10-11T14:33:11.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.684337_100.1', 3: ''} 2023-10-11T14:33:11.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:11.913 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.914 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.914 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.914 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.915 DEBUG:boto:Data: 2023-10-11T14:33:11.915 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.916 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.916 DEBUG:boto:Params: {} 2023-10-11T14:33:11.917 DEBUG:boto:Token: None 2023-10-11T14:33:11.917 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo= 2023-10-11T14:33:11.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo=', 'Content-Length': '0'} 2023-10-11T14:33:11.920 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000039792b549e33c3d2-006526b227-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.920 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.921 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.921 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.922 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.922 DEBUG:boto:Data: 2023-10-11T14:33:11.922 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.923 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.923 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.924 DEBUG:boto:Params: {} 2023-10-11T14:33:11.924 DEBUG:boto:Token: None 2023-10-11T14:33:11.925 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64= 2023-10-11T14:33:11.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64=', 'Content-Length': '0'} 2023-10-11T14:33:11.927 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000094575e23966fc476-006526b227-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.927 DEBUG:boto:path=/dghqvd-5/ 2023-10-11T14:33:11.928 DEBUG:boto:auth_path=/dghqvd-5/ 2023-10-11T14:33:11.928 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.928 DEBUG:boto:Path: /dghqvd-5/ 2023-10-11T14:33:11.929 DEBUG:boto:Data: 2023-10-11T14:33:11.929 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.930 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.930 DEBUG:boto:Params: {} 2023-10-11T14:33:11.931 DEBUG:boto:Token: None 2023-10-11T14:33:11.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-5/ 2023-10-11T14:33:11.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo= 2023-10-11T14:33:11.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tGm/9JZfyws/uHEMDpa5blrZxDo=', 'Content-Length': '0'} 2023-10-11T14:33:11.934 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000f5177506d9ae368f-006526b227-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.934 DEBUG:boto:path=/dghqvd-6/ 2023-10-11T14:33:11.935 DEBUG:boto:auth_path=/dghqvd-6/ 2023-10-11T14:33:11.935 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.935 DEBUG:boto:Path: /dghqvd-6/ 2023-10-11T14:33:11.936 DEBUG:boto:Data: 2023-10-11T14:33:11.936 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.937 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.937 DEBUG:boto:Params: {} 2023-10-11T14:33:11.938 DEBUG:boto:Token: None 2023-10-11T14:33:11.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-6/ 2023-10-11T14:33:11.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64= 2023-10-11T14:33:11.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7eRiS6rJIcu8bKzNT6wg5xXIV64=', 'Content-Length': '0'} 2023-10-11T14:33:11.940 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000944b282f534c4fe2-006526b227-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-10-11T14:33:11.942 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-7 2023-10-11T14:33:11.942 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:33:11.943 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:33:11.943 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.943 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:33:11.944 DEBUG:boto:Data: 2023-10-11T14:33:11.944 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.945 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.945 DEBUG:boto:Params: {} 2023-10-11T14:33:11.946 DEBUG:boto:Token: None 2023-10-11T14:33:11.946 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-7/ 2023-10-11T14:33:11.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYWj5495jbqEnOOsijyWGKeDUok= 2023-10-11T14:33:11.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYWj5495jbqEnOOsijyWGKeDUok=', 'Content-Length': '0'} 2023-10-11T14:33:11.958 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064fdbdb3896d6038-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.959 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-8 2023-10-11T14:33:11.959 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:33:11.960 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:33:11.960 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.960 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:33:11.961 DEBUG:boto:Data: 2023-10-11T14:33:11.961 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.962 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:11.962 DEBUG:boto:Port: 8001 2023-10-11T14:33:11.962 DEBUG:boto:Params: {} 2023-10-11T14:33:11.963 DEBUG:boto:Token: None 2023-10-11T14:33:11.963 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-8/ 2023-10-11T14:33:11.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:La9viZ6sgFxdGQoZGy6yKpeJvJE= 2023-10-11T14:33:11.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:La9viZ6sgFxdGQoZGy6yKpeJvJE=', 'Content-Length': '0'} 2023-10-11T14:33:11.983 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005fdb6f4c7f63ba88-006526b227-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.984 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:33:11.984 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:33:11.985 DEBUG:boto:Method: HEAD 2023-10-11T14:33:11.985 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:33:11.985 DEBUG:boto:Data: 2023-10-11T14:33:11.986 DEBUG:boto:Headers: {} 2023-10-11T14:33:11.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.987 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.987 DEBUG:boto:Params: {} 2023-10-11T14:33:11.987 DEBUG:boto:Token: None 2023-10-11T14:33:11.988 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-7/ 2023-10-11T14:33:11.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZejbDbalVvFq7KPzr945Ze88j8= 2023-10-11T14:33:11.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZejbDbalVvFq7KPzr945Ze88j8=', 'Content-Length': '0'} 2023-10-11T14:33:11.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8d218c08f69079f-006526b227-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:11.992 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:33:11.993 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:33:11.993 DEBUG:boto:Method: PUT 2023-10-11T14:33:11.994 DEBUG:boto:Path: /dghqvd-7/myobj 2023-10-11T14:33:11.994 DEBUG:boto:Data: 2023-10-11T14:33:11.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:11.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:11.995 DEBUG:boto:Port: 8000 2023-10-11T14:33:11.996 DEBUG:boto:Params: {} 2023-10-11T14:33:11.996 DEBUG:boto:Token: None 2023-10-11T14:33:11.997 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:11 GMT /dghqvd-7/myobj 2023-10-11T14:33:11.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m4AjHkJC2RobZfjgqxSuHRuml2s= 2023-10-11T14:33:11.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m4AjHkJC2RobZfjgqxSuHRuml2s='} 2023-10-11T14:33:12.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b17f4aed723077f-006526b227-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.043 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:33:12.044 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:33:12.044 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.044 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:33:12.045 DEBUG:boto:Data: 2023-10-11T14:33:12.045 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.046 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.046 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.047 DEBUG:boto:Params: {} 2023-10-11T14:33:12.047 DEBUG:boto:Token: None 2023-10-11T14:33:12.048 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/ 2023-10-11T14:33:12.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE= 2023-10-11T14:33:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE=', 'Content-Length': '0'} 2023-10-11T14:33:12.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a022c0e792c4eadd-006526b228-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.051 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:33:12.052 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:33:12.052 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.053 DEBUG:boto:Path: /dghqvd-7/_myobj 2023-10-11T14:33:12.053 DEBUG:boto:Data: 2023-10-11T14:33:12.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.054 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.055 DEBUG:boto:Params: {} 2023-10-11T14:33:12.056 DEBUG:boto:Token: None 2023-10-11T14:33:12.056 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/_myobj 2023-10-11T14:33:12.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sJn46hbsCtGs4DLw/d+t4WQP/lk= 2023-10-11T14:33:12.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sJn46hbsCtGs4DLw/d+t4WQP/lk='} 2023-10-11T14:33:12.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007164d295dee95dea-006526b228-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.066 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:33:12.066 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:33:12.067 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.067 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:33:12.067 DEBUG:boto:Data: 2023-10-11T14:33:12.068 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.068 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.069 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.069 DEBUG:boto:Params: {} 2023-10-11T14:33:12.069 DEBUG:boto:Token: None 2023-10-11T14:33:12.070 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/ 2023-10-11T14:33:12.070 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE= 2023-10-11T14:33:12.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE=', 'Content-Length': '0'} 2023-10-11T14:33:12.073 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084881a3574c34773-006526b228-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.074 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:33:12.075 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:33:12.075 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.075 DEBUG:boto:Path: /dghqvd-7/%3A 2023-10-11T14:33:12.076 DEBUG:boto:Data: 2023-10-11T14:33:12.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.077 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.077 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.077 DEBUG:boto:Params: {} 2023-10-11T14:33:12.078 DEBUG:boto:Token: None 2023-10-11T14:33:12.078 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/%3A 2023-10-11T14:33:12.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ysUwnMLbDli5M4Wi5kEiX17mRxo= 2023-10-11T14:33:12.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ysUwnMLbDli5M4Wi5kEiX17mRxo='} 2023-10-11T14:33:12.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099353dfe081f4e1d-006526b228-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.123 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:33:12.124 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:33:12.124 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.124 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:33:12.125 DEBUG:boto:Data: 2023-10-11T14:33:12.125 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.126 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.126 DEBUG:boto:Params: {} 2023-10-11T14:33:12.127 DEBUG:boto:Token: None 2023-10-11T14:33:12.127 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/ 2023-10-11T14:33:12.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE= 2023-10-11T14:33:12.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jML9MBqDVkUqIIBTzTvSw85HAdE=', 'Content-Length': '0'} 2023-10-11T14:33:12.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2230e7fce9022da-006526b228-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.131 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:33:12.131 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:33:12.132 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.132 DEBUG:boto:Path: /dghqvd-7/%26 2023-10-11T14:33:12.133 DEBUG:boto:Data: 2023-10-11T14:33:12.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:33:12.134 DEBUG:boto:Port: 8000 2023-10-11T14:33:12.134 DEBUG:boto:Params: {} 2023-10-11T14:33:12.135 DEBUG:boto:Token: None 2023-10-11T14:33:12.135 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-7/%26 2023-10-11T14:33:12.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V5FTi3a4QwWuLwk+RhqSQlgwGos= 2023-10-11T14:33:12.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V5FTi3a4QwWuLwk+RhqSQlgwGos='} 2023-10-11T14:33:12.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4198c0ea210ad9d-006526b228-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.142 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:33:12.143 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:33:12.143 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.144 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:33:12.144 DEBUG:boto:Data: 2023-10-11T14:33:12.144 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.145 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.146 DEBUG:boto:Params: {} 2023-10-11T14:33:12.146 DEBUG:boto:Token: None 2023-10-11T14:33:12.147 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/ 2023-10-11T14:33:12.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ= 2023-10-11T14:33:12.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ=', 'Content-Length': '0'} 2023-10-11T14:33:12.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008628231ad1930a4f-006526b228-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.150 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:33:12.151 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:33:12.151 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.151 DEBUG:boto:Path: /dghqvd-8/myobj 2023-10-11T14:33:12.152 DEBUG:boto:Data: 2023-10-11T14:33:12.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.153 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.153 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.153 DEBUG:boto:Params: {} 2023-10-11T14:33:12.154 DEBUG:boto:Token: None 2023-10-11T14:33:12.154 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/myobj 2023-10-11T14:33:12.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aGKbJRZWcNP8Y8a1xO/5LAp0+AM= 2023-10-11T14:33:12.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aGKbJRZWcNP8Y8a1xO/5LAp0+AM='} 2023-10-11T14:33:12.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c94252d31d9148eb-006526b228-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.164 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:33:12.164 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:33:12.165 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.165 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:33:12.166 DEBUG:boto:Data: 2023-10-11T14:33:12.166 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.167 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.167 DEBUG:boto:Params: {} 2023-10-11T14:33:12.168 DEBUG:boto:Token: None 2023-10-11T14:33:12.168 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/ 2023-10-11T14:33:12.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ= 2023-10-11T14:33:12.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ=', 'Content-Length': '0'} 2023-10-11T14:33:12.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045513c931b7c0930-006526b228-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.172 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:33:12.172 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:33:12.173 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.173 DEBUG:boto:Path: /dghqvd-8/_myobj 2023-10-11T14:33:12.173 DEBUG:boto:Data: 2023-10-11T14:33:12.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.175 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.175 DEBUG:boto:Params: {} 2023-10-11T14:33:12.175 DEBUG:boto:Token: None 2023-10-11T14:33:12.176 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/_myobj 2023-10-11T14:33:12.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W1aLI26wmJtSDknCz6cW4JDNH08= 2023-10-11T14:33:12.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W1aLI26wmJtSDknCz6cW4JDNH08='} 2023-10-11T14:33:12.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea92a91cdaf27c21-006526b228-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.219 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:33:12.220 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:33:12.220 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.220 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:33:12.221 DEBUG:boto:Data: 2023-10-11T14:33:12.221 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.222 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.222 DEBUG:boto:Params: {} 2023-10-11T14:33:12.223 DEBUG:boto:Token: None 2023-10-11T14:33:12.223 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/ 2023-10-11T14:33:12.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ= 2023-10-11T14:33:12.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ=', 'Content-Length': '0'} 2023-10-11T14:33:12.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fd76bb27062e200-006526b228-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.227 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:33:12.227 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:33:12.228 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.228 DEBUG:boto:Path: /dghqvd-8/%3A 2023-10-11T14:33:12.229 DEBUG:boto:Data: 2023-10-11T14:33:12.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.230 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.230 DEBUG:boto:Params: {} 2023-10-11T14:33:12.231 DEBUG:boto:Token: None 2023-10-11T14:33:12.231 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/%3A 2023-10-11T14:33:12.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gj20sOGrrYFsmGeMt0r3dAN2y58= 2023-10-11T14:33:12.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gj20sOGrrYFsmGeMt0r3dAN2y58='} 2023-10-11T14:33:12.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000454665170c154b4d-006526b228-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.240 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:33:12.240 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:33:12.241 DEBUG:boto:Method: HEAD 2023-10-11T14:33:12.241 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:33:12.241 DEBUG:boto:Data: 2023-10-11T14:33:12.242 DEBUG:boto:Headers: {} 2023-10-11T14:33:12.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.243 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.243 DEBUG:boto:Params: {} 2023-10-11T14:33:12.244 DEBUG:boto:Token: None 2023-10-11T14:33:12.244 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/ 2023-10-11T14:33:12.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ= 2023-10-11T14:33:12.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZMC1V0D/rRMTADIy/Ny9tFGI5LQ=', 'Content-Length': '0'} 2023-10-11T14:33:12.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000190f5422167c132e-006526b228-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.248 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:33:12.248 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:33:12.248 DEBUG:boto:Method: PUT 2023-10-11T14:33:12.249 DEBUG:boto:Path: /dghqvd-8/%26 2023-10-11T14:33:12.249 DEBUG:boto:Data: 2023-10-11T14:33:12.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:33:12.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:33:12.250 DEBUG:boto:Port: 8001 2023-10-11T14:33:12.251 DEBUG:boto:Params: {} 2023-10-11T14:33:12.251 DEBUG:boto:Token: None 2023-10-11T14:33:12.252 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:33:12 GMT /dghqvd-8/%26 2023-10-11T14:33:12.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5IqDb2oy9/idyu97kEZSAiPcX/A= 2023-10-11T14:33:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:33:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5IqDb2oy9/idyu97kEZSAiPcX/A='} 2023-10-11T14:33:12.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7d5061925b599a0-006526b228-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:33:12.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:12.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:12.296 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:12.347 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:12.375 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:33:12.385 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:33:12.386 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.386 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:12.386 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:33:12.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.387 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.387 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:12.387 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:12.387 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.388 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.388 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.974196_108.1", 2023-10-11T14:33:12.388 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:33:11.974196Z" 2023-10-11T14:33:12.388 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.388 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.389 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:12.389 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:33:12.389 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.389 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:33:12.390 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.974196_108.1', 3: ''} 2023-10-11T14:33:12.391 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:33:12.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:12.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:33:12.392 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:33:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:12.442 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:12.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:12.467+0000 7f8ca838c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:12.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:12.468+0000 7f8ca838c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:33:12.480 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:33:12.481 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:33:12.481 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:33:12.481 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:33:12.481 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:33:12.482 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:33:12.482 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:12.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.482 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:33:12.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.483 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:33:12.483 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:12.483 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:12.484 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:33:12.484 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:12.484 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:33:12.484 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:12.484 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:33:12.485 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:12.485 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.485 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.485 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:33:12.486 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:12.486 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:12.486 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:12.486 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:12.487 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:12.487 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:12.487 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:12.487 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:12.487 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.488 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.488 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.488 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:33:12.488 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:12.489 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:12.489 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034791.974196_108.1", 2023-10-11T14:33:12.489 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:12.489 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:12.489 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:12.490 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:33:11.974196Z", 2023-10-11T14:33:12.490 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:12.490 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:33:12.491 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:33:12.491 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:33:12.491 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:33:12.491 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:33:12.492 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:33:12.492 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:33:12.492 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:33:12.492 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:33:12.492 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:33:12.493 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.493 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:33:12.493 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:33:12.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.493 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:33:12.494 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:33:12.494 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:33:12.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:33:12.494 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:33:12Z" 2023-10-11T14:33:12.495 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:33:12.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034791.974196_108.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:33:11.974196Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:33:12Z" } 2023-10-11T14:33:12.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034791.974196_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:33:11.974196Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-10-11T14:33:12.497 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.974196_108.1', 3: ''} 2023-10-11T14:33:12.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034791.974196_108.1', 3: ''} 2023-10-11T14:33:12.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:33:12.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:33:12.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:33:12.499 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:33:12.556 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:33:12.556 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:33:12.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:12.586+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:17.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:17.587+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:22.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:22.587+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:27.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:27.588+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:32.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:32.589+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:37.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:37.589+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:42.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:42.590+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:47.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:47.591+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:52.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:52.590+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:33:57.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:33:57.591+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:02.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:02.592+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:07.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:07.593+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:12.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:12.593+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:17.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:17.594+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:22.594+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:27.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:27.595+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:32.596+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:37.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:37.597+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:42.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:42.598+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:47.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:47.598+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:52.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:52.598+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:34:57.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:34:57.599+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:35:02.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:02.600+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:35:07.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:07.600+0000 7f9d29c87780 1 waiting to reach incremental sync.. 2023-10-11T14:35:12.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:12.602+0000 7f9d29c87780 1 bucket sync caught up with source: 2023-10-11T14:35:12.605 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.6.6, , 00000000001.6.6, , , , 00000000001.6.6, 00000000001.4.6, , ] 2023-10-11T14:35:12.605 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.6.6, , 00000000001.6.6, , , , 00000000001.6.6, 00000000001.4.6, , ] 2023-10-11T14:35:12.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:12.602+0000 7f9d29c87780 0 bucket checkpoint complete 2023-10-11T14:35:12.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-7 zones={a2, a1} 2023-10-11T14:35:12.621 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.621 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.622 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.622 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:35:12.623 DEBUG:boto:Data: 2023-10-11T14:35:12.623 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.624 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.624 DEBUG:boto:Params: {} 2023-10-11T14:35:12.625 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:35:12.625 DEBUG:boto:Token: None 2023-10-11T14:35:12.626 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.626 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Ry0gtMEr3TF7/5cBVL+RSe6Urc= 2023-10-11T14:35:12.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Ry0gtMEr3TF7/5cBVL+RSe6Urc=', 'Content-Length': '0'} 2023-10-11T14:35:12.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab07a839db4f0b63-006526b2a0-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.632 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.632 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.633 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.633 DEBUG:boto:Path: /dghqvd-7/ 2023-10-11T14:35:12.633 DEBUG:boto:Data: 2023-10-11T14:35:12.634 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.635 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.635 DEBUG:boto:Params: {} 2023-10-11T14:35:12.635 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:35:12.636 DEBUG:boto:Token: None 2023-10-11T14:35:12.636 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.637 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Ry0gtMEr3TF7/5cBVL+RSe6Urc= 2023-10-11T14:35:12.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Ry0gtMEr3TF7/5cBVL+RSe6Urc=', 'Content-Length': '0'} 2023-10-11T14:35:12.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb3224d5255b67f4-006526b2a0-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:35:12.642 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.643 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.644 DEBUG:boto:path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.644 DEBUG:boto:auth_path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.644 DEBUG:boto:Method: GET 2023-10-11T14:35:12.645 DEBUG:boto:Path: /dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.645 DEBUG:boto:Data: 2023-10-11T14:35:12.646 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.647 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.647 DEBUG:boto:Params: {} 2023-10-11T14:35:12.647 DEBUG:boto:Token: None 2023-10-11T14:35:12.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/?versions 2023-10-11T14:35:12.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8= 2023-10-11T14:35:12.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8=', 'Content-Length': '0'} 2023-10-11T14:35:12.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086732cc0cd33c698-006526b2a0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.652 DEBUG:boto:b'dghqvd-7999false&nulltrue2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.654 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-11T14:35:12.654 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-11T14:35:12.655 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-11T14:35:12.655 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:35:12.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:35:12.656 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.656 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.657 DEBUG:boto:path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.657 DEBUG:boto:auth_path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.658 DEBUG:boto:Method: GET 2023-10-11T14:35:12.658 DEBUG:boto:Path: /dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.658 DEBUG:boto:Data: 2023-10-11T14:35:12.659 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.659 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.660 DEBUG:boto:Params: {} 2023-10-11T14:35:12.660 DEBUG:boto:Token: None 2023-10-11T14:35:12.661 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/?versions 2023-10-11T14:35:12.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8= 2023-10-11T14:35:12.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8=', 'Content-Length': '0'} 2023-10-11T14:35:12.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dbc2cd1a35633eb-006526b2a0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.664 DEBUG:boto:b'dghqvd-7999false&nulltrue2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.665 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-11T14:35:12.665 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-11T14:35:12.666 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-11T14:35:12.666 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:35:12.667 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.667 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.667 DEBUG:boto:path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.668 DEBUG:boto:auth_path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.668 DEBUG:boto:Method: GET 2023-10-11T14:35:12.669 DEBUG:boto:Path: /dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.669 DEBUG:boto:Data: 2023-10-11T14:35:12.669 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.670 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.671 DEBUG:boto:Params: {} 2023-10-11T14:35:12.671 DEBUG:boto:Token: None 2023-10-11T14:35:12.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/?versions 2023-10-11T14:35:12.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8= 2023-10-11T14:35:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8=', 'Content-Length': '0'} 2023-10-11T14:35:12.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005714f3f764e78bf6-006526b2a0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.675 DEBUG:boto:b'dghqvd-7999false&nulltrue2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.676 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.677 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.677 DEBUG:boto:path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.678 DEBUG:boto:auth_path=/dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.678 DEBUG:boto:Method: GET 2023-10-11T14:35:12.678 DEBUG:boto:Path: /dghqvd-7/?versions&max-keys=999 2023-10-11T14:35:12.679 DEBUG:boto:Data: 2023-10-11T14:35:12.679 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.680 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.681 DEBUG:boto:Params: {} 2023-10-11T14:35:12.681 DEBUG:boto:Token: None 2023-10-11T14:35:12.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/?versions 2023-10-11T14:35:12.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8= 2023-10-11T14:35:12.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:29FgIoRDFMg9NtIupcVbBWMDAz8=', 'Content-Length': '0'} 2023-10-11T14:35:12.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007da734446215f26e-006526b2a0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.686 DEBUG:boto:b'dghqvd-7999false&nulltrue2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:35:12.687 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.687 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.688 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.688 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.689 DEBUG:boto:Method: GET 2023-10-11T14:35:12.689 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.690 DEBUG:boto:Data: 2023-10-11T14:35:12.690 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.691 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.691 DEBUG:boto:Params: {} 2023-10-11T14:35:12.692 DEBUG:boto:Token: None 2023-10-11T14:35:12.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw= 2023-10-11T14:35:12.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw=', 'Content-Length': '0'} 2023-10-11T14:35:12.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043059023385ede75-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.742 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.743 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.744 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.744 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.744 DEBUG:boto:Method: GET 2023-10-11T14:35:12.745 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.745 DEBUG:boto:Data: 2023-10-11T14:35:12.746 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.747 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.747 DEBUG:boto:Params: {} 2023-10-11T14:35:12.747 DEBUG:boto:Token: None 2023-10-11T14:35:12.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw= 2023-10-11T14:35:12.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw=', 'Content-Length': '0'} 2023-10-11T14:35:12.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bd03ab7d83ee86e-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.752 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.752 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.753 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.753 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.754 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.754 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.754 DEBUG:boto:Data: 2023-10-11T14:35:12.755 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.756 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.756 DEBUG:boto:Params: {} 2023-10-11T14:35:12.756 DEBUG:boto:Token: None 2023-10-11T14:35:12.757 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pItyz6x4WTizlZrXHoWE+zlIntE= 2023-10-11T14:35:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pItyz6x4WTizlZrXHoWE+zlIntE=', 'Content-Length': '0'} 2023-10-11T14:35:12.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004458ae9fa2ff654e-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.761 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.761 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.761 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.762 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.762 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.763 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.763 DEBUG:boto:Data: 2023-10-11T14:35:12.763 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.764 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.765 DEBUG:boto:Params: {} 2023-10-11T14:35:12.765 DEBUG:boto:Token: None 2023-10-11T14:35:12.766 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pItyz6x4WTizlZrXHoWE+zlIntE= 2023-10-11T14:35:12.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pItyz6x4WTizlZrXHoWE+zlIntE=', 'Content-Length': '0'} 2023-10-11T14:35:12.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033377364ae148061-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:35:12.769 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.770 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.770 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.771 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.771 DEBUG:boto:Method: GET 2023-10-11T14:35:12.772 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.772 DEBUG:boto:Data: 2023-10-11T14:35:12.773 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.773 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.774 DEBUG:boto:Params: {} 2023-10-11T14:35:12.774 DEBUG:boto:Token: None 2023-10-11T14:35:12.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw= 2023-10-11T14:35:12.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw=', 'Content-Length': '0'} 2023-10-11T14:35:12.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001340363aec503373-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.778 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.779 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.779 DEBUG:boto:path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.780 DEBUG:boto:auth_path=/dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.780 DEBUG:boto:Method: GET 2023-10-11T14:35:12.780 DEBUG:boto:Path: /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.781 DEBUG:boto:Data: 2023-10-11T14:35:12.781 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.782 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.782 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.782 DEBUG:boto:Params: {} 2023-10-11T14:35:12.783 DEBUG:boto:Token: None 2023-10-11T14:35:12.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26?versionId=null 2023-10-11T14:35:12.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw= 2023-10-11T14:35:12.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j6yDulycnbJvPO7Z0cv6aZpGGOw=', 'Content-Length': '0'} 2023-10-11T14:35:12.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075209af38ab92545-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.787 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.788 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.788 DEBUG:boto:path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.789 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.789 DEBUG:boto:Method: GET 2023-10-11T14:35:12.789 DEBUG:boto:Path: /dghqvd-7/?prefix=%26 2023-10-11T14:35:12.790 DEBUG:boto:Data: 2023-10-11T14:35:12.790 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.791 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.792 DEBUG:boto:Params: {} 2023-10-11T14:35:12.792 DEBUG:boto:Token: None 2023-10-11T14:35:12.793 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e3a626e9d5b956-006526b2a0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.798 DEBUG:boto:b'dghqvd-7&1000false&2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.798 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.799 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.799 DEBUG:boto:path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.800 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.800 DEBUG:boto:Method: GET 2023-10-11T14:35:12.800 DEBUG:boto:Path: /dghqvd-7/?prefix=%26 2023-10-11T14:35:12.801 DEBUG:boto:Data: 2023-10-11T14:35:12.801 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.802 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.802 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.803 DEBUG:boto:Params: {} 2023-10-11T14:35:12.803 DEBUG:boto:Token: None 2023-10-11T14:35:12.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3ae4d06a1fa71c-006526b2a0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.808 DEBUG:boto:b'dghqvd-7&1000false&2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.808 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.809 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.809 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.810 DEBUG:boto:Path: /dghqvd-7/%26 2023-10-11T14:35:12.810 DEBUG:boto:Data: 2023-10-11T14:35:12.810 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.811 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.811 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.812 DEBUG:boto:Params: {} 2023-10-11T14:35:12.812 DEBUG:boto:Token: None 2023-10-11T14:35:12.813 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26 2023-10-11T14:35:12.813 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i3dMcoxW/Djp0pSWxF6ZF7m2No0= 2023-10-11T14:35:12.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i3dMcoxW/Djp0pSWxF6ZF7m2No0=', 'Content-Length': '0'} 2023-10-11T14:35:12.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005aa90496b239935-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.816 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.817 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.817 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.817 DEBUG:boto:Path: /dghqvd-7/%26 2023-10-11T14:35:12.818 DEBUG:boto:Data: 2023-10-11T14:35:12.818 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.819 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.819 DEBUG:boto:Params: {} 2023-10-11T14:35:12.820 DEBUG:boto:Token: None 2023-10-11T14:35:12.820 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26 2023-10-11T14:35:12.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i3dMcoxW/Djp0pSWxF6ZF7m2No0= 2023-10-11T14:35:12.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i3dMcoxW/Djp0pSWxF6ZF7m2No0=', 'Content-Length': '0'} 2023-10-11T14:35:12.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c1c3690101415c1-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:35:12.824 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.825 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.825 DEBUG:boto:Method: GET 2023-10-11T14:35:12.826 DEBUG:boto:Path: /dghqvd-7/%26 2023-10-11T14:35:12.826 DEBUG:boto:Data: 2023-10-11T14:35:12.827 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.827 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.828 DEBUG:boto:Params: {} 2023-10-11T14:35:12.828 DEBUG:boto:Token: None 2023-10-11T14:35:12.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26 2023-10-11T14:35:12.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UtdzIx1PgR3Z8k3ZXWFJL4YOMpY= 2023-10-11T14:35:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UtdzIx1PgR3Z8k3ZXWFJL4YOMpY=', 'Content-Length': '0'} 2023-10-11T14:35:12.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000091665d31694b2f6-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.833 DEBUG:boto:path=/dghqvd-7/%26 2023-10-11T14:35:12.833 DEBUG:boto:auth_path=/dghqvd-7/%26 2023-10-11T14:35:12.834 DEBUG:boto:Method: GET 2023-10-11T14:35:12.834 DEBUG:boto:Path: /dghqvd-7/%26 2023-10-11T14:35:12.834 DEBUG:boto:Data: 2023-10-11T14:35:12.835 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.836 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.836 DEBUG:boto:Params: {} 2023-10-11T14:35:12.837 DEBUG:boto:Token: None 2023-10-11T14:35:12.837 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%26 2023-10-11T14:35:12.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UtdzIx1PgR3Z8k3ZXWFJL4YOMpY= 2023-10-11T14:35:12.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UtdzIx1PgR3Z8k3ZXWFJL4YOMpY=', 'Content-Length': '0'} 2023-10-11T14:35:12.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ca7fb821e438ecd-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.882 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.883 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.884 DEBUG:boto:path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.884 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.884 DEBUG:boto:Method: GET 2023-10-11T14:35:12.885 DEBUG:boto:Path: /dghqvd-7/?prefix=%26 2023-10-11T14:35:12.885 DEBUG:boto:Data: 2023-10-11T14:35:12.886 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.887 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.887 DEBUG:boto:Params: {} 2023-10-11T14:35:12.887 DEBUG:boto:Token: None 2023-10-11T14:35:12.888 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c98cf26fe2d0c39a-006526b2a0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.892 DEBUG:boto:b'dghqvd-7&1000false&2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.893 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.894 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.894 DEBUG:boto:path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.895 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%26 2023-10-11T14:35:12.895 DEBUG:boto:Method: GET 2023-10-11T14:35:12.895 DEBUG:boto:Path: /dghqvd-7/?prefix=%26 2023-10-11T14:35:12.896 DEBUG:boto:Data: 2023-10-11T14:35:12.896 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.897 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.897 DEBUG:boto:Params: {} 2023-10-11T14:35:12.898 DEBUG:boto:Token: None 2023-10-11T14:35:12.899 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.899 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dca1b747078bc252-006526b2a0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.902 DEBUG:boto:b'dghqvd-7&1000false&2023-10-11T14:33:12.137Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:35:12.903 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.904 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.904 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.905 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.905 DEBUG:boto:Method: GET 2023-10-11T14:35:12.906 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.906 DEBUG:boto:Data: 2023-10-11T14:35:12.906 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.907 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.908 DEBUG:boto:Params: {} 2023-10-11T14:35:12.908 DEBUG:boto:Token: None 2023-10-11T14:35:12.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4= 2023-10-11T14:35:12.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4=', 'Content-Length': '0'} 2023-10-11T14:35:12.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001687aa1eef4562b8-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.913 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.913 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.914 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.914 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.915 DEBUG:boto:Method: GET 2023-10-11T14:35:12.915 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.915 DEBUG:boto:Data: 2023-10-11T14:35:12.916 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.917 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.917 DEBUG:boto:Params: {} 2023-10-11T14:35:12.918 DEBUG:boto:Token: None 2023-10-11T14:35:12.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.919 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4= 2023-10-11T14:35:12.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4=', 'Content-Length': '0'} 2023-10-11T14:35:12.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a25712437b9c51d-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.922 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.922 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.922 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.923 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.923 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.924 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.924 DEBUG:boto:Data: 2023-10-11T14:35:12.924 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.925 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.926 DEBUG:boto:Params: {} 2023-10-11T14:35:12.926 DEBUG:boto:Token: None 2023-10-11T14:35:12.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0IFy7SAN63fpZlEj/cVowz0iHQ= 2023-10-11T14:35:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0IFy7SAN63fpZlEj/cVowz0iHQ=', 'Content-Length': '0'} 2023-10-11T14:35:12.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d6de1439c0b049f-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.930 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.931 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.931 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.931 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.932 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.932 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.933 DEBUG:boto:Data: 2023-10-11T14:35:12.933 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.934 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.934 DEBUG:boto:Params: {} 2023-10-11T14:35:12.935 DEBUG:boto:Token: None 2023-10-11T14:35:12.935 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0IFy7SAN63fpZlEj/cVowz0iHQ= 2023-10-11T14:35:12.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0IFy7SAN63fpZlEj/cVowz0iHQ=', 'Content-Length': '0'} 2023-10-11T14:35:12.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaaf3ace01c125d0-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:35:12.939 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.940 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.940 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.940 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.941 DEBUG:boto:Method: GET 2023-10-11T14:35:12.941 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.942 DEBUG:boto:Data: 2023-10-11T14:35:12.942 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.943 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.943 DEBUG:boto:Params: {} 2023-10-11T14:35:12.944 DEBUG:boto:Token: None 2023-10-11T14:35:12.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4= 2023-10-11T14:35:12.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4=', 'Content-Length': '0'} 2023-10-11T14:35:12.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d19e2413842fcfa3-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.948 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.949 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.949 DEBUG:boto:path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.950 DEBUG:boto:auth_path=/dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.950 DEBUG:boto:Method: GET 2023-10-11T14:35:12.951 DEBUG:boto:Path: /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.951 DEBUG:boto:Data: 2023-10-11T14:35:12.951 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.952 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.953 DEBUG:boto:Params: {} 2023-10-11T14:35:12.953 DEBUG:boto:Token: None 2023-10-11T14:35:12.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A?versionId=null 2023-10-11T14:35:12.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4= 2023-10-11T14:35:12.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fiG+4qsJVww8P3/7WXRn8ON74T4=', 'Content-Length': '0'} 2023-10-11T14:35:12.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e176b2b1e9c43535-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.957 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.958 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.958 DEBUG:boto:path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.959 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.959 DEBUG:boto:Method: GET 2023-10-11T14:35:12.960 DEBUG:boto:Path: /dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.960 DEBUG:boto:Data: 2023-10-11T14:35:12.960 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.961 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.962 DEBUG:boto:Params: {} 2023-10-11T14:35:12.962 DEBUG:boto:Token: None 2023-10-11T14:35:12.963 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.963 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d658b77adb8c66e8-006526b2a0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.967 DEBUG:boto:b'dghqvd-7:1000false:2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.968 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:12.968 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:12.968 DEBUG:boto:path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.969 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.969 DEBUG:boto:Method: GET 2023-10-11T14:35:12.970 DEBUG:boto:Path: /dghqvd-7/?prefix=%3A 2023-10-11T14:35:12.970 DEBUG:boto:Data: 2023-10-11T14:35:12.971 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.971 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.972 DEBUG:boto:Params: {} 2023-10-11T14:35:12.972 DEBUG:boto:Token: None 2023-10-11T14:35:12.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/ 2023-10-11T14:35:12.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg= 2023-10-11T14:35:12.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B6gH8F92bu6z+VatCtSecMgSDQg=', 'Content-Length': '0'} 2023-10-11T14:35:12.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab98883ea7d21ac4-006526b2a0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.977 DEBUG:boto:b'dghqvd-7:1000false:2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:12.978 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.978 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.979 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.979 DEBUG:boto:Path: /dghqvd-7/%3A 2023-10-11T14:35:12.980 DEBUG:boto:Data: 2023-10-11T14:35:12.980 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.981 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.981 DEBUG:boto:Params: {} 2023-10-11T14:35:12.982 DEBUG:boto:Token: None 2023-10-11T14:35:12.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A 2023-10-11T14:35:12.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kXBclRY/kf7wcBUQweUFW72L7j8= 2023-10-11T14:35:12.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kXBclRY/kf7wcBUQweUFW72L7j8=', 'Content-Length': '0'} 2023-10-11T14:35:12.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ff179884cff63f7-006526b2a0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.985 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.986 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.986 DEBUG:boto:Method: HEAD 2023-10-11T14:35:12.987 DEBUG:boto:Path: /dghqvd-7/%3A 2023-10-11T14:35:12.987 DEBUG:boto:Data: 2023-10-11T14:35:12.987 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:12.988 DEBUG:boto:Port: 8000 2023-10-11T14:35:12.988 DEBUG:boto:Params: {} 2023-10-11T14:35:12.989 DEBUG:boto:Token: None 2023-10-11T14:35:12.989 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A 2023-10-11T14:35:12.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kXBclRY/kf7wcBUQweUFW72L7j8= 2023-10-11T14:35:12.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kXBclRY/kf7wcBUQweUFW72L7j8=', 'Content-Length': '0'} 2023-10-11T14:35:12.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038ed2f01539e189f-006526b2a0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:12.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:35:12.993 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:12.994 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:12.994 DEBUG:boto:Method: GET 2023-10-11T14:35:12.995 DEBUG:boto:Path: /dghqvd-7/%3A 2023-10-11T14:35:12.995 DEBUG:boto:Data: 2023-10-11T14:35:12.995 DEBUG:boto:Headers: {} 2023-10-11T14:35:12.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:12.996 DEBUG:boto:Port: 8001 2023-10-11T14:35:12.997 DEBUG:boto:Params: {} 2023-10-11T14:35:12.997 DEBUG:boto:Token: None 2023-10-11T14:35:12.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:12 GMT /dghqvd-7/%3A 2023-10-11T14:35:12.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lDj/t15k2b1pcLoAUjekoddAxWA= 2023-10-11T14:35:13.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lDj/t15k2b1pcLoAUjekoddAxWA=', 'Content-Length': '0'} 2023-10-11T14:35:13.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1e2d300bd8de911-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.050 DEBUG:boto:path=/dghqvd-7/%3A 2023-10-11T14:35:13.051 DEBUG:boto:auth_path=/dghqvd-7/%3A 2023-10-11T14:35:13.052 DEBUG:boto:Method: GET 2023-10-11T14:35:13.052 DEBUG:boto:Path: /dghqvd-7/%3A 2023-10-11T14:35:13.052 DEBUG:boto:Data: 2023-10-11T14:35:13.053 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.054 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.054 DEBUG:boto:Params: {} 2023-10-11T14:35:13.055 DEBUG:boto:Token: None 2023-10-11T14:35:13.055 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/%3A 2023-10-11T14:35:13.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NDTkTgPMAaiNbrZoXYG8oQVduTQ= 2023-10-11T14:35:13.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NDTkTgPMAaiNbrZoXYG8oQVduTQ=', 'Content-Length': '0'} 2023-10-11T14:35:13.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1365b318a6373c5-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.059 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.059 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.060 DEBUG:boto:path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.060 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.061 DEBUG:boto:Method: GET 2023-10-11T14:35:13.061 DEBUG:boto:Path: /dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.061 DEBUG:boto:Data: 2023-10-11T14:35:13.062 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.062 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.063 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.063 DEBUG:boto:Params: {} 2023-10-11T14:35:13.064 DEBUG:boto:Token: None 2023-10-11T14:35:13.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.065 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45eebef962f7e1d-006526b2a1-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.068 DEBUG:boto:b'dghqvd-7:1000false:2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.069 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.070 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.070 DEBUG:boto:path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.070 DEBUG:boto:auth_path=/dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.071 DEBUG:boto:Method: GET 2023-10-11T14:35:13.071 DEBUG:boto:Path: /dghqvd-7/?prefix=%3A 2023-10-11T14:35:13.072 DEBUG:boto:Data: 2023-10-11T14:35:13.072 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.073 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.073 DEBUG:boto:Params: {} 2023-10-11T14:35:13.074 DEBUG:boto:Token: None 2023-10-11T14:35:13.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c4967bebd8534b-006526b2a1-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.079 DEBUG:boto:b'dghqvd-7:1000false:2023-10-11T14:33:12.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:35:13.080 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.080 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.081 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.081 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.081 DEBUG:boto:Method: GET 2023-10-11T14:35:13.082 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.082 DEBUG:boto:Data: 2023-10-11T14:35:13.083 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.083 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.084 DEBUG:boto:Params: {} 2023-10-11T14:35:13.084 DEBUG:boto:Token: None 2023-10-11T14:35:13.085 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4= 2023-10-11T14:35:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4=', 'Content-Length': '0'} 2023-10-11T14:35:13.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2181db0d22df2a0-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.089 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.089 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.090 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.090 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.091 DEBUG:boto:Method: GET 2023-10-11T14:35:13.091 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.092 DEBUG:boto:Data: 2023-10-11T14:35:13.092 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.093 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.093 DEBUG:boto:Params: {} 2023-10-11T14:35:13.094 DEBUG:boto:Token: None 2023-10-11T14:35:13.094 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4= 2023-10-11T14:35:13.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4=', 'Content-Length': '0'} 2023-10-11T14:35:13.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007baf949299c9f29b-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.142 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.143 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.143 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.144 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.144 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.145 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.145 DEBUG:boto:Data: 2023-10-11T14:35:13.146 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.146 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.147 DEBUG:boto:Params: {} 2023-10-11T14:35:13.147 DEBUG:boto:Token: None 2023-10-11T14:35:13.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AGxDVja8HHveedOIJMvHDTq30vs= 2023-10-11T14:35:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AGxDVja8HHveedOIJMvHDTq30vs=', 'Content-Length': '0'} 2023-10-11T14:35:13.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092d3c7845052dbf3-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.152 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.152 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.152 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.153 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.153 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.154 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.154 DEBUG:boto:Data: 2023-10-11T14:35:13.154 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.155 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.156 DEBUG:boto:Params: {} 2023-10-11T14:35:13.156 DEBUG:boto:Token: None 2023-10-11T14:35:13.157 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AGxDVja8HHveedOIJMvHDTq30vs= 2023-10-11T14:35:13.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AGxDVja8HHveedOIJMvHDTq30vs=', 'Content-Length': '0'} 2023-10-11T14:35:13.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3d1faba4e025cfd-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:35:13.160 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.161 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.161 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.162 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.162 DEBUG:boto:Method: GET 2023-10-11T14:35:13.163 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.163 DEBUG:boto:Data: 2023-10-11T14:35:13.164 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.165 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.165 DEBUG:boto:Params: {} 2023-10-11T14:35:13.165 DEBUG:boto:Token: None 2023-10-11T14:35:13.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4= 2023-10-11T14:35:13.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4=', 'Content-Length': '0'} 2023-10-11T14:35:13.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb95e4566d2f121b-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.210 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.211 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.211 DEBUG:boto:path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.212 DEBUG:boto:auth_path=/dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.212 DEBUG:boto:Method: GET 2023-10-11T14:35:13.213 DEBUG:boto:Path: /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.213 DEBUG:boto:Data: 2023-10-11T14:35:13.214 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.215 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.215 DEBUG:boto:Params: {} 2023-10-11T14:35:13.215 DEBUG:boto:Token: None 2023-10-11T14:35:13.216 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj?versionId=null 2023-10-11T14:35:13.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4= 2023-10-11T14:35:13.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D3TKbdDpjRDe0QDKhleVy9npct4=', 'Content-Length': '0'} 2023-10-11T14:35:13.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d1c6618f276baec-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.220 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.220 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.220 DEBUG:boto:path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.221 DEBUG:boto:auth_path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.221 DEBUG:boto:Method: GET 2023-10-11T14:35:13.222 DEBUG:boto:Path: /dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.222 DEBUG:boto:Data: 2023-10-11T14:35:13.223 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.223 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.223 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.224 DEBUG:boto:Params: {} 2023-10-11T14:35:13.224 DEBUG:boto:Token: None 2023-10-11T14:35:13.225 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.225 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014fe662dbfd36a7b-006526b2a1-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.229 DEBUG:boto:b'dghqvd-7_myobj1000false_myobj2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.230 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.230 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.231 DEBUG:boto:path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.231 DEBUG:boto:auth_path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.232 DEBUG:boto:Method: GET 2023-10-11T14:35:13.232 DEBUG:boto:Path: /dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.233 DEBUG:boto:Data: 2023-10-11T14:35:13.233 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.233 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.234 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.234 DEBUG:boto:Params: {} 2023-10-11T14:35:13.235 DEBUG:boto:Token: None 2023-10-11T14:35:13.235 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa920ef58be2af9d-006526b2a1-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.240 DEBUG:boto:b'dghqvd-7_myobj1000false_myobj2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.240 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.241 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.241 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.241 DEBUG:boto:Path: /dghqvd-7/_myobj 2023-10-11T14:35:13.242 DEBUG:boto:Data: 2023-10-11T14:35:13.242 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.243 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.243 DEBUG:boto:Params: {} 2023-10-11T14:35:13.244 DEBUG:boto:Token: None 2023-10-11T14:35:13.244 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj 2023-10-11T14:35:13.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DR/k1D40qA/KSv/Pxy5Q8QZ2h4I= 2023-10-11T14:35:13.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DR/k1D40qA/KSv/Pxy5Q8QZ2h4I=', 'Content-Length': '0'} 2023-10-11T14:35:13.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002228be64b9351ce2-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.248 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.248 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.249 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.249 DEBUG:boto:Path: /dghqvd-7/_myobj 2023-10-11T14:35:13.250 DEBUG:boto:Data: 2023-10-11T14:35:13.250 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.251 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.251 DEBUG:boto:Params: {} 2023-10-11T14:35:13.252 DEBUG:boto:Token: None 2023-10-11T14:35:13.252 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj 2023-10-11T14:35:13.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DR/k1D40qA/KSv/Pxy5Q8QZ2h4I= 2023-10-11T14:35:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DR/k1D40qA/KSv/Pxy5Q8QZ2h4I=', 'Content-Length': '0'} 2023-10-11T14:35:13.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a97d0939f9ec62e-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:35:13.256 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.256 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.257 DEBUG:boto:Method: GET 2023-10-11T14:35:13.257 DEBUG:boto:Path: /dghqvd-7/_myobj 2023-10-11T14:35:13.258 DEBUG:boto:Data: 2023-10-11T14:35:13.258 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.259 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.259 DEBUG:boto:Params: {} 2023-10-11T14:35:13.260 DEBUG:boto:Token: None 2023-10-11T14:35:13.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj 2023-10-11T14:35:13.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pthmDiP4wXGzSnXjbC6NAxM+G50= 2023-10-11T14:35:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pthmDiP4wXGzSnXjbC6NAxM+G50=', 'Content-Length': '0'} 2023-10-11T14:35:13.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000620151528100661e-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.264 DEBUG:boto:path=/dghqvd-7/_myobj 2023-10-11T14:35:13.265 DEBUG:boto:auth_path=/dghqvd-7/_myobj 2023-10-11T14:35:13.265 DEBUG:boto:Method: GET 2023-10-11T14:35:13.265 DEBUG:boto:Path: /dghqvd-7/_myobj 2023-10-11T14:35:13.266 DEBUG:boto:Data: 2023-10-11T14:35:13.266 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.267 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.268 DEBUG:boto:Params: {} 2023-10-11T14:35:13.268 DEBUG:boto:Token: None 2023-10-11T14:35:13.269 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/_myobj 2023-10-11T14:35:13.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pthmDiP4wXGzSnXjbC6NAxM+G50= 2023-10-11T14:35:13.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pthmDiP4wXGzSnXjbC6NAxM+G50=', 'Content-Length': '0'} 2023-10-11T14:35:13.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020e68cba7ca8c274-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.314 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.315 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.316 DEBUG:boto:path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.316 DEBUG:boto:auth_path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.316 DEBUG:boto:Method: GET 2023-10-11T14:35:13.317 DEBUG:boto:Path: /dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.317 DEBUG:boto:Data: 2023-10-11T14:35:13.318 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.319 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.319 DEBUG:boto:Params: {} 2023-10-11T14:35:13.320 DEBUG:boto:Token: None 2023-10-11T14:35:13.320 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.321 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a4cf6cafca37ac-006526b2a1-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.324 DEBUG:boto:b'dghqvd-7_myobj1000false_myobj2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.325 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.326 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.326 DEBUG:boto:path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.327 DEBUG:boto:auth_path=/dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.327 DEBUG:boto:Method: GET 2023-10-11T14:35:13.327 DEBUG:boto:Path: /dghqvd-7/?prefix=_myobj 2023-10-11T14:35:13.328 DEBUG:boto:Data: 2023-10-11T14:35:13.328 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.329 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.329 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.330 DEBUG:boto:Params: {} 2023-10-11T14:35:13.330 DEBUG:boto:Token: None 2023-10-11T14:35:13.331 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37aabe2276e2135-006526b2a1-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.335 DEBUG:boto:b'dghqvd-7_myobj1000false_myobj2023-10-11T14:33:12.058Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:35:13.336 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.336 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.337 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.337 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.337 DEBUG:boto:Method: GET 2023-10-11T14:35:13.338 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.338 DEBUG:boto:Data: 2023-10-11T14:35:13.339 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.340 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.340 DEBUG:boto:Params: {} 2023-10-11T14:35:13.340 DEBUG:boto:Token: None 2023-10-11T14:35:13.341 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs= 2023-10-11T14:35:13.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs=', 'Content-Length': '0'} 2023-10-11T14:35:13.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001451449f759bcb18-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.386 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.387 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.387 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.388 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.388 DEBUG:boto:Method: GET 2023-10-11T14:35:13.389 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.389 DEBUG:boto:Data: 2023-10-11T14:35:13.390 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.390 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.390 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.391 DEBUG:boto:Params: {} 2023-10-11T14:35:13.391 DEBUG:boto:Token: None 2023-10-11T14:35:13.392 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs= 2023-10-11T14:35:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs=', 'Content-Length': '0'} 2023-10-11T14:35:13.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e81616c0640ad557-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.395 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.396 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.396 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.397 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.397 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.397 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.398 DEBUG:boto:Data: 2023-10-11T14:35:13.398 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.399 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.400 DEBUG:boto:Params: {} 2023-10-11T14:35:13.400 DEBUG:boto:Token: None 2023-10-11T14:35:13.400 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2hNVDXDFXt7zTV/L3bfk7X2oTZ0= 2023-10-11T14:35:13.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2hNVDXDFXt7zTV/L3bfk7X2oTZ0=', 'Content-Length': '0'} 2023-10-11T14:35:13.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006515fad877b8fc03-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.404 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.404 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.405 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.405 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.406 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.406 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.407 DEBUG:boto:Data: 2023-10-11T14:35:13.407 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.408 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.408 DEBUG:boto:Params: {} 2023-10-11T14:35:13.409 DEBUG:boto:Token: None 2023-10-11T14:35:13.409 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2hNVDXDFXt7zTV/L3bfk7X2oTZ0= 2023-10-11T14:35:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2hNVDXDFXt7zTV/L3bfk7X2oTZ0=', 'Content-Length': '0'} 2023-10-11T14:35:13.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a22f9b14be2104f9-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:35:13.414 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.415 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.415 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.415 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.416 DEBUG:boto:Method: GET 2023-10-11T14:35:13.416 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.417 DEBUG:boto:Data: 2023-10-11T14:35:13.417 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.418 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.418 DEBUG:boto:Params: {} 2023-10-11T14:35:13.419 DEBUG:boto:Token: None 2023-10-11T14:35:13.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs= 2023-10-11T14:35:13.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs=', 'Content-Length': '0'} 2023-10-11T14:35:13.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000829c90dcdf6506c0-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.423 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.423 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.424 DEBUG:boto:path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.424 DEBUG:boto:auth_path=/dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.425 DEBUG:boto:Method: GET 2023-10-11T14:35:13.425 DEBUG:boto:Path: /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.425 DEBUG:boto:Data: 2023-10-11T14:35:13.426 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.427 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.427 DEBUG:boto:Params: {} 2023-10-11T14:35:13.428 DEBUG:boto:Token: None 2023-10-11T14:35:13.428 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj?versionId=null 2023-10-11T14:35:13.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs= 2023-10-11T14:35:13.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wNwgCLTEdnOSF51ILtF2mLQcBBs=', 'Content-Length': '0'} 2023-10-11T14:35:13.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c84acd722cda24a-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.431 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.432 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.432 DEBUG:boto:path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.433 DEBUG:boto:auth_path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.433 DEBUG:boto:Method: GET 2023-10-11T14:35:13.433 DEBUG:boto:Path: /dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.434 DEBUG:boto:Data: 2023-10-11T14:35:13.434 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.435 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.435 DEBUG:boto:Params: {} 2023-10-11T14:35:13.436 DEBUG:boto:Token: None 2023-10-11T14:35:13.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000701e4a01e8a225d3-006526b2a1-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.482 DEBUG:boto:b'dghqvd-7myobj1000falsemyobj2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.483 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.484 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.484 DEBUG:boto:path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.484 DEBUG:boto:auth_path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.485 DEBUG:boto:Method: GET 2023-10-11T14:35:13.485 DEBUG:boto:Path: /dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.486 DEBUG:boto:Data: 2023-10-11T14:35:13.486 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.487 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.488 DEBUG:boto:Params: {} 2023-10-11T14:35:13.488 DEBUG:boto:Token: None 2023-10-11T14:35:13.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac829df8349a4fd3-006526b2a1-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.493 DEBUG:boto:b'dghqvd-7myobj1000falsemyobj2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.493 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.494 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.494 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.494 DEBUG:boto:Path: /dghqvd-7/myobj 2023-10-11T14:35:13.495 DEBUG:boto:Data: 2023-10-11T14:35:13.495 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.496 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.496 DEBUG:boto:Params: {} 2023-10-11T14:35:13.497 DEBUG:boto:Token: None 2023-10-11T14:35:13.497 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj 2023-10-11T14:35:13.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wVEN08U4E1F2Qt6OuDLDQGhes7U= 2023-10-11T14:35:13.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wVEN08U4E1F2Qt6OuDLDQGhes7U=', 'Content-Length': '0'} 2023-10-11T14:35:13.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000980fb5d27e97addd-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.501 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.501 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.502 DEBUG:boto:Method: HEAD 2023-10-11T14:35:13.502 DEBUG:boto:Path: /dghqvd-7/myobj 2023-10-11T14:35:13.503 DEBUG:boto:Data: 2023-10-11T14:35:13.503 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.504 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.504 DEBUG:boto:Params: {} 2023-10-11T14:35:13.505 DEBUG:boto:Token: None 2023-10-11T14:35:13.505 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj 2023-10-11T14:35:13.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wVEN08U4E1F2Qt6OuDLDQGhes7U= 2023-10-11T14:35:13.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wVEN08U4E1F2Qt6OuDLDQGhes7U=', 'Content-Length': '0'} 2023-10-11T14:35:13.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c65c04a5ecfda5f0-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:35:13.509 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.510 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.510 DEBUG:boto:Method: GET 2023-10-11T14:35:13.511 DEBUG:boto:Path: /dghqvd-7/myobj 2023-10-11T14:35:13.511 DEBUG:boto:Data: 2023-10-11T14:35:13.512 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.513 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.513 DEBUG:boto:Params: {} 2023-10-11T14:35:13.513 DEBUG:boto:Token: None 2023-10-11T14:35:13.514 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj 2023-10-11T14:35:13.514 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AQPeYKkrTMqEWUT+q7i5/kaA15Y= 2023-10-11T14:35:13.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AQPeYKkrTMqEWUT+q7i5/kaA15Y=', 'Content-Length': '0'} 2023-10-11T14:35:13.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000534b7b65b3f83985-006526b2a1-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.518 DEBUG:boto:path=/dghqvd-7/myobj 2023-10-11T14:35:13.518 DEBUG:boto:auth_path=/dghqvd-7/myobj 2023-10-11T14:35:13.519 DEBUG:boto:Method: GET 2023-10-11T14:35:13.519 DEBUG:boto:Path: /dghqvd-7/myobj 2023-10-11T14:35:13.519 DEBUG:boto:Data: 2023-10-11T14:35:13.520 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.521 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.521 DEBUG:boto:Params: {} 2023-10-11T14:35:13.522 DEBUG:boto:Token: None 2023-10-11T14:35:13.522 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/myobj 2023-10-11T14:35:13.523 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AQPeYKkrTMqEWUT+q7i5/kaA15Y= 2023-10-11T14:35:13.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AQPeYKkrTMqEWUT+q7i5/kaA15Y=', 'Content-Length': '0'} 2023-10-11T14:35:13.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5e9db88a0f30654-006526b2a1-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.526 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.526 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.527 DEBUG:boto:path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.527 DEBUG:boto:auth_path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.528 DEBUG:boto:Method: GET 2023-10-11T14:35:13.528 DEBUG:boto:Path: /dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.528 DEBUG:boto:Data: 2023-10-11T14:35:13.529 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:35:13.530 DEBUG:boto:Port: 8001 2023-10-11T14:35:13.530 DEBUG:boto:Params: {} 2023-10-11T14:35:13.531 DEBUG:boto:Token: None 2023-10-11T14:35:13.531 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006163284564469f14-006526b2a1-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.535 DEBUG:boto:b'dghqvd-7myobj1000falsemyobj2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.536 DEBUG:boto:path=/dghqvd-7/ 2023-10-11T14:35:13.536 DEBUG:boto:auth_path=/dghqvd-7/ 2023-10-11T14:35:13.537 DEBUG:boto:path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.537 DEBUG:boto:auth_path=/dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.538 DEBUG:boto:Method: GET 2023-10-11T14:35:13.538 DEBUG:boto:Path: /dghqvd-7/?prefix=myobj 2023-10-11T14:35:13.539 DEBUG:boto:Data: 2023-10-11T14:35:13.539 DEBUG:boto:Headers: {} 2023-10-11T14:35:13.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:35:13.540 DEBUG:boto:Port: 8000 2023-10-11T14:35:13.540 DEBUG:boto:Params: {} 2023-10-11T14:35:13.541 DEBUG:boto:Token: None 2023-10-11T14:35:13.541 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:35:13 GMT /dghqvd-7/ 2023-10-11T14:35:13.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so= 2023-10-11T14:35:13.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:35:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CLf+P252ial1PahjjCynhbco/so=', 'Content-Length': '0'} 2023-10-11T14:35:13.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9ece31ab6b3e6d4-006526b2a1-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:35:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:35:13.545 DEBUG:boto:b'dghqvd-7myobj1000falsemyobj2023-10-11T14:33:11.998Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:35:13.546 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-7 zones={a2, a1} 2023-10-11T14:35:13.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:35:13.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:35:13.548 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:35:13.611 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:35:13.612 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:35:13.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:13.645+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:18.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:18.646+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:23.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:23.646+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:28.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:28.646+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:33.647+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:38.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:38.647+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:43.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:43.648+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:48.652 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:48.649+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:53.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:53.650+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:35:58.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:35:58.650+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:36:03.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:03.651+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:36:08.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:08.652+0000 7fd8c7ccf780 1 waiting to reach incremental sync.. 2023-10-11T14:36:13.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:13.654+0000 7fd8c7ccf780 1 bucket sync caught up with source: 2023-10-11T14:36:13.657 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.9.6, , 00000000001.7.6, , , , 00000000001.10.6, 00000000001.9.6, , ] 2023-10-11T14:36:13.657 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.9.6, , 00000000001.7.6, , , , 00000000001.10.6, 00000000001.9.6, , ] 2023-10-11T14:36:13.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:13.654+0000 7fd8c7ccf780 0 bucket checkpoint complete 2023-10-11T14:36:13.672 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-8 zones={a1, a2} 2023-10-11T14:36:13.673 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.674 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.674 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.675 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:36:13.675 DEBUG:boto:Data: 2023-10-11T14:36:13.675 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.676 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.677 DEBUG:boto:Params: {} 2023-10-11T14:36:13.677 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:36:13.678 DEBUG:boto:Token: None 2023-10-11T14:36:13.678 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnmIfceHXyHgSJ//ncKeMzbX2Vo= 2023-10-11T14:36:13.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnmIfceHXyHgSJ//ncKeMzbX2Vo=', 'Content-Length': '0'} 2023-10-11T14:36:13.684 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b19f4e27dd0fe02-006526b2dd-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.685 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.685 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.686 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.686 DEBUG:boto:Path: /dghqvd-8/ 2023-10-11T14:36:13.687 DEBUG:boto:Data: 2023-10-11T14:36:13.687 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.688 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.688 DEBUG:boto:Params: {} 2023-10-11T14:36:13.689 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:36:13.689 DEBUG:boto:Token: None 2023-10-11T14:36:13.690 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnmIfceHXyHgSJ//ncKeMzbX2Vo= 2023-10-11T14:36:13.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnmIfceHXyHgSJ//ncKeMzbX2Vo=', 'Content-Length': '0'} 2023-10-11T14:36:13.695 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed345f69ddfa9816-006526b2dd-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.695 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:36:13.696 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.696 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.697 DEBUG:boto:path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.697 DEBUG:boto:auth_path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.697 DEBUG:boto:Method: GET 2023-10-11T14:36:13.698 DEBUG:boto:Path: /dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.698 DEBUG:boto:Data: 2023-10-11T14:36:13.699 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.699 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.700 DEBUG:boto:Params: {} 2023-10-11T14:36:13.700 DEBUG:boto:Token: None 2023-10-11T14:36:13.701 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/?versions 2023-10-11T14:36:13.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c= 2023-10-11T14:36:13.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c=', 'Content-Length': '0'} 2023-10-11T14:36:13.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01fbcf866e32f4d-006526b2dd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.705 DEBUG:boto:b'dghqvd-8999false&nulltrue2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.706 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-11T14:36:13.706 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-11T14:36:13.707 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-11T14:36:13.707 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:13.708 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:36:13.708 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.708 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.709 DEBUG:boto:path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.709 DEBUG:boto:auth_path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.710 DEBUG:boto:Method: GET 2023-10-11T14:36:13.710 DEBUG:boto:Path: /dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.711 DEBUG:boto:Data: 2023-10-11T14:36:13.711 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.712 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.712 DEBUG:boto:Params: {} 2023-10-11T14:36:13.713 DEBUG:boto:Token: None 2023-10-11T14:36:13.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/?versions 2023-10-11T14:36:13.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c= 2023-10-11T14:36:13.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c=', 'Content-Length': '0'} 2023-10-11T14:36:13.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cb77f1c9317a906-006526b2dd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.717 DEBUG:boto:b'dghqvd-8999false&nulltrue2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.718 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-10-11T14:36:13.718 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-10-11T14:36:13.719 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-10-11T14:36:13.719 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:13.719 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.720 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.720 DEBUG:boto:path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.721 DEBUG:boto:auth_path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.721 DEBUG:boto:Method: GET 2023-10-11T14:36:13.721 DEBUG:boto:Path: /dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.722 DEBUG:boto:Data: 2023-10-11T14:36:13.722 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.723 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.723 DEBUG:boto:Params: {} 2023-10-11T14:36:13.724 DEBUG:boto:Token: None 2023-10-11T14:36:13.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/?versions 2023-10-11T14:36:13.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c= 2023-10-11T14:36:13.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c=', 'Content-Length': '0'} 2023-10-11T14:36:13.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087b92a07fc0a9512-006526b2dd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.728 DEBUG:boto:b'dghqvd-8999false&nulltrue2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.729 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.729 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.730 DEBUG:boto:path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.730 DEBUG:boto:auth_path=/dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.730 DEBUG:boto:Method: GET 2023-10-11T14:36:13.731 DEBUG:boto:Path: /dghqvd-8/?versions&max-keys=999 2023-10-11T14:36:13.731 DEBUG:boto:Data: 2023-10-11T14:36:13.732 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.732 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.733 DEBUG:boto:Params: {} 2023-10-11T14:36:13.733 DEBUG:boto:Token: None 2023-10-11T14:36:13.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/?versions 2023-10-11T14:36:13.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c= 2023-10-11T14:36:13.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ3cJN7GmVQokHKHB58uALroe2c=', 'Content-Length': '0'} 2023-10-11T14:36:13.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004284d7b7b4ca8c5e-006526b2dd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.738 DEBUG:boto:b'dghqvd-8999false&nulltrue2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:36:13.739 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.740 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.740 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.740 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.741 DEBUG:boto:Method: GET 2023-10-11T14:36:13.741 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.742 DEBUG:boto:Data: 2023-10-11T14:36:13.742 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.743 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.743 DEBUG:boto:Params: {} 2023-10-11T14:36:13.744 DEBUG:boto:Token: None 2023-10-11T14:36:13.744 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro= 2023-10-11T14:36:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro=', 'Content-Length': '0'} 2023-10-11T14:36:13.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008efb5a29f3cdec58-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.748 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.749 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.749 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.749 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.750 DEBUG:boto:Method: GET 2023-10-11T14:36:13.750 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.751 DEBUG:boto:Data: 2023-10-11T14:36:13.751 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.752 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.752 DEBUG:boto:Params: {} 2023-10-11T14:36:13.753 DEBUG:boto:Token: None 2023-10-11T14:36:13.753 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro= 2023-10-11T14:36:13.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro=', 'Content-Length': '0'} 2023-10-11T14:36:13.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed410695db2e6edb-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.757 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.758 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.758 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.758 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.759 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.759 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.760 DEBUG:boto:Data: 2023-10-11T14:36:13.760 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.761 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.761 DEBUG:boto:Params: {} 2023-10-11T14:36:13.762 DEBUG:boto:Token: None 2023-10-11T14:36:13.762 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DKb+bN00HZtUjw+3WSJ1sIXukXY= 2023-10-11T14:36:13.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DKb+bN00HZtUjw+3WSJ1sIXukXY=', 'Content-Length': '0'} 2023-10-11T14:36:13.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d6460e85c9b5c81-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.766 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.766 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.767 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.767 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.767 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.768 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.768 DEBUG:boto:Data: 2023-10-11T14:36:13.769 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.770 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.770 DEBUG:boto:Params: {} 2023-10-11T14:36:13.771 DEBUG:boto:Token: None 2023-10-11T14:36:13.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DKb+bN00HZtUjw+3WSJ1sIXukXY= 2023-10-11T14:36:13.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DKb+bN00HZtUjw+3WSJ1sIXukXY=', 'Content-Length': '0'} 2023-10-11T14:36:13.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7c97cfc1b35ce23-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:36:13.775 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.776 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.776 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.777 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.777 DEBUG:boto:Method: GET 2023-10-11T14:36:13.777 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.778 DEBUG:boto:Data: 2023-10-11T14:36:13.778 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.779 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.780 DEBUG:boto:Params: {} 2023-10-11T14:36:13.780 DEBUG:boto:Token: None 2023-10-11T14:36:13.781 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro= 2023-10-11T14:36:13.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro=', 'Content-Length': '0'} 2023-10-11T14:36:13.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000434b9882a2e628d8-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.785 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.785 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.786 DEBUG:boto:path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.786 DEBUG:boto:auth_path=/dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.786 DEBUG:boto:Method: GET 2023-10-11T14:36:13.787 DEBUG:boto:Path: /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.787 DEBUG:boto:Data: 2023-10-11T14:36:13.788 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.789 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.789 DEBUG:boto:Params: {} 2023-10-11T14:36:13.789 DEBUG:boto:Token: None 2023-10-11T14:36:13.790 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26?versionId=null 2023-10-11T14:36:13.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro= 2023-10-11T14:36:13.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YBfLtBOuDC16arWmQYA/NxAfZro=', 'Content-Length': '0'} 2023-10-11T14:36:13.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087fb183f4bff963b-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.794 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.794 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.795 DEBUG:boto:path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.795 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.796 DEBUG:boto:Method: GET 2023-10-11T14:36:13.796 DEBUG:boto:Path: /dghqvd-8/?prefix=%26 2023-10-11T14:36:13.796 DEBUG:boto:Data: 2023-10-11T14:36:13.797 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.798 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.798 DEBUG:boto:Params: {} 2023-10-11T14:36:13.799 DEBUG:boto:Token: None 2023-10-11T14:36:13.799 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw= 2023-10-11T14:36:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw=', 'Content-Length': '0'} 2023-10-11T14:36:13.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087434cc689c32ef-006526b2dd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.803 DEBUG:boto:b'dghqvd-8&1000false&2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.804 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.804 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.805 DEBUG:boto:path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.805 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.805 DEBUG:boto:Method: GET 2023-10-11T14:36:13.806 DEBUG:boto:Path: /dghqvd-8/?prefix=%26 2023-10-11T14:36:13.806 DEBUG:boto:Data: 2023-10-11T14:36:13.807 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.807 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.808 DEBUG:boto:Params: {} 2023-10-11T14:36:13.808 DEBUG:boto:Token: None 2023-10-11T14:36:13.809 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw= 2023-10-11T14:36:13.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw=', 'Content-Length': '0'} 2023-10-11T14:36:13.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7eeb5b131fcd19b-006526b2dd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.813 DEBUG:boto:b'dghqvd-8&1000false&2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.813 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.814 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.814 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.815 DEBUG:boto:Path: /dghqvd-8/%26 2023-10-11T14:36:13.815 DEBUG:boto:Data: 2023-10-11T14:36:13.815 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.816 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.817 DEBUG:boto:Params: {} 2023-10-11T14:36:13.817 DEBUG:boto:Token: None 2023-10-11T14:36:13.818 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26 2023-10-11T14:36:13.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91EGKgik/AscuOolIeXMpSVk1B0= 2023-10-11T14:36:13.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91EGKgik/AscuOolIeXMpSVk1B0=', 'Content-Length': '0'} 2023-10-11T14:36:13.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003041917000610f2b-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.821 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.821 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.822 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.822 DEBUG:boto:Path: /dghqvd-8/%26 2023-10-11T14:36:13.823 DEBUG:boto:Data: 2023-10-11T14:36:13.823 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.824 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.824 DEBUG:boto:Params: {} 2023-10-11T14:36:13.825 DEBUG:boto:Token: None 2023-10-11T14:36:13.825 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26 2023-10-11T14:36:13.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91EGKgik/AscuOolIeXMpSVk1B0= 2023-10-11T14:36:13.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91EGKgik/AscuOolIeXMpSVk1B0=', 'Content-Length': '0'} 2023-10-11T14:36:13.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074ef515a43d093d9-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-10-11T14:36:13.829 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.830 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.830 DEBUG:boto:Method: GET 2023-10-11T14:36:13.830 DEBUG:boto:Path: /dghqvd-8/%26 2023-10-11T14:36:13.831 DEBUG:boto:Data: 2023-10-11T14:36:13.831 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.832 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.833 DEBUG:boto:Params: {} 2023-10-11T14:36:13.833 DEBUG:boto:Token: None 2023-10-11T14:36:13.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26 2023-10-11T14:36:13.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r9RJcyLHQVIWuH+xFUvvchniqzU= 2023-10-11T14:36:13.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r9RJcyLHQVIWuH+xFUvvchniqzU=', 'Content-Length': '0'} 2023-10-11T14:36:13.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070eca7b73e3d1d9e-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.878 DEBUG:boto:path=/dghqvd-8/%26 2023-10-11T14:36:13.879 DEBUG:boto:auth_path=/dghqvd-8/%26 2023-10-11T14:36:13.879 DEBUG:boto:Method: GET 2023-10-11T14:36:13.880 DEBUG:boto:Path: /dghqvd-8/%26 2023-10-11T14:36:13.880 DEBUG:boto:Data: 2023-10-11T14:36:13.880 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.881 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.882 DEBUG:boto:Params: {} 2023-10-11T14:36:13.882 DEBUG:boto:Token: None 2023-10-11T14:36:13.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%26 2023-10-11T14:36:13.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r9RJcyLHQVIWuH+xFUvvchniqzU= 2023-10-11T14:36:13.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r9RJcyLHQVIWuH+xFUvvchniqzU=', 'Content-Length': '0'} 2023-10-11T14:36:13.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8411b2f13bdb2c5-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.887 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.887 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.888 DEBUG:boto:path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.888 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.888 DEBUG:boto:Method: GET 2023-10-11T14:36:13.889 DEBUG:boto:Path: /dghqvd-8/?prefix=%26 2023-10-11T14:36:13.889 DEBUG:boto:Data: 2023-10-11T14:36:13.890 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.891 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.891 DEBUG:boto:Params: {} 2023-10-11T14:36:13.892 DEBUG:boto:Token: None 2023-10-11T14:36:13.892 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw= 2023-10-11T14:36:13.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw=', 'Content-Length': '0'} 2023-10-11T14:36:13.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48cccc41d7f3939-006526b2dd-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.896 DEBUG:boto:b'dghqvd-8&1000false&2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.897 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:13.898 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:13.898 DEBUG:boto:path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.898 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%26 2023-10-11T14:36:13.899 DEBUG:boto:Method: GET 2023-10-11T14:36:13.899 DEBUG:boto:Path: /dghqvd-8/?prefix=%26 2023-10-11T14:36:13.900 DEBUG:boto:Data: 2023-10-11T14:36:13.900 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.901 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.902 DEBUG:boto:Params: {} 2023-10-11T14:36:13.902 DEBUG:boto:Token: None 2023-10-11T14:36:13.903 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/ 2023-10-11T14:36:13.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw= 2023-10-11T14:36:13.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kDbdqis7haeTlQ7iirbHeUk18jw=', 'Content-Length': '0'} 2023-10-11T14:36:13.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6e8a6692c7169d-006526b2dd-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.907 DEBUG:boto:b'dghqvd-8&1000false&2023-10-11T14:33:12.253Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:13.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:36:13.908 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.908 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.909 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.909 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.909 DEBUG:boto:Method: GET 2023-10-11T14:36:13.910 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.910 DEBUG:boto:Data: 2023-10-11T14:36:13.911 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.911 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.912 DEBUG:boto:Params: {} 2023-10-11T14:36:13.912 DEBUG:boto:Token: None 2023-10-11T14:36:13.913 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU= 2023-10-11T14:36:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU=', 'Content-Length': '0'} 2023-10-11T14:36:13.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d50998e17e162f2-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.917 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.917 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.917 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.918 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.918 DEBUG:boto:Method: GET 2023-10-11T14:36:13.919 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.919 DEBUG:boto:Data: 2023-10-11T14:36:13.920 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.921 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.921 DEBUG:boto:Params: {} 2023-10-11T14:36:13.921 DEBUG:boto:Token: None 2023-10-11T14:36:13.922 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU= 2023-10-11T14:36:13.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU=', 'Content-Length': '0'} 2023-10-11T14:36:13.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047277036bfd3a2d5-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.926 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.926 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.926 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.927 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.927 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.928 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.928 DEBUG:boto:Data: 2023-10-11T14:36:13.929 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.929 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.930 DEBUG:boto:Params: {} 2023-10-11T14:36:13.930 DEBUG:boto:Token: None 2023-10-11T14:36:13.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YlJH3iSRzD1KdR9a+zCOF9GHaQA= 2023-10-11T14:36:13.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YlJH3iSRzD1KdR9a+zCOF9GHaQA=', 'Content-Length': '0'} 2023-10-11T14:36:13.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000920e1b437009cd5e-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.934 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.935 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.935 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.935 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.936 DEBUG:boto:Method: HEAD 2023-10-11T14:36:13.936 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.937 DEBUG:boto:Data: 2023-10-11T14:36:13.937 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.938 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.938 DEBUG:boto:Params: {} 2023-10-11T14:36:13.939 DEBUG:boto:Token: None 2023-10-11T14:36:13.939 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YlJH3iSRzD1KdR9a+zCOF9GHaQA= 2023-10-11T14:36:13.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YlJH3iSRzD1KdR9a+zCOF9GHaQA=', 'Content-Length': '0'} 2023-10-11T14:36:13.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cdc7812e92d3037-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:36:13.943 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.944 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.944 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.945 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.945 DEBUG:boto:Method: GET 2023-10-11T14:36:13.946 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.946 DEBUG:boto:Data: 2023-10-11T14:36:13.947 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:13.947 DEBUG:boto:Port: 8000 2023-10-11T14:36:13.948 DEBUG:boto:Params: {} 2023-10-11T14:36:13.948 DEBUG:boto:Token: None 2023-10-11T14:36:13.949 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU= 2023-10-11T14:36:13.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU=', 'Content-Length': '0'} 2023-10-11T14:36:13.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007ea14d392528967-006526b2dd-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:13.952 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:13.953 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:13.953 DEBUG:boto:path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.954 DEBUG:boto:auth_path=/dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.954 DEBUG:boto:Method: GET 2023-10-11T14:36:13.955 DEBUG:boto:Path: /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.955 DEBUG:boto:Data: 2023-10-11T14:36:13.955 DEBUG:boto:Headers: {} 2023-10-11T14:36:13.956 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:13.956 DEBUG:boto:Port: 8001 2023-10-11T14:36:13.957 DEBUG:boto:Params: {} 2023-10-11T14:36:13.957 DEBUG:boto:Token: None 2023-10-11T14:36:13.958 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:13 GMT /dghqvd-8/%3A?versionId=null 2023-10-11T14:36:13.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU= 2023-10-11T14:36:13.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mfAiljhlmaPGfYm3m8C0HOD4oLU=', 'Content-Length': '0'} 2023-10-11T14:36:13.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab35c368008b70de-006526b2dd-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.002 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.003 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.003 DEBUG:boto:path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.004 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.004 DEBUG:boto:Method: GET 2023-10-11T14:36:14.004 DEBUG:boto:Path: /dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.005 DEBUG:boto:Data: 2023-10-11T14:36:14.005 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.006 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.006 DEBUG:boto:Params: {} 2023-10-11T14:36:14.007 DEBUG:boto:Token: None 2023-10-11T14:36:14.007 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bda3065cbd643d97-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.011 DEBUG:boto:b'dghqvd-8:1000false:2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.012 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.012 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.013 DEBUG:boto:path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.013 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.014 DEBUG:boto:Method: GET 2023-10-11T14:36:14.014 DEBUG:boto:Path: /dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.015 DEBUG:boto:Data: 2023-10-11T14:36:14.015 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.016 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.016 DEBUG:boto:Params: {} 2023-10-11T14:36:14.017 DEBUG:boto:Token: None 2023-10-11T14:36:14.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f5f649d6248bc2-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.021 DEBUG:boto:b'dghqvd-8:1000false:2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.022 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:14.023 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:14.023 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.023 DEBUG:boto:Path: /dghqvd-8/%3A 2023-10-11T14:36:14.024 DEBUG:boto:Data: 2023-10-11T14:36:14.024 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.025 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.025 DEBUG:boto:Params: {} 2023-10-11T14:36:14.026 DEBUG:boto:Token: None 2023-10-11T14:36:14.026 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/%3A 2023-10-11T14:36:14.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJeFxZ5TNXGlaa3MGBmGWnKE7vs= 2023-10-11T14:36:14.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJeFxZ5TNXGlaa3MGBmGWnKE7vs=', 'Content-Length': '0'} 2023-10-11T14:36:14.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e2088825b43a45e-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.030 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:14.030 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:14.031 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.031 DEBUG:boto:Path: /dghqvd-8/%3A 2023-10-11T14:36:14.032 DEBUG:boto:Data: 2023-10-11T14:36:14.032 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.033 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.033 DEBUG:boto:Params: {} 2023-10-11T14:36:14.034 DEBUG:boto:Token: None 2023-10-11T14:36:14.034 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/%3A 2023-10-11T14:36:14.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJeFxZ5TNXGlaa3MGBmGWnKE7vs= 2023-10-11T14:36:14.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJeFxZ5TNXGlaa3MGBmGWnKE7vs=', 'Content-Length': '0'} 2023-10-11T14:36:14.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca2f0228ffb4faba-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-10-11T14:36:14.038 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:14.039 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:14.039 DEBUG:boto:Method: GET 2023-10-11T14:36:14.040 DEBUG:boto:Path: /dghqvd-8/%3A 2023-10-11T14:36:14.040 DEBUG:boto:Data: 2023-10-11T14:36:14.040 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.041 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.041 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.042 DEBUG:boto:Params: {} 2023-10-11T14:36:14.042 DEBUG:boto:Token: None 2023-10-11T14:36:14.043 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/%3A 2023-10-11T14:36:14.043 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n/rJcoh9mxkwwp9MZIk1S7eKTF4= 2023-10-11T14:36:14.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/rJcoh9mxkwwp9MZIk1S7eKTF4=', 'Content-Length': '0'} 2023-10-11T14:36:14.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000eea8dbfd6568a73-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.090 DEBUG:boto:path=/dghqvd-8/%3A 2023-10-11T14:36:14.091 DEBUG:boto:auth_path=/dghqvd-8/%3A 2023-10-11T14:36:14.091 DEBUG:boto:Method: GET 2023-10-11T14:36:14.092 DEBUG:boto:Path: /dghqvd-8/%3A 2023-10-11T14:36:14.092 DEBUG:boto:Data: 2023-10-11T14:36:14.092 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.093 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.094 DEBUG:boto:Params: {} 2023-10-11T14:36:14.094 DEBUG:boto:Token: None 2023-10-11T14:36:14.095 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/%3A 2023-10-11T14:36:14.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n/rJcoh9mxkwwp9MZIk1S7eKTF4= 2023-10-11T14:36:14.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/rJcoh9mxkwwp9MZIk1S7eKTF4=', 'Content-Length': '0'} 2023-10-11T14:36:14.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c393ac4119d866e3-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.098 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.099 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.099 DEBUG:boto:path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.099 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.100 DEBUG:boto:Method: GET 2023-10-11T14:36:14.100 DEBUG:boto:Path: /dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.101 DEBUG:boto:Data: 2023-10-11T14:36:14.101 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.102 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.102 DEBUG:boto:Params: {} 2023-10-11T14:36:14.103 DEBUG:boto:Token: None 2023-10-11T14:36:14.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1455d4d1cce11de-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.107 DEBUG:boto:b'dghqvd-8:1000false:2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.108 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.109 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.109 DEBUG:boto:path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.109 DEBUG:boto:auth_path=/dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.110 DEBUG:boto:Method: GET 2023-10-11T14:36:14.110 DEBUG:boto:Path: /dghqvd-8/?prefix=%3A 2023-10-11T14:36:14.111 DEBUG:boto:Data: 2023-10-11T14:36:14.111 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.112 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.112 DEBUG:boto:Params: {} 2023-10-11T14:36:14.113 DEBUG:boto:Token: None 2023-10-11T14:36:14.113 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3c0ae2e8ed6ea82-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.117 DEBUG:boto:b'dghqvd-8:1000false:2023-10-11T14:33:12.232Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:36:14.118 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.119 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.119 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.120 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.120 DEBUG:boto:Method: GET 2023-10-11T14:36:14.121 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.121 DEBUG:boto:Data: 2023-10-11T14:36:14.122 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.123 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.123 DEBUG:boto:Params: {} 2023-10-11T14:36:14.123 DEBUG:boto:Token: None 2023-10-11T14:36:14.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU= 2023-10-11T14:36:14.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU=', 'Content-Length': '0'} 2023-10-11T14:36:14.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bb4302b96bc7cba-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.128 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.128 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.129 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.129 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.130 DEBUG:boto:Method: GET 2023-10-11T14:36:14.130 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.131 DEBUG:boto:Data: 2023-10-11T14:36:14.131 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.132 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.133 DEBUG:boto:Params: {} 2023-10-11T14:36:14.133 DEBUG:boto:Token: None 2023-10-11T14:36:14.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU= 2023-10-11T14:36:14.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU=', 'Content-Length': '0'} 2023-10-11T14:36:14.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055c7998a6f45a01e-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.137 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.138 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.138 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.139 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.139 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.139 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.140 DEBUG:boto:Data: 2023-10-11T14:36:14.140 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.141 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.142 DEBUG:boto:Params: {} 2023-10-11T14:36:14.142 DEBUG:boto:Token: None 2023-10-11T14:36:14.143 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.143 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LPk1OJ50NQCqCluAfNSX6RNQ8+4= 2023-10-11T14:36:14.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LPk1OJ50NQCqCluAfNSX6RNQ8+4=', 'Content-Length': '0'} 2023-10-11T14:36:14.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004580d1c402953c8b-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.147 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.147 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.147 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.148 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.148 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.149 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.149 DEBUG:boto:Data: 2023-10-11T14:36:14.150 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.151 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.151 DEBUG:boto:Params: {} 2023-10-11T14:36:14.151 DEBUG:boto:Token: None 2023-10-11T14:36:14.152 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LPk1OJ50NQCqCluAfNSX6RNQ8+4= 2023-10-11T14:36:14.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LPk1OJ50NQCqCluAfNSX6RNQ8+4=', 'Content-Length': '0'} 2023-10-11T14:36:14.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000764a757ccba6aeb6-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:36:14.156 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.156 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.157 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.157 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.158 DEBUG:boto:Method: GET 2023-10-11T14:36:14.158 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.159 DEBUG:boto:Data: 2023-10-11T14:36:14.159 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.160 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.160 DEBUG:boto:Params: {} 2023-10-11T14:36:14.161 DEBUG:boto:Token: None 2023-10-11T14:36:14.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU= 2023-10-11T14:36:14.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU=', 'Content-Length': '0'} 2023-10-11T14:36:14.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fa8bbb23ed35db1-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.210 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.211 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.212 DEBUG:boto:path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.213 DEBUG:boto:auth_path=/dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.213 DEBUG:boto:Method: GET 2023-10-11T14:36:14.214 DEBUG:boto:Path: /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.214 DEBUG:boto:Data: 2023-10-11T14:36:14.215 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.216 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.217 DEBUG:boto:Params: {} 2023-10-11T14:36:14.217 DEBUG:boto:Token: None 2023-10-11T14:36:14.218 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj?versionId=null 2023-10-11T14:36:14.219 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU= 2023-10-11T14:36:14.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eK86A2R2BxAfzBoNFGWNVKz/oGU=', 'Content-Length': '0'} 2023-10-11T14:36:14.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef2cd7bd15793e4-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.222 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.223 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.223 DEBUG:boto:path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.224 DEBUG:boto:auth_path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.224 DEBUG:boto:Method: GET 2023-10-11T14:36:14.225 DEBUG:boto:Path: /dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.225 DEBUG:boto:Data: 2023-10-11T14:36:14.226 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.227 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.227 DEBUG:boto:Params: {} 2023-10-11T14:36:14.228 DEBUG:boto:Token: None 2023-10-11T14:36:14.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d627e605d240fbe4-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.233 DEBUG:boto:b'dghqvd-8_myobj1000false_myobj2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.234 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.234 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.235 DEBUG:boto:path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.235 DEBUG:boto:auth_path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.236 DEBUG:boto:Method: GET 2023-10-11T14:36:14.236 DEBUG:boto:Path: /dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.236 DEBUG:boto:Data: 2023-10-11T14:36:14.237 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.238 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.238 DEBUG:boto:Params: {} 2023-10-11T14:36:14.239 DEBUG:boto:Token: None 2023-10-11T14:36:14.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a48dca9b9facd96d-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.243 DEBUG:boto:b'dghqvd-8_myobj1000false_myobj2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.244 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.245 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.245 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.245 DEBUG:boto:Path: /dghqvd-8/_myobj 2023-10-11T14:36:14.246 DEBUG:boto:Data: 2023-10-11T14:36:14.246 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.247 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.248 DEBUG:boto:Params: {} 2023-10-11T14:36:14.248 DEBUG:boto:Token: None 2023-10-11T14:36:14.249 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj 2023-10-11T14:36:14.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yS9icd6WK3OAvb9z/opQLa7oohA= 2023-10-11T14:36:14.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yS9icd6WK3OAvb9z/opQLa7oohA=', 'Content-Length': '0'} 2023-10-11T14:36:14.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1f34a19cf4231c5-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.252 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.253 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.253 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.254 DEBUG:boto:Path: /dghqvd-8/_myobj 2023-10-11T14:36:14.254 DEBUG:boto:Data: 2023-10-11T14:36:14.254 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.255 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.256 DEBUG:boto:Params: {} 2023-10-11T14:36:14.256 DEBUG:boto:Token: None 2023-10-11T14:36:14.257 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj 2023-10-11T14:36:14.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yS9icd6WK3OAvb9z/opQLa7oohA= 2023-10-11T14:36:14.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yS9icd6WK3OAvb9z/opQLa7oohA=', 'Content-Length': '0'} 2023-10-11T14:36:14.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f43d8902170a81e-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-10-11T14:36:14.260 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.261 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.261 DEBUG:boto:Method: GET 2023-10-11T14:36:14.262 DEBUG:boto:Path: /dghqvd-8/_myobj 2023-10-11T14:36:14.262 DEBUG:boto:Data: 2023-10-11T14:36:14.263 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.263 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.263 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.264 DEBUG:boto:Params: {} 2023-10-11T14:36:14.264 DEBUG:boto:Token: None 2023-10-11T14:36:14.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj 2023-10-11T14:36:14.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h8k657c7e+KBggf6PwGp7X0AZYo= 2023-10-11T14:36:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h8k657c7e+KBggf6PwGp7X0AZYo=', 'Content-Length': '0'} 2023-10-11T14:36:14.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3cfdc15d7b7a5fd-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.269 DEBUG:boto:path=/dghqvd-8/_myobj 2023-10-11T14:36:14.269 DEBUG:boto:auth_path=/dghqvd-8/_myobj 2023-10-11T14:36:14.270 DEBUG:boto:Method: GET 2023-10-11T14:36:14.270 DEBUG:boto:Path: /dghqvd-8/_myobj 2023-10-11T14:36:14.271 DEBUG:boto:Data: 2023-10-11T14:36:14.271 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.272 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.272 DEBUG:boto:Params: {} 2023-10-11T14:36:14.273 DEBUG:boto:Token: None 2023-10-11T14:36:14.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/_myobj 2023-10-11T14:36:14.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h8k657c7e+KBggf6PwGp7X0AZYo= 2023-10-11T14:36:14.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h8k657c7e+KBggf6PwGp7X0AZYo=', 'Content-Length': '0'} 2023-10-11T14:36:14.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecd476a82e26099f-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.277 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.277 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.278 DEBUG:boto:path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.278 DEBUG:boto:auth_path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.279 DEBUG:boto:Method: GET 2023-10-11T14:36:14.279 DEBUG:boto:Path: /dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.280 DEBUG:boto:Data: 2023-10-11T14:36:14.280 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.281 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.282 DEBUG:boto:Params: {} 2023-10-11T14:36:14.282 DEBUG:boto:Token: None 2023-10-11T14:36:14.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b068a3a98a01cda2-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.288 DEBUG:boto:b'dghqvd-8_myobj1000false_myobj2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.288 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.289 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.289 DEBUG:boto:path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.290 DEBUG:boto:auth_path=/dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.290 DEBUG:boto:Method: GET 2023-10-11T14:36:14.291 DEBUG:boto:Path: /dghqvd-8/?prefix=_myobj 2023-10-11T14:36:14.291 DEBUG:boto:Data: 2023-10-11T14:36:14.292 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.292 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.293 DEBUG:boto:Params: {} 2023-10-11T14:36:14.293 DEBUG:boto:Token: None 2023-10-11T14:36:14.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.295 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed4a2e538d2e81d-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.299 DEBUG:boto:b'dghqvd-8_myobj1000false_myobj2023-10-11T14:33:12.177Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:14.300 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.301 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.303 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.303 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.304 DEBUG:boto:Method: GET 2023-10-11T14:36:14.304 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.305 DEBUG:boto:Data: 2023-10-11T14:36:14.305 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.306 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.307 DEBUG:boto:Params: {} 2023-10-11T14:36:14.308 DEBUG:boto:Token: None 2023-10-11T14:36:14.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw= 2023-10-11T14:36:14.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw=', 'Content-Length': '0'} 2023-10-11T14:36:14.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e89c3edc31588053-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.358 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.359 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.359 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.360 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.360 DEBUG:boto:Method: GET 2023-10-11T14:36:14.361 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.361 DEBUG:boto:Data: 2023-10-11T14:36:14.361 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.362 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.363 DEBUG:boto:Params: {} 2023-10-11T14:36:14.363 DEBUG:boto:Token: None 2023-10-11T14:36:14.364 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw= 2023-10-11T14:36:14.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw=', 'Content-Length': '0'} 2023-10-11T14:36:14.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f024a6c82bf59e8-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.368 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.368 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.369 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.369 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.370 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.370 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.371 DEBUG:boto:Data: 2023-10-11T14:36:14.371 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.372 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.372 DEBUG:boto:Params: {} 2023-10-11T14:36:14.373 DEBUG:boto:Token: None 2023-10-11T14:36:14.374 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RLPxrGIWCVZMdZ0U7ErvrAfI+FA= 2023-10-11T14:36:14.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RLPxrGIWCVZMdZ0U7ErvrAfI+FA=', 'Content-Length': '0'} 2023-10-11T14:36:14.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006474c7fa4c186b8c-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.377 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.378 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.378 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.379 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.379 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.380 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.380 DEBUG:boto:Data: 2023-10-11T14:36:14.380 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.381 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.382 DEBUG:boto:Params: {} 2023-10-11T14:36:14.382 DEBUG:boto:Token: None 2023-10-11T14:36:14.383 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RLPxrGIWCVZMdZ0U7ErvrAfI+FA= 2023-10-11T14:36:14.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RLPxrGIWCVZMdZ0U7ErvrAfI+FA=', 'Content-Length': '0'} 2023-10-11T14:36:14.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af4522668a992619-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:14.387 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.388 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.388 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.388 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.389 DEBUG:boto:Method: GET 2023-10-11T14:36:14.389 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.390 DEBUG:boto:Data: 2023-10-11T14:36:14.390 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.391 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.391 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.392 DEBUG:boto:Params: {} 2023-10-11T14:36:14.392 DEBUG:boto:Token: None 2023-10-11T14:36:14.393 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.393 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw= 2023-10-11T14:36:14.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw=', 'Content-Length': '0'} 2023-10-11T14:36:14.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000141cdbfb94e99f7a-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.397 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.397 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.397 DEBUG:boto:path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.398 DEBUG:boto:auth_path=/dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.398 DEBUG:boto:Method: GET 2023-10-11T14:36:14.399 DEBUG:boto:Path: /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.399 DEBUG:boto:Data: 2023-10-11T14:36:14.400 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.400 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.401 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.401 DEBUG:boto:Params: {} 2023-10-11T14:36:14.402 DEBUG:boto:Token: None 2023-10-11T14:36:14.402 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj?versionId=null 2023-10-11T14:36:14.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw= 2023-10-11T14:36:14.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9XFoM1BiBNn/wqK4RuGP4bttMiw=', 'Content-Length': '0'} 2023-10-11T14:36:14.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efdf621225c55762-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.450 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.451 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.451 DEBUG:boto:path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.452 DEBUG:boto:auth_path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.452 DEBUG:boto:Method: GET 2023-10-11T14:36:14.453 DEBUG:boto:Path: /dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.453 DEBUG:boto:Data: 2023-10-11T14:36:14.454 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.455 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.455 DEBUG:boto:Params: {} 2023-10-11T14:36:14.456 DEBUG:boto:Token: None 2023-10-11T14:36:14.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009773b4627cc2f2dd-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.461 DEBUG:boto:b'dghqvd-8myobj1000falsemyobj2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.462 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.462 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.463 DEBUG:boto:path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.463 DEBUG:boto:auth_path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.464 DEBUG:boto:Method: GET 2023-10-11T14:36:14.464 DEBUG:boto:Path: /dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.464 DEBUG:boto:Data: 2023-10-11T14:36:14.465 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.466 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.466 DEBUG:boto:Params: {} 2023-10-11T14:36:14.467 DEBUG:boto:Token: None 2023-10-11T14:36:14.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032256b92a4ddc694-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.471 DEBUG:boto:b'dghqvd-8myobj1000falsemyobj2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.472 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.472 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.473 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.473 DEBUG:boto:Path: /dghqvd-8/myobj 2023-10-11T14:36:14.474 DEBUG:boto:Data: 2023-10-11T14:36:14.474 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.475 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.476 DEBUG:boto:Params: {} 2023-10-11T14:36:14.476 DEBUG:boto:Token: None 2023-10-11T14:36:14.476 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj 2023-10-11T14:36:14.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wBCxlBd6y9Ct+FZ2X3u2VfRigZU= 2023-10-11T14:36:14.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wBCxlBd6y9Ct+FZ2X3u2VfRigZU=', 'Content-Length': '0'} 2023-10-11T14:36:14.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4d89e037f7d3462-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.480 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.481 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.481 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.482 DEBUG:boto:Path: /dghqvd-8/myobj 2023-10-11T14:36:14.482 DEBUG:boto:Data: 2023-10-11T14:36:14.482 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.483 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.484 DEBUG:boto:Params: {} 2023-10-11T14:36:14.484 DEBUG:boto:Token: None 2023-10-11T14:36:14.485 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj 2023-10-11T14:36:14.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wBCxlBd6y9Ct+FZ2X3u2VfRigZU= 2023-10-11T14:36:14.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wBCxlBd6y9Ct+FZ2X3u2VfRigZU=', 'Content-Length': '0'} 2023-10-11T14:36:14.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc3e70d2ddcc6add-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:14.488 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.489 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.489 DEBUG:boto:Method: GET 2023-10-11T14:36:14.490 DEBUG:boto:Path: /dghqvd-8/myobj 2023-10-11T14:36:14.491 DEBUG:boto:Data: 2023-10-11T14:36:14.491 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.492 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.492 DEBUG:boto:Params: {} 2023-10-11T14:36:14.493 DEBUG:boto:Token: None 2023-10-11T14:36:14.493 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj 2023-10-11T14:36:14.494 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bez3mP9hM3eNTXij/5Kw81EBtMI= 2023-10-11T14:36:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bez3mP9hM3eNTXij/5Kw81EBtMI=', 'Content-Length': '0'} 2023-10-11T14:36:14.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce7f9f34ff5f889f-006526b2de-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.497 DEBUG:boto:path=/dghqvd-8/myobj 2023-10-11T14:36:14.498 DEBUG:boto:auth_path=/dghqvd-8/myobj 2023-10-11T14:36:14.498 DEBUG:boto:Method: GET 2023-10-11T14:36:14.499 DEBUG:boto:Path: /dghqvd-8/myobj 2023-10-11T14:36:14.499 DEBUG:boto:Data: 2023-10-11T14:36:14.500 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.501 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.501 DEBUG:boto:Params: {} 2023-10-11T14:36:14.501 DEBUG:boto:Token: None 2023-10-11T14:36:14.502 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/myobj 2023-10-11T14:36:14.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bez3mP9hM3eNTXij/5Kw81EBtMI= 2023-10-11T14:36:14.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bez3mP9hM3eNTXij/5Kw81EBtMI=', 'Content-Length': '0'} 2023-10-11T14:36:14.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:33:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c57bd6a3f4a55276-006526b2de-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.505 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.506 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.506 DEBUG:boto:path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.507 DEBUG:boto:auth_path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.507 DEBUG:boto:Method: GET 2023-10-11T14:36:14.508 DEBUG:boto:Path: /dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.508 DEBUG:boto:Data: 2023-10-11T14:36:14.509 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.510 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.510 DEBUG:boto:Params: {} 2023-10-11T14:36:14.510 DEBUG:boto:Token: None 2023-10-11T14:36:14.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035fb6a9dc23c6657-006526b2de-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.515 DEBUG:boto:b'dghqvd-8myobj1000falsemyobj2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.516 DEBUG:boto:path=/dghqvd-8/ 2023-10-11T14:36:14.516 DEBUG:boto:auth_path=/dghqvd-8/ 2023-10-11T14:36:14.516 DEBUG:boto:path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.517 DEBUG:boto:auth_path=/dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.517 DEBUG:boto:Method: GET 2023-10-11T14:36:14.518 DEBUG:boto:Path: /dghqvd-8/?prefix=myobj 2023-10-11T14:36:14.518 DEBUG:boto:Data: 2023-10-11T14:36:14.519 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.520 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.520 DEBUG:boto:Params: {} 2023-10-11T14:36:14.521 DEBUG:boto:Token: None 2023-10-11T14:36:14.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-8/ 2023-10-11T14:36:14.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg= 2023-10-11T14:36:14.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xgZh9pG6vNAnhfJPKIWUxSTfFg=', 'Content-Length': '0'} 2023-10-11T14:36:14.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c848925107f3ad6-006526b2de-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.526 DEBUG:boto:b'dghqvd-8myobj1000falsemyobj2023-10-11T14:33:12.156Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:14.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-8 zones={a1, a2} 2023-10-11T14:36:14.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-10-11T14:36:14.528 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-9 2023-10-11T14:36:14.529 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:14.529 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:14.529 DEBUG:boto:Method: PUT 2023-10-11T14:36:14.530 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:14.530 DEBUG:boto:Data: 2023-10-11T14:36:14.531 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.532 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.532 DEBUG:boto:Params: {} 2023-10-11T14:36:14.532 DEBUG:boto:Token: None 2023-10-11T14:36:14.533 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-9/ 2023-10-11T14:36:14.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1IO+nIgi2r6q4q6odzjTMgT4RLY= 2023-10-11T14:36:14.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1IO+nIgi2r6q4q6odzjTMgT4RLY=', 'Content-Length': '0'} 2023-10-11T14:36:14.548 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007cc4d0e7c7cce349-006526b2de-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.549 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-10 2023-10-11T14:36:14.549 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:14.550 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:14.550 DEBUG:boto:Method: PUT 2023-10-11T14:36:14.550 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:36:14.551 DEBUG:boto:Data: 2023-10-11T14:36:14.551 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.552 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.552 DEBUG:boto:Params: {} 2023-10-11T14:36:14.553 DEBUG:boto:Token: None 2023-10-11T14:36:14.553 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-10/ 2023-10-11T14:36:14.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:drDlN7tveGLM07eiprhmNG3A2rk= 2023-10-11T14:36:14.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:drDlN7tveGLM07eiprhmNG3A2rk=', 'Content-Length': '0'} 2023-10-11T14:36:14.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000365d5ca2fd64eb2d-006526b2de-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.579 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:14.580 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:14.580 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.581 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:14.581 DEBUG:boto:Data: 2023-10-11T14:36:14.581 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.582 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.583 DEBUG:boto:Params: {} 2023-10-11T14:36:14.583 DEBUG:boto:Token: None 2023-10-11T14:36:14.584 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-9/ 2023-10-11T14:36:14.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MI9X2efykkf1HhkL5S/vfgpxOVE= 2023-10-11T14:36:14.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MI9X2efykkf1HhkL5S/vfgpxOVE=', 'Content-Length': '0'} 2023-10-11T14:36:14.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f09483e58a7911c-006526b2de-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.589 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:14.589 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:14.589 DEBUG:boto:Method: PUT 2023-10-11T14:36:14.590 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:14.590 DEBUG:boto:Data: 2023-10-11T14:36:14.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:36:14.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:14.591 DEBUG:boto:Port: 8000 2023-10-11T14:36:14.592 DEBUG:boto:Params: {} 2023-10-11T14:36:14.592 DEBUG:boto:Token: None 2023-10-11T14:36:14.593 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-9/myobj 2023-10-11T14:36:14.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L7Y+Hvkh8Cy90n1tXbWXOCgtnFY= 2023-10-11T14:36:14.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L7Y+Hvkh8Cy90n1tXbWXOCgtnFY='} 2023-10-11T14:36:14.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6e3929e91d2ea39-006526b2de-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.639 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:14.640 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:14.640 DEBUG:boto:Method: HEAD 2023-10-11T14:36:14.640 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:36:14.641 DEBUG:boto:Data: 2023-10-11T14:36:14.641 DEBUG:boto:Headers: {} 2023-10-11T14:36:14.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.642 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.642 DEBUG:boto:Params: {} 2023-10-11T14:36:14.643 DEBUG:boto:Token: None 2023-10-11T14:36:14.643 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-10/ 2023-10-11T14:36:14.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Go8LVXU9aO2GzJjzb73usywfWwk= 2023-10-11T14:36:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Go8LVXU9aO2GzJjzb73usywfWwk=', 'Content-Length': '0'} 2023-10-11T14:36:14.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dfe14368fcb11d9-006526b2de-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.648 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:14.648 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:14.648 DEBUG:boto:Method: PUT 2023-10-11T14:36:14.649 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:14.649 DEBUG:boto:Data: 2023-10-11T14:36:14.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:36:14.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:14.650 DEBUG:boto:Port: 8001 2023-10-11T14:36:14.651 DEBUG:boto:Params: {} 2023-10-11T14:36:14.651 DEBUG:boto:Token: None 2023-10-11T14:36:14.652 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:36:14 GMT /dghqvd-10/myobj 2023-10-11T14:36:14.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CDziAe35+IMae2mD2p9t3ruMpBA= 2023-10-11T14:36:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:36:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CDziAe35+IMae2mD2p9t3ruMpBA='} 2023-10-11T14:36:14.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091232600f09ddfb4-006526b2de-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:14.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:36:14.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:36:14.696 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:36:14.753 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:14.754 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:14.785 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:36:14.796 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:36:14.797 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.797 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:36:14.797 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:36:14.797 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.798 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.798 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:36:14.798 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:36:14.798 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.799 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.799 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:36:14.799 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:36:14.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.799 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.800 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.567478_40.1", 2023-10-11T14:36:14.800 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.567478Z" 2023-10-11T14:36:14.800 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.800 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:36:14.801 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697034974.567478_40.1'} 2023-10-11T14:36:14.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:36:14.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:36:14.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:36:14.803 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:36:14.853 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:14.853 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:14.880 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:14.878+0000 7fa74bbd2780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:36:14.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:14.879+0000 7fa74bbd2780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:36:14.893 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:36:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:36:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:36:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:36:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:36:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:36:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:36:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:36:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:36:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:36:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:36:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:36:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:36:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:36:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:36:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:36:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:36:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:36:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:36:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:36:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:36:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:36:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:36:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:36:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:36:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:36:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:36:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:36:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:36:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:36:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:36:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:36:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.567478_40.1", 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:36:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:36:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.567478Z", 2023-10-11T14:36:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:36:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:36:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:36:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:36:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:36:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:36:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:36:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:36:14Z" 2023-10-11T14:36:14.906 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:36:14.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697034974.567478_40.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.567478Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:36:14Z" } 2023-10-11T14:36:14.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697034974.567478_40.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.567478Z', 'realm_epoch': 2}}] 2023-10-11T14:36:14.908 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697034974.567478_40.1'} 2023-10-11T14:36:14.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697034974.567478_40.1'} 2023-10-11T14:36:14.909 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:36:14.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:14.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:14.910 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:36:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:14.967 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:15.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:14.998+0000 7fcd3a31e780 1 waiting to reach incremental sync.. 2023-10-11T14:36:20.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:19.999+0000 7fcd3a31e780 1 waiting to reach incremental sync.. 2023-10-11T14:36:25.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:25.000+0000 7fcd3a31e780 1 waiting to reach incremental sync.. 2023-10-11T14:36:30.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:30.000+0000 7fcd3a31e780 1 waiting to reach incremental sync.. 2023-10-11T14:36:35.004 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:35.002+0000 7fcd3a31e780 1 bucket sync caught up with source: 2023-10-11T14:36:35.005 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-10-11T14:36:35.005 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.11.6, , , , , , , , , ] 2023-10-11T14:36:35.005 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:35.002+0000 7fcd3a31e780 0 bucket checkpoint complete 2023-10-11T14:36:35.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-9 zones={a2, a1} 2023-10-11T14:36:35.020 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.020 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.021 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.021 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:35.022 DEBUG:boto:Data: 2023-10-11T14:36:35.022 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.023 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.024 DEBUG:boto:Params: {} 2023-10-11T14:36:35.025 DEBUG:boto:Token: None 2023-10-11T14:36:35.026 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q= 2023-10-11T14:36:35.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q=', 'Content-Length': '0'} 2023-10-11T14:36:35.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa75b0c3ff7c20b1-006526b2f3-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.030 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.031 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.031 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.032 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:35.032 DEBUG:boto:Data: 2023-10-11T14:36:35.033 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.033 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.034 DEBUG:boto:Params: {} 2023-10-11T14:36:35.034 DEBUG:boto:Token: None 2023-10-11T14:36:35.035 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q= 2023-10-11T14:36:35.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q=', 'Content-Length': '0'} 2023-10-11T14:36:35.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7991a97eae04173-006526b2f3-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.039 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:36:35.039 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.040 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.040 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.041 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.041 DEBUG:boto:Method: GET 2023-10-11T14:36:35.042 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.042 DEBUG:boto:Data: 2023-10-11T14:36:35.043 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.043 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.044 DEBUG:boto:Params: {} 2023-10-11T14:36:35.044 DEBUG:boto:Token: None 2023-10-11T14:36:35.045 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/?versions 2023-10-11T14:36:35.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc= 2023-10-11T14:36:35.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc=', 'Content-Length': '0'} 2023-10-11T14:36:35.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d90576148853bc-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.049 DEBUG:boto:b'dghqvd-9999falsemyobjnulltrue2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.050 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:35.051 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:36:35.051 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.051 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.052 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.052 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.053 DEBUG:boto:Method: GET 2023-10-11T14:36:35.053 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.054 DEBUG:boto:Data: 2023-10-11T14:36:35.054 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.055 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.055 DEBUG:boto:Params: {} 2023-10-11T14:36:35.056 DEBUG:boto:Token: None 2023-10-11T14:36:35.056 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/?versions 2023-10-11T14:36:35.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc= 2023-10-11T14:36:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc=', 'Content-Length': '0'} 2023-10-11T14:36:35.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001444cf79120f3a6a-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.060 DEBUG:boto:b'dghqvd-9999falsemyobjnulltrue2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.061 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:35.061 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.062 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.063 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.063 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.063 DEBUG:boto:Method: GET 2023-10-11T14:36:35.064 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.064 DEBUG:boto:Data: 2023-10-11T14:36:35.065 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.065 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.066 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.066 DEBUG:boto:Params: {} 2023-10-11T14:36:35.067 DEBUG:boto:Token: None 2023-10-11T14:36:35.067 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/?versions 2023-10-11T14:36:35.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc= 2023-10-11T14:36:35.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc=', 'Content-Length': '0'} 2023-10-11T14:36:35.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccf6f0339c5a32f-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.071 DEBUG:boto:b'dghqvd-9999falsemyobjnulltrue2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.072 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.072 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.073 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.073 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.074 DEBUG:boto:Method: GET 2023-10-11T14:36:35.074 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:35.074 DEBUG:boto:Data: 2023-10-11T14:36:35.075 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.076 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.076 DEBUG:boto:Params: {} 2023-10-11T14:36:35.077 DEBUG:boto:Token: None 2023-10-11T14:36:35.077 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/?versions 2023-10-11T14:36:35.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc= 2023-10-11T14:36:35.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LN+MteMGYtGzEzZcf0rtfSg2YOc=', 'Content-Length': '0'} 2023-10-11T14:36:35.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f94c8509b83664be-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.081 DEBUG:boto:b'dghqvd-9999falsemyobjnulltrue2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.082 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.083 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.083 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.083 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.084 DEBUG:boto:Method: GET 2023-10-11T14:36:35.084 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.085 DEBUG:boto:Data: 2023-10-11T14:36:35.085 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.086 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.086 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.086 DEBUG:boto:Params: {} 2023-10-11T14:36:35.087 DEBUG:boto:Token: None 2023-10-11T14:36:35.087 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.088 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw= 2023-10-11T14:36:35.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw=', 'Content-Length': '0'} 2023-10-11T14:36:35.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e896a9c85483de4-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.091 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.091 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.092 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.092 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.092 DEBUG:boto:Method: GET 2023-10-11T14:36:35.093 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.093 DEBUG:boto:Data: 2023-10-11T14:36:35.094 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.094 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.095 DEBUG:boto:Params: {} 2023-10-11T14:36:35.095 DEBUG:boto:Token: None 2023-10-11T14:36:35.096 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw= 2023-10-11T14:36:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw=', 'Content-Length': '0'} 2023-10-11T14:36:35.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbd305aac6e8f247-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.099 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.100 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.100 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.100 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.101 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.101 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.102 DEBUG:boto:Data: 2023-10-11T14:36:35.102 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.103 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.103 DEBUG:boto:Params: {} 2023-10-11T14:36:35.104 DEBUG:boto:Token: None 2023-10-11T14:36:35.104 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qg0OsCf9vDBeolHKF7kae9XTynM= 2023-10-11T14:36:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qg0OsCf9vDBeolHKF7kae9XTynM=', 'Content-Length': '0'} 2023-10-11T14:36:35.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007ad3e2e5ee5ccd1-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.108 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.108 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.109 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.109 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.109 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.110 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.110 DEBUG:boto:Data: 2023-10-11T14:36:35.111 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.111 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.112 DEBUG:boto:Params: {} 2023-10-11T14:36:35.112 DEBUG:boto:Token: None 2023-10-11T14:36:35.113 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qg0OsCf9vDBeolHKF7kae9XTynM= 2023-10-11T14:36:35.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qg0OsCf9vDBeolHKF7kae9XTynM=', 'Content-Length': '0'} 2023-10-11T14:36:35.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000852f08be11141513-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.116 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.117 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.117 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.118 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.118 DEBUG:boto:Method: GET 2023-10-11T14:36:35.119 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.119 DEBUG:boto:Data: 2023-10-11T14:36:35.119 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.120 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.121 DEBUG:boto:Params: {} 2023-10-11T14:36:35.121 DEBUG:boto:Token: None 2023-10-11T14:36:35.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw= 2023-10-11T14:36:35.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw=', 'Content-Length': '0'} 2023-10-11T14:36:35.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029de9a418348c9ad-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.170 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.171 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.171 DEBUG:boto:path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.172 DEBUG:boto:auth_path=/dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.172 DEBUG:boto:Method: GET 2023-10-11T14:36:35.172 DEBUG:boto:Path: /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.173 DEBUG:boto:Data: 2023-10-11T14:36:35.173 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.174 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.175 DEBUG:boto:Params: {} 2023-10-11T14:36:35.175 DEBUG:boto:Token: None 2023-10-11T14:36:35.176 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj?versionId=null 2023-10-11T14:36:35.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw= 2023-10-11T14:36:35.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YrUmdcMp9est8y8gDH/eetoGlw=', 'Content-Length': '0'} 2023-10-11T14:36:35.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023b1d4f251600bdc-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.180 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.180 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.181 DEBUG:boto:path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.181 DEBUG:boto:auth_path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.182 DEBUG:boto:Method: GET 2023-10-11T14:36:35.182 DEBUG:boto:Path: /dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.183 DEBUG:boto:Data: 2023-10-11T14:36:35.183 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.184 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.184 DEBUG:boto:Params: {} 2023-10-11T14:36:35.185 DEBUG:boto:Token: None 2023-10-11T14:36:35.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.186 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4= 2023-10-11T14:36:35.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4=', 'Content-Length': '0'} 2023-10-11T14:36:35.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffff6902b294bc97-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.189 DEBUG:boto:b'dghqvd-9myobj1000falsemyobj2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.190 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.190 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.191 DEBUG:boto:path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.191 DEBUG:boto:auth_path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.192 DEBUG:boto:Method: GET 2023-10-11T14:36:35.192 DEBUG:boto:Path: /dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.193 DEBUG:boto:Data: 2023-10-11T14:36:35.193 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.194 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.194 DEBUG:boto:Params: {} 2023-10-11T14:36:35.195 DEBUG:boto:Token: None 2023-10-11T14:36:35.195 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4= 2023-10-11T14:36:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4=', 'Content-Length': '0'} 2023-10-11T14:36:35.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012acb01c4409d28b-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.242 DEBUG:boto:b'dghqvd-9myobj1000falsemyobj2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.243 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.244 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.244 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.244 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.245 DEBUG:boto:Data: 2023-10-11T14:36:35.245 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.246 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.247 DEBUG:boto:Params: {} 2023-10-11T14:36:35.247 DEBUG:boto:Token: None 2023-10-11T14:36:35.248 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg= 2023-10-11T14:36:35.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg=', 'Content-Length': '0'} 2023-10-11T14:36:35.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000154e0eb31a16303e-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.251 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.252 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.253 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.253 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.254 DEBUG:boto:Data: 2023-10-11T14:36:35.254 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.255 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.255 DEBUG:boto:Params: {} 2023-10-11T14:36:35.256 DEBUG:boto:Token: None 2023-10-11T14:36:35.256 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg= 2023-10-11T14:36:35.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg=', 'Content-Length': '0'} 2023-10-11T14:36:35.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e80c8ca37f9c11eb-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.261 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.261 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.261 DEBUG:boto:Method: GET 2023-10-11T14:36:35.262 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.263 DEBUG:boto:Data: 2023-10-11T14:36:35.263 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.264 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.264 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.265 DEBUG:boto:Params: {} 2023-10-11T14:36:35.265 DEBUG:boto:Token: None 2023-10-11T14:36:35.266 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CT18xLbF1mqXWU0Pdn7AY2eCQvc= 2023-10-11T14:36:35.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CT18xLbF1mqXWU0Pdn7AY2eCQvc=', 'Content-Length': '0'} 2023-10-11T14:36:35.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064c337bf05f689cf-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.272 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.272 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.273 DEBUG:boto:Method: GET 2023-10-11T14:36:35.274 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.274 DEBUG:boto:Data: 2023-10-11T14:36:35.275 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.276 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.276 DEBUG:boto:Params: {} 2023-10-11T14:36:35.276 DEBUG:boto:Token: None 2023-10-11T14:36:35.277 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CT18xLbF1mqXWU0Pdn7AY2eCQvc= 2023-10-11T14:36:35.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CT18xLbF1mqXWU0Pdn7AY2eCQvc=', 'Content-Length': '0'} 2023-10-11T14:36:35.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbb70d9e6ee5dfa5-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.283 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.284 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.284 DEBUG:boto:path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.284 DEBUG:boto:auth_path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.285 DEBUG:boto:Method: GET 2023-10-11T14:36:35.285 DEBUG:boto:Path: /dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.286 DEBUG:boto:Data: 2023-10-11T14:36:35.286 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.287 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.287 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.288 DEBUG:boto:Params: {} 2023-10-11T14:36:35.288 DEBUG:boto:Token: None 2023-10-11T14:36:35.289 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4= 2023-10-11T14:36:35.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4=', 'Content-Length': '0'} 2023-10-11T14:36:35.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb5cbce33d7a23b3-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.334 DEBUG:boto:b'dghqvd-9myobj1000falsemyobj2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.335 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.335 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.336 DEBUG:boto:path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.336 DEBUG:boto:auth_path=/dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.337 DEBUG:boto:Method: GET 2023-10-11T14:36:35.337 DEBUG:boto:Path: /dghqvd-9/?prefix=myobj 2023-10-11T14:36:35.338 DEBUG:boto:Data: 2023-10-11T14:36:35.338 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.339 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.339 DEBUG:boto:Params: {} 2023-10-11T14:36:35.340 DEBUG:boto:Token: None 2023-10-11T14:36:35.340 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4= 2023-10-11T14:36:35.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i41zuC8ew6OR4B2SOdDJmuTjPJ4=', 'Content-Length': '0'} 2023-10-11T14:36:35.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534756c49406fe51-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.344 DEBUG:boto:b'dghqvd-9myobj1000falsemyobj2023-10-11T14:36:14.594Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-9 zones={a2, a1} 2023-10-11T14:36:35.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:35.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:35.346 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:36:35.410 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:35.411 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:35.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:35.444+0000 7f694c62a780 1 bucket sync caught up with source: 2023-10-11T14:36:35.447 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:36:35.447 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:36:35.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:35.444+0000 7f694c62a780 0 bucket checkpoint complete 2023-10-11T14:36:35.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-10 zones={a1, a2} 2023-10-11T14:36:35.459 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.460 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.460 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.461 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:36:35.461 DEBUG:boto:Data: 2023-10-11T14:36:35.462 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.463 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.463 DEBUG:boto:Params: {} 2023-10-11T14:36:35.464 DEBUG:boto:Token: None 2023-10-11T14:36:35.464 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:neIoNFDSKCjTUTO9+4hXazsTk/s= 2023-10-11T14:36:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:neIoNFDSKCjTUTO9+4hXazsTk/s=', 'Content-Length': '0'} 2023-10-11T14:36:35.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098ab1259abd9c428-006526b2f3-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.468 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.468 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.469 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.469 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:36:35.470 DEBUG:boto:Data: 2023-10-11T14:36:35.470 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.471 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.471 DEBUG:boto:Params: {} 2023-10-11T14:36:35.472 DEBUG:boto:Token: None 2023-10-11T14:36:35.472 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:neIoNFDSKCjTUTO9+4hXazsTk/s= 2023-10-11T14:36:35.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:neIoNFDSKCjTUTO9+4hXazsTk/s=', 'Content-Length': '0'} 2023-10-11T14:36:35.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3e62f846adda195-006526b2f3-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.476 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:36:35.476 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.476 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.477 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.477 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.478 DEBUG:boto:Method: GET 2023-10-11T14:36:35.478 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.479 DEBUG:boto:Data: 2023-10-11T14:36:35.479 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.480 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.480 DEBUG:boto:Params: {} 2023-10-11T14:36:35.481 DEBUG:boto:Token: None 2023-10-11T14:36:35.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/?versions 2023-10-11T14:36:35.482 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4= 2023-10-11T14:36:35.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4=', 'Content-Length': '0'} 2023-10-11T14:36:35.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235394fe7b67f499-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.488 DEBUG:boto:b'dghqvd-10999falsemyobjnulltrue2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.489 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:35.489 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:36:35.489 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.490 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.490 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.490 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.491 DEBUG:boto:Method: GET 2023-10-11T14:36:35.492 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.492 DEBUG:boto:Data: 2023-10-11T14:36:35.492 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.493 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.494 DEBUG:boto:Params: {} 2023-10-11T14:36:35.494 DEBUG:boto:Token: None 2023-10-11T14:36:35.495 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/?versions 2023-10-11T14:36:35.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4= 2023-10-11T14:36:35.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4=', 'Content-Length': '0'} 2023-10-11T14:36:35.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebf20e5d11558209-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.498 DEBUG:boto:b'dghqvd-10999falsemyobjnulltrue2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.499 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-10-11T14:36:35.499 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.500 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.500 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.501 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.501 DEBUG:boto:Method: GET 2023-10-11T14:36:35.502 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.502 DEBUG:boto:Data: 2023-10-11T14:36:35.502 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.503 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.503 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.504 DEBUG:boto:Params: {} 2023-10-11T14:36:35.504 DEBUG:boto:Token: None 2023-10-11T14:36:35.505 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/?versions 2023-10-11T14:36:35.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4= 2023-10-11T14:36:35.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4=', 'Content-Length': '0'} 2023-10-11T14:36:35.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066039f9d57bb2b43-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.554 DEBUG:boto:b'dghqvd-10999falsemyobjnulltrue2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.555 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.555 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.556 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.556 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.557 DEBUG:boto:Method: GET 2023-10-11T14:36:35.557 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:36:35.558 DEBUG:boto:Data: 2023-10-11T14:36:35.558 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.559 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.559 DEBUG:boto:Params: {} 2023-10-11T14:36:35.560 DEBUG:boto:Token: None 2023-10-11T14:36:35.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/?versions 2023-10-11T14:36:35.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4= 2023-10-11T14:36:35.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBJnfVAZtBeDtooHH4dewrH7nS4=', 'Content-Length': '0'} 2023-10-11T14:36:35.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000976832c4161b5f80-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.564 DEBUG:boto:b'dghqvd-10999falsemyobjnulltrue2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.565 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.565 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.566 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.566 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.567 DEBUG:boto:Method: GET 2023-10-11T14:36:35.567 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.568 DEBUG:boto:Data: 2023-10-11T14:36:35.568 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.568 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.569 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.569 DEBUG:boto:Params: {} 2023-10-11T14:36:35.570 DEBUG:boto:Token: None 2023-10-11T14:36:35.570 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY= 2023-10-11T14:36:35.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY=', 'Content-Length': '0'} 2023-10-11T14:36:35.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053311c67adbc8858-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.574 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.574 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.575 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.575 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.575 DEBUG:boto:Method: GET 2023-10-11T14:36:35.576 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.576 DEBUG:boto:Data: 2023-10-11T14:36:35.577 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.578 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.578 DEBUG:boto:Params: {} 2023-10-11T14:36:35.579 DEBUG:boto:Token: None 2023-10-11T14:36:35.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY= 2023-10-11T14:36:35.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY=', 'Content-Length': '0'} 2023-10-11T14:36:35.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007aac2a650f77b644-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.583 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.583 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.584 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.584 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.585 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.585 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.585 DEBUG:boto:Data: 2023-10-11T14:36:35.586 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.587 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.587 DEBUG:boto:Params: {} 2023-10-11T14:36:35.587 DEBUG:boto:Token: None 2023-10-11T14:36:35.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WzNbvoGI1Oi4vzfaAvUqJRJa+mk= 2023-10-11T14:36:35.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WzNbvoGI1Oi4vzfaAvUqJRJa+mk=', 'Content-Length': '0'} 2023-10-11T14:36:35.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d46ef1052d6aa4d5-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.591 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.591 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.592 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.592 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.593 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.593 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.593 DEBUG:boto:Data: 2023-10-11T14:36:35.594 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.595 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.595 DEBUG:boto:Params: {} 2023-10-11T14:36:35.595 DEBUG:boto:Token: None 2023-10-11T14:36:35.596 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WzNbvoGI1Oi4vzfaAvUqJRJa+mk= 2023-10-11T14:36:35.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WzNbvoGI1Oi4vzfaAvUqJRJa+mk=', 'Content-Length': '0'} 2023-10-11T14:36:35.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ddc3f57d615be4f-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.599 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.600 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.600 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.601 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.601 DEBUG:boto:Method: GET 2023-10-11T14:36:35.602 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.602 DEBUG:boto:Data: 2023-10-11T14:36:35.602 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.603 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.604 DEBUG:boto:Params: {} 2023-10-11T14:36:35.604 DEBUG:boto:Token: None 2023-10-11T14:36:35.604 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY= 2023-10-11T14:36:35.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY=', 'Content-Length': '0'} 2023-10-11T14:36:35.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fb80a2ed86960c5-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.608 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.609 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.609 DEBUG:boto:path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.610 DEBUG:boto:auth_path=/dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.610 DEBUG:boto:Method: GET 2023-10-11T14:36:35.610 DEBUG:boto:Path: /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.611 DEBUG:boto:Data: 2023-10-11T14:36:35.611 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.612 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.612 DEBUG:boto:Params: {} 2023-10-11T14:36:35.613 DEBUG:boto:Token: None 2023-10-11T14:36:35.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj?versionId=null 2023-10-11T14:36:35.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY= 2023-10-11T14:36:35.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ggTh5TdRCng+uB+ku/ji28c6kYY=', 'Content-Length': '0'} 2023-10-11T14:36:35.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d208cad7367cef4-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.658 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.659 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.659 DEBUG:boto:path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.660 DEBUG:boto:auth_path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.660 DEBUG:boto:Method: GET 2023-10-11T14:36:35.660 DEBUG:boto:Path: /dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.661 DEBUG:boto:Data: 2023-10-11T14:36:35.661 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.662 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.663 DEBUG:boto:Params: {} 2023-10-11T14:36:35.663 DEBUG:boto:Token: None 2023-10-11T14:36:35.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM= 2023-10-11T14:36:35.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM=', 'Content-Length': '0'} 2023-10-11T14:36:35.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92d6c19a5750697-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.668 DEBUG:boto:b'dghqvd-10myobj1000falsemyobj2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.669 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.669 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.670 DEBUG:boto:path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.670 DEBUG:boto:auth_path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.671 DEBUG:boto:Method: GET 2023-10-11T14:36:35.671 DEBUG:boto:Path: /dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.672 DEBUG:boto:Data: 2023-10-11T14:36:35.672 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.673 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.673 DEBUG:boto:Params: {} 2023-10-11T14:36:35.674 DEBUG:boto:Token: None 2023-10-11T14:36:35.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM= 2023-10-11T14:36:35.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM=', 'Content-Length': '0'} 2023-10-11T14:36:35.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fab5e0ba8a9638b5-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.678 DEBUG:boto:b'dghqvd-10myobj1000falsemyobj2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.679 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.679 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.680 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.680 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:35.681 DEBUG:boto:Data: 2023-10-11T14:36:35.681 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.682 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.682 DEBUG:boto:Params: {} 2023-10-11T14:36:35.683 DEBUG:boto:Token: None 2023-10-11T14:36:35.683 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj 2023-10-11T14:36:35.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LLtUjSFF42YyqIJW7F0GA1HBpL0= 2023-10-11T14:36:35.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LLtUjSFF42YyqIJW7F0GA1HBpL0=', 'Content-Length': '0'} 2023-10-11T14:36:35.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e6d7dc55304d9a3-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.687 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.687 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.688 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.688 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:35.688 DEBUG:boto:Data: 2023-10-11T14:36:35.689 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.689 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.690 DEBUG:boto:Params: {} 2023-10-11T14:36:35.690 DEBUG:boto:Token: None 2023-10-11T14:36:35.691 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj 2023-10-11T14:36:35.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LLtUjSFF42YyqIJW7F0GA1HBpL0= 2023-10-11T14:36:35.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LLtUjSFF42YyqIJW7F0GA1HBpL0=', 'Content-Length': '0'} 2023-10-11T14:36:35.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000322dae496164b483-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-10-11T14:36:35.696 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.697 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.697 DEBUG:boto:Method: GET 2023-10-11T14:36:35.697 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:35.698 DEBUG:boto:Data: 2023-10-11T14:36:35.698 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.699 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.699 DEBUG:boto:Params: {} 2023-10-11T14:36:35.700 DEBUG:boto:Token: None 2023-10-11T14:36:35.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj 2023-10-11T14:36:35.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cFjsD3QOKVJzBwvBToRN4/UbX4Q= 2023-10-11T14:36:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cFjsD3QOKVJzBwvBToRN4/UbX4Q=', 'Content-Length': '0'} 2023-10-11T14:36:35.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026a18633d25dd04e-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.750 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:35.751 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:35.751 DEBUG:boto:Method: GET 2023-10-11T14:36:35.752 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:35.752 DEBUG:boto:Data: 2023-10-11T14:36:35.753 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.754 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.754 DEBUG:boto:Params: {} 2023-10-11T14:36:35.755 DEBUG:boto:Token: None 2023-10-11T14:36:35.755 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/myobj 2023-10-11T14:36:35.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cFjsD3QOKVJzBwvBToRN4/UbX4Q= 2023-10-11T14:36:35.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cFjsD3QOKVJzBwvBToRN4/UbX4Q=', 'Content-Length': '0'} 2023-10-11T14:36:35.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ac531c2057901f6-006526b2f3-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.759 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.759 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.760 DEBUG:boto:path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.760 DEBUG:boto:auth_path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.760 DEBUG:boto:Method: GET 2023-10-11T14:36:35.761 DEBUG:boto:Path: /dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.761 DEBUG:boto:Data: 2023-10-11T14:36:35.762 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.763 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.763 DEBUG:boto:Params: {} 2023-10-11T14:36:35.764 DEBUG:boto:Token: None 2023-10-11T14:36:35.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM= 2023-10-11T14:36:35.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM=', 'Content-Length': '0'} 2023-10-11T14:36:35.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2a1c31bc9a295a-006526b2f3-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.768 DEBUG:boto:b'dghqvd-10myobj1000falsemyobj2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.769 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:35.769 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:35.769 DEBUG:boto:path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.770 DEBUG:boto:auth_path=/dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.770 DEBUG:boto:Method: GET 2023-10-11T14:36:35.771 DEBUG:boto:Path: /dghqvd-10/?prefix=myobj 2023-10-11T14:36:35.771 DEBUG:boto:Data: 2023-10-11T14:36:35.772 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.772 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:35.772 DEBUG:boto:Port: 8001 2023-10-11T14:36:35.773 DEBUG:boto:Params: {} 2023-10-11T14:36:35.773 DEBUG:boto:Token: None 2023-10-11T14:36:35.774 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-10/ 2023-10-11T14:36:35.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM= 2023-10-11T14:36:35.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:43BdsJPWrbga1bTeaa8L1uD1DWM=', 'Content-Length': '0'} 2023-10-11T14:36:35.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5fddb86c2bd033-006526b2f3-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.777 DEBUG:boto:b'dghqvd-10myobj1000falsemyobj2023-10-11T14:36:14.653Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:36:35.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-10 zones={a1, a2} 2023-10-11T14:36:35.778 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:35.779 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:35.779 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.780 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:35.780 DEBUG:boto:Data: 2023-10-11T14:36:35.781 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.781 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.782 DEBUG:boto:Params: {} 2023-10-11T14:36:35.782 DEBUG:boto:Token: None 2023-10-11T14:36:35.783 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/ 2023-10-11T14:36:35.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q= 2023-10-11T14:36:35.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:72xulre+6dZ0lKX/sZ0NxojZN6Q=', 'Content-Length': '0'} 2023-10-11T14:36:35.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b5a7843f3b24fd5-006526b2f3-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.786 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.787 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.787 DEBUG:boto:Method: HEAD 2023-10-11T14:36:35.788 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.788 DEBUG:boto:Data: 2023-10-11T14:36:35.788 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.789 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.790 DEBUG:boto:Params: {} 2023-10-11T14:36:35.790 DEBUG:boto:Token: None 2023-10-11T14:36:35.791 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg= 2023-10-11T14:36:35.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EHV8m41NDS+e9sPFYnkEnTJ2Ztg=', 'Content-Length': '0'} 2023-10-11T14:36:35.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e2be0ee4cc9c319-006526b2f3-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.794 DEBUG:boto:path=/dghqvd-9/myobj 2023-10-11T14:36:35.794 DEBUG:boto:auth_path=/dghqvd-9/myobj 2023-10-11T14:36:35.795 DEBUG:boto:Method: DELETE 2023-10-11T14:36:35.795 DEBUG:boto:Path: /dghqvd-9/myobj 2023-10-11T14:36:35.795 DEBUG:boto:Data: 2023-10-11T14:36:35.796 DEBUG:boto:Headers: {} 2023-10-11T14:36:35.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:35.797 DEBUG:boto:Port: 8000 2023-10-11T14:36:35.797 DEBUG:boto:Params: {} 2023-10-11T14:36:35.797 DEBUG:boto:Token: None 2023-10-11T14:36:35.798 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:36:35 GMT /dghqvd-9/myobj 2023-10-11T14:36:35.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dRiZiCUrOfQDH4K9g1rmLZCEKxA= 2023-10-11T14:36:35.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dRiZiCUrOfQDH4K9g1rmLZCEKxA=', 'Content-Length': '0'} 2023-10-11T14:36:35.803 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a78986cacdbd4f12-006526b2f3-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:36:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:35.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:35.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:35.804 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:36:35.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:35.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:35.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:35.893+0000 7f97291d8780 1 waiting for incremental sync to catch up: 2023-10-11T14:36:35.898 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-10-11T14:36:35.898 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2023-10-11T14:36:40.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:40.894+0000 7f97291d8780 1 waiting for incremental sync to catch up: 2023-10-11T14:36:40.897 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-10-11T14:36:40.897 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2023-10-11T14:36:45.898 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:45.895+0000 7f97291d8780 1 waiting for incremental sync to catch up: 2023-10-11T14:36:45.898 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-10-11T14:36:45.898 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2023-10-11T14:36:50.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:50.897+0000 7f97291d8780 1 bucket sync caught up with source: 2023-10-11T14:36:50.899 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.13.6, , , , , , , , , ] 2023-10-11T14:36:50.900 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.13.6, , , , , , , , , ] 2023-10-11T14:36:50.900 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:50.897+0000 7f97291d8780 0 bucket checkpoint complete 2023-10-11T14:36:50.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-9 zones={a2, a1} 2023-10-11T14:36:50.915 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:50.915 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:50.916 DEBUG:boto:Method: HEAD 2023-10-11T14:36:50.916 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:50.916 DEBUG:boto:Data: 2023-10-11T14:36:50.917 DEBUG:boto:Headers: {} 2023-10-11T14:36:50.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:50.918 DEBUG:boto:Port: 8001 2023-10-11T14:36:50.918 DEBUG:boto:Params: {} 2023-10-11T14:36:50.919 DEBUG:boto:Token: None 2023-10-11T14:36:50.919 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:50 GMT /dghqvd-9/ 2023-10-11T14:36:50.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g93aqCPpuiuP66W5DGWkvSrB+j0= 2023-10-11T14:36:50.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g93aqCPpuiuP66W5DGWkvSrB+j0=', 'Content-Length': '0'} 2023-10-11T14:36:50.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ac972fdf413e69b-006526b302-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:50.923 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:50.924 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:50.924 DEBUG:boto:Method: HEAD 2023-10-11T14:36:50.925 DEBUG:boto:Path: /dghqvd-9/ 2023-10-11T14:36:50.925 DEBUG:boto:Data: 2023-10-11T14:36:50.926 DEBUG:boto:Headers: {} 2023-10-11T14:36:50.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:50.927 DEBUG:boto:Port: 8000 2023-10-11T14:36:50.927 DEBUG:boto:Params: {} 2023-10-11T14:36:50.927 DEBUG:boto:Token: None 2023-10-11T14:36:50.928 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:50 GMT /dghqvd-9/ 2023-10-11T14:36:50.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g93aqCPpuiuP66W5DGWkvSrB+j0= 2023-10-11T14:36:50.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g93aqCPpuiuP66W5DGWkvSrB+j0=', 'Content-Length': '0'} 2023-10-11T14:36:50.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004046e9c0de1e5bab-006526b302-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:50.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:36:50.932 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:50.933 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:50.933 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.934 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.934 DEBUG:boto:Method: GET 2023-10-11T14:36:50.935 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.935 DEBUG:boto:Data: 2023-10-11T14:36:50.935 DEBUG:boto:Headers: {} 2023-10-11T14:36:50.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:50.936 DEBUG:boto:Port: 8001 2023-10-11T14:36:50.937 DEBUG:boto:Params: {} 2023-10-11T14:36:50.937 DEBUG:boto:Token: None 2023-10-11T14:36:50.938 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:50 GMT /dghqvd-9/?versions 2023-10-11T14:36:50.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:plu6BZAmQUYXNlOkE2KT4DxZKIc= 2023-10-11T14:36:50.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:plu6BZAmQUYXNlOkE2KT4DxZKIc=', 'Content-Length': '0'} 2023-10-11T14:36:50.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099d895ffcd5fc559-006526b302-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:50.982 DEBUG:boto:b'dghqvd-9999false' 2023-10-11T14:36:50.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:36:50.986 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:50.986 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:50.987 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.987 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.988 DEBUG:boto:Method: GET 2023-10-11T14:36:50.988 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.989 DEBUG:boto:Data: 2023-10-11T14:36:50.989 DEBUG:boto:Headers: {} 2023-10-11T14:36:50.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:50.990 DEBUG:boto:Port: 8000 2023-10-11T14:36:50.991 DEBUG:boto:Params: {} 2023-10-11T14:36:50.991 DEBUG:boto:Token: None 2023-10-11T14:36:50.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:50 GMT /dghqvd-9/?versions 2023-10-11T14:36:50.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:plu6BZAmQUYXNlOkE2KT4DxZKIc= 2023-10-11T14:36:50.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:plu6BZAmQUYXNlOkE2KT4DxZKIc=', 'Content-Length': '0'} 2023-10-11T14:36:50.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003537fe7189645b31-006526b302-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:50.995 DEBUG:boto:b'dghqvd-9999false' 2023-10-11T14:36:50.996 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:50.997 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:50.997 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.997 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.998 DEBUG:boto:Method: GET 2023-10-11T14:36:50.998 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:50.999 DEBUG:boto:Data: 2023-10-11T14:36:50.999 DEBUG:boto:Headers: {} 2023-10-11T14:36:51.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:51.000 DEBUG:boto:Port: 8001 2023-10-11T14:36:51.000 DEBUG:boto:Params: {} 2023-10-11T14:36:51.001 DEBUG:boto:Token: None 2023-10-11T14:36:51.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:51 GMT /dghqvd-9/?versions 2023-10-11T14:36:51.002 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFd4csD/3hdhqhfFTk1Ue7N5QvM= 2023-10-11T14:36:51.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFd4csD/3hdhqhfFTk1Ue7N5QvM=', 'Content-Length': '0'} 2023-10-11T14:36:51.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d0c22117af2232f-006526b303-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:51.005 DEBUG:boto:b'dghqvd-9999false' 2023-10-11T14:36:51.006 DEBUG:boto:path=/dghqvd-9/ 2023-10-11T14:36:51.006 DEBUG:boto:auth_path=/dghqvd-9/ 2023-10-11T14:36:51.007 DEBUG:boto:path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:51.007 DEBUG:boto:auth_path=/dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:51.008 DEBUG:boto:Method: GET 2023-10-11T14:36:51.009 DEBUG:boto:Path: /dghqvd-9/?versions&max-keys=999 2023-10-11T14:36:51.009 DEBUG:boto:Data: 2023-10-11T14:36:51.010 DEBUG:boto:Headers: {} 2023-10-11T14:36:51.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:36:51.010 DEBUG:boto:Port: 8000 2023-10-11T14:36:51.011 DEBUG:boto:Params: {} 2023-10-11T14:36:51.011 DEBUG:boto:Token: None 2023-10-11T14:36:51.012 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:36:51 GMT /dghqvd-9/?versions 2023-10-11T14:36:51.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFd4csD/3hdhqhfFTk1Ue7N5QvM= 2023-10-11T14:36:51.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFd4csD/3hdhqhfFTk1Ue7N5QvM=', 'Content-Length': '0'} 2023-10-11T14:36:51.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b66de64af89fbafe-006526b303-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:36:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:51.058 DEBUG:boto:b'dghqvd-9999false' 2023-10-11T14:36:51.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-9 zones={a2, a1} 2023-10-11T14:36:51.060 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:36:51.060 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:36:51.060 DEBUG:boto:Method: HEAD 2023-10-11T14:36:51.061 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:36:51.061 DEBUG:boto:Data: 2023-10-11T14:36:51.062 DEBUG:boto:Headers: {} 2023-10-11T14:36:51.062 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:51.063 DEBUG:boto:Port: 8001 2023-10-11T14:36:51.063 DEBUG:boto:Params: {} 2023-10-11T14:36:51.063 DEBUG:boto:Token: None 2023-10-11T14:36:51.064 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:51 GMT /dghqvd-10/ 2023-10-11T14:36:51.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G4Vjrr2W1CpocTahVWeBnOY3mS4= 2023-10-11T14:36:51.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G4Vjrr2W1CpocTahVWeBnOY3mS4=', 'Content-Length': '0'} 2023-10-11T14:36:51.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad53d9e402bf66e2-006526b303-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:36:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:51.068 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:51.068 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:51.069 DEBUG:boto:Method: HEAD 2023-10-11T14:36:51.069 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:51.070 DEBUG:boto:Data: 2023-10-11T14:36:51.070 DEBUG:boto:Headers: {} 2023-10-11T14:36:51.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:51.071 DEBUG:boto:Port: 8001 2023-10-11T14:36:51.071 DEBUG:boto:Params: {} 2023-10-11T14:36:51.073 DEBUG:boto:Token: None 2023-10-11T14:36:51.074 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:36:51 GMT /dghqvd-10/myobj 2023-10-11T14:36:51.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fqmaTexOPIRr4c53ev1EbBGI2MA= 2023-10-11T14:36:51.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fqmaTexOPIRr4c53ev1EbBGI2MA=', 'Content-Length': '0'} 2023-10-11T14:36:51.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:36:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a36117f3e97b714-006526b303-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:36:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:51.078 DEBUG:boto:path=/dghqvd-10/myobj 2023-10-11T14:36:51.078 DEBUG:boto:auth_path=/dghqvd-10/myobj 2023-10-11T14:36:51.079 DEBUG:boto:Method: DELETE 2023-10-11T14:36:51.079 DEBUG:boto:Path: /dghqvd-10/myobj 2023-10-11T14:36:51.079 DEBUG:boto:Data: 2023-10-11T14:36:51.080 DEBUG:boto:Headers: {} 2023-10-11T14:36:51.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:36:51.080 DEBUG:boto:Port: 8001 2023-10-11T14:36:51.081 DEBUG:boto:Params: {} 2023-10-11T14:36:51.081 DEBUG:boto:Token: None 2023-10-11T14:36:51.082 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:36:51 GMT /dghqvd-10/myobj 2023-10-11T14:36:51.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZcrJJWwPeW8HfihD5ebXHAgNsRk= 2023-10-11T14:36:51.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:36:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZcrJJWwPeW8HfihD5ebXHAgNsRk=', 'Content-Length': '0'} 2023-10-11T14:36:51.089 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d739b5d6bec83db-006526b303-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:36:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:36:51.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:51.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:36:51.090 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:36:51.156 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:36:51.156 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:36:51.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:51.190+0000 7fc35a665780 1 waiting for incremental sync to catch up: 2023-10-11T14:36:51.193 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:36:51.193 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:36:56.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:36:56.190+0000 7fc35a665780 1 waiting for incremental sync to catch up: 2023-10-11T14:36:56.194 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:36:56.194 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:37:01.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:01.192+0000 7fc35a665780 1 waiting for incremental sync to catch up: 2023-10-11T14:37:01.196 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:37:01.196 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:37:06.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:06.193+0000 7fc35a665780 1 waiting for incremental sync to catch up: 2023-10-11T14:37:06.196 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.15.6, , , , , , , , , ] 2023-10-11T14:37:06.197 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:37:11.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:11.194+0000 7fc35a665780 1 bucket sync caught up with source: 2023-10-11T14:37:11.198 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:37:11.198 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.18.6, , , , , , , , , ] 2023-10-11T14:37:11.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:11.194+0000 7fc35a665780 0 bucket checkpoint complete 2023-10-11T14:37:11.213 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-10 zones={a1, a2} 2023-10-11T14:37:11.213 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.214 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.214 DEBUG:boto:Method: HEAD 2023-10-11T14:37:11.215 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:37:11.216 DEBUG:boto:Data: 2023-10-11T14:37:11.216 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.217 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:11.217 DEBUG:boto:Port: 8000 2023-10-11T14:37:11.217 DEBUG:boto:Params: {} 2023-10-11T14:37:11.218 DEBUG:boto:Token: None 2023-10-11T14:37:11.219 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/ 2023-10-11T14:37:11.219 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hGIYhOTfl6SgoNayUrpY+1jCWQw= 2023-10-11T14:37:11.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hGIYhOTfl6SgoNayUrpY+1jCWQw=', 'Content-Length': '0'} 2023-10-11T14:37:11.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e4f75992c264e15-006526b317-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.223 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.223 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.224 DEBUG:boto:Method: HEAD 2023-10-11T14:37:11.224 DEBUG:boto:Path: /dghqvd-10/ 2023-10-11T14:37:11.225 DEBUG:boto:Data: 2023-10-11T14:37:11.225 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:37:11.226 DEBUG:boto:Port: 8001 2023-10-11T14:37:11.227 DEBUG:boto:Params: {} 2023-10-11T14:37:11.227 DEBUG:boto:Token: None 2023-10-11T14:37:11.228 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/ 2023-10-11T14:37:11.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hGIYhOTfl6SgoNayUrpY+1jCWQw= 2023-10-11T14:37:11.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hGIYhOTfl6SgoNayUrpY+1jCWQw=', 'Content-Length': '0'} 2023-10-11T14:37:11.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d979ef2a42aab3f-006526b317-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:37:11.233 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.234 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.234 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.235 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.236 DEBUG:boto:Method: GET 2023-10-11T14:37:11.236 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.236 DEBUG:boto:Data: 2023-10-11T14:37:11.237 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:11.238 DEBUG:boto:Port: 8000 2023-10-11T14:37:11.239 DEBUG:boto:Params: {} 2023-10-11T14:37:11.239 DEBUG:boto:Token: None 2023-10-11T14:37:11.240 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/?versions 2023-10-11T14:37:11.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk= 2023-10-11T14:37:11.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk=', 'Content-Length': '0'} 2023-10-11T14:37:11.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fca98015c5b2461-006526b317-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.244 DEBUG:boto:b'dghqvd-10999false' 2023-10-11T14:37:11.245 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:37:11.245 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.246 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.246 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.247 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.247 DEBUG:boto:Method: GET 2023-10-11T14:37:11.248 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.248 DEBUG:boto:Data: 2023-10-11T14:37:11.248 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:37:11.249 DEBUG:boto:Port: 8001 2023-10-11T14:37:11.250 DEBUG:boto:Params: {} 2023-10-11T14:37:11.250 DEBUG:boto:Token: None 2023-10-11T14:37:11.251 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/?versions 2023-10-11T14:37:11.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk= 2023-10-11T14:37:11.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk=', 'Content-Length': '0'} 2023-10-11T14:37:11.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20ebcd2a2e5a3e3-006526b317-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.298 DEBUG:boto:b'dghqvd-10999false' 2023-10-11T14:37:11.299 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.300 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.301 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.301 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.302 DEBUG:boto:Method: GET 2023-10-11T14:37:11.303 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.304 DEBUG:boto:Data: 2023-10-11T14:37:11.304 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:11.305 DEBUG:boto:Port: 8000 2023-10-11T14:37:11.306 DEBUG:boto:Params: {} 2023-10-11T14:37:11.307 DEBUG:boto:Token: None 2023-10-11T14:37:11.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/?versions 2023-10-11T14:37:11.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk= 2023-10-11T14:37:11.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk=', 'Content-Length': '0'} 2023-10-11T14:37:11.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5bda2c1731cc7a1-006526b317-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.312 DEBUG:boto:b'dghqvd-10999false' 2023-10-11T14:37:11.313 DEBUG:boto:path=/dghqvd-10/ 2023-10-11T14:37:11.314 DEBUG:boto:auth_path=/dghqvd-10/ 2023-10-11T14:37:11.315 DEBUG:boto:path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.315 DEBUG:boto:auth_path=/dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.316 DEBUG:boto:Method: GET 2023-10-11T14:37:11.317 DEBUG:boto:Path: /dghqvd-10/?versions&max-keys=999 2023-10-11T14:37:11.318 DEBUG:boto:Data: 2023-10-11T14:37:11.319 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:37:11.320 DEBUG:boto:Port: 8001 2023-10-11T14:37:11.320 DEBUG:boto:Params: {} 2023-10-11T14:37:11.320 DEBUG:boto:Token: None 2023-10-11T14:37:11.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-10/?versions 2023-10-11T14:37:11.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk= 2023-10-11T14:37:11.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lam82iMwvReIe/TPn+v/Ga+y1Vk=', 'Content-Length': '0'} 2023-10-11T14:37:11.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f031016181e31331-006526b317-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.327 DEBUG:boto:b'dghqvd-10999false' 2023-10-11T14:37:11.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-10 zones={a1, a2} 2023-10-11T14:37:11.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-10-11T14:37:11.329 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-11 2023-10-11T14:37:11.330 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:37:11.330 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:37:11.331 DEBUG:boto:Method: PUT 2023-10-11T14:37:11.331 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:37:11.332 DEBUG:boto:Data: 2023-10-11T14:37:11.332 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:11.333 DEBUG:boto:Port: 8000 2023-10-11T14:37:11.333 DEBUG:boto:Params: {} 2023-10-11T14:37:11.333 DEBUG:boto:Token: None 2023-10-11T14:37:11.334 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-11/ 2023-10-11T14:37:11.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JI3sal2zD/uyDedkJstFqKDUIus= 2023-10-11T14:37:11.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JI3sal2zD/uyDedkJstFqKDUIus=', 'Content-Length': '0'} 2023-10-11T14:37:11.349 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028252f1121d5fa63-006526b317-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.350 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-12 2023-10-11T14:37:11.351 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:37:11.351 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:37:11.352 DEBUG:boto:Method: PUT 2023-10-11T14:37:11.352 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:37:11.353 DEBUG:boto:Data: 2023-10-11T14:37:11.353 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:37:11.355 DEBUG:boto:Port: 8001 2023-10-11T14:37:11.355 DEBUG:boto:Params: {} 2023-10-11T14:37:11.356 DEBUG:boto:Token: None 2023-10-11T14:37:11.356 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-12/ 2023-10-11T14:37:11.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fS/w0SrYogCy0o9bCElYXX9Qzuk= 2023-10-11T14:37:11.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fS/w0SrYogCy0o9bCElYXX9Qzuk=', 'Content-Length': '0'} 2023-10-11T14:37:11.381 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f473c23065d8c066-006526b317-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.382 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:37:11.382 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:37:11.383 DEBUG:boto:path=/dghqvd-11/?versioning 2023-10-11T14:37:11.383 DEBUG:boto:auth_path=/dghqvd-11/?versioning 2023-10-11T14:37:11.383 DEBUG:boto:Method: PUT 2023-10-11T14:37:11.384 DEBUG:boto:Path: /dghqvd-11/?versioning 2023-10-11T14:37:11.384 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:37:11.385 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:11.385 DEBUG:boto:Port: 8000 2023-10-11T14:37:11.386 DEBUG:boto:Params: {} 2023-10-11T14:37:11.386 DEBUG:boto:Token: None 2023-10-11T14:37:11.387 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-11/?versioning 2023-10-11T14:37:11.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RV4gk3tW0r/CAuhJlQwkBeTvqQo= 2023-10-11T14:37:11.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RV4gk3tW0r/CAuhJlQwkBeTvqQo=', 'Content-Length': '228'} 2023-10-11T14:37:11.393 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077f1d7eaed676e34-006526b317-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.394 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:37:11.394 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:37:11.395 DEBUG:boto:path=/dghqvd-12/?versioning 2023-10-11T14:37:11.395 DEBUG:boto:auth_path=/dghqvd-12/?versioning 2023-10-11T14:37:11.396 DEBUG:boto:Method: PUT 2023-10-11T14:37:11.396 DEBUG:boto:Path: /dghqvd-12/?versioning 2023-10-11T14:37:11.396 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:37:11.397 DEBUG:boto:Headers: {} 2023-10-11T14:37:11.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:37:11.398 DEBUG:boto:Port: 8001 2023-10-11T14:37:11.398 DEBUG:boto:Params: {} 2023-10-11T14:37:11.399 DEBUG:boto:Token: None 2023-10-11T14:37:11.399 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:37:11 GMT /dghqvd-12/?versioning 2023-10-11T14:37:11.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EK9elxgusKPBfM3uL79hGx631F4= 2023-10-11T14:37:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EK9elxgusKPBfM3uL79hGx631F4=', 'Content-Length': '228'} 2023-10-11T14:37:11.409 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c806fcef65770965-006526b317-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:11.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:11.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:11.411 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:37:11.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:37:11.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:37:11.492 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:37:11.503 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:37:11.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.504 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:37:11.504 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:37:11.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.505 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:37:11.505 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:37:11.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.506 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:37:11.506 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:37:11.506 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.506 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.507 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035031.404819_54.1", 2023-10-11T14:37:11.507 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:37:11.404819Z" 2023-10-11T14:37:11.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.507 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:37:11.508 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035031.404819_54.1'} 2023-10-11T14:37:11.509 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:37:11.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:11.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:11.510 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:37:11.562 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:37:11.562 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:37:11.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:11.589+0000 7f6534651780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:37:11.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:11.590+0000 7f6534651780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:37:11.606 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:37:11.607 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:37:11.607 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:37:11.607 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:37:11.607 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:37:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:37:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:37:11.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:37:11.609 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:11.609 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:11.609 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:37:11.609 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:11.610 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:37:11.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:11.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:37:11.610 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:11.611 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.611 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.611 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.611 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:37:11.611 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:11.612 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:11.612 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:37:11.612 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:11.612 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:11.612 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:11.613 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:37:11.613 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:11.613 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.613 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.614 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:37:11.614 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:11.614 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:11.614 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:37:11.615 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:11.615 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:11.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:11.615 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:37:11.615 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:11.616 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.616 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:11.616 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:37:11.616 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:11.617 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:11.617 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.567478_40.1", 2023-10-11T14:37:11.617 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:11.617 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:11.617 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:11.618 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.567478Z", 2023-10-11T14:37:11.618 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:11.618 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.618 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:11.619 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:37:11.619 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.619 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:37:11.619 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:37:11.619 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:37:11.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:11.620 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:37:11Z" 2023-10-11T14:37:11.620 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:37:11.621 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697034974.567478_40.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.567478Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:37:11Z" } 2023-10-11T14:37:11.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697034974.567478_40.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.567478Z', 'realm_epoch': 2}}] 2023-10-11T14:37:11.622 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035031.404819_54.1'} 2023-10-11T14:37:11.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697034974.567478_40.1'} 2023-10-11T14:37:11.623 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035031.404819_54.1 target=1_1697034974.567478_40.1 2023-10-11T14:37:16.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:16.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:37:16.625 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:37:16.682 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:37:16.682 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:37:16.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:16.709+0000 7f60682e3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:37:16.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:16.710+0000 7f60682e3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:37:16.725 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:37:16.726 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:37:16.726 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:37:16.726 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:37:16.726 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:37:16.727 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:37:16.727 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:16.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.727 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:37:16.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:16.728 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:37:16.728 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:16.728 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:16.729 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:37:16.729 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:16.729 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:37:16.729 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:16.729 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:37:16.730 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:16.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:16.730 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.730 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:16.730 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:16.731 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:16.732 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:37:16.732 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:16.732 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:16.732 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.732 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:16.733 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:37:16.733 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:16.733 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:16.733 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:37:16.734 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:16.734 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:16.734 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:16.734 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:37:16.734 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:16.735 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:16.735 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.735 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:37:16.735 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:37:16.736 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:37:16.736 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:37:16.736 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035031.404819_54.1", 2023-10-11T14:37:16.736 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:37:16.736 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:37:16.737 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:37:16.737 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:37:11.404819Z", 2023-10-11T14:37:16.737 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:37:16.737 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:16.737 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:37:16.738 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:37:16.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.738 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:37:16.738 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:37:16.738 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:37:16.739 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:37:16.739 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:37:16Z" 2023-10-11T14:37:16.739 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:37:16.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035031.404819_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:37:11.404819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:37:16Z" } 2023-10-11T14:37:16.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035031.404819_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:37:11.404819Z', 'realm_epoch': 2}}] 2023-10-11T14:37:16.741 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035031.404819_54.1'} 2023-10-11T14:37:16.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035031.404819_54.1'} 2023-10-11T14:37:16.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:37:16.742 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:37:16.743 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:37:16.743 DEBUG:boto:Method: HEAD 2023-10-11T14:37:16.743 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:37:16.744 DEBUG:boto:Data: 2023-10-11T14:37:16.744 DEBUG:boto:Headers: {} 2023-10-11T14:37:16.745 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:16.745 DEBUG:boto:Port: 8000 2023-10-11T14:37:16.746 DEBUG:boto:Params: {} 2023-10-11T14:37:16.746 DEBUG:boto:Token: None 2023-10-11T14:37:16.747 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:37:16 GMT /dghqvd-11/ 2023-10-11T14:37:16.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZDmwU4U9VTs0c+dHUoTn8ItZVJ0= 2023-10-11T14:37:16.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:37:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZDmwU4U9VTs0c+dHUoTn8ItZVJ0=', 'Content-Length': '0'} 2023-10-11T14:37:16.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee74511f73265855-006526b31c-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:16.751 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:37:16.751 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:37:16.751 DEBUG:boto:Method: PUT 2023-10-11T14:37:16.752 DEBUG:boto:Path: /dghqvd-11/dummy 2023-10-11T14:37:16.752 DEBUG:boto:Data: 2023-10-11T14:37:16.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:37:16.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:37:16.753 DEBUG:boto:Port: 8000 2023-10-11T14:37:16.754 DEBUG:boto:Params: {} 2023-10-11T14:37:16.754 DEBUG:boto:Token: None 2023-10-11T14:37:16.755 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:37:16 GMT /dghqvd-11/dummy 2023-10-11T14:37:16.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cemt46CrKl+a5Eh1BYWjbe0Z1+w= 2023-10-11T14:37:16.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:37:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cemt46CrKl+a5Eh1BYWjbe0Z1+w='} 2023-10-11T14:37:16.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-amz-request-id', 'tx00000c0d5b3f318deb387-006526b31c-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:37:16.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:37:16.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:37:16.767 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:37:16.825 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:37:16.826 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:37:16.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:16.855+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:21.856+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:26.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:26.856+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:31.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:31.857+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:36.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:36.858+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:41.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:41.859+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:46.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:46.860+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:51.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:51.860+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:37:56.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:37:56.861+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:01.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:01.862+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:06.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:06.862+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:11.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:11.863+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:16.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:16.864+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:21.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:21.865+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:26.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:26.864+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:31.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:31.865+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:36.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:36.866+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:41.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:41.866+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:46.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:46.867+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:51.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:51.868+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:38:56.871 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:38:56.868+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:01.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:01.869+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:06.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:06.870+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:11.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:11.871+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:16.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:16.871+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:21.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:21.872+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:26.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:26.873+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:31.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:31.873+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:36.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:36.874+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:41.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:41.874+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:46.877 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:46.875+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:51.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:51.875+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:39:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:39:56.874+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:40:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:01.875+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:40:06.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:06.875+0000 7fef7020f780 1 waiting to reach incremental sync.. 2023-10-11T14:40:11.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:11.877+0000 7fef7020f780 1 bucket sync caught up with source: 2023-10-11T14:40:11.881 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:11.881 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:11.881 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:11.877+0000 7fef7020f780 0 bucket checkpoint complete 2023-10-11T14:40:11.896 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:40:11.897 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:40:11.897 DEBUG:boto:Method: HEAD 2023-10-11T14:40:11.898 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:40:11.898 DEBUG:boto:Data: 2023-10-11T14:40:11.899 DEBUG:boto:Headers: {} 2023-10-11T14:40:11.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:11.900 DEBUG:boto:Port: 8001 2023-10-11T14:40:11.900 DEBUG:boto:Params: {} 2023-10-11T14:40:11.901 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:40:11.902 DEBUG:boto:Token: None 2023-10-11T14:40:11.903 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:11 GMT /dghqvd-12/ 2023-10-11T14:40:11.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BKw61/rr+yipPKRpgyO6PJgHsn8= 2023-10-11T14:40:11.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BKw61/rr+yipPKRpgyO6PJgHsn8=', 'Content-Length': '0'} 2023-10-11T14:40:11.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd27d23b2e1a58fa-006526b3cb-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:11.910 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:40:11.911 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:40:11.911 DEBUG:boto:Method: PUT 2023-10-11T14:40:11.912 DEBUG:boto:Path: /dghqvd-12/dummy 2023-10-11T14:40:11.912 DEBUG:boto:Data: 2023-10-11T14:40:11.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:40:11.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:11.914 DEBUG:boto:Port: 8001 2023-10-11T14:40:11.914 DEBUG:boto:Params: {} 2023-10-11T14:40:11.914 DEBUG:boto:Token: None 2023-10-11T14:40:11.915 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:40:11 GMT /dghqvd-12/dummy 2023-10-11T14:40:11.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8uI2cUpjfPE90XxztTYV6tk2/04= 2023-10-11T14:40:11.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8uI2cUpjfPE90XxztTYV6tk2/04='} 2023-10-11T14:40:11.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-amz-request-id', 'tx0000025c8709a7d904636-006526b3cb-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:11.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:11.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:11.932 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:40:11.997 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:40:11.998 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:40:12.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:12.028+0000 7f05d0697780 1 waiting to reach incremental sync.. 2023-10-11T14:40:17.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:17.029+0000 7f05d0697780 1 waiting to reach incremental sync.. 2023-10-11T14:40:22.033 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:22.030+0000 7f05d0697780 1 waiting to reach incremental sync.. 2023-10-11T14:40:27.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:27.030+0000 7f05d0697780 1 waiting to reach incremental sync.. 2023-10-11T14:40:32.035 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:32.032+0000 7f05d0697780 1 bucket sync caught up with source: 2023-10-11T14:40:32.036 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , , , ] 2023-10-11T14:40:32.036 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.16.11, , , , , ] 2023-10-11T14:40:32.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:32.032+0000 7f05d0697780 0 bucket checkpoint complete 2023-10-11T14:40:32.051 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:32.052 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:32.052 DEBUG:boto:Method: HEAD 2023-10-11T14:40:32.052 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:40:32.053 DEBUG:boto:Data: 2023-10-11T14:40:32.053 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.054 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.055 DEBUG:boto:Params: {} 2023-10-11T14:40:32.055 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:40:32.056 DEBUG:boto:Token: None 2023-10-11T14:40:32.056 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/ 2023-10-11T14:40:32.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DM26yYjO11bJQNVgdoIgYqhEKps= 2023-10-11T14:40:32.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DM26yYjO11bJQNVgdoIgYqhEKps=', 'Content-Length': '0'} 2023-10-11T14:40:32.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1530f7622ef935e-006526b3e0-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.062 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.063 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.063 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.064 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:40:32.064 DEBUG:boto:Data: 2023-10-11T14:40:32.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.065 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.065 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.066 DEBUG:boto:Params: {} 2023-10-11T14:40:32.066 DEBUG:boto:Token: None 2023-10-11T14:40:32.066 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a1 2023-10-11T14:40:32.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aD9MQVlgssue3f8HrUIfXv7ZpIc= 2023-10-11T14:40:32.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aD9MQVlgssue3f8HrUIfXv7ZpIc='} 2023-10-11T14:40:32.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-amz-request-id', 'tx00000039ae4624da46fac-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.079 DEBUG:rgw_multi.tests:version1 id=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:32.079 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.080 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.080 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.080 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:40:32.081 DEBUG:boto:Data: 2023-10-11T14:40:32.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.082 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.082 DEBUG:boto:Params: {} 2023-10-11T14:40:32.083 DEBUG:boto:Token: None 2023-10-11T14:40:32.083 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a1 2023-10-11T14:40:32.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:np8Yb39GFGpislWQxcL9Tq07HKs= 2023-10-11T14:40:32.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:np8Yb39GFGpislWQxcL9Tq07HKs='} 2023-10-11T14:40:32.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb'), ('x-amz-request-id', 'tx000003a3bd2d4f22eaf99-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.094 DEBUG:rgw_multi.tests:version2 id=lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb 2023-10-11T14:40:32.097 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.099 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.099 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb 2023-10-11T14:40:32.099 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb 2023-10-11T14:40:32.100 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.100 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb 2023-10-11T14:40:32.101 DEBUG:boto:Data: 2023-10-11T14:40:32.101 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.102 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.102 DEBUG:boto:Params: {} 2023-10-11T14:40:32.103 DEBUG:boto:Token: None 2023-10-11T14:40:32.103 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a1?versionId=lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb 2023-10-11T14:40:32.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6sdZGkiqPMCOtEuAufuwP3cA7cc= 2023-10-11T14:40:32.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6sdZGkiqPMCOtEuAufuwP3cA7cc=', 'Content-Length': '0'} 2023-10-11T14:40:32.114 DEBUG:boto:Response headers: [('x-amz-version-id', 'lm.cyTcwQ2gfyOBhuHYqBCqAmZF95qb'), ('x-amz-request-id', 'tx000008ad5e2c494e07440-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.114 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.115 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.115 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.116 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:40:32.116 DEBUG:boto:Data: 2023-10-11T14:40:32.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.117 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.117 DEBUG:boto:Params: {} 2023-10-11T14:40:32.118 DEBUG:boto:Token: None 2023-10-11T14:40:32.118 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a1 2023-10-11T14:40:32.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hh5UeRiR54+NZyGGI0MaFrxRuic= 2023-10-11T14:40:32.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hh5UeRiR54+NZyGGI0MaFrxRuic='} 2023-10-11T14:40:32.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR'), ('x-amz-request-id', 'tx00000a299ba33aabae15c-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.167 DEBUG:rgw_multi.tests:version3 id=WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR 2023-10-11T14:40:32.168 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.168 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:32.168 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR 2023-10-11T14:40:32.169 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR 2023-10-11T14:40:32.169 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.170 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR 2023-10-11T14:40:32.170 DEBUG:boto:Data: 2023-10-11T14:40:32.171 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.171 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.172 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.172 DEBUG:boto:Params: {} 2023-10-11T14:40:32.173 DEBUG:boto:Token: None 2023-10-11T14:40:32.173 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a1?versionId=WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR 2023-10-11T14:40:32.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8HkCwRijmGMz0D28c1akC0XTjEI= 2023-10-11T14:40:32.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8HkCwRijmGMz0D28c1akC0XTjEI=', 'Content-Length': '0'} 2023-10-11T14:40:32.184 DEBUG:boto:Response headers: [('x-amz-version-id', 'WAvoUljsXdO1r0pCuFGuDwuoQY9n0sR'), ('x-amz-request-id', 'tx00000b434586ce0641c8a-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.184 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:32.185 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:32.185 DEBUG:boto:Method: HEAD 2023-10-11T14:40:32.186 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:40:32.186 DEBUG:boto:Data: 2023-10-11T14:40:32.186 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.187 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.188 DEBUG:boto:Params: {} 2023-10-11T14:40:32.188 DEBUG:boto:Token: None 2023-10-11T14:40:32.189 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/ 2023-10-11T14:40:32.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DM26yYjO11bJQNVgdoIgYqhEKps= 2023-10-11T14:40:32.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DM26yYjO11bJQNVgdoIgYqhEKps=', 'Content-Length': '0'} 2023-10-11T14:40:32.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c3ec25c279b9dfd-006526b3e0-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.192 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.193 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.193 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.194 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:40:32.194 DEBUG:boto:Data: 2023-10-11T14:40:32.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.196 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.196 DEBUG:boto:Params: {} 2023-10-11T14:40:32.196 DEBUG:boto:Token: None 2023-10-11T14:40:32.197 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a2 2023-10-11T14:40:32.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N6B2ihPtAqGawzyKjFR1y/EKQbc= 2023-10-11T14:40:32.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N6B2ihPtAqGawzyKjFR1y/EKQbc='} 2023-10-11T14:40:32.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-amz-request-id', 'tx000004a65c5ff95be4d43-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.208 DEBUG:rgw_multi.tests:version1 id=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:32.208 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.209 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.209 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.210 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:40:32.210 DEBUG:boto:Data: 2023-10-11T14:40:32.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.211 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.212 DEBUG:boto:Params: {} 2023-10-11T14:40:32.212 DEBUG:boto:Token: None 2023-10-11T14:40:32.212 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a2 2023-10-11T14:40:32.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zhwjBwqrWRRn46gFtgfPuW3M0a0= 2023-10-11T14:40:32.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zhwjBwqrWRRn46gFtgfPuW3M0a0='} 2023-10-11T14:40:32.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz'), ('x-amz-request-id', 'tx00000bd29bb4e08ebddc4-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.224 DEBUG:rgw_multi.tests:version2 id=muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz 2023-10-11T14:40:32.224 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.225 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.225 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz 2023-10-11T14:40:32.225 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz 2023-10-11T14:40:32.226 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.226 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz 2023-10-11T14:40:32.227 DEBUG:boto:Data: 2023-10-11T14:40:32.227 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.229 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.229 DEBUG:boto:Params: {} 2023-10-11T14:40:32.229 DEBUG:boto:Token: None 2023-10-11T14:40:32.230 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a2?versionId=muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz 2023-10-11T14:40:32.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KYwRW9ir7N3ZnaAwH/Ic208kme0= 2023-10-11T14:40:32.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KYwRW9ir7N3ZnaAwH/Ic208kme0=', 'Content-Length': '0'} 2023-10-11T14:40:32.239 DEBUG:boto:Response headers: [('x-amz-version-id', 'muPkJbI2Z2VQJKTG3mp7lzMC1wkAxjz'), ('x-amz-request-id', 'tx000001499970b0cb0a0f5-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.240 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.240 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.241 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.241 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:40:32.242 DEBUG:boto:Data: 2023-10-11T14:40:32.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.243 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.243 DEBUG:boto:Params: {} 2023-10-11T14:40:32.244 DEBUG:boto:Token: None 2023-10-11T14:40:32.244 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a2 2023-10-11T14:40:32.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zopVWFyD3rIbxa3KxEiiclP6I34= 2023-10-11T14:40:32.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zopVWFyD3rIbxa3KxEiiclP6I34='} 2023-10-11T14:40:32.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OvxnNPdWFF13s80qzVnde7lQDZETOXt'), ('x-amz-request-id', 'tx0000030908e9482bfe989-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.295 DEBUG:rgw_multi.tests:version3 id=OvxnNPdWFF13s80qzVnde7lQDZETOXt 2023-10-11T14:40:32.295 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.296 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:32.296 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=OvxnNPdWFF13s80qzVnde7lQDZETOXt 2023-10-11T14:40:32.297 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=OvxnNPdWFF13s80qzVnde7lQDZETOXt 2023-10-11T14:40:32.297 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.297 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=OvxnNPdWFF13s80qzVnde7lQDZETOXt 2023-10-11T14:40:32.298 DEBUG:boto:Data: 2023-10-11T14:40:32.298 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.299 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.299 DEBUG:boto:Params: {} 2023-10-11T14:40:32.300 DEBUG:boto:Token: None 2023-10-11T14:40:32.300 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-11/obj-a2?versionId=OvxnNPdWFF13s80qzVnde7lQDZETOXt 2023-10-11T14:40:32.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yy2F7OJ6LnAekieNF9qGRdsWWAc= 2023-10-11T14:40:32.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yy2F7OJ6LnAekieNF9qGRdsWWAc=', 'Content-Length': '0'} 2023-10-11T14:40:32.310 DEBUG:boto:Response headers: [('x-amz-version-id', 'OvxnNPdWFF13s80qzVnde7lQDZETOXt'), ('x-amz-request-id', 'tx00000aa2bc9e798fc8f28-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.311 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:40:32.311 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:40:32.312 DEBUG:boto:Method: HEAD 2023-10-11T14:40:32.312 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:40:32.313 DEBUG:boto:Data: 2023-10-11T14:40:32.313 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.314 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.316 DEBUG:boto:Params: {} 2023-10-11T14:40:32.317 DEBUG:boto:Token: None 2023-10-11T14:40:32.318 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/ 2023-10-11T14:40:32.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hvacdv/CMOiXEBtIufBb/89RJcY= 2023-10-11T14:40:32.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hvacdv/CMOiXEBtIufBb/89RJcY=', 'Content-Length': '0'} 2023-10-11T14:40:32.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000965ae2f2f86a5902-006526b3e0-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.324 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.324 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.325 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.326 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:40:32.326 DEBUG:boto:Data: 2023-10-11T14:40:32.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.331 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.331 DEBUG:boto:Params: {} 2023-10-11T14:40:32.332 DEBUG:boto:Token: None 2023-10-11T14:40:32.333 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a1 2023-10-11T14:40:32.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0st5C4bwxQVI9CsMt7prfS50A0Q= 2023-10-11T14:40:32.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0st5C4bwxQVI9CsMt7prfS50A0Q='} 2023-10-11T14:40:32.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-amz-request-id', 'tx00000606b12d8a999e739-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.345 DEBUG:rgw_multi.tests:version1 id=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:40:32.346 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.347 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.348 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.348 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:40:32.349 DEBUG:boto:Data: 2023-10-11T14:40:32.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.351 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.352 DEBUG:boto:Params: {} 2023-10-11T14:40:32.353 DEBUG:boto:Token: None 2023-10-11T14:40:32.353 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a1 2023-10-11T14:40:32.354 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZbjnpY3ikf3+WGBkeiJDhSnNKp4= 2023-10-11T14:40:32.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZbjnpY3ikf3+WGBkeiJDhSnNKp4='} 2023-10-11T14:40:32.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT'), ('x-amz-request-id', 'tx00000bdfdee26dc479c58-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.399 DEBUG:rgw_multi.tests:version2 id=2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT 2023-10-11T14:40:32.400 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.400 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.400 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT 2023-10-11T14:40:32.401 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT 2023-10-11T14:40:32.401 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.402 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT 2023-10-11T14:40:32.402 DEBUG:boto:Data: 2023-10-11T14:40:32.402 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.403 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.404 DEBUG:boto:Params: {} 2023-10-11T14:40:32.404 DEBUG:boto:Token: None 2023-10-11T14:40:32.404 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a1?versionId=2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT 2023-10-11T14:40:32.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6saD9etyv7wwZNsq6txqWjXA+/k= 2023-10-11T14:40:32.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6saD9etyv7wwZNsq6txqWjXA+/k=', 'Content-Length': '0'} 2023-10-11T14:40:32.414 DEBUG:boto:Response headers: [('x-amz-version-id', '2zD1PZWuCMlOWAH.avP6cI0w-Ee2sHT'), ('x-amz-request-id', 'tx00000e073dc22ab2701d6-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.414 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.415 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.415 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.416 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:40:32.416 DEBUG:boto:Data: 2023-10-11T14:40:32.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.418 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.418 DEBUG:boto:Params: {} 2023-10-11T14:40:32.418 DEBUG:boto:Token: None 2023-10-11T14:40:32.419 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a1 2023-10-11T14:40:32.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lPICmX0LFYIMcTRNoESd0675Wis= 2023-10-11T14:40:32.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lPICmX0LFYIMcTRNoESd0675Wis='} 2023-10-11T14:40:32.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk'), ('x-amz-request-id', 'tx00000e07c8b759f6128dd-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.428 DEBUG:rgw_multi.tests:version3 id=x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk 2023-10-11T14:40:32.429 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.429 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:40:32.430 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk 2023-10-11T14:40:32.430 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk 2023-10-11T14:40:32.430 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.431 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk 2023-10-11T14:40:32.431 DEBUG:boto:Data: 2023-10-11T14:40:32.432 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.432 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:32.432 DEBUG:boto:Port: 8000 2023-10-11T14:40:32.433 DEBUG:boto:Params: {} 2023-10-11T14:40:32.433 DEBUG:boto:Token: None 2023-10-11T14:40:32.434 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a1?versionId=x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk 2023-10-11T14:40:32.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lnjGCfM9vT+TOkiVIsRlTrM+HJE= 2023-10-11T14:40:32.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lnjGCfM9vT+TOkiVIsRlTrM+HJE=', 'Content-Length': '0'} 2023-10-11T14:40:32.443 DEBUG:boto:Response headers: [('x-amz-version-id', 'x93Btnq-G7sZ7vPfm9PXqEp8T9DnKmk'), ('x-amz-request-id', 'tx0000017f7276266992434-006526b3e0-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.444 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:40:32.444 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:40:32.445 DEBUG:boto:Method: HEAD 2023-10-11T14:40:32.446 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:40:32.446 DEBUG:boto:Data: 2023-10-11T14:40:32.447 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.448 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.448 DEBUG:boto:Params: {} 2023-10-11T14:40:32.448 DEBUG:boto:Token: None 2023-10-11T14:40:32.449 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/ 2023-10-11T14:40:32.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hvacdv/CMOiXEBtIufBb/89RJcY= 2023-10-11T14:40:32.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hvacdv/CMOiXEBtIufBb/89RJcY=', 'Content-Length': '0'} 2023-10-11T14:40:32.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e44640e96f89fc48-006526b3e0-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.453 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.453 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.454 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.454 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:40:32.454 DEBUG:boto:Data: 2023-10-11T14:40:32.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.456 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.456 DEBUG:boto:Params: {} 2023-10-11T14:40:32.457 DEBUG:boto:Token: None 2023-10-11T14:40:32.457 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a2 2023-10-11T14:40:32.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SJMhGKLUcumtNxdyIpy1bZvM7tk= 2023-10-11T14:40:32.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SJMhGKLUcumtNxdyIpy1bZvM7tk='} 2023-10-11T14:40:32.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-amz-request-id', 'tx0000050bd08f9123d5c63-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.467 DEBUG:rgw_multi.tests:version1 id=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:40:32.468 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.468 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.469 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.469 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:40:32.469 DEBUG:boto:Data: 2023-10-11T14:40:32.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.471 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.471 DEBUG:boto:Params: {} 2023-10-11T14:40:32.472 DEBUG:boto:Token: None 2023-10-11T14:40:32.472 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a2 2023-10-11T14:40:32.473 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Eq52+QuasJZNJaBPpQO+5TpLu2A= 2023-10-11T14:40:32.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Eq52+QuasJZNJaBPpQO+5TpLu2A='} 2023-10-11T14:40:32.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9AsJlF-SCwJlgdUNexT1.x3g86kcMBn'), ('x-amz-request-id', 'tx000003c0e9b16d5965a8c-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.481 DEBUG:rgw_multi.tests:version2 id=9AsJlF-SCwJlgdUNexT1.x3g86kcMBn 2023-10-11T14:40:32.482 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.482 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.482 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=9AsJlF-SCwJlgdUNexT1.x3g86kcMBn 2023-10-11T14:40:32.483 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=9AsJlF-SCwJlgdUNexT1.x3g86kcMBn 2023-10-11T14:40:32.483 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.484 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=9AsJlF-SCwJlgdUNexT1.x3g86kcMBn 2023-10-11T14:40:32.484 DEBUG:boto:Data: 2023-10-11T14:40:32.484 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.485 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.486 DEBUG:boto:Params: {} 2023-10-11T14:40:32.486 DEBUG:boto:Token: None 2023-10-11T14:40:32.487 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a2?versionId=9AsJlF-SCwJlgdUNexT1.x3g86kcMBn 2023-10-11T14:40:32.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q5koeuQlFZMDJ7mbXi4bSIzesYY= 2023-10-11T14:40:32.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q5koeuQlFZMDJ7mbXi4bSIzesYY=', 'Content-Length': '0'} 2023-10-11T14:40:32.519 DEBUG:boto:Response headers: [('x-amz-version-id', '9AsJlF-SCwJlgdUNexT1.x3g86kcMBn'), ('x-amz-request-id', 'tx000001831cec17b3322e6-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.520 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.520 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.521 DEBUG:boto:Method: PUT 2023-10-11T14:40:32.521 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:40:32.522 DEBUG:boto:Data: 2023-10-11T14:40:32.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:40:32.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.523 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.523 DEBUG:boto:Params: {} 2023-10-11T14:40:32.524 DEBUG:boto:Token: None 2023-10-11T14:40:32.524 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a2 2023-10-11T14:40:32.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7TOKTd9M20+anaBZU3t5VQ93YGc= 2023-10-11T14:40:32.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7TOKTd9M20+anaBZU3t5VQ93YGc='} 2023-10-11T14:40:32.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU'), ('x-amz-request-id', 'tx00000f850eeda31a2ab81-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.571 DEBUG:rgw_multi.tests:version3 id=0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU 2023-10-11T14:40:32.572 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.572 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:40:32.573 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU 2023-10-11T14:40:32.573 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU 2023-10-11T14:40:32.573 DEBUG:boto:Method: DELETE 2023-10-11T14:40:32.574 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU 2023-10-11T14:40:32.574 DEBUG:boto:Data: 2023-10-11T14:40:32.575 DEBUG:boto:Headers: {} 2023-10-11T14:40:32.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:32.575 DEBUG:boto:Port: 8001 2023-10-11T14:40:32.576 DEBUG:boto:Params: {} 2023-10-11T14:40:32.576 DEBUG:boto:Token: None 2023-10-11T14:40:32.577 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:40:32 GMT /dghqvd-12/obj-a2?versionId=0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU 2023-10-11T14:40:32.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MnlRlgXT+4uyxKj3dlyr69GkdDU= 2023-10-11T14:40:32.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MnlRlgXT+4uyxKj3dlyr69GkdDU=', 'Content-Length': '0'} 2023-10-11T14:40:32.586 DEBUG:boto:Response headers: [('x-amz-version-id', '0xDPQdQcHCwVYF1fE3NeX4JRQgZi0AU'), ('x-amz-request-id', 'tx00000269a26d98a713243-006526b3e0-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:32.587 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-11 2023-10-11T14:40:32.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:32.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:32.588 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:40:32.645 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:40:32.646 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:40:32.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:32.676+0000 7f05b5321780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:32.680 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:32.680 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.21.6, ] 2023-10-11T14:40:37.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:37.678+0000 7f05b5321780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:37.681 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:37.681 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.21.6, ] 2023-10-11T14:40:42.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:42.679+0000 7f05b5321780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:42.682 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:42.682 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.21.6, ] 2023-10-11T14:40:47.683 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:47.681+0000 7f05b5321780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:47.684 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-10-11T14:40:47.684 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.21.6, ] 2023-10-11T14:40:52.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:52.682+0000 7f05b5321780 1 bucket sync caught up with source: 2023-10-11T14:40:52.685 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.21.6, ] 2023-10-11T14:40:52.685 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.21.6, ] 2023-10-11T14:40:52.685 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:52.682+0000 7f05b5321780 0 bucket checkpoint complete 2023-10-11T14:40:52.700 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-11 2023-10-11T14:40:52.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:52.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:52.701 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:40:52.763 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:40:52.764 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:40:52.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:52.797+0000 7fd707d67780 1 bucket sync caught up with source: 2023-10-11T14:40:52.800 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000010.31.6, 00000000002.11.11, ] 2023-10-11T14:40:52.800 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.16.11, , , 00000000010.31.6, 00000000002.11.11, ] 2023-10-11T14:40:52.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:52.797+0000 7fd707d67780 0 bucket checkpoint complete 2023-10-11T14:40:52.813 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-11 zones={a2, a1} 2023-10-11T14:40:52.814 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.814 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.815 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.815 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:40:52.815 DEBUG:boto:Data: 2023-10-11T14:40:52.816 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.818 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.819 DEBUG:boto:Params: {} 2023-10-11T14:40:52.819 DEBUG:boto:Token: None 2023-10-11T14:40:52.820 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/ 2023-10-11T14:40:52.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vERgfyu2CM6TWmkaDxaFyVzZ9ok= 2023-10-11T14:40:52.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vERgfyu2CM6TWmkaDxaFyVzZ9ok=', 'Content-Length': '0'} 2023-10-11T14:40:52.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f22a86bc43413286-006526b3f4-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.825 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.826 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.826 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.826 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:40:52.827 DEBUG:boto:Data: 2023-10-11T14:40:52.827 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.828 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.828 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.828 DEBUG:boto:Params: {} 2023-10-11T14:40:52.829 DEBUG:boto:Token: None 2023-10-11T14:40:52.829 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/ 2023-10-11T14:40:52.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vERgfyu2CM6TWmkaDxaFyVzZ9ok= 2023-10-11T14:40:52.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vERgfyu2CM6TWmkaDxaFyVzZ9ok=', 'Content-Length': '0'} 2023-10-11T14:40:52.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d54dd5c2fce5fea6-006526b3f4-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:40:52.834 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.834 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.834 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.835 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.835 DEBUG:boto:Method: GET 2023-10-11T14:40:52.836 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.836 DEBUG:boto:Data: 2023-10-11T14:40:52.837 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.838 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.838 DEBUG:boto:Params: {} 2023-10-11T14:40:52.838 DEBUG:boto:Token: None 2023-10-11T14:40:52.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/?versions 2023-10-11T14:40:52.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI= 2023-10-11T14:40:52.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI=', 'Content-Length': '0'} 2023-10-11T14:40:52.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c305d9675df3e54d-006526b3f4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.843 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.844 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:40:52.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:40:52.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:40:52.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:40:52.846 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.847 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.847 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.847 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.848 DEBUG:boto:Method: GET 2023-10-11T14:40:52.848 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.849 DEBUG:boto:Data: 2023-10-11T14:40:52.849 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.849 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.850 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.850 DEBUG:boto:Params: {} 2023-10-11T14:40:52.851 DEBUG:boto:Token: None 2023-10-11T14:40:52.851 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/?versions 2023-10-11T14:40:52.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI= 2023-10-11T14:40:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI=', 'Content-Length': '0'} 2023-10-11T14:40:52.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b866727cba35d978-006526b3f4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.855 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:40:52.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:40:52.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:40:52.857 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.858 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.858 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.859 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.859 DEBUG:boto:Method: GET 2023-10-11T14:40:52.859 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.860 DEBUG:boto:Data: 2023-10-11T14:40:52.860 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.861 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.861 DEBUG:boto:Params: {} 2023-10-11T14:40:52.862 DEBUG:boto:Token: None 2023-10-11T14:40:52.862 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/?versions 2023-10-11T14:40:52.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI= 2023-10-11T14:40:52.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI=', 'Content-Length': '0'} 2023-10-11T14:40:52.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5d20872690bbfe0-006526b3f4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.866 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.867 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.868 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.868 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.868 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.869 DEBUG:boto:Method: GET 2023-10-11T14:40:52.869 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:40:52.870 DEBUG:boto:Data: 2023-10-11T14:40:52.870 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.871 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.871 DEBUG:boto:Params: {} 2023-10-11T14:40:52.872 DEBUG:boto:Token: None 2023-10-11T14:40:52.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/?versions 2023-10-11T14:40:52.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI= 2023-10-11T14:40:52.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WukMNz2MLH7m78TQBkBEH4C51OI=', 'Content-Length': '0'} 2023-10-11T14:40:52.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513e739af13715bc-006526b3f4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.876 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:40:52.877 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.878 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.878 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.878 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.879 DEBUG:boto:Method: GET 2023-10-11T14:40:52.879 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.880 DEBUG:boto:Data: 2023-10-11T14:40:52.880 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.881 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.881 DEBUG:boto:Params: {} 2023-10-11T14:40:52.882 DEBUG:boto:Token: None 2023-10-11T14:40:52.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.885 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed9a5518a82be86f-006526b3f4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.886 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.887 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.887 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.888 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.888 DEBUG:boto:Method: GET 2023-10-11T14:40:52.889 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.889 DEBUG:boto:Data: 2023-10-11T14:40:52.889 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.890 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.891 DEBUG:boto:Params: {} 2023-10-11T14:40:52.891 DEBUG:boto:Token: None 2023-10-11T14:40:52.892 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a99f8421a8d3eaa-006526b3f4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.896 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.897 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.897 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.898 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.898 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.898 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.899 DEBUG:boto:Data: 2023-10-11T14:40:52.899 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.900 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.900 DEBUG:boto:Params: {} 2023-10-11T14:40:52.901 DEBUG:boto:Token: None 2023-10-11T14:40:52.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TL3MaZds4Nsquhy4CvRDZ50SvIw= 2023-10-11T14:40:52.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TL3MaZds4Nsquhy4CvRDZ50SvIw=', 'Content-Length': '0'} 2023-10-11T14:40:52.904 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000008c1492d7052c0e-006526b3f4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.905 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.905 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.906 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.906 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.907 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.907 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.908 DEBUG:boto:Data: 2023-10-11T14:40:52.908 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.909 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.909 DEBUG:boto:Params: {} 2023-10-11T14:40:52.910 DEBUG:boto:Token: None 2023-10-11T14:40:52.910 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TL3MaZds4Nsquhy4CvRDZ50SvIw= 2023-10-11T14:40:52.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TL3MaZds4Nsquhy4CvRDZ50SvIw=', 'Content-Length': '0'} 2023-10-11T14:40:52.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096b48c52f4016f1e-006526b3f4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:40:52.914 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.915 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.915 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.915 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.916 DEBUG:boto:Method: GET 2023-10-11T14:40:52.916 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.917 DEBUG:boto:Data: 2023-10-11T14:40:52.917 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.918 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.918 DEBUG:boto:Params: {} 2023-10-11T14:40:52.919 DEBUG:boto:Token: None 2023-10-11T14:40:52.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e19a5f15354af49d-006526b3f4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.923 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.923 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.924 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.924 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.924 DEBUG:boto:Method: GET 2023-10-11T14:40:52.925 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.925 DEBUG:boto:Data: 2023-10-11T14:40:52.926 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.927 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.927 DEBUG:boto:Params: {} 2023-10-11T14:40:52.927 DEBUG:boto:Token: None 2023-10-11T14:40:52.928 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046beb56689f5324a-006526b3f4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.933 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.934 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.934 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.935 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.935 DEBUG:boto:Method: GET 2023-10-11T14:40:52.935 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.936 DEBUG:boto:Data: 2023-10-11T14:40:52.936 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.937 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.938 DEBUG:boto:Params: {} 2023-10-11T14:40:52.939 DEBUG:boto:Token: None 2023-10-11T14:40:52.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/ 2023-10-11T14:40:52.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM= 2023-10-11T14:40:52.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM=', 'Content-Length': '0'} 2023-10-11T14:40:52.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3df848c7f738f84-006526b3f4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.943 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.944 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.944 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.945 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.945 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.946 DEBUG:boto:Method: GET 2023-10-11T14:40:52.946 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.947 DEBUG:boto:Data: 2023-10-11T14:40:52.947 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.948 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.948 DEBUG:boto:Params: {} 2023-10-11T14:40:52.949 DEBUG:boto:Token: None 2023-10-11T14:40:52.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/ 2023-10-11T14:40:52.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM= 2023-10-11T14:40:52.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM=', 'Content-Length': '0'} 2023-10-11T14:40:52.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa2924f55df68c4-006526b3f4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.954 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:52.954 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.955 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.955 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.956 DEBUG:boto:Path: /dghqvd-11/dummy 2023-10-11T14:40:52.956 DEBUG:boto:Data: 2023-10-11T14:40:52.956 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.957 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.958 DEBUG:boto:Params: {} 2023-10-11T14:40:52.958 DEBUG:boto:Token: None 2023-10-11T14:40:52.959 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy 2023-10-11T14:40:52.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pzCh03P2P/Fmle+0ezPvUxg/i0Y= 2023-10-11T14:40:52.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pzCh03P2P/Fmle+0ezPvUxg/i0Y=', 'Content-Length': '0'} 2023-10-11T14:40:52.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4badfef5ea016e5-006526b3f4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.963 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.963 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.963 DEBUG:boto:Method: HEAD 2023-10-11T14:40:52.964 DEBUG:boto:Path: /dghqvd-11/dummy 2023-10-11T14:40:52.964 DEBUG:boto:Data: 2023-10-11T14:40:52.965 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.965 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.966 DEBUG:boto:Params: {} 2023-10-11T14:40:52.967 DEBUG:boto:Token: None 2023-10-11T14:40:52.967 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy 2023-10-11T14:40:52.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pzCh03P2P/Fmle+0ezPvUxg/i0Y= 2023-10-11T14:40:52.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pzCh03P2P/Fmle+0ezPvUxg/i0Y=', 'Content-Length': '0'} 2023-10-11T14:40:52.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a5b36cf2b6a111b-006526b3f4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:40:52.972 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.972 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.972 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.973 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.973 DEBUG:boto:Method: GET 2023-10-11T14:40:52.974 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.974 DEBUG:boto:Data: 2023-10-11T14:40:52.975 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.975 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.976 DEBUG:boto:Params: {} 2023-10-11T14:40:52.976 DEBUG:boto:Token: None 2023-10-11T14:40:52.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.977 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e44f9d58bcb27bf-006526b3f4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.980 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:40:52.981 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:40:52.981 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.982 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.982 DEBUG:boto:Method: GET 2023-10-11T14:40:52.983 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.983 DEBUG:boto:Data: 2023-10-11T14:40:52.984 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.984 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:52.985 DEBUG:boto:Port: 8000 2023-10-11T14:40:52.985 DEBUG:boto:Params: {} 2023-10-11T14:40:52.986 DEBUG:boto:Token: None 2023-10-11T14:40:52.986 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:40:52.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI= 2023-10-11T14:40:52.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzdAyE3qmBC7PSscdYNd81FyOiI=', 'Content-Length': '0'} 2023-10-11T14:40:52.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002c206031d2961503-006526b3f4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:52.990 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:52.990 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:52.991 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.991 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.992 DEBUG:boto:Method: GET 2023-10-11T14:40:52.992 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:40:52.992 DEBUG:boto:Data: 2023-10-11T14:40:52.993 DEBUG:boto:Headers: {} 2023-10-11T14:40:52.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:52.994 DEBUG:boto:Port: 8001 2023-10-11T14:40:52.994 DEBUG:boto:Params: {} 2023-10-11T14:40:52.995 DEBUG:boto:Token: None 2023-10-11T14:40:52.995 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:52 GMT /dghqvd-11/ 2023-10-11T14:40:52.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM= 2023-10-11T14:40:52.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TR6pQDWUdjoHpG6IyOxVaw3DSmM=', 'Content-Length': '0'} 2023-10-11T14:40:52.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dbaec4a7ef6851-006526b3f4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.042 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.044 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.044 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.045 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:53.046 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:40:53.047 DEBUG:boto:Method: GET 2023-10-11T14:40:53.047 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:40:53.048 DEBUG:boto:Data: 2023-10-11T14:40:53.049 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.050 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.051 DEBUG:boto:Params: {} 2023-10-11T14:40:53.052 DEBUG:boto:Token: None 2023-10-11T14:40:53.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000174d586ed09cb6a3-006526b3f5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.057 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:40:53.059 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.059 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.060 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.061 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.062 DEBUG:boto:Method: GET 2023-10-11T14:40:53.062 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.063 DEBUG:boto:Data: 2023-10-11T14:40:53.063 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.065 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.066 DEBUG:boto:Params: {} 2023-10-11T14:40:53.067 DEBUG:boto:Token: None 2023-10-11T14:40:53.067 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1e0d401177d22a5-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.072 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.074 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.074 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.076 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.076 DEBUG:boto:Method: GET 2023-10-11T14:40:53.076 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.077 DEBUG:boto:Data: 2023-10-11T14:40:53.077 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.078 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.079 DEBUG:boto:Params: {} 2023-10-11T14:40:53.079 DEBUG:boto:Token: None 2023-10-11T14:40:53.080 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e25b23031146622f-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.085 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.086 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.086 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.087 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.087 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.088 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.088 DEBUG:boto:Data: 2023-10-11T14:40:53.089 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.090 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.090 DEBUG:boto:Params: {} 2023-10-11T14:40:53.091 DEBUG:boto:Token: None 2023-10-11T14:40:53.091 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VdRSLg0ytwaOc7UnibI0Y1nRTPM= 2023-10-11T14:40:53.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VdRSLg0ytwaOc7UnibI0Y1nRTPM=', 'Content-Length': '0'} 2023-10-11T14:40:53.094 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e85704b7b0968de-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.095 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.095 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.096 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.096 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.096 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.097 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.097 DEBUG:boto:Data: 2023-10-11T14:40:53.098 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.099 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.099 DEBUG:boto:Params: {} 2023-10-11T14:40:53.100 DEBUG:boto:Token: None 2023-10-11T14:40:53.100 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VdRSLg0ytwaOc7UnibI0Y1nRTPM= 2023-10-11T14:40:53.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VdRSLg0ytwaOc7UnibI0Y1nRTPM=', 'Content-Length': '0'} 2023-10-11T14:40:53.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077b4e1dca9df23ea-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:40:53.104 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.105 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.105 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.105 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.106 DEBUG:boto:Method: GET 2023-10-11T14:40:53.106 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.107 DEBUG:boto:Data: 2023-10-11T14:40:53.107 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.108 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.109 DEBUG:boto:Params: {} 2023-10-11T14:40:53.109 DEBUG:boto:Token: None 2023-10-11T14:40:53.110 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfbb3e655d115f68-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.158 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.159 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.160 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.160 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.161 DEBUG:boto:Method: GET 2023-10-11T14:40:53.162 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.163 DEBUG:boto:Data: 2023-10-11T14:40:53.164 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.165 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.166 DEBUG:boto:Params: {} 2023-10-11T14:40:53.167 DEBUG:boto:Token: None 2023-10-11T14:40:53.168 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.171 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007896bfc97fb950ae-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.171 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.172 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.172 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.173 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.173 DEBUG:boto:Method: GET 2023-10-11T14:40:53.174 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.174 DEBUG:boto:Data: 2023-10-11T14:40:53.175 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.176 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.176 DEBUG:boto:Params: {} 2023-10-11T14:40:53.177 DEBUG:boto:Token: None 2023-10-11T14:40:53.177 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.178 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42633a23f7a2c83-006526b3f5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.222 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.223 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.224 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.224 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.225 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.225 DEBUG:boto:Method: GET 2023-10-11T14:40:53.225 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.226 DEBUG:boto:Data: 2023-10-11T14:40:53.226 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.227 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.228 DEBUG:boto:Params: {} 2023-10-11T14:40:53.228 DEBUG:boto:Token: None 2023-10-11T14:40:53.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002896656fb1620da8-006526b3f5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.233 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.234 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.234 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.234 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.235 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:40:53.236 DEBUG:boto:Data: 2023-10-11T14:40:53.236 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.237 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.237 DEBUG:boto:Params: {} 2023-10-11T14:40:53.238 DEBUG:boto:Token: None 2023-10-11T14:40:53.238 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1 2023-10-11T14:40:53.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJM9PzZiyW5OBT9StB+0mpSqH9c= 2023-10-11T14:40:53.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJM9PzZiyW5OBT9StB+0mpSqH9c=', 'Content-Length': '0'} 2023-10-11T14:40:53.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000867e59b70870e965-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.243 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.243 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.246 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.246 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:40:53.246 DEBUG:boto:Data: 2023-10-11T14:40:53.247 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.248 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.248 DEBUG:boto:Params: {} 2023-10-11T14:40:53.249 DEBUG:boto:Token: None 2023-10-11T14:40:53.249 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1 2023-10-11T14:40:53.250 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJM9PzZiyW5OBT9StB+0mpSqH9c= 2023-10-11T14:40:53.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJM9PzZiyW5OBT9StB+0mpSqH9c=', 'Content-Length': '0'} 2023-10-11T14:40:53.253 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2f445b73a4cd8e3-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:40:53.254 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.255 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.256 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.257 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.257 DEBUG:boto:Method: GET 2023-10-11T14:40:53.258 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.259 DEBUG:boto:Data: 2023-10-11T14:40:53.259 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.260 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.260 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.261 DEBUG:boto:Params: {} 2023-10-11T14:40:53.262 DEBUG:boto:Token: None 2023-10-11T14:40:53.262 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000548c2644a8197a9e-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.310 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.311 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:40:53.311 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.312 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.312 DEBUG:boto:Method: GET 2023-10-11T14:40:53.313 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.313 DEBUG:boto:Data: 2023-10-11T14:40:53.314 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.315 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.315 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.316 DEBUG:boto:Params: {} 2023-10-11T14:40:53.316 DEBUG:boto:Token: None 2023-10-11T14:40:53.317 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:40:53.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4= 2023-10-11T14:40:53.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYGuZRDYmG8RuUOZarS9+Hs37d4=', 'Content-Length': '0'} 2023-10-11T14:40:53.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007daaec454a51a5bc-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.320 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.321 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.321 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.322 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.322 DEBUG:boto:Method: GET 2023-10-11T14:40:53.323 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.323 DEBUG:boto:Data: 2023-10-11T14:40:53.323 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.324 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.325 DEBUG:boto:Params: {} 2023-10-11T14:40:53.326 DEBUG:boto:Token: None 2023-10-11T14:40:53.326 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908b33800399c2ce-006526b3f5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.370 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.371 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.372 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.372 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.373 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.373 DEBUG:boto:Method: GET 2023-10-11T14:40:53.373 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:40:53.374 DEBUG:boto:Data: 2023-10-11T14:40:53.374 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.375 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.375 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.376 DEBUG:boto:Params: {} 2023-10-11T14:40:53.376 DEBUG:boto:Token: None 2023-10-11T14:40:53.377 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882844e4f6e77066-006526b3f5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.381 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:40:53.382 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.382 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.383 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.383 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.384 DEBUG:boto:Method: GET 2023-10-11T14:40:53.384 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.385 DEBUG:boto:Data: 2023-10-11T14:40:53.385 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.386 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.386 DEBUG:boto:Params: {} 2023-10-11T14:40:53.387 DEBUG:boto:Token: None 2023-10-11T14:40:53.387 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000059d4a8b9c9d363b-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.391 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.391 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.392 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.392 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.393 DEBUG:boto:Method: GET 2023-10-11T14:40:53.393 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.393 DEBUG:boto:Data: 2023-10-11T14:40:53.394 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.395 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.395 DEBUG:boto:Params: {} 2023-10-11T14:40:53.396 DEBUG:boto:Token: None 2023-10-11T14:40:53.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.397 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae7b99fa2f88306d-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.400 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.400 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.401 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.401 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.401 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.402 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.402 DEBUG:boto:Data: 2023-10-11T14:40:53.403 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.403 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.404 DEBUG:boto:Params: {} 2023-10-11T14:40:53.404 DEBUG:boto:Token: None 2023-10-11T14:40:53.405 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r5+6pAOYphqsZBy8EOHY9M+m5hg= 2023-10-11T14:40:53.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r5+6pAOYphqsZBy8EOHY9M+m5hg=', 'Content-Length': '0'} 2023-10-11T14:40:53.407 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000122580a3efab6fab-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.408 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.408 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.409 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.409 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.410 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.410 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.410 DEBUG:boto:Data: 2023-10-11T14:40:53.411 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.411 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.412 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.412 DEBUG:boto:Params: {} 2023-10-11T14:40:53.413 DEBUG:boto:Token: None 2023-10-11T14:40:53.413 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.413 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r5+6pAOYphqsZBy8EOHY9M+m5hg= 2023-10-11T14:40:53.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r5+6pAOYphqsZBy8EOHY9M+m5hg=', 'Content-Length': '0'} 2023-10-11T14:40:53.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc3288dfc2a8fefb-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:40:53.417 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.417 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.418 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.418 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.419 DEBUG:boto:Method: GET 2023-10-11T14:40:53.419 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.420 DEBUG:boto:Data: 2023-10-11T14:40:53.420 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.421 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.421 DEBUG:boto:Params: {} 2023-10-11T14:40:53.422 DEBUG:boto:Token: None 2023-10-11T14:40:53.422 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001879c62ce39caac3-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.426 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.426 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.427 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.427 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.427 DEBUG:boto:Method: GET 2023-10-11T14:40:53.428 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.428 DEBUG:boto:Data: 2023-10-11T14:40:53.429 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.430 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.430 DEBUG:boto:Params: {} 2023-10-11T14:40:53.431 DEBUG:boto:Token: None 2023-10-11T14:40:53.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.434 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000234477c63b161c0-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.474 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.475 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.476 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.476 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.476 DEBUG:boto:Method: GET 2023-10-11T14:40:53.477 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.477 DEBUG:boto:Data: 2023-10-11T14:40:53.478 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.479 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.479 DEBUG:boto:Params: {} 2023-10-11T14:40:53.480 DEBUG:boto:Token: None 2023-10-11T14:40:53.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd6f2a71450a9cf-006526b3f5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.484 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.485 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.485 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.485 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.486 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.486 DEBUG:boto:Method: GET 2023-10-11T14:40:53.487 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.487 DEBUG:boto:Data: 2023-10-11T14:40:53.488 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.488 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.489 DEBUG:boto:Params: {} 2023-10-11T14:40:53.489 DEBUG:boto:Token: None 2023-10-11T14:40:53.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087c9b25876932de8-006526b3f5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.494 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.494 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.495 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.495 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.496 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:40:53.496 DEBUG:boto:Data: 2023-10-11T14:40:53.497 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.498 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.498 DEBUG:boto:Params: {} 2023-10-11T14:40:53.498 DEBUG:boto:Token: None 2023-10-11T14:40:53.499 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2 2023-10-11T14:40:53.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NaIlxOfozaw0OCQbe1ACyM5kbVk= 2023-10-11T14:40:53.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NaIlxOfozaw0OCQbe1ACyM5kbVk=', 'Content-Length': '0'} 2023-10-11T14:40:53.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c9abf363495f14f-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.503 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.503 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.504 DEBUG:boto:Method: HEAD 2023-10-11T14:40:53.504 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:40:53.504 DEBUG:boto:Data: 2023-10-11T14:40:53.505 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.506 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.506 DEBUG:boto:Params: {} 2023-10-11T14:40:53.507 DEBUG:boto:Token: None 2023-10-11T14:40:53.507 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2 2023-10-11T14:40:53.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NaIlxOfozaw0OCQbe1ACyM5kbVk= 2023-10-11T14:40:53.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NaIlxOfozaw0OCQbe1ACyM5kbVk=', 'Content-Length': '0'} 2023-10-11T14:40:53.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000622447cbef3b903b-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:40:53.511 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.511 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.512 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.512 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.513 DEBUG:boto:Method: GET 2023-10-11T14:40:53.513 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.514 DEBUG:boto:Data: 2023-10-11T14:40:53.514 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.515 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.515 DEBUG:boto:Params: {} 2023-10-11T14:40:53.516 DEBUG:boto:Token: None 2023-10-11T14:40:53.516 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac73bed3141de012-006526b3f5-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.566 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.567 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:40:53.567 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.568 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.568 DEBUG:boto:Method: GET 2023-10-11T14:40:53.569 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.569 DEBUG:boto:Data: 2023-10-11T14:40:53.570 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.570 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.571 DEBUG:boto:Params: {} 2023-10-11T14:40:53.571 DEBUG:boto:Token: None 2023-10-11T14:40:53.572 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:40:53.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM= 2023-10-11T14:40:53.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEC6QhVhyuqntfnwr0AFFH2k4fM=', 'Content-Length': '0'} 2023-10-11T14:40:53.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e5830f3668c2841-006526b3f5-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.575 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.576 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.576 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.577 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.577 DEBUG:boto:Method: GET 2023-10-11T14:40:53.577 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.578 DEBUG:boto:Data: 2023-10-11T14:40:53.578 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:40:53.579 DEBUG:boto:Port: 8001 2023-10-11T14:40:53.580 DEBUG:boto:Params: {} 2023-10-11T14:40:53.580 DEBUG:boto:Token: None 2023-10-11T14:40:53.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e9db4af8d0db286-006526b3f5-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.585 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.586 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:40:53.586 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:40:53.586 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.587 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.587 DEBUG:boto:Method: GET 2023-10-11T14:40:53.588 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:40:53.588 DEBUG:boto:Data: 2023-10-11T14:40:53.589 DEBUG:boto:Headers: {} 2023-10-11T14:40:53.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:40:53.589 DEBUG:boto:Port: 8000 2023-10-11T14:40:53.590 DEBUG:boto:Params: {} 2023-10-11T14:40:53.590 DEBUG:boto:Token: None 2023-10-11T14:40:53.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:40:53 GMT /dghqvd-11/ 2023-10-11T14:40:53.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos= 2023-10-11T14:40:53.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:40:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xOBmoyBc2Rhg5+TDjtU2zbcyos=', 'Content-Length': '0'} 2023-10-11T14:40:53.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae56844eaad393f-006526b3f5-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:40:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:40:53.642 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:40:53.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-11 zones={a2, a1} 2023-10-11T14:40:53.644 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-12 2023-10-11T14:40:53.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:53.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:53.645 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:40:53.706 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:40:53.706 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:40:53.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:53.738+0000 7f1bbd4ce780 1 bucket sync caught up with source: 2023-10-11T14:40:53.741 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000002.7.11, 00000000010.23.6, ] 2023-10-11T14:40:53.741 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000002.7.11, 00000000010.23.6, ] 2023-10-11T14:40:53.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:53.738+0000 7f1bbd4ce780 0 bucket checkpoint complete 2023-10-11T14:40:53.753 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-12 2023-10-11T14:40:53.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:53.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:40:53.754 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:40:53.810 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:40:53.811 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:40:53.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:53.842+0000 7f56fc8f4780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:53.844 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000010.31.6, , ] 2023-10-11T14:40:53.845 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:40:58.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:40:58.843+0000 7f56fc8f4780 1 waiting for incremental sync to catch up: 2023-10-11T14:40:58.846 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000010.31.6, , ] 2023-10-11T14:40:58.846 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:41:03.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:03.844+0000 7f56fc8f4780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:03.847 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000010.31.6, , ] 2023-10-11T14:41:03.847 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:41:08.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:08.845+0000 7f56fc8f4780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:08.848 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000010.31.6, , ] 2023-10-11T14:41:08.848 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:41:13.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:13.846+0000 7f56fc8f4780 1 bucket sync caught up with source: 2023-10-11T14:41:13.849 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:41:13.850 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.31.6, 00000000002.20.11, ] 2023-10-11T14:41:13.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:13.846+0000 7f56fc8f4780 0 bucket checkpoint complete 2023-10-11T14:41:13.865 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-12 zones={a2, a1} 2023-10-11T14:41:13.865 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.866 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.866 DEBUG:boto:Method: HEAD 2023-10-11T14:41:13.867 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:13.867 DEBUG:boto:Data: 2023-10-11T14:41:13.868 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.868 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.869 DEBUG:boto:Params: {} 2023-10-11T14:41:13.872 DEBUG:boto:Token: None 2023-10-11T14:41:13.873 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/ 2023-10-11T14:41:13.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+6K68a8K9S2Ub/G2aCVn94cXbJM= 2023-10-11T14:41:13.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+6K68a8K9S2Ub/G2aCVn94cXbJM=', 'Content-Length': '0'} 2023-10-11T14:41:13.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022b5ad9f005d33dc-006526b409-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.877 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.878 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.878 DEBUG:boto:Method: HEAD 2023-10-11T14:41:13.878 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:13.879 DEBUG:boto:Data: 2023-10-11T14:41:13.880 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.881 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.881 DEBUG:boto:Params: {} 2023-10-11T14:41:13.881 DEBUG:boto:Token: None 2023-10-11T14:41:13.882 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/ 2023-10-11T14:41:13.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+6K68a8K9S2Ub/G2aCVn94cXbJM= 2023-10-11T14:41:13.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+6K68a8K9S2Ub/G2aCVn94cXbJM=', 'Content-Length': '0'} 2023-10-11T14:41:13.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad8b6df883c9ba9-006526b409-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:41:13.887 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.888 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.889 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.889 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.890 DEBUG:boto:Method: GET 2023-10-11T14:41:13.890 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.891 DEBUG:boto:Data: 2023-10-11T14:41:13.892 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.893 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.893 DEBUG:boto:Params: {} 2023-10-11T14:41:13.894 DEBUG:boto:Token: None 2023-10-11T14:41:13.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/?versions 2023-10-11T14:41:13.895 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s= 2023-10-11T14:41:13.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s=', 'Content-Length': '0'} 2023-10-11T14:41:13.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c80556b28e7d18-006526b409-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.900 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:13.901 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:13.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:13.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:13.902 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:41:13.903 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.903 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.904 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.904 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.904 DEBUG:boto:Method: GET 2023-10-11T14:41:13.905 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.905 DEBUG:boto:Data: 2023-10-11T14:41:13.906 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.906 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.907 DEBUG:boto:Params: {} 2023-10-11T14:41:13.907 DEBUG:boto:Token: None 2023-10-11T14:41:13.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/?versions 2023-10-11T14:41:13.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s= 2023-10-11T14:41:13.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s=', 'Content-Length': '0'} 2023-10-11T14:41:13.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189aa76dc0620aaf-006526b409-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.913 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:13.914 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:13.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:13.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:13.915 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.916 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.916 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.917 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.917 DEBUG:boto:Method: GET 2023-10-11T14:41:13.917 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.918 DEBUG:boto:Data: 2023-10-11T14:41:13.919 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.920 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.920 DEBUG:boto:Params: {} 2023-10-11T14:41:13.921 DEBUG:boto:Token: None 2023-10-11T14:41:13.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/?versions 2023-10-11T14:41:13.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s= 2023-10-11T14:41:13.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s=', 'Content-Length': '0'} 2023-10-11T14:41:13.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000971accd6649673d-006526b409-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.927 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:13.927 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.928 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.928 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.929 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.929 DEBUG:boto:Method: GET 2023-10-11T14:41:13.930 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:13.930 DEBUG:boto:Data: 2023-10-11T14:41:13.930 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.931 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.931 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.932 DEBUG:boto:Params: {} 2023-10-11T14:41:13.932 DEBUG:boto:Token: None 2023-10-11T14:41:13.933 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/?versions 2023-10-11T14:41:13.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s= 2023-10-11T14:41:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o2A8ig64g/I4lGXgFPTZcQdVd4s=', 'Content-Length': '0'} 2023-10-11T14:41:13.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb4e07baf2c2f3e-006526b409-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.937 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:13.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:13.938 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.938 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.939 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.939 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.940 DEBUG:boto:Method: GET 2023-10-11T14:41:13.940 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.940 DEBUG:boto:Data: 2023-10-11T14:41:13.941 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.942 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.942 DEBUG:boto:Params: {} 2023-10-11T14:41:13.943 DEBUG:boto:Token: None 2023-10-11T14:41:13.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw= 2023-10-11T14:41:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw=', 'Content-Length': '0'} 2023-10-11T14:41:13.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000667c1b9b2201fd2b-006526b409-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.948 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.948 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.948 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.949 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.949 DEBUG:boto:Method: GET 2023-10-11T14:41:13.950 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.950 DEBUG:boto:Data: 2023-10-11T14:41:13.951 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.951 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.952 DEBUG:boto:Params: {} 2023-10-11T14:41:13.952 DEBUG:boto:Token: None 2023-10-11T14:41:13.953 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw= 2023-10-11T14:41:13.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw=', 'Content-Length': '0'} 2023-10-11T14:41:13.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b07a33040b708d5d-006526b409-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.957 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.957 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.958 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.958 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.958 DEBUG:boto:Method: HEAD 2023-10-11T14:41:13.959 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.959 DEBUG:boto:Data: 2023-10-11T14:41:13.960 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.961 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.961 DEBUG:boto:Params: {} 2023-10-11T14:41:13.961 DEBUG:boto:Token: None 2023-10-11T14:41:13.962 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NNQADg4nIu7OR0iG0t9RA9ApHo4= 2023-10-11T14:41:13.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NNQADg4nIu7OR0iG0t9RA9ApHo4=', 'Content-Length': '0'} 2023-10-11T14:41:13.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008ac2d569cf8a90c-006526b409-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.966 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.966 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.967 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.967 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.967 DEBUG:boto:Method: HEAD 2023-10-11T14:41:13.968 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.968 DEBUG:boto:Data: 2023-10-11T14:41:13.969 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.970 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.971 DEBUG:boto:Params: {} 2023-10-11T14:41:13.971 DEBUG:boto:Token: None 2023-10-11T14:41:13.972 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NNQADg4nIu7OR0iG0t9RA9ApHo4= 2023-10-11T14:41:13.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NNQADg4nIu7OR0iG0t9RA9ApHo4=', 'Content-Length': '0'} 2023-10-11T14:41:13.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007946e284d4694e16-006526b409-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:13.976 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.976 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.977 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.977 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.978 DEBUG:boto:Method: GET 2023-10-11T14:41:13.978 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.979 DEBUG:boto:Data: 2023-10-11T14:41:13.979 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:13.980 DEBUG:boto:Port: 8001 2023-10-11T14:41:13.980 DEBUG:boto:Params: {} 2023-10-11T14:41:13.981 DEBUG:boto:Token: None 2023-10-11T14:41:13.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw= 2023-10-11T14:41:13.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw=', 'Content-Length': '0'} 2023-10-11T14:41:13.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000817fe589616fc4f9-006526b409-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.985 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:13.985 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:13.986 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.986 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.987 DEBUG:boto:Method: GET 2023-10-11T14:41:13.987 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.988 DEBUG:boto:Data: 2023-10-11T14:41:13.989 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:13.990 DEBUG:boto:Port: 8000 2023-10-11T14:41:13.990 DEBUG:boto:Params: {} 2023-10-11T14:41:13.991 DEBUG:boto:Token: None 2023-10-11T14:41:13.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:13 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:13.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw= 2023-10-11T14:41:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pv4aQSW9C8DdUm5wbsW9Oo36qYw=', 'Content-Length': '0'} 2023-10-11T14:41:13.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000028ef714edc329979-006526b409-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:13.995 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:13.996 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:13.996 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:13.997 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:13.997 DEBUG:boto:Method: GET 2023-10-11T14:41:13.998 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:13.998 DEBUG:boto:Data: 2023-10-11T14:41:13.999 DEBUG:boto:Headers: {} 2023-10-11T14:41:13.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.000 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.000 DEBUG:boto:Params: {} 2023-10-11T14:41:14.001 DEBUG:boto:Token: None 2023-10-11T14:41:14.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.002 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a10816b5affb78c-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.046 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.047 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.048 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.048 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.048 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.049 DEBUG:boto:Method: GET 2023-10-11T14:41:14.049 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.050 DEBUG:boto:Data: 2023-10-11T14:41:14.050 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.051 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.051 DEBUG:boto:Params: {} 2023-10-11T14:41:14.052 DEBUG:boto:Token: None 2023-10-11T14:41:14.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645fab14f834e245-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.057 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.058 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:14.058 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:14.059 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.059 DEBUG:boto:Path: /dghqvd-12/dummy 2023-10-11T14:41:14.060 DEBUG:boto:Data: 2023-10-11T14:41:14.060 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.060 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.061 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.061 DEBUG:boto:Params: {} 2023-10-11T14:41:14.062 DEBUG:boto:Token: None 2023-10-11T14:41:14.062 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/dummy 2023-10-11T14:41:14.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dhvsmeO4LXvE1Hkf6oLnlR/izAI= 2023-10-11T14:41:14.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dhvsmeO4LXvE1Hkf6oLnlR/izAI=', 'Content-Length': '0'} 2023-10-11T14:41:14.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f52cfd5e64ac6670-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.066 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:14.067 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:14.067 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.068 DEBUG:boto:Path: /dghqvd-12/dummy 2023-10-11T14:41:14.068 DEBUG:boto:Data: 2023-10-11T14:41:14.069 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.070 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.070 DEBUG:boto:Params: {} 2023-10-11T14:41:14.071 DEBUG:boto:Token: None 2023-10-11T14:41:14.071 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/dummy 2023-10-11T14:41:14.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dhvsmeO4LXvE1Hkf6oLnlR/izAI= 2023-10-11T14:41:14.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dhvsmeO4LXvE1Hkf6oLnlR/izAI=', 'Content-Length': '0'} 2023-10-11T14:41:14.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8fa7f42278ba23f-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:14.077 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:14.077 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:14.077 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.078 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.078 DEBUG:boto:Method: GET 2023-10-11T14:41:14.079 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.079 DEBUG:boto:Data: 2023-10-11T14:41:14.080 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.080 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.081 DEBUG:boto:Params: {} 2023-10-11T14:41:14.081 DEBUG:boto:Token: None 2023-10-11T14:41:14.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0hyZK5rvK2sze5feKfip03VHHl0= 2023-10-11T14:41:14.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0hyZK5rvK2sze5feKfip03VHHl0=', 'Content-Length': '0'} 2023-10-11T14:41:14.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013a84fe4cf899d02-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.086 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:14.086 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:14.086 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.087 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.087 DEBUG:boto:Method: GET 2023-10-11T14:41:14.088 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.088 DEBUG:boto:Data: 2023-10-11T14:41:14.089 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.089 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.090 DEBUG:boto:Params: {} 2023-10-11T14:41:14.090 DEBUG:boto:Token: None 2023-10-11T14:41:14.091 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:14.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0hyZK5rvK2sze5feKfip03VHHl0= 2023-10-11T14:41:14.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0hyZK5rvK2sze5feKfip03VHHl0=', 'Content-Length': '0'} 2023-10-11T14:41:14.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e5a1d3f4305082c-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.094 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.095 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.096 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.096 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.097 DEBUG:boto:Method: GET 2023-10-11T14:41:14.098 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.098 DEBUG:boto:Data: 2023-10-11T14:41:14.098 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.100 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.100 DEBUG:boto:Params: {} 2023-10-11T14:41:14.101 DEBUG:boto:Token: None 2023-10-11T14:41:14.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c092843ca42db40-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.106 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.106 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.107 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.108 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.108 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.109 DEBUG:boto:Method: GET 2023-10-11T14:41:14.109 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:14.110 DEBUG:boto:Data: 2023-10-11T14:41:14.110 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.111 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.111 DEBUG:boto:Params: {} 2023-10-11T14:41:14.112 DEBUG:boto:Token: None 2023-10-11T14:41:14.112 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4283972c9098c4a-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.116 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:14.118 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.119 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.119 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.120 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.120 DEBUG:boto:Method: GET 2023-10-11T14:41:14.120 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.121 DEBUG:boto:Data: 2023-10-11T14:41:14.121 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.122 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.123 DEBUG:boto:Params: {} 2023-10-11T14:41:14.123 DEBUG:boto:Token: None 2023-10-11T14:41:14.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e4e2ec72489aad8-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.127 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.128 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.129 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.129 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.130 DEBUG:boto:Method: GET 2023-10-11T14:41:14.130 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.131 DEBUG:boto:Data: 2023-10-11T14:41:14.131 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.132 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.132 DEBUG:boto:Params: {} 2023-10-11T14:41:14.133 DEBUG:boto:Token: None 2023-10-11T14:41:14.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007697be1f6e28d57e-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.178 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.179 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.179 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.180 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.180 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.181 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.181 DEBUG:boto:Data: 2023-10-11T14:41:14.182 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.183 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.183 DEBUG:boto:Params: {} 2023-10-11T14:41:14.184 DEBUG:boto:Token: None 2023-10-11T14:41:14.184 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9BKjzDHhHXTJPuam/u6ZQ6j40Dk= 2023-10-11T14:41:14.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9BKjzDHhHXTJPuam/u6ZQ6j40Dk=', 'Content-Length': '0'} 2023-10-11T14:41:14.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a4f345e62b23dec-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.188 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.188 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.189 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.190 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.190 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.190 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.191 DEBUG:boto:Data: 2023-10-11T14:41:14.192 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.192 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.193 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.194 DEBUG:boto:Params: {} 2023-10-11T14:41:14.194 DEBUG:boto:Token: None 2023-10-11T14:41:14.195 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9BKjzDHhHXTJPuam/u6ZQ6j40Dk= 2023-10-11T14:41:14.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9BKjzDHhHXTJPuam/u6ZQ6j40Dk=', 'Content-Length': '0'} 2023-10-11T14:41:14.197 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d91ceef510a7e102-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:14.199 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.199 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.199 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.200 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.200 DEBUG:boto:Method: GET 2023-10-11T14:41:14.201 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.201 DEBUG:boto:Data: 2023-10-11T14:41:14.202 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.202 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.203 DEBUG:boto:Params: {} 2023-10-11T14:41:14.207 DEBUG:boto:Token: None 2023-10-11T14:41:14.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000872eb0e0f8049a3e-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.254 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.255 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.255 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.256 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.256 DEBUG:boto:Method: GET 2023-10-11T14:41:14.257 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.257 DEBUG:boto:Data: 2023-10-11T14:41:14.258 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.258 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.258 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.259 DEBUG:boto:Params: {} 2023-10-11T14:41:14.259 DEBUG:boto:Token: None 2023-10-11T14:41:14.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2c2810f9446f11e-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.265 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.265 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.267 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.268 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.268 DEBUG:boto:Method: GET 2023-10-11T14:41:14.268 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.269 DEBUG:boto:Data: 2023-10-11T14:41:14.269 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.270 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.270 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.271 DEBUG:boto:Params: {} 2023-10-11T14:41:14.271 DEBUG:boto:Token: None 2023-10-11T14:41:14.272 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004239c3f8b3fc19db-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.318 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.319 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.320 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.320 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.320 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.321 DEBUG:boto:Method: GET 2023-10-11T14:41:14.321 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.322 DEBUG:boto:Data: 2023-10-11T14:41:14.322 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.323 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.323 DEBUG:boto:Params: {} 2023-10-11T14:41:14.324 DEBUG:boto:Token: None 2023-10-11T14:41:14.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de64efa96fd4781a-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.329 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.330 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.330 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.331 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.331 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:41:14.332 DEBUG:boto:Data: 2023-10-11T14:41:14.333 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.334 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.335 DEBUG:boto:Params: {} 2023-10-11T14:41:14.335 DEBUG:boto:Token: None 2023-10-11T14:41:14.336 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1 2023-10-11T14:41:14.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rQF/7lg4HyXOkqznejZrGqU9YbE= 2023-10-11T14:41:14.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rQF/7lg4HyXOkqznejZrGqU9YbE=', 'Content-Length': '0'} 2023-10-11T14:41:14.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019a375c40e83b29a-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.340 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.341 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.341 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.342 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:41:14.342 DEBUG:boto:Data: 2023-10-11T14:41:14.342 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.343 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.344 DEBUG:boto:Params: {} 2023-10-11T14:41:14.344 DEBUG:boto:Token: None 2023-10-11T14:41:14.344 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1 2023-10-11T14:41:14.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rQF/7lg4HyXOkqznejZrGqU9YbE= 2023-10-11T14:41:14.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rQF/7lg4HyXOkqznejZrGqU9YbE=', 'Content-Length': '0'} 2023-10-11T14:41:14.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006514f675722f0e64-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:14.349 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.349 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.350 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.350 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.351 DEBUG:boto:Method: GET 2023-10-11T14:41:14.351 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.352 DEBUG:boto:Data: 2023-10-11T14:41:14.352 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.353 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.354 DEBUG:boto:Params: {} 2023-10-11T14:41:14.355 DEBUG:boto:Token: None 2023-10-11T14:41:14.355 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022d2c6e56cb55435-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.402 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.403 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.403 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.404 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.404 DEBUG:boto:Method: GET 2023-10-11T14:41:14.405 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.405 DEBUG:boto:Data: 2023-10-11T14:41:14.406 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.406 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.407 DEBUG:boto:Params: {} 2023-10-11T14:41:14.407 DEBUG:boto:Token: None 2023-10-11T14:41:14.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:14.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4= 2023-10-11T14:41:14.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/kD3nZse/4L3izosKI1IDKylx4=', 'Content-Length': '0'} 2023-10-11T14:41:14.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a7b79b016f84466-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.412 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.412 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.413 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.413 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.414 DEBUG:boto:Method: GET 2023-10-11T14:41:14.414 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.414 DEBUG:boto:Data: 2023-10-11T14:41:14.415 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.416 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.416 DEBUG:boto:Params: {} 2023-10-11T14:41:14.417 DEBUG:boto:Token: None 2023-10-11T14:41:14.417 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000225b50b3a8b1d4ec-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.462 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.463 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.464 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.464 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.465 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.465 DEBUG:boto:Method: GET 2023-10-11T14:41:14.465 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:14.466 DEBUG:boto:Data: 2023-10-11T14:41:14.466 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.467 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.467 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.468 DEBUG:boto:Params: {} 2023-10-11T14:41:14.468 DEBUG:boto:Token: None 2023-10-11T14:41:14.469 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.469 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f50551d08d8b62-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.473 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:14.474 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.475 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.475 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.475 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.476 DEBUG:boto:Method: GET 2023-10-11T14:41:14.476 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.477 DEBUG:boto:Data: 2023-10-11T14:41:14.477 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.478 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.478 DEBUG:boto:Params: {} 2023-10-11T14:41:14.479 DEBUG:boto:Token: None 2023-10-11T14:41:14.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4925dd71856eb8e-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.483 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.484 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.484 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.485 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.485 DEBUG:boto:Method: GET 2023-10-11T14:41:14.486 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.486 DEBUG:boto:Data: 2023-10-11T14:41:14.486 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.487 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.488 DEBUG:boto:Params: {} 2023-10-11T14:41:14.488 DEBUG:boto:Token: None 2023-10-11T14:41:14.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.492 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ca471d70dcba4d4-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.492 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.493 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.493 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.494 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.494 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.495 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.495 DEBUG:boto:Data: 2023-10-11T14:41:14.495 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.496 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.497 DEBUG:boto:Params: {} 2023-10-11T14:41:14.497 DEBUG:boto:Token: None 2023-10-11T14:41:14.498 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yiRZSUBTZNwZg0lixHG3C1lbQ18= 2023-10-11T14:41:14.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yiRZSUBTZNwZg0lixHG3C1lbQ18=', 'Content-Length': '0'} 2023-10-11T14:41:14.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027588ff53144a779-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.501 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.501 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.502 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.502 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.503 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.503 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.503 DEBUG:boto:Data: 2023-10-11T14:41:14.504 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.505 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.505 DEBUG:boto:Params: {} 2023-10-11T14:41:14.506 DEBUG:boto:Token: None 2023-10-11T14:41:14.506 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yiRZSUBTZNwZg0lixHG3C1lbQ18= 2023-10-11T14:41:14.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yiRZSUBTZNwZg0lixHG3C1lbQ18=', 'Content-Length': '0'} 2023-10-11T14:41:14.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3ff2e4d7c77c04f-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:14.510 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.511 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.511 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.511 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.512 DEBUG:boto:Method: GET 2023-10-11T14:41:14.512 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.513 DEBUG:boto:Data: 2023-10-11T14:41:14.513 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.514 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.514 DEBUG:boto:Params: {} 2023-10-11T14:41:14.515 DEBUG:boto:Token: None 2023-10-11T14:41:14.515 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.518 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f13a650e9c9b6b43-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.519 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.519 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.520 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.520 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.521 DEBUG:boto:Method: GET 2023-10-11T14:41:14.521 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.521 DEBUG:boto:Data: 2023-10-11T14:41:14.522 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.523 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.523 DEBUG:boto:Params: {} 2023-10-11T14:41:14.524 DEBUG:boto:Token: None 2023-10-11T14:41:14.524 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf4db8f2321eb114-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.570 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.571 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.571 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.572 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.572 DEBUG:boto:Method: GET 2023-10-11T14:41:14.573 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.573 DEBUG:boto:Data: 2023-10-11T14:41:14.573 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.574 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.577 DEBUG:boto:Params: {} 2023-10-11T14:41:14.577 DEBUG:boto:Token: None 2023-10-11T14:41:14.578 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.578 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd10fc0a97e5ef6d-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.582 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.583 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.584 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.584 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.585 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.585 DEBUG:boto:Method: GET 2023-10-11T14:41:14.585 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.586 DEBUG:boto:Data: 2023-10-11T14:41:14.587 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.588 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.588 DEBUG:boto:Params: {} 2023-10-11T14:41:14.589 DEBUG:boto:Token: None 2023-10-11T14:41:14.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1b33d9d565ea85-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.594 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.594 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.595 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.595 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.596 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:41:14.596 DEBUG:boto:Data: 2023-10-11T14:41:14.596 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.597 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.598 DEBUG:boto:Params: {} 2023-10-11T14:41:14.598 DEBUG:boto:Token: None 2023-10-11T14:41:14.599 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2 2023-10-11T14:41:14.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bjPYWejzHQUh+dmIhdecs6aMecA= 2023-10-11T14:41:14.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bjPYWejzHQUh+dmIhdecs6aMecA=', 'Content-Length': '0'} 2023-10-11T14:41:14.603 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5727928471a2aa3-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.603 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.604 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.604 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.605 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:41:14.605 DEBUG:boto:Data: 2023-10-11T14:41:14.606 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.606 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.606 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.607 DEBUG:boto:Params: {} 2023-10-11T14:41:14.607 DEBUG:boto:Token: None 2023-10-11T14:41:14.608 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2 2023-10-11T14:41:14.608 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bjPYWejzHQUh+dmIhdecs6aMecA= 2023-10-11T14:41:14.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bjPYWejzHQUh+dmIhdecs6aMecA=', 'Content-Length': '0'} 2023-10-11T14:41:14.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000906e7849c77f0315-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:14.612 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.613 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.613 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.614 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.614 DEBUG:boto:Method: GET 2023-10-11T14:41:14.615 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.615 DEBUG:boto:Data: 2023-10-11T14:41:14.616 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.616 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.617 DEBUG:boto:Params: {} 2023-10-11T14:41:14.617 DEBUG:boto:Token: None 2023-10-11T14:41:14.618 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002551d9c3ac1a722-006526b40a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.665 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.666 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.666 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.667 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.667 DEBUG:boto:Method: GET 2023-10-11T14:41:14.668 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.668 DEBUG:boto:Data: 2023-10-11T14:41:14.669 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.669 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.669 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.670 DEBUG:boto:Params: {} 2023-10-11T14:41:14.670 DEBUG:boto:Token: None 2023-10-11T14:41:14.671 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:14.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E= 2023-10-11T14:41:14.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+0Bayo8/rqQXK5fiTDPWv0Bp5E=', 'Content-Length': '0'} 2023-10-11T14:41:14.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8afadeec8cc5eb7-006526b40a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.722 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.723 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.723 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.724 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.724 DEBUG:boto:Method: GET 2023-10-11T14:41:14.725 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.725 DEBUG:boto:Data: 2023-10-11T14:41:14.726 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.726 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.726 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.727 DEBUG:boto:Params: {} 2023-10-11T14:41:14.727 DEBUG:boto:Token: None 2023-10-11T14:41:14.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068febea535333ea2-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.732 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.733 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.733 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.734 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.734 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.735 DEBUG:boto:Method: GET 2023-10-11T14:41:14.735 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:14.735 DEBUG:boto:Data: 2023-10-11T14:41:14.736 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.737 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.737 DEBUG:boto:Params: {} 2023-10-11T14:41:14.738 DEBUG:boto:Token: None 2023-10-11T14:41:14.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs= 2023-10-11T14:41:14.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:79hj62Lla5K0IVFdVUiz5cPrCUs=', 'Content-Length': '0'} 2023-10-11T14:41:14.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b20cb709f9ade7-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.786 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.788 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-12 zones={a2, a1} 2023-10-11T14:41:14.788 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:14.789 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:14.790 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.790 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:41:14.791 DEBUG:boto:Data: 2023-10-11T14:41:14.791 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.793 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.794 DEBUG:boto:Params: {} 2023-10-11T14:41:14.795 DEBUG:boto:Token: None 2023-10-11T14:41:14.795 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-11/ 2023-10-11T14:41:14.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qULopZx00vmqAg4wfK3vb/R7Prs= 2023-10-11T14:41:14.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qULopZx00vmqAg4wfK3vb/R7Prs=', 'Content-Length': '0'} 2023-10-11T14:41:14.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000130507c6126ee664-006526b40a-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.800 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:14.801 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:14.801 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.802 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.803 DEBUG:boto:Method: GET 2023-10-11T14:41:14.804 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.804 DEBUG:boto:Data: 2023-10-11T14:41:14.805 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.806 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.807 DEBUG:boto:Params: {} 2023-10-11T14:41:14.808 DEBUG:boto:Token: None 2023-10-11T14:41:14.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-11/?versions 2023-10-11T14:41:14.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wvb8ytz9SKOF+5i7y19ONvdHIMM= 2023-10-11T14:41:14.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wvb8ytz9SKOF+5i7y19ONvdHIMM=', 'Content-Length': '0'} 2023-10-11T14:41:14.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf1604fa2129cb7-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.814 DEBUG:boto:b'dghqvd-11obj-a1999falseobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.815 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:14.816 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:14.817 DEBUG:boto:path=/dghqvd-11/obj-a1?acl 2023-10-11T14:41:14.817 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?acl 2023-10-11T14:41:14.818 DEBUG:boto:Method: PUT 2023-10-11T14:41:14.819 DEBUG:boto:Path: /dghqvd-11/obj-a1?acl 2023-10-11T14:41:14.819 DEBUG:boto:Data: 2023-10-11T14:41:14.819 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:41:14.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.820 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.821 DEBUG:boto:Params: {} 2023-10-11T14:41:14.822 DEBUG:boto:Token: None 2023-10-11T14:41:14.822 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:14 GMT x-amz-acl:public-read /dghqvd-11/obj-a1?acl 2023-10-11T14:41:14.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fR9I0V1v/naPVawcZG9CdQ/DQ3k= 2023-10-11T14:41:14.823 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fR9I0V1v/naPVawcZG9CdQ/DQ3k=', 'Content-Length': '0'} 2023-10-11T14:41:14.830 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006c31d43445ffaa40-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.831 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:14.832 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:14.833 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.834 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:41:14.834 DEBUG:boto:Data: 2023-10-11T14:41:14.835 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.837 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.837 DEBUG:boto:Params: {} 2023-10-11T14:41:14.838 DEBUG:boto:Token: None 2023-10-11T14:41:14.839 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-11/ 2023-10-11T14:41:14.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qULopZx00vmqAg4wfK3vb/R7Prs= 2023-10-11T14:41:14.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qULopZx00vmqAg4wfK3vb/R7Prs=', 'Content-Length': '0'} 2023-10-11T14:41:14.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000351c17f43a61c3f3-006526b40a-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.844 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:14.845 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:14.845 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.846 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.846 DEBUG:boto:Method: GET 2023-10-11T14:41:14.846 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.847 DEBUG:boto:Data: 2023-10-11T14:41:14.847 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.848 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.848 DEBUG:boto:Params: {} 2023-10-11T14:41:14.849 DEBUG:boto:Token: None 2023-10-11T14:41:14.849 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-11/?versions 2023-10-11T14:41:14.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wvb8ytz9SKOF+5i7y19ONvdHIMM= 2023-10-11T14:41:14.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wvb8ytz9SKOF+5i7y19ONvdHIMM=', 'Content-Length': '0'} 2023-10-11T14:41:14.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2fd681aa9c5d6b-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.853 DEBUG:boto:b'dghqvd-11obj-a2999falseobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.854 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:14.855 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:14.855 DEBUG:boto:path=/dghqvd-11/obj-a2?acl 2023-10-11T14:41:14.856 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?acl 2023-10-11T14:41:14.856 DEBUG:boto:Method: PUT 2023-10-11T14:41:14.857 DEBUG:boto:Path: /dghqvd-11/obj-a2?acl 2023-10-11T14:41:14.857 DEBUG:boto:Data: 2023-10-11T14:41:14.857 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:41:14.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.858 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.859 DEBUG:boto:Params: {} 2023-10-11T14:41:14.859 DEBUG:boto:Token: None 2023-10-11T14:41:14.859 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:14 GMT x-amz-acl:public-read /dghqvd-11/obj-a2?acl 2023-10-11T14:41:14.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iHKrEa2WjykVSEbBN+WYkQuvkPY= 2023-10-11T14:41:14.860 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iHKrEa2WjykVSEbBN+WYkQuvkPY=', 'Content-Length': '0'} 2023-10-11T14:41:14.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000076357b5a8996b927-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.868 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.869 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.870 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.870 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:14.871 DEBUG:boto:Data: 2023-10-11T14:41:14.872 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.873 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.874 DEBUG:boto:Params: {} 2023-10-11T14:41:14.875 DEBUG:boto:Token: None 2023-10-11T14:41:14.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r6UVEZcY2Xz71u29+yYZVH8zgfo= 2023-10-11T14:41:14.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r6UVEZcY2Xz71u29+yYZVH8zgfo=', 'Content-Length': '0'} 2023-10-11T14:41:14.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4a0bfedb8b1af63-006526b40a-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.881 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.881 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.882 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.882 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.883 DEBUG:boto:Method: GET 2023-10-11T14:41:14.883 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999&prefix=obj-a1 2023-10-11T14:41:14.884 DEBUG:boto:Data: 2023-10-11T14:41:14.884 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.885 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.885 DEBUG:boto:Params: {} 2023-10-11T14:41:14.886 DEBUG:boto:Token: None 2023-10-11T14:41:14.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/?versions 2023-10-11T14:41:14.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ddLnFCOAdm3yfzOOQslQJOOmB9w= 2023-10-11T14:41:14.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ddLnFCOAdm3yfzOOQslQJOOmB9w=', 'Content-Length': '0'} 2023-10-11T14:41:14.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009724d20d882406fa-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.891 DEBUG:boto:b'dghqvd-12obj-a1999falseobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.892 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.893 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:14.893 DEBUG:boto:path=/dghqvd-12/obj-a1?acl 2023-10-11T14:41:14.894 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?acl 2023-10-11T14:41:14.895 DEBUG:boto:Method: PUT 2023-10-11T14:41:14.895 DEBUG:boto:Path: /dghqvd-12/obj-a1?acl 2023-10-11T14:41:14.896 DEBUG:boto:Data: 2023-10-11T14:41:14.897 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:41:14.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:14.898 DEBUG:boto:Port: 8000 2023-10-11T14:41:14.899 DEBUG:boto:Params: {} 2023-10-11T14:41:14.900 DEBUG:boto:Token: None 2023-10-11T14:41:14.900 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:14 GMT x-amz-acl:public-read /dghqvd-12/obj-a1?acl 2023-10-11T14:41:14.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E9EVI/POfvP4CiaD5YH4hrfSqYc= 2023-10-11T14:41:14.902 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E9EVI/POfvP4CiaD5YH4hrfSqYc=', 'Content-Length': '0'} 2023-10-11T14:41:14.909 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eeb2b66973047df9-006526b40a-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.910 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.910 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.911 DEBUG:boto:Method: HEAD 2023-10-11T14:41:14.911 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:14.911 DEBUG:boto:Data: 2023-10-11T14:41:14.912 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.913 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.913 DEBUG:boto:Params: {} 2023-10-11T14:41:14.914 DEBUG:boto:Token: None 2023-10-11T14:41:14.914 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/ 2023-10-11T14:41:14.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r6UVEZcY2Xz71u29+yYZVH8zgfo= 2023-10-11T14:41:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r6UVEZcY2Xz71u29+yYZVH8zgfo=', 'Content-Length': '0'} 2023-10-11T14:41:14.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b54650cb61c5172-006526b40a-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.919 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:14.919 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:14.919 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.920 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.920 DEBUG:boto:Method: GET 2023-10-11T14:41:14.921 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999&prefix=obj-a2 2023-10-11T14:41:14.921 DEBUG:boto:Data: 2023-10-11T14:41:14.922 DEBUG:boto:Headers: {} 2023-10-11T14:41:14.922 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.923 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.923 DEBUG:boto:Params: {} 2023-10-11T14:41:14.924 DEBUG:boto:Token: None 2023-10-11T14:41:14.924 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:14 GMT /dghqvd-12/?versions 2023-10-11T14:41:14.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ddLnFCOAdm3yfzOOQslQJOOmB9w= 2023-10-11T14:41:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ddLnFCOAdm3yfzOOQslQJOOmB9w=', 'Content-Length': '0'} 2023-10-11T14:41:14.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069194e8e54851317-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.929 DEBUG:boto:b'dghqvd-12obj-a2999falseobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:14.929 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.930 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:14.930 DEBUG:boto:path=/dghqvd-12/obj-a2?acl 2023-10-11T14:41:14.931 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?acl 2023-10-11T14:41:14.931 DEBUG:boto:Method: PUT 2023-10-11T14:41:14.931 DEBUG:boto:Path: /dghqvd-12/obj-a2?acl 2023-10-11T14:41:14.932 DEBUG:boto:Data: 2023-10-11T14:41:14.932 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:41:14.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:14.933 DEBUG:boto:Port: 8001 2023-10-11T14:41:14.934 DEBUG:boto:Params: {} 2023-10-11T14:41:14.934 DEBUG:boto:Token: None 2023-10-11T14:41:14.934 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:14 GMT x-amz-acl:public-read /dghqvd-12/obj-a2?acl 2023-10-11T14:41:14.935 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XbvcCdZ7khbOaTWrRtHXBeOyc9E= 2023-10-11T14:41:14.935 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XbvcCdZ7khbOaTWrRtHXBeOyc9E=', 'Content-Length': '0'} 2023-10-11T14:41:14.945 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000629dcdd341f36da-006526b40a-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:14.946 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-11 2023-10-11T14:41:14.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:14.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:14.947 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:15.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:15.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:15.049 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:15.047+0000 7f19f2a70780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:15.049 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.21.6, ] 2023-10-11T14:41:15.050 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:20.051 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:20.048+0000 7f19f2a70780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:20.051 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.21.6, ] 2023-10-11T14:41:20.052 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:25.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:25.048+0000 7f19f2a70780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:25.052 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.21.6, ] 2023-10-11T14:41:25.053 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:30.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:30.050+0000 7f19f2a70780 1 waiting for incremental sync to catch up: 2023-10-11T14:41:30.054 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000010.21.6, ] 2023-10-11T14:41:30.054 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:35.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:35.051+0000 7f19f2a70780 1 bucket sync caught up with source: 2023-10-11T14:41:35.055 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:35.055 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.6.11, 00000000011.24.6, ] 2023-10-11T14:41:35.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:35.051+0000 7f19f2a70780 0 bucket checkpoint complete 2023-10-11T14:41:35.071 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-11 2023-10-11T14:41:35.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:35.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:35.072 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:35.136 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:35.137 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:35.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:35.169+0000 7fe044727780 1 bucket sync caught up with source: 2023-10-11T14:41:35.173 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.16.11, , , 00000000011.34.6, 00000000002.11.11, ] 2023-10-11T14:41:35.173 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.16.11, , , 00000000011.34.6, 00000000002.11.11, ] 2023-10-11T14:41:35.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:35.169+0000 7fe044727780 0 bucket checkpoint complete 2023-10-11T14:41:35.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-11 zones={a2, a1} 2023-10-11T14:41:35.186 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.186 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.186 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.187 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:41:35.187 DEBUG:boto:Data: 2023-10-11T14:41:35.188 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.188 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.189 DEBUG:boto:Params: {} 2023-10-11T14:41:35.190 DEBUG:boto:Token: None 2023-10-11T14:41:35.190 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z9DmdY67eHEBzBIryeninJE/FbE= 2023-10-11T14:41:35.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z9DmdY67eHEBzBIryeninJE/FbE=', 'Content-Length': '0'} 2023-10-11T14:41:35.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a05eb096cfbec0b0-006526b41f-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.198 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.198 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.198 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.199 DEBUG:boto:Path: /dghqvd-11/ 2023-10-11T14:41:35.199 DEBUG:boto:Data: 2023-10-11T14:41:35.200 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.200 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.200 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.201 DEBUG:boto:Params: {} 2023-10-11T14:41:35.201 DEBUG:boto:Token: None 2023-10-11T14:41:35.202 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.202 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z9DmdY67eHEBzBIryeninJE/FbE= 2023-10-11T14:41:35.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z9DmdY67eHEBzBIryeninJE/FbE=', 'Content-Length': '0'} 2023-10-11T14:41:35.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9946c79d86d38ef-006526b41f-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.205 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:41:35.206 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.206 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.207 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.207 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.208 DEBUG:boto:Method: GET 2023-10-11T14:41:35.208 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.208 DEBUG:boto:Data: 2023-10-11T14:41:35.209 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.210 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.210 DEBUG:boto:Params: {} 2023-10-11T14:41:35.211 DEBUG:boto:Token: None 2023-10-11T14:41:35.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/?versions 2023-10-11T14:41:35.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8= 2023-10-11T14:41:35.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8=', 'Content-Length': '0'} 2023-10-11T14:41:35.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032790b3f87bf75ec-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.215 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.216 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:35.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:35.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:35.217 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:41:35.218 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.218 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.218 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.219 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.219 DEBUG:boto:Method: GET 2023-10-11T14:41:35.220 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.220 DEBUG:boto:Data: 2023-10-11T14:41:35.221 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.222 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.222 DEBUG:boto:Params: {} 2023-10-11T14:41:35.222 DEBUG:boto:Token: None 2023-10-11T14:41:35.223 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/?versions 2023-10-11T14:41:35.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8= 2023-10-11T14:41:35.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8=', 'Content-Length': '0'} 2023-10-11T14:41:35.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004663ca96e9852372-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.227 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:35.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:35.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:35.230 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.231 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.231 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.232 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.232 DEBUG:boto:Method: GET 2023-10-11T14:41:35.233 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.233 DEBUG:boto:Data: 2023-10-11T14:41:35.234 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.235 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.235 DEBUG:boto:Params: {} 2023-10-11T14:41:35.235 DEBUG:boto:Token: None 2023-10-11T14:41:35.236 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/?versions 2023-10-11T14:41:35.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8= 2023-10-11T14:41:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8=', 'Content-Length': '0'} 2023-10-11T14:41:35.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007371372a4df67071-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.240 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.241 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.241 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.242 DEBUG:boto:path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.242 DEBUG:boto:auth_path=/dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.242 DEBUG:boto:Method: GET 2023-10-11T14:41:35.243 DEBUG:boto:Path: /dghqvd-11/?versions&max-keys=999 2023-10-11T14:41:35.243 DEBUG:boto:Data: 2023-10-11T14:41:35.244 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.244 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.245 DEBUG:boto:Params: {} 2023-10-11T14:41:35.245 DEBUG:boto:Token: None 2023-10-11T14:41:35.246 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/?versions 2023-10-11T14:41:35.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8= 2023-10-11T14:41:35.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6yPKKKO5uawlFbF2Hpc/MblXGJ8=', 'Content-Length': '0'} 2023-10-11T14:41:35.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d98b4d792b0e993-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.249 DEBUG:boto:b'dghqvd-11999falsedummyalMaXFwfL-O8TRkqV1b4q1hFWF6.ysXtrue2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a17MXlRLzDnNsDwcdWqn6Q8e4OQMfApYmtrue2023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a24rWwWcCI-wbpAWklU32Du67A8cgYeSftrue2023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:35.251 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.251 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.251 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.252 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.252 DEBUG:boto:Method: GET 2023-10-11T14:41:35.253 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.253 DEBUG:boto:Data: 2023-10-11T14:41:35.254 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.254 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.255 DEBUG:boto:Params: {} 2023-10-11T14:41:35.255 DEBUG:boto:Token: None 2023-10-11T14:41:35.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdff525292f11512-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.262 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.263 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.263 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.264 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.264 DEBUG:boto:Method: GET 2023-10-11T14:41:35.265 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.265 DEBUG:boto:Data: 2023-10-11T14:41:35.265 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.266 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.267 DEBUG:boto:Params: {} 2023-10-11T14:41:35.267 DEBUG:boto:Token: None 2023-10-11T14:41:35.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5e4adb7f902168a-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.274 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.275 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.276 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.276 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.277 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.278 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.278 DEBUG:boto:Data: 2023-10-11T14:41:35.279 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.279 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.279 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.280 DEBUG:boto:Params: {} 2023-10-11T14:41:35.280 DEBUG:boto:Token: None 2023-10-11T14:41:35.281 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.281 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sdRpJzkH0wh4TgChmW1PXJ23TH8= 2023-10-11T14:41:35.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sdRpJzkH0wh4TgChmW1PXJ23TH8=', 'Content-Length': '0'} 2023-10-11T14:41:35.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c25fead944f06d2-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.285 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.285 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.286 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.286 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.287 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.287 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.287 DEBUG:boto:Data: 2023-10-11T14:41:35.288 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.288 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.289 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.289 DEBUG:boto:Params: {} 2023-10-11T14:41:35.290 DEBUG:boto:Token: None 2023-10-11T14:41:35.290 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sdRpJzkH0wh4TgChmW1PXJ23TH8= 2023-10-11T14:41:35.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sdRpJzkH0wh4TgChmW1PXJ23TH8=', 'Content-Length': '0'} 2023-10-11T14:41:35.293 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a0851c2483e381e4-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:35.294 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.295 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.295 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.295 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.296 DEBUG:boto:Method: GET 2023-10-11T14:41:35.296 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.297 DEBUG:boto:Data: 2023-10-11T14:41:35.297 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.298 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.298 DEBUG:boto:Params: {} 2023-10-11T14:41:35.299 DEBUG:boto:Token: None 2023-10-11T14:41:35.299 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079bd75c2240856d7-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.303 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.304 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.306 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.307 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.307 DEBUG:boto:Method: GET 2023-10-11T14:41:35.308 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.308 DEBUG:boto:Data: 2023-10-11T14:41:35.308 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.309 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.310 DEBUG:boto:Params: {} 2023-10-11T14:41:35.310 DEBUG:boto:Token: None 2023-10-11T14:41:35.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.313 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a610d52d1df65d89-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.314 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.315 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.315 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.316 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.317 DEBUG:boto:Method: GET 2023-10-11T14:41:35.317 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.317 DEBUG:boto:Data: 2023-10-11T14:41:35.318 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.319 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.319 DEBUG:boto:Params: {} 2023-10-11T14:41:35.320 DEBUG:boto:Token: None 2023-10-11T14:41:35.320 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000620128faec4fb4ad-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.366 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.367 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.368 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.368 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.368 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.369 DEBUG:boto:Method: GET 2023-10-11T14:41:35.369 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.370 DEBUG:boto:Data: 2023-10-11T14:41:35.370 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.372 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.372 DEBUG:boto:Params: {} 2023-10-11T14:41:35.372 DEBUG:boto:Token: None 2023-10-11T14:41:35.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e829f47d9440a50d-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.377 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.378 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.378 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.379 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.380 DEBUG:boto:Path: /dghqvd-11/dummy 2023-10-11T14:41:35.380 DEBUG:boto:Data: 2023-10-11T14:41:35.381 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.382 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.383 DEBUG:boto:Params: {} 2023-10-11T14:41:35.384 DEBUG:boto:Token: None 2023-10-11T14:41:35.385 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy 2023-10-11T14:41:35.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1cmA/KyJsg6MG/t2CD41Ljd+u18= 2023-10-11T14:41:35.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1cmA/KyJsg6MG/t2CD41Ljd+u18=', 'Content-Length': '0'} 2023-10-11T14:41:35.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000035fce22102d0fb36-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.391 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.391 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.392 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.392 DEBUG:boto:Path: /dghqvd-11/dummy 2023-10-11T14:41:35.392 DEBUG:boto:Data: 2023-10-11T14:41:35.393 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.394 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.394 DEBUG:boto:Params: {} 2023-10-11T14:41:35.394 DEBUG:boto:Token: None 2023-10-11T14:41:35.395 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy 2023-10-11T14:41:35.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1cmA/KyJsg6MG/t2CD41Ljd+u18= 2023-10-11T14:41:35.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1cmA/KyJsg6MG/t2CD41Ljd+u18=', 'Content-Length': '0'} 2023-10-11T14:41:35.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fa76ddb939f20c7-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:35.399 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.399 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.400 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.400 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.401 DEBUG:boto:Method: GET 2023-10-11T14:41:35.401 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.402 DEBUG:boto:Data: 2023-10-11T14:41:35.402 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.403 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.403 DEBUG:boto:Params: {} 2023-10-11T14:41:35.404 DEBUG:boto:Token: None 2023-10-11T14:41:35.404 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d7d8acc2450d274-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.408 DEBUG:boto:path=/dghqvd-11/dummy 2023-10-11T14:41:35.408 DEBUG:boto:auth_path=/dghqvd-11/dummy 2023-10-11T14:41:35.409 DEBUG:boto:path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.409 DEBUG:boto:auth_path=/dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.410 DEBUG:boto:Method: GET 2023-10-11T14:41:35.410 DEBUG:boto:Path: /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.410 DEBUG:boto:Data: 2023-10-11T14:41:35.411 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.411 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.412 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.412 DEBUG:boto:Params: {} 2023-10-11T14:41:35.413 DEBUG:boto:Token: None 2023-10-11T14:41:35.413 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/dummy?versionId=alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX 2023-10-11T14:41:35.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI= 2023-10-11T14:41:35.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0bVh7SWRHiEcICJE/KTkT7A6oJI=', 'Content-Length': '0'} 2023-10-11T14:41:35.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:37:16 GMT'), ('x-amz-version-id', 'alMaXFwfL-O8TRkqV1b4q1hFWF6.ysX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e65c81206cd4b352-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.417 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.417 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.418 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.418 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.419 DEBUG:boto:Method: GET 2023-10-11T14:41:35.419 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.420 DEBUG:boto:Data: 2023-10-11T14:41:35.420 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.421 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.421 DEBUG:boto:Params: {} 2023-10-11T14:41:35.422 DEBUG:boto:Token: None 2023-10-11T14:41:35.422 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007628872664e18612-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.426 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.427 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.428 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.428 DEBUG:boto:path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.428 DEBUG:boto:auth_path=/dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.429 DEBUG:boto:Method: GET 2023-10-11T14:41:35.429 DEBUG:boto:Path: /dghqvd-11/?prefix=dummy 2023-10-11T14:41:35.430 DEBUG:boto:Data: 2023-10-11T14:41:35.430 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.431 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.431 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.432 DEBUG:boto:Params: {} 2023-10-11T14:41:35.432 DEBUG:boto:Token: None 2023-10-11T14:41:35.433 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000526f9a56703e3ba8-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.436 DEBUG:boto:b'dghqvd-11dummy1000falsedummy2023-10-11T14:37:16.756Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:35.438 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.438 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.439 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.439 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.440 DEBUG:boto:Method: GET 2023-10-11T14:41:35.440 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.441 DEBUG:boto:Data: 2023-10-11T14:41:35.441 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.441 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.442 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.442 DEBUG:boto:Params: {} 2023-10-11T14:41:35.443 DEBUG:boto:Token: None 2023-10-11T14:41:35.443 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.446 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055ce249dca54ce48-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.447 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.448 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.448 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.449 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.449 DEBUG:boto:Method: GET 2023-10-11T14:41:35.450 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.450 DEBUG:boto:Data: 2023-10-11T14:41:35.450 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.451 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.452 DEBUG:boto:Params: {} 2023-10-11T14:41:35.452 DEBUG:boto:Token: None 2023-10-11T14:41:35.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008378ee05e2c3f07c-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.456 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.457 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.457 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.458 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.459 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.459 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.460 DEBUG:boto:Data: 2023-10-11T14:41:35.460 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.461 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.461 DEBUG:boto:Params: {} 2023-10-11T14:41:35.462 DEBUG:boto:Token: None 2023-10-11T14:41:35.462 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SYvyiVr0QnYNjQPV8CakpvyzdYU= 2023-10-11T14:41:35.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SYvyiVr0QnYNjQPV8CakpvyzdYU=', 'Content-Length': '0'} 2023-10-11T14:41:35.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092c8e3cdbaa6a5a7-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.466 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.467 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.467 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.467 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.468 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.468 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.469 DEBUG:boto:Data: 2023-10-11T14:41:35.469 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.470 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.470 DEBUG:boto:Params: {} 2023-10-11T14:41:35.471 DEBUG:boto:Token: None 2023-10-11T14:41:35.471 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SYvyiVr0QnYNjQPV8CakpvyzdYU= 2023-10-11T14:41:35.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SYvyiVr0QnYNjQPV8CakpvyzdYU=', 'Content-Length': '0'} 2023-10-11T14:41:35.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6be8d22354fea0d-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:35.475 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.476 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.476 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.477 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.477 DEBUG:boto:Method: GET 2023-10-11T14:41:35.477 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.478 DEBUG:boto:Data: 2023-10-11T14:41:35.478 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.479 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.480 DEBUG:boto:Params: {} 2023-10-11T14:41:35.480 DEBUG:boto:Token: None 2023-10-11T14:41:35.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a3d3418675c0255-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.526 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.527 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.527 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.528 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.528 DEBUG:boto:Method: GET 2023-10-11T14:41:35.529 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.529 DEBUG:boto:Data: 2023-10-11T14:41:35.529 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.530 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.531 DEBUG:boto:Params: {} 2023-10-11T14:41:35.531 DEBUG:boto:Token: None 2023-10-11T14:41:35.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.532 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000204c2fb0526d7d4c-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.535 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.536 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.536 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.536 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.537 DEBUG:boto:Method: GET 2023-10-11T14:41:35.537 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.538 DEBUG:boto:Data: 2023-10-11T14:41:35.538 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.539 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.539 DEBUG:boto:Params: {} 2023-10-11T14:41:35.540 DEBUG:boto:Token: None 2023-10-11T14:41:35.540 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.541 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4eae8e08148013-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.586 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.587 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.588 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.588 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.588 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.589 DEBUG:boto:Method: GET 2023-10-11T14:41:35.589 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.590 DEBUG:boto:Data: 2023-10-11T14:41:35.590 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.591 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.591 DEBUG:boto:Params: {} 2023-10-11T14:41:35.592 DEBUG:boto:Token: None 2023-10-11T14:41:35.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89cb039eb7c5f80-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.596 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.597 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.598 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.598 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.598 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:41:35.599 DEBUG:boto:Data: 2023-10-11T14:41:35.599 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.600 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.600 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.600 DEBUG:boto:Params: {} 2023-10-11T14:41:35.601 DEBUG:boto:Token: None 2023-10-11T14:41:35.601 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1 2023-10-11T14:41:35.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r/nC124/7/gfyTdKiOAZqGJwYbE= 2023-10-11T14:41:35.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r/nC124/7/gfyTdKiOAZqGJwYbE=', 'Content-Length': '0'} 2023-10-11T14:41:35.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1f4841290466892-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.605 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.606 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.607 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.607 DEBUG:boto:Path: /dghqvd-11/obj-a1 2023-10-11T14:41:35.608 DEBUG:boto:Data: 2023-10-11T14:41:35.608 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.609 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.609 DEBUG:boto:Params: {} 2023-10-11T14:41:35.610 DEBUG:boto:Token: None 2023-10-11T14:41:35.610 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1 2023-10-11T14:41:35.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r/nC124/7/gfyTdKiOAZqGJwYbE= 2023-10-11T14:41:35.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r/nC124/7/gfyTdKiOAZqGJwYbE=', 'Content-Length': '0'} 2023-10-11T14:41:35.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041929961f8de8489-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:35.615 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.615 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.615 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.616 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.616 DEBUG:boto:Method: GET 2023-10-11T14:41:35.617 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.619 DEBUG:boto:Data: 2023-10-11T14:41:35.620 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.620 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.621 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.621 DEBUG:boto:Params: {} 2023-10-11T14:41:35.622 DEBUG:boto:Token: None 2023-10-11T14:41:35.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e91f892f8654b77d-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.670 DEBUG:boto:path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.671 DEBUG:boto:auth_path=/dghqvd-11/obj-a1 2023-10-11T14:41:35.671 DEBUG:boto:path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.672 DEBUG:boto:auth_path=/dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.672 DEBUG:boto:Method: GET 2023-10-11T14:41:35.673 DEBUG:boto:Path: /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.673 DEBUG:boto:Data: 2023-10-11T14:41:35.673 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.674 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.675 DEBUG:boto:Params: {} 2023-10-11T14:41:35.675 DEBUG:boto:Token: None 2023-10-11T14:41:35.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a1?versionId=7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm 2023-10-11T14:41:35.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag= 2023-10-11T14:41:35.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BIk8qZlqtT5Bnq63EXq5AtW6zag=', 'Content-Length': '0'} 2023-10-11T14:41:35.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '7MXlRLzDnNsDwcdWqn6Q8e4OQMfApYm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030543a3d92270325-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.680 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.680 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.680 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.681 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.681 DEBUG:boto:Method: GET 2023-10-11T14:41:35.682 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.682 DEBUG:boto:Data: 2023-10-11T14:41:35.683 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.683 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.684 DEBUG:boto:Params: {} 2023-10-11T14:41:35.684 DEBUG:boto:Token: None 2023-10-11T14:41:35.685 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000223cc1a27ca6c6e3-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.689 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.689 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.690 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.690 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.691 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.691 DEBUG:boto:Method: GET 2023-10-11T14:41:35.691 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a1 2023-10-11T14:41:35.692 DEBUG:boto:Data: 2023-10-11T14:41:35.692 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.693 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.694 DEBUG:boto:Params: {} 2023-10-11T14:41:35.694 DEBUG:boto:Token: None 2023-10-11T14:41:35.695 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.695 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ef1bd8725dd810f-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.738 DEBUG:boto:b'dghqvd-11obj-a11000falseobj-a12023-10-11T14:40:32.068Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:35.740 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.740 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.741 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.741 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.742 DEBUG:boto:Method: GET 2023-10-11T14:41:35.742 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.743 DEBUG:boto:Data: 2023-10-11T14:41:35.743 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.744 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.744 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.745 DEBUG:boto:Params: {} 2023-10-11T14:41:35.746 DEBUG:boto:Token: None 2023-10-11T14:41:35.746 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e010c2fb58a28585-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.750 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.751 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.751 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.751 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.752 DEBUG:boto:Method: GET 2023-10-11T14:41:35.752 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.753 DEBUG:boto:Data: 2023-10-11T14:41:35.753 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.754 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.755 DEBUG:boto:Params: {} 2023-10-11T14:41:35.755 DEBUG:boto:Token: None 2023-10-11T14:41:35.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010c7680463deb1a9-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.760 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.760 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.761 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.761 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.762 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.762 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.763 DEBUG:boto:Data: 2023-10-11T14:41:35.763 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.764 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.764 DEBUG:boto:Params: {} 2023-10-11T14:41:35.765 DEBUG:boto:Token: None 2023-10-11T14:41:35.765 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JgUAFTeG+pe3rKDh7BL+P+bWOk4= 2023-10-11T14:41:35.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JgUAFTeG+pe3rKDh7BL+P+bWOk4=', 'Content-Length': '0'} 2023-10-11T14:41:35.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000900010a6288c213b-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.768 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.769 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.769 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.770 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.770 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.771 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.771 DEBUG:boto:Data: 2023-10-11T14:41:35.771 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.772 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.772 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.773 DEBUG:boto:Params: {} 2023-10-11T14:41:35.773 DEBUG:boto:Token: None 2023-10-11T14:41:35.773 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JgUAFTeG+pe3rKDh7BL+P+bWOk4= 2023-10-11T14:41:35.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JgUAFTeG+pe3rKDh7BL+P+bWOk4=', 'Content-Length': '0'} 2023-10-11T14:41:35.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcd24fde73ad64cf-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:35.778 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.778 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.779 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.779 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.779 DEBUG:boto:Method: GET 2023-10-11T14:41:35.780 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.780 DEBUG:boto:Data: 2023-10-11T14:41:35.781 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.782 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.782 DEBUG:boto:Params: {} 2023-10-11T14:41:35.782 DEBUG:boto:Token: None 2023-10-11T14:41:35.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.786 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4c5ea1e08418d5-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.787 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.788 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.788 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.788 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.789 DEBUG:boto:Method: GET 2023-10-11T14:41:35.789 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.790 DEBUG:boto:Data: 2023-10-11T14:41:35.790 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.791 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.791 DEBUG:boto:Params: {} 2023-10-11T14:41:35.792 DEBUG:boto:Token: None 2023-10-11T14:41:35.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.795 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fbfb058a4a8a212-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.842 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.843 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.843 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.844 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.844 DEBUG:boto:Method: GET 2023-10-11T14:41:35.845 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.845 DEBUG:boto:Data: 2023-10-11T14:41:35.846 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.847 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.848 DEBUG:boto:Params: {} 2023-10-11T14:41:35.848 DEBUG:boto:Token: None 2023-10-11T14:41:35.849 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f8be6fe35d2438-006526b41f-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.854 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.854 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:35.855 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:35.856 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.856 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.856 DEBUG:boto:Method: GET 2023-10-11T14:41:35.858 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:35.858 DEBUG:boto:Data: 2023-10-11T14:41:35.859 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.859 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.860 DEBUG:boto:Params: {} 2023-10-11T14:41:35.860 DEBUG:boto:Token: None 2023-10-11T14:41:35.861 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/ 2023-10-11T14:41:35.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc= 2023-10-11T14:41:35.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aOnLU+luyW2GsVg28Edwyp00PWc=', 'Content-Length': '0'} 2023-10-11T14:41:35.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df55f8ea0d3d68dc-006526b41f-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.906 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:35.907 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.908 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.908 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.908 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:41:35.909 DEBUG:boto:Data: 2023-10-11T14:41:35.909 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.910 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.911 DEBUG:boto:Params: {} 2023-10-11T14:41:35.912 DEBUG:boto:Token: None 2023-10-11T14:41:35.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2 2023-10-11T14:41:35.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7Ul0XO/Utjxx7wc5RRAvdKr44Y= 2023-10-11T14:41:35.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7Ul0XO/Utjxx7wc5RRAvdKr44Y=', 'Content-Length': '0'} 2023-10-11T14:41:35.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084857356189aac55-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.918 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.918 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.919 DEBUG:boto:Method: HEAD 2023-10-11T14:41:35.919 DEBUG:boto:Path: /dghqvd-11/obj-a2 2023-10-11T14:41:35.919 DEBUG:boto:Data: 2023-10-11T14:41:35.920 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.921 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.921 DEBUG:boto:Params: {} 2023-10-11T14:41:35.922 DEBUG:boto:Token: None 2023-10-11T14:41:35.922 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2 2023-10-11T14:41:35.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7Ul0XO/Utjxx7wc5RRAvdKr44Y= 2023-10-11T14:41:35.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7Ul0XO/Utjxx7wc5RRAvdKr44Y=', 'Content-Length': '0'} 2023-10-11T14:41:35.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e81bfa596946d76-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:35.927 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.927 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.928 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.928 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.929 DEBUG:boto:Method: GET 2023-10-11T14:41:35.929 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.930 DEBUG:boto:Data: 2023-10-11T14:41:35.930 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:35.931 DEBUG:boto:Port: 8001 2023-10-11T14:41:35.932 DEBUG:boto:Params: {} 2023-10-11T14:41:35.932 DEBUG:boto:Token: None 2023-10-11T14:41:35.933 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b78870675ef83ddd-006526b41f-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:35.982 DEBUG:boto:path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.983 DEBUG:boto:auth_path=/dghqvd-11/obj-a2 2023-10-11T14:41:35.983 DEBUG:boto:path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.984 DEBUG:boto:auth_path=/dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.984 DEBUG:boto:Method: GET 2023-10-11T14:41:35.985 DEBUG:boto:Path: /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.985 DEBUG:boto:Data: 2023-10-11T14:41:35.986 DEBUG:boto:Headers: {} 2023-10-11T14:41:35.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:35.987 DEBUG:boto:Port: 8000 2023-10-11T14:41:35.987 DEBUG:boto:Params: {} 2023-10-11T14:41:35.988 DEBUG:boto:Token: None 2023-10-11T14:41:35.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:35 GMT /dghqvd-11/obj-a2?versionId=4rWwWcCI-wbpAWklU32Du67A8cgYeSf 2023-10-11T14:41:35.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk= 2023-10-11T14:41:35.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:otbQoHvT7tW//SUQRj3nNGTxznk=', 'Content-Length': '0'} 2023-10-11T14:41:35.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', '4rWwWcCI-wbpAWklU32Du67A8cgYeSf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005429d3753b4d9d64-006526b41f-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.038 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:36.039 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:36.039 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.040 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.040 DEBUG:boto:Method: GET 2023-10-11T14:41:36.041 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.041 DEBUG:boto:Data: 2023-10-11T14:41:36.042 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.042 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.043 DEBUG:boto:Params: {} 2023-10-11T14:41:36.043 DEBUG:boto:Token: None 2023-10-11T14:41:36.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-11/ 2023-10-11T14:41:36.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+aS4M27bqARC1UM7nVHP6OkOGVo= 2023-10-11T14:41:36.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+aS4M27bqARC1UM7nVHP6OkOGVo=', 'Content-Length': '0'} 2023-10-11T14:41:36.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064e7b384bbd8618b-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.048 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.049 DEBUG:boto:path=/dghqvd-11/ 2023-10-11T14:41:36.049 DEBUG:boto:auth_path=/dghqvd-11/ 2023-10-11T14:41:36.050 DEBUG:boto:path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.050 DEBUG:boto:auth_path=/dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.051 DEBUG:boto:Method: GET 2023-10-11T14:41:36.051 DEBUG:boto:Path: /dghqvd-11/?prefix=obj-a2 2023-10-11T14:41:36.051 DEBUG:boto:Data: 2023-10-11T14:41:36.052 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.053 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.053 DEBUG:boto:Params: {} 2023-10-11T14:41:36.054 DEBUG:boto:Token: None 2023-10-11T14:41:36.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-11/ 2023-10-11T14:41:36.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+aS4M27bqARC1UM7nVHP6OkOGVo= 2023-10-11T14:41:36.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+aS4M27bqARC1UM7nVHP6OkOGVo=', 'Content-Length': '0'} 2023-10-11T14:41:36.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a60ea1217cd38af-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.059 DEBUG:boto:b'dghqvd-11obj-a21000falseobj-a22023-10-11T14:40:32.198Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.060 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-11 zones={a2, a1} 2023-10-11T14:41:36.060 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-12 2023-10-11T14:41:36.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:36.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:36.062 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:36.128 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:36.128 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:36.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:36.161+0000 7f18802c8780 1 bucket sync caught up with source: 2023-10-11T14:41:36.165 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000002.7.11, 00000000011.26.6, ] 2023-10-11T14:41:36.165 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000002.7.11, 00000000011.26.6, ] 2023-10-11T14:41:36.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:36.161+0000 7f18802c8780 0 bucket checkpoint complete 2023-10-11T14:41:36.177 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-12 2023-10-11T14:41:36.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:36.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:36.179 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:36.238 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:36.239 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:36.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:36.268+0000 7f82e26ea780 1 bucket sync caught up with source: 2023-10-11T14:41:36.272 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000011.35.6, 00000000002.20.11, ] 2023-10-11T14:41:36.272 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000011.35.6, 00000000002.20.11, ] 2023-10-11T14:41:36.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:36.268+0000 7f82e26ea780 0 bucket checkpoint complete 2023-10-11T14:41:36.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-12 zones={a2, a1} 2023-10-11T14:41:36.284 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.284 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.284 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.285 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:36.285 DEBUG:boto:Data: 2023-10-11T14:41:36.286 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.287 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.287 DEBUG:boto:Params: {} 2023-10-11T14:41:36.288 DEBUG:boto:Token: None 2023-10-11T14:41:36.288 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fYUWa3wpJeQ7Ibz2X3bSsQUSkJk= 2023-10-11T14:41:36.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fYUWa3wpJeQ7Ibz2X3bSsQUSkJk=', 'Content-Length': '0'} 2023-10-11T14:41:36.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b25e93dff50522e6-006526b420-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.292 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.292 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.293 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.293 DEBUG:boto:Path: /dghqvd-12/ 2023-10-11T14:41:36.294 DEBUG:boto:Data: 2023-10-11T14:41:36.294 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.295 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.295 DEBUG:boto:Params: {} 2023-10-11T14:41:36.296 DEBUG:boto:Token: None 2023-10-11T14:41:36.296 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fYUWa3wpJeQ7Ibz2X3bSsQUSkJk= 2023-10-11T14:41:36.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fYUWa3wpJeQ7Ibz2X3bSsQUSkJk=', 'Content-Length': '0'} 2023-10-11T14:41:36.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032cb15ac1acbff1d-006526b420-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:41:36.301 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.301 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.302 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.303 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.303 DEBUG:boto:Method: GET 2023-10-11T14:41:36.304 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.304 DEBUG:boto:Data: 2023-10-11T14:41:36.305 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.306 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.306 DEBUG:boto:Params: {} 2023-10-11T14:41:36.307 DEBUG:boto:Token: None 2023-10-11T14:41:36.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/?versions 2023-10-11T14:41:36.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc= 2023-10-11T14:41:36.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc=', 'Content-Length': '0'} 2023-10-11T14:41:36.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a321a44d5ed53e0-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.313 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.314 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:36.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:36.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:36.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:41:36.316 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.316 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.317 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.317 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.318 DEBUG:boto:Method: GET 2023-10-11T14:41:36.318 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.318 DEBUG:boto:Data: 2023-10-11T14:41:36.319 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.320 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.320 DEBUG:boto:Params: {} 2023-10-11T14:41:36.321 DEBUG:boto:Token: None 2023-10-11T14:41:36.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/?versions 2023-10-11T14:41:36.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc= 2023-10-11T14:41:36.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc=', 'Content-Length': '0'} 2023-10-11T14:41:36.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019618d59c18845b3-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.325 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.326 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:41:36.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-10-11T14:41:36.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-10-11T14:41:36.329 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.330 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.330 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.331 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.331 DEBUG:boto:Method: GET 2023-10-11T14:41:36.331 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.332 DEBUG:boto:Data: 2023-10-11T14:41:36.332 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.333 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.333 DEBUG:boto:Params: {} 2023-10-11T14:41:36.334 DEBUG:boto:Token: None 2023-10-11T14:41:36.334 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/?versions 2023-10-11T14:41:36.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc= 2023-10-11T14:41:36.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc=', 'Content-Length': '0'} 2023-10-11T14:41:36.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065d9b16356b4a054-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.338 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.339 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.339 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.340 DEBUG:boto:path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.340 DEBUG:boto:auth_path=/dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.341 DEBUG:boto:Method: GET 2023-10-11T14:41:36.341 DEBUG:boto:Path: /dghqvd-12/?versions&max-keys=999 2023-10-11T14:41:36.341 DEBUG:boto:Data: 2023-10-11T14:41:36.342 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.344 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.344 DEBUG:boto:Params: {} 2023-10-11T14:41:36.345 DEBUG:boto:Token: None 2023-10-11T14:41:36.345 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/?versions 2023-10-11T14:41:36.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc= 2023-10-11T14:41:36.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aS7xlLP8vFEw5ZWJ9d9VP9Dtvhc=', 'Content-Length': '0'} 2023-10-11T14:41:36.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7d09050bc43614-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.350 DEBUG:boto:b'dghqvd-12999falsedummyWr3fKGqxmF2PJvp.bI7inJBd4sJJX98true2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.Ftrue2023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwVtrue2023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:36.352 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.352 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.353 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.353 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.353 DEBUG:boto:Method: GET 2023-10-11T14:41:36.354 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.354 DEBUG:boto:Data: 2023-10-11T14:41:36.355 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.356 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.357 DEBUG:boto:Params: {} 2023-10-11T14:41:36.357 DEBUG:boto:Token: None 2023-10-11T14:41:36.358 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.359 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001fdda35220a4cfaf-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.363 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.363 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.363 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.364 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.364 DEBUG:boto:Method: GET 2023-10-11T14:41:36.365 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.365 DEBUG:boto:Data: 2023-10-11T14:41:36.365 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.366 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.367 DEBUG:boto:Params: {} 2023-10-11T14:41:36.367 DEBUG:boto:Token: None 2023-10-11T14:41:36.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a989577e497b7f82-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.372 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.372 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.373 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.373 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.374 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.374 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.375 DEBUG:boto:Data: 2023-10-11T14:41:36.375 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.377 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.377 DEBUG:boto:Params: {} 2023-10-11T14:41:36.378 DEBUG:boto:Token: None 2023-10-11T14:41:36.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:56r7qBeZVAmLWw5DHJknslxNk/g= 2023-10-11T14:41:36.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:56r7qBeZVAmLWw5DHJknslxNk/g=', 'Content-Length': '0'} 2023-10-11T14:41:36.381 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006adeeb3ef6d6488-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.381 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.382 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.382 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.383 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.383 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.383 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.384 DEBUG:boto:Data: 2023-10-11T14:41:36.384 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.385 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.386 DEBUG:boto:Params: {} 2023-10-11T14:41:36.387 DEBUG:boto:Token: None 2023-10-11T14:41:36.387 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:56r7qBeZVAmLWw5DHJknslxNk/g= 2023-10-11T14:41:36.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:56r7qBeZVAmLWw5DHJknslxNk/g=', 'Content-Length': '0'} 2023-10-11T14:41:36.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1330612f4de8f81-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:36.392 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.392 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.393 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.393 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.393 DEBUG:boto:Method: GET 2023-10-11T14:41:36.394 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.394 DEBUG:boto:Data: 2023-10-11T14:41:36.395 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.395 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.396 DEBUG:boto:Params: {} 2023-10-11T14:41:36.396 DEBUG:boto:Token: None 2023-10-11T14:41:36.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.397 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009260eba5689c3007-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.401 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.402 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.402 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.403 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.403 DEBUG:boto:Method: GET 2023-10-11T14:41:36.404 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.404 DEBUG:boto:Data: 2023-10-11T14:41:36.404 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.405 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.406 DEBUG:boto:Params: {} 2023-10-11T14:41:36.406 DEBUG:boto:Token: None 2023-10-11T14:41:36.407 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.407 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008af8865fd89c51d7-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.410 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.410 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.411 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.411 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.412 DEBUG:boto:Method: GET 2023-10-11T14:41:36.412 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.413 DEBUG:boto:Data: 2023-10-11T14:41:36.413 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.414 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.415 DEBUG:boto:Params: {} 2023-10-11T14:41:36.415 DEBUG:boto:Token: None 2023-10-11T14:41:36.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89eba995761eb8e-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.420 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.421 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.421 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.421 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.422 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.422 DEBUG:boto:Method: GET 2023-10-11T14:41:36.423 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.423 DEBUG:boto:Data: 2023-10-11T14:41:36.424 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.424 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.425 DEBUG:boto:Params: {} 2023-10-11T14:41:36.425 DEBUG:boto:Token: None 2023-10-11T14:41:36.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000762af68f249833c9-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.474 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.475 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.476 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.476 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.477 DEBUG:boto:Path: /dghqvd-12/dummy 2023-10-11T14:41:36.477 DEBUG:boto:Data: 2023-10-11T14:41:36.477 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.478 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.479 DEBUG:boto:Params: {} 2023-10-11T14:41:36.479 DEBUG:boto:Token: None 2023-10-11T14:41:36.480 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy 2023-10-11T14:41:36.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVx/64oE20CkVfkA9xVjTpVghtw= 2023-10-11T14:41:36.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVx/64oE20CkVfkA9xVjTpVghtw=', 'Content-Length': '0'} 2023-10-11T14:41:36.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd20d44f18252ebb-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.484 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.484 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.485 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.485 DEBUG:boto:Path: /dghqvd-12/dummy 2023-10-11T14:41:36.486 DEBUG:boto:Data: 2023-10-11T14:41:36.486 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.487 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.487 DEBUG:boto:Params: {} 2023-10-11T14:41:36.488 DEBUG:boto:Token: None 2023-10-11T14:41:36.488 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy 2023-10-11T14:41:36.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVx/64oE20CkVfkA9xVjTpVghtw= 2023-10-11T14:41:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVx/64oE20CkVfkA9xVjTpVghtw=', 'Content-Length': '0'} 2023-10-11T14:41:36.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ea23229f28e32142-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:41:36.493 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.493 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.494 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.494 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.495 DEBUG:boto:Method: GET 2023-10-11T14:41:36.495 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.496 DEBUG:boto:Data: 2023-10-11T14:41:36.496 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.497 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.497 DEBUG:boto:Params: {} 2023-10-11T14:41:36.498 DEBUG:boto:Token: None 2023-10-11T14:41:36.498 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f857cce0b75382e-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.502 DEBUG:boto:path=/dghqvd-12/dummy 2023-10-11T14:41:36.502 DEBUG:boto:auth_path=/dghqvd-12/dummy 2023-10-11T14:41:36.503 DEBUG:boto:path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.504 DEBUG:boto:auth_path=/dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.504 DEBUG:boto:Method: GET 2023-10-11T14:41:36.505 DEBUG:boto:Path: /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.505 DEBUG:boto:Data: 2023-10-11T14:41:36.506 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.507 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.507 DEBUG:boto:Params: {} 2023-10-11T14:41:36.507 DEBUG:boto:Token: None 2023-10-11T14:41:36.508 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/dummy?versionId=Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98 2023-10-11T14:41:36.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI= 2023-10-11T14:41:36.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0CKfBvlS6zvVKJLZv+Y174AVJwI=', 'Content-Length': '0'} 2023-10-11T14:41:36.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:11 GMT'), ('x-amz-version-id', 'Wr3fKGqxmF2PJvp.bI7inJBd4sJJX98'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002ebbac038b4636be-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.511 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.512 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.512 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.512 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.513 DEBUG:boto:Method: GET 2023-10-11T14:41:36.513 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.514 DEBUG:boto:Data: 2023-10-11T14:41:36.515 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.516 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.516 DEBUG:boto:Params: {} 2023-10-11T14:41:36.517 DEBUG:boto:Token: None 2023-10-11T14:41:36.517 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b04ca6da2fae64a-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.521 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.522 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.523 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.523 DEBUG:boto:path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.523 DEBUG:boto:auth_path=/dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.524 DEBUG:boto:Method: GET 2023-10-11T14:41:36.524 DEBUG:boto:Path: /dghqvd-12/?prefix=dummy 2023-10-11T14:41:36.525 DEBUG:boto:Data: 2023-10-11T14:41:36.525 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.526 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.526 DEBUG:boto:Params: {} 2023-10-11T14:41:36.527 DEBUG:boto:Token: None 2023-10-11T14:41:36.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098035d01a993500c-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.531 DEBUG:boto:b'dghqvd-12dummy1000falsedummy2023-10-11T14:40:11.916Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:36.533 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.533 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.533 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.534 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.534 DEBUG:boto:Method: GET 2023-10-11T14:41:36.535 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.535 DEBUG:boto:Data: 2023-10-11T14:41:36.537 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.538 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.538 DEBUG:boto:Params: {} 2023-10-11T14:41:36.539 DEBUG:boto:Token: None 2023-10-11T14:41:36.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9f02d37c693d82a-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.543 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.543 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.544 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.544 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.545 DEBUG:boto:Method: GET 2023-10-11T14:41:36.545 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.546 DEBUG:boto:Data: 2023-10-11T14:41:36.546 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.547 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.547 DEBUG:boto:Params: {} 2023-10-11T14:41:36.548 DEBUG:boto:Token: None 2023-10-11T14:41:36.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.551 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be2dd0a26b0446bc-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.598 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.599 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.599 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.600 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.600 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.601 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.601 DEBUG:boto:Data: 2023-10-11T14:41:36.601 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.602 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.603 DEBUG:boto:Params: {} 2023-10-11T14:41:36.603 DEBUG:boto:Token: None 2023-10-11T14:41:36.604 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jb1mmtElA627NEVZFSm4kjNWLyM= 2023-10-11T14:41:36.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jb1mmtElA627NEVZFSm4kjNWLyM=', 'Content-Length': '0'} 2023-10-11T14:41:36.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090ab661586cae2a6-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.608 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.608 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.609 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.609 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.609 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.610 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.610 DEBUG:boto:Data: 2023-10-11T14:41:36.611 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.612 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.612 DEBUG:boto:Params: {} 2023-10-11T14:41:36.613 DEBUG:boto:Token: None 2023-10-11T14:41:36.613 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jb1mmtElA627NEVZFSm4kjNWLyM= 2023-10-11T14:41:36.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jb1mmtElA627NEVZFSm4kjNWLyM=', 'Content-Length': '0'} 2023-10-11T14:41:36.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002733519f179aa077-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:36.617 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.618 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.618 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.619 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.619 DEBUG:boto:Method: GET 2023-10-11T14:41:36.620 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.620 DEBUG:boto:Data: 2023-10-11T14:41:36.621 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.622 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.622 DEBUG:boto:Params: {} 2023-10-11T14:41:36.623 DEBUG:boto:Token: None 2023-10-11T14:41:36.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99f4f09ad82c913-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.674 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.675 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.675 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.676 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.676 DEBUG:boto:Method: GET 2023-10-11T14:41:36.677 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.677 DEBUG:boto:Data: 2023-10-11T14:41:36.678 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.680 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.680 DEBUG:boto:Params: {} 2023-10-11T14:41:36.681 DEBUG:boto:Token: None 2023-10-11T14:41:36.681 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f3ad612a290095d-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.685 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.685 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.686 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.686 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.687 DEBUG:boto:Method: GET 2023-10-11T14:41:36.687 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.688 DEBUG:boto:Data: 2023-10-11T14:41:36.688 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.690 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.691 DEBUG:boto:Params: {} 2023-10-11T14:41:36.691 DEBUG:boto:Token: None 2023-10-11T14:41:36.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c320d1623508e2-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.738 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.739 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.741 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.741 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.742 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.743 DEBUG:boto:Method: GET 2023-10-11T14:41:36.743 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.744 DEBUG:boto:Data: 2023-10-11T14:41:36.744 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.744 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.745 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.746 DEBUG:boto:Params: {} 2023-10-11T14:41:36.746 DEBUG:boto:Token: None 2023-10-11T14:41:36.747 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149a35edde2cb794-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.751 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.753 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.753 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.754 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.754 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:41:36.754 DEBUG:boto:Data: 2023-10-11T14:41:36.755 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.756 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.756 DEBUG:boto:Params: {} 2023-10-11T14:41:36.757 DEBUG:boto:Token: None 2023-10-11T14:41:36.757 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1 2023-10-11T14:41:36.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QLz2rMq8VrUKTG+vA9gpmYjl5t8= 2023-10-11T14:41:36.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QLz2rMq8VrUKTG+vA9gpmYjl5t8=', 'Content-Length': '0'} 2023-10-11T14:41:36.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000492eb1320c2d201d-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.761 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.762 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.762 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.763 DEBUG:boto:Path: /dghqvd-12/obj-a1 2023-10-11T14:41:36.763 DEBUG:boto:Data: 2023-10-11T14:41:36.764 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.765 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.765 DEBUG:boto:Params: {} 2023-10-11T14:41:36.766 DEBUG:boto:Token: None 2023-10-11T14:41:36.766 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1 2023-10-11T14:41:36.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QLz2rMq8VrUKTG+vA9gpmYjl5t8= 2023-10-11T14:41:36.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QLz2rMq8VrUKTG+vA9gpmYjl5t8=', 'Content-Length': '0'} 2023-10-11T14:41:36.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb5f6583fa88b18a-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-10-11T14:41:36.771 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.771 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.771 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.772 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.772 DEBUG:boto:Method: GET 2023-10-11T14:41:36.773 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.773 DEBUG:boto:Data: 2023-10-11T14:41:36.774 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.774 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.775 DEBUG:boto:Params: {} 2023-10-11T14:41:36.775 DEBUG:boto:Token: None 2023-10-11T14:41:36.776 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002039b3955793cc6b-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.822 DEBUG:boto:path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.823 DEBUG:boto:auth_path=/dghqvd-12/obj-a1 2023-10-11T14:41:36.823 DEBUG:boto:path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.824 DEBUG:boto:auth_path=/dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.824 DEBUG:boto:Method: GET 2023-10-11T14:41:36.825 DEBUG:boto:Path: /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.825 DEBUG:boto:Data: 2023-10-11T14:41:36.825 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.826 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.827 DEBUG:boto:Params: {} 2023-10-11T14:41:36.827 DEBUG:boto:Token: None 2023-10-11T14:41:36.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a1?versionId=GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F 2023-10-11T14:41:36.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o= 2023-10-11T14:41:36.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W53Wrqx3cZKIVBhFVQYa6ssL+8o=', 'Content-Length': '0'} 2023-10-11T14:41:36.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'GZtfy.tD2UjjV0dIJ5OPHdUV39QXO.F'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ae42c9988c8d4b2-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.833 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.834 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.834 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.835 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.835 DEBUG:boto:Method: GET 2023-10-11T14:41:36.836 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.836 DEBUG:boto:Data: 2023-10-11T14:41:36.837 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.837 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.838 DEBUG:boto:Params: {} 2023-10-11T14:41:36.838 DEBUG:boto:Token: None 2023-10-11T14:41:36.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f79a0ab56605c913-006526b420-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.843 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.844 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:36.844 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:36.845 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.845 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.845 DEBUG:boto:Method: GET 2023-10-11T14:41:36.853 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a1 2023-10-11T14:41:36.853 DEBUG:boto:Data: 2023-10-11T14:41:36.854 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.855 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.855 DEBUG:boto:Params: {} 2023-10-11T14:41:36.855 DEBUG:boto:Token: None 2023-10-11T14:41:36.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/ 2023-10-11T14:41:36.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY= 2023-10-11T14:41:36.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OCWl8+awKyPzAemDby5QgbG1wkY=', 'Content-Length': '0'} 2023-10-11T14:41:36.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f895ecd09388f6e3-006526b420-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.860 DEBUG:boto:b'dghqvd-12obj-a11000falseobj-a12023-10-11T14:40:32.335Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:36.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:36.861 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.862 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.862 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.863 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.864 DEBUG:boto:Method: GET 2023-10-11T14:41:36.864 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.864 DEBUG:boto:Data: 2023-10-11T14:41:36.865 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.866 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.866 DEBUG:boto:Params: {} 2023-10-11T14:41:36.867 DEBUG:boto:Token: None 2023-10-11T14:41:36.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4= 2023-10-11T14:41:36.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4=', 'Content-Length': '0'} 2023-10-11T14:41:36.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb3ca0fea5ebd3f5-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.910 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.911 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.911 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.912 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.912 DEBUG:boto:Method: GET 2023-10-11T14:41:36.913 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.913 DEBUG:boto:Data: 2023-10-11T14:41:36.914 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.915 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.915 DEBUG:boto:Params: {} 2023-10-11T14:41:36.915 DEBUG:boto:Token: None 2023-10-11T14:41:36.916 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.916 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4= 2023-10-11T14:41:36.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4=', 'Content-Length': '0'} 2023-10-11T14:41:36.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e503c07e9e9a1d1c-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.966 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.967 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.967 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.968 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.968 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.969 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.969 DEBUG:boto:Data: 2023-10-11T14:41:36.969 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.970 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.971 DEBUG:boto:Params: {} 2023-10-11T14:41:36.971 DEBUG:boto:Token: None 2023-10-11T14:41:36.972 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fdTdAR1wc0TnPYWYeWNUcchGncM= 2023-10-11T14:41:36.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fdTdAR1wc0TnPYWYeWNUcchGncM=', 'Content-Length': '0'} 2023-10-11T14:41:36.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5a158bc910ee37b-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.975 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.975 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.976 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.976 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.977 DEBUG:boto:Method: HEAD 2023-10-11T14:41:36.977 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.977 DEBUG:boto:Data: 2023-10-11T14:41:36.978 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.979 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.979 DEBUG:boto:Params: {} 2023-10-11T14:41:36.980 DEBUG:boto:Token: None 2023-10-11T14:41:36.980 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fdTdAR1wc0TnPYWYeWNUcchGncM= 2023-10-11T14:41:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fdTdAR1wc0TnPYWYeWNUcchGncM=', 'Content-Length': '0'} 2023-10-11T14:41:36.983 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a756ca4fcc1fa0-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:36.984 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.984 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.985 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.985 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.986 DEBUG:boto:Method: GET 2023-10-11T14:41:36.986 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.986 DEBUG:boto:Data: 2023-10-11T14:41:36.987 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:36.988 DEBUG:boto:Port: 8001 2023-10-11T14:41:36.988 DEBUG:boto:Params: {} 2023-10-11T14:41:36.989 DEBUG:boto:Token: None 2023-10-11T14:41:36.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4= 2023-10-11T14:41:36.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4=', 'Content-Length': '0'} 2023-10-11T14:41:36.992 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035439da97f3a096d-006526b420-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:36.993 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.993 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:36.993 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.994 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.994 DEBUG:boto:Method: GET 2023-10-11T14:41:36.995 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.995 DEBUG:boto:Data: 2023-10-11T14:41:36.996 DEBUG:boto:Headers: {} 2023-10-11T14:41:36.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:36.996 DEBUG:boto:Port: 8000 2023-10-11T14:41:36.997 DEBUG:boto:Params: {} 2023-10-11T14:41:36.997 DEBUG:boto:Token: None 2023-10-11T14:41:36.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:36 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:36.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4= 2023-10-11T14:41:36.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MxwQfF3mg5J/ihP9JEBxTNFmMs4=', 'Content-Length': '0'} 2023-10-11T14:41:37.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003efc9ef57a5f1c5-006526b420-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.046 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:37.048 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:37.048 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.048 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.049 DEBUG:boto:Method: GET 2023-10-11T14:41:37.049 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.050 DEBUG:boto:Data: 2023-10-11T14:41:37.050 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:37.051 DEBUG:boto:Port: 8001 2023-10-11T14:41:37.052 DEBUG:boto:Params: {} 2023-10-11T14:41:37.052 DEBUG:boto:Token: None 2023-10-11T14:41:37.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/ 2023-10-11T14:41:37.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo= 2023-10-11T14:41:37.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo=', 'Content-Length': '0'} 2023-10-11T14:41:37.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128d77d84138c2c3-006526b421-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.056 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:37.057 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:37.058 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:37.058 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.059 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.059 DEBUG:boto:Method: GET 2023-10-11T14:41:37.060 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.060 DEBUG:boto:Data: 2023-10-11T14:41:37.060 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.061 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.062 DEBUG:boto:Params: {} 2023-10-11T14:41:37.062 DEBUG:boto:Token: None 2023-10-11T14:41:37.063 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/ 2023-10-11T14:41:37.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo= 2023-10-11T14:41:37.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo=', 'Content-Length': '0'} 2023-10-11T14:41:37.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e1342730cbbf9c-006526b421-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.106 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:37.107 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.108 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.108 DEBUG:boto:Method: HEAD 2023-10-11T14:41:37.108 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:41:37.109 DEBUG:boto:Data: 2023-10-11T14:41:37.109 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:37.110 DEBUG:boto:Port: 8001 2023-10-11T14:41:37.110 DEBUG:boto:Params: {} 2023-10-11T14:41:37.111 DEBUG:boto:Token: None 2023-10-11T14:41:37.111 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/obj-a2 2023-10-11T14:41:37.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vgORdtMtbAVTYiPD67RTxYl93uI= 2023-10-11T14:41:37.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vgORdtMtbAVTYiPD67RTxYl93uI=', 'Content-Length': '0'} 2023-10-11T14:41:37.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd68954bc4a4e209-006526b421-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.116 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.116 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.117 DEBUG:boto:Method: HEAD 2023-10-11T14:41:37.117 DEBUG:boto:Path: /dghqvd-12/obj-a2 2023-10-11T14:41:37.118 DEBUG:boto:Data: 2023-10-11T14:41:37.118 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.119 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.119 DEBUG:boto:Params: {} 2023-10-11T14:41:37.120 DEBUG:boto:Token: None 2023-10-11T14:41:37.120 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/obj-a2 2023-10-11T14:41:37.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vgORdtMtbAVTYiPD67RTxYl93uI= 2023-10-11T14:41:37.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vgORdtMtbAVTYiPD67RTxYl93uI=', 'Content-Length': '0'} 2023-10-11T14:41:37.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f71a6c19f2f85d1d-006526b421-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-10-11T14:41:37.128 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.129 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.129 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.129 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.130 DEBUG:boto:Method: GET 2023-10-11T14:41:37.130 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.131 DEBUG:boto:Data: 2023-10-11T14:41:37.131 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:37.132 DEBUG:boto:Port: 8001 2023-10-11T14:41:37.132 DEBUG:boto:Params: {} 2023-10-11T14:41:37.133 DEBUG:boto:Token: None 2023-10-11T14:41:37.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tcC944kZlsQPsiYX/vn8XEOdWN0= 2023-10-11T14:41:37.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tcC944kZlsQPsiYX/vn8XEOdWN0=', 'Content-Length': '0'} 2023-10-11T14:41:37.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ab00dab4de52049-006526b421-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.137 DEBUG:boto:path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.138 DEBUG:boto:auth_path=/dghqvd-12/obj-a2 2023-10-11T14:41:37.138 DEBUG:boto:path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.139 DEBUG:boto:auth_path=/dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.139 DEBUG:boto:Method: GET 2023-10-11T14:41:37.139 DEBUG:boto:Path: /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.140 DEBUG:boto:Data: 2023-10-11T14:41:37.140 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.141 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.142 DEBUG:boto:Params: {} 2023-10-11T14:41:37.142 DEBUG:boto:Token: None 2023-10-11T14:41:37.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/obj-a2?versionId=qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV 2023-10-11T14:41:37.143 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tcC944kZlsQPsiYX/vn8XEOdWN0= 2023-10-11T14:41:37.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tcC944kZlsQPsiYX/vn8XEOdWN0=', 'Content-Length': '0'} 2023-10-11T14:41:37.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:40:32 GMT'), ('x-amz-version-id', 'qanxd6-I4Gp6WrWJL9OGOQfMm.oiSwV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f7a344082106dab-006526b421-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.190 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:37.191 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:37.191 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.192 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.192 DEBUG:boto:Method: GET 2023-10-11T14:41:37.193 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.193 DEBUG:boto:Data: 2023-10-11T14:41:37.194 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:41:37.194 DEBUG:boto:Port: 8001 2023-10-11T14:41:37.195 DEBUG:boto:Params: {} 2023-10-11T14:41:37.195 DEBUG:boto:Token: None 2023-10-11T14:41:37.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/ 2023-10-11T14:41:37.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo= 2023-10-11T14:41:37.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo=', 'Content-Length': '0'} 2023-10-11T14:41:37.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046538f219666d2ea-006526b421-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.200 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:37.201 DEBUG:boto:path=/dghqvd-12/ 2023-10-11T14:41:37.201 DEBUG:boto:auth_path=/dghqvd-12/ 2023-10-11T14:41:37.202 DEBUG:boto:path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.202 DEBUG:boto:auth_path=/dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.202 DEBUG:boto:Method: GET 2023-10-11T14:41:37.203 DEBUG:boto:Path: /dghqvd-12/?prefix=obj-a2 2023-10-11T14:41:37.203 DEBUG:boto:Data: 2023-10-11T14:41:37.204 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.204 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.205 DEBUG:boto:Params: {} 2023-10-11T14:41:37.206 DEBUG:boto:Token: None 2023-10-11T14:41:37.206 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-12/ 2023-10-11T14:41:37.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo= 2023-10-11T14:41:37.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JNMz8/KSfezIhB+y8AIcTtRjYzo=', 'Content-Length': '0'} 2023-10-11T14:41:37.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d0da32ba079221-006526b421-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.250 DEBUG:boto:b'dghqvd-12obj-a21000falseobj-a22023-10-11T14:40:32.458Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:41:37.251 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-12 zones={a2, a1} 2023-10-11T14:41:37.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-10-11T14:41:37.253 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:41:37.254 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:41:37.254 DEBUG:boto:Method: PUT 2023-10-11T14:41:37.255 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:41:37.255 DEBUG:boto:Data: 2023-10-11T14:41:37.255 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.256 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.256 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.257 DEBUG:boto:Params: {} 2023-10-11T14:41:37.257 DEBUG:boto:Token: None 2023-10-11T14:41:37.258 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-13/ 2023-10-11T14:41:37.258 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXWv0uKml3u7Nfln37EUAHA63Qc= 2023-10-11T14:41:37.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXWv0uKml3u7Nfln37EUAHA63Qc=', 'Content-Length': '0'} 2023-10-11T14:41:37.271 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000061ad70e57ce9bf46-006526b421-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.272 DEBUG:rgw_multi.tests:created bucket=dghqvd-13 2023-10-11T14:41:37.272 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:41:37.273 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:41:37.273 DEBUG:boto:path=/dghqvd-13/?versioning 2023-10-11T14:41:37.273 DEBUG:boto:auth_path=/dghqvd-13/?versioning 2023-10-11T14:41:37.274 DEBUG:boto:Method: PUT 2023-10-11T14:41:37.274 DEBUG:boto:Path: /dghqvd-13/?versioning 2023-10-11T14:41:37.275 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:41:37.275 DEBUG:boto:Headers: {} 2023-10-11T14:41:37.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:37.276 DEBUG:boto:Port: 8000 2023-10-11T14:41:37.276 DEBUG:boto:Params: {} 2023-10-11T14:41:37.277 DEBUG:boto:Token: None 2023-10-11T14:41:37.277 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:41:37 GMT /dghqvd-13/?versioning 2023-10-11T14:41:37.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i8UJe7dDadMHZ0K7929CicmpaX4= 2023-10-11T14:41:37.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i8UJe7dDadMHZ0K7929CicmpaX4=', 'Content-Length': '228'} 2023-10-11T14:41:37.284 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d362c952a79c354-006526b421-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:37.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:37.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:37.285 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:41:37.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:37.336 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:37.364 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:41:37.376 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:41:37.376 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:41:37.376 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:41:37.377 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.377 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.377 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:41:37.377 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:41:37.377 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.378 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:41:37.378 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:41:37.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.379 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035297.281616_66.1", 2023-10-11T14:41:37.379 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:41:37.281616Z" 2023-10-11T14:41:37.379 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.379 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:41:37.381 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035297.281616_66.1'} 2023-10-11T14:41:37.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:41:37.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:37.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:37.383 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:41:37.433 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:37.433 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:37.465+0000 7f6ab4697780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:41:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:37.466+0000 7f6ab4697780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:41:37.480 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:41:37.481 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:41:37.481 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:41:37.481 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:41:37.481 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:41:37.481 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:41:37.482 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:37.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.482 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:41:37.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.482 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:41:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:41:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:41:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:41:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:41:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:41:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:41:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:41:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:41:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:41:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:41:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035297.262747_61.1", 2023-10-11T14:41:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:37.262747Z", 2023-10-11T14:41:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:41:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:41:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:41:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:41:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:41:37Z" 2023-10-11T14:41:37.494 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:41:37.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035297.262747_61.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:37.262747Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:41:37Z" } 2023-10-11T14:41:37.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035297.262747_61.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:37.262747Z', 'realm_epoch': 2}}] 2023-10-11T14:41:37.496 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035297.281616_66.1'} 2023-10-11T14:41:37.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035297.262747_61.1'} 2023-10-11T14:41:37.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035297.281616_66.1 target=1_1697035297.262747_61.1 2023-10-11T14:41:42.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:42.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:41:42.500 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:41:42.557 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:42.558 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:42.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:42.585+0000 7fbdcfc23780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:41:42.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:42.586+0000 7fbdcfc23780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:41:42.601 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:41:42.601 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:41:42.602 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:41:42.602 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:41:42.602 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:41:42.602 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:41:42.603 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:42.603 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.603 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:41:42.603 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:42.603 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:41:42.604 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:42.604 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:42.604 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:41:42.604 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:42.605 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:41:42.605 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:42.605 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:41:42.605 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:42.605 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:42.606 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.606 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:42.606 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:41:42.606 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:42.606 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:42.607 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:41:42.607 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:42.607 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:42.607 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:42.607 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:41:42.608 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:42.608 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:42.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.609 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:42.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:41:42.609 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:42.609 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:42.610 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:41:42.610 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:42.610 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:42.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:42.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:41:42.611 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:42.611 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:42.611 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.611 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:41:42.612 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:41:42.613 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:41:42.614 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:41:42.614 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035297.281616_66.1", 2023-10-11T14:41:42.615 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:41:42.615 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:41:42.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:41:42.615 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:37.281616Z", 2023-10-11T14:41:42.616 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:41:42.616 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:42.616 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:41:42.617 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:41:42.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.617 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:41:42.617 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:41:42.617 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:41:42.618 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:41:42.618 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:41:42Z" 2023-10-11T14:41:42.618 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:41:42.620 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035297.281616_66.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:37.281616Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:41:42Z" } 2023-10-11T14:41:42.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035297.281616_66.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:37.281616Z', 'realm_epoch': 2}}] 2023-10-11T14:41:42.621 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035297.281616_66.1'} 2023-10-11T14:41:42.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035297.281616_66.1'} 2023-10-11T14:41:42.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:41:42.622 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:41:42.622 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:41:42.623 DEBUG:boto:Method: HEAD 2023-10-11T14:41:42.623 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:41:42.624 DEBUG:boto:Data: 2023-10-11T14:41:42.624 DEBUG:boto:Headers: {} 2023-10-11T14:41:42.625 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:42.625 DEBUG:boto:Port: 8000 2023-10-11T14:41:42.625 DEBUG:boto:Params: {} 2023-10-11T14:41:42.626 DEBUG:boto:Token: None 2023-10-11T14:41:42.627 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:41:42 GMT /dghqvd-13/ 2023-10-11T14:41:42.627 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DuJvaHdPaJ92ShCf/ZpkA8o1JAs= 2023-10-11T14:41:42.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:41:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DuJvaHdPaJ92ShCf/ZpkA8o1JAs=', 'Content-Length': '0'} 2023-10-11T14:41:42.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b09f7cb2aaec2574-006526b426-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:42.632 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:41:42.632 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:41:42.632 DEBUG:boto:Method: PUT 2023-10-11T14:41:42.633 DEBUG:boto:Path: /dghqvd-13/dummy 2023-10-11T14:41:42.633 DEBUG:boto:Data: 2023-10-11T14:41:42.634 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:41:42.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:41:42.635 DEBUG:boto:Port: 8000 2023-10-11T14:41:42.635 DEBUG:boto:Params: {} 2023-10-11T14:41:42.636 DEBUG:boto:Token: None 2023-10-11T14:41:42.636 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:41:42 GMT /dghqvd-13/dummy 2023-10-11T14:41:42.637 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7l/ST1m1158lNvE+anHsZeRiI4M= 2023-10-11T14:41:42.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:41:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7l/ST1m1158lNvE+anHsZeRiI4M='} 2023-10-11T14:41:42.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-amz-request-id', 'tx000003bc29a81db51f02c-006526b426-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:41:42.648 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-13 2023-10-11T14:41:42.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:42.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:42.649 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:42.706 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:42.706 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:42.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:42.735+0000 7fea80955780 1 waiting to reach incremental sync.. 2023-10-11T14:41:47.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:47.736+0000 7fea80955780 1 waiting to reach incremental sync.. 2023-10-11T14:41:52.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:52.737+0000 7fea80955780 1 bucket sync caught up with source: 2023-10-11T14:41:52.741 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:41:52.741 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:41:52.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:52.737+0000 7fea80955780 0 bucket checkpoint complete 2023-10-11T14:41:52.756 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-13 2023-10-11T14:41:52.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:52.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:41:52.758 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:41:52.820 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:41:52.820 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:41:52.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:52.851+0000 7f5ad8628780 1 waiting to reach incremental sync.. 2023-10-11T14:41:57.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:41:57.852+0000 7f5ad8628780 1 waiting to reach incremental sync.. 2023-10-11T14:42:02.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:02.853+0000 7f5ad8628780 1 waiting to reach incremental sync.. 2023-10-11T14:42:07.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:07.854+0000 7f5ad8628780 1 waiting to reach incremental sync.. 2023-10-11T14:42:12.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:12.856+0000 7f5ad8628780 1 bucket sync caught up with source: 2023-10-11T14:42:12.859 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.40.11, , , , , ] 2023-10-11T14:42:12.859 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.40.11, , , , , ] 2023-10-11T14:42:12.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:12.856+0000 7f5ad8628780 0 bucket checkpoint complete 2023-10-11T14:42:12.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-13 zones={a2, a1} 2023-10-11T14:42:12.875 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.875 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.876 DEBUG:boto:Method: HEAD 2023-10-11T14:42:12.876 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:12.876 DEBUG:boto:Data: 2023-10-11T14:42:12.877 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:12.878 DEBUG:boto:Port: 8001 2023-10-11T14:42:12.878 DEBUG:boto:Params: {} 2023-10-11T14:42:12.879 DEBUG:boto:Token: None 2023-10-11T14:42:12.879 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/ 2023-10-11T14:42:12.880 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oT3uvU01zzPnVzjyUC28KFTW7j4= 2023-10-11T14:42:12.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oT3uvU01zzPnVzjyUC28KFTW7j4=', 'Content-Length': '0'} 2023-10-11T14:42:12.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a738d42e6b0cd35-006526b444-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.884 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.884 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.885 DEBUG:boto:Method: HEAD 2023-10-11T14:42:12.885 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:12.886 DEBUG:boto:Data: 2023-10-11T14:42:12.886 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:12.887 DEBUG:boto:Port: 8000 2023-10-11T14:42:12.887 DEBUG:boto:Params: {} 2023-10-11T14:42:12.888 DEBUG:boto:Token: None 2023-10-11T14:42:12.888 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/ 2023-10-11T14:42:12.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oT3uvU01zzPnVzjyUC28KFTW7j4= 2023-10-11T14:42:12.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oT3uvU01zzPnVzjyUC28KFTW7j4=', 'Content-Length': '0'} 2023-10-11T14:42:12.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dc7030bd2186270-006526b444-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:42:12.893 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.894 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.894 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.895 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.895 DEBUG:boto:Method: GET 2023-10-11T14:42:12.896 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.896 DEBUG:boto:Data: 2023-10-11T14:42:12.897 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:12.897 DEBUG:boto:Port: 8001 2023-10-11T14:42:12.898 DEBUG:boto:Params: {} 2023-10-11T14:42:12.898 DEBUG:boto:Token: None 2023-10-11T14:42:12.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/?versions 2023-10-11T14:42:12.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8= 2023-10-11T14:42:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8=', 'Content-Length': '0'} 2023-10-11T14:42:12.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7ceed128524cf2b-006526b444-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.946 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:12.947 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:42:12.948 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:42:12.948 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.949 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.949 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.950 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.950 DEBUG:boto:Method: GET 2023-10-11T14:42:12.951 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.951 DEBUG:boto:Data: 2023-10-11T14:42:12.951 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:12.952 DEBUG:boto:Port: 8000 2023-10-11T14:42:12.953 DEBUG:boto:Params: {} 2023-10-11T14:42:12.953 DEBUG:boto:Token: None 2023-10-11T14:42:12.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/?versions 2023-10-11T14:42:12.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8= 2023-10-11T14:42:12.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8=', 'Content-Length': '0'} 2023-10-11T14:42:12.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa23c7b37db185b-006526b444-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.958 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:12.959 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:42:12.960 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.960 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.960 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.961 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.961 DEBUG:boto:Method: GET 2023-10-11T14:42:12.962 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.962 DEBUG:boto:Data: 2023-10-11T14:42:12.963 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:12.964 DEBUG:boto:Port: 8001 2023-10-11T14:42:12.964 DEBUG:boto:Params: {} 2023-10-11T14:42:12.965 DEBUG:boto:Token: None 2023-10-11T14:42:12.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/?versions 2023-10-11T14:42:12.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8= 2023-10-11T14:42:12.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8=', 'Content-Length': '0'} 2023-10-11T14:42:12.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519c33ca05a8b0b2-006526b444-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.970 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:12.970 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:12.971 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:12.971 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.972 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.972 DEBUG:boto:Method: GET 2023-10-11T14:42:12.972 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:42:12.973 DEBUG:boto:Data: 2023-10-11T14:42:12.973 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.974 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:12.974 DEBUG:boto:Port: 8000 2023-10-11T14:42:12.975 DEBUG:boto:Params: {} 2023-10-11T14:42:12.975 DEBUG:boto:Token: None 2023-10-11T14:42:12.976 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/?versions 2023-10-11T14:42:12.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8= 2023-10-11T14:42:12.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BtJkgUqXLmMTl633YC4Jzuf/kv8=', 'Content-Length': '0'} 2023-10-11T14:42:12.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312f3af762d467ca-006526b444-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.980 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:12.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:42:12.981 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:12.982 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:12.982 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.983 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.983 DEBUG:boto:Method: GET 2023-10-11T14:42:12.983 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.984 DEBUG:boto:Data: 2023-10-11T14:42:12.985 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:12.986 DEBUG:boto:Port: 8001 2023-10-11T14:42:12.986 DEBUG:boto:Params: {} 2023-10-11T14:42:12.987 DEBUG:boto:Token: None 2023-10-11T14:42:12.987 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HlSFxGKU66iA/CXFPprTd7Euu/I= 2023-10-11T14:42:12.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HlSFxGKU66iA/CXFPprTd7Euu/I=', 'Content-Length': '0'} 2023-10-11T14:42:12.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb167c62962a714e-006526b444-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:12.991 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:12.991 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:12.992 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.992 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.993 DEBUG:boto:Method: GET 2023-10-11T14:42:12.993 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.993 DEBUG:boto:Data: 2023-10-11T14:42:12.994 DEBUG:boto:Headers: {} 2023-10-11T14:42:12.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:12.995 DEBUG:boto:Port: 8000 2023-10-11T14:42:12.995 DEBUG:boto:Params: {} 2023-10-11T14:42:12.996 DEBUG:boto:Token: None 2023-10-11T14:42:12.996 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:12 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:12.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HlSFxGKU66iA/CXFPprTd7Euu/I= 2023-10-11T14:42:12.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HlSFxGKU66iA/CXFPprTd7Euu/I=', 'Content-Length': '0'} 2023-10-11T14:42:12.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006f20789dbd1d304-006526b444-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.000 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.000 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.001 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.001 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.002 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.002 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.003 DEBUG:boto:Data: 2023-10-11T14:42:13.003 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.004 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.004 DEBUG:boto:Params: {} 2023-10-11T14:42:13.005 DEBUG:boto:Token: None 2023-10-11T14:42:13.005 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.006 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HH9LruWDQ0eCRRcwbcc8S6e4= 2023-10-11T14:42:13.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HH9LruWDQ0eCRRcwbcc8S6e4=', 'Content-Length': '0'} 2023-10-11T14:42:13.008 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c185bc2f5ff7c993-006526b445-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.009 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.009 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.010 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.010 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.010 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.011 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.011 DEBUG:boto:Data: 2023-10-11T14:42:13.012 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.012 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.013 DEBUG:boto:Params: {} 2023-10-11T14:42:13.013 DEBUG:boto:Token: None 2023-10-11T14:42:13.014 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HH9LruWDQ0eCRRcwbcc8S6e4= 2023-10-11T14:42:13.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HH9LruWDQ0eCRRcwbcc8S6e4=', 'Content-Length': '0'} 2023-10-11T14:42:13.017 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cb728148675791f-006526b445-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:42:13.018 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.018 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.019 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.019 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.019 DEBUG:boto:Method: GET 2023-10-11T14:42:13.020 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.020 DEBUG:boto:Data: 2023-10-11T14:42:13.021 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.022 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.022 DEBUG:boto:Params: {} 2023-10-11T14:42:13.023 DEBUG:boto:Token: None 2023-10-11T14:42:13.023 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go= 2023-10-11T14:42:13.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go=', 'Content-Length': '0'} 2023-10-11T14:42:13.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000083b87d5a7b7bde50-006526b445-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.027 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.027 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.028 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.028 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.029 DEBUG:boto:Method: GET 2023-10-11T14:42:13.029 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.030 DEBUG:boto:Data: 2023-10-11T14:42:13.030 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.031 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.031 DEBUG:boto:Params: {} 2023-10-11T14:42:13.032 DEBUG:boto:Token: None 2023-10-11T14:42:13.032 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go= 2023-10-11T14:42:13.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go=', 'Content-Length': '0'} 2023-10-11T14:42:13.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000dcb2091ee627000-006526b445-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.036 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.037 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.037 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.038 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.038 DEBUG:boto:Method: GET 2023-10-11T14:42:13.039 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.039 DEBUG:boto:Data: 2023-10-11T14:42:13.039 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.040 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.041 DEBUG:boto:Params: {} 2023-10-11T14:42:13.041 DEBUG:boto:Token: None 2023-10-11T14:42:13.042 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI= 2023-10-11T14:42:13.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI=', 'Content-Length': '0'} 2023-10-11T14:42:13.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d9c76309fb59d30-006526b445-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.048 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:13.049 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.049 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.050 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.050 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.051 DEBUG:boto:Method: GET 2023-10-11T14:42:13.051 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.052 DEBUG:boto:Data: 2023-10-11T14:42:13.052 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.053 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.053 DEBUG:boto:Params: {} 2023-10-11T14:42:13.054 DEBUG:boto:Token: None 2023-10-11T14:42:13.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI= 2023-10-11T14:42:13.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI=', 'Content-Length': '0'} 2023-10-11T14:42:13.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d88b575fe613ca5-006526b445-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.106 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:13.107 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.108 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.108 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.109 DEBUG:boto:Path: /dghqvd-13/dummy 2023-10-11T14:42:13.109 DEBUG:boto:Data: 2023-10-11T14:42:13.109 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.110 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.111 DEBUG:boto:Params: {} 2023-10-11T14:42:13.111 DEBUG:boto:Token: None 2023-10-11T14:42:13.112 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy 2023-10-11T14:42:13.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HicB3esksJY6cjyBj3gRy/B3PgY= 2023-10-11T14:42:13.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HicB3esksJY6cjyBj3gRy/B3PgY=', 'Content-Length': '0'} 2023-10-11T14:42:13.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa0f0d065c06d301-006526b445-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.116 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.117 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.117 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.117 DEBUG:boto:Path: /dghqvd-13/dummy 2023-10-11T14:42:13.118 DEBUG:boto:Data: 2023-10-11T14:42:13.118 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.119 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.119 DEBUG:boto:Params: {} 2023-10-11T14:42:13.120 DEBUG:boto:Token: None 2023-10-11T14:42:13.120 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy 2023-10-11T14:42:13.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HicB3esksJY6cjyBj3gRy/B3PgY= 2023-10-11T14:42:13.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HicB3esksJY6cjyBj3gRy/B3PgY=', 'Content-Length': '0'} 2023-10-11T14:42:13.124 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f8e4fe9d2092fd7-006526b445-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:42:13.125 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.125 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.126 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.126 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.126 DEBUG:boto:Method: GET 2023-10-11T14:42:13.127 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.127 DEBUG:boto:Data: 2023-10-11T14:42:13.128 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.129 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.129 DEBUG:boto:Params: {} 2023-10-11T14:42:13.130 DEBUG:boto:Token: None 2023-10-11T14:42:13.130 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go= 2023-10-11T14:42:13.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go=', 'Content-Length': '0'} 2023-10-11T14:42:13.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c297a66eda03ff87-006526b445-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.134 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:42:13.134 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:42:13.135 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.135 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.136 DEBUG:boto:Method: GET 2023-10-11T14:42:13.136 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.136 DEBUG:boto:Data: 2023-10-11T14:42:13.137 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.138 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.138 DEBUG:boto:Params: {} 2023-10-11T14:42:13.139 DEBUG:boto:Token: None 2023-10-11T14:42:13.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:42:13.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go= 2023-10-11T14:42:13.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RpKMf0/wk+n7f+KuBhRZsenL7Go=', 'Content-Length': '0'} 2023-10-11T14:42:13.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009efa546a8d91968b-006526b445-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.142 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.143 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.143 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.144 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.144 DEBUG:boto:Method: GET 2023-10-11T14:42:13.145 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.145 DEBUG:boto:Data: 2023-10-11T14:42:13.145 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.146 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.147 DEBUG:boto:Params: {} 2023-10-11T14:42:13.147 DEBUG:boto:Token: None 2023-10-11T14:42:13.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI= 2023-10-11T14:42:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI=', 'Content-Length': '0'} 2023-10-11T14:42:13.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea7d3322595f3222-006526b445-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.198 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:13.199 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.200 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.200 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.201 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.201 DEBUG:boto:Method: GET 2023-10-11T14:42:13.201 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:42:13.202 DEBUG:boto:Data: 2023-10-11T14:42:13.202 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.203 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.204 DEBUG:boto:Params: {} 2023-10-11T14:42:13.204 DEBUG:boto:Token: None 2023-10-11T14:42:13.205 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI= 2023-10-11T14:42:13.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DsOX5YFrIqnnLcaHHwMXQoYdJUI=', 'Content-Length': '0'} 2023-10-11T14:42:13.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000573b880d2dd59c7a-006526b445-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.209 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:42:13.210 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-13 zones={a2, a1} 2023-10-11T14:42:13.210 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.211 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.211 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.211 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.212 DEBUG:boto:Data: 2023-10-11T14:42:13.212 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.213 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.213 DEBUG:boto:Params: {} 2023-10-11T14:42:13.214 DEBUG:boto:Token: None 2023-10-11T14:42:13.214 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9e23271fb36bd67-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.219 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.219 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.219 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.220 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.220 DEBUG:boto:Data: 2023-10-11T14:42:13.221 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.222 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.222 DEBUG:boto:Params: {} 2023-10-11T14:42:13.222 DEBUG:boto:Token: None 2023-10-11T14:42:13.223 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-amz-request-id', 'tx000001aba06556b03f247-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.236 DEBUG:rgw_multi.tests:zone=a1 version=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:42:13.237 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.237 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.238 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.239 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.239 DEBUG:boto:Data: 2023-10-11T14:42:13.240 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.240 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.241 DEBUG:boto:Params: {} 2023-10-11T14:42:13.241 DEBUG:boto:Token: None 2023-10-11T14:42:13.242 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003468d9764d9738cf-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.246 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.246 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.247 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.247 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.248 DEBUG:boto:Data: 2023-10-11T14:42:13.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.248 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.249 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.249 DEBUG:boto:Params: {} 2023-10-11T14:42:13.250 DEBUG:boto:Token: None 2023-10-11T14:42:13.250 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-amz-request-id', 'tx000003822461dc3ca5b8c-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.263 DEBUG:rgw_multi.tests:zone=a2 version=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:42:13.263 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.264 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.264 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.265 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.265 DEBUG:boto:Data: 2023-10-11T14:42:13.265 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.266 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.267 DEBUG:boto:Params: {} 2023-10-11T14:42:13.267 DEBUG:boto:Token: None 2023-10-11T14:42:13.267 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cb207d185772833-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.272 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.272 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.272 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.273 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.273 DEBUG:boto:Data: 2023-10-11T14:42:13.274 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.274 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.275 DEBUG:boto:Params: {} 2023-10-11T14:42:13.275 DEBUG:boto:Token: None 2023-10-11T14:42:13.276 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.276 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.319 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-amz-request-id', 'tx000008636329f0794542b-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.319 DEBUG:rgw_multi.tests:zone=a1 version=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:42:13.320 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.320 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.320 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.321 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.321 DEBUG:boto:Data: 2023-10-11T14:42:13.322 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.323 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.323 DEBUG:boto:Params: {} 2023-10-11T14:42:13.323 DEBUG:boto:Token: None 2023-10-11T14:42:13.324 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f26395450649a97-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.328 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.328 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.329 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.329 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.330 DEBUG:boto:Data: 2023-10-11T14:42:13.330 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.331 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.331 DEBUG:boto:Params: {} 2023-10-11T14:42:13.332 DEBUG:boto:Token: None 2023-10-11T14:42:13.332 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.333 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-amz-request-id', 'tx00000e9a763e73285d793-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.343 DEBUG:rgw_multi.tests:zone=a2 version=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:42:13.344 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.344 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.344 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.345 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.345 DEBUG:boto:Data: 2023-10-11T14:42:13.346 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.346 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.347 DEBUG:boto:Params: {} 2023-10-11T14:42:13.347 DEBUG:boto:Token: None 2023-10-11T14:42:13.348 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000675dff817aecd234-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.352 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.352 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.352 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.353 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.353 DEBUG:boto:Data: 2023-10-11T14:42:13.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.355 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.355 DEBUG:boto:Params: {} 2023-10-11T14:42:13.355 DEBUG:boto:Token: None 2023-10-11T14:42:13.356 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-amz-request-id', 'tx00000bc01650d8f1fec6f-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.367 DEBUG:rgw_multi.tests:zone=a1 version=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:42:13.367 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.367 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.368 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.368 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.369 DEBUG:boto:Data: 2023-10-11T14:42:13.369 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.370 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.371 DEBUG:boto:Params: {} 2023-10-11T14:42:13.371 DEBUG:boto:Token: None 2023-10-11T14:42:13.372 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006da3b457f69ce224-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.376 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.376 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.377 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.377 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.378 DEBUG:boto:Data: 2023-10-11T14:42:13.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.379 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.379 DEBUG:boto:Params: {} 2023-10-11T14:42:13.380 DEBUG:boto:Token: None 2023-10-11T14:42:13.380 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-amz-request-id', 'tx00000270fb1c0527f4cdc-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.427 DEBUG:rgw_multi.tests:zone=a2 version=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:42:13.428 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.428 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.429 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.429 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.429 DEBUG:boto:Data: 2023-10-11T14:42:13.430 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.431 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.431 DEBUG:boto:Params: {} 2023-10-11T14:42:13.431 DEBUG:boto:Token: None 2023-10-11T14:42:13.432 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b796967e208cc3ec-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.436 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.436 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.437 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.437 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.438 DEBUG:boto:Data: 2023-10-11T14:42:13.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.439 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.439 DEBUG:boto:Params: {} 2023-10-11T14:42:13.440 DEBUG:boto:Token: None 2023-10-11T14:42:13.440 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-amz-request-id', 'tx000002c566e250b597678-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.491 DEBUG:rgw_multi.tests:zone=a1 version=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:42:13.492 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.492 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.492 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.493 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.493 DEBUG:boto:Data: 2023-10-11T14:42:13.494 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.495 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.495 DEBUG:boto:Params: {} 2023-10-11T14:42:13.495 DEBUG:boto:Token: None 2023-10-11T14:42:13.496 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.496 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000270529b36bd593b5-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.500 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.501 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.501 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.502 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.502 DEBUG:boto:Data: 2023-10-11T14:42:13.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.503 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.503 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.504 DEBUG:boto:Params: {} 2023-10-11T14:42:13.504 DEBUG:boto:Token: None 2023-10-11T14:42:13.505 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.505 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-amz-request-id', 'tx0000088e83a71e90b92ed-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.516 DEBUG:rgw_multi.tests:zone=a2 version=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:42:13.516 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.517 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.517 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.518 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.518 DEBUG:boto:Data: 2023-10-11T14:42:13.519 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.519 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.520 DEBUG:boto:Params: {} 2023-10-11T14:42:13.520 DEBUG:boto:Token: None 2023-10-11T14:42:13.521 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084a69bae19d55de7-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.525 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.525 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.526 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.526 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.527 DEBUG:boto:Data: 2023-10-11T14:42:13.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.528 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.528 DEBUG:boto:Params: {} 2023-10-11T14:42:13.529 DEBUG:boto:Token: None 2023-10-11T14:42:13.529 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-amz-request-id', 'tx000003363f94a8377e70b-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.540 DEBUG:rgw_multi.tests:zone=a1 version=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:42:13.541 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.541 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.541 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.542 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.542 DEBUG:boto:Data: 2023-10-11T14:42:13.543 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.543 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.544 DEBUG:boto:Params: {} 2023-10-11T14:42:13.545 DEBUG:boto:Token: None 2023-10-11T14:42:13.545 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a908285a0cdf39-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.549 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.549 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.550 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.550 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.551 DEBUG:boto:Data: 2023-10-11T14:42:13.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.552 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.553 DEBUG:boto:Params: {} 2023-10-11T14:42:13.553 DEBUG:boto:Token: None 2023-10-11T14:42:13.554 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-amz-request-id', 'tx0000094700ae9bdfb4971-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.564 DEBUG:rgw_multi.tests:zone=a2 version=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:42:13.565 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.565 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.566 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.566 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.567 DEBUG:boto:Data: 2023-10-11T14:42:13.567 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.568 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.568 DEBUG:boto:Params: {} 2023-10-11T14:42:13.569 DEBUG:boto:Token: None 2023-10-11T14:42:13.569 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.570 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5b9247fc535db2f-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.573 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.574 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.574 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.575 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.576 DEBUG:boto:Data: 2023-10-11T14:42:13.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.577 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.578 DEBUG:boto:Params: {} 2023-10-11T14:42:13.578 DEBUG:boto:Token: None 2023-10-11T14:42:13.579 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-amz-request-id', 'tx000002f9c21ac285d73d9-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.590 DEBUG:rgw_multi.tests:zone=a1 version=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:42:13.590 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.591 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.591 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.591 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.592 DEBUG:boto:Data: 2023-10-11T14:42:13.592 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.593 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.593 DEBUG:boto:Params: {} 2023-10-11T14:42:13.594 DEBUG:boto:Token: None 2023-10-11T14:42:13.594 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b6490a3e1171fbf-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.599 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.599 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.599 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.600 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.600 DEBUG:boto:Data: 2023-10-11T14:42:13.601 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.602 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.602 DEBUG:boto:Params: {} 2023-10-11T14:42:13.603 DEBUG:boto:Token: None 2023-10-11T14:42:13.603 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-amz-request-id', 'tx0000013117c2b75987c48-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.651 DEBUG:rgw_multi.tests:zone=a2 version=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:42:13.652 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.652 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.652 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.653 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.653 DEBUG:boto:Data: 2023-10-11T14:42:13.654 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.658 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.659 DEBUG:boto:Params: {} 2023-10-11T14:42:13.660 DEBUG:boto:Token: None 2023-10-11T14:42:13.660 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.664 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d04a376d2fd432a5-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.665 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.665 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.666 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.666 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.667 DEBUG:boto:Data: 2023-10-11T14:42:13.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.668 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.668 DEBUG:boto:Params: {} 2023-10-11T14:42:13.669 DEBUG:boto:Token: None 2023-10-11T14:42:13.669 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-amz-request-id', 'tx000000b283de2d14d1810-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.715 DEBUG:rgw_multi.tests:zone=a1 version=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:42:13.716 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.716 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.717 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.717 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.717 DEBUG:boto:Data: 2023-10-11T14:42:13.718 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.718 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.719 DEBUG:boto:Params: {} 2023-10-11T14:42:13.719 DEBUG:boto:Token: None 2023-10-11T14:42:13.720 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073d80bfe0a98307d-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.724 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.724 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.725 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.725 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.726 DEBUG:boto:Data: 2023-10-11T14:42:13.726 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.727 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.727 DEBUG:boto:Params: {} 2023-10-11T14:42:13.728 DEBUG:boto:Token: None 2023-10-11T14:42:13.728 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-amz-request-id', 'tx000003c98b791aa1587a8-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.739 DEBUG:rgw_multi.tests:zone=a2 version=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:42:13.739 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.740 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.740 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.740 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.741 DEBUG:boto:Data: 2023-10-11T14:42:13.741 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.742 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.743 DEBUG:boto:Params: {} 2023-10-11T14:42:13.743 DEBUG:boto:Token: None 2023-10-11T14:42:13.743 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021d41e05d2ae2415-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.747 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.748 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.748 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.748 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.749 DEBUG:boto:Data: 2023-10-11T14:42:13.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.750 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.750 DEBUG:boto:Params: {} 2023-10-11T14:42:13.751 DEBUG:boto:Token: None 2023-10-11T14:42:13.751 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-amz-request-id', 'tx00000556db80b56e6740c-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.762 DEBUG:rgw_multi.tests:zone=a1 version=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:42:13.762 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.762 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.763 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.763 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.764 DEBUG:boto:Data: 2023-10-11T14:42:13.764 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.765 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.765 DEBUG:boto:Params: {} 2023-10-11T14:42:13.766 DEBUG:boto:Token: None 2023-10-11T14:42:13.766 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000609425d2ba2f51a7-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.770 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.771 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.771 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.771 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.772 DEBUG:boto:Data: 2023-10-11T14:42:13.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.773 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.773 DEBUG:boto:Params: {} 2023-10-11T14:42:13.774 DEBUG:boto:Token: None 2023-10-11T14:42:13.774 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-amz-request-id', 'tx00000269112ed629f6a5c-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.785 DEBUG:rgw_multi.tests:zone=a2 version=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:42:13.785 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.786 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.786 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.786 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.787 DEBUG:boto:Data: 2023-10-11T14:42:13.787 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.788 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.788 DEBUG:boto:Params: {} 2023-10-11T14:42:13.789 DEBUG:boto:Token: None 2023-10-11T14:42:13.789 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e84f37bdb083d64-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.793 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.794 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.794 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.795 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.795 DEBUG:boto:Data: 2023-10-11T14:42:13.795 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.797 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.797 DEBUG:boto:Params: {} 2023-10-11T14:42:13.797 DEBUG:boto:Token: None 2023-10-11T14:42:13.798 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-amz-request-id', 'tx00000c9fdc07ebcddbfbe-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.843 DEBUG:rgw_multi.tests:zone=a1 version=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:42:13.844 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.844 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.844 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.845 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.845 DEBUG:boto:Data: 2023-10-11T14:42:13.846 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.847 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.847 DEBUG:boto:Params: {} 2023-10-11T14:42:13.847 DEBUG:boto:Token: None 2023-10-11T14:42:13.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006056f74e59b2119b-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.852 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.853 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.853 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.854 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.854 DEBUG:boto:Data: 2023-10-11T14:42:13.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.855 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.856 DEBUG:boto:Params: {} 2023-10-11T14:42:13.856 DEBUG:boto:Token: None 2023-10-11T14:42:13.857 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-amz-request-id', 'tx0000099925820c768c59b-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.903 DEBUG:rgw_multi.tests:zone=a2 version=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:42:13.904 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.904 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.904 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.905 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.905 DEBUG:boto:Data: 2023-10-11T14:42:13.906 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.906 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.907 DEBUG:boto:Params: {} 2023-10-11T14:42:13.907 DEBUG:boto:Token: None 2023-10-11T14:42:13.908 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014ee2ddf6e52f839-006526b445-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.912 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.912 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.913 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.913 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.914 DEBUG:boto:Data: 2023-10-11T14:42:13.914 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:42:13.915 DEBUG:boto:Port: 8000 2023-10-11T14:42:13.915 DEBUG:boto:Params: {} 2023-10-11T14:42:13.916 DEBUG:boto:Token: None 2023-10-11T14:42:13.916 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.917 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-amz-request-id', 'tx0000009279dae6b242afb-006526b445-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.928 DEBUG:rgw_multi.tests:zone=a1 version=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:42:13.928 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:42:13.929 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:42:13.929 DEBUG:boto:Method: HEAD 2023-10-11T14:42:13.930 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:42:13.930 DEBUG:boto:Data: 2023-10-11T14:42:13.931 DEBUG:boto:Headers: {} 2023-10-11T14:42:13.932 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.933 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.933 DEBUG:boto:Params: {} 2023-10-11T14:42:13.934 DEBUG:boto:Token: None 2023-10-11T14:42:13.934 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/ 2023-10-11T14:42:13.935 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s= 2023-10-11T14:42:13.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0C2H8/+MOAabuKD7aKCo5eo6s=', 'Content-Length': '0'} 2023-10-11T14:42:13.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d8b166396aab965-006526b445-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.939 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:42:13.939 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:42:13.939 DEBUG:boto:Method: PUT 2023-10-11T14:42:13.940 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:42:13.940 DEBUG:boto:Data: 2023-10-11T14:42:13.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-10-11T14:42:13.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:42:13.942 DEBUG:boto:Port: 8001 2023-10-11T14:42:13.943 DEBUG:boto:Params: {} 2023-10-11T14:42:13.943 DEBUG:boto:Token: None 2023-10-11T14:42:13.944 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 11 Oct 2023 14:42:13 GMT /dghqvd-13/obj 2023-10-11T14:42:13.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok= 2023-10-11T14:42:13.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:42:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud42coZRb4RxkVmk6gu0FLP5Mok='} 2023-10-11T14:42:13.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-amz-request-id', 'tx00000b931a5cdf03522b2-006526b445-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:42:13.957 DEBUG:rgw_multi.tests:zone=a2 version=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:42:13.958 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-13 2023-10-11T14:42:13.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:42:13.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:42:13.960 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:42:14.022 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:42:14.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:42:14.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:14.055+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:14.058 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:42:14.059 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:19.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:19.057+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:19.060 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:42:19.060 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:24.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:24.058+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:24.061 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:42:24.061 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:29.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:29.059+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:29.062 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-10-11T14:42:29.062 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:34.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:34.060+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:34.063 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000001.9.6, , ] 2023-10-11T14:42:34.063 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:39.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:39.062+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:39.067 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000001.9.6, , ] 2023-10-11T14:42:39.067 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:44.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:44.063+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:44.066 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000001.9.6, , ] 2023-10-11T14:42:44.066 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:49.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:49.064+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:49.067 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000001.9.6, , ] 2023-10-11T14:42:49.067 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:54.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:54.065+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:54.068 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:42:54.068 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:42:59.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:42:59.066+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:42:59.070 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:42:59.070 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:04.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:04.067+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:04.071 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:04.071 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:09.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:09.068+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:09.072 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:09.072 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:14.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:14.069+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:14.073 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:14.073 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:19.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:19.070+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:19.075 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:19.075 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:24.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:24.072+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:24.076 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:24.076 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:29.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:29.073+0000 7f4e41c51780 1 waiting for incremental sync to catch up: 2023-10-11T14:43:29.077 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000015.37.6, , ] 2023-10-11T14:43:29.077 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:34.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:34.074+0000 7f4e41c51780 1 bucket sync caught up with source: 2023-10-11T14:43:34.078 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000003.14.10, , , 00000000031.77.12, , ] 2023-10-11T14:43:34.078 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000003.14.10, , , 00000000020.46.12, , ] 2023-10-11T14:43:34.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:34.074+0000 7f4e41c51780 0 bucket checkpoint complete 2023-10-11T14:43:34.095 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-13 2023-10-11T14:43:34.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:43:34.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:43:34.097 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:43:34.159 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:34.160 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:34.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:34.194+0000 7f2044756780 1 bucket sync caught up with source: 2023-10-11T14:43:34.197 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000002.40.11, , , 00000000031.86.9, , ] 2023-10-11T14:43:34.198 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000002.40.11, , , 00000000031.86.9, , ] 2023-10-11T14:43:34.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:34.194+0000 7f2044756780 0 bucket checkpoint complete 2023-10-11T14:43:34.209 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-13 zones={a2, a1} 2023-10-11T14:43:34.210 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.211 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.211 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.211 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:43:34.212 DEBUG:boto:Data: 2023-10-11T14:43:34.212 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.213 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.214 DEBUG:boto:Params: {} 2023-10-11T14:43:34.214 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:43:34.215 DEBUG:boto:Token: None 2023-10-11T14:43:34.215 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vCWcIxDk78tEYpTLJ+qoMB+wTE0= 2023-10-11T14:43:34.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vCWcIxDk78tEYpTLJ+qoMB+wTE0=', 'Content-Length': '0'} 2023-10-11T14:43:34.221 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002561fa7bf1820f6-006526b496-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.221 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.222 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.222 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.223 DEBUG:boto:Path: /dghqvd-13/ 2023-10-11T14:43:34.223 DEBUG:boto:Data: 2023-10-11T14:43:34.224 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.224 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.225 DEBUG:boto:Params: {} 2023-10-11T14:43:34.225 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:43:34.226 DEBUG:boto:Token: None 2023-10-11T14:43:34.226 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.227 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vCWcIxDk78tEYpTLJ+qoMB+wTE0= 2023-10-11T14:43:34.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vCWcIxDk78tEYpTLJ+qoMB+wTE0=', 'Content-Length': '0'} 2023-10-11T14:43:34.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002296eabcb9bef6c3-006526b496-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:43:34.233 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.233 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.233 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.234 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.234 DEBUG:boto:Method: GET 2023-10-11T14:43:34.235 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.235 DEBUG:boto:Data: 2023-10-11T14:43:34.235 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.236 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.237 DEBUG:boto:Params: {} 2023-10-11T14:43:34.237 DEBUG:boto:Token: None 2023-10-11T14:43:34.238 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/?versions 2023-10-11T14:43:34.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg= 2023-10-11T14:43:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg=', 'Content-Length': '0'} 2023-10-11T14:43:34.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005853c61366949d55-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.243 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6RJDWkrU2bTatsG495AuYikbVYE8YABtrue2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGKfWqmsw-z7Ejg408xKI8ldqyGd8QG6false2023-10-11T14:42:13.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ7nxbudJbYMNy0UJbaIe.Exc7to7dPDfalse2023-10-11T14:42:13.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjndu40zrXogdzY4ng84E1Mr3Hsuv.QWifalse2023-10-11T14:42:13.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhOSuegsak97W3gRnOp3hGGXBGQQ7wLofalse2023-10-11T14:42:13.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjo6LBfgE9sJDVNtvypTH15ie5WNqeEJzfalse2023-10-11T14:42:13.775Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiV91cxetZ.TuW1p2X6OhMF5NHHPhncmfalse2023-10-11T14:42:13.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsEmMWLSV3yOk-qo2EZbn45L5gbVKztDfalse2023-10-11T14:42:13.729Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFwvPjusD13V5DqHCMJ1Pz3g4uLrBghhfalse2023-10-11T14:42:13.604Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgmNHLswwMCcYpP-.C9uSLG-aq2BsTCyfalse2023-10-11T14:42:13.580Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCPjUmICKQIYvCqxTqzrmqcmRuLODPGRfalse2023-10-11T14:42:13.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJZa50GpVI1vCx28Kb5-tNsEeGXK-dS8false2023-10-11T14:42:13.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2wgGYzfzykE51dH4B1WhGC.VF8my1wRfalse2023-10-11T14:42:13.506Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhPp7YWKoT9JsCeZlK-TF18LWy1XGaslfalse2023-10-11T14:42:13.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2AjfwA1MAw87MYsGuLUFeOQn6xDjNFUfalse2023-10-11T14:42:13.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjg-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrlfalse2023-10-11T14:42:13.357Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oTfalse2023-10-11T14:42:13.333Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiYRMCVFlXjwyNAxeAe25Cl73oqKNfq6false2023-10-11T14:42:13.277Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60YNBZiNsGEzuTbT1jnFi1diB1uidQbfalse2023-10-11T14:42:13.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sxfalse2023-10-11T14:42:13.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:43:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:43:34.254 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.254 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.255 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.255 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.255 DEBUG:boto:Method: GET 2023-10-11T14:43:34.256 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.256 DEBUG:boto:Data: 2023-10-11T14:43:34.257 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.258 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.258 DEBUG:boto:Params: {} 2023-10-11T14:43:34.259 DEBUG:boto:Token: None 2023-10-11T14:43:34.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/?versions 2023-10-11T14:43:34.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg= 2023-10-11T14:43:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg=', 'Content-Length': '0'} 2023-10-11T14:43:34.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb470d32be7a76a-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.264 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6RJDWkrU2bTatsG495AuYikbVYE8YABtrue2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGKfWqmsw-z7Ejg408xKI8ldqyGd8QG6false2023-10-11T14:42:13.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ7nxbudJbYMNy0UJbaIe.Exc7to7dPDfalse2023-10-11T14:42:13.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjndu40zrXogdzY4ng84E1Mr3Hsuv.QWifalse2023-10-11T14:42:13.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhOSuegsak97W3gRnOp3hGGXBGQQ7wLofalse2023-10-11T14:42:13.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjo6LBfgE9sJDVNtvypTH15ie5WNqeEJzfalse2023-10-11T14:42:13.775Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiV91cxetZ.TuW1p2X6OhMF5NHHPhncmfalse2023-10-11T14:42:13.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsEmMWLSV3yOk-qo2EZbn45L5gbVKztDfalse2023-10-11T14:42:13.729Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFwvPjusD13V5DqHCMJ1Pz3g4uLrBghhfalse2023-10-11T14:42:13.604Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgmNHLswwMCcYpP-.C9uSLG-aq2BsTCyfalse2023-10-11T14:42:13.580Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCPjUmICKQIYvCqxTqzrmqcmRuLODPGRfalse2023-10-11T14:42:13.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJZa50GpVI1vCx28Kb5-tNsEeGXK-dS8false2023-10-11T14:42:13.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2wgGYzfzykE51dH4B1WhGC.VF8my1wRfalse2023-10-11T14:42:13.506Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhPp7YWKoT9JsCeZlK-TF18LWy1XGaslfalse2023-10-11T14:42:13.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2AjfwA1MAw87MYsGuLUFeOQn6xDjNFUfalse2023-10-11T14:42:13.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjg-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrlfalse2023-10-11T14:42:13.357Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oTfalse2023-10-11T14:42:13.333Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiYRMCVFlXjwyNAxeAe25Cl73oqKNfq6false2023-10-11T14:42:13.277Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60YNBZiNsGEzuTbT1jnFi1diB1uidQbfalse2023-10-11T14:42:13.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sxfalse2023-10-11T14:42:13.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.266 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T14:43:34.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:43:34.279 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.279 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.280 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.280 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.281 DEBUG:boto:Method: GET 2023-10-11T14:43:34.281 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.282 DEBUG:boto:Data: 2023-10-11T14:43:34.282 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.283 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.283 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.283 DEBUG:boto:Params: {} 2023-10-11T14:43:34.284 DEBUG:boto:Token: None 2023-10-11T14:43:34.284 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/?versions 2023-10-11T14:43:34.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg= 2023-10-11T14:43:34.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg=', 'Content-Length': '0'} 2023-10-11T14:43:34.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fae8183b4b8c0c6-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.289 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6RJDWkrU2bTatsG495AuYikbVYE8YABtrue2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGKfWqmsw-z7Ejg408xKI8ldqyGd8QG6false2023-10-11T14:42:13.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ7nxbudJbYMNy0UJbaIe.Exc7to7dPDfalse2023-10-11T14:42:13.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjndu40zrXogdzY4ng84E1Mr3Hsuv.QWifalse2023-10-11T14:42:13.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhOSuegsak97W3gRnOp3hGGXBGQQ7wLofalse2023-10-11T14:42:13.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjo6LBfgE9sJDVNtvypTH15ie5WNqeEJzfalse2023-10-11T14:42:13.775Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiV91cxetZ.TuW1p2X6OhMF5NHHPhncmfalse2023-10-11T14:42:13.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsEmMWLSV3yOk-qo2EZbn45L5gbVKztDfalse2023-10-11T14:42:13.729Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFwvPjusD13V5DqHCMJ1Pz3g4uLrBghhfalse2023-10-11T14:42:13.604Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgmNHLswwMCcYpP-.C9uSLG-aq2BsTCyfalse2023-10-11T14:42:13.580Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCPjUmICKQIYvCqxTqzrmqcmRuLODPGRfalse2023-10-11T14:42:13.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJZa50GpVI1vCx28Kb5-tNsEeGXK-dS8false2023-10-11T14:42:13.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2wgGYzfzykE51dH4B1WhGC.VF8my1wRfalse2023-10-11T14:42:13.506Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhPp7YWKoT9JsCeZlK-TF18LWy1XGaslfalse2023-10-11T14:42:13.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2AjfwA1MAw87MYsGuLUFeOQn6xDjNFUfalse2023-10-11T14:42:13.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjg-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrlfalse2023-10-11T14:42:13.357Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oTfalse2023-10-11T14:42:13.333Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiYRMCVFlXjwyNAxeAe25Cl73oqKNfq6false2023-10-11T14:42:13.277Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60YNBZiNsGEzuTbT1jnFi1diB1uidQbfalse2023-10-11T14:42:13.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sxfalse2023-10-11T14:42:13.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.290 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.291 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.291 DEBUG:boto:path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.292 DEBUG:boto:auth_path=/dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.293 DEBUG:boto:Method: GET 2023-10-11T14:43:34.293 DEBUG:boto:Path: /dghqvd-13/?versions&max-keys=999 2023-10-11T14:43:34.294 DEBUG:boto:Data: 2023-10-11T14:43:34.294 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.295 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.296 DEBUG:boto:Params: {} 2023-10-11T14:43:34.296 DEBUG:boto:Token: None 2023-10-11T14:43:34.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/?versions 2023-10-11T14:43:34.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg= 2023-10-11T14:43:34.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wDd5bhYLhcyygJxWUHIXZPv8aEg=', 'Content-Length': '0'} 2023-10-11T14:43:34.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1dec30e0837e365-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.301 DEBUG:boto:b'dghqvd-13999falsedummyvJ7bRQzARA5p8AFwrsTAUwAxehEXSxdtrue2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6RJDWkrU2bTatsG495AuYikbVYE8YABtrue2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjGKfWqmsw-z7Ejg408xKI8ldqyGd8QG6false2023-10-11T14:42:13.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ7nxbudJbYMNy0UJbaIe.Exc7to7dPDfalse2023-10-11T14:42:13.858Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjndu40zrXogdzY4ng84E1Mr3Hsuv.QWifalse2023-10-11T14:42:13.799Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhOSuegsak97W3gRnOp3hGGXBGQQ7wLofalse2023-10-11T14:42:13.752Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjo6LBfgE9sJDVNtvypTH15ie5WNqeEJzfalse2023-10-11T14:42:13.775Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiV91cxetZ.TuW1p2X6OhMF5NHHPhncmfalse2023-10-11T14:42:13.670Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjsEmMWLSV3yOk-qo2EZbn45L5gbVKztDfalse2023-10-11T14:42:13.729Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFwvPjusD13V5DqHCMJ1Pz3g4uLrBghhfalse2023-10-11T14:42:13.604Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgmNHLswwMCcYpP-.C9uSLG-aq2BsTCyfalse2023-10-11T14:42:13.580Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCPjUmICKQIYvCqxTqzrmqcmRuLODPGRfalse2023-10-11T14:42:13.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJZa50GpVI1vCx28Kb5-tNsEeGXK-dS8false2023-10-11T14:42:13.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2wgGYzfzykE51dH4B1WhGC.VF8my1wRfalse2023-10-11T14:42:13.506Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjhPp7YWKoT9JsCeZlK-TF18LWy1XGaslfalse2023-10-11T14:42:13.441Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2AjfwA1MAw87MYsGuLUFeOQn6xDjNFUfalse2023-10-11T14:42:13.381Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjg-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrlfalse2023-10-11T14:42:13.357Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oTfalse2023-10-11T14:42:13.333Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiYRMCVFlXjwyNAxeAe25Cl73oqKNfq6false2023-10-11T14:42:13.277Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj60YNBZiNsGEzuTbT1jnFi1diB1uidQbfalse2023-10-11T14:42:13.224Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sxfalse2023-10-11T14:42:13.251Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:43:34.303 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.304 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.304 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.305 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.305 DEBUG:boto:Method: GET 2023-10-11T14:43:34.306 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.306 DEBUG:boto:Data: 2023-10-11T14:43:34.307 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.307 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.308 DEBUG:boto:Params: {} 2023-10-11T14:43:34.308 DEBUG:boto:Token: None 2023-10-11T14:43:34.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acc241e2d7e0f303-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.312 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.313 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.313 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.314 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.314 DEBUG:boto:Method: GET 2023-10-11T14:43:34.314 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.315 DEBUG:boto:Data: 2023-10-11T14:43:34.315 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.316 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.317 DEBUG:boto:Params: {} 2023-10-11T14:43:34.317 DEBUG:boto:Token: None 2023-10-11T14:43:34.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ab8f9d541529067-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.321 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.322 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.322 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.323 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.323 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.323 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.324 DEBUG:boto:Data: 2023-10-11T14:43:34.324 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.325 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.326 DEBUG:boto:Params: {} 2023-10-11T14:43:34.326 DEBUG:boto:Token: None 2023-10-11T14:43:34.327 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EzST/we0/2/rpACxKDc1X6X9RVc= 2023-10-11T14:43:34.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EzST/we0/2/rpACxKDc1X6X9RVc=', 'Content-Length': '0'} 2023-10-11T14:43:34.330 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000038c1735765f1db4e-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.330 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.331 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.331 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.332 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.332 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.332 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.333 DEBUG:boto:Data: 2023-10-11T14:43:34.333 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.334 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.334 DEBUG:boto:Params: {} 2023-10-11T14:43:34.335 DEBUG:boto:Token: None 2023-10-11T14:43:34.335 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EzST/we0/2/rpACxKDc1X6X9RVc= 2023-10-11T14:43:34.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EzST/we0/2/rpACxKDc1X6X9RVc=', 'Content-Length': '0'} 2023-10-11T14:43:34.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a7e6e9d3b27a363-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:43:34.339 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.340 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.340 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.340 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.341 DEBUG:boto:Method: GET 2023-10-11T14:43:34.341 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.342 DEBUG:boto:Data: 2023-10-11T14:43:34.342 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.343 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.344 DEBUG:boto:Params: {} 2023-10-11T14:43:34.344 DEBUG:boto:Token: None 2023-10-11T14:43:34.345 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db6913c1b17f6da0-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.348 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.349 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.349 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.349 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.350 DEBUG:boto:Method: GET 2023-10-11T14:43:34.351 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.351 DEBUG:boto:Data: 2023-10-11T14:43:34.351 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.352 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.353 DEBUG:boto:Params: {} 2023-10-11T14:43:34.353 DEBUG:boto:Token: None 2023-10-11T14:43:34.354 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.354 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fbb62ee7fcae1b63-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.357 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.358 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.358 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.359 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.359 DEBUG:boto:Method: GET 2023-10-11T14:43:34.359 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.360 DEBUG:boto:Data: 2023-10-11T14:43:34.360 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.361 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.362 DEBUG:boto:Params: {} 2023-10-11T14:43:34.362 DEBUG:boto:Token: None 2023-10-11T14:43:34.362 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000add232cacec34f2a-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.366 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.367 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.368 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.368 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.369 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.369 DEBUG:boto:Method: GET 2023-10-11T14:43:34.370 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.370 DEBUG:boto:Data: 2023-10-11T14:43:34.371 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.372 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.373 DEBUG:boto:Params: {} 2023-10-11T14:43:34.373 DEBUG:boto:Token: None 2023-10-11T14:43:34.374 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027113543993b9b61-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.378 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.378 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.379 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.379 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.380 DEBUG:boto:Path: /dghqvd-13/dummy 2023-10-11T14:43:34.380 DEBUG:boto:Data: 2023-10-11T14:43:34.380 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.381 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.382 DEBUG:boto:Params: {} 2023-10-11T14:43:34.382 DEBUG:boto:Token: None 2023-10-11T14:43:34.383 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy 2023-10-11T14:43:34.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:exLAviVaCqy/s7WFTSAw42mDPmo= 2023-10-11T14:43:34.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:exLAviVaCqy/s7WFTSAw42mDPmo=', 'Content-Length': '0'} 2023-10-11T14:43:34.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd92227ed3e86979-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.387 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.387 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.388 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.388 DEBUG:boto:Path: /dghqvd-13/dummy 2023-10-11T14:43:34.389 DEBUG:boto:Data: 2023-10-11T14:43:34.389 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.390 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.390 DEBUG:boto:Params: {} 2023-10-11T14:43:34.391 DEBUG:boto:Token: None 2023-10-11T14:43:34.391 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy 2023-10-11T14:43:34.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:exLAviVaCqy/s7WFTSAw42mDPmo= 2023-10-11T14:43:34.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:exLAviVaCqy/s7WFTSAw42mDPmo=', 'Content-Length': '0'} 2023-10-11T14:43:34.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060867c68388ad950-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T14:43:34.396 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.397 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.397 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.397 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.398 DEBUG:boto:Method: GET 2023-10-11T14:43:34.398 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.399 DEBUG:boto:Data: 2023-10-11T14:43:34.399 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.400 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.400 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.400 DEBUG:boto:Params: {} 2023-10-11T14:43:34.401 DEBUG:boto:Token: None 2023-10-11T14:43:34.401 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.402 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b1a25e516f476d3-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.405 DEBUG:boto:path=/dghqvd-13/dummy 2023-10-11T14:43:34.405 DEBUG:boto:auth_path=/dghqvd-13/dummy 2023-10-11T14:43:34.406 DEBUG:boto:path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.407 DEBUG:boto:auth_path=/dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.407 DEBUG:boto:Method: GET 2023-10-11T14:43:34.408 DEBUG:boto:Path: /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.408 DEBUG:boto:Data: 2023-10-11T14:43:34.409 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.410 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.410 DEBUG:boto:Params: {} 2023-10-11T14:43:34.411 DEBUG:boto:Token: None 2023-10-11T14:43:34.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/dummy?versionId=vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd 2023-10-11T14:43:34.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U= 2023-10-11T14:43:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwlcJOloxpnHNyTSOM6DS5kxr0U=', 'Content-Length': '0'} 2023-10-11T14:43:34.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:41:42 GMT'), ('x-amz-version-id', 'vJ7bRQzARA5p8AFwrsTAUwAxehEXSxd'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020e8a6801074460d-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.415 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.416 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.416 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.417 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.417 DEBUG:boto:Method: GET 2023-10-11T14:43:34.418 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.418 DEBUG:boto:Data: 2023-10-11T14:43:34.419 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.419 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.420 DEBUG:boto:Params: {} 2023-10-11T14:43:34.420 DEBUG:boto:Token: None 2023-10-11T14:43:34.421 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a004e8754461c225-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.425 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.425 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.426 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.426 DEBUG:boto:path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.427 DEBUG:boto:auth_path=/dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.427 DEBUG:boto:Method: GET 2023-10-11T14:43:34.428 DEBUG:boto:Path: /dghqvd-13/?prefix=dummy 2023-10-11T14:43:34.428 DEBUG:boto:Data: 2023-10-11T14:43:34.428 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.429 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.430 DEBUG:boto:Params: {} 2023-10-11T14:43:34.430 DEBUG:boto:Token: None 2023-10-11T14:43:34.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.431 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d29932be45438f13-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.478 DEBUG:boto:b'dghqvd-13dummy1000falsedummy2023-10-11T14:41:42.637Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.480 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.480 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.481 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.481 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.482 DEBUG:boto:Method: GET 2023-10-11T14:43:34.482 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.483 DEBUG:boto:Data: 2023-10-11T14:43:34.483 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.485 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.486 DEBUG:boto:Params: {} 2023-10-11T14:43:34.486 DEBUG:boto:Token: None 2023-10-11T14:43:34.487 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006dbcbbe34cb57311-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.491 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.491 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.492 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.492 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.493 DEBUG:boto:Method: GET 2023-10-11T14:43:34.493 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.494 DEBUG:boto:Data: 2023-10-11T14:43:34.494 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.495 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.495 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.497 DEBUG:boto:Params: {} 2023-10-11T14:43:34.498 DEBUG:boto:Token: None 2023-10-11T14:43:34.498 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c62f116bdf886758-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.502 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.502 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.503 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.503 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.503 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.504 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.504 DEBUG:boto:Data: 2023-10-11T14:43:34.505 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.505 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.506 DEBUG:boto:Params: {} 2023-10-11T14:43:34.506 DEBUG:boto:Token: None 2023-10-11T14:43:34.507 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GtVXzjad8/4eS1o7ynRLrxm6TJA= 2023-10-11T14:43:34.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GtVXzjad8/4eS1o7ynRLrxm6TJA=', 'Content-Length': '0'} 2023-10-11T14:43:34.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4af3f85082cc5ae-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.510 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.511 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.511 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.512 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.513 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.513 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.513 DEBUG:boto:Data: 2023-10-11T14:43:34.514 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.515 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.515 DEBUG:boto:Params: {} 2023-10-11T14:43:34.515 DEBUG:boto:Token: None 2023-10-11T14:43:34.516 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GtVXzjad8/4eS1o7ynRLrxm6TJA= 2023-10-11T14:43:34.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GtVXzjad8/4eS1o7ynRLrxm6TJA=', 'Content-Length': '0'} 2023-10-11T14:43:34.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000d9526273fa38d8-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.521 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.521 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.522 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.522 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.523 DEBUG:boto:Method: GET 2023-10-11T14:43:34.523 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.524 DEBUG:boto:Data: 2023-10-11T14:43:34.524 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.525 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.526 DEBUG:boto:Params: {} 2023-10-11T14:43:34.527 DEBUG:boto:Token: None 2023-10-11T14:43:34.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec7672c476fb7efc-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.531 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.532 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.532 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.533 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.533 DEBUG:boto:Method: GET 2023-10-11T14:43:34.534 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.534 DEBUG:boto:Data: 2023-10-11T14:43:34.534 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.535 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.536 DEBUG:boto:Params: {} 2023-10-11T14:43:34.537 DEBUG:boto:Token: None 2023-10-11T14:43:34.537 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e31c5fd52b416da9-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.582 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.583 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.584 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.584 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.584 DEBUG:boto:Method: GET 2023-10-11T14:43:34.585 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.585 DEBUG:boto:Data: 2023-10-11T14:43:34.586 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.587 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.587 DEBUG:boto:Params: {} 2023-10-11T14:43:34.588 DEBUG:boto:Token: None 2023-10-11T14:43:34.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff75e4da2c33ced5-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.594 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.595 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.595 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.596 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.596 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.597 DEBUG:boto:Method: GET 2023-10-11T14:43:34.597 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.598 DEBUG:boto:Data: 2023-10-11T14:43:34.598 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.599 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.600 DEBUG:boto:Params: {} 2023-10-11T14:43:34.600 DEBUG:boto:Token: None 2023-10-11T14:43:34.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e70a1d4ef9d365-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.605 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.605 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.606 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.606 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.607 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:34.608 DEBUG:boto:Data: 2023-10-11T14:43:34.608 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.609 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.609 DEBUG:boto:Params: {} 2023-10-11T14:43:34.610 DEBUG:boto:Token: None 2023-10-11T14:43:34.610 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj 2023-10-11T14:43:34.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ= 2023-10-11T14:43:34.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ=', 'Content-Length': '0'} 2023-10-11T14:43:34.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1670c1c62ff8d6e-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.615 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.616 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.616 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.617 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:34.617 DEBUG:boto:Data: 2023-10-11T14:43:34.617 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.618 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.619 DEBUG:boto:Params: {} 2023-10-11T14:43:34.622 DEBUG:boto:Token: None 2023-10-11T14:43:34.623 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj 2023-10-11T14:43:34.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ= 2023-10-11T14:43:34.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ=', 'Content-Length': '0'} 2023-10-11T14:43:34.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d60aafc5d70c8d46-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.627 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.628 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.628 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.629 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.629 DEBUG:boto:Method: GET 2023-10-11T14:43:34.630 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.630 DEBUG:boto:Data: 2023-10-11T14:43:34.631 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.631 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.632 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.632 DEBUG:boto:Params: {} 2023-10-11T14:43:34.633 DEBUG:boto:Token: None 2023-10-11T14:43:34.633 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004aad56ea499630d5-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.637 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.637 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.638 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.638 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.638 DEBUG:boto:Method: GET 2023-10-11T14:43:34.639 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.639 DEBUG:boto:Data: 2023-10-11T14:43:34.640 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.640 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.641 DEBUG:boto:Params: {} 2023-10-11T14:43:34.641 DEBUG:boto:Token: None 2023-10-11T14:43:34.642 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.642 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8f610b241aea3f3-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.686 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.687 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.689 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.690 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.691 DEBUG:boto:Method: GET 2023-10-11T14:43:34.691 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.692 DEBUG:boto:Data: 2023-10-11T14:43:34.693 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.694 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.695 DEBUG:boto:Params: {} 2023-10-11T14:43:34.695 DEBUG:boto:Token: None 2023-10-11T14:43:34.696 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895d554cf7a367b3-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.702 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.703 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.704 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.704 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.704 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.705 DEBUG:boto:Method: GET 2023-10-11T14:43:34.705 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.706 DEBUG:boto:Data: 2023-10-11T14:43:34.706 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.707 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.708 DEBUG:boto:Params: {} 2023-10-11T14:43:34.708 DEBUG:boto:Token: None 2023-10-11T14:43:34.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5c8749c1fd6ae3-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.713 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.714 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.714 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.715 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.715 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.716 DEBUG:boto:Method: GET 2023-10-11T14:43:34.716 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.716 DEBUG:boto:Data: 2023-10-11T14:43:34.717 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.718 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.718 DEBUG:boto:Params: {} 2023-10-11T14:43:34.719 DEBUG:boto:Token: None 2023-10-11T14:43:34.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g= 2023-10-11T14:43:34.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g=', 'Content-Length': '0'} 2023-10-11T14:43:34.722 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011018731329f7a41-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.723 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.723 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.724 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.724 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.725 DEBUG:boto:Method: GET 2023-10-11T14:43:34.725 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.726 DEBUG:boto:Data: 2023-10-11T14:43:34.726 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.726 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.727 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.727 DEBUG:boto:Params: {} 2023-10-11T14:43:34.728 DEBUG:boto:Token: None 2023-10-11T14:43:34.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g= 2023-10-11T14:43:34.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g=', 'Content-Length': '0'} 2023-10-11T14:43:34.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e743602b53b38cc7-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.732 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.732 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.733 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.733 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.733 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.734 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.734 DEBUG:boto:Data: 2023-10-11T14:43:34.735 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.735 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.735 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.736 DEBUG:boto:Params: {} 2023-10-11T14:43:34.736 DEBUG:boto:Token: None 2023-10-11T14:43:34.737 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.737 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ozElHRPEY6Yd3E2SRsxlPBm5Bvo= 2023-10-11T14:43:34.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ozElHRPEY6Yd3E2SRsxlPBm5Bvo=', 'Content-Length': '0'} 2023-10-11T14:43:34.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000851735645909eef6-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.740 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.740 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.741 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.741 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.742 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.742 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.743 DEBUG:boto:Data: 2023-10-11T14:43:34.743 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.744 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.744 DEBUG:boto:Params: {} 2023-10-11T14:43:34.745 DEBUG:boto:Token: None 2023-10-11T14:43:34.745 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ozElHRPEY6Yd3E2SRsxlPBm5Bvo= 2023-10-11T14:43:34.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ozElHRPEY6Yd3E2SRsxlPBm5Bvo=', 'Content-Length': '0'} 2023-10-11T14:43:34.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002621619b9a9973db-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.749 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.749 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.750 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.750 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.751 DEBUG:boto:Method: GET 2023-10-11T14:43:34.751 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.752 DEBUG:boto:Data: 2023-10-11T14:43:34.752 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.753 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.753 DEBUG:boto:Params: {} 2023-10-11T14:43:34.754 DEBUG:boto:Token: None 2023-10-11T14:43:34.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g= 2023-10-11T14:43:34.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g=', 'Content-Length': '0'} 2023-10-11T14:43:34.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6108f03b9fb7d29-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.798 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.802 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.802 DEBUG:boto:path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.803 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.803 DEBUG:boto:Method: GET 2023-10-11T14:43:34.804 DEBUG:boto:Path: /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.804 DEBUG:boto:Data: 2023-10-11T14:43:34.804 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.805 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.805 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.806 DEBUG:boto:Params: {} 2023-10-11T14:43:34.806 DEBUG:boto:Token: None 2023-10-11T14:43:34.807 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6 2023-10-11T14:43:34.807 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g= 2023-10-11T14:43:34.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LKNXPer0v8am8VA4Niu+PklhF4g=', 'Content-Length': '0'} 2023-10-11T14:43:34.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'GKfWqmsw-z7Ejg408xKI8ldqyGd8QG6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfb2e086b29d71f8-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.810 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.811 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.811 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.811 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.812 DEBUG:boto:Method: GET 2023-10-11T14:43:34.812 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.813 DEBUG:boto:Data: 2023-10-11T14:43:34.813 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.814 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.814 DEBUG:boto:Params: {} 2023-10-11T14:43:34.815 DEBUG:boto:Token: None 2023-10-11T14:43:34.815 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca8a40a44b0dfba5-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.819 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.820 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.820 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.821 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.821 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.822 DEBUG:boto:Method: GET 2023-10-11T14:43:34.822 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.822 DEBUG:boto:Data: 2023-10-11T14:43:34.823 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.824 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.824 DEBUG:boto:Params: {} 2023-10-11T14:43:34.825 DEBUG:boto:Token: None 2023-10-11T14:43:34.825 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025c7b22c1be6aec4-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.829 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.830 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.830 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.831 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.831 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:34.832 DEBUG:boto:Data: 2023-10-11T14:43:34.832 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.833 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.833 DEBUG:boto:Params: {} 2023-10-11T14:43:34.834 DEBUG:boto:Token: None 2023-10-11T14:43:34.834 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj 2023-10-11T14:43:34.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ= 2023-10-11T14:43:34.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ=', 'Content-Length': '0'} 2023-10-11T14:43:34.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000990ba7de3447ca58-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.839 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.839 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.840 DEBUG:boto:Method: HEAD 2023-10-11T14:43:34.840 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:34.841 DEBUG:boto:Data: 2023-10-11T14:43:34.841 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.843 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.843 DEBUG:boto:Params: {} 2023-10-11T14:43:34.844 DEBUG:boto:Token: None 2023-10-11T14:43:34.844 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj 2023-10-11T14:43:34.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ= 2023-10-11T14:43:34.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BYQbGppJn1/sItERyQLNyoSzeyQ=', 'Content-Length': '0'} 2023-10-11T14:43:34.848 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc5a2a3a35af44dd-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.849 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.849 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.850 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.850 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.851 DEBUG:boto:Method: GET 2023-10-11T14:43:34.852 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.852 DEBUG:boto:Data: 2023-10-11T14:43:34.852 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.853 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.854 DEBUG:boto:Params: {} 2023-10-11T14:43:34.855 DEBUG:boto:Token: None 2023-10-11T14:43:34.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c480bd6c2e285f44-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.859 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.859 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.860 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.860 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.860 DEBUG:boto:Method: GET 2023-10-11T14:43:34.861 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.861 DEBUG:boto:Data: 2023-10-11T14:43:34.862 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.863 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.863 DEBUG:boto:Params: {} 2023-10-11T14:43:34.864 DEBUG:boto:Token: None 2023-10-11T14:43:34.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:34.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M= 2023-10-11T14:43:34.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6dRRoJIp6PchKDLWdNu/615M5M=', 'Content-Length': '0'} 2023-10-11T14:43:34.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1f30987ae24d191-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.910 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.911 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.912 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.912 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.912 DEBUG:boto:Method: GET 2023-10-11T14:43:34.913 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.913 DEBUG:boto:Data: 2023-10-11T14:43:34.914 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.914 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.915 DEBUG:boto:Params: {} 2023-10-11T14:43:34.915 DEBUG:boto:Token: None 2023-10-11T14:43:34.916 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.916 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb9d800f143c316-006526b496-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.920 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.920 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:34.921 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:34.921 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.922 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:34.922 DEBUG:boto:Method: GET 2023-10-11T14:43:34.923 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:34.923 DEBUG:boto:Data: 2023-10-11T14:43:34.924 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.924 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.925 DEBUG:boto:Params: {} 2023-10-11T14:43:34.925 DEBUG:boto:Token: None 2023-10-11T14:43:34.926 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/ 2023-10-11T14:43:34.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8= 2023-10-11T14:43:34.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bgmepb9vmsY1Ds52If0wUqZnzG8=', 'Content-Length': '0'} 2023-10-11T14:43:34.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff7d38e66e543e54-006526b496-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.930 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:34.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:34.932 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.932 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.933 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.933 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.933 DEBUG:boto:Method: GET 2023-10-11T14:43:34.934 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.934 DEBUG:boto:Data: 2023-10-11T14:43:34.935 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:34.936 DEBUG:boto:Port: 8001 2023-10-11T14:43:34.936 DEBUG:boto:Params: {} 2023-10-11T14:43:34.936 DEBUG:boto:Token: None 2023-10-11T14:43:34.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1oKATQPrrqEcSZGRkt6Ia/o5Ws= 2023-10-11T14:43:34.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1oKATQPrrqEcSZGRkt6Ia/o5Ws=', 'Content-Length': '0'} 2023-10-11T14:43:34.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c84251a3d2f5a1a9-006526b496-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:34.986 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:34.987 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:34.987 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.988 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.988 DEBUG:boto:Method: GET 2023-10-11T14:43:34.989 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.989 DEBUG:boto:Data: 2023-10-11T14:43:34.990 DEBUG:boto:Headers: {} 2023-10-11T14:43:34.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:34.991 DEBUG:boto:Port: 8000 2023-10-11T14:43:34.991 DEBUG:boto:Params: {} 2023-10-11T14:43:34.992 DEBUG:boto:Token: None 2023-10-11T14:43:34.993 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:34 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:34.995 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1oKATQPrrqEcSZGRkt6Ia/o5Ws= 2023-10-11T14:43:34.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1oKATQPrrqEcSZGRkt6Ia/o5Ws=', 'Content-Length': '0'} 2023-10-11T14:43:34.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b51a53f95297b993-006526b496-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.000 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.000 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.001 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.001 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.002 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.002 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.003 DEBUG:boto:Data: 2023-10-11T14:43:35.003 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.004 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.004 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.005 DEBUG:boto:Params: {} 2023-10-11T14:43:35.005 DEBUG:boto:Token: None 2023-10-11T14:43:35.006 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.006 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jGoFTBvkRVB0FErXxnJd5FQG77g= 2023-10-11T14:43:35.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jGoFTBvkRVB0FErXxnJd5FQG77g=', 'Content-Length': '0'} 2023-10-11T14:43:35.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff665d30354a99a1-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.009 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.010 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.010 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.011 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.011 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.012 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.012 DEBUG:boto:Data: 2023-10-11T14:43:35.013 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.014 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.014 DEBUG:boto:Params: {} 2023-10-11T14:43:35.015 DEBUG:boto:Token: None 2023-10-11T14:43:35.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jGoFTBvkRVB0FErXxnJd5FQG77g= 2023-10-11T14:43:35.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jGoFTBvkRVB0FErXxnJd5FQG77g=', 'Content-Length': '0'} 2023-10-11T14:43:35.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000374ceb1a588b412e-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.019 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.019 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.020 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.020 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.021 DEBUG:boto:Method: GET 2023-10-11T14:43:35.021 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.021 DEBUG:boto:Data: 2023-10-11T14:43:35.022 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.023 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.023 DEBUG:boto:Params: {} 2023-10-11T14:43:35.024 DEBUG:boto:Token: None 2023-10-11T14:43:35.024 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LNSvNjKRhqNwZnM4ECvXtZZrzcs= 2023-10-11T14:43:35.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LNSvNjKRhqNwZnM4ECvXtZZrzcs=', 'Content-Length': '0'} 2023-10-11T14:43:35.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eac116c62a7baa2a-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.028 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.028 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.028 DEBUG:boto:path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.029 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.029 DEBUG:boto:Method: GET 2023-10-11T14:43:35.030 DEBUG:boto:Path: /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.030 DEBUG:boto:Data: 2023-10-11T14:43:35.031 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.031 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.032 DEBUG:boto:Params: {} 2023-10-11T14:43:35.032 DEBUG:boto:Token: None 2023-10-11T14:43:35.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=J7nxbudJbYMNy0UJbaIe.Exc7to7dPD 2023-10-11T14:43:35.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LNSvNjKRhqNwZnM4ECvXtZZrzcs= 2023-10-11T14:43:35.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LNSvNjKRhqNwZnM4ECvXtZZrzcs=', 'Content-Length': '0'} 2023-10-11T14:43:35.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'J7nxbudJbYMNy0UJbaIe.Exc7to7dPD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddcd110b8d038578-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.036 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.037 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.037 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.037 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.038 DEBUG:boto:Method: GET 2023-10-11T14:43:35.038 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.039 DEBUG:boto:Data: 2023-10-11T14:43:35.039 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.040 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.040 DEBUG:boto:Params: {} 2023-10-11T14:43:35.041 DEBUG:boto:Token: None 2023-10-11T14:43:35.041 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000438f54055f6e2456-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.045 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.046 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.047 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.047 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.048 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.048 DEBUG:boto:Method: GET 2023-10-11T14:43:35.048 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.049 DEBUG:boto:Data: 2023-10-11T14:43:35.049 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.050 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.050 DEBUG:boto:Params: {} 2023-10-11T14:43:35.051 DEBUG:boto:Token: None 2023-10-11T14:43:35.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cb068ae1090f39d-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.055 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.056 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.056 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.057 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.057 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.058 DEBUG:boto:Data: 2023-10-11T14:43:35.058 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.059 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.059 DEBUG:boto:Params: {} 2023-10-11T14:43:35.060 DEBUG:boto:Token: None 2023-10-11T14:43:35.060 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054ab659e2e8cd8c3-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.064 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.065 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.065 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.066 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.066 DEBUG:boto:Data: 2023-10-11T14:43:35.067 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.067 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.068 DEBUG:boto:Params: {} 2023-10-11T14:43:35.068 DEBUG:boto:Token: None 2023-10-11T14:43:35.069 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048c866219631fb6f-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.073 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.074 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.074 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.075 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.075 DEBUG:boto:Method: GET 2023-10-11T14:43:35.075 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.076 DEBUG:boto:Data: 2023-10-11T14:43:35.076 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.077 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.077 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.078 DEBUG:boto:Params: {} 2023-10-11T14:43:35.078 DEBUG:boto:Token: None 2023-10-11T14:43:35.079 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb45d6395ed32cff-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.122 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.123 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.123 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.124 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.124 DEBUG:boto:Method: GET 2023-10-11T14:43:35.125 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.125 DEBUG:boto:Data: 2023-10-11T14:43:35.126 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.127 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.127 DEBUG:boto:Params: {} 2023-10-11T14:43:35.128 DEBUG:boto:Token: None 2023-10-11T14:43:35.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.129 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cf4ac4e92e7a1c8-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.132 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.132 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.133 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.133 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.133 DEBUG:boto:Method: GET 2023-10-11T14:43:35.134 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.134 DEBUG:boto:Data: 2023-10-11T14:43:35.135 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.136 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.136 DEBUG:boto:Params: {} 2023-10-11T14:43:35.136 DEBUG:boto:Token: None 2023-10-11T14:43:35.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea11e10acf572ec-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.141 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.142 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.142 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.143 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.143 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.143 DEBUG:boto:Method: GET 2023-10-11T14:43:35.144 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.144 DEBUG:boto:Data: 2023-10-11T14:43:35.145 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.145 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.146 DEBUG:boto:Params: {} 2023-10-11T14:43:35.146 DEBUG:boto:Token: None 2023-10-11T14:43:35.147 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c56038f74ec7a8-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.151 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.152 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.153 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.153 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.153 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.154 DEBUG:boto:Method: GET 2023-10-11T14:43:35.154 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.155 DEBUG:boto:Data: 2023-10-11T14:43:35.155 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.156 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.157 DEBUG:boto:Params: {} 2023-10-11T14:43:35.157 DEBUG:boto:Token: None 2023-10-11T14:43:35.158 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs= 2023-10-11T14:43:35.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034d4da813158276c-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.161 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.162 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.162 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.162 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.163 DEBUG:boto:Method: GET 2023-10-11T14:43:35.163 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.164 DEBUG:boto:Data: 2023-10-11T14:43:35.164 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.165 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.165 DEBUG:boto:Params: {} 2023-10-11T14:43:35.166 DEBUG:boto:Token: None 2023-10-11T14:43:35.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs= 2023-10-11T14:43:35.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c42c1f75c8dd922c-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.170 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.170 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.171 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.171 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.172 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.173 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.173 DEBUG:boto:Data: 2023-10-11T14:43:35.173 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.174 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.175 DEBUG:boto:Params: {} 2023-10-11T14:43:35.175 DEBUG:boto:Token: None 2023-10-11T14:43:35.175 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k5cs7OdsDySkcIe+ou7wmhFo8K0= 2023-10-11T14:43:35.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k5cs7OdsDySkcIe+ou7wmhFo8K0=', 'Content-Length': '0'} 2023-10-11T14:43:35.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026b9c2fec07b4415-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.179 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.179 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.180 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.180 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.180 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.181 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.181 DEBUG:boto:Data: 2023-10-11T14:43:35.182 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.182 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.183 DEBUG:boto:Params: {} 2023-10-11T14:43:35.183 DEBUG:boto:Token: None 2023-10-11T14:43:35.184 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k5cs7OdsDySkcIe+ou7wmhFo8K0= 2023-10-11T14:43:35.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k5cs7OdsDySkcIe+ou7wmhFo8K0=', 'Content-Length': '0'} 2023-10-11T14:43:35.186 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e18b3768ef9c3730-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.187 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.188 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.188 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.189 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.189 DEBUG:boto:Method: GET 2023-10-11T14:43:35.189 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.190 DEBUG:boto:Data: 2023-10-11T14:43:35.190 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.191 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.191 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.192 DEBUG:boto:Params: {} 2023-10-11T14:43:35.192 DEBUG:boto:Token: None 2023-10-11T14:43:35.192 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.193 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs= 2023-10-11T14:43:35.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f14e61e8dcc716f-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.238 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.239 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.240 DEBUG:boto:path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.240 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.240 DEBUG:boto:Method: GET 2023-10-11T14:43:35.241 DEBUG:boto:Path: /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.241 DEBUG:boto:Data: 2023-10-11T14:43:35.242 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.243 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.243 DEBUG:boto:Params: {} 2023-10-11T14:43:35.244 DEBUG:boto:Token: None 2023-10-11T14:43:35.244 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi 2023-10-11T14:43:35.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs= 2023-10-11T14:43:35.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7apcw3mbtfahCXTbyvxHnIQt8hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'ndu40zrXogdzY4ng84E1Mr3Hsuv.QWi'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2d5666225172713-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.248 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.248 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.248 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.249 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.249 DEBUG:boto:Method: GET 2023-10-11T14:43:35.250 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.250 DEBUG:boto:Data: 2023-10-11T14:43:35.251 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.251 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.252 DEBUG:boto:Params: {} 2023-10-11T14:43:35.252 DEBUG:boto:Token: None 2023-10-11T14:43:35.253 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b267ec88c02a38c-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.257 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.257 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.258 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.258 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.259 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.259 DEBUG:boto:Method: GET 2023-10-11T14:43:35.260 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.260 DEBUG:boto:Data: 2023-10-11T14:43:35.261 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.261 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.261 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.262 DEBUG:boto:Params: {} 2023-10-11T14:43:35.262 DEBUG:boto:Token: None 2023-10-11T14:43:35.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ced0fcdabcf5c5-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.310 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.311 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.312 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.312 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.313 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.313 DEBUG:boto:Data: 2023-10-11T14:43:35.313 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.314 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.315 DEBUG:boto:Params: {} 2023-10-11T14:43:35.315 DEBUG:boto:Token: None 2023-10-11T14:43:35.316 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006160313b5f01182d-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.321 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.321 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.322 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.322 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.323 DEBUG:boto:Data: 2023-10-11T14:43:35.323 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.324 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.324 DEBUG:boto:Params: {} 2023-10-11T14:43:35.325 DEBUG:boto:Token: None 2023-10-11T14:43:35.325 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000212265682dd62d58-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.330 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.330 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.331 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.331 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.332 DEBUG:boto:Method: GET 2023-10-11T14:43:35.332 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.332 DEBUG:boto:Data: 2023-10-11T14:43:35.333 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.334 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.334 DEBUG:boto:Params: {} 2023-10-11T14:43:35.335 DEBUG:boto:Token: None 2023-10-11T14:43:35.335 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.338 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d91c928fe1457cd0-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.339 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.339 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.340 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.340 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.341 DEBUG:boto:Method: GET 2023-10-11T14:43:35.341 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.341 DEBUG:boto:Data: 2023-10-11T14:43:35.342 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.343 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.343 DEBUG:boto:Params: {} 2023-10-11T14:43:35.344 DEBUG:boto:Token: None 2023-10-11T14:43:35.344 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c03930bed5517e-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.348 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.348 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.349 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.349 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.349 DEBUG:boto:Method: GET 2023-10-11T14:43:35.350 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.350 DEBUG:boto:Data: 2023-10-11T14:43:35.351 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.352 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.352 DEBUG:boto:Params: {} 2023-10-11T14:43:35.353 DEBUG:boto:Token: None 2023-10-11T14:43:35.353 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f3cec5a1490be7b-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.402 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.403 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.404 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.404 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.405 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.405 DEBUG:boto:Method: GET 2023-10-11T14:43:35.405 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.406 DEBUG:boto:Data: 2023-10-11T14:43:35.406 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.407 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.408 DEBUG:boto:Params: {} 2023-10-11T14:43:35.408 DEBUG:boto:Token: None 2023-10-11T14:43:35.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560939be97388b69-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.412 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.414 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.414 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.414 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.415 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.415 DEBUG:boto:Method: GET 2023-10-11T14:43:35.416 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.416 DEBUG:boto:Data: 2023-10-11T14:43:35.417 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.417 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.418 DEBUG:boto:Params: {} 2023-10-11T14:43:35.418 DEBUG:boto:Token: None 2023-10-11T14:43:35.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M= 2023-10-11T14:43:35.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M=', 'Content-Length': '0'} 2023-10-11T14:43:35.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dd59d02ef1952b6-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.422 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.423 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.423 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.424 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.424 DEBUG:boto:Method: GET 2023-10-11T14:43:35.424 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.425 DEBUG:boto:Data: 2023-10-11T14:43:35.425 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.426 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.427 DEBUG:boto:Params: {} 2023-10-11T14:43:35.427 DEBUG:boto:Token: None 2023-10-11T14:43:35.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M= 2023-10-11T14:43:35.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M=', 'Content-Length': '0'} 2023-10-11T14:43:35.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c17868478a2e121b-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.431 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.431 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.432 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.432 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.433 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.433 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.433 DEBUG:boto:Data: 2023-10-11T14:43:35.434 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.435 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.435 DEBUG:boto:Params: {} 2023-10-11T14:43:35.436 DEBUG:boto:Token: None 2023-10-11T14:43:35.436 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.436 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFN05g2EZcVBqEJUAU6X1MrnvXM= 2023-10-11T14:43:35.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFN05g2EZcVBqEJUAU6X1MrnvXM=', 'Content-Length': '0'} 2023-10-11T14:43:35.439 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d08b710a232a314a-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.440 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.440 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.441 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.441 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.441 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.442 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.442 DEBUG:boto:Data: 2023-10-11T14:43:35.443 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.443 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.444 DEBUG:boto:Params: {} 2023-10-11T14:43:35.444 DEBUG:boto:Token: None 2023-10-11T14:43:35.445 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFN05g2EZcVBqEJUAU6X1MrnvXM= 2023-10-11T14:43:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFN05g2EZcVBqEJUAU6X1MrnvXM=', 'Content-Length': '0'} 2023-10-11T14:43:35.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001235ab783a428ea9-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.448 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.449 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.449 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.449 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.450 DEBUG:boto:Method: GET 2023-10-11T14:43:35.450 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.451 DEBUG:boto:Data: 2023-10-11T14:43:35.451 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.452 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.452 DEBUG:boto:Params: {} 2023-10-11T14:43:35.453 DEBUG:boto:Token: None 2023-10-11T14:43:35.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M= 2023-10-11T14:43:35.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M=', 'Content-Length': '0'} 2023-10-11T14:43:35.456 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d8908f269317003-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.502 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.503 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.503 DEBUG:boto:path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.504 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.504 DEBUG:boto:Method: GET 2023-10-11T14:43:35.505 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.505 DEBUG:boto:Data: 2023-10-11T14:43:35.506 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.507 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.507 DEBUG:boto:Params: {} 2023-10-11T14:43:35.508 DEBUG:boto:Token: None 2023-10-11T14:43:35.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=hOSuegsak97W3gRnOp3hGGXBGQQ7wLo 2023-10-11T14:43:35.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M= 2023-10-11T14:43:35.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9M+6f+xdV8NOpadRm+8Ok0rm04M=', 'Content-Length': '0'} 2023-10-11T14:43:35.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hOSuegsak97W3gRnOp3hGGXBGQQ7wLo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e2c010dbd44e5d5-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.513 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.513 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.514 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.514 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.515 DEBUG:boto:Method: GET 2023-10-11T14:43:35.515 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.515 DEBUG:boto:Data: 2023-10-11T14:43:35.516 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.517 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.517 DEBUG:boto:Params: {} 2023-10-11T14:43:35.518 DEBUG:boto:Token: None 2023-10-11T14:43:35.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd5d7390af29bc68-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.566 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.567 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.568 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.568 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.569 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.569 DEBUG:boto:Method: GET 2023-10-11T14:43:35.569 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.570 DEBUG:boto:Data: 2023-10-11T14:43:35.571 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.572 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.572 DEBUG:boto:Params: {} 2023-10-11T14:43:35.573 DEBUG:boto:Token: None 2023-10-11T14:43:35.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2605e84407816e-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.578 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.578 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.579 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.579 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.580 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.580 DEBUG:boto:Data: 2023-10-11T14:43:35.580 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.581 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.582 DEBUG:boto:Params: {} 2023-10-11T14:43:35.582 DEBUG:boto:Token: None 2023-10-11T14:43:35.583 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.587 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c69f955513b2ec9-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.588 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.588 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.589 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.589 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.589 DEBUG:boto:Data: 2023-10-11T14:43:35.590 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.591 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.591 DEBUG:boto:Params: {} 2023-10-11T14:43:35.591 DEBUG:boto:Token: None 2023-10-11T14:43:35.592 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f83636e67dc7a2dd-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.597 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.598 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.598 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.599 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.599 DEBUG:boto:Method: GET 2023-10-11T14:43:35.600 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.600 DEBUG:boto:Data: 2023-10-11T14:43:35.601 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.601 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.602 DEBUG:boto:Params: {} 2023-10-11T14:43:35.603 DEBUG:boto:Token: None 2023-10-11T14:43:35.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068c819182eb03ca8-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.607 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.608 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.608 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.609 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.609 DEBUG:boto:Method: GET 2023-10-11T14:43:35.610 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.610 DEBUG:boto:Data: 2023-10-11T14:43:35.611 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.612 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.612 DEBUG:boto:Params: {} 2023-10-11T14:43:35.613 DEBUG:boto:Token: None 2023-10-11T14:43:35.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004002a6845bfaf668-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.658 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.659 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.659 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.660 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.660 DEBUG:boto:Method: GET 2023-10-11T14:43:35.661 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.661 DEBUG:boto:Data: 2023-10-11T14:43:35.661 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.662 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.663 DEBUG:boto:Params: {} 2023-10-11T14:43:35.663 DEBUG:boto:Token: None 2023-10-11T14:43:35.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee59a2f4c4637c1-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.668 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.668 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.669 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.669 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.670 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.670 DEBUG:boto:Method: GET 2023-10-11T14:43:35.671 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.671 DEBUG:boto:Data: 2023-10-11T14:43:35.671 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.672 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.673 DEBUG:boto:Params: {} 2023-10-11T14:43:35.673 DEBUG:boto:Token: None 2023-10-11T14:43:35.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cf8937873529fe1-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.678 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.679 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.679 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.680 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.680 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.681 DEBUG:boto:Method: GET 2023-10-11T14:43:35.681 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.681 DEBUG:boto:Data: 2023-10-11T14:43:35.682 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.683 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.683 DEBUG:boto:Params: {} 2023-10-11T14:43:35.684 DEBUG:boto:Token: None 2023-10-11T14:43:35.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg= 2023-10-11T14:43:35.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg=', 'Content-Length': '0'} 2023-10-11T14:43:35.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000335b9e2b3a6cb30d-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.730 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.731 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.731 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.732 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.732 DEBUG:boto:Method: GET 2023-10-11T14:43:35.733 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.733 DEBUG:boto:Data: 2023-10-11T14:43:35.734 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.734 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.735 DEBUG:boto:Params: {} 2023-10-11T14:43:35.735 DEBUG:boto:Token: None 2023-10-11T14:43:35.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg= 2023-10-11T14:43:35.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg=', 'Content-Length': '0'} 2023-10-11T14:43:35.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f74cfe61e28d71c-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.740 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.740 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.741 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.741 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.741 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.742 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.742 DEBUG:boto:Data: 2023-10-11T14:43:35.743 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.743 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.744 DEBUG:boto:Params: {} 2023-10-11T14:43:35.744 DEBUG:boto:Token: None 2023-10-11T14:43:35.745 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1O01F7442r6LS53bIdtzj2fmmNs= 2023-10-11T14:43:35.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1O01F7442r6LS53bIdtzj2fmmNs=', 'Content-Length': '0'} 2023-10-11T14:43:35.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f526b65c2c85d9d6-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.748 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.748 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.749 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.749 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.750 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.750 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.751 DEBUG:boto:Data: 2023-10-11T14:43:35.751 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.752 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.752 DEBUG:boto:Params: {} 2023-10-11T14:43:35.753 DEBUG:boto:Token: None 2023-10-11T14:43:35.753 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1O01F7442r6LS53bIdtzj2fmmNs= 2023-10-11T14:43:35.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1O01F7442r6LS53bIdtzj2fmmNs=', 'Content-Length': '0'} 2023-10-11T14:43:35.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab483551762adaf0-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.757 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.758 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.758 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.758 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.759 DEBUG:boto:Method: GET 2023-10-11T14:43:35.759 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.760 DEBUG:boto:Data: 2023-10-11T14:43:35.760 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.761 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.761 DEBUG:boto:Params: {} 2023-10-11T14:43:35.762 DEBUG:boto:Token: None 2023-10-11T14:43:35.763 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg= 2023-10-11T14:43:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg=', 'Content-Length': '0'} 2023-10-11T14:43:35.766 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084963502308dac1e-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.767 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.767 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.768 DEBUG:boto:path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.768 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.768 DEBUG:boto:Method: GET 2023-10-11T14:43:35.769 DEBUG:boto:Path: /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.769 DEBUG:boto:Data: 2023-10-11T14:43:35.770 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.771 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.771 DEBUG:boto:Params: {} 2023-10-11T14:43:35.772 DEBUG:boto:Token: None 2023-10-11T14:43:35.772 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=o6LBfgE9sJDVNtvypTH15ie5WNqeEJz 2023-10-11T14:43:35.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg= 2023-10-11T14:43:35.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0KsSLCdzWrsELJbn28k06fBGwNg=', 'Content-Length': '0'} 2023-10-11T14:43:35.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'o6LBfgE9sJDVNtvypTH15ie5WNqeEJz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cebaaa3d47294e7-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.776 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.776 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.777 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.777 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.777 DEBUG:boto:Method: GET 2023-10-11T14:43:35.778 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.778 DEBUG:boto:Data: 2023-10-11T14:43:35.779 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.780 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.780 DEBUG:boto:Params: {} 2023-10-11T14:43:35.781 DEBUG:boto:Token: None 2023-10-11T14:43:35.781 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070f88d3fd82522c-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.786 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.786 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.787 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.787 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.788 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.788 DEBUG:boto:Method: GET 2023-10-11T14:43:35.789 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.789 DEBUG:boto:Data: 2023-10-11T14:43:35.789 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.790 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.791 DEBUG:boto:Params: {} 2023-10-11T14:43:35.791 DEBUG:boto:Token: None 2023-10-11T14:43:35.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113556ba44ccd4b6-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.842 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.844 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.844 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.844 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.845 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.846 DEBUG:boto:Data: 2023-10-11T14:43:35.846 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.847 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.847 DEBUG:boto:Params: {} 2023-10-11T14:43:35.848 DEBUG:boto:Token: None 2023-10-11T14:43:35.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.854 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bd95c566716ab5f-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.854 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.855 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.855 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.856 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:35.856 DEBUG:boto:Data: 2023-10-11T14:43:35.857 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.858 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.858 DEBUG:boto:Params: {} 2023-10-11T14:43:35.858 DEBUG:boto:Token: None 2023-10-11T14:43:35.859 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj 2023-10-11T14:43:35.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag= 2023-10-11T14:43:35.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqw6tkfnXMmN3P6QyIictXrT8Ag=', 'Content-Length': '0'} 2023-10-11T14:43:35.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003980575e4bcf5b18-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.864 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.864 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.865 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.865 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.866 DEBUG:boto:Method: GET 2023-10-11T14:43:35.866 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.867 DEBUG:boto:Data: 2023-10-11T14:43:35.867 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.868 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.869 DEBUG:boto:Params: {} 2023-10-11T14:43:35.869 DEBUG:boto:Token: None 2023-10-11T14:43:35.869 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.872 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b25a749257cf6dfb-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.873 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.874 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.874 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.874 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.875 DEBUG:boto:Method: GET 2023-10-11T14:43:35.875 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.876 DEBUG:boto:Data: 2023-10-11T14:43:35.876 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.877 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.877 DEBUG:boto:Params: {} 2023-10-11T14:43:35.878 DEBUG:boto:Token: None 2023-10-11T14:43:35.878 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:35.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk= 2023-10-11T14:43:35.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s5QylaFjVipOwoFketvCBu7ACMk=', 'Content-Length': '0'} 2023-10-11T14:43:35.881 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aeed91f9a2619f3d-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.882 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.883 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.883 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.883 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.884 DEBUG:boto:Method: GET 2023-10-11T14:43:35.884 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.885 DEBUG:boto:Data: 2023-10-11T14:43:35.885 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.886 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.886 DEBUG:boto:Params: {} 2023-10-11T14:43:35.887 DEBUG:boto:Token: None 2023-10-11T14:43:35.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000033a04a11f23fe7f-006526b497-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.892 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.892 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:35.893 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:35.893 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.894 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:35.894 DEBUG:boto:Method: GET 2023-10-11T14:43:35.894 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:35.895 DEBUG:boto:Data: 2023-10-11T14:43:35.895 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.896 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.897 DEBUG:boto:Params: {} 2023-10-11T14:43:35.897 DEBUG:boto:Token: None 2023-10-11T14:43:35.897 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/ 2023-10-11T14:43:35.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys= 2023-10-11T14:43:35.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NW/iAVFVqZwBKwb64Lfqzf2XKys=', 'Content-Length': '0'} 2023-10-11T14:43:35.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab43751d066feaf-006526b497-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.902 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:35.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.903 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.903 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.904 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.904 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.905 DEBUG:boto:Method: GET 2023-10-11T14:43:35.905 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.905 DEBUG:boto:Data: 2023-10-11T14:43:35.906 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.907 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.907 DEBUG:boto:Params: {} 2023-10-11T14:43:35.908 DEBUG:boto:Token: None 2023-10-11T14:43:35.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0= 2023-10-11T14:43:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0=', 'Content-Length': '0'} 2023-10-11T14:43:35.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008eede4ef8ced19e1-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.914 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.914 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.915 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.915 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.916 DEBUG:boto:Method: GET 2023-10-11T14:43:35.916 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.916 DEBUG:boto:Data: 2023-10-11T14:43:35.917 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.918 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.918 DEBUG:boto:Params: {} 2023-10-11T14:43:35.919 DEBUG:boto:Token: None 2023-10-11T14:43:35.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0= 2023-10-11T14:43:35.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0=', 'Content-Length': '0'} 2023-10-11T14:43:35.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000224332934a4e7dee-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.925 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.926 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.926 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.927 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.927 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.928 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.928 DEBUG:boto:Data: 2023-10-11T14:43:35.928 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.929 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.930 DEBUG:boto:Params: {} 2023-10-11T14:43:35.930 DEBUG:boto:Token: None 2023-10-11T14:43:35.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dl7WOy6m10l2IbqpLWSKZS2u0hs= 2023-10-11T14:43:35.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dl7WOy6m10l2IbqpLWSKZS2u0hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8edfcdd3d344200-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.934 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.934 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.935 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.935 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.936 DEBUG:boto:Method: HEAD 2023-10-11T14:43:35.936 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.936 DEBUG:boto:Data: 2023-10-11T14:43:35.937 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.938 DEBUG:boto:Port: 8000 2023-10-11T14:43:35.938 DEBUG:boto:Params: {} 2023-10-11T14:43:35.938 DEBUG:boto:Token: None 2023-10-11T14:43:35.939 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dl7WOy6m10l2IbqpLWSKZS2u0hs= 2023-10-11T14:43:35.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dl7WOy6m10l2IbqpLWSKZS2u0hs=', 'Content-Length': '0'} 2023-10-11T14:43:35.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027052a276c11113d-006526b497-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:35.943 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.944 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.944 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.945 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.945 DEBUG:boto:Method: GET 2023-10-11T14:43:35.945 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.946 DEBUG:boto:Data: 2023-10-11T14:43:35.946 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:35.947 DEBUG:boto:Port: 8001 2023-10-11T14:43:35.948 DEBUG:boto:Params: {} 2023-10-11T14:43:35.948 DEBUG:boto:Token: None 2023-10-11T14:43:35.949 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:35 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0= 2023-10-11T14:43:35.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vdymZqPdiISfaEBha+1FgpZJcM0=', 'Content-Length': '0'} 2023-10-11T14:43:35.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086992bf6c9c82e5b-006526b497-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:35.994 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:35.995 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:35.995 DEBUG:boto:path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.996 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.996 DEBUG:boto:Method: GET 2023-10-11T14:43:35.997 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:35.997 DEBUG:boto:Data: 2023-10-11T14:43:35.998 DEBUG:boto:Headers: {} 2023-10-11T14:43:35.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:35.998 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.000 DEBUG:boto:Params: {} 2023-10-11T14:43:36.000 DEBUG:boto:Token: None 2023-10-11T14:43:36.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=iV91cxetZ.TuW1p2X6OhMF5NHHPhncm 2023-10-11T14:43:36.002 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0P5Kd6eF96YFfY+ioqQEAqzkBlk= 2023-10-11T14:43:36.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0P5Kd6eF96YFfY+ioqQEAqzkBlk=', 'Content-Length': '0'} 2023-10-11T14:43:36.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iV91cxetZ.TuW1p2X6OhMF5NHHPhncm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000646f12ab5eaa95b0-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.005 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.005 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.006 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.006 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.007 DEBUG:boto:Method: GET 2023-10-11T14:43:36.007 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.008 DEBUG:boto:Data: 2023-10-11T14:43:36.008 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.009 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.010 DEBUG:boto:Params: {} 2023-10-11T14:43:36.011 DEBUG:boto:Token: None 2023-10-11T14:43:36.011 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3651dbb19a60fb-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.017 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.019 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.019 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.019 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.020 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.020 DEBUG:boto:Method: GET 2023-10-11T14:43:36.021 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.021 DEBUG:boto:Data: 2023-10-11T14:43:36.022 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.022 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.023 DEBUG:boto:Params: {} 2023-10-11T14:43:36.024 DEBUG:boto:Token: None 2023-10-11T14:43:36.025 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c020f3686617e99f-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.031 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.032 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.032 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.033 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.033 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.033 DEBUG:boto:Data: 2023-10-11T14:43:36.034 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.035 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.036 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.036 DEBUG:boto:Params: {} 2023-10-11T14:43:36.037 DEBUG:boto:Token: None 2023-10-11T14:43:36.037 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.038 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f2ba25636af4011-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.042 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.043 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.043 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.044 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.044 DEBUG:boto:Data: 2023-10-11T14:43:36.045 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.045 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.046 DEBUG:boto:Params: {} 2023-10-11T14:43:36.046 DEBUG:boto:Token: None 2023-10-11T14:43:36.048 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f6a15327d83ac07-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.053 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.054 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.054 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.054 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.055 DEBUG:boto:Method: GET 2023-10-11T14:43:36.055 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.056 DEBUG:boto:Data: 2023-10-11T14:43:36.056 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.057 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.057 DEBUG:boto:Params: {} 2023-10-11T14:43:36.058 DEBUG:boto:Token: None 2023-10-11T14:43:36.058 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.059 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b613662a8efc5f2-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.062 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.062 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.063 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.063 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.064 DEBUG:boto:Method: GET 2023-10-11T14:43:36.064 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.065 DEBUG:boto:Data: 2023-10-11T14:43:36.065 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.066 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.067 DEBUG:boto:Params: {} 2023-10-11T14:43:36.067 DEBUG:boto:Token: None 2023-10-11T14:43:36.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056ff462346afd098-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.071 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.072 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.072 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.073 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.073 DEBUG:boto:Method: GET 2023-10-11T14:43:36.074 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.074 DEBUG:boto:Data: 2023-10-11T14:43:36.075 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.075 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.076 DEBUG:boto:Params: {} 2023-10-11T14:43:36.076 DEBUG:boto:Token: None 2023-10-11T14:43:36.077 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.077 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2d93feb7ea0c54-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.081 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.081 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.082 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.082 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.083 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.083 DEBUG:boto:Method: GET 2023-10-11T14:43:36.084 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.084 DEBUG:boto:Data: 2023-10-11T14:43:36.084 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.085 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.086 DEBUG:boto:Params: {} 2023-10-11T14:43:36.086 DEBUG:boto:Token: None 2023-10-11T14:43:36.087 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf3f14fb3032441-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.091 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.092 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.092 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.093 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.093 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.094 DEBUG:boto:Method: GET 2023-10-11T14:43:36.094 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.094 DEBUG:boto:Data: 2023-10-11T14:43:36.095 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.096 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.096 DEBUG:boto:Params: {} 2023-10-11T14:43:36.097 DEBUG:boto:Token: None 2023-10-11T14:43:36.097 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o= 2023-10-11T14:43:36.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o=', 'Content-Length': '0'} 2023-10-11T14:43:36.100 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000248513f8a7b09816-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.101 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.101 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.101 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.102 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.102 DEBUG:boto:Method: GET 2023-10-11T14:43:36.103 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.103 DEBUG:boto:Data: 2023-10-11T14:43:36.104 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.105 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.105 DEBUG:boto:Params: {} 2023-10-11T14:43:36.105 DEBUG:boto:Token: None 2023-10-11T14:43:36.106 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o= 2023-10-11T14:43:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o=', 'Content-Length': '0'} 2023-10-11T14:43:36.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032ec3d741bbcd70f-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.150 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.151 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.151 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.152 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.152 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.153 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.153 DEBUG:boto:Data: 2023-10-11T14:43:36.153 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.154 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.155 DEBUG:boto:Params: {} 2023-10-11T14:43:36.155 DEBUG:boto:Token: None 2023-10-11T14:43:36.156 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5KFJFFQCT7W2XvKt7XbNmtq2ogA= 2023-10-11T14:43:36.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5KFJFFQCT7W2XvKt7XbNmtq2ogA=', 'Content-Length': '0'} 2023-10-11T14:43:36.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f1f80789dda7625-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.160 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.161 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.161 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.161 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.162 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.162 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.163 DEBUG:boto:Data: 2023-10-11T14:43:36.163 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.164 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.165 DEBUG:boto:Params: {} 2023-10-11T14:43:36.165 DEBUG:boto:Token: None 2023-10-11T14:43:36.166 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.166 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5KFJFFQCT7W2XvKt7XbNmtq2ogA= 2023-10-11T14:43:36.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5KFJFFQCT7W2XvKt7XbNmtq2ogA=', 'Content-Length': '0'} 2023-10-11T14:43:36.169 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb86717e60f44dca-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.170 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.170 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.171 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.171 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.172 DEBUG:boto:Method: GET 2023-10-11T14:43:36.172 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.172 DEBUG:boto:Data: 2023-10-11T14:43:36.173 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.174 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.174 DEBUG:boto:Params: {} 2023-10-11T14:43:36.175 DEBUG:boto:Token: None 2023-10-11T14:43:36.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o= 2023-10-11T14:43:36.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o=', 'Content-Length': '0'} 2023-10-11T14:43:36.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c97a85ddefeeab53-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.179 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.179 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.180 DEBUG:boto:path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.180 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.180 DEBUG:boto:Method: GET 2023-10-11T14:43:36.181 DEBUG:boto:Path: /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.182 DEBUG:boto:Data: 2023-10-11T14:43:36.182 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.183 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.184 DEBUG:boto:Params: {} 2023-10-11T14:43:36.184 DEBUG:boto:Token: None 2023-10-11T14:43:36.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=sEmMWLSV3yOk-qo2EZbn45L5gbVKztD 2023-10-11T14:43:36.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o= 2023-10-11T14:43:36.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUo9v/o3ulBGv5MHAaIylY2ov0o=', 'Content-Length': '0'} 2023-10-11T14:43:36.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'sEmMWLSV3yOk-qo2EZbn45L5gbVKztD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4146cdae19f02be-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.189 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.189 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.190 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.190 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.190 DEBUG:boto:Method: GET 2023-10-11T14:43:36.191 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.191 DEBUG:boto:Data: 2023-10-11T14:43:36.192 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.192 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.193 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.193 DEBUG:boto:Params: {} 2023-10-11T14:43:36.193 DEBUG:boto:Token: None 2023-10-11T14:43:36.194 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.194 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe86a77b7aa47369-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.242 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.243 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.244 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.244 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.245 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.245 DEBUG:boto:Method: GET 2023-10-11T14:43:36.246 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.246 DEBUG:boto:Data: 2023-10-11T14:43:36.247 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.247 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.248 DEBUG:boto:Params: {} 2023-10-11T14:43:36.248 DEBUG:boto:Token: None 2023-10-11T14:43:36.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098c0aace3f1b736b-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.253 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.254 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.255 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.255 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.255 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.256 DEBUG:boto:Data: 2023-10-11T14:43:36.257 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.258 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.258 DEBUG:boto:Params: {} 2023-10-11T14:43:36.258 DEBUG:boto:Token: None 2023-10-11T14:43:36.259 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeae40d5298f8387-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.264 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.266 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.266 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.267 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.267 DEBUG:boto:Data: 2023-10-11T14:43:36.267 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.268 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.269 DEBUG:boto:Params: {} 2023-10-11T14:43:36.270 DEBUG:boto:Token: None 2023-10-11T14:43:36.270 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b71fbf94ae396919-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.276 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.276 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.277 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.277 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.278 DEBUG:boto:Method: GET 2023-10-11T14:43:36.278 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.279 DEBUG:boto:Data: 2023-10-11T14:43:36.279 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.280 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.281 DEBUG:boto:Params: {} 2023-10-11T14:43:36.281 DEBUG:boto:Token: None 2023-10-11T14:43:36.282 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.285 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000938ce4abbfc0809c-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.286 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.286 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.287 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.288 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.288 DEBUG:boto:Method: GET 2023-10-11T14:43:36.289 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.289 DEBUG:boto:Data: 2023-10-11T14:43:36.290 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.291 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.291 DEBUG:boto:Params: {} 2023-10-11T14:43:36.292 DEBUG:boto:Token: None 2023-10-11T14:43:36.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ecd3e1b6e1c67b1-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.296 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.297 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.297 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.297 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.298 DEBUG:boto:Method: GET 2023-10-11T14:43:36.298 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.299 DEBUG:boto:Data: 2023-10-11T14:43:36.299 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.300 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.300 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.301 DEBUG:boto:Params: {} 2023-10-11T14:43:36.301 DEBUG:boto:Token: None 2023-10-11T14:43:36.302 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000232d9e7683471922-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.306 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.307 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.307 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.308 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.308 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.309 DEBUG:boto:Method: GET 2023-10-11T14:43:36.309 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.310 DEBUG:boto:Data: 2023-10-11T14:43:36.310 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.311 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.311 DEBUG:boto:Params: {} 2023-10-11T14:43:36.312 DEBUG:boto:Token: None 2023-10-11T14:43:36.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e91c304acfb010-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.363 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.364 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.364 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.365 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.365 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.366 DEBUG:boto:Method: GET 2023-10-11T14:43:36.366 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.367 DEBUG:boto:Data: 2023-10-11T14:43:36.367 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.368 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.368 DEBUG:boto:Params: {} 2023-10-11T14:43:36.369 DEBUG:boto:Token: None 2023-10-11T14:43:36.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI= 2023-10-11T14:43:36.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI=', 'Content-Length': '0'} 2023-10-11T14:43:36.372 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b42dca1d83b6ce9-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.373 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.373 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.374 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.375 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.375 DEBUG:boto:Method: GET 2023-10-11T14:43:36.375 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.376 DEBUG:boto:Data: 2023-10-11T14:43:36.376 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.378 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.379 DEBUG:boto:Params: {} 2023-10-11T14:43:36.379 DEBUG:boto:Token: None 2023-10-11T14:43:36.380 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI= 2023-10-11T14:43:36.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI=', 'Content-Length': '0'} 2023-10-11T14:43:36.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000746c5c9d1ab91db-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.383 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.384 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.385 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.385 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.386 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.386 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.387 DEBUG:boto:Data: 2023-10-11T14:43:36.387 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.389 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.389 DEBUG:boto:Params: {} 2023-10-11T14:43:36.390 DEBUG:boto:Token: None 2023-10-11T14:43:36.391 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qAGzOevnlh6Q13/jse4+VLyEzeg= 2023-10-11T14:43:36.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qAGzOevnlh6Q13/jse4+VLyEzeg=', 'Content-Length': '0'} 2023-10-11T14:43:36.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9d6dafa2a2d4e51-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.394 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.395 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.395 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.396 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.396 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.396 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.397 DEBUG:boto:Data: 2023-10-11T14:43:36.397 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.398 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.398 DEBUG:boto:Params: {} 2023-10-11T14:43:36.399 DEBUG:boto:Token: None 2023-10-11T14:43:36.399 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qAGzOevnlh6Q13/jse4+VLyEzeg= 2023-10-11T14:43:36.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qAGzOevnlh6Q13/jse4+VLyEzeg=', 'Content-Length': '0'} 2023-10-11T14:43:36.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b193a88de3fb61b-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.404 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.405 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.405 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.406 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.407 DEBUG:boto:Method: GET 2023-10-11T14:43:36.408 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.408 DEBUG:boto:Data: 2023-10-11T14:43:36.409 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.410 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.411 DEBUG:boto:Params: {} 2023-10-11T14:43:36.412 DEBUG:boto:Token: None 2023-10-11T14:43:36.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.413 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI= 2023-10-11T14:43:36.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI=', 'Content-Length': '0'} 2023-10-11T14:43:36.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e8c337552dcc3eb-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.416 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.417 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.417 DEBUG:boto:path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.418 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.418 DEBUG:boto:Method: GET 2023-10-11T14:43:36.419 DEBUG:boto:Path: /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.419 DEBUG:boto:Data: 2023-10-11T14:43:36.419 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.420 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.421 DEBUG:boto:Params: {} 2023-10-11T14:43:36.421 DEBUG:boto:Token: None 2023-10-11T14:43:36.422 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh 2023-10-11T14:43:36.422 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI= 2023-10-11T14:43:36.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrFT0gSs77vy1xUn3fGZfzjUQdI=', 'Content-Length': '0'} 2023-10-11T14:43:36.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'FwvPjusD13V5DqHCMJ1Pz3g4uLrBghh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e07d3e67fb26b8d1-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.470 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.471 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.471 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.472 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.473 DEBUG:boto:Method: GET 2023-10-11T14:43:36.473 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.474 DEBUG:boto:Data: 2023-10-11T14:43:36.475 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.476 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.477 DEBUG:boto:Params: {} 2023-10-11T14:43:36.477 DEBUG:boto:Token: None 2023-10-11T14:43:36.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000331f5b47d2c6f28c-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.483 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.484 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.484 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.485 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.485 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.485 DEBUG:boto:Method: GET 2023-10-11T14:43:36.486 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.486 DEBUG:boto:Data: 2023-10-11T14:43:36.487 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.488 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.488 DEBUG:boto:Params: {} 2023-10-11T14:43:36.489 DEBUG:boto:Token: None 2023-10-11T14:43:36.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bbb84277910881-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.495 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.496 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.497 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.497 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.498 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.499 DEBUG:boto:Data: 2023-10-11T14:43:36.500 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.501 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.502 DEBUG:boto:Params: {} 2023-10-11T14:43:36.503 DEBUG:boto:Token: None 2023-10-11T14:43:36.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000256dd52005cfa4f0-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.510 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.511 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.511 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.512 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.513 DEBUG:boto:Data: 2023-10-11T14:43:36.513 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.514 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.515 DEBUG:boto:Params: {} 2023-10-11T14:43:36.516 DEBUG:boto:Token: None 2023-10-11T14:43:36.516 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c56f6c5616a8addd-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.522 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.523 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.524 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.525 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.525 DEBUG:boto:Method: GET 2023-10-11T14:43:36.526 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.527 DEBUG:boto:Data: 2023-10-11T14:43:36.529 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.531 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.531 DEBUG:boto:Params: {} 2023-10-11T14:43:36.532 DEBUG:boto:Token: None 2023-10-11T14:43:36.533 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b98ccc3eaf3a12ba-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.578 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.579 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.580 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.580 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.580 DEBUG:boto:Method: GET 2023-10-11T14:43:36.581 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.581 DEBUG:boto:Data: 2023-10-11T14:43:36.582 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.583 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.584 DEBUG:boto:Params: {} 2023-10-11T14:43:36.585 DEBUG:boto:Token: None 2023-10-11T14:43:36.586 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.587 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd3fdfbc03f7435d-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.591 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.591 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.592 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.592 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.593 DEBUG:boto:Method: GET 2023-10-11T14:43:36.593 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.594 DEBUG:boto:Data: 2023-10-11T14:43:36.595 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.596 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.597 DEBUG:boto:Params: {} 2023-10-11T14:43:36.598 DEBUG:boto:Token: None 2023-10-11T14:43:36.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000337df1f9553b56bc-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.604 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.606 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.606 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.607 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.608 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.609 DEBUG:boto:Method: GET 2023-10-11T14:43:36.609 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.610 DEBUG:boto:Data: 2023-10-11T14:43:36.611 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.613 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.613 DEBUG:boto:Params: {} 2023-10-11T14:43:36.614 DEBUG:boto:Token: None 2023-10-11T14:43:36.615 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ded08ca72048a61-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.621 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.623 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.624 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.624 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.625 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.626 DEBUG:boto:Method: GET 2023-10-11T14:43:36.627 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.627 DEBUG:boto:Data: 2023-10-11T14:43:36.628 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.629 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.630 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.630 DEBUG:boto:Params: {} 2023-10-11T14:43:36.631 DEBUG:boto:Token: None 2023-10-11T14:43:36.632 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.633 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ= 2023-10-11T14:43:36.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ=', 'Content-Length': '0'} 2023-10-11T14:43:36.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000675bb23eccb933d3-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.637 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.637 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.638 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.638 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.639 DEBUG:boto:Method: GET 2023-10-11T14:43:36.639 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.640 DEBUG:boto:Data: 2023-10-11T14:43:36.640 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.641 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.641 DEBUG:boto:Params: {} 2023-10-11T14:43:36.642 DEBUG:boto:Token: None 2023-10-11T14:43:36.642 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.643 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ= 2023-10-11T14:43:36.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ=', 'Content-Length': '0'} 2023-10-11T14:43:36.645 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000109f31124420f98c-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.646 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.647 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.647 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.647 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.648 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.648 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.649 DEBUG:boto:Data: 2023-10-11T14:43:36.649 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.650 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.650 DEBUG:boto:Params: {} 2023-10-11T14:43:36.651 DEBUG:boto:Token: None 2023-10-11T14:43:36.651 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2a4Ij7pfzla4PzAzxN4+HSXf0/4= 2023-10-11T14:43:36.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2a4Ij7pfzla4PzAzxN4+HSXf0/4=', 'Content-Length': '0'} 2023-10-11T14:43:36.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5b29c608ba9f0ab-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.655 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.656 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.657 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.657 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.658 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.659 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.659 DEBUG:boto:Data: 2023-10-11T14:43:36.660 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.662 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.662 DEBUG:boto:Params: {} 2023-10-11T14:43:36.663 DEBUG:boto:Token: None 2023-10-11T14:43:36.664 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2a4Ij7pfzla4PzAzxN4+HSXf0/4= 2023-10-11T14:43:36.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2a4Ij7pfzla4PzAzxN4+HSXf0/4=', 'Content-Length': '0'} 2023-10-11T14:43:36.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003156d0f7b007c3f2-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.667 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.668 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.668 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.669 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.669 DEBUG:boto:Method: GET 2023-10-11T14:43:36.670 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.670 DEBUG:boto:Data: 2023-10-11T14:43:36.670 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.671 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.672 DEBUG:boto:Params: {} 2023-10-11T14:43:36.672 DEBUG:boto:Token: None 2023-10-11T14:43:36.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ= 2023-10-11T14:43:36.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ=', 'Content-Length': '0'} 2023-10-11T14:43:36.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e41b26ecd4c801f9-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.718 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.719 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.719 DEBUG:boto:path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.720 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.720 DEBUG:boto:Method: GET 2023-10-11T14:43:36.721 DEBUG:boto:Path: /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.721 DEBUG:boto:Data: 2023-10-11T14:43:36.722 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.722 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.723 DEBUG:boto:Params: {} 2023-10-11T14:43:36.723 DEBUG:boto:Token: None 2023-10-11T14:43:36.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy 2023-10-11T14:43:36.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ= 2023-10-11T14:43:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qJsMw9ejcPtkZfc1bAEF1xBELZQ=', 'Content-Length': '0'} 2023-10-11T14:43:36.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'gmNHLswwMCcYpP-.C9uSLG-aq2BsTCy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f65892a688cc46a-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.728 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.728 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.728 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.729 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.729 DEBUG:boto:Method: GET 2023-10-11T14:43:36.730 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.730 DEBUG:boto:Data: 2023-10-11T14:43:36.731 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.732 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.733 DEBUG:boto:Params: {} 2023-10-11T14:43:36.734 DEBUG:boto:Token: None 2023-10-11T14:43:36.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007177d4d31e8d3a9f-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.738 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.739 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.740 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.740 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.741 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.741 DEBUG:boto:Method: GET 2023-10-11T14:43:36.741 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.742 DEBUG:boto:Data: 2023-10-11T14:43:36.742 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.743 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.743 DEBUG:boto:Params: {} 2023-10-11T14:43:36.744 DEBUG:boto:Token: None 2023-10-11T14:43:36.744 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a08dd5527d2b5193-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.794 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.795 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.796 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.796 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.797 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.797 DEBUG:boto:Data: 2023-10-11T14:43:36.797 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.798 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.799 DEBUG:boto:Params: {} 2023-10-11T14:43:36.799 DEBUG:boto:Token: None 2023-10-11T14:43:36.800 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ec879e85c558785-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.806 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.807 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.807 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.808 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:36.808 DEBUG:boto:Data: 2023-10-11T14:43:36.808 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.809 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.810 DEBUG:boto:Params: {} 2023-10-11T14:43:36.810 DEBUG:boto:Token: None 2023-10-11T14:43:36.811 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj 2023-10-11T14:43:36.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk= 2023-10-11T14:43:36.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0Of99CZd/iYtlG2edB8lxBfyTk=', 'Content-Length': '0'} 2023-10-11T14:43:36.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b67f4ed866285c77-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.816 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.817 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.818 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.819 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.819 DEBUG:boto:Method: GET 2023-10-11T14:43:36.819 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.820 DEBUG:boto:Data: 2023-10-11T14:43:36.820 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.821 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.822 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.822 DEBUG:boto:Params: {} 2023-10-11T14:43:36.823 DEBUG:boto:Token: None 2023-10-11T14:43:36.823 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.824 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edfb10f944461937-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.827 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.829 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.829 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.830 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.830 DEBUG:boto:Method: GET 2023-10-11T14:43:36.830 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.831 DEBUG:boto:Data: 2023-10-11T14:43:36.831 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.832 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.833 DEBUG:boto:Params: {} 2023-10-11T14:43:36.834 DEBUG:boto:Token: None 2023-10-11T14:43:36.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:36.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws= 2023-10-11T14:43:36.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSKSOj/a0Yo00lCKKspY2Hcxws=', 'Content-Length': '0'} 2023-10-11T14:43:36.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff3e3d2ba14f0aab-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.840 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.841 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.841 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.841 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.842 DEBUG:boto:Method: GET 2023-10-11T14:43:36.842 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.843 DEBUG:boto:Data: 2023-10-11T14:43:36.843 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.844 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.845 DEBUG:boto:Params: {} 2023-10-11T14:43:36.845 DEBUG:boto:Token: None 2023-10-11T14:43:36.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.846 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d366b626ac21da-006526b498-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.890 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.891 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:36.892 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:36.892 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.892 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:36.893 DEBUG:boto:Method: GET 2023-10-11T14:43:36.893 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:36.894 DEBUG:boto:Data: 2023-10-11T14:43:36.894 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.895 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.896 DEBUG:boto:Params: {} 2023-10-11T14:43:36.896 DEBUG:boto:Token: None 2023-10-11T14:43:36.897 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/ 2023-10-11T14:43:36.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18= 2023-10-11T14:43:36.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1GILqwIMDzv659Yo/uinWxzd18=', 'Content-Length': '0'} 2023-10-11T14:43:36.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022dfcdee9b133982-006526b498-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.901 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:36.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.902 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.903 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.903 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.903 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.904 DEBUG:boto:Method: GET 2023-10-11T14:43:36.904 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.905 DEBUG:boto:Data: 2023-10-11T14:43:36.905 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.905 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.906 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.906 DEBUG:boto:Params: {} 2023-10-11T14:43:36.907 DEBUG:boto:Token: None 2023-10-11T14:43:36.907 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw= 2023-10-11T14:43:36.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw=', 'Content-Length': '0'} 2023-10-11T14:43:36.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a84f6c104b442674-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.911 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.911 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.912 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.912 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.913 DEBUG:boto:Method: GET 2023-10-11T14:43:36.913 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.914 DEBUG:boto:Data: 2023-10-11T14:43:36.914 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.915 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.915 DEBUG:boto:Params: {} 2023-10-11T14:43:36.916 DEBUG:boto:Token: None 2023-10-11T14:43:36.916 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.917 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw= 2023-10-11T14:43:36.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw=', 'Content-Length': '0'} 2023-10-11T14:43:36.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055db8565a6b8de80-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.920 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.921 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.921 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.922 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.922 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.922 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.923 DEBUG:boto:Data: 2023-10-11T14:43:36.923 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.924 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.924 DEBUG:boto:Params: {} 2023-10-11T14:43:36.925 DEBUG:boto:Token: None 2023-10-11T14:43:36.925 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.926 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R0QoSGjsPld5TCoq0Wl1C8+GDQE= 2023-10-11T14:43:36.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R0QoSGjsPld5TCoq0Wl1C8+GDQE=', 'Content-Length': '0'} 2023-10-11T14:43:36.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a841ee546006dbb4-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.929 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.929 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.930 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.930 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.931 DEBUG:boto:Method: HEAD 2023-10-11T14:43:36.933 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.933 DEBUG:boto:Data: 2023-10-11T14:43:36.934 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.935 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.935 DEBUG:boto:Params: {} 2023-10-11T14:43:36.936 DEBUG:boto:Token: None 2023-10-11T14:43:36.936 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R0QoSGjsPld5TCoq0Wl1C8+GDQE= 2023-10-11T14:43:36.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R0QoSGjsPld5TCoq0Wl1C8+GDQE=', 'Content-Length': '0'} 2023-10-11T14:43:36.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000617b91d5e95a1472-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:36.941 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.941 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.942 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.942 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.943 DEBUG:boto:Method: GET 2023-10-11T14:43:36.943 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.944 DEBUG:boto:Data: 2023-10-11T14:43:36.944 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:36.946 DEBUG:boto:Port: 8001 2023-10-11T14:43:36.946 DEBUG:boto:Params: {} 2023-10-11T14:43:36.946 DEBUG:boto:Token: None 2023-10-11T14:43:36.947 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw= 2023-10-11T14:43:36.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw=', 'Content-Length': '0'} 2023-10-11T14:43:36.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018039e734b21422b-006526b498-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:36.951 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:36.952 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:36.953 DEBUG:boto:path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.953 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.953 DEBUG:boto:Method: GET 2023-10-11T14:43:36.954 DEBUG:boto:Path: /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.954 DEBUG:boto:Data: 2023-10-11T14:43:36.955 DEBUG:boto:Headers: {} 2023-10-11T14:43:36.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:36.956 DEBUG:boto:Port: 8000 2023-10-11T14:43:36.956 DEBUG:boto:Params: {} 2023-10-11T14:43:36.957 DEBUG:boto:Token: None 2023-10-11T14:43:36.957 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:36 GMT /dghqvd-13/obj?versionId=CPjUmICKQIYvCqxTqzrmqcmRuLODPGR 2023-10-11T14:43:36.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw= 2023-10-11T14:43:36.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4E8+afnhQarkmCfW94K5iiPj7sw=', 'Content-Length': '0'} 2023-10-11T14:43:36.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'CPjUmICKQIYvCqxTqzrmqcmRuLODPGR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088b0153b1dc5abcd-006526b498-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.002 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.003 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.004 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.004 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.004 DEBUG:boto:Method: GET 2023-10-11T14:43:37.005 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.005 DEBUG:boto:Data: 2023-10-11T14:43:37.006 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.007 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.007 DEBUG:boto:Params: {} 2023-10-11T14:43:37.008 DEBUG:boto:Token: None 2023-10-11T14:43:37.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cebf7b483f72df5e-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.012 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.013 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.014 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.014 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.015 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.015 DEBUG:boto:Method: GET 2023-10-11T14:43:37.015 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.016 DEBUG:boto:Data: 2023-10-11T14:43:37.016 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.017 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.017 DEBUG:boto:Params: {} 2023-10-11T14:43:37.018 DEBUG:boto:Token: None 2023-10-11T14:43:37.019 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d6830db511fd85b-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.023 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.023 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.024 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.024 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.025 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.025 DEBUG:boto:Data: 2023-10-11T14:43:37.026 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.026 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.027 DEBUG:boto:Params: {} 2023-10-11T14:43:37.027 DEBUG:boto:Token: None 2023-10-11T14:43:37.028 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000579b6285d6566d3-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.033 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.033 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.034 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.034 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.035 DEBUG:boto:Data: 2023-10-11T14:43:37.035 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.036 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.037 DEBUG:boto:Params: {} 2023-10-11T14:43:37.037 DEBUG:boto:Token: None 2023-10-11T14:43:37.037 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.038 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eaefd93e2583490a-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.043 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.043 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.044 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.044 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.045 DEBUG:boto:Method: GET 2023-10-11T14:43:37.045 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.046 DEBUG:boto:Data: 2023-10-11T14:43:37.046 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.047 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.047 DEBUG:boto:Params: {} 2023-10-11T14:43:37.048 DEBUG:boto:Token: None 2023-10-11T14:43:37.048 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002031d2d7cd4bdd1c-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.098 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.099 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.099 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.100 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.100 DEBUG:boto:Method: GET 2023-10-11T14:43:37.101 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.101 DEBUG:boto:Data: 2023-10-11T14:43:37.102 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.103 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.103 DEBUG:boto:Params: {} 2023-10-11T14:43:37.104 DEBUG:boto:Token: None 2023-10-11T14:43:37.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdc326cc3746ac73-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.154 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.155 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.155 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.156 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.156 DEBUG:boto:Method: GET 2023-10-11T14:43:37.157 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.157 DEBUG:boto:Data: 2023-10-11T14:43:37.158 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.159 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.159 DEBUG:boto:Params: {} 2023-10-11T14:43:37.160 DEBUG:boto:Token: None 2023-10-11T14:43:37.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55287c32557d523-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.165 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.166 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.166 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.166 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.167 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.167 DEBUG:boto:Method: GET 2023-10-11T14:43:37.168 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.168 DEBUG:boto:Data: 2023-10-11T14:43:37.169 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.169 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.170 DEBUG:boto:Params: {} 2023-10-11T14:43:37.170 DEBUG:boto:Token: None 2023-10-11T14:43:37.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae149d8edebbdb9-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.218 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.220 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.220 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.220 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.221 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.221 DEBUG:boto:Method: GET 2023-10-11T14:43:37.222 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.222 DEBUG:boto:Data: 2023-10-11T14:43:37.223 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.223 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.223 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.224 DEBUG:boto:Params: {} 2023-10-11T14:43:37.224 DEBUG:boto:Token: None 2023-10-11T14:43:37.225 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.225 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o= 2023-10-11T14:43:37.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o=', 'Content-Length': '0'} 2023-10-11T14:43:37.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000292fab5ecc79925b-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.229 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.229 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.230 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.230 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.230 DEBUG:boto:Method: GET 2023-10-11T14:43:37.231 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.231 DEBUG:boto:Data: 2023-10-11T14:43:37.232 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.233 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.233 DEBUG:boto:Params: {} 2023-10-11T14:43:37.233 DEBUG:boto:Token: None 2023-10-11T14:43:37.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o= 2023-10-11T14:43:37.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o=', 'Content-Length': '0'} 2023-10-11T14:43:37.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e970e1f23e43e789-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.238 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.239 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.239 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.239 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.240 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.240 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.241 DEBUG:boto:Data: 2023-10-11T14:43:37.241 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.242 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.243 DEBUG:boto:Params: {} 2023-10-11T14:43:37.243 DEBUG:boto:Token: None 2023-10-11T14:43:37.244 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h+3YYYVZJBOfW8HE2BZL7B75sx8= 2023-10-11T14:43:37.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h+3YYYVZJBOfW8HE2BZL7B75sx8=', 'Content-Length': '0'} 2023-10-11T14:43:37.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8f9b0c41da525c3-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.248 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.248 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.249 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.249 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.249 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.250 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.250 DEBUG:boto:Data: 2023-10-11T14:43:37.251 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.252 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.252 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.253 DEBUG:boto:Params: {} 2023-10-11T14:43:37.253 DEBUG:boto:Token: None 2023-10-11T14:43:37.253 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.254 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h+3YYYVZJBOfW8HE2BZL7B75sx8= 2023-10-11T14:43:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h+3YYYVZJBOfW8HE2BZL7B75sx8=', 'Content-Length': '0'} 2023-10-11T14:43:37.256 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006feb3c86b9fa0d6b-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.258 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.258 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.259 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.259 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.259 DEBUG:boto:Method: GET 2023-10-11T14:43:37.260 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.260 DEBUG:boto:Data: 2023-10-11T14:43:37.261 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.261 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.262 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.262 DEBUG:boto:Params: {} 2023-10-11T14:43:37.263 DEBUG:boto:Token: None 2023-10-11T14:43:37.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o= 2023-10-11T14:43:37.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o=', 'Content-Length': '0'} 2023-10-11T14:43:37.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000039a01f2c983d86c4-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.310 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.311 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.311 DEBUG:boto:path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.312 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.312 DEBUG:boto:Method: GET 2023-10-11T14:43:37.313 DEBUG:boto:Path: /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.313 DEBUG:boto:Data: 2023-10-11T14:43:37.314 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.316 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.316 DEBUG:boto:Params: {} 2023-10-11T14:43:37.317 DEBUG:boto:Token: None 2023-10-11T14:43:37.317 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8 2023-10-11T14:43:37.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o= 2023-10-11T14:43:37.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LpgA5EKp2D6CEbg2bDQdB3j4d/o=', 'Content-Length': '0'} 2023-10-11T14:43:37.321 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'JZa50GpVI1vCx28Kb5-tNsEeGXK-dS8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013d86b00fb50c74b-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.321 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.322 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.322 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.323 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.323 DEBUG:boto:Method: GET 2023-10-11T14:43:37.324 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.324 DEBUG:boto:Data: 2023-10-11T14:43:37.325 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.326 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.326 DEBUG:boto:Params: {} 2023-10-11T14:43:37.327 DEBUG:boto:Token: None 2023-10-11T14:43:37.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b03509b815ec7d-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.332 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.332 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.333 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.333 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.334 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.334 DEBUG:boto:Method: GET 2023-10-11T14:43:37.335 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.335 DEBUG:boto:Data: 2023-10-11T14:43:37.336 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.337 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.337 DEBUG:boto:Params: {} 2023-10-11T14:43:37.338 DEBUG:boto:Token: None 2023-10-11T14:43:37.338 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b45af9fe303c4541-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.346 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.347 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.347 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.348 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.348 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.349 DEBUG:boto:Data: 2023-10-11T14:43:37.349 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.350 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.350 DEBUG:boto:Params: {} 2023-10-11T14:43:37.351 DEBUG:boto:Token: None 2023-10-11T14:43:37.352 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f13e270b89f5f569-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.357 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.357 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.358 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.358 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.359 DEBUG:boto:Data: 2023-10-11T14:43:37.359 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.360 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.360 DEBUG:boto:Params: {} 2023-10-11T14:43:37.361 DEBUG:boto:Token: None 2023-10-11T14:43:37.361 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c2820a54c4f95e0-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.367 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.367 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.368 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.368 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.369 DEBUG:boto:Method: GET 2023-10-11T14:43:37.369 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.370 DEBUG:boto:Data: 2023-10-11T14:43:37.370 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.371 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.371 DEBUG:boto:Params: {} 2023-10-11T14:43:37.372 DEBUG:boto:Token: None 2023-10-11T14:43:37.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003807bdd628626046-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.423 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.424 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.424 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.424 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.425 DEBUG:boto:Method: GET 2023-10-11T14:43:37.425 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.426 DEBUG:boto:Data: 2023-10-11T14:43:37.426 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.427 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.427 DEBUG:boto:Params: {} 2023-10-11T14:43:37.428 DEBUG:boto:Token: None 2023-10-11T14:43:37.428 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.432 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ba28219658d6856-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.478 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.479 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.480 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.480 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.480 DEBUG:boto:Method: GET 2023-10-11T14:43:37.481 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.481 DEBUG:boto:Data: 2023-10-11T14:43:37.482 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.484 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.484 DEBUG:boto:Params: {} 2023-10-11T14:43:37.485 DEBUG:boto:Token: None 2023-10-11T14:43:37.485 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.486 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a900d434c6d6ef37-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.490 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.491 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.492 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.493 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.493 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.493 DEBUG:boto:Method: GET 2023-10-11T14:43:37.494 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.495 DEBUG:boto:Data: 2023-10-11T14:43:37.495 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.497 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.497 DEBUG:boto:Params: {} 2023-10-11T14:43:37.498 DEBUG:boto:Token: None 2023-10-11T14:43:37.499 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071edd64f236c1cf0-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.503 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.504 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.504 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.505 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.505 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.506 DEBUG:boto:Method: GET 2023-10-11T14:43:37.506 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.507 DEBUG:boto:Data: 2023-10-11T14:43:37.507 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.508 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.508 DEBUG:boto:Params: {} 2023-10-11T14:43:37.509 DEBUG:boto:Token: None 2023-10-11T14:43:37.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.510 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU= 2023-10-11T14:43:37.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU=', 'Content-Length': '0'} 2023-10-11T14:43:37.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009865e441b14428f1-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.513 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.514 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.514 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.515 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.515 DEBUG:boto:Method: GET 2023-10-11T14:43:37.516 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.516 DEBUG:boto:Data: 2023-10-11T14:43:37.516 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.517 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.518 DEBUG:boto:Params: {} 2023-10-11T14:43:37.518 DEBUG:boto:Token: None 2023-10-11T14:43:37.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU= 2023-10-11T14:43:37.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU=', 'Content-Length': '0'} 2023-10-11T14:43:37.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6092f8951fee8c0-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.566 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.567 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.567 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.568 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.568 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.569 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.569 DEBUG:boto:Data: 2023-10-11T14:43:37.569 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.570 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.571 DEBUG:boto:Params: {} 2023-10-11T14:43:37.571 DEBUG:boto:Token: None 2023-10-11T14:43:37.572 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JeL7ZAQa/1Ktjvp4wuAZbRa8WjE= 2023-10-11T14:43:37.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JeL7ZAQa/1Ktjvp4wuAZbRa8WjE=', 'Content-Length': '0'} 2023-10-11T14:43:37.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a72ae40d4540bc1-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.575 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.576 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.576 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.576 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.577 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.577 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.578 DEBUG:boto:Data: 2023-10-11T14:43:37.578 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.579 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.579 DEBUG:boto:Params: {} 2023-10-11T14:43:37.580 DEBUG:boto:Token: None 2023-10-11T14:43:37.580 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JeL7ZAQa/1Ktjvp4wuAZbRa8WjE= 2023-10-11T14:43:37.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JeL7ZAQa/1Ktjvp4wuAZbRa8WjE=', 'Content-Length': '0'} 2023-10-11T14:43:37.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085df1213d8bd88fe-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.585 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.585 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.586 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.586 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.587 DEBUG:boto:Method: GET 2023-10-11T14:43:37.587 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.588 DEBUG:boto:Data: 2023-10-11T14:43:37.588 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.589 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.589 DEBUG:boto:Params: {} 2023-10-11T14:43:37.590 DEBUG:boto:Token: None 2023-10-11T14:43:37.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU= 2023-10-11T14:43:37.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU=', 'Content-Length': '0'} 2023-10-11T14:43:37.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000368a2559750c539c-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.595 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.596 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.596 DEBUG:boto:path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.597 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.597 DEBUG:boto:Method: GET 2023-10-11T14:43:37.598 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.598 DEBUG:boto:Data: 2023-10-11T14:43:37.599 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.600 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.600 DEBUG:boto:Params: {} 2023-10-11T14:43:37.601 DEBUG:boto:Token: None 2023-10-11T14:43:37.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=2wgGYzfzykE51dH4B1WhGC.VF8my1wR 2023-10-11T14:43:37.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU= 2023-10-11T14:43:37.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeICtL1MRj4+zJ2gWXpSHmm5vcU=', 'Content-Length': '0'} 2023-10-11T14:43:37.604 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2wgGYzfzykE51dH4B1WhGC.VF8my1wR'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000327ad4aa72556fdc-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.646 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.647 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.647 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.648 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.648 DEBUG:boto:Method: GET 2023-10-11T14:43:37.649 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.649 DEBUG:boto:Data: 2023-10-11T14:43:37.650 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.650 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.651 DEBUG:boto:Params: {} 2023-10-11T14:43:37.651 DEBUG:boto:Token: None 2023-10-11T14:43:37.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003723888f1f27623-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.656 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.657 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.658 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.658 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.658 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.659 DEBUG:boto:Method: GET 2023-10-11T14:43:37.659 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.660 DEBUG:boto:Data: 2023-10-11T14:43:37.660 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.661 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.662 DEBUG:boto:Params: {} 2023-10-11T14:43:37.662 DEBUG:boto:Token: None 2023-10-11T14:43:37.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e64d443e1c4887b-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.714 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.715 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.716 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.716 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.716 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.717 DEBUG:boto:Data: 2023-10-11T14:43:37.717 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.718 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.718 DEBUG:boto:Params: {} 2023-10-11T14:43:37.720 DEBUG:boto:Token: None 2023-10-11T14:43:37.720 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.725 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0b2ce9533694c37-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.725 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.726 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.726 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.727 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:37.727 DEBUG:boto:Data: 2023-10-11T14:43:37.727 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.728 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.729 DEBUG:boto:Params: {} 2023-10-11T14:43:37.729 DEBUG:boto:Token: None 2023-10-11T14:43:37.730 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj 2023-10-11T14:43:37.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs= 2023-10-11T14:43:37.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4LdQvrBE0965RQV8RnFit0U/wzs=', 'Content-Length': '0'} 2023-10-11T14:43:37.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003f12fe31f391d472-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.735 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.735 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.736 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.736 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.736 DEBUG:boto:Method: GET 2023-10-11T14:43:37.737 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.737 DEBUG:boto:Data: 2023-10-11T14:43:37.738 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.739 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.739 DEBUG:boto:Params: {} 2023-10-11T14:43:37.739 DEBUG:boto:Token: None 2023-10-11T14:43:37.740 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.743 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000283b95059e7b90a7-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.786 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.787 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.788 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.788 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.789 DEBUG:boto:Method: GET 2023-10-11T14:43:37.789 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.790 DEBUG:boto:Data: 2023-10-11T14:43:37.790 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.791 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.791 DEBUG:boto:Params: {} 2023-10-11T14:43:37.792 DEBUG:boto:Token: None 2023-10-11T14:43:37.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:37.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE= 2023-10-11T14:43:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WB4hmNbApB5BdAPAMyY3Fs5hgnE=', 'Content-Length': '0'} 2023-10-11T14:43:37.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000adbc4de3144c6c03-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.838 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.839 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.839 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.840 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.840 DEBUG:boto:Method: GET 2023-10-11T14:43:37.841 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.841 DEBUG:boto:Data: 2023-10-11T14:43:37.842 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.843 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.843 DEBUG:boto:Params: {} 2023-10-11T14:43:37.844 DEBUG:boto:Token: None 2023-10-11T14:43:37.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345ab7f08e59f604-006526b499-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.849 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.849 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:37.850 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:37.850 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.851 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:37.851 DEBUG:boto:Method: GET 2023-10-11T14:43:37.852 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:37.852 DEBUG:boto:Data: 2023-10-11T14:43:37.853 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.853 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.854 DEBUG:boto:Params: {} 2023-10-11T14:43:37.854 DEBUG:boto:Token: None 2023-10-11T14:43:37.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/ 2023-10-11T14:43:37.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI= 2023-10-11T14:43:37.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jWU7yGJzjxR2HxCvR6P4SI6GClI=', 'Content-Length': '0'} 2023-10-11T14:43:37.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035491bd2d05fbdf2-006526b499-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.860 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:37.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.861 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.862 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.862 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.863 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.863 DEBUG:boto:Method: GET 2023-10-11T14:43:37.863 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.864 DEBUG:boto:Data: 2023-10-11T14:43:37.864 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.865 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.866 DEBUG:boto:Params: {} 2023-10-11T14:43:37.866 DEBUG:boto:Token: None 2023-10-11T14:43:37.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4= 2023-10-11T14:43:37.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4=', 'Content-Length': '0'} 2023-10-11T14:43:37.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072795c44b14d76f6-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.871 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.871 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.872 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.872 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.873 DEBUG:boto:Method: GET 2023-10-11T14:43:37.873 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.874 DEBUG:boto:Data: 2023-10-11T14:43:37.874 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.875 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.875 DEBUG:boto:Params: {} 2023-10-11T14:43:37.876 DEBUG:boto:Token: None 2023-10-11T14:43:37.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4= 2023-10-11T14:43:37.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4=', 'Content-Length': '0'} 2023-10-11T14:43:37.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b07f5fbcb2d1e437-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.922 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.923 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.924 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.924 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.924 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.925 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.925 DEBUG:boto:Data: 2023-10-11T14:43:37.926 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.926 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.927 DEBUG:boto:Params: {} 2023-10-11T14:43:37.927 DEBUG:boto:Token: None 2023-10-11T14:43:37.928 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oXMmv+S/2AZYkyU2BWFq7TL/Vvs= 2023-10-11T14:43:37.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oXMmv+S/2AZYkyU2BWFq7TL/Vvs=', 'Content-Length': '0'} 2023-10-11T14:43:37.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d16e3b2cf2288951-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.932 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.933 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.933 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.934 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.934 DEBUG:boto:Method: HEAD 2023-10-11T14:43:37.935 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.935 DEBUG:boto:Data: 2023-10-11T14:43:37.936 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:37.937 DEBUG:boto:Port: 8000 2023-10-11T14:43:37.937 DEBUG:boto:Params: {} 2023-10-11T14:43:37.938 DEBUG:boto:Token: None 2023-10-11T14:43:37.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oXMmv+S/2AZYkyU2BWFq7TL/Vvs= 2023-10-11T14:43:37.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oXMmv+S/2AZYkyU2BWFq7TL/Vvs=', 'Content-Length': '0'} 2023-10-11T14:43:37.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bb4d5440b67cfb9-006526b499-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:37.943 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.943 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:37.944 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.944 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.945 DEBUG:boto:Method: GET 2023-10-11T14:43:37.945 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.945 DEBUG:boto:Data: 2023-10-11T14:43:37.946 DEBUG:boto:Headers: {} 2023-10-11T14:43:37.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:37.947 DEBUG:boto:Port: 8001 2023-10-11T14:43:37.947 DEBUG:boto:Params: {} 2023-10-11T14:43:37.948 DEBUG:boto:Token: None 2023-10-11T14:43:37.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:37 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:37.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4= 2023-10-11T14:43:37.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NCflxToT+dRjU4zVisROLqvEFP4=', 'Content-Length': '0'} 2023-10-11T14:43:37.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090e02ba59724ae7c-006526b499-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:37.998 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:37.999 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.000 DEBUG:boto:path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:38.000 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:38.000 DEBUG:boto:Method: GET 2023-10-11T14:43:38.001 DEBUG:boto:Path: /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:38.001 DEBUG:boto:Data: 2023-10-11T14:43:38.002 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.003 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.003 DEBUG:boto:Params: {} 2023-10-11T14:43:38.004 DEBUG:boto:Token: None 2023-10-11T14:43:38.004 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=hPp7YWKoT9JsCeZlK-TF18LWy1XGasl 2023-10-11T14:43:38.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Afw/dCLswFRWggCbY8OfWwUXF4Q= 2023-10-11T14:43:38.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Afw/dCLswFRWggCbY8OfWwUXF4Q=', 'Content-Length': '0'} 2023-10-11T14:43:38.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'hPp7YWKoT9JsCeZlK-TF18LWy1XGasl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006002ade59fc45f90-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.008 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.008 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.008 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.009 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.009 DEBUG:boto:Method: GET 2023-10-11T14:43:38.010 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.010 DEBUG:boto:Data: 2023-10-11T14:43:38.011 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.011 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.011 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.012 DEBUG:boto:Params: {} 2023-10-11T14:43:38.012 DEBUG:boto:Token: None 2023-10-11T14:43:38.013 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25ce68cf8d88615-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.062 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.063 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.064 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.064 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.064 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.065 DEBUG:boto:Method: GET 2023-10-11T14:43:38.065 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.066 DEBUG:boto:Data: 2023-10-11T14:43:38.066 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.067 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.067 DEBUG:boto:Params: {} 2023-10-11T14:43:38.068 DEBUG:boto:Token: None 2023-10-11T14:43:38.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5086b72636e119a-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.073 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.073 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.074 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.074 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.075 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.075 DEBUG:boto:Data: 2023-10-11T14:43:38.075 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.076 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.077 DEBUG:boto:Params: {} 2023-10-11T14:43:38.077 DEBUG:boto:Token: None 2023-10-11T14:43:38.078 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4c3d7fe1826c3bd-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.084 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.084 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.085 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.085 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.086 DEBUG:boto:Data: 2023-10-11T14:43:38.086 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.086 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.087 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.087 DEBUG:boto:Params: {} 2023-10-11T14:43:38.088 DEBUG:boto:Token: None 2023-10-11T14:43:38.088 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057a4838984dec2bd-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.094 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.094 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.095 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.095 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.096 DEBUG:boto:Method: GET 2023-10-11T14:43:38.097 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.098 DEBUG:boto:Data: 2023-10-11T14:43:38.098 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.099 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.100 DEBUG:boto:Params: {} 2023-10-11T14:43:38.100 DEBUG:boto:Token: None 2023-10-11T14:43:38.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dea851f07e9935e5-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.105 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.106 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.106 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.107 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.108 DEBUG:boto:Method: GET 2023-10-11T14:43:38.108 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.108 DEBUG:boto:Data: 2023-10-11T14:43:38.109 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.110 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.110 DEBUG:boto:Params: {} 2023-10-11T14:43:38.111 DEBUG:boto:Token: None 2023-10-11T14:43:38.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000161cf617364d81a0-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.158 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.159 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.160 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.160 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.160 DEBUG:boto:Method: GET 2023-10-11T14:43:38.161 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.161 DEBUG:boto:Data: 2023-10-11T14:43:38.162 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.163 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.163 DEBUG:boto:Params: {} 2023-10-11T14:43:38.165 DEBUG:boto:Token: None 2023-10-11T14:43:38.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ced39f7413ed48d-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.172 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.173 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.173 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.174 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.174 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.174 DEBUG:boto:Method: GET 2023-10-11T14:43:38.175 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.176 DEBUG:boto:Data: 2023-10-11T14:43:38.176 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.177 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.178 DEBUG:boto:Params: {} 2023-10-11T14:43:38.179 DEBUG:boto:Token: None 2023-10-11T14:43:38.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043d80743282c387-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.226 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.228 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.228 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.229 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.229 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.229 DEBUG:boto:Method: GET 2023-10-11T14:43:38.230 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.230 DEBUG:boto:Data: 2023-10-11T14:43:38.231 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.232 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.232 DEBUG:boto:Params: {} 2023-10-11T14:43:38.232 DEBUG:boto:Token: None 2023-10-11T14:43:38.233 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y= 2023-10-11T14:43:38.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y=', 'Content-Length': '0'} 2023-10-11T14:43:38.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6425d3864e64818-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.282 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.283 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.283 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.284 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.284 DEBUG:boto:Method: GET 2023-10-11T14:43:38.285 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.285 DEBUG:boto:Data: 2023-10-11T14:43:38.286 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.286 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.287 DEBUG:boto:Params: {} 2023-10-11T14:43:38.287 DEBUG:boto:Token: None 2023-10-11T14:43:38.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y= 2023-10-11T14:43:38.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y=', 'Content-Length': '0'} 2023-10-11T14:43:38.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037244d2203f76daf-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.292 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.293 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.293 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.294 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.294 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.295 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.295 DEBUG:boto:Data: 2023-10-11T14:43:38.295 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.296 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.297 DEBUG:boto:Params: {} 2023-10-11T14:43:38.297 DEBUG:boto:Token: None 2023-10-11T14:43:38.298 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9mVearwsiQFE67Luaah1HdjsjNY= 2023-10-11T14:43:38.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9mVearwsiQFE67Luaah1HdjsjNY=', 'Content-Length': '0'} 2023-10-11T14:43:38.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091cca5c05dfbcc3d-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.301 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.302 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.303 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.303 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.303 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.304 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.304 DEBUG:boto:Data: 2023-10-11T14:43:38.305 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.305 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.306 DEBUG:boto:Params: {} 2023-10-11T14:43:38.306 DEBUG:boto:Token: None 2023-10-11T14:43:38.307 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9mVearwsiQFE67Luaah1HdjsjNY= 2023-10-11T14:43:38.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9mVearwsiQFE67Luaah1HdjsjNY=', 'Content-Length': '0'} 2023-10-11T14:43:38.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000189b4f491677a0fd-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.311 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.311 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.311 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.312 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.312 DEBUG:boto:Method: GET 2023-10-11T14:43:38.313 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.313 DEBUG:boto:Data: 2023-10-11T14:43:38.313 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.314 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.315 DEBUG:boto:Params: {} 2023-10-11T14:43:38.315 DEBUG:boto:Token: None 2023-10-11T14:43:38.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y= 2023-10-11T14:43:38.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y=', 'Content-Length': '0'} 2023-10-11T14:43:38.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009c07043fb1366616-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.319 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.320 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.320 DEBUG:boto:path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.321 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.321 DEBUG:boto:Method: GET 2023-10-11T14:43:38.321 DEBUG:boto:Path: /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.322 DEBUG:boto:Data: 2023-10-11T14:43:38.322 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.323 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.324 DEBUG:boto:Params: {} 2023-10-11T14:43:38.324 DEBUG:boto:Token: None 2023-10-11T14:43:38.325 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU 2023-10-11T14:43:38.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y= 2023-10-11T14:43:38.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkaOkZWLubJ1DJXDC8uMk59Zv1Y=', 'Content-Length': '0'} 2023-10-11T14:43:38.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '2AjfwA1MAw87MYsGuLUFeOQn6xDjNFU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059a637697868638c-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.370 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.371 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.371 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.372 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.372 DEBUG:boto:Method: GET 2023-10-11T14:43:38.373 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.373 DEBUG:boto:Data: 2023-10-11T14:43:38.373 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.374 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.375 DEBUG:boto:Params: {} 2023-10-11T14:43:38.375 DEBUG:boto:Token: None 2023-10-11T14:43:38.376 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921a1122924ffce0-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.381 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.382 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.382 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.382 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.383 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.383 DEBUG:boto:Method: GET 2023-10-11T14:43:38.384 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.384 DEBUG:boto:Data: 2023-10-11T14:43:38.385 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.387 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.387 DEBUG:boto:Params: {} 2023-10-11T14:43:38.388 DEBUG:boto:Token: None 2023-10-11T14:43:38.388 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001234db8c4fa0b070-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.434 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.435 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.436 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.436 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.437 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.437 DEBUG:boto:Data: 2023-10-11T14:43:38.438 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.439 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.439 DEBUG:boto:Params: {} 2023-10-11T14:43:38.439 DEBUG:boto:Token: None 2023-10-11T14:43:38.440 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a8c031d12117cbb-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.446 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.446 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.447 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.447 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.448 DEBUG:boto:Data: 2023-10-11T14:43:38.448 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.449 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.449 DEBUG:boto:Params: {} 2023-10-11T14:43:38.450 DEBUG:boto:Token: None 2023-10-11T14:43:38.450 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.451 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9aa4878ae525e25-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.455 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.456 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.456 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.457 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.457 DEBUG:boto:Method: GET 2023-10-11T14:43:38.458 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.458 DEBUG:boto:Data: 2023-10-11T14:43:38.459 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.459 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.460 DEBUG:boto:Params: {} 2023-10-11T14:43:38.460 DEBUG:boto:Token: None 2023-10-11T14:43:38.461 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c13930b31e77036d-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.465 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.465 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.465 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.466 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.466 DEBUG:boto:Method: GET 2023-10-11T14:43:38.467 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.467 DEBUG:boto:Data: 2023-10-11T14:43:38.468 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.468 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.469 DEBUG:boto:Params: {} 2023-10-11T14:43:38.469 DEBUG:boto:Token: None 2023-10-11T14:43:38.470 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d1b904ce4d1798a-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.514 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.515 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.516 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.516 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.517 DEBUG:boto:Method: GET 2023-10-11T14:43:38.517 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.518 DEBUG:boto:Data: 2023-10-11T14:43:38.518 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.519 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.519 DEBUG:boto:Params: {} 2023-10-11T14:43:38.520 DEBUG:boto:Token: None 2023-10-11T14:43:38.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a34a2f722a14ff-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.525 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.527 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.527 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.528 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.529 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.530 DEBUG:boto:Method: GET 2023-10-11T14:43:38.530 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.531 DEBUG:boto:Data: 2023-10-11T14:43:38.532 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.534 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.535 DEBUG:boto:Params: {} 2023-10-11T14:43:38.536 DEBUG:boto:Token: None 2023-10-11T14:43:38.536 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d387c4e01f00f236-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.582 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.584 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.585 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.585 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.586 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.586 DEBUG:boto:Method: GET 2023-10-11T14:43:38.587 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.587 DEBUG:boto:Data: 2023-10-11T14:43:38.588 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.589 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.589 DEBUG:boto:Params: {} 2023-10-11T14:43:38.590 DEBUG:boto:Token: None 2023-10-11T14:43:38.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs= 2023-10-11T14:43:38.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs=', 'Content-Length': '0'} 2023-10-11T14:43:38.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000889c35f06cc400b0-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.594 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.595 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.596 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.596 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.597 DEBUG:boto:Method: GET 2023-10-11T14:43:38.598 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.598 DEBUG:boto:Data: 2023-10-11T14:43:38.599 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.600 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.600 DEBUG:boto:Params: {} 2023-10-11T14:43:38.601 DEBUG:boto:Token: None 2023-10-11T14:43:38.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs= 2023-10-11T14:43:38.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs=', 'Content-Length': '0'} 2023-10-11T14:43:38.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e00f7161ab7f62a5-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.606 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.606 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.607 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.607 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.608 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.608 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.608 DEBUG:boto:Data: 2023-10-11T14:43:38.609 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.610 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.610 DEBUG:boto:Params: {} 2023-10-11T14:43:38.610 DEBUG:boto:Token: None 2023-10-11T14:43:38.611 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yKOyvw9OWMZRfqZlAMdYyiSvEu8= 2023-10-11T14:43:38.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yKOyvw9OWMZRfqZlAMdYyiSvEu8=', 'Content-Length': '0'} 2023-10-11T14:43:38.614 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b08bb00903783fe1-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.614 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.615 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.615 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.616 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.616 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.616 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.617 DEBUG:boto:Data: 2023-10-11T14:43:38.617 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.618 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.618 DEBUG:boto:Params: {} 2023-10-11T14:43:38.619 DEBUG:boto:Token: None 2023-10-11T14:43:38.619 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yKOyvw9OWMZRfqZlAMdYyiSvEu8= 2023-10-11T14:43:38.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yKOyvw9OWMZRfqZlAMdYyiSvEu8=', 'Content-Length': '0'} 2023-10-11T14:43:38.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e304fedf2de4569-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.623 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.624 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.624 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.624 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.625 DEBUG:boto:Method: GET 2023-10-11T14:43:38.625 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.626 DEBUG:boto:Data: 2023-10-11T14:43:38.626 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.626 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.627 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.627 DEBUG:boto:Params: {} 2023-10-11T14:43:38.628 DEBUG:boto:Token: None 2023-10-11T14:43:38.628 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.629 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs= 2023-10-11T14:43:38.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs=', 'Content-Length': '0'} 2023-10-11T14:43:38.631 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001160a18bf899a9f1-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.674 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.675 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.675 DEBUG:boto:path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.676 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.676 DEBUG:boto:Method: GET 2023-10-11T14:43:38.677 DEBUG:boto:Path: /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.677 DEBUG:boto:Data: 2023-10-11T14:43:38.678 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.678 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.679 DEBUG:boto:Params: {} 2023-10-11T14:43:38.679 DEBUG:boto:Token: None 2023-10-11T14:43:38.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl 2023-10-11T14:43:38.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs= 2023-10-11T14:43:38.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lZLOEFj43waScHxGvtvRcZUHwxs=', 'Content-Length': '0'} 2023-10-11T14:43:38.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'g-PPa0iZ0WRsFDVd6bbtr3oIMuKLyrl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036c6995075dd3e40-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.683 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.684 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.684 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.685 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.685 DEBUG:boto:Method: GET 2023-10-11T14:43:38.686 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.686 DEBUG:boto:Data: 2023-10-11T14:43:38.686 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.687 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.688 DEBUG:boto:Params: {} 2023-10-11T14:43:38.688 DEBUG:boto:Token: None 2023-10-11T14:43:38.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000346158b1f487727a-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.734 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.735 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.736 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.736 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.737 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.737 DEBUG:boto:Method: GET 2023-10-11T14:43:38.738 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.738 DEBUG:boto:Data: 2023-10-11T14:43:38.738 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.739 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.740 DEBUG:boto:Params: {} 2023-10-11T14:43:38.740 DEBUG:boto:Token: None 2023-10-11T14:43:38.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d3ded5ba474a7e-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.745 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.746 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.747 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.748 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.748 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.748 DEBUG:boto:Data: 2023-10-11T14:43:38.749 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.750 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.750 DEBUG:boto:Params: {} 2023-10-11T14:43:38.751 DEBUG:boto:Token: None 2023-10-11T14:43:38.752 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba007ce1840e49a9-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.757 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.757 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.758 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.758 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:38.759 DEBUG:boto:Data: 2023-10-11T14:43:38.759 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.760 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.761 DEBUG:boto:Params: {} 2023-10-11T14:43:38.762 DEBUG:boto:Token: None 2023-10-11T14:43:38.762 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj 2023-10-11T14:43:38.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II= 2023-10-11T14:43:38.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSZFyR74FLbCGuvLko1MU7Ye9II=', 'Content-Length': '0'} 2023-10-11T14:43:38.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d302f32101479378-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.768 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.769 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.769 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.770 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.770 DEBUG:boto:Method: GET 2023-10-11T14:43:38.771 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.771 DEBUG:boto:Data: 2023-10-11T14:43:38.772 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.772 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.772 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.773 DEBUG:boto:Params: {} 2023-10-11T14:43:38.773 DEBUG:boto:Token: None 2023-10-11T14:43:38.774 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.777 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec64e7a5f7b8ac99-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.778 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.778 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.781 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.782 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.782 DEBUG:boto:Method: GET 2023-10-11T14:43:38.783 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.783 DEBUG:boto:Data: 2023-10-11T14:43:38.784 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.785 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.785 DEBUG:boto:Params: {} 2023-10-11T14:43:38.786 DEBUG:boto:Token: None 2023-10-11T14:43:38.786 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:38.787 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c= 2023-10-11T14:43:38.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UBonS57JP4fVqafdFltdWUn9n7c=', 'Content-Length': '0'} 2023-10-11T14:43:38.789 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcb1282a72c6cea5-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.834 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.835 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.836 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.836 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.837 DEBUG:boto:Method: GET 2023-10-11T14:43:38.837 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.838 DEBUG:boto:Data: 2023-10-11T14:43:38.838 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.839 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.839 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.840 DEBUG:boto:Params: {} 2023-10-11T14:43:38.841 DEBUG:boto:Token: None 2023-10-11T14:43:38.841 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.842 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c4b6f4187be752b-006526b49a-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.846 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.847 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.847 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.848 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.848 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:38.849 DEBUG:boto:Method: GET 2023-10-11T14:43:38.849 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:38.850 DEBUG:boto:Data: 2023-10-11T14:43:38.850 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.851 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.851 DEBUG:boto:Params: {} 2023-10-11T14:43:38.852 DEBUG:boto:Token: None 2023-10-11T14:43:38.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/ 2023-10-11T14:43:38.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM= 2023-10-11T14:43:38.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ostGbvWE10jPMccDBGE0bwB4FHM=', 'Content-Length': '0'} 2023-10-11T14:43:38.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aadecb15665eee2-006526b49a-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.902 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:38.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.904 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.904 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.905 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.905 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.906 DEBUG:boto:Method: GET 2023-10-11T14:43:38.906 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.907 DEBUG:boto:Data: 2023-10-11T14:43:38.907 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.908 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.908 DEBUG:boto:Params: {} 2023-10-11T14:43:38.909 DEBUG:boto:Token: None 2023-10-11T14:43:38.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms= 2023-10-11T14:43:38.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms=', 'Content-Length': '0'} 2023-10-11T14:43:38.912 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038296f136fd1d30a-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.913 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.914 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.914 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.914 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.915 DEBUG:boto:Method: GET 2023-10-11T14:43:38.915 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.916 DEBUG:boto:Data: 2023-10-11T14:43:38.916 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.917 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.917 DEBUG:boto:Params: {} 2023-10-11T14:43:38.918 DEBUG:boto:Token: None 2023-10-11T14:43:38.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.919 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms= 2023-10-11T14:43:38.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms=', 'Content-Length': '0'} 2023-10-11T14:43:38.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a2ff6839d043143-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.922 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.923 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.923 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.923 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.924 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.924 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.925 DEBUG:boto:Data: 2023-10-11T14:43:38.925 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.926 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.926 DEBUG:boto:Params: {} 2023-10-11T14:43:38.927 DEBUG:boto:Token: None 2023-10-11T14:43:38.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OWcpAPsgjcMgu82WiFQ4dZMdRu4= 2023-10-11T14:43:38.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OWcpAPsgjcMgu82WiFQ4dZMdRu4=', 'Content-Length': '0'} 2023-10-11T14:43:38.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab44c1f8b124f832-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.931 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.931 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.932 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.932 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.932 DEBUG:boto:Method: HEAD 2023-10-11T14:43:38.933 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.933 DEBUG:boto:Data: 2023-10-11T14:43:38.934 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.934 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.935 DEBUG:boto:Params: {} 2023-10-11T14:43:38.935 DEBUG:boto:Token: None 2023-10-11T14:43:38.936 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OWcpAPsgjcMgu82WiFQ4dZMdRu4= 2023-10-11T14:43:38.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OWcpAPsgjcMgu82WiFQ4dZMdRu4=', 'Content-Length': '0'} 2023-10-11T14:43:38.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045eea3da54d53957-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:38.940 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.940 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.941 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.941 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.941 DEBUG:boto:Method: GET 2023-10-11T14:43:38.942 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.942 DEBUG:boto:Data: 2023-10-11T14:43:38.943 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:38.944 DEBUG:boto:Port: 8001 2023-10-11T14:43:38.944 DEBUG:boto:Params: {} 2023-10-11T14:43:38.944 DEBUG:boto:Token: None 2023-10-11T14:43:38.945 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms= 2023-10-11T14:43:38.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms=', 'Content-Length': '0'} 2023-10-11T14:43:38.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003378933e943443a8-006526b49a-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.949 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:38.949 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:38.950 DEBUG:boto:path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.950 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.951 DEBUG:boto:Method: GET 2023-10-11T14:43:38.951 DEBUG:boto:Path: /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.951 DEBUG:boto:Data: 2023-10-11T14:43:38.952 DEBUG:boto:Headers: {} 2023-10-11T14:43:38.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:38.953 DEBUG:boto:Port: 8000 2023-10-11T14:43:38.953 DEBUG:boto:Params: {} 2023-10-11T14:43:38.954 DEBUG:boto:Token: None 2023-10-11T14:43:38.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:38 GMT /dghqvd-13/obj?versionId=9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT 2023-10-11T14:43:38.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms= 2023-10-11T14:43:38.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XynNYDg/TwgABHvr6N5F5qK4+Ms=', 'Content-Length': '0'} 2023-10-11T14:43:38.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '9YkBW4ZzkElB0NC9RM8ybyUGwzvA1oT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ade65ed82722b200-006526b49a-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:38.998 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:38.999 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:38.999 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.000 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.000 DEBUG:boto:Method: GET 2023-10-11T14:43:39.001 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.001 DEBUG:boto:Data: 2023-10-11T14:43:39.002 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.003 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.003 DEBUG:boto:Params: {} 2023-10-11T14:43:39.004 DEBUG:boto:Token: None 2023-10-11T14:43:39.004 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e70b369988ea0d-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.009 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.010 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.010 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.011 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.011 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.011 DEBUG:boto:Method: GET 2023-10-11T14:43:39.012 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.012 DEBUG:boto:Data: 2023-10-11T14:43:39.013 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.013 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.014 DEBUG:boto:Params: {} 2023-10-11T14:43:39.014 DEBUG:boto:Token: None 2023-10-11T14:43:39.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006820f4abf6bc70c5-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.062 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.063 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.064 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.064 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.065 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.065 DEBUG:boto:Data: 2023-10-11T14:43:39.065 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.066 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.067 DEBUG:boto:Params: {} 2023-10-11T14:43:39.067 DEBUG:boto:Token: None 2023-10-11T14:43:39.068 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.072 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000936c5d7311273f00-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.073 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.073 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.074 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.074 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.075 DEBUG:boto:Data: 2023-10-11T14:43:39.075 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.076 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.076 DEBUG:boto:Params: {} 2023-10-11T14:43:39.077 DEBUG:boto:Token: None 2023-10-11T14:43:39.078 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.082 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008331c0d2620f84d0-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.084 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.084 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.084 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.085 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.085 DEBUG:boto:Method: GET 2023-10-11T14:43:39.086 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.086 DEBUG:boto:Data: 2023-10-11T14:43:39.087 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.088 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.088 DEBUG:boto:Params: {} 2023-10-11T14:43:39.088 DEBUG:boto:Token: None 2023-10-11T14:43:39.089 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.092 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac5d48250ee487af-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.134 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.135 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.135 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.136 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.136 DEBUG:boto:Method: GET 2023-10-11T14:43:39.137 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.137 DEBUG:boto:Data: 2023-10-11T14:43:39.138 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.138 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.139 DEBUG:boto:Params: {} 2023-10-11T14:43:39.139 DEBUG:boto:Token: None 2023-10-11T14:43:39.140 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093287d6ec1e67193-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.190 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.191 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.192 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.193 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.194 DEBUG:boto:Method: GET 2023-10-11T14:43:39.194 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.195 DEBUG:boto:Data: 2023-10-11T14:43:39.195 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.196 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.196 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.197 DEBUG:boto:Params: {} 2023-10-11T14:43:39.197 DEBUG:boto:Token: None 2023-10-11T14:43:39.198 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.198 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068eab3546d9dc7c9-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.203 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.203 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.204 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.204 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.205 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.205 DEBUG:boto:Method: GET 2023-10-11T14:43:39.206 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.206 DEBUG:boto:Data: 2023-10-11T14:43:39.206 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.207 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.208 DEBUG:boto:Params: {} 2023-10-11T14:43:39.208 DEBUG:boto:Token: None 2023-10-11T14:43:39.209 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.209 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034812e65481612fe-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.213 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.214 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.215 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.215 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.216 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.216 DEBUG:boto:Method: GET 2023-10-11T14:43:39.216 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.217 DEBUG:boto:Data: 2023-10-11T14:43:39.217 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.218 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.219 DEBUG:boto:Params: {} 2023-10-11T14:43:39.219 DEBUG:boto:Token: None 2023-10-11T14:43:39.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ= 2023-10-11T14:43:39.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ=', 'Content-Length': '0'} 2023-10-11T14:43:39.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011e73b3b0026a411-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.224 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.224 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.224 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.225 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.225 DEBUG:boto:Method: GET 2023-10-11T14:43:39.226 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.226 DEBUG:boto:Data: 2023-10-11T14:43:39.227 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.227 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.228 DEBUG:boto:Params: {} 2023-10-11T14:43:39.228 DEBUG:boto:Token: None 2023-10-11T14:43:39.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ= 2023-10-11T14:43:39.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ=', 'Content-Length': '0'} 2023-10-11T14:43:39.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d0bbd1f072c3976-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.233 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.233 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.234 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.234 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.234 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.235 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.235 DEBUG:boto:Data: 2023-10-11T14:43:39.236 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.236 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.237 DEBUG:boto:Params: {} 2023-10-11T14:43:39.237 DEBUG:boto:Token: None 2023-10-11T14:43:39.238 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ioTL4Ul2e5iSmnhUeqCmBS3mCEw= 2023-10-11T14:43:39.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ioTL4Ul2e5iSmnhUeqCmBS3mCEw=', 'Content-Length': '0'} 2023-10-11T14:43:39.241 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006938e0648545feb9-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.241 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.242 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.243 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.243 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.243 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.244 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.244 DEBUG:boto:Data: 2023-10-11T14:43:39.245 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.245 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.246 DEBUG:boto:Params: {} 2023-10-11T14:43:39.246 DEBUG:boto:Token: None 2023-10-11T14:43:39.247 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ioTL4Ul2e5iSmnhUeqCmBS3mCEw= 2023-10-11T14:43:39.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ioTL4Ul2e5iSmnhUeqCmBS3mCEw=', 'Content-Length': '0'} 2023-10-11T14:43:39.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000291fc67ea5c2c98b-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.251 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.251 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.251 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.252 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.252 DEBUG:boto:Method: GET 2023-10-11T14:43:39.253 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.253 DEBUG:boto:Data: 2023-10-11T14:43:39.254 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.254 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.255 DEBUG:boto:Params: {} 2023-10-11T14:43:39.255 DEBUG:boto:Token: None 2023-10-11T14:43:39.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ= 2023-10-11T14:43:39.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ=', 'Content-Length': '0'} 2023-10-11T14:43:39.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001356ee0392ececff-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.306 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.307 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.307 DEBUG:boto:path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.308 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.308 DEBUG:boto:Method: GET 2023-10-11T14:43:39.309 DEBUG:boto:Path: /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.309 DEBUG:boto:Data: 2023-10-11T14:43:39.309 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.310 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.311 DEBUG:boto:Params: {} 2023-10-11T14:43:39.311 DEBUG:boto:Token: None 2023-10-11T14:43:39.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6 2023-10-11T14:43:39.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ= 2023-10-11T14:43:39.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+roe5/GktE8q0Mat2UgJBBmidOQ=', 'Content-Length': '0'} 2023-10-11T14:43:39.315 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'iYRMCVFlXjwyNAxeAe25Cl73oqKNfq6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db01eab749ad30ad-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.315 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.316 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.316 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.317 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.317 DEBUG:boto:Method: GET 2023-10-11T14:43:39.318 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.318 DEBUG:boto:Data: 2023-10-11T14:43:39.319 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.320 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.320 DEBUG:boto:Params: {} 2023-10-11T14:43:39.321 DEBUG:boto:Token: None 2023-10-11T14:43:39.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a23d122e25f77ee6-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.326 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.327 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.327 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.328 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.329 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.329 DEBUG:boto:Method: GET 2023-10-11T14:43:39.330 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.330 DEBUG:boto:Data: 2023-10-11T14:43:39.331 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.332 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.332 DEBUG:boto:Params: {} 2023-10-11T14:43:39.333 DEBUG:boto:Token: None 2023-10-11T14:43:39.333 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd83956dbda7429-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.382 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.383 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.384 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.384 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.385 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.385 DEBUG:boto:Data: 2023-10-11T14:43:39.385 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.386 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.387 DEBUG:boto:Params: {} 2023-10-11T14:43:39.387 DEBUG:boto:Token: None 2023-10-11T14:43:39.388 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce2586d2cada55ef-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.393 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.393 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.394 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.394 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.398 DEBUG:boto:Data: 2023-10-11T14:43:39.398 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.399 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.399 DEBUG:boto:Params: {} 2023-10-11T14:43:39.400 DEBUG:boto:Token: None 2023-10-11T14:43:39.400 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9522cad7ea1741d-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.408 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.408 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.409 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.409 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.410 DEBUG:boto:Method: GET 2023-10-11T14:43:39.410 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.410 DEBUG:boto:Data: 2023-10-11T14:43:39.411 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.411 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.412 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.412 DEBUG:boto:Params: {} 2023-10-11T14:43:39.413 DEBUG:boto:Token: None 2023-10-11T14:43:39.413 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d88f845f8a3c90ba-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.458 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.460 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.461 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.461 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.461 DEBUG:boto:Method: GET 2023-10-11T14:43:39.462 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.462 DEBUG:boto:Data: 2023-10-11T14:43:39.463 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.464 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.464 DEBUG:boto:Params: {} 2023-10-11T14:43:39.464 DEBUG:boto:Token: None 2023-10-11T14:43:39.465 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000509eb91f069db940-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.510 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.511 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.511 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.512 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.512 DEBUG:boto:Method: GET 2023-10-11T14:43:39.513 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.513 DEBUG:boto:Data: 2023-10-11T14:43:39.514 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.514 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.515 DEBUG:boto:Params: {} 2023-10-11T14:43:39.515 DEBUG:boto:Token: None 2023-10-11T14:43:39.516 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c293da6767832ed8-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.520 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.521 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.521 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.522 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.522 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.523 DEBUG:boto:Method: GET 2023-10-11T14:43:39.523 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.524 DEBUG:boto:Data: 2023-10-11T14:43:39.524 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.525 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.525 DEBUG:boto:Params: {} 2023-10-11T14:43:39.526 DEBUG:boto:Token: None 2023-10-11T14:43:39.526 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d56a54c31c51d2-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.531 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.532 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.533 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.533 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.534 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.534 DEBUG:boto:Method: GET 2023-10-11T14:43:39.534 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.535 DEBUG:boto:Data: 2023-10-11T14:43:39.535 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.536 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.537 DEBUG:boto:Params: {} 2023-10-11T14:43:39.537 DEBUG:boto:Token: None 2023-10-11T14:43:39.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY= 2023-10-11T14:43:39.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY=', 'Content-Length': '0'} 2023-10-11T14:43:39.541 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be6e6c688d67e1e0-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.541 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.542 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.542 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.543 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.543 DEBUG:boto:Method: GET 2023-10-11T14:43:39.544 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.544 DEBUG:boto:Data: 2023-10-11T14:43:39.545 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.545 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.545 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.546 DEBUG:boto:Params: {} 2023-10-11T14:43:39.546 DEBUG:boto:Token: None 2023-10-11T14:43:39.547 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY= 2023-10-11T14:43:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY=', 'Content-Length': '0'} 2023-10-11T14:43:39.550 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1c1b080ff51cbcf-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.590 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.591 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.591 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.592 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.592 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.593 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.593 DEBUG:boto:Data: 2023-10-11T14:43:39.593 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.594 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.595 DEBUG:boto:Params: {} 2023-10-11T14:43:39.595 DEBUG:boto:Token: None 2023-10-11T14:43:39.595 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s/cFwsyAW7YvELdSuOOA8TceMC4= 2023-10-11T14:43:39.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s/cFwsyAW7YvELdSuOOA8TceMC4=', 'Content-Length': '0'} 2023-10-11T14:43:39.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074607c738394d380-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.599 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.600 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.600 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.600 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.601 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.601 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.602 DEBUG:boto:Data: 2023-10-11T14:43:39.602 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.603 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.603 DEBUG:boto:Params: {} 2023-10-11T14:43:39.604 DEBUG:boto:Token: None 2023-10-11T14:43:39.604 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s/cFwsyAW7YvELdSuOOA8TceMC4= 2023-10-11T14:43:39.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s/cFwsyAW7YvELdSuOOA8TceMC4=', 'Content-Length': '0'} 2023-10-11T14:43:39.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085cc75354d16551a-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.608 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.609 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.609 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.609 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.610 DEBUG:boto:Method: GET 2023-10-11T14:43:39.610 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.611 DEBUG:boto:Data: 2023-10-11T14:43:39.611 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.612 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.612 DEBUG:boto:Params: {} 2023-10-11T14:43:39.613 DEBUG:boto:Token: None 2023-10-11T14:43:39.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY= 2023-10-11T14:43:39.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY=', 'Content-Length': '0'} 2023-10-11T14:43:39.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a6c3bfaa5fc3b50-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.659 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.659 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.660 DEBUG:boto:path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.661 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.661 DEBUG:boto:Method: GET 2023-10-11T14:43:39.662 DEBUG:boto:Path: /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.662 DEBUG:boto:Data: 2023-10-11T14:43:39.663 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.664 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.664 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.664 DEBUG:boto:Params: {} 2023-10-11T14:43:39.665 DEBUG:boto:Token: None 2023-10-11T14:43:39.665 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=60YNBZiNsGEzuTbT1jnFi1diB1uidQb 2023-10-11T14:43:39.666 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY= 2023-10-11T14:43:39.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:488+4MNj6akIMRu+FDuDO6Eu1wY=', 'Content-Length': '0'} 2023-10-11T14:43:39.668 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '60YNBZiNsGEzuTbT1jnFi1diB1uidQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000213a45146c31d928-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.669 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.670 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.670 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.671 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.671 DEBUG:boto:Method: GET 2023-10-11T14:43:39.671 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.672 DEBUG:boto:Data: 2023-10-11T14:43:39.672 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.673 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.674 DEBUG:boto:Params: {} 2023-10-11T14:43:39.674 DEBUG:boto:Token: None 2023-10-11T14:43:39.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9fb9f1be52b143c-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.722 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.723 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.724 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.724 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.725 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.725 DEBUG:boto:Method: GET 2023-10-11T14:43:39.725 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.726 DEBUG:boto:Data: 2023-10-11T14:43:39.726 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.727 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.728 DEBUG:boto:Params: {} 2023-10-11T14:43:39.728 DEBUG:boto:Token: None 2023-10-11T14:43:39.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c604b07023a7abfa-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.733 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.734 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.734 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.735 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.735 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.736 DEBUG:boto:Data: 2023-10-11T14:43:39.736 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.737 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.738 DEBUG:boto:Params: {} 2023-10-11T14:43:39.738 DEBUG:boto:Token: None 2023-10-11T14:43:39.739 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ef722dbea470901-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.745 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.745 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.746 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.746 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:39.747 DEBUG:boto:Data: 2023-10-11T14:43:39.747 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.749 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.749 DEBUG:boto:Params: {} 2023-10-11T14:43:39.749 DEBUG:boto:Token: None 2023-10-11T14:43:39.750 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj 2023-10-11T14:43:39.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0= 2023-10-11T14:43:39.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MA0j+xlFpHtQzPVipo+LdcQqSF0=', 'Content-Length': '0'} 2023-10-11T14:43:39.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f24b380c8784e7a-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.756 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.757 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.757 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.757 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.758 DEBUG:boto:Method: GET 2023-10-11T14:43:39.758 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.759 DEBUG:boto:Data: 2023-10-11T14:43:39.759 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.760 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.761 DEBUG:boto:Params: {} 2023-10-11T14:43:39.761 DEBUG:boto:Token: None 2023-10-11T14:43:39.762 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.765 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a94dbe07b8b97f84-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.766 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.766 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.767 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.767 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.768 DEBUG:boto:Method: GET 2023-10-11T14:43:39.768 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.768 DEBUG:boto:Data: 2023-10-11T14:43:39.769 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.770 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.770 DEBUG:boto:Params: {} 2023-10-11T14:43:39.771 DEBUG:boto:Token: None 2023-10-11T14:43:39.771 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:39.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU= 2023-10-11T14:43:39.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AblKkZ2+Vfxqhplx/KIprb5BqJU=', 'Content-Length': '0'} 2023-10-11T14:43:39.774 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef0bdc602b369b15-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.822 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.823 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.823 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.824 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.824 DEBUG:boto:Method: GET 2023-10-11T14:43:39.825 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.825 DEBUG:boto:Data: 2023-10-11T14:43:39.826 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.826 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.827 DEBUG:boto:Params: {} 2023-10-11T14:43:39.827 DEBUG:boto:Token: None 2023-10-11T14:43:39.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc595a496b1c56b-006526b49b-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.833 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.833 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:39.834 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:39.834 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.835 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:39.835 DEBUG:boto:Method: GET 2023-10-11T14:43:39.836 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:39.836 DEBUG:boto:Data: 2023-10-11T14:43:39.836 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.837 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.838 DEBUG:boto:Params: {} 2023-10-11T14:43:39.838 DEBUG:boto:Token: None 2023-10-11T14:43:39.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/ 2023-10-11T14:43:39.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A= 2023-10-11T14:43:39.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KCZtUtudzW1uNRatE04GcxoF7A=', 'Content-Length': '0'} 2023-10-11T14:43:39.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a36051d703bb61eb-006526b49b-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.882 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:39.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.884 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.884 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.885 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.885 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.885 DEBUG:boto:Method: GET 2023-10-11T14:43:39.886 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.886 DEBUG:boto:Data: 2023-10-11T14:43:39.887 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.888 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.888 DEBUG:boto:Params: {} 2023-10-11T14:43:39.889 DEBUG:boto:Token: None 2023-10-11T14:43:39.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.890 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo= 2023-10-11T14:43:39.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo=', 'Content-Length': '0'} 2023-10-11T14:43:39.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e4d48be6e91355b-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.934 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.935 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.935 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.936 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.936 DEBUG:boto:Method: GET 2023-10-11T14:43:39.937 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.937 DEBUG:boto:Data: 2023-10-11T14:43:39.938 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.938 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.939 DEBUG:boto:Params: {} 2023-10-11T14:43:39.939 DEBUG:boto:Token: None 2023-10-11T14:43:39.940 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo= 2023-10-11T14:43:39.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo=', 'Content-Length': '0'} 2023-10-11T14:43:39.943 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0e7126c013bed93-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.944 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.944 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.945 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.945 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.946 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.946 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.946 DEBUG:boto:Data: 2023-10-11T14:43:39.947 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.948 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.948 DEBUG:boto:Params: {} 2023-10-11T14:43:39.949 DEBUG:boto:Token: None 2023-10-11T14:43:39.949 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:196BJc0oY/EFN6SYShilvuDt63I= 2023-10-11T14:43:39.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:196BJc0oY/EFN6SYShilvuDt63I=', 'Content-Length': '0'} 2023-10-11T14:43:39.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b577563e05a36f3f-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.953 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.953 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.954 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.954 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.954 DEBUG:boto:Method: HEAD 2023-10-11T14:43:39.955 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.955 DEBUG:boto:Data: 2023-10-11T14:43:39.956 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.956 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.956 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.957 DEBUG:boto:Params: {} 2023-10-11T14:43:39.957 DEBUG:boto:Token: None 2023-10-11T14:43:39.958 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:196BJc0oY/EFN6SYShilvuDt63I= 2023-10-11T14:43:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:196BJc0oY/EFN6SYShilvuDt63I=', 'Content-Length': '0'} 2023-10-11T14:43:39.961 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b479aef16fb4ec9e-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:39.962 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.962 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.962 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.963 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.963 DEBUG:boto:Method: GET 2023-10-11T14:43:39.964 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.964 DEBUG:boto:Data: 2023-10-11T14:43:39.965 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:39.965 DEBUG:boto:Port: 8001 2023-10-11T14:43:39.966 DEBUG:boto:Params: {} 2023-10-11T14:43:39.966 DEBUG:boto:Token: None 2023-10-11T14:43:39.967 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.967 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo= 2023-10-11T14:43:39.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo=', 'Content-Length': '0'} 2023-10-11T14:43:39.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaada11a2fa39bc6-006526b49b-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:39.970 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:39.971 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:39.971 DEBUG:boto:path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.972 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.972 DEBUG:boto:Method: GET 2023-10-11T14:43:39.973 DEBUG:boto:Path: /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.973 DEBUG:boto:Data: 2023-10-11T14:43:39.973 DEBUG:boto:Headers: {} 2023-10-11T14:43:39.974 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:39.974 DEBUG:boto:Port: 8000 2023-10-11T14:43:39.975 DEBUG:boto:Params: {} 2023-10-11T14:43:39.975 DEBUG:boto:Token: None 2023-10-11T14:43:39.976 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:39 GMT /dghqvd-13/obj?versionId=MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx 2023-10-11T14:43:39.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo= 2023-10-11T14:43:39.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8qjmMzfo8fE56HMRfqQGSRYGlfo=', 'Content-Length': '0'} 2023-10-11T14:43:39.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', 'MV2BZ6FSJzxa1GY5UWnkQnj6KEU9-sx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eff3563de17c702f-006526b49b-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.026 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:40.027 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:40.027 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.028 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.028 DEBUG:boto:Method: GET 2023-10-11T14:43:40.029 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:40.029 DEBUG:boto:Data: 2023-10-11T14:43:40.030 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:40.031 DEBUG:boto:Port: 8001 2023-10-11T14:43:40.032 DEBUG:boto:Params: {} 2023-10-11T14:43:40.032 DEBUG:boto:Token: None 2023-10-11T14:43:40.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/ 2023-10-11T14:43:40.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw= 2023-10-11T14:43:40.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw=', 'Content-Length': '0'} 2023-10-11T14:43:40.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000237847c0cd803c4-006526b49c-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.037 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:40.038 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:40.038 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:40.039 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.039 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.040 DEBUG:boto:Method: GET 2023-10-11T14:43:40.040 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:40.040 DEBUG:boto:Data: 2023-10-11T14:43:40.041 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.041 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:40.042 DEBUG:boto:Port: 8000 2023-10-11T14:43:40.042 DEBUG:boto:Params: {} 2023-10-11T14:43:40.043 DEBUG:boto:Token: None 2023-10-11T14:43:40.043 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/ 2023-10-11T14:43:40.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw= 2023-10-11T14:43:40.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw=', 'Content-Length': '0'} 2023-10-11T14:43:40.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c881118b014c9155-006526b49c-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.090 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:40.091 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:40.092 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:40.092 DEBUG:boto:Method: HEAD 2023-10-11T14:43:40.092 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:40.093 DEBUG:boto:Data: 2023-10-11T14:43:40.093 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:40.094 DEBUG:boto:Port: 8001 2023-10-11T14:43:40.095 DEBUG:boto:Params: {} 2023-10-11T14:43:40.098 DEBUG:boto:Token: None 2023-10-11T14:43:40.099 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/obj 2023-10-11T14:43:40.099 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D/0GWocixQ8pTk2T1akv8PTNUys= 2023-10-11T14:43:40.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D/0GWocixQ8pTk2T1akv8PTNUys=', 'Content-Length': '0'} 2023-10-11T14:43:40.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000014f0b7a1bb8d51f5-006526b49c-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.104 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:40.104 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:40.105 DEBUG:boto:Method: HEAD 2023-10-11T14:43:40.105 DEBUG:boto:Path: /dghqvd-13/obj 2023-10-11T14:43:40.106 DEBUG:boto:Data: 2023-10-11T14:43:40.106 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:40.107 DEBUG:boto:Port: 8000 2023-10-11T14:43:40.108 DEBUG:boto:Params: {} 2023-10-11T14:43:40.108 DEBUG:boto:Token: None 2023-10-11T14:43:40.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/obj 2023-10-11T14:43:40.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D/0GWocixQ8pTk2T1akv8PTNUys= 2023-10-11T14:43:40.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D/0GWocixQ8pTk2T1akv8PTNUys=', 'Content-Length': '0'} 2023-10-11T14:43:40.113 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044654a121a2febf4-006526b49c-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:43:40.114 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:40.115 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:40.115 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.116 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.117 DEBUG:boto:Method: GET 2023-10-11T14:43:40.117 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.118 DEBUG:boto:Data: 2023-10-11T14:43:40.118 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:40.119 DEBUG:boto:Port: 8001 2023-10-11T14:43:40.119 DEBUG:boto:Params: {} 2023-10-11T14:43:40.120 DEBUG:boto:Token: None 2023-10-11T14:43:40.121 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bCWKVbvTaBKPZkpcYki5luKefaI= 2023-10-11T14:43:40.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bCWKVbvTaBKPZkpcYki5luKefaI=', 'Content-Length': '0'} 2023-10-11T14:43:40.124 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a60a5e054d39a2a-006526b49c-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.124 DEBUG:boto:path=/dghqvd-13/obj 2023-10-11T14:43:40.125 DEBUG:boto:auth_path=/dghqvd-13/obj 2023-10-11T14:43:40.125 DEBUG:boto:path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.126 DEBUG:boto:auth_path=/dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.126 DEBUG:boto:Method: GET 2023-10-11T14:43:40.127 DEBUG:boto:Path: /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.127 DEBUG:boto:Data: 2023-10-11T14:43:40.128 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:40.128 DEBUG:boto:Port: 8000 2023-10-11T14:43:40.129 DEBUG:boto:Params: {} 2023-10-11T14:43:40.129 DEBUG:boto:Token: None 2023-10-11T14:43:40.130 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/obj?versionId=6RJDWkrU2bTatsG495AuYikbVYE8YAB 2023-10-11T14:43:40.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bCWKVbvTaBKPZkpcYki5luKefaI= 2023-10-11T14:43:40.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bCWKVbvTaBKPZkpcYki5luKefaI=', 'Content-Length': '0'} 2023-10-11T14:43:40.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:42:13 GMT'), ('x-amz-version-id', '6RJDWkrU2bTatsG495AuYikbVYE8YAB'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007dbeb290ee819a23-006526b49c-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.178 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:40.179 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:40.179 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.180 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.180 DEBUG:boto:Method: GET 2023-10-11T14:43:40.181 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:40.181 DEBUG:boto:Data: 2023-10-11T14:43:40.182 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:43:40.183 DEBUG:boto:Port: 8001 2023-10-11T14:43:40.183 DEBUG:boto:Params: {} 2023-10-11T14:43:40.184 DEBUG:boto:Token: None 2023-10-11T14:43:40.184 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/ 2023-10-11T14:43:40.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw= 2023-10-11T14:43:40.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw=', 'Content-Length': '0'} 2023-10-11T14:43:40.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fa44aecf7d5b415-006526b49c-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.189 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:40.190 DEBUG:boto:path=/dghqvd-13/ 2023-10-11T14:43:40.190 DEBUG:boto:auth_path=/dghqvd-13/ 2023-10-11T14:43:40.190 DEBUG:boto:path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.191 DEBUG:boto:auth_path=/dghqvd-13/?prefix=obj 2023-10-11T14:43:40.191 DEBUG:boto:Method: GET 2023-10-11T14:43:40.192 DEBUG:boto:Path: /dghqvd-13/?prefix=obj 2023-10-11T14:43:40.192 DEBUG:boto:Data: 2023-10-11T14:43:40.193 DEBUG:boto:Headers: {} 2023-10-11T14:43:40.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:40.193 DEBUG:boto:Port: 8000 2023-10-11T14:43:40.194 DEBUG:boto:Params: {} 2023-10-11T14:43:40.194 DEBUG:boto:Token: None 2023-10-11T14:43:40.195 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:43:40 GMT /dghqvd-13/ 2023-10-11T14:43:40.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw= 2023-10-11T14:43:40.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1QRXn3RmmjjvgLIhoNBiEzA6adw=', 'Content-Length': '0'} 2023-10-11T14:43:40.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000872de0b68d42ae74-006526b49c-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:43:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:40.199 DEBUG:boto:b'dghqvd-13obj1000falseobj2023-10-11T14:42:13.947Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:43:40.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-13 zones={a2, a1} 2023-10-11T14:43:40.201 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T14:43:40.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.202 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:40.258 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:40.259 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:40.306 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:43:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:41.278145Z" 2023-10-11T14:43:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.308 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:40.308 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:43:40.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.308 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.308 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001491", 2023-10-11T14:43:40.309 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:42:33.285244Z" 2023-10-11T14:43:40.309 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.309 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.309 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:43:40.310 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:43:40.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.310 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:43:40.311 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000001491', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:43:40.312 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:43:40.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.315 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:40.368 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:40.409 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:40.409 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:40.409 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:40.410 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:40.410 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:40.410 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:43:40.410 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.411 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:40.411 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.411 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:40.411 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.411 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.412 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:40.412 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.412 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.412 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.413 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:41.278145Z" 2023-10-11T14:43:40.413 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.413 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.414 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.414 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:40.414 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.414 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.415 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:40.415 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.415 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.415 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.416 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:43:40.416 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.416 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.416 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.416 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:40.417 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.417 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.417 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001491", 2023-10-11T14:43:40.417 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.418 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.418 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:42:33.285244Z" 2023-10-11T14:43:40.418 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.419 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.419 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:40.419 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.419 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.419 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:43:40.420 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.420 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.420 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:43:40.420 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.421 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.421 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:40.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.421 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:40.422 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:43:40.422 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:43:40.422 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.422 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:40Z" 2023-10-11T14:43:40.422 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:40.424 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:41.278145Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001491", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:42:33.285244Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:42:11.283392Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:43:40Z" } 2023-10-11T14:43:40.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:41.278145Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001491', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:42:33.285244Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:42:11.283392Z'}}] 2023-10-11T14:43:40.425 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000001491', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:43:40.425 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000001491', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:43:40.426 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:43:40.426 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T14:43:40.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.427 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:40.475 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:40.476 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:40.518 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:43:40.518 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:37:03.159417Z" 2023-10-11T14:43:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:43:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002619", 2023-10-11T14:43:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:43:39.171673Z" 2023-10-11T14:43:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:43:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:43:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.522 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:43:40.523 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:40.524 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:43:40.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:40.525 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:40.576 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:40.577 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:40.616 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:40.617 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:40.617 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:40.617 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:40.617 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:40.618 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:43:40.618 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.618 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:40.618 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.618 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:40.619 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.621 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.621 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:40.622 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.622 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.622 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.622 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:37:03.159417Z" 2023-10-11T14:43:40.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.623 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.623 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.623 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:40.623 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.624 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.624 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:40.624 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.624 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.624 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.625 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:43:40.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.625 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:40.626 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.626 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.626 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002478", 2023-10-11T14:43:40.626 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.627 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.627 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.627 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:43:30.508740Z" 2023-10-11T14:43:40.627 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:40.628 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:40.628 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:40.628 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:40.628 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:43:40.629 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:40.629 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:40.629 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:40.629 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:43:40.629 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:40.630 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:40.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.630 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:40.630 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:43:40.631 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:43:40.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:40.631 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:40Z" 2023-10-11T14:43:40.631 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:40.633 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:37:03.159417Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002478", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:43:30.508740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:43:40Z" } 2023-10-11T14:43:40.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:37:03.159417Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002478', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:43:30.508740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:43:40.634 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:40.634 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002478', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:40.635 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=2 master=00000000000000000000:00000000000000002619 target=00000000000000000000:00000000000000002478 2023-10-11T14:43:45.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:45.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:45.637 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:45.696 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:45.697 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:45.741 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:45.742 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:45.742 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:45.742 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:45.743 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:45.743 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:43:45.743 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.743 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:45.743 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:45.744 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:45.744 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:45.744 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:45.744 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:45.745 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:45.745 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:45.745 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:45.745 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:37:03.159417Z" 2023-10-11T14:43:45.746 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:45.746 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.746 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:45.746 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:45.747 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:45.747 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:45.747 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:45.747 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:45.747 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:45.748 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:45.748 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:43:45.748 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:45.748 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.748 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:45.749 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:45.749 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:45.749 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:45.749 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002478", 2023-10-11T14:43:45.749 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:45.750 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:45.750 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:45.750 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:43:30.508740Z" 2023-10-11T14:43:45.750 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:45.753 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.754 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:45.754 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:45.754 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:45.754 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:45.754 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:43:45.755 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:45.755 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:45.755 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:45.755 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:43:45.756 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:45.756 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:45.756 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:45.756 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.756 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:45.757 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:43:45.757 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:43:45.757 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:45.757 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:45Z" 2023-10-11T14:43:45.757 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:45.759 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:37:03.159417Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002478", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:43:30.508740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:43:45Z" } 2023-10-11T14:43:45.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:37:03.159417Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002478', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:43:30.508740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:43:45.760 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:45.760 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002478', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:45.761 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=2 master=00000000000000000000:00000000000000002619 target=00000000000000000000:00000000000000002478 2023-10-11T14:43:50.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:50.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:50.763 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:50.822 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:50.823 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:50.867 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:50.868 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:50.868 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:50.868 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:50.868 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:50.869 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:43:50.869 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.869 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:50.869 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:50.869 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:50.870 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:37:03.159417Z" 2023-10-11T14:43:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:50.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:50.872 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:50.873 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:43:50.874 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:50.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.874 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:50.874 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:50.875 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:50.875 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:50.875 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002478", 2023-10-11T14:43:50.875 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:50.877 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:50.877 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:50.877 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:43:30.508740Z" 2023-10-11T14:43:50.877 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:50.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.878 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:50.878 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:50.878 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:50.878 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:50.879 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:43:50.879 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:50.879 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:50.879 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:50.879 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:43:50.880 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:50.880 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:50.880 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:50.880 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.880 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:50.881 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:43:50.881 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:43:50.881 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:50.881 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:50Z" 2023-10-11T14:43:50.881 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:50.882 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:37:03.159417Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002478", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:43:30.508740Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:43:50Z" } 2023-10-11T14:43:50.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:37:03.159417Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002478', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:43:30.508740Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:43:50.883 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:50.884 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002478', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:50.884 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=2 master=00000000000000000000:00000000000000002619 target=00000000000000000000:00000000000000002478 2023-10-11T14:43:55.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:55.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:55.887 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:55.946 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:55.990 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:55.991 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:55.991 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:55.991 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:55.992 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:55.992 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:43:55.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:55.992 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:55.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:55.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:55.993 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:55.993 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:55.993 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:43:55.993 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:55.994 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:55.994 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:55.994 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:37:03.159417Z" 2023-10-11T14:43:55.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:55.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:55.995 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:55.995 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:55.995 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:55.995 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:55.996 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:55.996 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:55.996 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:55.996 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:55.996 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:43:55.997 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:55.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:55.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:55.997 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:55.997 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:55.998 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:55.998 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002619", 2023-10-11T14:43:55.998 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:55.998 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:56.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.001 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:43:39.171673Z" 2023-10-11T14:43:56.001 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.002 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.002 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:56.002 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:56.003 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:43:56.003 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:43:56.003 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:56.003 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:56.003 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.004 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:43:56.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.004 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:56.004 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.005 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:56.005 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:43:56.005 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:43:56.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.005 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:55Z" 2023-10-11T14:43:56.006 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:56.007 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:37:03.159417Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002619", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:43:39.171673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:43:55Z" } 2023-10-11T14:43:56.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:37:03.159417Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002619', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:43:39.171673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:43:56.008 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:56.008 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002619', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:43:56.009 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:43:56.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-10-11T14:43:56.011 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:43:56.011 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:43:56.012 DEBUG:boto:Method: PUT 2023-10-11T14:43:56.012 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:43:56.013 DEBUG:boto:Data: 2023-10-11T14:43:56.013 DEBUG:boto:Headers: {} 2023-10-11T14:43:56.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:56.014 DEBUG:boto:Port: 8000 2023-10-11T14:43:56.014 DEBUG:boto:Params: {} 2023-10-11T14:43:56.015 DEBUG:boto:Token: None 2023-10-11T14:43:56.015 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:43:56 GMT /dghqvd-14/ 2023-10-11T14:43:56.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3cHMcIwsVhBiiGnQSe+Xo8G1L6Q= 2023-10-11T14:43:56.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3cHMcIwsVhBiiGnQSe+Xo8G1L6Q=', 'Content-Length': '0'} 2023-10-11T14:43:56.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001b91c1230df71b0b-006526b4ac-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:56.030 DEBUG:rgw_multi.tests:created bucket=dghqvd-14 2023-10-11T14:43:56.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:56.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:56.032 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:56.082 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:56.082 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:56.110 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:43:56.122 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:43:56.122 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.122 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:43:56.123 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:43:56.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.123 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.123 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:56.124 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:43:56.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.124 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.124 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:43:56.124 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:43:56.125 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.125 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.125 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035436.026469_70.1", 2023-10-11T14:43:56.125 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:43:56.026469Z" 2023-10-11T14:43:56.125 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.126 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:43:56.127 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035436.026469_70.1'} 2023-10-11T14:43:56.127 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:43:56.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:56.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:43:56.129 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:43:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:56.181 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:56.206+0000 7f6e6ab9e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:43:56.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:56.207+0000 7f6e6ab9e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:43:56.221 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:43:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:43:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:43:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:43:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:43:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:43:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:43:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:43:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:43:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:43:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:43:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:43:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:43:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:43:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:43:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:43:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:43:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:43:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:43:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:43:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:43:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:43:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:43:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:43:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:43:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:43:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:43:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:43:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035436.026469_70.1", 2023-10-11T14:43:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:43:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:43:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:43:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:43:56.026469Z", 2023-10-11T14:43:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:43:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:43:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:43:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:43:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:43:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:43:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:43:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:43:56Z" 2023-10-11T14:43:56.236 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:43:56.237 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035436.026469_70.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:43:56.026469Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:43:56Z" } 2023-10-11T14:43:56.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035436.026469_70.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:43:56.026469Z', 'realm_epoch': 2}}] 2023-10-11T14:43:56.238 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035436.026469_70.1'} 2023-10-11T14:43:56.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035436.026469_70.1'} 2023-10-11T14:43:56.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:43:56.239 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:43:56.240 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:43:56.240 DEBUG:boto:Method: HEAD 2023-10-11T14:43:56.241 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:43:56.241 DEBUG:boto:Data: 2023-10-11T14:43:56.242 DEBUG:boto:Headers: {} 2023-10-11T14:43:56.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:56.242 DEBUG:boto:Port: 8000 2023-10-11T14:43:56.243 DEBUG:boto:Params: {} 2023-10-11T14:43:56.243 DEBUG:boto:Token: None 2023-10-11T14:43:56.244 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:43:56 GMT /dghqvd-14/ 2023-10-11T14:43:56.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yR+ti3SBqk4eMmjZNGxn5u61GJs= 2023-10-11T14:43:56.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:43:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yR+ti3SBqk4eMmjZNGxn5u61GJs=', 'Content-Length': '0'} 2023-10-11T14:43:56.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d178b8e35385439d-006526b4ac-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:56.248 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:43:56.248 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:43:56.249 DEBUG:boto:Method: PUT 2023-10-11T14:43:56.249 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:43:56.250 DEBUG:boto:Data: 2023-10-11T14:43:56.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:43:56.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:43:56.251 DEBUG:boto:Port: 8000 2023-10-11T14:43:56.252 DEBUG:boto:Params: {} 2023-10-11T14:43:56.252 DEBUG:boto:Token: None 2023-10-11T14:43:56.253 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:43:56 GMT /dghqvd-14/obj 2023-10-11T14:43:56.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b4omFiQK0PhBy3ZqYJ54Jl58psA= 2023-10-11T14:43:56.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:43:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b4omFiQK0PhBy3ZqYJ54Jl58psA='} 2023-10-11T14:43:56.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cef536634dba0809-006526b4ac-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:43:56.299 DEBUG:rgw_multi.tests:created initial version id=None 2023-10-11T14:43:56.300 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-14 2023-10-11T14:43:56.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:43:56.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:43:56.301 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:43:56.359 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:43:56.359 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:43:56.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:43:56.389+0000 7fb402a9b780 1 waiting to reach incremental sync.. 2023-10-11T14:44:01.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:01.390+0000 7fb402a9b780 1 waiting to reach incremental sync.. 2023-10-11T14:44:06.393 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:06.390+0000 7fb402a9b780 1 waiting to reach incremental sync.. 2023-10-11T14:44:11.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:11.392+0000 7fb402a9b780 1 bucket sync caught up with source: 2023-10-11T14:44:11.395 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.3.6, , ] 2023-10-11T14:44:11.395 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.3.6, , ] 2023-10-11T14:44:11.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:11.392+0000 7fb402a9b780 0 bucket checkpoint complete 2023-10-11T14:44:11.411 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-14 2023-10-11T14:44:11.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:11.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:11.412 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:44:11.475 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:11.476 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:11.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:11.512+0000 7f5b11a46780 1 waiting to reach incremental sync.. 2023-10-11T14:44:16.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:16.514+0000 7f5b11a46780 1 bucket sync caught up with source: 2023-10-11T14:44:16.517 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:16.517 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:16.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:16.514+0000 7f5b11a46780 0 bucket checkpoint complete 2023-10-11T14:44:16.532 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:44:16.533 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.533 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.534 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.534 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:16.535 DEBUG:boto:Data: 2023-10-11T14:44:16.535 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.536 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.536 DEBUG:boto:Params: {} 2023-10-11T14:44:16.537 DEBUG:boto:Token: None 2023-10-11T14:44:16.537 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4c1tWKX2nusiLZWwY3QKo3d1IcM= 2023-10-11T14:44:16.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4c1tWKX2nusiLZWwY3QKo3d1IcM=', 'Content-Length': '0'} 2023-10-11T14:44:16.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073b8f56e99103c62-006526b4c0-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.542 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.542 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.543 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.543 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:16.544 DEBUG:boto:Data: 2023-10-11T14:44:16.544 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.545 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.545 DEBUG:boto:Params: {} 2023-10-11T14:44:16.546 DEBUG:boto:Token: None 2023-10-11T14:44:16.546 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4c1tWKX2nusiLZWwY3QKo3d1IcM= 2023-10-11T14:44:16.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4c1tWKX2nusiLZWwY3QKo3d1IcM=', 'Content-Length': '0'} 2023-10-11T14:44:16.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fda6e075b71d1a6d-006526b4c0-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.550 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:44:16.551 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.551 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.552 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.552 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.553 DEBUG:boto:Method: GET 2023-10-11T14:44:16.553 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.553 DEBUG:boto:Data: 2023-10-11T14:44:16.554 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.555 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.555 DEBUG:boto:Params: {} 2023-10-11T14:44:16.556 DEBUG:boto:Token: None 2023-10-11T14:44:16.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/?versions 2023-10-11T14:44:16.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo= 2023-10-11T14:44:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo=', 'Content-Length': '0'} 2023-10-11T14:44:16.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fad1aca9cbfafc3-006526b4c0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.560 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:16.562 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:44:16.562 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.563 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.563 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.564 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.564 DEBUG:boto:Method: GET 2023-10-11T14:44:16.565 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.565 DEBUG:boto:Data: 2023-10-11T14:44:16.566 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.567 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.567 DEBUG:boto:Params: {} 2023-10-11T14:44:16.567 DEBUG:boto:Token: None 2023-10-11T14:44:16.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/?versions 2023-10-11T14:44:16.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo= 2023-10-11T14:44:16.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo=', 'Content-Length': '0'} 2023-10-11T14:44:16.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de4ff093ab3abb4e-006526b4c0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.572 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:16.573 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.573 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.574 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.575 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.575 DEBUG:boto:Method: GET 2023-10-11T14:44:16.576 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.576 DEBUG:boto:Data: 2023-10-11T14:44:16.576 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.577 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.578 DEBUG:boto:Params: {} 2023-10-11T14:44:16.578 DEBUG:boto:Token: None 2023-10-11T14:44:16.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/?versions 2023-10-11T14:44:16.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo= 2023-10-11T14:44:16.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo=', 'Content-Length': '0'} 2023-10-11T14:44:16.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a5999e6d7b8a93-006526b4c0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.626 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.627 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.628 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.628 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.629 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.629 DEBUG:boto:Method: GET 2023-10-11T14:44:16.630 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:16.630 DEBUG:boto:Data: 2023-10-11T14:44:16.631 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.631 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.632 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.632 DEBUG:boto:Params: {} 2023-10-11T14:44:16.633 DEBUG:boto:Token: None 2023-10-11T14:44:16.633 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/?versions 2023-10-11T14:44:16.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo= 2023-10-11T14:44:16.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j9QToB+/XOuDNTOmfwrpsj6htpo=', 'Content-Length': '0'} 2023-10-11T14:44:16.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0b231ccc33c4b7-006526b4c0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.638 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:16.639 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.639 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.640 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.640 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.640 DEBUG:boto:Method: GET 2023-10-11T14:44:16.641 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.641 DEBUG:boto:Data: 2023-10-11T14:44:16.642 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.644 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.645 DEBUG:boto:Params: {} 2023-10-11T14:44:16.645 DEBUG:boto:Token: None 2023-10-11T14:44:16.646 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI= 2023-10-11T14:44:16.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI=', 'Content-Length': '0'} 2023-10-11T14:44:16.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f28feeb2c4c463af-006526b4c0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.650 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.651 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.651 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.652 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.652 DEBUG:boto:Method: GET 2023-10-11T14:44:16.652 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.653 DEBUG:boto:Data: 2023-10-11T14:44:16.653 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.655 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.655 DEBUG:boto:Params: {} 2023-10-11T14:44:16.655 DEBUG:boto:Token: None 2023-10-11T14:44:16.656 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.657 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI= 2023-10-11T14:44:16.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI=', 'Content-Length': '0'} 2023-10-11T14:44:16.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000432d9f2cc3a513dd-006526b4c0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.660 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.661 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.661 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.662 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.662 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.662 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.663 DEBUG:boto:Data: 2023-10-11T14:44:16.663 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.664 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.664 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.665 DEBUG:boto:Params: {} 2023-10-11T14:44:16.665 DEBUG:boto:Token: None 2023-10-11T14:44:16.666 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.666 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WJaHq6/vI/pl52ret15RfmCI0Uk= 2023-10-11T14:44:16.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WJaHq6/vI/pl52ret15RfmCI0Uk=', 'Content-Length': '0'} 2023-10-11T14:44:16.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b942631b05ea4205-006526b4c0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.670 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.670 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.671 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.671 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.671 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.672 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.672 DEBUG:boto:Data: 2023-10-11T14:44:16.673 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.673 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.674 DEBUG:boto:Params: {} 2023-10-11T14:44:16.674 DEBUG:boto:Token: None 2023-10-11T14:44:16.675 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WJaHq6/vI/pl52ret15RfmCI0Uk= 2023-10-11T14:44:16.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WJaHq6/vI/pl52ret15RfmCI0Uk=', 'Content-Length': '0'} 2023-10-11T14:44:16.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cb6f6ec6d3ec4f7-006526b4c0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:16.679 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.680 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.680 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.680 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.681 DEBUG:boto:Method: GET 2023-10-11T14:44:16.681 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.682 DEBUG:boto:Data: 2023-10-11T14:44:16.682 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.683 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.684 DEBUG:boto:Params: {} 2023-10-11T14:44:16.684 DEBUG:boto:Token: None 2023-10-11T14:44:16.685 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI= 2023-10-11T14:44:16.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI=', 'Content-Length': '0'} 2023-10-11T14:44:16.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006aa5f4cec04024d3-006526b4c0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.688 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.689 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.689 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.690 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.690 DEBUG:boto:Method: GET 2023-10-11T14:44:16.691 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.691 DEBUG:boto:Data: 2023-10-11T14:44:16.691 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.692 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.692 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.693 DEBUG:boto:Params: {} 2023-10-11T14:44:16.693 DEBUG:boto:Token: None 2023-10-11T14:44:16.694 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:16.694 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI= 2023-10-11T14:44:16.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weSg7417BLPa318uutCx0r/BnlI=', 'Content-Length': '0'} 2023-10-11T14:44:16.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f47c09f7a4f854f4-006526b4c0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.697 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.698 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.698 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.699 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.699 DEBUG:boto:Method: GET 2023-10-11T14:44:16.700 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:16.700 DEBUG:boto:Data: 2023-10-11T14:44:16.700 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.702 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.702 DEBUG:boto:Params: {} 2023-10-11T14:44:16.703 DEBUG:boto:Token: None 2023-10-11T14:44:16.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.704 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8= 2023-10-11T14:44:16.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8=', 'Content-Length': '0'} 2023-10-11T14:44:16.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495012b8aabb24a0-006526b4c0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.707 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.708 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.709 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.709 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.709 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.710 DEBUG:boto:Method: GET 2023-10-11T14:44:16.710 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:16.711 DEBUG:boto:Data: 2023-10-11T14:44:16.711 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.712 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.712 DEBUG:boto:Params: {} 2023-10-11T14:44:16.713 DEBUG:boto:Token: None 2023-10-11T14:44:16.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8= 2023-10-11T14:44:16.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8=', 'Content-Length': '0'} 2023-10-11T14:44:16.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405434b12f4f16be-006526b4c0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.717 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.718 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.719 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.719 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.720 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:16.720 DEBUG:boto:Data: 2023-10-11T14:44:16.720 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.721 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.721 DEBUG:boto:Params: {} 2023-10-11T14:44:16.722 DEBUG:boto:Token: None 2023-10-11T14:44:16.722 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj 2023-10-11T14:44:16.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VFuJ1Xbg/0VOhs7OMzhnxIwMsoo= 2023-10-11T14:44:16.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VFuJ1Xbg/0VOhs7OMzhnxIwMsoo=', 'Content-Length': '0'} 2023-10-11T14:44:16.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6178d26900d3ec7-006526b4c0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.727 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.727 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.727 DEBUG:boto:Method: HEAD 2023-10-11T14:44:16.728 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:16.728 DEBUG:boto:Data: 2023-10-11T14:44:16.729 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.729 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.730 DEBUG:boto:Params: {} 2023-10-11T14:44:16.730 DEBUG:boto:Token: None 2023-10-11T14:44:16.731 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj 2023-10-11T14:44:16.731 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VFuJ1Xbg/0VOhs7OMzhnxIwMsoo= 2023-10-11T14:44:16.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VFuJ1Xbg/0VOhs7OMzhnxIwMsoo=', 'Content-Length': '0'} 2023-10-11T14:44:16.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065f663ae0e3c8b20-006526b4c0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:16.736 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.736 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.737 DEBUG:boto:Method: GET 2023-10-11T14:44:16.737 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:16.738 DEBUG:boto:Data: 2023-10-11T14:44:16.738 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.739 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.739 DEBUG:boto:Params: {} 2023-10-11T14:44:16.740 DEBUG:boto:Token: None 2023-10-11T14:44:16.740 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj 2023-10-11T14:44:16.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X4q7/DOQETMIEhnCvhw0dSLCPF4= 2023-10-11T14:44:16.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X4q7/DOQETMIEhnCvhw0dSLCPF4=', 'Content-Length': '0'} 2023-10-11T14:44:16.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002374ad8af488f2d8-006526b4c0-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.744 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:16.744 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:16.745 DEBUG:boto:Method: GET 2023-10-11T14:44:16.745 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:16.745 DEBUG:boto:Data: 2023-10-11T14:44:16.746 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.747 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.747 DEBUG:boto:Params: {} 2023-10-11T14:44:16.748 DEBUG:boto:Token: None 2023-10-11T14:44:16.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/obj 2023-10-11T14:44:16.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X4q7/DOQETMIEhnCvhw0dSLCPF4= 2023-10-11T14:44:16.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X4q7/DOQETMIEhnCvhw0dSLCPF4=', 'Content-Length': '0'} 2023-10-11T14:44:16.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003990bf7935d343f7-006526b4c0-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.751 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.752 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.752 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.753 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.753 DEBUG:boto:Method: GET 2023-10-11T14:44:16.753 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:16.754 DEBUG:boto:Data: 2023-10-11T14:44:16.754 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:16.755 DEBUG:boto:Port: 8001 2023-10-11T14:44:16.756 DEBUG:boto:Params: {} 2023-10-11T14:44:16.756 DEBUG:boto:Token: None 2023-10-11T14:44:16.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8= 2023-10-11T14:44:16.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8=', 'Content-Length': '0'} 2023-10-11T14:44:16.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009714331281fba15b-006526b4c0-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.802 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.803 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.804 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.804 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.805 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:16.805 DEBUG:boto:Method: GET 2023-10-11T14:44:16.805 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:16.806 DEBUG:boto:Data: 2023-10-11T14:44:16.806 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.807 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.808 DEBUG:boto:Params: {} 2023-10-11T14:44:16.808 DEBUG:boto:Token: None 2023-10-11T14:44:16.809 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/ 2023-10-11T14:44:16.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8= 2023-10-11T14:44:16.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rta/F25nGBHEZ5KXKpw2NbupYy8=', 'Content-Length': '0'} 2023-10-11T14:44:16.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a98e668561a2ae33-006526b4c0-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.813 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:16.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:44:16.814 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:16.814 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:16.815 DEBUG:boto:path=/dghqvd-14/?versioning 2023-10-11T14:44:16.815 DEBUG:boto:auth_path=/dghqvd-14/?versioning 2023-10-11T14:44:16.815 DEBUG:boto:Method: PUT 2023-10-11T14:44:16.816 DEBUG:boto:Path: /dghqvd-14/?versioning 2023-10-11T14:44:16.816 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:44:16.817 DEBUG:boto:Headers: {} 2023-10-11T14:44:16.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:16.817 DEBUG:boto:Port: 8000 2023-10-11T14:44:16.818 DEBUG:boto:Params: {} 2023-10-11T14:44:16.818 DEBUG:boto:Token: None 2023-10-11T14:44:16.819 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:44:16 GMT /dghqvd-14/?versioning 2023-10-11T14:44:16.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2XFaGlS6cCHmP0MUjOkr5HLuWWk= 2023-10-11T14:44:16.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2XFaGlS6cCHmP0MUjOkr5HLuWWk=', 'Content-Length': '228'} 2023-10-11T14:44:16.826 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c279e29c9644934-006526b4c0-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:16.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:16.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:16.827 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:44:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:16.883 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:16.914 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:44:16.927 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:44:16.928 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:16.928 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:44:16.928 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:44:16.928 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:16.928 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:16.929 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:44:16.929 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:44:16.929 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:16.929 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:16.929 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:44:16.930 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:44:16.930 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:16.930 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:16.930 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035456.823162_72.1", 2023-10-11T14:44:16.931 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:44:16.823162Z" 2023-10-11T14:44:16.931 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:16.931 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:44:16.932 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035456.823162_72.1'} 2023-10-11T14:44:16.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:44:16.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:16.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:16.934 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:44:16.990 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:16.990 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:17.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:17.015+0000 7f8f165e7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:17.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:17.016+0000 7f8f165e7780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:17.029 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:44:17.030 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:44:17.030 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:44:17.030 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:44:17.030 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:44:17.030 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:44:17.031 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:17.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.031 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:44:17.031 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:17.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:44:17.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:17.032 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:17.032 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:44:17.032 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:17.033 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:44:17.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:17.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:44:17.033 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:17.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:17.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:17.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:44:17.034 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:17.034 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:17.035 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:44:17.035 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:17.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:17.035 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:17.035 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:44:17.036 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:17.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:17.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.036 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:17.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:44:17.037 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:17.037 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:17.037 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:44:17.037 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:17.037 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:17.038 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:17.038 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:44:17.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:17.038 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:17.039 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035456.823162_72.1", 2023-10-11T14:44:17.040 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:17.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:17.040 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:17.040 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:44:16.823162Z", 2023-10-11T14:44:17.040 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:17.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:17.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:17.041 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:44:17.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.041 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:44:17.042 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:44:17.042 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:44:17.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:17.042 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:44:17Z" 2023-10-11T14:44:17.043 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:44:17.044 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035456.823162_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:44:16.823162Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:44:17Z" } 2023-10-11T14:44:17.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035456.823162_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:44:16.823162Z', 'realm_epoch': 2}}] 2023-10-11T14:44:17.045 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035456.823162_72.1'} 2023-10-11T14:44:17.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035456.823162_72.1'} 2023-10-11T14:44:17.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:44:17.046 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:17.046 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:17.047 DEBUG:boto:Method: HEAD 2023-10-11T14:44:17.047 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:17.047 DEBUG:boto:Data: 2023-10-11T14:44:17.048 DEBUG:boto:Headers: {} 2023-10-11T14:44:17.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:17.048 DEBUG:boto:Port: 8000 2023-10-11T14:44:17.049 DEBUG:boto:Params: {} 2023-10-11T14:44:17.049 DEBUG:boto:Token: None 2023-10-11T14:44:17.050 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:17 GMT /dghqvd-14/ 2023-10-11T14:44:17.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tylwqL8JdYujBe/Me/SXjHOhdAw= 2023-10-11T14:44:17.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tylwqL8JdYujBe/Me/SXjHOhdAw=', 'Content-Length': '0'} 2023-10-11T14:44:17.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d9cdcf1d46becb-006526b4c1-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:17.054 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:17.054 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:17.054 DEBUG:boto:Method: PUT 2023-10-11T14:44:17.055 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:17.055 DEBUG:boto:Data: 2023-10-11T14:44:17.056 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:44:17.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:17.056 DEBUG:boto:Port: 8000 2023-10-11T14:44:17.057 DEBUG:boto:Params: {} 2023-10-11T14:44:17.057 DEBUG:boto:Token: None 2023-10-11T14:44:17.058 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:44:17 GMT /dghqvd-14/obj 2023-10-11T14:44:17.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ceGLToEDHKwW06M7jPSVelECWbE= 2023-10-11T14:44:17.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:44:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ceGLToEDHKwW06M7jPSVelECWbE='} 2023-10-11T14:44:17.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-amz-request-id', 'tx00000c9a4ea9badbcc2c5-006526b4c1-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:17.067 DEBUG:rgw_multi.tests:created new version id=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:17.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-14 2023-10-11T14:44:17.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:17.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:17.069 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:44:17.125 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:17.126 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:17.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:17.156+0000 7fe95dca5780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:17.159 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.3.6, , ] 2023-10-11T14:44:17.160 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:22.158+0000 7fe95dca5780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:22.161 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.3.6, , ] 2023-10-11T14:44:22.161 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:27.158+0000 7fe95dca5780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:27.162 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.3.6, , ] 2023-10-11T14:44:27.162 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:32.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:32.159+0000 7fe95dca5780 1 bucket sync caught up with source: 2023-10-11T14:44:32.163 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:32.163 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:32.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:32.159+0000 7fe95dca5780 0 bucket checkpoint complete 2023-10-11T14:44:32.178 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-14 2023-10-11T14:44:32.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:32.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:32.180 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:44:32.244 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:32.245 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:32.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:32.277+0000 7f641f89f780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:32.281 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:32.282 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:37.282 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:37.279+0000 7f641f89f780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:37.283 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:37.283 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:42.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:42.280+0000 7f641f89f780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:42.284 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:42.284 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:47.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:47.281+0000 7f641f89f780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:47.285 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.6.6, , ] 2023-10-11T14:44:47.286 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:52.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:52.283+0000 7f641f89f780 1 bucket sync caught up with source: 2023-10-11T14:44:52.287 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:52.287 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:44:52.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:52.283+0000 7f641f89f780 0 bucket checkpoint complete 2023-10-11T14:44:52.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:44:52.302 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.303 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.303 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.304 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:52.304 DEBUG:boto:Data: 2023-10-11T14:44:52.304 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.305 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.306 DEBUG:boto:Params: {} 2023-10-11T14:44:52.306 DEBUG:boto:Token: None 2023-10-11T14:44:52.307 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x4ifVYqPmp9gOYRxG59TPNanqIo= 2023-10-11T14:44:52.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x4ifVYqPmp9gOYRxG59TPNanqIo=', 'Content-Length': '0'} 2023-10-11T14:44:52.310 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a63a8dae038a616-006526b4e4-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.311 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.312 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.313 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.314 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:52.314 DEBUG:boto:Data: 2023-10-11T14:44:52.315 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.315 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.316 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.316 DEBUG:boto:Params: {} 2023-10-11T14:44:52.317 DEBUG:boto:Token: None 2023-10-11T14:44:52.317 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x4ifVYqPmp9gOYRxG59TPNanqIo= 2023-10-11T14:44:52.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x4ifVYqPmp9gOYRxG59TPNanqIo=', 'Content-Length': '0'} 2023-10-11T14:44:52.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002422cf23c20fb31f-006526b4e4-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.321 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:44:52.321 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.322 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.322 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.323 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.323 DEBUG:boto:Method: GET 2023-10-11T14:44:52.324 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.324 DEBUG:boto:Data: 2023-10-11T14:44:52.324 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.325 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.326 DEBUG:boto:Params: {} 2023-10-11T14:44:52.326 DEBUG:boto:Token: None 2023-10-11T14:44:52.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/?versions 2023-10-11T14:44:52.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg= 2023-10-11T14:44:52.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg=', 'Content-Length': '0'} 2023-10-11T14:44:52.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da7920e6a2dd285c-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.331 DEBUG:boto:b'dghqvd-14999falseobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0true2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:52.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:52.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:44:52.333 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.333 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.334 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.334 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.334 DEBUG:boto:Method: GET 2023-10-11T14:44:52.335 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.335 DEBUG:boto:Data: 2023-10-11T14:44:52.336 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.336 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.337 DEBUG:boto:Params: {} 2023-10-11T14:44:52.337 DEBUG:boto:Token: None 2023-10-11T14:44:52.338 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/?versions 2023-10-11T14:44:52.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg= 2023-10-11T14:44:52.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg=', 'Content-Length': '0'} 2023-10-11T14:44:52.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004333c6c7ebb0d7b1-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.342 DEBUG:boto:b'dghqvd-14999falseobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0true2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:52.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:44:52.343 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.344 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.345 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.346 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.346 DEBUG:boto:Method: GET 2023-10-11T14:44:52.347 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.347 DEBUG:boto:Data: 2023-10-11T14:44:52.348 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.348 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.349 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.349 DEBUG:boto:Params: {} 2023-10-11T14:44:52.350 DEBUG:boto:Token: None 2023-10-11T14:44:52.350 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/?versions 2023-10-11T14:44:52.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg= 2023-10-11T14:44:52.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg=', 'Content-Length': '0'} 2023-10-11T14:44:52.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b223797da57376b-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.354 DEBUG:boto:b'dghqvd-14999falseobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0true2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.355 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.356 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.356 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.356 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.357 DEBUG:boto:Method: GET 2023-10-11T14:44:52.357 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:44:52.358 DEBUG:boto:Data: 2023-10-11T14:44:52.358 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.359 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.359 DEBUG:boto:Params: {} 2023-10-11T14:44:52.360 DEBUG:boto:Token: None 2023-10-11T14:44:52.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/?versions 2023-10-11T14:44:52.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg= 2023-10-11T14:44:52.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q8LeAo1D7rHpVkLnZW2WrLeqoLg=', 'Content-Length': '0'} 2023-10-11T14:44:52.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5f4bdc075dbb6af-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.365 DEBUG:boto:b'dghqvd-14999falseobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0true2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-10-11T14:43:56.253Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.366 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.367 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.367 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.367 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.368 DEBUG:boto:Method: GET 2023-10-11T14:44:52.369 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.369 DEBUG:boto:Data: 2023-10-11T14:44:52.369 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.370 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.371 DEBUG:boto:Params: {} 2023-10-11T14:44:52.371 DEBUG:boto:Token: None 2023-10-11T14:44:52.372 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000769b550fc56f8887-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.375 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.376 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.376 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.377 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.377 DEBUG:boto:Method: GET 2023-10-11T14:44:52.377 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.378 DEBUG:boto:Data: 2023-10-11T14:44:52.378 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.379 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.380 DEBUG:boto:Params: {} 2023-10-11T14:44:52.380 DEBUG:boto:Token: None 2023-10-11T14:44:52.381 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5ee418de26e1962-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.385 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.385 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.385 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.386 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.386 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.387 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.387 DEBUG:boto:Data: 2023-10-11T14:44:52.387 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.388 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.389 DEBUG:boto:Params: {} 2023-10-11T14:44:52.389 DEBUG:boto:Token: None 2023-10-11T14:44:52.390 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.390 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7kw2f5j3xvAYVmqFE1RFFmdP5eY= 2023-10-11T14:44:52.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7kw2f5j3xvAYVmqFE1RFFmdP5eY=', 'Content-Length': '0'} 2023-10-11T14:44:52.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005637e23084e7928d-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.393 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.393 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.394 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.394 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.395 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.395 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.395 DEBUG:boto:Data: 2023-10-11T14:44:52.396 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.397 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.397 DEBUG:boto:Params: {} 2023-10-11T14:44:52.397 DEBUG:boto:Token: None 2023-10-11T14:44:52.398 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7kw2f5j3xvAYVmqFE1RFFmdP5eY= 2023-10-11T14:44:52.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7kw2f5j3xvAYVmqFE1RFFmdP5eY=', 'Content-Length': '0'} 2023-10-11T14:44:52.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000385e0404673c33ea-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.402 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.402 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.403 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.403 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.404 DEBUG:boto:Method: GET 2023-10-11T14:44:52.404 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.404 DEBUG:boto:Data: 2023-10-11T14:44:52.405 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.406 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.406 DEBUG:boto:Params: {} 2023-10-11T14:44:52.407 DEBUG:boto:Token: None 2023-10-11T14:44:52.407 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5d2774421e20387-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.411 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.411 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.412 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.412 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.412 DEBUG:boto:Method: GET 2023-10-11T14:44:52.413 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.413 DEBUG:boto:Data: 2023-10-11T14:44:52.414 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.415 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.415 DEBUG:boto:Params: {} 2023-10-11T14:44:52.415 DEBUG:boto:Token: None 2023-10-11T14:44:52.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c8ee4a4084e3d6bc-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.420 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.420 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.421 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.421 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.421 DEBUG:boto:Method: GET 2023-10-11T14:44:52.422 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.422 DEBUG:boto:Data: 2023-10-11T14:44:52.423 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.424 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.424 DEBUG:boto:Params: {} 2023-10-11T14:44:52.425 DEBUG:boto:Token: None 2023-10-11T14:44:52.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2bc6c2355d0d569-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.474 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.475 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.476 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.476 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.477 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.477 DEBUG:boto:Method: GET 2023-10-11T14:44:52.477 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.478 DEBUG:boto:Data: 2023-10-11T14:44:52.478 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.479 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.480 DEBUG:boto:Params: {} 2023-10-11T14:44:52.480 DEBUG:boto:Token: None 2023-10-11T14:44:52.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab30354927367ee2-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.484 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.485 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.486 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.486 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.486 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:52.487 DEBUG:boto:Data: 2023-10-11T14:44:52.487 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.488 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.488 DEBUG:boto:Params: {} 2023-10-11T14:44:52.489 DEBUG:boto:Token: None 2023-10-11T14:44:52.489 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj 2023-10-11T14:44:52.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI= 2023-10-11T14:44:52.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI=', 'Content-Length': '0'} 2023-10-11T14:44:52.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000050cfcc153181168e-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.495 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.495 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.495 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.496 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:52.496 DEBUG:boto:Data: 2023-10-11T14:44:52.497 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.497 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.498 DEBUG:boto:Params: {} 2023-10-11T14:44:52.498 DEBUG:boto:Token: None 2023-10-11T14:44:52.499 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj 2023-10-11T14:44:52.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI= 2023-10-11T14:44:52.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI=', 'Content-Length': '0'} 2023-10-11T14:44:52.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcf68d084973f391-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.503 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.504 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.504 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.504 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.505 DEBUG:boto:Method: GET 2023-10-11T14:44:52.505 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.506 DEBUG:boto:Data: 2023-10-11T14:44:52.506 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.507 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.507 DEBUG:boto:Params: {} 2023-10-11T14:44:52.508 DEBUG:boto:Token: None 2023-10-11T14:44:52.508 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e6a78535659aa0b1-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.513 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.514 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.514 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.514 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.515 DEBUG:boto:Method: GET 2023-10-11T14:44:52.515 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.516 DEBUG:boto:Data: 2023-10-11T14:44:52.516 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.517 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.517 DEBUG:boto:Params: {} 2023-10-11T14:44:52.518 DEBUG:boto:Token: None 2023-10-11T14:44:52.518 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f2da144f4b7258b8-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.522 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.523 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.523 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.524 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.524 DEBUG:boto:Method: GET 2023-10-11T14:44:52.525 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.525 DEBUG:boto:Data: 2023-10-11T14:44:52.525 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.526 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.527 DEBUG:boto:Params: {} 2023-10-11T14:44:52.527 DEBUG:boto:Token: None 2023-10-11T14:44:52.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090aa1d3e5e25cf7f-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.532 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.533 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.533 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.533 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.534 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.534 DEBUG:boto:Method: GET 2023-10-11T14:44:52.535 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.535 DEBUG:boto:Data: 2023-10-11T14:44:52.536 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.537 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.537 DEBUG:boto:Params: {} 2023-10-11T14:44:52.537 DEBUG:boto:Token: None 2023-10-11T14:44:52.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000068d6f0844b378ae-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.542 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.543 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.544 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.544 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.544 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.545 DEBUG:boto:Method: GET 2023-10-11T14:44:52.545 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.546 DEBUG:boto:Data: 2023-10-11T14:44:52.546 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.547 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.547 DEBUG:boto:Params: {} 2023-10-11T14:44:52.548 DEBUG:boto:Token: None 2023-10-11T14:44:52.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s= 2023-10-11T14:44:52.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s=', 'Content-Length': '0'} 2023-10-11T14:44:52.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000090d51786e8a54220-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.552 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.553 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.553 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.553 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.554 DEBUG:boto:Method: GET 2023-10-11T14:44:52.554 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.555 DEBUG:boto:Data: 2023-10-11T14:44:52.555 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.556 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.556 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.556 DEBUG:boto:Params: {} 2023-10-11T14:44:52.557 DEBUG:boto:Token: None 2023-10-11T14:44:52.557 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s= 2023-10-11T14:44:52.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s=', 'Content-Length': '0'} 2023-10-11T14:44:52.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f04beefe515696f-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.561 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.562 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.562 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.562 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.563 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.563 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.564 DEBUG:boto:Data: 2023-10-11T14:44:52.564 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.565 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.565 DEBUG:boto:Params: {} 2023-10-11T14:44:52.566 DEBUG:boto:Token: None 2023-10-11T14:44:52.566 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzpoBso0PzfXmz9VcKX1QUZ3mZY= 2023-10-11T14:44:52.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzpoBso0PzfXmz9VcKX1QUZ3mZY=', 'Content-Length': '0'} 2023-10-11T14:44:52.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dbbe152c49cb5640-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.569 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.570 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.570 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.571 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.571 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.572 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.573 DEBUG:boto:Data: 2023-10-11T14:44:52.573 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.574 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.575 DEBUG:boto:Params: {} 2023-10-11T14:44:52.575 DEBUG:boto:Token: None 2023-10-11T14:44:52.576 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.576 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzpoBso0PzfXmz9VcKX1QUZ3mZY= 2023-10-11T14:44:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzpoBso0PzfXmz9VcKX1QUZ3mZY=', 'Content-Length': '0'} 2023-10-11T14:44:52.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092ceac3b31e25c2e-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.579 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.580 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.580 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.581 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.581 DEBUG:boto:Method: GET 2023-10-11T14:44:52.582 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.582 DEBUG:boto:Data: 2023-10-11T14:44:52.582 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.583 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.584 DEBUG:boto:Params: {} 2023-10-11T14:44:52.584 DEBUG:boto:Token: None 2023-10-11T14:44:52.585 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s= 2023-10-11T14:44:52.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s=', 'Content-Length': '0'} 2023-10-11T14:44:52.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b1f8e4e3cf1599a6-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.589 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.589 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.590 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.590 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.591 DEBUG:boto:Method: GET 2023-10-11T14:44:52.591 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.592 DEBUG:boto:Data: 2023-10-11T14:44:52.592 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.593 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.593 DEBUG:boto:Params: {} 2023-10-11T14:44:52.594 DEBUG:boto:Token: None 2023-10-11T14:44:52.594 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:44:52.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s= 2023-10-11T14:44:52.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SNJ3VEkfv8BOWmFwwZAS/s6J45s=', 'Content-Length': '0'} 2023-10-11T14:44:52.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:43:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002ab919d911bf46a-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.598 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.599 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.599 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.600 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.600 DEBUG:boto:Method: GET 2023-10-11T14:44:52.601 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.601 DEBUG:boto:Data: 2023-10-11T14:44:52.601 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.602 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.603 DEBUG:boto:Params: {} 2023-10-11T14:44:52.603 DEBUG:boto:Token: None 2023-10-11T14:44:52.604 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a758f042e4288d-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.608 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.609 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.609 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.609 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.610 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.610 DEBUG:boto:Method: GET 2023-10-11T14:44:52.611 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.611 DEBUG:boto:Data: 2023-10-11T14:44:52.612 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.612 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.613 DEBUG:boto:Params: {} 2023-10-11T14:44:52.613 DEBUG:boto:Token: None 2023-10-11T14:44:52.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c497d74c88eff0-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.659 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.660 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.660 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.661 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.661 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:52.662 DEBUG:boto:Data: 2023-10-11T14:44:52.662 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.663 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.663 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.664 DEBUG:boto:Params: {} 2023-10-11T14:44:52.664 DEBUG:boto:Token: None 2023-10-11T14:44:52.665 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj 2023-10-11T14:44:52.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI= 2023-10-11T14:44:52.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI=', 'Content-Length': '0'} 2023-10-11T14:44:52.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000035deb8288aa2053-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.669 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.669 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.670 DEBUG:boto:Method: HEAD 2023-10-11T14:44:52.670 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:52.670 DEBUG:boto:Data: 2023-10-11T14:44:52.671 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.672 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.672 DEBUG:boto:Params: {} 2023-10-11T14:44:52.673 DEBUG:boto:Token: None 2023-10-11T14:44:52.673 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj 2023-10-11T14:44:52.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI= 2023-10-11T14:44:52.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wEHEefwvAcBQN3QBLCVJNEcFiCI=', 'Content-Length': '0'} 2023-10-11T14:44:52.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091d0acd6eec65c42-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:44:52.677 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.678 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.678 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.679 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.679 DEBUG:boto:Method: GET 2023-10-11T14:44:52.680 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.680 DEBUG:boto:Data: 2023-10-11T14:44:52.681 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.681 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.682 DEBUG:boto:Params: {} 2023-10-11T14:44:52.682 DEBUG:boto:Token: None 2023-10-11T14:44:52.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c2aea5e9a58c831-006526b4e4-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.687 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:52.687 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:52.688 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.688 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.688 DEBUG:boto:Method: GET 2023-10-11T14:44:52.689 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.689 DEBUG:boto:Data: 2023-10-11T14:44:52.690 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.691 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.691 DEBUG:boto:Params: {} 2023-10-11T14:44:52.692 DEBUG:boto:Token: None 2023-10-11T14:44:52.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:44:52.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ= 2023-10-11T14:44:52.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pyHOdDaQ4ftzlAo8helQEkb3QPQ=', 'Content-Length': '0'} 2023-10-11T14:44:52.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbdba05ab08db930-006526b4e4-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.696 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.697 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.697 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.698 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.698 DEBUG:boto:Method: GET 2023-10-11T14:44:52.699 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.699 DEBUG:boto:Data: 2023-10-11T14:44:52.700 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.700 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:44:52.701 DEBUG:boto:Port: 8001 2023-10-11T14:44:52.701 DEBUG:boto:Params: {} 2023-10-11T14:44:52.702 DEBUG:boto:Token: None 2023-10-11T14:44:52.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000349df8790bd0773e-006526b4e4-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.754 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.755 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.758 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.759 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.759 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:44:52.759 DEBUG:boto:Method: GET 2023-10-11T14:44:52.760 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:44:52.761 DEBUG:boto:Data: 2023-10-11T14:44:52.761 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.763 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.763 DEBUG:boto:Params: {} 2023-10-11T14:44:52.764 DEBUG:boto:Token: None 2023-10-11T14:44:52.765 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/ 2023-10-11T14:44:52.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc= 2023-10-11T14:44:52.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d8Nu4WvGosU7uUp61UCGEFuDsWc=', 'Content-Length': '0'} 2023-10-11T14:44:52.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014994fd0682cb1a-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.772 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:44:52.773 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:44:52.774 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:52.774 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:52.775 DEBUG:boto:path=/dghqvd-14/?versioning 2023-10-11T14:44:52.776 DEBUG:boto:auth_path=/dghqvd-14/?versioning 2023-10-11T14:44:52.776 DEBUG:boto:Method: PUT 2023-10-11T14:44:52.777 DEBUG:boto:Path: /dghqvd-14/?versioning 2023-10-11T14:44:52.778 DEBUG:boto:Data: Suspended Disabled 2023-10-11T14:44:52.778 DEBUG:boto:Headers: {} 2023-10-11T14:44:52.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:52.780 DEBUG:boto:Port: 8000 2023-10-11T14:44:52.780 DEBUG:boto:Params: {} 2023-10-11T14:44:52.781 DEBUG:boto:Token: None 2023-10-11T14:44:52.781 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:44:52 GMT /dghqvd-14/?versioning 2023-10-11T14:44:52.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d05QS7dNsL52bOaeBmp8hufEzb4= 2023-10-11T14:44:52.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d05QS7dNsL52bOaeBmp8hufEzb4=', 'Content-Length': '230'} 2023-10-11T14:44:52.789 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000024cbaa77db1b1614-006526b4e4-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:52.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:52.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:52.791 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:44:52.848 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:52.848 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:52.880 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:44:52.892 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:44:52.893 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:52.893 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:44:52.893 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:44:52.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:52.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:52.894 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:44:52.894 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:44:52.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:52.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:52.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:44:52.895 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:44:52.895 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:52.896 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:52.896 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035492.786269_74.1", 2023-10-11T14:44:52.896 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:44:52.786269Z" 2023-10-11T14:44:52.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:52.897 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:44:52.898 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035492.786269_74.1'} 2023-10-11T14:44:52.898 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:44:52.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:52.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:52.900 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:44:52.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:52.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:52.980 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:52.976+0000 7fd015dcf780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:52.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:52.977+0000 7fd015dcf780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:52.995 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:44:52.996 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:44:52.996 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:44:52.996 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:44:52.996 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:44:52.997 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:44:52.997 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:52.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:52.997 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:44:52.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:52.998 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:44:52.998 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:52.998 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:52.998 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:44:52.999 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:52.999 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:44:52.999 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:53.000 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:44:53.000 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:53.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:53.000 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:53.000 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:53.001 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:44:53.001 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:53.001 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:53.001 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:44:53.002 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:53.002 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:53.002 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:53.002 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:44:53.002 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:53.003 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:53.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:53.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:53.003 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:44:53.004 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:53.004 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:53.004 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:44:53.004 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:53.004 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:53.005 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:53.005 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:44:53.005 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:53.005 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:53.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:44:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035492.782951_73.1", 2023-10-11T14:44:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:44:52.782951Z", 2023-10-11T14:44:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:44:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:44:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:44:53.010 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:44:53.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:53.010 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:44:52Z" 2023-10-11T14:44:53.010 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:44:53.011 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035492.782951_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:44:52.782951Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:44:52Z" } 2023-10-11T14:44:53.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035492.782951_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:44:52.782951Z', 'realm_epoch': 2}}] 2023-10-11T14:44:53.012 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035492.786269_74.1'} 2023-10-11T14:44:53.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035492.782951_73.1'} 2023-10-11T14:44:53.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035492.786269_74.1 target=1_1697035492.782951_73.1 2023-10-11T14:44:58.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:58.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:44:58.015 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:44:58.072 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:58.073 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:58.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:58.106+0000 7fe4f02a9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:58.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:58.107+0000 7fe4f02a9780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:44:58.123 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:44:58.123 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:44:58.124 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:44:58.124 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:44:58.124 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:44:58.124 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:44:58.124 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:58.125 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.125 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:44:58.125 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:58.125 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:44:58.125 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:58.126 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:58.126 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:44:58.126 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:58.126 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:44:58.127 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:58.127 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:44:58.127 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:58.127 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:58.127 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.128 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:58.128 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:44:58.128 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:58.128 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:58.128 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:44:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:44:58.129 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:58.130 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:58.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:58.131 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:44:58.131 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:58.131 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:58.131 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:44:58.132 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:58.132 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:58.132 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:58.132 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:44:58.132 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:58.133 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:58.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.133 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:44:58.133 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:44:58.133 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:44:58.134 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:44:58.134 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035492.786269_74.1", 2023-10-11T14:44:58.134 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:44:58.134 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:44:58.134 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:44:58.135 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:44:52.786269Z", 2023-10-11T14:44:58.135 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:44:58.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:58.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:44:58.135 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:44:58.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.136 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:44:58.136 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:44:58.136 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:44:58.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:44:58.137 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:44:58Z" 2023-10-11T14:44:58.137 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:44:58.138 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035492.786269_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:44:52.786269Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:44:58Z" } 2023-10-11T14:44:58.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035492.786269_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:44:52.786269Z', 'realm_epoch': 2}}] 2023-10-11T14:44:58.139 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035492.786269_74.1'} 2023-10-11T14:44:58.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035492.786269_74.1'} 2023-10-11T14:44:58.140 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:44:58.141 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:44:58.141 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:44:58.142 DEBUG:boto:Method: HEAD 2023-10-11T14:44:58.142 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:44:58.143 DEBUG:boto:Data: 2023-10-11T14:44:58.143 DEBUG:boto:Headers: {} 2023-10-11T14:44:58.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:58.144 DEBUG:boto:Port: 8000 2023-10-11T14:44:58.144 DEBUG:boto:Params: {} 2023-10-11T14:44:58.145 DEBUG:boto:Token: None 2023-10-11T14:44:58.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:44:58 GMT /dghqvd-14/ 2023-10-11T14:44:58.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:78yaAV+EnFg1dA+yDCI4R9+JRdg= 2023-10-11T14:44:58.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:44:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:78yaAV+EnFg1dA+yDCI4R9+JRdg=', 'Content-Length': '0'} 2023-10-11T14:44:58.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1deb08c3eb95fb1-006526b4ea-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:58.150 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:44:58.151 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:44:58.152 DEBUG:boto:Method: PUT 2023-10-11T14:44:58.152 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:44:58.153 DEBUG:boto:Data: 2023-10-11T14:44:58.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:44:58.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:44:58.154 DEBUG:boto:Port: 8000 2023-10-11T14:44:58.154 DEBUG:boto:Params: {} 2023-10-11T14:44:58.155 DEBUG:boto:Token: None 2023-10-11T14:44:58.155 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:44:58 GMT /dghqvd-14/obj 2023-10-11T14:44:58.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UePd3PKOPhgaN/vCq24f9lRyXo4= 2023-10-11T14:44:58.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:44:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UePd3PKOPhgaN/vCq24f9lRyXo4='} 2023-10-11T14:44:58.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fdad3e56fb075a2-006526b4ea-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:44:58.168 DEBUG:rgw_multi.tests:created null version id=None 2023-10-11T14:44:58.168 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-14 2023-10-11T14:44:58.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:58.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:44:58.169 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:44:58.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:44:58.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:44:58.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:44:58.260+0000 7f8dfb7b0780 1 waiting for incremental sync to catch up: 2023-10-11T14:44:58.264 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:44:58.264 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.10.18, , ] 2023-10-11T14:45:03.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:03.261+0000 7f8dfb7b0780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:03.265 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:45:03.265 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.10.18, , ] 2023-10-11T14:45:08.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:08.263+0000 7f8dfb7b0780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:08.266 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.6.13, , ] 2023-10-11T14:45:08.266 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.10.18, , ] 2023-10-11T14:45:13.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:13.264+0000 7f8dfb7b0780 1 bucket sync caught up with source: 2023-10-11T14:45:13.267 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000005.10.18, , ] 2023-10-11T14:45:13.268 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.10.18, , ] 2023-10-11T14:45:13.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:13.264+0000 7f8dfb7b0780 0 bucket checkpoint complete 2023-10-11T14:45:13.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-14 2023-10-11T14:45:13.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:13.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:13.285 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:45:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:13.347 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:13.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:13.380+0000 7f666b4a6780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:13.383 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:45:13.383 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:18.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:18.381+0000 7f666b4a6780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:18.384 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:45:18.384 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:23.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:23.382+0000 7f666b4a6780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:23.385 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:45:23.385 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:28.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:28.383+0000 7f666b4a6780 1 waiting for incremental sync to catch up: 2023-10-11T14:45:28.386 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.10.13, , ] 2023-10-11T14:45:28.387 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:33.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:33.385+0000 7f666b4a6780 1 bucket sync caught up with source: 2023-10-11T14:45:33.388 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:33.388 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000005.15.18, , ] 2023-10-11T14:45:33.388 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:33.385+0000 7f666b4a6780 0 bucket checkpoint complete 2023-10-11T14:45:33.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:45:33.404 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.405 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.405 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.406 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:45:33.406 DEBUG:boto:Data: 2023-10-11T14:45:33.407 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.408 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.408 DEBUG:boto:Params: {} 2023-10-11T14:45:33.409 DEBUG:boto:Token: None 2023-10-11T14:45:33.409 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.410 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1sEvz7lddcWhXc+5SXRWDTDJwU= 2023-10-11T14:45:33.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1sEvz7lddcWhXc+5SXRWDTDJwU=', 'Content-Length': '0'} 2023-10-11T14:45:33.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081e80c764b764ce6-006526b50d-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.414 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.414 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.414 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.415 DEBUG:boto:Path: /dghqvd-14/ 2023-10-11T14:45:33.415 DEBUG:boto:Data: 2023-10-11T14:45:33.416 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.416 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.418 DEBUG:boto:Params: {} 2023-10-11T14:45:33.418 DEBUG:boto:Token: None 2023-10-11T14:45:33.419 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e1sEvz7lddcWhXc+5SXRWDTDJwU= 2023-10-11T14:45:33.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e1sEvz7lddcWhXc+5SXRWDTDJwU=', 'Content-Length': '0'} 2023-10-11T14:45:33.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005a3b572e72c5207-006526b50d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:45:33.423 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.424 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.424 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.424 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.425 DEBUG:boto:Method: GET 2023-10-11T14:45:33.425 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.426 DEBUG:boto:Data: 2023-10-11T14:45:33.426 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.427 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.428 DEBUG:boto:Params: {} 2023-10-11T14:45:33.428 DEBUG:boto:Token: None 2023-10-11T14:45:33.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/?versions 2023-10-11T14:45:33.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY= 2023-10-11T14:45:33.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY=', 'Content-Length': '0'} 2023-10-11T14:45:33.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f89a417ccc1db645-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.433 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0false2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:33.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:33.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:45:33.436 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.436 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.437 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.437 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.438 DEBUG:boto:Method: GET 2023-10-11T14:45:33.438 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.439 DEBUG:boto:Data: 2023-10-11T14:45:33.439 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.440 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.441 DEBUG:boto:Params: {} 2023-10-11T14:45:33.441 DEBUG:boto:Token: None 2023-10-11T14:45:33.442 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/?versions 2023-10-11T14:45:33.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY= 2023-10-11T14:45:33.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY=', 'Content-Length': '0'} 2023-10-11T14:45:33.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118347563d74361e-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.447 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0false2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:33.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:33.448 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.449 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.449 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.449 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.450 DEBUG:boto:Method: GET 2023-10-11T14:45:33.450 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.451 DEBUG:boto:Data: 2023-10-11T14:45:33.451 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.452 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.452 DEBUG:boto:Params: {} 2023-10-11T14:45:33.453 DEBUG:boto:Token: None 2023-10-11T14:45:33.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/?versions 2023-10-11T14:45:33.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY= 2023-10-11T14:45:33.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY=', 'Content-Length': '0'} 2023-10-11T14:45:33.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9102c01c5348998-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.457 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0false2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.458 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.459 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.459 DEBUG:boto:path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.459 DEBUG:boto:auth_path=/dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.460 DEBUG:boto:Method: GET 2023-10-11T14:45:33.460 DEBUG:boto:Path: /dghqvd-14/?versions&max-keys=999 2023-10-11T14:45:33.461 DEBUG:boto:Data: 2023-10-11T14:45:33.461 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.462 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.462 DEBUG:boto:Params: {} 2023-10-11T14:45:33.463 DEBUG:boto:Token: None 2023-10-11T14:45:33.463 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/?versions 2023-10-11T14:45:33.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY= 2023-10-11T14:45:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NtVcwDriqavUVmmyk0JV7j3vgSY=', 'Content-Length': '0'} 2023-10-11T14:45:33.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c535e1e9943124c5-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.468 DEBUG:boto:b'dghqvd-14999falseobjnulltrue2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0false2023-10-11T14:44:17.058Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.469 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.469 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.470 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.470 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.471 DEBUG:boto:Method: GET 2023-10-11T14:45:33.471 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.472 DEBUG:boto:Data: 2023-10-11T14:45:33.473 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.473 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.474 DEBUG:boto:Params: {} 2023-10-11T14:45:33.475 DEBUG:boto:Token: None 2023-10-11T14:45:33.475 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk= 2023-10-11T14:45:33.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk=', 'Content-Length': '0'} 2023-10-11T14:45:33.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000061673836f99bcf67-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.479 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.479 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.480 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.480 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.481 DEBUG:boto:Method: GET 2023-10-11T14:45:33.481 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.481 DEBUG:boto:Data: 2023-10-11T14:45:33.482 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.483 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.483 DEBUG:boto:Params: {} 2023-10-11T14:45:33.484 DEBUG:boto:Token: None 2023-10-11T14:45:33.484 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk= 2023-10-11T14:45:33.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk=', 'Content-Length': '0'} 2023-10-11T14:45:33.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006962d9f80c1e9e07-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.488 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.488 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.488 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.489 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.489 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.490 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.490 DEBUG:boto:Data: 2023-10-11T14:45:33.491 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.491 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.492 DEBUG:boto:Params: {} 2023-10-11T14:45:33.492 DEBUG:boto:Token: None 2023-10-11T14:45:33.493 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gyi2L0KG2Sm++m1E7W8j7AL4hQI= 2023-10-11T14:45:33.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gyi2L0KG2Sm++m1E7W8j7AL4hQI=', 'Content-Length': '0'} 2023-10-11T14:45:33.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000034b8f2926197cf39-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.496 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.497 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.497 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.498 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.498 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.499 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.499 DEBUG:boto:Data: 2023-10-11T14:45:33.499 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.500 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.501 DEBUG:boto:Params: {} 2023-10-11T14:45:33.501 DEBUG:boto:Token: None 2023-10-11T14:45:33.502 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gyi2L0KG2Sm++m1E7W8j7AL4hQI= 2023-10-11T14:45:33.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gyi2L0KG2Sm++m1E7W8j7AL4hQI=', 'Content-Length': '0'} 2023-10-11T14:45:33.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036c9b1520c0af332-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.506 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.507 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.507 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.507 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.508 DEBUG:boto:Method: GET 2023-10-11T14:45:33.508 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.509 DEBUG:boto:Data: 2023-10-11T14:45:33.509 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.510 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.511 DEBUG:boto:Params: {} 2023-10-11T14:45:33.511 DEBUG:boto:Token: None 2023-10-11T14:45:33.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk= 2023-10-11T14:45:33.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk=', 'Content-Length': '0'} 2023-10-11T14:45:33.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051e75cb91154d315-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.515 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.516 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.516 DEBUG:boto:path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.517 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.517 DEBUG:boto:Method: GET 2023-10-11T14:45:33.518 DEBUG:boto:Path: /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.518 DEBUG:boto:Data: 2023-10-11T14:45:33.518 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.519 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.520 DEBUG:boto:Params: {} 2023-10-11T14:45:33.520 DEBUG:boto:Token: None 2023-10-11T14:45:33.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=null 2023-10-11T14:45:33.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk= 2023-10-11T14:45:33.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QrQPmGQR8cFt5qyvjirrZdjVsnk=', 'Content-Length': '0'} 2023-10-11T14:45:33.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000091a710e217f0e845-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.524 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.525 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.525 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.526 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.526 DEBUG:boto:Method: GET 2023-10-11T14:45:33.527 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.527 DEBUG:boto:Data: 2023-10-11T14:45:33.528 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.529 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.529 DEBUG:boto:Params: {} 2023-10-11T14:45:33.529 DEBUG:boto:Token: None 2023-10-11T14:45:33.530 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079edd47cf27d2871-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.534 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.535 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.537 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.538 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.538 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.538 DEBUG:boto:Method: GET 2023-10-11T14:45:33.539 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.539 DEBUG:boto:Data: 2023-10-11T14:45:33.540 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.541 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.541 DEBUG:boto:Params: {} 2023-10-11T14:45:33.542 DEBUG:boto:Token: None 2023-10-11T14:45:33.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab3a24523f7dc39-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.590 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.591 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.592 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.592 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.593 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.593 DEBUG:boto:Data: 2023-10-11T14:45:33.593 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.594 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.595 DEBUG:boto:Params: {} 2023-10-11T14:45:33.595 DEBUG:boto:Token: None 2023-10-11T14:45:33.596 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA= 2023-10-11T14:45:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA=', 'Content-Length': '0'} 2023-10-11T14:45:33.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f4afdbf9386a0fbe-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.599 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.600 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.600 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.601 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.601 DEBUG:boto:Data: 2023-10-11T14:45:33.602 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.602 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.603 DEBUG:boto:Params: {} 2023-10-11T14:45:33.603 DEBUG:boto:Token: None 2023-10-11T14:45:33.604 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA= 2023-10-11T14:45:33.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA=', 'Content-Length': '0'} 2023-10-11T14:45:33.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000049e107b181fd602f-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.608 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.608 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.609 DEBUG:boto:Method: GET 2023-10-11T14:45:33.609 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.610 DEBUG:boto:Data: 2023-10-11T14:45:33.610 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.611 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.611 DEBUG:boto:Params: {} 2023-10-11T14:45:33.612 DEBUG:boto:Token: None 2023-10-11T14:45:33.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ= 2023-10-11T14:45:33.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ=', 'Content-Length': '0'} 2023-10-11T14:45:33.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9a061365a1e5c8f-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.616 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.617 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.617 DEBUG:boto:Method: GET 2023-10-11T14:45:33.617 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.618 DEBUG:boto:Data: 2023-10-11T14:45:33.618 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.619 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.620 DEBUG:boto:Params: {} 2023-10-11T14:45:33.620 DEBUG:boto:Token: None 2023-10-11T14:45:33.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ= 2023-10-11T14:45:33.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ=', 'Content-Length': '0'} 2023-10-11T14:45:33.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d38d58307affb901-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.624 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.625 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.625 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.626 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.626 DEBUG:boto:Method: GET 2023-10-11T14:45:33.626 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.627 DEBUG:boto:Data: 2023-10-11T14:45:33.627 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.628 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.629 DEBUG:boto:Params: {} 2023-10-11T14:45:33.629 DEBUG:boto:Token: None 2023-10-11T14:45:33.630 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfb4485deb31332-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.678 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.679 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.680 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.680 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.681 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.681 DEBUG:boto:Method: GET 2023-10-11T14:45:33.681 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.682 DEBUG:boto:Data: 2023-10-11T14:45:33.682 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.683 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.684 DEBUG:boto:Params: {} 2023-10-11T14:45:33.684 DEBUG:boto:Token: None 2023-10-11T14:45:33.685 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006176a93ee4e993c7-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.689 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.691 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.691 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.692 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.692 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.692 DEBUG:boto:Method: GET 2023-10-11T14:45:33.693 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.693 DEBUG:boto:Data: 2023-10-11T14:45:33.694 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.695 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.695 DEBUG:boto:Params: {} 2023-10-11T14:45:33.696 DEBUG:boto:Token: None 2023-10-11T14:45:33.696 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI= 2023-10-11T14:45:33.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI=', 'Content-Length': '0'} 2023-10-11T14:45:33.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023e6169d1e158880-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.700 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.700 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.701 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.701 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.702 DEBUG:boto:Method: GET 2023-10-11T14:45:33.702 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.702 DEBUG:boto:Data: 2023-10-11T14:45:33.703 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.704 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.704 DEBUG:boto:Params: {} 2023-10-11T14:45:33.705 DEBUG:boto:Token: None 2023-10-11T14:45:33.705 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.706 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI= 2023-10-11T14:45:33.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI=', 'Content-Length': '0'} 2023-10-11T14:45:33.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003b757f004ff4b7ee-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.709 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.709 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.710 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.710 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.711 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.711 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.711 DEBUG:boto:Data: 2023-10-11T14:45:33.712 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.713 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.713 DEBUG:boto:Params: {} 2023-10-11T14:45:33.714 DEBUG:boto:Token: None 2023-10-11T14:45:33.714 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ykefujsW9xWl35AUipBaoCWZMFE= 2023-10-11T14:45:33.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ykefujsW9xWl35AUipBaoCWZMFE=', 'Content-Length': '0'} 2023-10-11T14:45:33.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003566319abaef6f1-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.717 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.718 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.718 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.719 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.719 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.719 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.720 DEBUG:boto:Data: 2023-10-11T14:45:33.720 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.721 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.721 DEBUG:boto:Params: {} 2023-10-11T14:45:33.722 DEBUG:boto:Token: None 2023-10-11T14:45:33.722 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ykefujsW9xWl35AUipBaoCWZMFE= 2023-10-11T14:45:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ykefujsW9xWl35AUipBaoCWZMFE=', 'Content-Length': '0'} 2023-10-11T14:45:33.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000616d3ad90eb1b9e7-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.726 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.727 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.727 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.728 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.728 DEBUG:boto:Method: GET 2023-10-11T14:45:33.728 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.729 DEBUG:boto:Data: 2023-10-11T14:45:33.729 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.730 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.731 DEBUG:boto:Params: {} 2023-10-11T14:45:33.731 DEBUG:boto:Token: None 2023-10-11T14:45:33.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI= 2023-10-11T14:45:33.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI=', 'Content-Length': '0'} 2023-10-11T14:45:33.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070c90f477cf1bd19-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.735 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.736 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.736 DEBUG:boto:path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.737 DEBUG:boto:auth_path=/dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.737 DEBUG:boto:Method: GET 2023-10-11T14:45:33.737 DEBUG:boto:Path: /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.738 DEBUG:boto:Data: 2023-10-11T14:45:33.738 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.739 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.740 DEBUG:boto:Params: {} 2023-10-11T14:45:33.740 DEBUG:boto:Token: None 2023-10-11T14:45:33.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj?versionId=Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0 2023-10-11T14:45:33.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI= 2023-10-11T14:45:33.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8jsCiDzFeJRkC0lYVez2oZ7aHI=', 'Content-Length': '0'} 2023-10-11T14:45:33.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:17 GMT'), ('x-amz-version-id', 'Sh2SzaqAJvBwUbmkSsTVwkW4bW8HFn0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f2dd62c57db0cc5-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.746 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.746 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.747 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.747 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.748 DEBUG:boto:Method: GET 2023-10-11T14:45:33.748 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.748 DEBUG:boto:Data: 2023-10-11T14:45:33.749 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.750 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.750 DEBUG:boto:Params: {} 2023-10-11T14:45:33.751 DEBUG:boto:Token: None 2023-10-11T14:45:33.751 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2536900baf1f76-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.802 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.803 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.804 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.804 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.805 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.805 DEBUG:boto:Method: GET 2023-10-11T14:45:33.805 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.806 DEBUG:boto:Data: 2023-10-11T14:45:33.806 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.807 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.808 DEBUG:boto:Params: {} 2023-10-11T14:45:33.808 DEBUG:boto:Token: None 2023-10-11T14:45:33.809 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007512e472025e1f48-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.813 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.813 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.814 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.814 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.815 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.815 DEBUG:boto:Data: 2023-10-11T14:45:33.816 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.816 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.817 DEBUG:boto:Params: {} 2023-10-11T14:45:33.817 DEBUG:boto:Token: None 2023-10-11T14:45:33.818 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA= 2023-10-11T14:45:33.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA=', 'Content-Length': '0'} 2023-10-11T14:45:33.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005bfff863941a5ddf-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.821 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.822 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.822 DEBUG:boto:Method: HEAD 2023-10-11T14:45:33.823 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.823 DEBUG:boto:Data: 2023-10-11T14:45:33.823 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.824 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.824 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.825 DEBUG:boto:Params: {} 2023-10-11T14:45:33.825 DEBUG:boto:Token: None 2023-10-11T14:45:33.826 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA= 2023-10-11T14:45:33.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsJXiKUx9I/HnrdBNR30tRMXGOA=', 'Content-Length': '0'} 2023-10-11T14:45:33.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001885b2c581211864-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:33.830 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.830 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.831 DEBUG:boto:Method: GET 2023-10-11T14:45:33.831 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.831 DEBUG:boto:Data: 2023-10-11T14:45:33.832 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.833 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.833 DEBUG:boto:Params: {} 2023-10-11T14:45:33.834 DEBUG:boto:Token: None 2023-10-11T14:45:33.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ= 2023-10-11T14:45:33.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ=', 'Content-Length': '0'} 2023-10-11T14:45:33.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004057ca7285fa6531-006526b50d-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.838 DEBUG:boto:path=/dghqvd-14/obj 2023-10-11T14:45:33.838 DEBUG:boto:auth_path=/dghqvd-14/obj 2023-10-11T14:45:33.839 DEBUG:boto:Method: GET 2023-10-11T14:45:33.839 DEBUG:boto:Path: /dghqvd-14/obj 2023-10-11T14:45:33.840 DEBUG:boto:Data: 2023-10-11T14:45:33.840 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.840 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.841 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.841 DEBUG:boto:Params: {} 2023-10-11T14:45:33.842 DEBUG:boto:Token: None 2023-10-11T14:45:33.842 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/obj 2023-10-11T14:45:33.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ= 2023-10-11T14:45:33.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQw9GF2zFyMNBypnTkp0ewOQQgQ=', 'Content-Length': '0'} 2023-10-11T14:45:33.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f6ee2713beda5734-006526b50d-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.846 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.846 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.847 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.847 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.847 DEBUG:boto:Method: GET 2023-10-11T14:45:33.848 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.848 DEBUG:boto:Data: 2023-10-11T14:45:33.849 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.849 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.849 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.850 DEBUG:boto:Params: {} 2023-10-11T14:45:33.850 DEBUG:boto:Token: None 2023-10-11T14:45:33.851 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001818ad6497dce6ea-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.855 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.856 DEBUG:boto:path=/dghqvd-14/ 2023-10-11T14:45:33.856 DEBUG:boto:auth_path=/dghqvd-14/ 2023-10-11T14:45:33.857 DEBUG:boto:path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.857 DEBUG:boto:auth_path=/dghqvd-14/?prefix=obj 2023-10-11T14:45:33.857 DEBUG:boto:Method: GET 2023-10-11T14:45:33.858 DEBUG:boto:Path: /dghqvd-14/?prefix=obj 2023-10-11T14:45:33.858 DEBUG:boto:Data: 2023-10-11T14:45:33.859 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.860 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.860 DEBUG:boto:Params: {} 2023-10-11T14:45:33.860 DEBUG:boto:Token: None 2023-10-11T14:45:33.861 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-14/ 2023-10-11T14:45:33.861 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E= 2023-10-11T14:45:33.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o+Cz4dwMT3jr3o0qBheiH+D6B0E=', 'Content-Length': '0'} 2023-10-11T14:45:33.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5e9f06d4bcf7ef-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.865 DEBUG:boto:b'dghqvd-14obj1000falseobj2023-10-11T14:44:58.156Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:33.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-14 zones={a2, a1} 2023-10-11T14:45:33.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-10-11T14:45:33.867 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-15 2023-10-11T14:45:33.868 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:33.868 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:33.869 DEBUG:boto:Method: PUT 2023-10-11T14:45:33.869 DEBUG:boto:Path: /dghqvd-15/ 2023-10-11T14:45:33.869 DEBUG:boto:Data: 2023-10-11T14:45:33.870 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.871 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.871 DEBUG:boto:Params: {} 2023-10-11T14:45:33.872 DEBUG:boto:Token: None 2023-10-11T14:45:33.872 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-15/ 2023-10-11T14:45:33.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PXgHrQkfwORupDmede8b27NS8LE= 2023-10-11T14:45:33.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PXgHrQkfwORupDmede8b27NS8LE=', 'Content-Length': '0'} 2023-10-11T14:45:33.889 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009fb778ee0ded6fd0-006526b50d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.889 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-16 2023-10-11T14:45:33.890 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:45:33.890 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:45:33.891 DEBUG:boto:Method: PUT 2023-10-11T14:45:33.891 DEBUG:boto:Path: /dghqvd-16/ 2023-10-11T14:45:33.891 DEBUG:boto:Data: 2023-10-11T14:45:33.892 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.892 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.893 DEBUG:boto:Params: {} 2023-10-11T14:45:33.893 DEBUG:boto:Token: None 2023-10-11T14:45:33.894 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-16/ 2023-10-11T14:45:33.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ED5AB+Xv3Q8tyjpYlLQdM3ICTdo= 2023-10-11T14:45:33.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ED5AB+Xv3Q8tyjpYlLQdM3ICTdo=', 'Content-Length': '0'} 2023-10-11T14:45:33.920 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b9c60422875da537-006526b50d-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.921 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:33.921 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:33.922 DEBUG:boto:path=/dghqvd-15/?versioning 2023-10-11T14:45:33.922 DEBUG:boto:auth_path=/dghqvd-15/?versioning 2023-10-11T14:45:33.922 DEBUG:boto:Method: PUT 2023-10-11T14:45:33.923 DEBUG:boto:Path: /dghqvd-15/?versioning 2023-10-11T14:45:33.923 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:45:33.923 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:33.924 DEBUG:boto:Port: 8000 2023-10-11T14:45:33.925 DEBUG:boto:Params: {} 2023-10-11T14:45:33.925 DEBUG:boto:Token: None 2023-10-11T14:45:33.926 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-15/?versioning 2023-10-11T14:45:33.926 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EUNXZerBKSxM3x5w81vNy45EPAs= 2023-10-11T14:45:33.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EUNXZerBKSxM3x5w81vNy45EPAs=', 'Content-Length': '228'} 2023-10-11T14:45:33.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004cb80082d5748428-006526b50d-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.933 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:45:33.933 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:45:33.934 DEBUG:boto:path=/dghqvd-16/?versioning 2023-10-11T14:45:33.934 DEBUG:boto:auth_path=/dghqvd-16/?versioning 2023-10-11T14:45:33.935 DEBUG:boto:Method: PUT 2023-10-11T14:45:33.935 DEBUG:boto:Path: /dghqvd-16/?versioning 2023-10-11T14:45:33.935 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:45:33.936 DEBUG:boto:Headers: {} 2023-10-11T14:45:33.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:33.936 DEBUG:boto:Port: 8001 2023-10-11T14:45:33.937 DEBUG:boto:Params: {} 2023-10-11T14:45:33.937 DEBUG:boto:Token: None 2023-10-11T14:45:33.938 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:45:33 GMT /dghqvd-16/?versioning 2023-10-11T14:45:33.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LLH1H0tbprT6gdwjPari2Fl/4gI= 2023-10-11T14:45:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LLH1H0tbprT6gdwjPari2Fl/4gI=', 'Content-Length': '228'} 2023-10-11T14:45:33.947 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2685c4679668aa9-006526b50d-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:33.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:33.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:33.948 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:45:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:34.008 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:34.042 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:45:34.054 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:45:34.055 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.055 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:45:34.055 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:45:34.055 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.055 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.056 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:45:34.056 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:45:34.056 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.056 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.056 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:45:34.057 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:45:34.057 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.057 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.057 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035533.942557_86.1", 2023-10-11T14:45:34.058 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:45:33.942557Z" 2023-10-11T14:45:34.058 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.058 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:45:34.059 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035533.942557_86.1'} 2023-10-11T14:45:34.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:45:34.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:34.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:34.061 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:45:34.111 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:34.112 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:34.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:34.135+0000 7f3e703e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:45:34.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:34.136+0000 7f3e703e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:45:34.150 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:45:34.150 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:45:34.150 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:45:34.151 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:45:34.151 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:45:34.151 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:45:34.151 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:34.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.152 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:45:34.152 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.152 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:45:34.153 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:34.153 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:34.153 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:45:34.153 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:34.154 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:45:34.154 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:34.154 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:45:34.154 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:34.155 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.155 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.155 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:45:34.157 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:34.158 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:34.158 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:45:34.158 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:34.158 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:34.159 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:34.159 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:45:34.159 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:34.159 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.160 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.160 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.160 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:45:34.160 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:34.161 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:34.161 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:45:34.161 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:34.161 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:34.162 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:34.162 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:45:34.162 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:34.162 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.163 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.163 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:34.163 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:45:34.163 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:34.164 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:34.164 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035533.878107_75.1", 2023-10-11T14:45:34.164 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:34.164 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:34.164 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:34.165 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:45:33.878107Z", 2023-10-11T14:45:34.165 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:34.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:34.166 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:45:34.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.166 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:45:34.166 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:45:34.167 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:45:34.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:34.167 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:45:34Z" 2023-10-11T14:45:34.167 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:45:34.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035533.878107_75.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:45:33.878107Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:45:34Z" } 2023-10-11T14:45:34.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035533.878107_75.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:45:33.878107Z', 'realm_epoch': 2}}] 2023-10-11T14:45:34.170 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035533.942557_86.1'} 2023-10-11T14:45:34.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035533.878107_75.1'} 2023-10-11T14:45:34.171 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035533.942557_86.1 target=1_1697035533.878107_75.1 2023-10-11T14:45:39.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:39.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:45:39.173 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:45:39.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:39.231 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:39.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:39.259+0000 7f2da32cb780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:45:39.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:39.260+0000 7f2da32cb780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:45:39.275 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:45:39.275 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:45:39.275 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:45:39.276 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:45:39.276 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:45:39.276 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:45:39.276 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:39.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.277 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:45:39.277 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:39.277 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:45:39.277 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:39.277 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:39.278 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:45:39.278 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:39.278 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:45:39.278 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:39.279 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:45:39.279 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:39.279 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:39.279 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.280 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:39.280 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:45:39.280 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:39.280 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:39.281 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:45:39.281 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:39.281 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:39.281 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:39.281 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:45:39.282 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:39.282 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:39.282 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.283 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:39.283 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:45:39.283 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:39.283 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:39.284 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:45:39.284 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:39.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:39.284 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:39.284 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:45:39.285 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:39.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:39.285 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.285 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:45:39.286 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:45:39.286 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:45:39.286 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:45:39.286 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035533.942557_86.1", 2023-10-11T14:45:39.286 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:45:39.287 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:45:39.287 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:45:39.287 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:45:33.942557Z", 2023-10-11T14:45:39.287 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:45:39.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:39.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:45:39.288 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:45:39.288 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.288 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:45:39.289 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:45:39.289 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:45:39.289 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:45:39.289 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:45:39Z" 2023-10-11T14:45:39.290 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:45:39.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035533.942557_86.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:45:33.942557Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:45:39Z" } 2023-10-11T14:45:39.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035533.942557_86.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:45:33.942557Z', 'realm_epoch': 2}}] 2023-10-11T14:45:39.292 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035533.942557_86.1'} 2023-10-11T14:45:39.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035533.942557_86.1'} 2023-10-11T14:45:39.293 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:45:39.293 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:39.294 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:39.294 DEBUG:boto:Method: HEAD 2023-10-11T14:45:39.295 DEBUG:boto:Path: /dghqvd-15/ 2023-10-11T14:45:39.295 DEBUG:boto:Data: 2023-10-11T14:45:39.295 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:39.296 DEBUG:boto:Port: 8000 2023-10-11T14:45:39.297 DEBUG:boto:Params: {} 2023-10-11T14:45:39.297 DEBUG:boto:Token: None 2023-10-11T14:45:39.298 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-15/ 2023-10-11T14:45:39.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iO3Iep8PGdBKUxE4galqMw08u4o= 2023-10-11T14:45:39.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iO3Iep8PGdBKUxE4galqMw08u4o=', 'Content-Length': '0'} 2023-10-11T14:45:39.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b81ce1fb7fffcf3d-006526b513-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.304 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:39.305 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:39.305 DEBUG:boto:Method: PUT 2023-10-11T14:45:39.306 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:39.306 DEBUG:boto:Data: 2023-10-11T14:45:39.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:45:39.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:39.307 DEBUG:boto:Port: 8000 2023-10-11T14:45:39.308 DEBUG:boto:Params: {} 2023-10-11T14:45:39.308 DEBUG:boto:Token: None 2023-10-11T14:45:39.309 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-15/obj 2023-10-11T14:45:39.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9/0sf+vHfl0Kd5KvS07aPsMw2g4= 2023-10-11T14:45:39.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9/0sf+vHfl0Kd5KvS07aPsMw2g4='} 2023-10-11T14:45:39.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-amz-request-id', 'tx00000bda798db15ad51ad-006526b513-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.321 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:39.322 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:39.322 DEBUG:boto:Method: HEAD 2023-10-11T14:45:39.323 DEBUG:boto:Path: /dghqvd-15/ 2023-10-11T14:45:39.323 DEBUG:boto:Data: 2023-10-11T14:45:39.323 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:39.324 DEBUG:boto:Port: 8000 2023-10-11T14:45:39.324 DEBUG:boto:Params: {} 2023-10-11T14:45:39.325 DEBUG:boto:Token: None 2023-10-11T14:45:39.325 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-15/ 2023-10-11T14:45:39.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iO3Iep8PGdBKUxE4galqMw08u4o= 2023-10-11T14:45:39.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iO3Iep8PGdBKUxE4galqMw08u4o=', 'Content-Length': '0'} 2023-10-11T14:45:39.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c68658fa2266a6a-006526b513-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.330 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:39.330 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:39.331 DEBUG:boto:Method: DELETE 2023-10-11T14:45:39.331 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:39.331 DEBUG:boto:Data: 2023-10-11T14:45:39.332 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:39.333 DEBUG:boto:Port: 8000 2023-10-11T14:45:39.333 DEBUG:boto:Params: {} 2023-10-11T14:45:39.334 DEBUG:boto:Token: None 2023-10-11T14:45:39.334 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-15/obj 2023-10-11T14:45:39.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EJ4kFpmUosBXeK87Yduu2bN769g= 2023-10-11T14:45:39.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EJ4kFpmUosBXeK87Yduu2bN769g=', 'Content-Length': '0'} 2023-10-11T14:45:39.341 DEBUG:boto:Response headers: [('x-amz-version-id', '2quM94cb.vRwEj2D59bmmqbUMacpSpq'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000a27ac3ec49c11a11-006526b513-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.342 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:45:39.343 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:45:39.344 DEBUG:boto:Method: HEAD 2023-10-11T14:45:39.344 DEBUG:boto:Path: /dghqvd-16/ 2023-10-11T14:45:39.345 DEBUG:boto:Data: 2023-10-11T14:45:39.345 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:39.346 DEBUG:boto:Port: 8001 2023-10-11T14:45:39.346 DEBUG:boto:Params: {} 2023-10-11T14:45:39.347 DEBUG:boto:Token: None 2023-10-11T14:45:39.347 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-16/ 2023-10-11T14:45:39.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3FshIMIFjH7odr4MULdQ+RJZkcQ= 2023-10-11T14:45:39.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3FshIMIFjH7odr4MULdQ+RJZkcQ=', 'Content-Length': '0'} 2023-10-11T14:45:39.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e282dbc2706c9b3-006526b513-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.351 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:45:39.352 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:45:39.352 DEBUG:boto:Method: PUT 2023-10-11T14:45:39.353 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:45:39.353 DEBUG:boto:Data: 2023-10-11T14:45:39.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:45:39.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:39.354 DEBUG:boto:Port: 8001 2023-10-11T14:45:39.355 DEBUG:boto:Params: {} 2023-10-11T14:45:39.355 DEBUG:boto:Token: None 2023-10-11T14:45:39.356 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-16/obj 2023-10-11T14:45:39.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JMVKk3fuCI+CnunlLWCe9MGturE= 2023-10-11T14:45:39.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JMVKk3fuCI+CnunlLWCe9MGturE='} 2023-10-11T14:45:39.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-amz-request-id', 'tx0000062e1121ec15685fb-006526b513-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.403 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:45:39.404 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:45:39.404 DEBUG:boto:Method: HEAD 2023-10-11T14:45:39.404 DEBUG:boto:Path: /dghqvd-16/ 2023-10-11T14:45:39.405 DEBUG:boto:Data: 2023-10-11T14:45:39.405 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:39.406 DEBUG:boto:Port: 8001 2023-10-11T14:45:39.407 DEBUG:boto:Params: {} 2023-10-11T14:45:39.407 DEBUG:boto:Token: None 2023-10-11T14:45:39.408 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-16/ 2023-10-11T14:45:39.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3FshIMIFjH7odr4MULdQ+RJZkcQ= 2023-10-11T14:45:39.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3FshIMIFjH7odr4MULdQ+RJZkcQ=', 'Content-Length': '0'} 2023-10-11T14:45:39.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b89ff622b933d9af-006526b513-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.411 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:45:39.412 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:45:39.412 DEBUG:boto:Method: DELETE 2023-10-11T14:45:39.413 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:45:39.413 DEBUG:boto:Data: 2023-10-11T14:45:39.414 DEBUG:boto:Headers: {} 2023-10-11T14:45:39.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:39.415 DEBUG:boto:Port: 8001 2023-10-11T14:45:39.415 DEBUG:boto:Params: {} 2023-10-11T14:45:39.416 DEBUG:boto:Token: None 2023-10-11T14:45:39.416 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:45:39 GMT /dghqvd-16/obj 2023-10-11T14:45:39.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zzvr10IkacDrf/JP+Hq5uyQSuZM= 2023-10-11T14:45:39.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zzvr10IkacDrf/JP+Hq5uyQSuZM=', 'Content-Length': '0'} 2023-10-11T14:45:39.424 DEBUG:boto:Response headers: [('x-amz-version-id', 'e5w-LVyLPKu.UNIyZtdqIE.oD0XsEeM'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000007f05aadc1e1403c6-006526b513-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:39.424 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-15 2023-10-11T14:45:39.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:39.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:39.426 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:45:39.483 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:39.483 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:39.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:39.514+0000 7f9fdc153780 1 waiting to reach incremental sync.. 2023-10-11T14:45:44.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:44.515+0000 7f9fdc153780 1 waiting to reach incremental sync.. 2023-10-11T14:45:49.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:49.515+0000 7f9fdc153780 1 waiting to reach incremental sync.. 2023-10-11T14:45:54.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:54.517+0000 7f9fdc153780 1 bucket sync caught up with source: 2023-10-11T14:45:54.520 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.22.12, , ] 2023-10-11T14:45:54.521 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2023-10-11T14:45:54.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:54.517+0000 7f9fdc153780 0 bucket checkpoint complete 2023-10-11T14:45:54.536 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-15 2023-10-11T14:45:54.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:54.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:54.538 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:45:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:54.600 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:54.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:54.634+0000 7f89fd385780 1 bucket sync caught up with source: 2023-10-11T14:45:54.637 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000002.52.6, , ] 2023-10-11T14:45:54.637 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000002.52.6, , ] 2023-10-11T14:45:54.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:54.634+0000 7f89fd385780 0 bucket checkpoint complete 2023-10-11T14:45:54.649 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-15 zones={a2, a1} 2023-10-11T14:45:54.650 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.650 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.654 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.655 DEBUG:boto:Path: /dghqvd-15/ 2023-10-11T14:45:54.655 DEBUG:boto:Data: 2023-10-11T14:45:54.656 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.657 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.657 DEBUG:boto:Params: {} 2023-10-11T14:45:54.658 DEBUG:boto:Token: None 2023-10-11T14:45:54.658 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/ 2023-10-11T14:45:54.659 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D+mY3LXCejvh/Txr7STyh1IUUuw= 2023-10-11T14:45:54.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D+mY3LXCejvh/Txr7STyh1IUUuw=', 'Content-Length': '0'} 2023-10-11T14:45:54.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003bee0d24238a3ad2-006526b522-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.662 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.663 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.663 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.664 DEBUG:boto:Path: /dghqvd-15/ 2023-10-11T14:45:54.664 DEBUG:boto:Data: 2023-10-11T14:45:54.664 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.666 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.666 DEBUG:boto:Params: {} 2023-10-11T14:45:54.667 DEBUG:boto:Token: None 2023-10-11T14:45:54.667 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/ 2023-10-11T14:45:54.667 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D+mY3LXCejvh/Txr7STyh1IUUuw= 2023-10-11T14:45:54.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D+mY3LXCejvh/Txr7STyh1IUUuw=', 'Content-Length': '0'} 2023-10-11T14:45:54.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d386d32e230cfec1-006526b522-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:45:54.671 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.672 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.672 DEBUG:boto:path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.672 DEBUG:boto:auth_path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.673 DEBUG:boto:Method: GET 2023-10-11T14:45:54.673 DEBUG:boto:Path: /dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.674 DEBUG:boto:Data: 2023-10-11T14:45:54.674 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.675 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.675 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.675 DEBUG:boto:Params: {} 2023-10-11T14:45:54.676 DEBUG:boto:Token: None 2023-10-11T14:45:54.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/?versions 2023-10-11T14:45:54.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI= 2023-10-11T14:45:54.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI=', 'Content-Length': '0'} 2023-10-11T14:45:54.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f65768646fb3695e-006526b522-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.680 DEBUG:boto:b'dghqvd-15999falseobj2quM94cb.vRwEj2D59bmmqbUMacpSpqtrue2023-10-11T14:45:39.335Zrgw-multisite-test-userMultisite Test UserNormalobjHW5hBmBm5ow7OKNwTaIcJXhi5F7BoNUfalse2023-10-11T14:45:39.310Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:54.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:54.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:54.682 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:45:54.683 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.683 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.683 DEBUG:boto:path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.684 DEBUG:boto:auth_path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.684 DEBUG:boto:Method: GET 2023-10-11T14:45:54.685 DEBUG:boto:Path: /dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.685 DEBUG:boto:Data: 2023-10-11T14:45:54.685 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.686 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.687 DEBUG:boto:Params: {} 2023-10-11T14:45:54.687 DEBUG:boto:Token: None 2023-10-11T14:45:54.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/?versions 2023-10-11T14:45:54.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI= 2023-10-11T14:45:54.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI=', 'Content-Length': '0'} 2023-10-11T14:45:54.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e81f83fd3d03a5-006526b522-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.738 DEBUG:boto:b'dghqvd-15999falseobj2quM94cb.vRwEj2D59bmmqbUMacpSpqtrue2023-10-11T14:45:39.335Zrgw-multisite-test-userMultisite Test UserNormalobjHW5hBmBm5ow7OKNwTaIcJXhi5F7BoNUfalse2023-10-11T14:45:39.310Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:54.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:54.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:45:54.740 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.741 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.741 DEBUG:boto:path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.741 DEBUG:boto:auth_path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.742 DEBUG:boto:Method: GET 2023-10-11T14:45:54.742 DEBUG:boto:Path: /dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.743 DEBUG:boto:Data: 2023-10-11T14:45:54.743 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.744 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.744 DEBUG:boto:Params: {} 2023-10-11T14:45:54.745 DEBUG:boto:Token: None 2023-10-11T14:45:54.745 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/?versions 2023-10-11T14:45:54.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI= 2023-10-11T14:45:54.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI=', 'Content-Length': '0'} 2023-10-11T14:45:54.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035be10b04b7bdef1-006526b522-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.749 DEBUG:boto:b'dghqvd-15999falseobj2quM94cb.vRwEj2D59bmmqbUMacpSpqtrue2023-10-11T14:45:39.335Zrgw-multisite-test-userMultisite Test UserNormalobjHW5hBmBm5ow7OKNwTaIcJXhi5F7BoNUfalse2023-10-11T14:45:39.310Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:54.750 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.750 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.751 DEBUG:boto:path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.751 DEBUG:boto:auth_path=/dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.751 DEBUG:boto:Method: GET 2023-10-11T14:45:54.752 DEBUG:boto:Path: /dghqvd-15/?versions&max-keys=999 2023-10-11T14:45:54.752 DEBUG:boto:Data: 2023-10-11T14:45:54.753 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.753 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.754 DEBUG:boto:Params: {} 2023-10-11T14:45:54.754 DEBUG:boto:Token: None 2023-10-11T14:45:54.755 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/?versions 2023-10-11T14:45:54.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI= 2023-10-11T14:45:54.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pNPDnH2VN+kP9d2N+UQWIo4RMI=', 'Content-Length': '0'} 2023-10-11T14:45:54.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ab4fe2b5664254e-006526b522-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.758 DEBUG:boto:b'dghqvd-15999falseobj2quM94cb.vRwEj2D59bmmqbUMacpSpqtrue2023-10-11T14:45:39.335Zrgw-multisite-test-userMultisite Test UserNormalobjHW5hBmBm5ow7OKNwTaIcJXhi5F7BoNUfalse2023-10-11T14:45:39.310Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:45:54.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:54.760 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.760 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.760 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.761 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:54.761 DEBUG:boto:Data: 2023-10-11T14:45:54.762 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.763 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.763 DEBUG:boto:Params: {} 2023-10-11T14:45:54.763 DEBUG:boto:Token: None 2023-10-11T14:45:54.764 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj 2023-10-11T14:45:54.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE= 2023-10-11T14:45:54.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE=', 'Content-Length': '0'} 2023-10-11T14:45:54.767 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000005849e11f5392e83-006526b522-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.767 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.768 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.768 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.769 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:54.769 DEBUG:boto:Data: 2023-10-11T14:45:54.769 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.770 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.771 DEBUG:boto:Params: {} 2023-10-11T14:45:54.771 DEBUG:boto:Token: None 2023-10-11T14:45:54.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj 2023-10-11T14:45:54.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE= 2023-10-11T14:45:54.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE=', 'Content-Length': '0'} 2023-10-11T14:45:54.774 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000823d679b91c6d695-006526b522-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:54.776 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.776 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.776 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.777 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.777 DEBUG:boto:Method: GET 2023-10-11T14:45:54.778 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.778 DEBUG:boto:Data: 2023-10-11T14:45:54.779 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.779 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.780 DEBUG:boto:Params: {} 2023-10-11T14:45:54.780 DEBUG:boto:Token: None 2023-10-11T14:45:54.781 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc= 2023-10-11T14:45:54.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc=', 'Content-Length': '0'} 2023-10-11T14:45:54.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e2674310c92e8ea2-006526b522-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.784 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.785 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.785 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.786 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.786 DEBUG:boto:Method: GET 2023-10-11T14:45:54.787 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.787 DEBUG:boto:Data: 2023-10-11T14:45:54.787 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.788 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.789 DEBUG:boto:Params: {} 2023-10-11T14:45:54.789 DEBUG:boto:Token: None 2023-10-11T14:45:54.790 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc= 2023-10-11T14:45:54.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc=', 'Content-Length': '0'} 2023-10-11T14:45:54.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000000bc64d99e10b5c6-006526b522-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.793 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.793 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.794 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.794 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.795 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.795 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.796 DEBUG:boto:Data: 2023-10-11T14:45:54.796 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.797 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.797 DEBUG:boto:Params: {} 2023-10-11T14:45:54.798 DEBUG:boto:Token: None 2023-10-11T14:45:54.798 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BpfJvePhs4Z/rt/5SVifxQA6j+o= 2023-10-11T14:45:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BpfJvePhs4Z/rt/5SVifxQA6j+o=', 'Content-Length': '0'} 2023-10-11T14:45:54.801 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069f7162291c1c268-006526b522-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.801 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.802 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.802 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.802 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.803 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.803 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.804 DEBUG:boto:Data: 2023-10-11T14:45:54.804 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.804 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.805 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.805 DEBUG:boto:Params: {} 2023-10-11T14:45:54.806 DEBUG:boto:Token: None 2023-10-11T14:45:54.806 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.807 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BpfJvePhs4Z/rt/5SVifxQA6j+o= 2023-10-11T14:45:54.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BpfJvePhs4Z/rt/5SVifxQA6j+o=', 'Content-Length': '0'} 2023-10-11T14:45:54.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ffa7e94ed3145b5-006526b522-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:45:54.810 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.811 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.811 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.811 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.812 DEBUG:boto:Method: GET 2023-10-11T14:45:54.812 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.813 DEBUG:boto:Data: 2023-10-11T14:45:54.813 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.814 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.815 DEBUG:boto:Params: {} 2023-10-11T14:45:54.815 DEBUG:boto:Token: None 2023-10-11T14:45:54.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc= 2023-10-11T14:45:54.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc=', 'Content-Length': '0'} 2023-10-11T14:45:54.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005cfaff3d512ba63f-006526b522-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.819 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.820 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.820 DEBUG:boto:path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.820 DEBUG:boto:auth_path=/dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.821 DEBUG:boto:Method: GET 2023-10-11T14:45:54.821 DEBUG:boto:Path: /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.822 DEBUG:boto:Data: 2023-10-11T14:45:54.822 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.823 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.823 DEBUG:boto:Params: {} 2023-10-11T14:45:54.824 DEBUG:boto:Token: None 2023-10-11T14:45:54.824 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj?versionId=HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU 2023-10-11T14:45:54.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc= 2023-10-11T14:45:54.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uwcWuKzL6qJc81Wj3xT8UfIlNYc=', 'Content-Length': '0'} 2023-10-11T14:45:54.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'HW5hBmBm5ow7OKNwTaIcJXhi5F7BoNU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006d1812ceda9714fb-006526b522-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.828 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.828 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.829 DEBUG:boto:path=/dghqvd-15/?prefix=obj 2023-10-11T14:45:54.829 DEBUG:boto:auth_path=/dghqvd-15/?prefix=obj 2023-10-11T14:45:54.830 DEBUG:boto:Method: GET 2023-10-11T14:45:54.830 DEBUG:boto:Path: /dghqvd-15/?prefix=obj 2023-10-11T14:45:54.830 DEBUG:boto:Data: 2023-10-11T14:45:54.831 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.831 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.832 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.832 DEBUG:boto:Params: {} 2023-10-11T14:45:54.833 DEBUG:boto:Token: None 2023-10-11T14:45:54.833 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/ 2023-10-11T14:45:54.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qML7ihe3JG/loYf/MTIriHhnhjU= 2023-10-11T14:45:54.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qML7ihe3JG/loYf/MTIriHhnhjU=', 'Content-Length': '0'} 2023-10-11T14:45:54.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa62adf2dfb3268-006526b522-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.878 DEBUG:boto:b'dghqvd-15obj1000false' 2023-10-11T14:45:54.879 DEBUG:boto:path=/dghqvd-15/ 2023-10-11T14:45:54.880 DEBUG:boto:auth_path=/dghqvd-15/ 2023-10-11T14:45:54.880 DEBUG:boto:path=/dghqvd-15/?prefix=obj 2023-10-11T14:45:54.881 DEBUG:boto:auth_path=/dghqvd-15/?prefix=obj 2023-10-11T14:45:54.881 DEBUG:boto:Method: GET 2023-10-11T14:45:54.882 DEBUG:boto:Path: /dghqvd-15/?prefix=obj 2023-10-11T14:45:54.883 DEBUG:boto:Data: 2023-10-11T14:45:54.883 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.885 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.886 DEBUG:boto:Params: {} 2023-10-11T14:45:54.886 DEBUG:boto:Token: None 2023-10-11T14:45:54.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/ 2023-10-11T14:45:54.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qML7ihe3JG/loYf/MTIriHhnhjU= 2023-10-11T14:45:54.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qML7ihe3JG/loYf/MTIriHhnhjU=', 'Content-Length': '0'} 2023-10-11T14:45:54.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef221a9f137ed662-006526b522-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.893 DEBUG:boto:b'dghqvd-15obj1000false' 2023-10-11T14:45:54.894 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.894 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.895 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.896 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:54.896 DEBUG:boto:Data: 2023-10-11T14:45:54.897 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:45:54.898 DEBUG:boto:Port: 8001 2023-10-11T14:45:54.899 DEBUG:boto:Params: {} 2023-10-11T14:45:54.900 DEBUG:boto:Token: None 2023-10-11T14:45:54.900 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj 2023-10-11T14:45:54.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE= 2023-10-11T14:45:54.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE=', 'Content-Length': '0'} 2023-10-11T14:45:54.905 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000058dbac0a988a7c46-006526b522-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.905 DEBUG:boto:path=/dghqvd-15/obj 2023-10-11T14:45:54.906 DEBUG:boto:auth_path=/dghqvd-15/obj 2023-10-11T14:45:54.906 DEBUG:boto:Method: HEAD 2023-10-11T14:45:54.907 DEBUG:boto:Path: /dghqvd-15/obj 2023-10-11T14:45:54.907 DEBUG:boto:Data: 2023-10-11T14:45:54.908 DEBUG:boto:Headers: {} 2023-10-11T14:45:54.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:45:54.908 DEBUG:boto:Port: 8000 2023-10-11T14:45:54.909 DEBUG:boto:Params: {} 2023-10-11T14:45:54.909 DEBUG:boto:Token: None 2023-10-11T14:45:54.910 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:45:54 GMT /dghqvd-15/obj 2023-10-11T14:45:54.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE= 2023-10-11T14:45:54.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:45:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9m9h+ONasEI8m9RfPbaRyZlA3yE=', 'Content-Length': '0'} 2023-10-11T14:45:54.914 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000030ac6c1ecd2bd32f-006526b522-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:45:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:45:54.915 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-15 zones={a2, a1} 2023-10-11T14:45:54.915 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-16 2023-10-11T14:45:54.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:54.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:45:54.917 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:45:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:45:54.978 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:45:55.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:45:55.007+0000 7f56138f1780 1 waiting to reach incremental sync.. 2023-10-11T14:46:00.010 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:00.007+0000 7f56138f1780 1 waiting to reach incremental sync.. 2023-10-11T14:46:05.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:05.007+0000 7f56138f1780 1 waiting to reach incremental sync.. 2023-10-11T14:46:10.011 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:10.008+0000 7f56138f1780 1 waiting to reach incremental sync.. 2023-10-11T14:46:15.013 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.010+0000 7f56138f1780 1 bucket sync caught up with source: 2023-10-11T14:46:15.014 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000002.30.6, , ] 2023-10-11T14:46:15.014 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000002.30.6, , ] 2023-10-11T14:46:15.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.010+0000 7f56138f1780 0 bucket checkpoint complete 2023-10-11T14:46:15.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-16 2023-10-11T14:46:15.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:15.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:15.031 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:46:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:15.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.125+0000 7ff917328780 1 bucket sync caught up with source: 2023-10-11T14:46:15.129 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.24.12, , ] 2023-10-11T14:46:15.129 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.24.12, , ] 2023-10-11T14:46:15.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.125+0000 7ff917328780 0 bucket checkpoint complete 2023-10-11T14:46:15.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-16 zones={a2, a1} 2023-10-11T14:46:15.142 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.142 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.143 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.143 DEBUG:boto:Path: /dghqvd-16/ 2023-10-11T14:46:15.143 DEBUG:boto:Data: 2023-10-11T14:46:15.144 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.145 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.145 DEBUG:boto:Params: {} 2023-10-11T14:46:15.146 DEBUG:boto:Token: None 2023-10-11T14:46:15.146 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/ 2023-10-11T14:46:15.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TQ2D6DpzwKZkSEIXECCpf+gdEb4= 2023-10-11T14:46:15.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TQ2D6DpzwKZkSEIXECCpf+gdEb4=', 'Content-Length': '0'} 2023-10-11T14:46:15.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fde2721c43f0e59-006526b537-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.150 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.150 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.151 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.151 DEBUG:boto:Path: /dghqvd-16/ 2023-10-11T14:46:15.152 DEBUG:boto:Data: 2023-10-11T14:46:15.152 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.153 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.153 DEBUG:boto:Params: {} 2023-10-11T14:46:15.154 DEBUG:boto:Token: None 2023-10-11T14:46:15.154 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/ 2023-10-11T14:46:15.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TQ2D6DpzwKZkSEIXECCpf+gdEb4= 2023-10-11T14:46:15.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TQ2D6DpzwKZkSEIXECCpf+gdEb4=', 'Content-Length': '0'} 2023-10-11T14:46:15.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097b4fcaff7a18207-006526b537-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:46:15.159 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.160 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.160 DEBUG:boto:path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.160 DEBUG:boto:auth_path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.161 DEBUG:boto:Method: GET 2023-10-11T14:46:15.161 DEBUG:boto:Path: /dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.162 DEBUG:boto:Data: 2023-10-11T14:46:15.162 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.163 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.163 DEBUG:boto:Params: {} 2023-10-11T14:46:15.164 DEBUG:boto:Token: None 2023-10-11T14:46:15.164 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/?versions 2023-10-11T14:46:15.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE= 2023-10-11T14:46:15.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE=', 'Content-Length': '0'} 2023-10-11T14:46:15.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ecde44b43590ec2-006526b537-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.168 DEBUG:boto:b'dghqvd-16999falseobje5w-LVyLPKu.UNIyZtdqIE.oD0XsEeMtrue2023-10-11T14:45:39.418Zrgw-multisite-test-userMultisite Test UserNormalobjEtqjWoQ69dv-ryO2uXypaBeqZ0iy-kWfalse2023-10-11T14:45:39.357Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:15.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:15.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:15.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:46:15.170 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.170 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.171 DEBUG:boto:path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.171 DEBUG:boto:auth_path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.172 DEBUG:boto:Method: GET 2023-10-11T14:46:15.172 DEBUG:boto:Path: /dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.172 DEBUG:boto:Data: 2023-10-11T14:46:15.173 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.174 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.174 DEBUG:boto:Params: {} 2023-10-11T14:46:15.175 DEBUG:boto:Token: None 2023-10-11T14:46:15.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/?versions 2023-10-11T14:46:15.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE= 2023-10-11T14:46:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE=', 'Content-Length': '0'} 2023-10-11T14:46:15.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc9be53e4b620e38-006526b537-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.222 DEBUG:boto:b'dghqvd-16999falseobje5w-LVyLPKu.UNIyZtdqIE.oD0XsEeMtrue2023-10-11T14:45:39.418Zrgw-multisite-test-userMultisite Test UserNormalobjEtqjWoQ69dv-ryO2uXypaBeqZ0iy-kWfalse2023-10-11T14:45:39.357Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:15.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:15.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:15.224 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.224 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.225 DEBUG:boto:path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.225 DEBUG:boto:auth_path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.226 DEBUG:boto:Method: GET 2023-10-11T14:46:15.226 DEBUG:boto:Path: /dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.227 DEBUG:boto:Data: 2023-10-11T14:46:15.227 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.228 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.228 DEBUG:boto:Params: {} 2023-10-11T14:46:15.229 DEBUG:boto:Token: None 2023-10-11T14:46:15.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/?versions 2023-10-11T14:46:15.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE= 2023-10-11T14:46:15.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE=', 'Content-Length': '0'} 2023-10-11T14:46:15.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053079690623ed308-006526b537-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.234 DEBUG:boto:b'dghqvd-16999falseobje5w-LVyLPKu.UNIyZtdqIE.oD0XsEeMtrue2023-10-11T14:45:39.418Zrgw-multisite-test-userMultisite Test UserNormalobjEtqjWoQ69dv-ryO2uXypaBeqZ0iy-kWfalse2023-10-11T14:45:39.357Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:15.235 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.235 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.236 DEBUG:boto:path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.236 DEBUG:boto:auth_path=/dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.237 DEBUG:boto:Method: GET 2023-10-11T14:46:15.237 DEBUG:boto:Path: /dghqvd-16/?versions&max-keys=999 2023-10-11T14:46:15.237 DEBUG:boto:Data: 2023-10-11T14:46:15.238 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.239 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.240 DEBUG:boto:Params: {} 2023-10-11T14:46:15.242 DEBUG:boto:Token: None 2023-10-11T14:46:15.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/?versions 2023-10-11T14:46:15.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE= 2023-10-11T14:46:15.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y750KYhaFuHcstb8BHMj3f7eVJE=', 'Content-Length': '0'} 2023-10-11T14:46:15.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a173c8ad6b2b3bcd-006526b537-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.246 DEBUG:boto:b'dghqvd-16999falseobje5w-LVyLPKu.UNIyZtdqIE.oD0XsEeMtrue2023-10-11T14:45:39.418Zrgw-multisite-test-userMultisite Test UserNormalobjEtqjWoQ69dv-ryO2uXypaBeqZ0iy-kWfalse2023-10-11T14:45:39.357Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:15.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:46:15.247 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.248 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.248 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.249 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:46:15.249 DEBUG:boto:Data: 2023-10-11T14:46:15.250 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.251 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.251 DEBUG:boto:Params: {} 2023-10-11T14:46:15.252 DEBUG:boto:Token: None 2023-10-11T14:46:15.253 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj 2023-10-11T14:46:15.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ= 2023-10-11T14:46:15.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.256 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a9e3b85f232ea3e9-006526b537-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.256 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.257 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.257 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.258 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:46:15.258 DEBUG:boto:Data: 2023-10-11T14:46:15.259 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.259 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.260 DEBUG:boto:Params: {} 2023-10-11T14:46:15.260 DEBUG:boto:Token: None 2023-10-11T14:46:15.261 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj 2023-10-11T14:46:15.262 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ= 2023-10-11T14:46:15.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.264 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000039c9e2cb836a6fe5-006526b537-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:46:15.265 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.266 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.266 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.266 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.267 DEBUG:boto:Method: GET 2023-10-11T14:46:15.267 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.268 DEBUG:boto:Data: 2023-10-11T14:46:15.268 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.269 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.269 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.269 DEBUG:boto:Params: {} 2023-10-11T14:46:15.270 DEBUG:boto:Token: None 2023-10-11T14:46:15.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ= 2023-10-11T14:46:15.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c069091bde090a4c-006526b537-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.274 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.275 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.275 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.276 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.276 DEBUG:boto:Method: GET 2023-10-11T14:46:15.276 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.277 DEBUG:boto:Data: 2023-10-11T14:46:15.277 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.278 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.279 DEBUG:boto:Params: {} 2023-10-11T14:46:15.279 DEBUG:boto:Token: None 2023-10-11T14:46:15.280 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ= 2023-10-11T14:46:15.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a61629b859c3eebc-006526b537-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.284 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.285 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.285 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.286 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.287 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.288 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.288 DEBUG:boto:Data: 2023-10-11T14:46:15.289 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.290 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.291 DEBUG:boto:Params: {} 2023-10-11T14:46:15.292 DEBUG:boto:Token: None 2023-10-11T14:46:15.294 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.295 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RJaYoTylnICErJQMt0ItQJbnUjA= 2023-10-11T14:46:15.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RJaYoTylnICErJQMt0ItQJbnUjA=', 'Content-Length': '0'} 2023-10-11T14:46:15.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3fcc1ab7e2e1798-006526b537-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.299 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.299 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.300 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.301 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.302 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.302 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.303 DEBUG:boto:Data: 2023-10-11T14:46:15.303 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.304 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.304 DEBUG:boto:Params: {} 2023-10-11T14:46:15.305 DEBUG:boto:Token: None 2023-10-11T14:46:15.306 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RJaYoTylnICErJQMt0ItQJbnUjA= 2023-10-11T14:46:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RJaYoTylnICErJQMt0ItQJbnUjA=', 'Content-Length': '0'} 2023-10-11T14:46:15.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c185f1887b5401f3-006526b537-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:46:15.310 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.310 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.311 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.311 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.311 DEBUG:boto:Method: GET 2023-10-11T14:46:15.312 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.313 DEBUG:boto:Data: 2023-10-11T14:46:15.313 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.315 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.315 DEBUG:boto:Params: {} 2023-10-11T14:46:15.316 DEBUG:boto:Token: None 2023-10-11T14:46:15.317 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ= 2023-10-11T14:46:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb18446100043423-006526b537-110f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.320 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.321 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.321 DEBUG:boto:path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.322 DEBUG:boto:auth_path=/dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.322 DEBUG:boto:Method: GET 2023-10-11T14:46:15.322 DEBUG:boto:Path: /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.323 DEBUG:boto:Data: 2023-10-11T14:46:15.323 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.324 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.324 DEBUG:boto:Params: {} 2023-10-11T14:46:15.325 DEBUG:boto:Token: None 2023-10-11T14:46:15.325 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj?versionId=EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW 2023-10-11T14:46:15.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ= 2023-10-11T14:46:15.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63NO/qaNnjxRIXVQvMXpyP2gUgQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:45:39 GMT'), ('x-amz-version-id', 'EtqjWoQ69dv-ryO2uXypaBeqZ0iy-kW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcf22e3e8baf2f28-006526b537-1113-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.329 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.330 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.331 DEBUG:boto:path=/dghqvd-16/?prefix=obj 2023-10-11T14:46:15.331 DEBUG:boto:auth_path=/dghqvd-16/?prefix=obj 2023-10-11T14:46:15.331 DEBUG:boto:Method: GET 2023-10-11T14:46:15.332 DEBUG:boto:Path: /dghqvd-16/?prefix=obj 2023-10-11T14:46:15.332 DEBUG:boto:Data: 2023-10-11T14:46:15.333 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.334 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.334 DEBUG:boto:Params: {} 2023-10-11T14:46:15.335 DEBUG:boto:Token: None 2023-10-11T14:46:15.335 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/ 2023-10-11T14:46:15.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xyrLMCQSyA1LCx2Odcl3fqY7a/I= 2023-10-11T14:46:15.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xyrLMCQSyA1LCx2Odcl3fqY7a/I=', 'Content-Length': '0'} 2023-10-11T14:46:15.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4238bb822daf91-006526b537-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.340 DEBUG:boto:b'dghqvd-16obj1000false' 2023-10-11T14:46:15.340 DEBUG:boto:path=/dghqvd-16/ 2023-10-11T14:46:15.341 DEBUG:boto:auth_path=/dghqvd-16/ 2023-10-11T14:46:15.341 DEBUG:boto:path=/dghqvd-16/?prefix=obj 2023-10-11T14:46:15.341 DEBUG:boto:auth_path=/dghqvd-16/?prefix=obj 2023-10-11T14:46:15.342 DEBUG:boto:Method: GET 2023-10-11T14:46:15.342 DEBUG:boto:Path: /dghqvd-16/?prefix=obj 2023-10-11T14:46:15.343 DEBUG:boto:Data: 2023-10-11T14:46:15.343 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.344 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.344 DEBUG:boto:Params: {} 2023-10-11T14:46:15.345 DEBUG:boto:Token: None 2023-10-11T14:46:15.345 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/ 2023-10-11T14:46:15.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xyrLMCQSyA1LCx2Odcl3fqY7a/I= 2023-10-11T14:46:15.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xyrLMCQSyA1LCx2Odcl3fqY7a/I=', 'Content-Length': '0'} 2023-10-11T14:46:15.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000edca45f33e4715a-006526b537-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.349 DEBUG:boto:b'dghqvd-16obj1000false' 2023-10-11T14:46:15.350 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.350 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.351 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.351 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:46:15.352 DEBUG:boto:Data: 2023-10-11T14:46:15.352 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.353 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.353 DEBUG:boto:Params: {} 2023-10-11T14:46:15.354 DEBUG:boto:Token: None 2023-10-11T14:46:15.354 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj 2023-10-11T14:46:15.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ= 2023-10-11T14:46:15.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.357 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000883eb5aaf9195bb9-006526b537-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.358 DEBUG:boto:path=/dghqvd-16/obj 2023-10-11T14:46:15.358 DEBUG:boto:auth_path=/dghqvd-16/obj 2023-10-11T14:46:15.359 DEBUG:boto:Method: HEAD 2023-10-11T14:46:15.359 DEBUG:boto:Path: /dghqvd-16/obj 2023-10-11T14:46:15.359 DEBUG:boto:Data: 2023-10-11T14:46:15.360 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.360 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.361 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.361 DEBUG:boto:Params: {} 2023-10-11T14:46:15.361 DEBUG:boto:Token: None 2023-10-11T14:46:15.362 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-16/obj 2023-10-11T14:46:15.362 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ= 2023-10-11T14:46:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:toU3nkREsTY1FyajMRSBJjxJGJQ=', 'Content-Length': '0'} 2023-10-11T14:46:15.365 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007da64cbceb58aec2-006526b537-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-16 zones={a2, a1} 2023-10-11T14:46:15.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-10-11T14:46:15.367 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-17 2023-10-11T14:46:15.367 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:15.368 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:15.368 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.368 DEBUG:boto:Path: /dghqvd-17/ 2023-10-11T14:46:15.369 DEBUG:boto:Data: 2023-10-11T14:46:15.369 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.370 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.370 DEBUG:boto:Params: {} 2023-10-11T14:46:15.371 DEBUG:boto:Token: None 2023-10-11T14:46:15.371 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-17/ 2023-10-11T14:46:15.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnkUA7GVybB1BDc/+GwpEQQ+kPo= 2023-10-11T14:46:15.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnkUA7GVybB1BDc/+GwpEQQ+kPo=', 'Content-Length': '0'} 2023-10-11T14:46:15.385 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006fb8493218bb794a-006526b537-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.386 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-18 2023-10-11T14:46:15.386 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:46:15.387 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:46:15.387 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.387 DEBUG:boto:Path: /dghqvd-18/ 2023-10-11T14:46:15.388 DEBUG:boto:Data: 2023-10-11T14:46:15.388 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.389 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.389 DEBUG:boto:Params: {} 2023-10-11T14:46:15.390 DEBUG:boto:Token: None 2023-10-11T14:46:15.390 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-18/ 2023-10-11T14:46:15.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Hmzik2TTo8qpBKZRJg/uqFpKIc= 2023-10-11T14:46:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Hmzik2TTo8qpBKZRJg/uqFpKIc=', 'Content-Length': '0'} 2023-10-11T14:46:15.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000026f35e734baed17d-006526b537-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.412 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:15.412 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:15.413 DEBUG:boto:path=/dghqvd-17/?versioning 2023-10-11T14:46:15.414 DEBUG:boto:auth_path=/dghqvd-17/?versioning 2023-10-11T14:46:15.414 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.414 DEBUG:boto:Path: /dghqvd-17/?versioning 2023-10-11T14:46:15.415 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:46:15.415 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.416 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.416 DEBUG:boto:Params: {} 2023-10-11T14:46:15.417 DEBUG:boto:Token: None 2023-10-11T14:46:15.417 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-17/?versioning 2023-10-11T14:46:15.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7UZFpHby/yQWyoiTE49RCmXJEx0= 2023-10-11T14:46:15.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7UZFpHby/yQWyoiTE49RCmXJEx0=', 'Content-Length': '228'} 2023-10-11T14:46:15.423 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b99992253c3abe7-006526b537-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.425 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:15.426 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:15.426 DEBUG:boto:path=/dghqvd-17/?versioning 2023-10-11T14:46:15.427 DEBUG:boto:auth_path=/dghqvd-17/?versioning 2023-10-11T14:46:15.428 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.428 DEBUG:boto:Path: /dghqvd-17/?versioning 2023-10-11T14:46:15.429 DEBUG:boto:Data: Suspended Disabled 2023-10-11T14:46:15.430 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:15.431 DEBUG:boto:Port: 8000 2023-10-11T14:46:15.432 DEBUG:boto:Params: {} 2023-10-11T14:46:15.433 DEBUG:boto:Token: None 2023-10-11T14:46:15.433 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-17/?versioning 2023-10-11T14:46:15.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7UZFpHby/yQWyoiTE49RCmXJEx0= 2023-10-11T14:46:15.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7UZFpHby/yQWyoiTE49RCmXJEx0=', 'Content-Length': '230'} 2023-10-11T14:46:15.440 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b11b8552473c91a-006526b537-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.441 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:46:15.441 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:46:15.442 DEBUG:boto:path=/dghqvd-18/?versioning 2023-10-11T14:46:15.442 DEBUG:boto:auth_path=/dghqvd-18/?versioning 2023-10-11T14:46:15.443 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.443 DEBUG:boto:Path: /dghqvd-18/?versioning 2023-10-11T14:46:15.443 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:46:15.444 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.445 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.446 DEBUG:boto:Params: {} 2023-10-11T14:46:15.447 DEBUG:boto:Token: None 2023-10-11T14:46:15.447 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-18/?versioning 2023-10-11T14:46:15.447 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CjX7zS4cbSwuYDsHkQmBQK+v+Lg= 2023-10-11T14:46:15.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CjX7zS4cbSwuYDsHkQmBQK+v+Lg=', 'Content-Length': '228'} 2023-10-11T14:46:15.455 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000045308fa9598e4f6a-006526b537-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.455 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:46:15.456 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:46:15.456 DEBUG:boto:path=/dghqvd-18/?versioning 2023-10-11T14:46:15.457 DEBUG:boto:auth_path=/dghqvd-18/?versioning 2023-10-11T14:46:15.457 DEBUG:boto:Method: PUT 2023-10-11T14:46:15.458 DEBUG:boto:Path: /dghqvd-18/?versioning 2023-10-11T14:46:15.458 DEBUG:boto:Data: Suspended Disabled 2023-10-11T14:46:15.458 DEBUG:boto:Headers: {} 2023-10-11T14:46:15.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:15.459 DEBUG:boto:Port: 8001 2023-10-11T14:46:15.460 DEBUG:boto:Params: {} 2023-10-11T14:46:15.460 DEBUG:boto:Token: None 2023-10-11T14:46:15.460 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:46:15 GMT /dghqvd-18/?versioning 2023-10-11T14:46:15.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CjX7zS4cbSwuYDsHkQmBQK+v+Lg= 2023-10-11T14:46:15.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CjX7zS4cbSwuYDsHkQmBQK+v+Lg=', 'Content-Length': '230'} 2023-10-11T14:46:15.468 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000712e817403b183de-006526b537-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:15.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:15.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:15.469 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:46:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:15.518 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:15.546 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:46:15.557 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:46:15.558 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.558 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:46:15.558 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:46:15.558 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.559 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.559 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:46:15.559 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:46:15.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.560 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.560 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:46:15.560 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:46:15.560 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.560 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.561 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035575.464339_102.1", 2023-10-11T14:46:15.561 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:46:15.464339Z" 2023-10-11T14:46:15.561 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.561 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:46:15.562 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035575.464339_102.1'} 2023-10-11T14:46:15.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:46:15.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:15.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:15.564 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:46:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:15.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:15.654 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.648+0000 7f33faf4c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:46:15.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:15.649+0000 7f33faf4c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:46:15.663 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:46:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:46:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:46:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:46:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:46:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:46:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:46:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:46:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:46:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:46:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:46:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:46:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:46:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:15.670 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:15.670 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:46:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:46:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:46:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:46:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:46:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035575.462100_101.1", 2023-10-11T14:46:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:46:15.462100Z", 2023-10-11T14:46:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:46:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:46:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:46:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:46:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:46:15Z" 2023-10-11T14:46:15.678 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:46:15.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035575.462100_101.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:46:15.462100Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:46:15Z" } 2023-10-11T14:46:15.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035575.462100_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:46:15.462100Z', 'realm_epoch': 2}}] 2023-10-11T14:46:15.680 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035575.464339_102.1'} 2023-10-11T14:46:15.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035575.462100_101.1'} 2023-10-11T14:46:15.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035575.464339_102.1 target=1_1697035575.462100_101.1 2023-10-11T14:46:20.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:20.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:46:20.683 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:46:20.743 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:20.744 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:20.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:20.771+0000 7f21c9841780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:46:20.775 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:20.772+0000 7f21c9841780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:46:20.787 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:46:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:46:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:46:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:46:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:46:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:46:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:46:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:46:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:46:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:46:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:46:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:46:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:46:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:46:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:46:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:46:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:46:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:46:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:46:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:46:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035575.464339_102.1", 2023-10-11T14:46:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:46:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:46:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:46:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:46:15.464339Z", 2023-10-11T14:46:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:46:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:46:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:46:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:46:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:46:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:46:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:46:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:46:20Z" 2023-10-11T14:46:20.801 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:46:20.802 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035575.464339_102.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:46:15.464339Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:46:20Z" } 2023-10-11T14:46:20.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035575.464339_102.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:46:15.464339Z', 'realm_epoch': 2}}] 2023-10-11T14:46:20.803 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035575.464339_102.1'} 2023-10-11T14:46:20.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035575.464339_102.1'} 2023-10-11T14:46:20.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:46:20.804 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:20.805 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:20.805 DEBUG:boto:Method: HEAD 2023-10-11T14:46:20.805 DEBUG:boto:Path: /dghqvd-17/ 2023-10-11T14:46:20.806 DEBUG:boto:Data: 2023-10-11T14:46:20.806 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:20.807 DEBUG:boto:Port: 8000 2023-10-11T14:46:20.808 DEBUG:boto:Params: {} 2023-10-11T14:46:20.808 DEBUG:boto:Token: None 2023-10-11T14:46:20.809 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-17/ 2023-10-11T14:46:20.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iBgFKRWfHumx3WtXpwxfZ9Wa9SE= 2023-10-11T14:46:20.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iBgFKRWfHumx3WtXpwxfZ9Wa9SE=', 'Content-Length': '0'} 2023-10-11T14:46:20.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010714e6f3be3cb99-006526b53c-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.812 DEBUG:boto:path=/dghqvd-17/obj 2023-10-11T14:46:20.813 DEBUG:boto:auth_path=/dghqvd-17/obj 2023-10-11T14:46:20.813 DEBUG:boto:Method: PUT 2023-10-11T14:46:20.814 DEBUG:boto:Path: /dghqvd-17/obj 2023-10-11T14:46:20.814 DEBUG:boto:Data: 2023-10-11T14:46:20.814 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:46:20.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:20.815 DEBUG:boto:Port: 8000 2023-10-11T14:46:20.816 DEBUG:boto:Params: {} 2023-10-11T14:46:20.816 DEBUG:boto:Token: None 2023-10-11T14:46:20.817 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-17/obj 2023-10-11T14:46:20.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZnBo6tqvQVV1RV2o+wczClym7pU= 2023-10-11T14:46:20.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZnBo6tqvQVV1RV2o+wczClym7pU='} 2023-10-11T14:46:20.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035b551ad4f361f0a-006526b53c-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.824 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:20.824 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:20.824 DEBUG:boto:Method: HEAD 2023-10-11T14:46:20.825 DEBUG:boto:Path: /dghqvd-17/ 2023-10-11T14:46:20.825 DEBUG:boto:Data: 2023-10-11T14:46:20.826 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:20.827 DEBUG:boto:Port: 8000 2023-10-11T14:46:20.827 DEBUG:boto:Params: {} 2023-10-11T14:46:20.827 DEBUG:boto:Token: None 2023-10-11T14:46:20.828 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-17/ 2023-10-11T14:46:20.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iBgFKRWfHumx3WtXpwxfZ9Wa9SE= 2023-10-11T14:46:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iBgFKRWfHumx3WtXpwxfZ9Wa9SE=', 'Content-Length': '0'} 2023-10-11T14:46:20.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c95d3f4b6c5b4d5c-006526b53c-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.831 DEBUG:boto:path=/dghqvd-17/obj 2023-10-11T14:46:20.832 DEBUG:boto:auth_path=/dghqvd-17/obj 2023-10-11T14:46:20.832 DEBUG:boto:Method: DELETE 2023-10-11T14:46:20.833 DEBUG:boto:Path: /dghqvd-17/obj 2023-10-11T14:46:20.833 DEBUG:boto:Data: 2023-10-11T14:46:20.833 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.834 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:20.834 DEBUG:boto:Port: 8000 2023-10-11T14:46:20.835 DEBUG:boto:Params: {} 2023-10-11T14:46:20.835 DEBUG:boto:Token: None 2023-10-11T14:46:20.835 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-17/obj 2023-10-11T14:46:20.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XByXvUyKcLPQpI+gkCWobFotjl0= 2023-10-11T14:46:20.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XByXvUyKcLPQpI+gkCWobFotjl0=', 'Content-Length': '0'} 2023-10-11T14:46:20.844 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000aaaf2cc2fa8d75a2-006526b53c-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.845 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:46:20.845 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:46:20.846 DEBUG:boto:Method: HEAD 2023-10-11T14:46:20.846 DEBUG:boto:Path: /dghqvd-18/ 2023-10-11T14:46:20.847 DEBUG:boto:Data: 2023-10-11T14:46:20.847 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:20.848 DEBUG:boto:Port: 8001 2023-10-11T14:46:20.848 DEBUG:boto:Params: {} 2023-10-11T14:46:20.849 DEBUG:boto:Token: None 2023-10-11T14:46:20.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-18/ 2023-10-11T14:46:20.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GcKfYLoPl8REVo0JazTl7jVGdXk= 2023-10-11T14:46:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GcKfYLoPl8REVo0JazTl7jVGdXk=', 'Content-Length': '0'} 2023-10-11T14:46:20.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3200c5e6dbe79b-006526b53c-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.853 DEBUG:boto:path=/dghqvd-18/obj 2023-10-11T14:46:20.854 DEBUG:boto:auth_path=/dghqvd-18/obj 2023-10-11T14:46:20.854 DEBUG:boto:Method: PUT 2023-10-11T14:46:20.854 DEBUG:boto:Path: /dghqvd-18/obj 2023-10-11T14:46:20.855 DEBUG:boto:Data: 2023-10-11T14:46:20.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-10-11T14:46:20.856 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:20.856 DEBUG:boto:Port: 8001 2023-10-11T14:46:20.856 DEBUG:boto:Params: {} 2023-10-11T14:46:20.857 DEBUG:boto:Token: None 2023-10-11T14:46:20.857 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-18/obj 2023-10-11T14:46:20.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iKB5aXSJEm65bLur2j4niBApP6k= 2023-10-11T14:46:20.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iKB5aXSJEm65bLur2j4niBApP6k='} 2023-10-11T14:46:20.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000040f9417a5491b1b-006526b53c-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.866 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:46:20.866 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:46:20.866 DEBUG:boto:Method: HEAD 2023-10-11T14:46:20.867 DEBUG:boto:Path: /dghqvd-18/ 2023-10-11T14:46:20.867 DEBUG:boto:Data: 2023-10-11T14:46:20.868 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:20.868 DEBUG:boto:Port: 8001 2023-10-11T14:46:20.869 DEBUG:boto:Params: {} 2023-10-11T14:46:20.869 DEBUG:boto:Token: None 2023-10-11T14:46:20.870 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-18/ 2023-10-11T14:46:20.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GcKfYLoPl8REVo0JazTl7jVGdXk= 2023-10-11T14:46:20.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GcKfYLoPl8REVo0JazTl7jVGdXk=', 'Content-Length': '0'} 2023-10-11T14:46:20.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000512c47099f27bf1a-006526b53c-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.874 DEBUG:boto:path=/dghqvd-18/obj 2023-10-11T14:46:20.874 DEBUG:boto:auth_path=/dghqvd-18/obj 2023-10-11T14:46:20.875 DEBUG:boto:Method: DELETE 2023-10-11T14:46:20.875 DEBUG:boto:Path: /dghqvd-18/obj 2023-10-11T14:46:20.875 DEBUG:boto:Data: 2023-10-11T14:46:20.876 DEBUG:boto:Headers: {} 2023-10-11T14:46:20.876 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:20.877 DEBUG:boto:Port: 8001 2023-10-11T14:46:20.877 DEBUG:boto:Params: {} 2023-10-11T14:46:20.877 DEBUG:boto:Token: None 2023-10-11T14:46:20.878 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:46:20 GMT /dghqvd-18/obj 2023-10-11T14:46:20.878 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XitihawuareTNQeOSzQcpUrrYr8= 2023-10-11T14:46:20.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XitihawuareTNQeOSzQcpUrrYr8=', 'Content-Length': '0'} 2023-10-11T14:46:20.886 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000a888f98fcf4f4c0a-006526b53c-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:46:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:20.887 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-17 2023-10-11T14:46:20.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:20.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:20.889 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:46:20.945 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:20.946 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:20.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:20.974+0000 7f048bbff780 1 waiting to reach incremental sync.. 2023-10-11T14:46:25.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:25.975+0000 7f048bbff780 1 waiting to reach incremental sync.. 2023-10-11T14:46:30.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:30.976+0000 7f048bbff780 1 waiting to reach incremental sync.. 2023-10-11T14:46:35.981 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:35.977+0000 7f048bbff780 1 bucket sync caught up with source: 2023-10-11T14:46:35.981 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.12.7, , ] 2023-10-11T14:46:35.981 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.12.7, , ] 2023-10-11T14:46:35.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:35.977+0000 7f048bbff780 0 bucket checkpoint complete 2023-10-11T14:46:35.997 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-17 2023-10-11T14:46:35.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:35.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:35.998 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:46:36.061 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:36.061 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:36.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:36.094+0000 7f8352905780 1 waiting to reach incremental sync.. 2023-10-11T14:46:41.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:41.095+0000 7f8352905780 1 waiting to reach incremental sync.. 2023-10-11T14:46:46.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:46.096+0000 7f8352905780 1 waiting to reach incremental sync.. 2023-10-11T14:46:51.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:51.097+0000 7f8352905780 1 waiting to reach incremental sync.. 2023-10-11T14:46:56.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:56.098+0000 7f8352905780 1 bucket sync caught up with source: 2023-10-11T14:46:56.101 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.17.14, , ] 2023-10-11T14:46:56.101 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.17.14, , ] 2023-10-11T14:46:56.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:56.098+0000 7f8352905780 0 bucket checkpoint complete 2023-10-11T14:46:56.116 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-17 zones={a2, a1} 2023-10-11T14:46:56.117 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.118 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.118 DEBUG:boto:Method: HEAD 2023-10-11T14:46:56.118 DEBUG:boto:Path: /dghqvd-17/ 2023-10-11T14:46:56.119 DEBUG:boto:Data: 2023-10-11T14:46:56.119 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:56.120 DEBUG:boto:Port: 8001 2023-10-11T14:46:56.120 DEBUG:boto:Params: {} 2023-10-11T14:46:56.121 DEBUG:boto:Token: None 2023-10-11T14:46:56.122 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/ 2023-10-11T14:46:56.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhAI9fglB7dwFtgmM38lOu0tFzE= 2023-10-11T14:46:56.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhAI9fglB7dwFtgmM38lOu0tFzE=', 'Content-Length': '0'} 2023-10-11T14:46:56.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014029ef8a01b7381-006526b560-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.126 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.126 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.127 DEBUG:boto:Method: HEAD 2023-10-11T14:46:56.127 DEBUG:boto:Path: /dghqvd-17/ 2023-10-11T14:46:56.127 DEBUG:boto:Data: 2023-10-11T14:46:56.128 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:56.129 DEBUG:boto:Port: 8000 2023-10-11T14:46:56.129 DEBUG:boto:Params: {} 2023-10-11T14:46:56.129 DEBUG:boto:Token: None 2023-10-11T14:46:56.130 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/ 2023-10-11T14:46:56.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhAI9fglB7dwFtgmM38lOu0tFzE= 2023-10-11T14:46:56.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhAI9fglB7dwFtgmM38lOu0tFzE=', 'Content-Length': '0'} 2023-10-11T14:46:56.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cb3404965b78704-006526b560-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:46:56.135 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.135 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.135 DEBUG:boto:path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.136 DEBUG:boto:auth_path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.136 DEBUG:boto:Method: GET 2023-10-11T14:46:56.137 DEBUG:boto:Path: /dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.137 DEBUG:boto:Data: 2023-10-11T14:46:56.138 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:56.138 DEBUG:boto:Port: 8001 2023-10-11T14:46:56.139 DEBUG:boto:Params: {} 2023-10-11T14:46:56.139 DEBUG:boto:Token: None 2023-10-11T14:46:56.140 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/?versions 2023-10-11T14:46:56.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw= 2023-10-11T14:46:56.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw=', 'Content-Length': '0'} 2023-10-11T14:46:56.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060043edd6d66ed9c-006526b560-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.144 DEBUG:boto:b'dghqvd-17999falseobjnulltrue2023-10-11T14:46:20.837Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:56.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:56.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:46:56.146 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.146 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.146 DEBUG:boto:path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.147 DEBUG:boto:auth_path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.147 DEBUG:boto:Method: GET 2023-10-11T14:46:56.148 DEBUG:boto:Path: /dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.148 DEBUG:boto:Data: 2023-10-11T14:46:56.148 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:56.149 DEBUG:boto:Port: 8000 2023-10-11T14:46:56.150 DEBUG:boto:Params: {} 2023-10-11T14:46:56.150 DEBUG:boto:Token: None 2023-10-11T14:46:56.151 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/?versions 2023-10-11T14:46:56.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw= 2023-10-11T14:46:56.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw=', 'Content-Length': '0'} 2023-10-11T14:46:56.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f18aecdae9605233-006526b560-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.154 DEBUG:boto:b'dghqvd-17999falseobjnulltrue2023-10-11T14:46:20.837Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:46:56.156 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.156 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.156 DEBUG:boto:path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.157 DEBUG:boto:auth_path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.157 DEBUG:boto:Method: GET 2023-10-11T14:46:56.158 DEBUG:boto:Path: /dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.158 DEBUG:boto:Data: 2023-10-11T14:46:56.159 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:56.159 DEBUG:boto:Port: 8001 2023-10-11T14:46:56.160 DEBUG:boto:Params: {} 2023-10-11T14:46:56.160 DEBUG:boto:Token: None 2023-10-11T14:46:56.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/?versions 2023-10-11T14:46:56.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw= 2023-10-11T14:46:56.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw=', 'Content-Length': '0'} 2023-10-11T14:46:56.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc884b2afde02e20-006526b560-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.164 DEBUG:boto:b'dghqvd-17999falseobjnulltrue2023-10-11T14:46:20.837Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:56.165 DEBUG:boto:path=/dghqvd-17/ 2023-10-11T14:46:56.166 DEBUG:boto:auth_path=/dghqvd-17/ 2023-10-11T14:46:56.166 DEBUG:boto:path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.166 DEBUG:boto:auth_path=/dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.167 DEBUG:boto:Method: GET 2023-10-11T14:46:56.167 DEBUG:boto:Path: /dghqvd-17/?versions&max-keys=999 2023-10-11T14:46:56.168 DEBUG:boto:Data: 2023-10-11T14:46:56.168 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:56.169 DEBUG:boto:Port: 8000 2023-10-11T14:46:56.169 DEBUG:boto:Params: {} 2023-10-11T14:46:56.170 DEBUG:boto:Token: None 2023-10-11T14:46:56.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/?versions 2023-10-11T14:46:56.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw= 2023-10-11T14:46:56.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YX3uhwTzmxBaI9gscTjVs+cHQCw=', 'Content-Length': '0'} 2023-10-11T14:46:56.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae13c21e6e321d3-006526b560-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.174 DEBUG:boto:b'dghqvd-17999falseobjnulltrue2023-10-11T14:46:20.837Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:46:56.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:46:56.175 DEBUG:boto:path=/dghqvd-17/obj 2023-10-11T14:46:56.175 DEBUG:boto:auth_path=/dghqvd-17/obj 2023-10-11T14:46:56.176 DEBUG:boto:Method: HEAD 2023-10-11T14:46:56.176 DEBUG:boto:Path: /dghqvd-17/obj 2023-10-11T14:46:56.177 DEBUG:boto:Data: 2023-10-11T14:46:56.177 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:46:56.178 DEBUG:boto:Port: 8001 2023-10-11T14:46:56.178 DEBUG:boto:Params: {} 2023-10-11T14:46:56.179 DEBUG:boto:Token: None 2023-10-11T14:46:56.179 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/obj 2023-10-11T14:46:56.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E/60MVYN0t4hJDuxrHlcXAxqelI= 2023-10-11T14:46:56.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E/60MVYN0t4hJDuxrHlcXAxqelI=', 'Content-Length': '0'} 2023-10-11T14:46:56.182 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cb66ff9680934621-006526b560-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.182 DEBUG:boto:path=/dghqvd-17/obj 2023-10-11T14:46:56.183 DEBUG:boto:auth_path=/dghqvd-17/obj 2023-10-11T14:46:56.183 DEBUG:boto:Method: HEAD 2023-10-11T14:46:56.184 DEBUG:boto:Path: /dghqvd-17/obj 2023-10-11T14:46:56.184 DEBUG:boto:Data: 2023-10-11T14:46:56.184 DEBUG:boto:Headers: {} 2023-10-11T14:46:56.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:46:56.185 DEBUG:boto:Port: 8000 2023-10-11T14:46:56.185 DEBUG:boto:Params: {} 2023-10-11T14:46:56.186 DEBUG:boto:Token: None 2023-10-11T14:46:56.186 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:46:56 GMT /dghqvd-17/obj 2023-10-11T14:46:56.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E/60MVYN0t4hJDuxrHlcXAxqelI= 2023-10-11T14:46:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:46:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E/60MVYN0t4hJDuxrHlcXAxqelI=', 'Content-Length': '0'} 2023-10-11T14:46:56.189 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001f8eb887a7af5155-006526b560-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:46:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:46:56.189 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-17 zones={a2, a1} 2023-10-11T14:46:56.190 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-18 2023-10-11T14:46:56.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:56.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:46:56.191 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:46:56.254 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:46:56.255 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:46:56.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:46:56.286+0000 7fd7110f5780 1 waiting to reach incremental sync.. 2023-10-11T14:47:01.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:01.287+0000 7fd7110f5780 1 waiting to reach incremental sync.. 2023-10-11T14:47:06.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:06.288+0000 7fd7110f5780 1 waiting to reach incremental sync.. 2023-10-11T14:47:11.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:11.288+0000 7fd7110f5780 1 waiting to reach incremental sync.. 2023-10-11T14:47:16.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:16.290+0000 7fd7110f5780 1 bucket sync caught up with source: 2023-10-11T14:47:16.293 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.5.14, , ] 2023-10-11T14:47:16.293 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.5.14, , ] 2023-10-11T14:47:16.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:16.290+0000 7fd7110f5780 0 bucket checkpoint complete 2023-10-11T14:47:16.309 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-18 2023-10-11T14:47:16.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:47:16.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:47:16.311 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:47:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:16.377 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:16.414 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:16.410+0000 7f944e627780 1 bucket sync caught up with source: 2023-10-11T14:47:16.415 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000003.20.7, , ] 2023-10-11T14:47:16.415 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000003.20.7, , ] 2023-10-11T14:47:16.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:16.410+0000 7f944e627780 0 bucket checkpoint complete 2023-10-11T14:47:16.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-18 zones={a2, a1} 2023-10-11T14:47:16.426 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.426 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.427 DEBUG:boto:Method: HEAD 2023-10-11T14:47:16.427 DEBUG:boto:Path: /dghqvd-18/ 2023-10-11T14:47:16.428 DEBUG:boto:Data: 2023-10-11T14:47:16.428 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.430 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.430 DEBUG:boto:Params: {} 2023-10-11T14:47:16.431 DEBUG:boto:Token: None 2023-10-11T14:47:16.431 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/ 2023-10-11T14:47:16.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zslJ6xcHMBub89GawhEKROhLbxs= 2023-10-11T14:47:16.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zslJ6xcHMBub89GawhEKROhLbxs=', 'Content-Length': '0'} 2023-10-11T14:47:16.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007abe0455b7270901-006526b574-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.436 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.437 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.437 DEBUG:boto:Method: HEAD 2023-10-11T14:47:16.438 DEBUG:boto:Path: /dghqvd-18/ 2023-10-11T14:47:16.438 DEBUG:boto:Data: 2023-10-11T14:47:16.439 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.440 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.441 DEBUG:boto:Params: {} 2023-10-11T14:47:16.442 DEBUG:boto:Token: None 2023-10-11T14:47:16.443 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/ 2023-10-11T14:47:16.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zslJ6xcHMBub89GawhEKROhLbxs= 2023-10-11T14:47:16.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zslJ6xcHMBub89GawhEKROhLbxs=', 'Content-Length': '0'} 2023-10-11T14:47:16.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d91bf40c10102c6-006526b574-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.448 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:47:16.448 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.449 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.450 DEBUG:boto:path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.450 DEBUG:boto:auth_path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.451 DEBUG:boto:Method: GET 2023-10-11T14:47:16.452 DEBUG:boto:Path: /dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.452 DEBUG:boto:Data: 2023-10-11T14:47:16.453 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.453 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.454 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.455 DEBUG:boto:Params: {} 2023-10-11T14:47:16.455 DEBUG:boto:Token: None 2023-10-11T14:47:16.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/?versions 2023-10-11T14:47:16.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE= 2023-10-11T14:47:16.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE=', 'Content-Length': '0'} 2023-10-11T14:47:16.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000076a89b1a941eb10-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.506 DEBUG:boto:b'dghqvd-18999falseobjnulltrue2023-10-11T14:46:20.879Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:47:16.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:47:16.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:47:16.508 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.508 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.509 DEBUG:boto:path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.509 DEBUG:boto:auth_path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.510 DEBUG:boto:Method: GET 2023-10-11T14:47:16.510 DEBUG:boto:Path: /dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.510 DEBUG:boto:Data: 2023-10-11T14:47:16.511 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.512 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.512 DEBUG:boto:Params: {} 2023-10-11T14:47:16.513 DEBUG:boto:Token: None 2023-10-11T14:47:16.513 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/?versions 2023-10-11T14:47:16.514 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE= 2023-10-11T14:47:16.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE=', 'Content-Length': '0'} 2023-10-11T14:47:16.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae764d8077dde8c9-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.517 DEBUG:boto:b'dghqvd-18999falseobjnulltrue2023-10-11T14:46:20.879Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:47:16.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-10-11T14:47:16.518 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.519 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.521 DEBUG:boto:path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.522 DEBUG:boto:auth_path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.522 DEBUG:boto:Method: GET 2023-10-11T14:47:16.523 DEBUG:boto:Path: /dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.523 DEBUG:boto:Data: 2023-10-11T14:47:16.523 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.527 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.527 DEBUG:boto:Params: {} 2023-10-11T14:47:16.528 DEBUG:boto:Token: None 2023-10-11T14:47:16.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/?versions 2023-10-11T14:47:16.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE= 2023-10-11T14:47:16.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE=', 'Content-Length': '0'} 2023-10-11T14:47:16.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8df1243771394b-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.533 DEBUG:boto:b'dghqvd-18999falseobjnulltrue2023-10-11T14:46:20.879Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:47:16.533 DEBUG:boto:path=/dghqvd-18/ 2023-10-11T14:47:16.534 DEBUG:boto:auth_path=/dghqvd-18/ 2023-10-11T14:47:16.534 DEBUG:boto:path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.535 DEBUG:boto:auth_path=/dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.535 DEBUG:boto:Method: GET 2023-10-11T14:47:16.536 DEBUG:boto:Path: /dghqvd-18/?versions&max-keys=999 2023-10-11T14:47:16.536 DEBUG:boto:Data: 2023-10-11T14:47:16.537 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.537 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.538 DEBUG:boto:Params: {} 2023-10-11T14:47:16.538 DEBUG:boto:Token: None 2023-10-11T14:47:16.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/?versions 2023-10-11T14:47:16.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE= 2023-10-11T14:47:16.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nMMGNosfR5AFQzdA9NiRuyewlrE=', 'Content-Length': '0'} 2023-10-11T14:47:16.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065dcb36ce19e1396-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.545 DEBUG:boto:b'dghqvd-18999falseobjnulltrue2023-10-11T14:46:20.879Zrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:47:16.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-10-11T14:47:16.546 DEBUG:boto:path=/dghqvd-18/obj 2023-10-11T14:47:16.546 DEBUG:boto:auth_path=/dghqvd-18/obj 2023-10-11T14:47:16.547 DEBUG:boto:Method: HEAD 2023-10-11T14:47:16.547 DEBUG:boto:Path: /dghqvd-18/obj 2023-10-11T14:47:16.547 DEBUG:boto:Data: 2023-10-11T14:47:16.548 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.549 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.549 DEBUG:boto:Params: {} 2023-10-11T14:47:16.550 DEBUG:boto:Token: None 2023-10-11T14:47:16.550 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/obj 2023-10-11T14:47:16.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XKlVlJASfuNpRVq+duO/rCmteEo= 2023-10-11T14:47:16.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XKlVlJASfuNpRVq+duO/rCmteEo=', 'Content-Length': '0'} 2023-10-11T14:47:16.553 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bf1a02f98d0d9b7a-006526b574-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.554 DEBUG:boto:path=/dghqvd-18/obj 2023-10-11T14:47:16.555 DEBUG:boto:auth_path=/dghqvd-18/obj 2023-10-11T14:47:16.555 DEBUG:boto:Method: HEAD 2023-10-11T14:47:16.556 DEBUG:boto:Path: /dghqvd-18/obj 2023-10-11T14:47:16.556 DEBUG:boto:Data: 2023-10-11T14:47:16.556 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.557 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.558 DEBUG:boto:Params: {} 2023-10-11T14:47:16.558 DEBUG:boto:Token: None 2023-10-11T14:47:16.559 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-18/obj 2023-10-11T14:47:16.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XKlVlJASfuNpRVq+duO/rCmteEo= 2023-10-11T14:47:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XKlVlJASfuNpRVq+duO/rCmteEo=', 'Content-Length': '0'} 2023-10-11T14:47:16.561 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009527bc2c72c98ee6-006526b574-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-18 zones={a2, a1} 2023-10-11T14:47:16.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-10-11T14:47:16.563 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-19 2023-10-11T14:47:16.564 DEBUG:boto:path=/dghqvd-19/ 2023-10-11T14:47:16.565 DEBUG:boto:auth_path=/dghqvd-19/ 2023-10-11T14:47:16.565 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.566 DEBUG:boto:Path: /dghqvd-19/ 2023-10-11T14:47:16.566 DEBUG:boto:Data: 2023-10-11T14:47:16.567 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.567 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.568 DEBUG:boto:Params: {} 2023-10-11T14:47:16.568 DEBUG:boto:Token: None 2023-10-11T14:47:16.569 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-19/ 2023-10-11T14:47:16.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FgFRJbCpkwUuQe9AArxd40l1ds4= 2023-10-11T14:47:16.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FgFRJbCpkwUuQe9AArxd40l1ds4=', 'Content-Length': '0'} 2023-10-11T14:47:16.583 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e52b540eea8d74e4-006526b574-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.584 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-20 2023-10-11T14:47:16.584 DEBUG:boto:path=/dghqvd-20/ 2023-10-11T14:47:16.584 DEBUG:boto:auth_path=/dghqvd-20/ 2023-10-11T14:47:16.585 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.586 DEBUG:boto:Path: /dghqvd-20/ 2023-10-11T14:47:16.586 DEBUG:boto:Data: 2023-10-11T14:47:16.587 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.587 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.588 DEBUG:boto:Params: {} 2023-10-11T14:47:16.588 DEBUG:boto:Token: None 2023-10-11T14:47:16.589 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-20/ 2023-10-11T14:47:16.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3MDghFVMddrlcXxfPQEgdaCJK6I= 2023-10-11T14:47:16.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3MDghFVMddrlcXxfPQEgdaCJK6I=', 'Content-Length': '0'} 2023-10-11T14:47:16.611 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd90d4bd28bdeb09-006526b574-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.612 DEBUG:boto:Using access key provided by client. 2023-10-11T14:47:16.612 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:47:16.613 DEBUG:boto:Using access key provided by client. 2023-10-11T14:47:16.614 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:47:16.615 DEBUG:boto:Using access key provided by client. 2023-10-11T14:47:16.615 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:47:16.616 DEBUG:boto:Using access key provided by client. 2023-10-11T14:47:16.616 DEBUG:boto:Using secret key provided by client. 2023-10-11T14:47:16.617 INFO:rgw_multi.tests:create bucket zone=b1 name=dghqvd-21 2023-10-11T14:47:16.618 DEBUG:boto:path=/dghqvd-21/ 2023-10-11T14:47:16.618 DEBUG:boto:auth_path=/dghqvd-21/ 2023-10-11T14:47:16.619 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.620 DEBUG:boto:Path: /dghqvd-21/ 2023-10-11T14:47:16.620 DEBUG:boto:Data: 2023-10-11T14:47:16.621 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.621 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:16.622 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.622 DEBUG:boto:Params: {} 2023-10-11T14:47:16.623 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:47:16.623 DEBUG:boto:Token: None 2023-10-11T14:47:16.624 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-21/ 2023-10-11T14:47:16.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xS63Wj5gNCd6llCMCXNKbCenZRE= 2023-10-11T14:47:16.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xS63Wj5gNCd6llCMCXNKbCenZRE=', 'Content-Length': '0'} 2023-10-11T14:47:16.656 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000245d82e8dd05794d-006526b574-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.656 INFO:rgw_multi.tests:create bucket zone=b2 name=dghqvd-22 2023-10-11T14:47:16.657 DEBUG:boto:path=/dghqvd-22/ 2023-10-11T14:47:16.657 DEBUG:boto:auth_path=/dghqvd-22/ 2023-10-11T14:47:16.658 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.658 DEBUG:boto:Path: /dghqvd-22/ 2023-10-11T14:47:16.659 DEBUG:boto:Data: 2023-10-11T14:47:16.659 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.659 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:16.660 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.660 DEBUG:boto:Params: {} 2023-10-11T14:47:16.661 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:47:16.661 DEBUG:boto:Token: None 2023-10-11T14:47:16.661 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-22/ 2023-10-11T14:47:16.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W5MtkOaYwaLx7Zl1GKd7qCCWbcQ= 2023-10-11T14:47:16.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W5MtkOaYwaLx7Zl1GKd7qCCWbcQ=', 'Content-Length': '0'} 2023-10-11T14:47:16.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ff65d168edb7926-006526b574-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.694 DEBUG:boto:path=/dghqvd-19/ 2023-10-11T14:47:16.694 DEBUG:boto:auth_path=/dghqvd-19/ 2023-10-11T14:47:16.695 DEBUG:boto:path=/dghqvd-19/?versioning 2023-10-11T14:47:16.695 DEBUG:boto:auth_path=/dghqvd-19/?versioning 2023-10-11T14:47:16.696 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.696 DEBUG:boto:Path: /dghqvd-19/?versioning 2023-10-11T14:47:16.696 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:47:16.697 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.697 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.698 DEBUG:boto:Params: {} 2023-10-11T14:47:16.698 DEBUG:boto:Token: None 2023-10-11T14:47:16.699 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-19/?versioning 2023-10-11T14:47:16.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X3HqWeXOHX5DwSpz5Keju+L1QfI= 2023-10-11T14:47:16.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X3HqWeXOHX5DwSpz5Keju+L1QfI=', 'Content-Length': '228'} 2023-10-11T14:47:16.704 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc596374efbf1154-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.705 DEBUG:boto:path=/dghqvd-19/ 2023-10-11T14:47:16.705 DEBUG:boto:auth_path=/dghqvd-19/ 2023-10-11T14:47:16.705 DEBUG:boto:path=/dghqvd-19/?versioning 2023-10-11T14:47:16.706 DEBUG:boto:auth_path=/dghqvd-19/?versioning 2023-10-11T14:47:16.706 DEBUG:boto:Method: GET 2023-10-11T14:47:16.707 DEBUG:boto:Path: /dghqvd-19/?versioning 2023-10-11T14:47:16.707 DEBUG:boto:Data: 2023-10-11T14:47:16.707 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.708 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.709 DEBUG:boto:Params: {} 2023-10-11T14:47:16.709 DEBUG:boto:Token: None 2023-10-11T14:47:16.709 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-19/?versioning 2023-10-11T14:47:16.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XHPTXqBnEYCatLUNGon6OiPHsy4= 2023-10-11T14:47:16.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XHPTXqBnEYCatLUNGon6OiPHsy4=', 'Content-Length': '0'} 2023-10-11T14:47:16.712 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2258959b4264ae2-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.712 DEBUG:boto:EnabledDisabled 2023-10-11T14:47:16.713 DEBUG:boto:path=/dghqvd-20/ 2023-10-11T14:47:16.714 DEBUG:boto:auth_path=/dghqvd-20/ 2023-10-11T14:47:16.714 DEBUG:boto:path=/dghqvd-20/?versioning 2023-10-11T14:47:16.714 DEBUG:boto:auth_path=/dghqvd-20/?versioning 2023-10-11T14:47:16.715 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.715 DEBUG:boto:Path: /dghqvd-20/?versioning 2023-10-11T14:47:16.716 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:47:16.716 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.716 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.717 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.717 DEBUG:boto:Params: {} 2023-10-11T14:47:16.718 DEBUG:boto:Token: None 2023-10-11T14:47:16.718 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-20/?versioning 2023-10-11T14:47:16.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oWvwOGCvOdrfyNKqwrThCsEkOqE= 2023-10-11T14:47:16.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oWvwOGCvOdrfyNKqwrThCsEkOqE=', 'Content-Length': '228'} 2023-10-11T14:47:16.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000060efc83d38c8c44e-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.726 DEBUG:boto:path=/dghqvd-20/ 2023-10-11T14:47:16.726 DEBUG:boto:auth_path=/dghqvd-20/ 2023-10-11T14:47:16.729 DEBUG:boto:path=/dghqvd-20/?versioning 2023-10-11T14:47:16.730 DEBUG:boto:auth_path=/dghqvd-20/?versioning 2023-10-11T14:47:16.730 DEBUG:boto:Method: GET 2023-10-11T14:47:16.730 DEBUG:boto:Path: /dghqvd-20/?versioning 2023-10-11T14:47:16.731 DEBUG:boto:Data: 2023-10-11T14:47:16.731 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.732 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.733 DEBUG:boto:Params: {} 2023-10-11T14:47:16.733 DEBUG:boto:Token: None 2023-10-11T14:47:16.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-20/?versioning 2023-10-11T14:47:16.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KXnR9Cd8DB/z6w30BhRKzACgWTU= 2023-10-11T14:47:16.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KXnR9Cd8DB/z6w30BhRKzACgWTU=', 'Content-Length': '0'} 2023-10-11T14:47:16.736 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c2bef9be050d81c-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.737 DEBUG:boto:EnabledDisabled 2023-10-11T14:47:16.737 DEBUG:boto:path=/dghqvd-21/ 2023-10-11T14:47:16.738 DEBUG:boto:auth_path=/dghqvd-21/ 2023-10-11T14:47:16.738 DEBUG:boto:path=/dghqvd-21/?versioning 2023-10-11T14:47:16.738 DEBUG:boto:auth_path=/dghqvd-21/?versioning 2023-10-11T14:47:16.739 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.739 DEBUG:boto:Path: /dghqvd-21/?versioning 2023-10-11T14:47:16.740 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:47:16.740 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.740 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:16.741 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.741 DEBUG:boto:Params: {} 2023-10-11T14:47:16.742 DEBUG:boto:Token: None 2023-10-11T14:47:16.742 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-21/?versioning 2023-10-11T14:47:16.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VWqc8KYSXR5Ti/9Nd7kGSURQwPA= 2023-10-11T14:47:16.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VWqc8KYSXR5Ti/9Nd7kGSURQwPA=', 'Content-Length': '228'} 2023-10-11T14:47:16.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2003d7e138d0ac9-006526b574-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.752 DEBUG:boto:path=/dghqvd-21/ 2023-10-11T14:47:16.752 DEBUG:boto:auth_path=/dghqvd-21/ 2023-10-11T14:47:16.753 DEBUG:boto:path=/dghqvd-21/?versioning 2023-10-11T14:47:16.753 DEBUG:boto:auth_path=/dghqvd-21/?versioning 2023-10-11T14:47:16.753 DEBUG:boto:Method: GET 2023-10-11T14:47:16.754 DEBUG:boto:Path: /dghqvd-21/?versioning 2023-10-11T14:47:16.754 DEBUG:boto:Data: 2023-10-11T14:47:16.755 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.755 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:16.756 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.757 DEBUG:boto:Params: {} 2023-10-11T14:47:16.757 DEBUG:boto:Token: None 2023-10-11T14:47:16.758 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-21/?versioning 2023-10-11T14:47:16.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BNVIB4beABfyVoPlluSIlanI7iA= 2023-10-11T14:47:16.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BNVIB4beABfyVoPlluSIlanI7iA=', 'Content-Length': '0'} 2023-10-11T14:47:16.760 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008fec2047eec88bfb-006526b574-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.760 DEBUG:boto:EnabledDisabled 2023-10-11T14:47:16.761 DEBUG:boto:path=/dghqvd-22/ 2023-10-11T14:47:16.761 DEBUG:boto:auth_path=/dghqvd-22/ 2023-10-11T14:47:16.762 DEBUG:boto:path=/dghqvd-22/?versioning 2023-10-11T14:47:16.762 DEBUG:boto:auth_path=/dghqvd-22/?versioning 2023-10-11T14:47:16.762 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.763 DEBUG:boto:Path: /dghqvd-22/?versioning 2023-10-11T14:47:16.763 DEBUG:boto:Data: Enabled Disabled 2023-10-11T14:47:16.764 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.764 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:16.764 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.765 DEBUG:boto:Params: {} 2023-10-11T14:47:16.765 DEBUG:boto:Token: None 2023-10-11T14:47:16.766 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-22/?versioning 2023-10-11T14:47:16.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZebropBm18AP4jcAyFAHhfC+hoY= 2023-10-11T14:47:16.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZebropBm18AP4jcAyFAHhfC+hoY=', 'Content-Length': '228'} 2023-10-11T14:47:16.774 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037ad3b53bd6fdeb0-006526b574-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.775 DEBUG:boto:path=/dghqvd-22/ 2023-10-11T14:47:16.775 DEBUG:boto:auth_path=/dghqvd-22/ 2023-10-11T14:47:16.776 DEBUG:boto:path=/dghqvd-22/?versioning 2023-10-11T14:47:16.776 DEBUG:boto:auth_path=/dghqvd-22/?versioning 2023-10-11T14:47:16.777 DEBUG:boto:Method: GET 2023-10-11T14:47:16.777 DEBUG:boto:Path: /dghqvd-22/?versioning 2023-10-11T14:47:16.778 DEBUG:boto:Data: 2023-10-11T14:47:16.778 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.778 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:16.779 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.779 DEBUG:boto:Params: {} 2023-10-11T14:47:16.780 DEBUG:boto:Token: None 2023-10-11T14:47:16.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-22/?versioning 2023-10-11T14:47:16.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pdf+s5s7EyTzkHFR4elQAEXShwU= 2023-10-11T14:47:16.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pdf+s5s7EyTzkHFR4elQAEXShwU=', 'Content-Length': '0'} 2023-10-11T14:47:16.782 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090e28ef050a4c32d-006526b574-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.783 DEBUG:boto:EnabledDisabled 2023-10-11T14:47:16.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-10-11T14:47:16.784 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-23 2023-10-11T14:47:16.785 DEBUG:boto:path=/dghqvd-23/ 2023-10-11T14:47:16.785 DEBUG:boto:auth_path=/dghqvd-23/ 2023-10-11T14:47:16.785 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.786 DEBUG:boto:Path: /dghqvd-23/ 2023-10-11T14:47:16.786 DEBUG:boto:Data: 2023-10-11T14:47:16.787 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.788 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.788 DEBUG:boto:Params: {} 2023-10-11T14:47:16.789 DEBUG:boto:Token: None 2023-10-11T14:47:16.789 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-23/ 2023-10-11T14:47:16.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tW6w+626yWONAbIieRgb/f66YwY= 2023-10-11T14:47:16.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tW6w+626yWONAbIieRgb/f66YwY=', 'Content-Length': '0'} 2023-10-11T14:47:16.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7d8318105d6c02d-006526b574-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.803 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-24 2023-10-11T14:47:16.803 DEBUG:boto:path=/dghqvd-24/ 2023-10-11T14:47:16.804 DEBUG:boto:auth_path=/dghqvd-24/ 2023-10-11T14:47:16.804 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.805 DEBUG:boto:Path: /dghqvd-24/ 2023-10-11T14:47:16.805 DEBUG:boto:Data: 2023-10-11T14:47:16.806 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.806 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.807 DEBUG:boto:Params: {} 2023-10-11T14:47:16.807 DEBUG:boto:Token: None 2023-10-11T14:47:16.808 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-24/ 2023-10-11T14:47:16.808 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OV1Br6HaAmZkHwa542zHS4vRDTw= 2023-10-11T14:47:16.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OV1Br6HaAmZkHwa542zHS4vRDTw=', 'Content-Length': '0'} 2023-10-11T14:47:16.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006a100d355f67e700-006526b574-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.839 INFO:rgw_multi.tests:create bucket zone=b1 name=dghqvd-25 2023-10-11T14:47:16.839 DEBUG:boto:path=/dghqvd-25/ 2023-10-11T14:47:16.840 DEBUG:boto:auth_path=/dghqvd-25/ 2023-10-11T14:47:16.840 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.841 DEBUG:boto:Path: /dghqvd-25/ 2023-10-11T14:47:16.841 DEBUG:boto:Data: 2023-10-11T14:47:16.841 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.842 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:16.842 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.843 DEBUG:boto:Params: {} 2023-10-11T14:47:16.843 DEBUG:boto:Token: None 2023-10-11T14:47:16.844 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-25/ 2023-10-11T14:47:16.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dAvLZHbsG//PIA/6qcxNI6+cqvA= 2023-10-11T14:47:16.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dAvLZHbsG//PIA/6qcxNI6+cqvA=', 'Content-Length': '0'} 2023-10-11T14:47:16.872 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005d2657258baf092-006526b574-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.873 INFO:rgw_multi.tests:create bucket zone=b2 name=dghqvd-26 2023-10-11T14:47:16.873 DEBUG:boto:path=/dghqvd-26/ 2023-10-11T14:47:16.874 DEBUG:boto:auth_path=/dghqvd-26/ 2023-10-11T14:47:16.874 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.874 DEBUG:boto:Path: /dghqvd-26/ 2023-10-11T14:47:16.875 DEBUG:boto:Data: 2023-10-11T14:47:16.876 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.877 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:16.877 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.878 DEBUG:boto:Params: {} 2023-10-11T14:47:16.878 DEBUG:boto:Token: None 2023-10-11T14:47:16.879 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-26/ 2023-10-11T14:47:16.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbn8ku+F96EZKlWUS2bIpoxn7xY= 2023-10-11T14:47:16.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbn8ku+F96EZKlWUS2bIpoxn7xY=', 'Content-Length': '0'} 2023-10-11T14:47:16.907 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad63335d61f87091-006526b574-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.908 DEBUG:boto:path=/dghqvd-23/ 2023-10-11T14:47:16.908 DEBUG:boto:auth_path=/dghqvd-23/ 2023-10-11T14:47:16.908 DEBUG:boto:path=/dghqvd-23/?acl 2023-10-11T14:47:16.909 DEBUG:boto:auth_path=/dghqvd-23/?acl 2023-10-11T14:47:16.909 DEBUG:boto:Method: GET 2023-10-11T14:47:16.910 DEBUG:boto:Path: /dghqvd-23/?acl 2023-10-11T14:47:16.910 DEBUG:boto:Data: 2023-10-11T14:47:16.911 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.911 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.912 DEBUG:boto:Params: {} 2023-10-11T14:47:16.912 DEBUG:boto:Token: None 2023-10-11T14:47:16.913 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-23/?acl 2023-10-11T14:47:16.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cdSGBYbOvP1NX1z/P6HRpm1t/c= 2023-10-11T14:47:16.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cdSGBYbOvP1NX1z/P6HRpm1t/c=', 'Content-Length': '0'} 2023-10-11T14:47:16.915 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000136d398c8205ddb0-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.916 DEBUG:boto:path=/dghqvd-23/ 2023-10-11T14:47:16.916 DEBUG:boto:auth_path=/dghqvd-23/ 2023-10-11T14:47:16.917 DEBUG:boto:path=/dghqvd-23/?acl 2023-10-11T14:47:16.917 DEBUG:boto:auth_path=/dghqvd-23/?acl 2023-10-11T14:47:16.917 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.918 DEBUG:boto:Path: /dghqvd-23/?acl 2023-10-11T14:47:16.918 DEBUG:boto:Data: 2023-10-11T14:47:16.919 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:47:16.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.919 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.920 DEBUG:boto:Params: {} 2023-10-11T14:47:16.920 DEBUG:boto:Token: None 2023-10-11T14:47:16.921 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT x-amz-acl:public-read /dghqvd-23/?acl 2023-10-11T14:47:16.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a3y5iIbmt3NYVLABRWPvKDZbM0k= 2023-10-11T14:47:16.921 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a3y5iIbmt3NYVLABRWPvKDZbM0k=', 'Content-Length': '0'} 2023-10-11T14:47:16.926 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d9b5ef67083f38c-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.927 DEBUG:boto:path=/dghqvd-23/ 2023-10-11T14:47:16.927 DEBUG:boto:auth_path=/dghqvd-23/ 2023-10-11T14:47:16.927 DEBUG:boto:path=/dghqvd-23/?acl 2023-10-11T14:47:16.928 DEBUG:boto:auth_path=/dghqvd-23/?acl 2023-10-11T14:47:16.928 DEBUG:boto:Method: GET 2023-10-11T14:47:16.929 DEBUG:boto:Path: /dghqvd-23/?acl 2023-10-11T14:47:16.929 DEBUG:boto:Data: 2023-10-11T14:47:16.929 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:16.930 DEBUG:boto:Port: 8000 2023-10-11T14:47:16.931 DEBUG:boto:Params: {} 2023-10-11T14:47:16.931 DEBUG:boto:Token: None 2023-10-11T14:47:16.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-23/?acl 2023-10-11T14:47:16.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cdSGBYbOvP1NX1z/P6HRpm1t/c= 2023-10-11T14:47:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cdSGBYbOvP1NX1z/P6HRpm1t/c=', 'Content-Length': '0'} 2023-10-11T14:47:16.934 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009eed9bc807691005-006526b574-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.934 DEBUG:boto:path=/dghqvd-24/ 2023-10-11T14:47:16.935 DEBUG:boto:auth_path=/dghqvd-24/ 2023-10-11T14:47:16.935 DEBUG:boto:path=/dghqvd-24/?acl 2023-10-11T14:47:16.936 DEBUG:boto:auth_path=/dghqvd-24/?acl 2023-10-11T14:47:16.936 DEBUG:boto:Method: GET 2023-10-11T14:47:16.936 DEBUG:boto:Path: /dghqvd-24/?acl 2023-10-11T14:47:16.937 DEBUG:boto:Data: 2023-10-11T14:47:16.937 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.938 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.938 DEBUG:boto:Params: {} 2023-10-11T14:47:16.939 DEBUG:boto:Token: None 2023-10-11T14:47:16.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-24/?acl 2023-10-11T14:47:16.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbxWN5jJl7YfKcIvc9qDKsy8//s= 2023-10-11T14:47:16.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbxWN5jJl7YfKcIvc9qDKsy8//s=', 'Content-Length': '0'} 2023-10-11T14:47:16.942 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000035e53573de9e140-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.943 DEBUG:boto:path=/dghqvd-24/ 2023-10-11T14:47:16.943 DEBUG:boto:auth_path=/dghqvd-24/ 2023-10-11T14:47:16.944 DEBUG:boto:path=/dghqvd-24/?acl 2023-10-11T14:47:16.944 DEBUG:boto:auth_path=/dghqvd-24/?acl 2023-10-11T14:47:16.945 DEBUG:boto:Method: PUT 2023-10-11T14:47:16.945 DEBUG:boto:Path: /dghqvd-24/?acl 2023-10-11T14:47:16.945 DEBUG:boto:Data: 2023-10-11T14:47:16.946 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:47:16.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.947 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.947 DEBUG:boto:Params: {} 2023-10-11T14:47:16.948 DEBUG:boto:Token: None 2023-10-11T14:47:16.948 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:16 GMT x-amz-acl:public-read /dghqvd-24/?acl 2023-10-11T14:47:16.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2s0si11+7i0PyBZwioFt/MxqQQA= 2023-10-11T14:47:16.949 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2s0si11+7i0PyBZwioFt/MxqQQA=', 'Content-Length': '0'} 2023-10-11T14:47:16.970 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000782c5d7cf0abc0a4-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:16.971 DEBUG:boto:path=/dghqvd-24/ 2023-10-11T14:47:16.971 DEBUG:boto:auth_path=/dghqvd-24/ 2023-10-11T14:47:16.972 DEBUG:boto:path=/dghqvd-24/?acl 2023-10-11T14:47:16.973 DEBUG:boto:auth_path=/dghqvd-24/?acl 2023-10-11T14:47:16.973 DEBUG:boto:Method: GET 2023-10-11T14:47:16.973 DEBUG:boto:Path: /dghqvd-24/?acl 2023-10-11T14:47:16.974 DEBUG:boto:Data: 2023-10-11T14:47:16.974 DEBUG:boto:Headers: {} 2023-10-11T14:47:16.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:16.975 DEBUG:boto:Port: 8001 2023-10-11T14:47:16.975 DEBUG:boto:Params: {} 2023-10-11T14:47:16.976 DEBUG:boto:Token: None 2023-10-11T14:47:16.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:16 GMT /dghqvd-24/?acl 2023-10-11T14:47:16.977 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbxWN5jJl7YfKcIvc9qDKsy8//s= 2023-10-11T14:47:16.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbxWN5jJl7YfKcIvc9qDKsy8//s=', 'Content-Length': '0'} 2023-10-11T14:47:16.979 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ca511e35fe04df2-006526b574-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 11 Oct 2023 14:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.023 DEBUG:boto:path=/dghqvd-25/ 2023-10-11T14:47:17.023 DEBUG:boto:auth_path=/dghqvd-25/ 2023-10-11T14:47:17.024 DEBUG:boto:path=/dghqvd-25/?acl 2023-10-11T14:47:17.024 DEBUG:boto:auth_path=/dghqvd-25/?acl 2023-10-11T14:47:17.025 DEBUG:boto:Method: GET 2023-10-11T14:47:17.025 DEBUG:boto:Path: /dghqvd-25/?acl 2023-10-11T14:47:17.025 DEBUG:boto:Data: 2023-10-11T14:47:17.026 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.026 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:17.027 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.027 DEBUG:boto:Params: {} 2023-10-11T14:47:17.027 DEBUG:boto:Token: None 2023-10-11T14:47:17.028 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-25/?acl 2023-10-11T14:47:17.028 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9yjqzoj4KIuVDKqGOB7vuOu50Fo= 2023-10-11T14:47:17.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9yjqzoj4KIuVDKqGOB7vuOu50Fo=', 'Content-Length': '0'} 2023-10-11T14:47:17.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052a2d52fb62a8b3c-006526b575-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.031 DEBUG:boto:path=/dghqvd-25/ 2023-10-11T14:47:17.031 DEBUG:boto:auth_path=/dghqvd-25/ 2023-10-11T14:47:17.032 DEBUG:boto:path=/dghqvd-25/?acl 2023-10-11T14:47:17.032 DEBUG:boto:auth_path=/dghqvd-25/?acl 2023-10-11T14:47:17.033 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.033 DEBUG:boto:Path: /dghqvd-25/?acl 2023-10-11T14:47:17.033 DEBUG:boto:Data: 2023-10-11T14:47:17.034 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:47:17.034 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:17.035 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.035 DEBUG:boto:Params: {} 2023-10-11T14:47:17.035 DEBUG:boto:Token: None 2023-10-11T14:47:17.036 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT x-amz-acl:public-read /dghqvd-25/?acl 2023-10-11T14:47:17.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IOmXdrE04YyrIYC472h7kBkV4Q4= 2023-10-11T14:47:17.037 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IOmXdrE04YyrIYC472h7kBkV4Q4=', 'Content-Length': '0'} 2023-10-11T14:47:17.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c560ad7206b6d3e6-006526b575-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.046 DEBUG:boto:path=/dghqvd-25/ 2023-10-11T14:47:17.046 DEBUG:boto:auth_path=/dghqvd-25/ 2023-10-11T14:47:17.047 DEBUG:boto:path=/dghqvd-25/?acl 2023-10-11T14:47:17.047 DEBUG:boto:auth_path=/dghqvd-25/?acl 2023-10-11T14:47:17.048 DEBUG:boto:Method: GET 2023-10-11T14:47:17.048 DEBUG:boto:Path: /dghqvd-25/?acl 2023-10-11T14:47:17.049 DEBUG:boto:Data: 2023-10-11T14:47:17.049 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.049 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:17.050 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.050 DEBUG:boto:Params: {} 2023-10-11T14:47:17.051 DEBUG:boto:Token: None 2023-10-11T14:47:17.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-25/?acl 2023-10-11T14:47:17.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9yjqzoj4KIuVDKqGOB7vuOu50Fo= 2023-10-11T14:47:17.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9yjqzoj4KIuVDKqGOB7vuOu50Fo=', 'Content-Length': '0'} 2023-10-11T14:47:17.053 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa6397d5577ff72f-006526b575-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.099 DEBUG:boto:path=/dghqvd-26/ 2023-10-11T14:47:17.099 DEBUG:boto:auth_path=/dghqvd-26/ 2023-10-11T14:47:17.100 DEBUG:boto:path=/dghqvd-26/?acl 2023-10-11T14:47:17.100 DEBUG:boto:auth_path=/dghqvd-26/?acl 2023-10-11T14:47:17.100 DEBUG:boto:Method: GET 2023-10-11T14:47:17.101 DEBUG:boto:Path: /dghqvd-26/?acl 2023-10-11T14:47:17.101 DEBUG:boto:Data: 2023-10-11T14:47:17.102 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.102 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:17.102 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.103 DEBUG:boto:Params: {} 2023-10-11T14:47:17.103 DEBUG:boto:Token: None 2023-10-11T14:47:17.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-26/?acl 2023-10-11T14:47:17.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tVTy1JvRZGhjF2O/Z7PqSzVDeCw= 2023-10-11T14:47:17.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tVTy1JvRZGhjF2O/Z7PqSzVDeCw=', 'Content-Length': '0'} 2023-10-11T14:47:17.106 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006fbdfaf90459b1d6-006526b575-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.107 DEBUG:boto:path=/dghqvd-26/ 2023-10-11T14:47:17.107 DEBUG:boto:auth_path=/dghqvd-26/ 2023-10-11T14:47:17.108 DEBUG:boto:path=/dghqvd-26/?acl 2023-10-11T14:47:17.108 DEBUG:boto:auth_path=/dghqvd-26/?acl 2023-10-11T14:47:17.109 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.109 DEBUG:boto:Path: /dghqvd-26/?acl 2023-10-11T14:47:17.110 DEBUG:boto:Data: 2023-10-11T14:47:17.110 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-10-11T14:47:17.111 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:17.111 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.112 DEBUG:boto:Params: {} 2023-10-11T14:47:17.112 DEBUG:boto:Token: None 2023-10-11T14:47:17.113 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT x-amz-acl:public-read /dghqvd-26/?acl 2023-10-11T14:47:17.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pqESPzN+NtoQrBZn6BPgEaQu4ww= 2023-10-11T14:47:17.114 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pqESPzN+NtoQrBZn6BPgEaQu4ww=', 'Content-Length': '0'} 2023-10-11T14:47:17.122 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c1c01b0b4f6d11e-006526b575-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.122 DEBUG:boto:path=/dghqvd-26/ 2023-10-11T14:47:17.123 DEBUG:boto:auth_path=/dghqvd-26/ 2023-10-11T14:47:17.123 DEBUG:boto:path=/dghqvd-26/?acl 2023-10-11T14:47:17.123 DEBUG:boto:auth_path=/dghqvd-26/?acl 2023-10-11T14:47:17.124 DEBUG:boto:Method: GET 2023-10-11T14:47:17.124 DEBUG:boto:Path: /dghqvd-26/?acl 2023-10-11T14:47:17.124 DEBUG:boto:Data: 2023-10-11T14:47:17.125 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.125 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:17.126 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.126 DEBUG:boto:Params: {} 2023-10-11T14:47:17.126 DEBUG:boto:Token: None 2023-10-11T14:47:17.127 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-26/?acl 2023-10-11T14:47:17.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tVTy1JvRZGhjF2O/Z7PqSzVDeCw= 2023-10-11T14:47:17.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tVTy1JvRZGhjF2O/Z7PqSzVDeCw=', 'Content-Length': '0'} 2023-10-11T14:47:17.129 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000acd50146f53c71fe-006526b575-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-10-11T14:47:17.172 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-27 2023-10-11T14:47:17.172 DEBUG:boto:path=/dghqvd-27/ 2023-10-11T14:47:17.173 DEBUG:boto:auth_path=/dghqvd-27/ 2023-10-11T14:47:17.173 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.174 DEBUG:boto:Path: /dghqvd-27/ 2023-10-11T14:47:17.175 DEBUG:boto:Data: 2023-10-11T14:47:17.176 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:17.177 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.177 DEBUG:boto:Params: {} 2023-10-11T14:47:17.178 DEBUG:boto:Token: None 2023-10-11T14:47:17.178 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-27/ 2023-10-11T14:47:17.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QTr5GDtK+KyM2h8/+YtyIZi26wA= 2023-10-11T14:47:17.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QTr5GDtK+KyM2h8/+YtyIZi26wA=', 'Content-Length': '0'} 2023-10-11T14:47:17.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc25b5412eaabe35-006526b575-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.192 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-28 2023-10-11T14:47:17.193 DEBUG:boto:path=/dghqvd-28/ 2023-10-11T14:47:17.193 DEBUG:boto:auth_path=/dghqvd-28/ 2023-10-11T14:47:17.194 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.194 DEBUG:boto:Path: /dghqvd-28/ 2023-10-11T14:47:17.195 DEBUG:boto:Data: 2023-10-11T14:47:17.195 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.196 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:17.196 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.196 DEBUG:boto:Params: {} 2023-10-11T14:47:17.197 DEBUG:boto:Token: None 2023-10-11T14:47:17.197 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-28/ 2023-10-11T14:47:17.198 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W5TDi9+B5/O4MOAzNkV4k2U4kK8= 2023-10-11T14:47:17.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W5TDi9+B5/O4MOAzNkV4k2U4kK8=', 'Content-Length': '0'} 2023-10-11T14:47:17.221 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000859327fd6a029c55-006526b575-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.222 INFO:rgw_multi.tests:create bucket zone=b1 name=dghqvd-29 2023-10-11T14:47:17.223 DEBUG:boto:path=/dghqvd-29/ 2023-10-11T14:47:17.224 DEBUG:boto:auth_path=/dghqvd-29/ 2023-10-11T14:47:17.224 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.225 DEBUG:boto:Path: /dghqvd-29/ 2023-10-11T14:47:17.226 DEBUG:boto:Data: 2023-10-11T14:47:17.227 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.228 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:17.229 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.229 DEBUG:boto:Params: {} 2023-10-11T14:47:17.230 DEBUG:boto:Token: None 2023-10-11T14:47:17.231 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-29/ 2023-10-11T14:47:17.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rzyQCHo85qmQqx9ZH2kBRmjgwZU= 2023-10-11T14:47:17.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rzyQCHo85qmQqx9ZH2kBRmjgwZU=', 'Content-Length': '0'} 2023-10-11T14:47:17.258 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8e7e047dacf1bca-006526b575-10fc-b1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.259 INFO:rgw_multi.tests:create bucket zone=b2 name=dghqvd-30 2023-10-11T14:47:17.260 DEBUG:boto:path=/dghqvd-30/ 2023-10-11T14:47:17.260 DEBUG:boto:auth_path=/dghqvd-30/ 2023-10-11T14:47:17.260 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.261 DEBUG:boto:Path: /dghqvd-30/ 2023-10-11T14:47:17.261 DEBUG:boto:Data: 2023-10-11T14:47:17.262 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.262 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:17.263 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.263 DEBUG:boto:Params: {} 2023-10-11T14:47:17.264 DEBUG:boto:Token: None 2023-10-11T14:47:17.264 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-30/ 2023-10-11T14:47:17.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EqC1a0qWrXmxs6n/TW1z1FxiK3k= 2023-10-11T14:47:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EqC1a0qWrXmxs6n/TW1z1FxiK3k=', 'Content-Length': '0'} 2023-10-11T14:47:17.288 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd93c7d38bfc60fe-006526b575-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.289 DEBUG:boto:path=/dghqvd-27/ 2023-10-11T14:47:17.289 DEBUG:boto:auth_path=/dghqvd-27/ 2023-10-11T14:47:17.290 DEBUG:boto:path=/dghqvd-27/?cors 2023-10-11T14:47:17.290 DEBUG:boto:auth_path=/dghqvd-27/?cors 2023-10-11T14:47:17.291 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.291 DEBUG:boto:Path: /dghqvd-27/?cors 2023-10-11T14:47:17.291 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-11T14:47:17.292 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-11T14:47:17.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:17.293 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.293 DEBUG:boto:Params: {} 2023-10-11T14:47:17.294 DEBUG:boto:Token: None 2023-10-11T14:47:17.294 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-27/?cors 2023-10-11T14:47:17.295 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R3ZZAnJ6E0VLeexY2rAdc6k9yjk= 2023-10-11T14:47:17.295 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R3ZZAnJ6E0VLeexY2rAdc6k9yjk=', 'Content-Length': '218'} 2023-10-11T14:47:17.299 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006907e00563eb4611-006526b575-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.300 DEBUG:boto:path=/dghqvd-27/ 2023-10-11T14:47:17.300 DEBUG:boto:auth_path=/dghqvd-27/ 2023-10-11T14:47:17.301 DEBUG:boto:path=/dghqvd-27/?cors 2023-10-11T14:47:17.301 DEBUG:boto:auth_path=/dghqvd-27/?cors 2023-10-11T14:47:17.302 DEBUG:boto:Method: GET 2023-10-11T14:47:17.302 DEBUG:boto:Path: /dghqvd-27/?cors 2023-10-11T14:47:17.302 DEBUG:boto:Data: 2023-10-11T14:47:17.303 DEBUG:boto:Headers: {} 2023-10-11T14:47:17.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:17.304 DEBUG:boto:Port: 8000 2023-10-11T14:47:17.304 DEBUG:boto:Params: {} 2023-10-11T14:47:17.305 DEBUG:boto:Token: None 2023-10-11T14:47:17.305 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-27/?cors 2023-10-11T14:47:17.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odwze4/rpEUvdrGiOLX0XDmqDLI= 2023-10-11T14:47:17.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odwze4/rpEUvdrGiOLX0XDmqDLI=', 'Content-Length': '0'} 2023-10-11T14:47:17.307 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ee177ed158b3a36a-006526b575-1113-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 11 Oct 2023 14:47:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:17.308 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-11T14:47:17.309 DEBUG:boto:path=/dghqvd-28/ 2023-10-11T14:47:17.310 DEBUG:boto:auth_path=/dghqvd-28/ 2023-10-11T14:47:17.313 DEBUG:boto:path=/dghqvd-28/?cors 2023-10-11T14:47:17.314 DEBUG:boto:auth_path=/dghqvd-28/?cors 2023-10-11T14:47:17.315 DEBUG:boto:Method: PUT 2023-10-11T14:47:17.315 DEBUG:boto:Path: /dghqvd-28/?cors 2023-10-11T14:47:17.316 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-11T14:47:17.317 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-11T14:47:17.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:17.318 DEBUG:boto:Port: 8001 2023-10-11T14:47:17.318 DEBUG:boto:Params: {} 2023-10-11T14:47:17.319 DEBUG:boto:Token: None 2023-10-11T14:47:17.320 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 11 Oct 2023 14:47:17 GMT /dghqvd-28/?cors 2023-10-11T14:47:17.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8liLUHWIuyWdp+wxX05pChmnTF4= 2023-10-11T14:47:17.321 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8liLUHWIuyWdp+wxX05pChmnTF4=', 'Content-Length': '218'} 2023-10-11T14:47:18.331 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dddc810777c796ba-006526b575-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:18.332 DEBUG:boto:path=/dghqvd-28/ 2023-10-11T14:47:18.332 DEBUG:boto:auth_path=/dghqvd-28/ 2023-10-11T14:47:18.333 DEBUG:boto:path=/dghqvd-28/?cors 2023-10-11T14:47:18.333 DEBUG:boto:auth_path=/dghqvd-28/?cors 2023-10-11T14:47:18.333 DEBUG:boto:Method: GET 2023-10-11T14:47:18.334 DEBUG:boto:Path: /dghqvd-28/?cors 2023-10-11T14:47:18.334 DEBUG:boto:Data: 2023-10-11T14:47:18.335 DEBUG:boto:Headers: {} 2023-10-11T14:47:18.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:18.335 DEBUG:boto:Port: 8001 2023-10-11T14:47:18.336 DEBUG:boto:Params: {} 2023-10-11T14:47:18.336 DEBUG:boto:Token: None 2023-10-11T14:47:18.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:18 GMT /dghqvd-28/?cors 2023-10-11T14:47:18.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dSBEly3WJzVWbbz58bLnzMNw6NM= 2023-10-11T14:47:18.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dSBEly3WJzVWbbz58bLnzMNw6NM=', 'Content-Length': '0'} 2023-10-11T14:47:18.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f9593036b780e07-006526b576-110f-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 11 Oct 2023 14:47:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:18.339 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-11T14:47:18.340 DEBUG:boto:path=/dghqvd-29/ 2023-10-11T14:47:18.340 DEBUG:boto:auth_path=/dghqvd-29/ 2023-10-11T14:47:18.341 DEBUG:boto:path=/dghqvd-29/?cors 2023-10-11T14:47:18.341 DEBUG:boto:auth_path=/dghqvd-29/?cors 2023-10-11T14:47:18.341 DEBUG:boto:Method: PUT 2023-10-11T14:47:18.342 DEBUG:boto:Path: /dghqvd-29/?cors 2023-10-11T14:47:18.342 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-11T14:47:18.343 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-11T14:47:18.343 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:18.343 DEBUG:boto:Port: 8000 2023-10-11T14:47:18.344 DEBUG:boto:Params: {} 2023-10-11T14:47:18.344 DEBUG:boto:Token: None 2023-10-11T14:47:18.345 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 11 Oct 2023 14:47:18 GMT /dghqvd-29/?cors 2023-10-11T14:47:18.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VQMiXKmAr/eZTsFpIyFKPg+CTBg= 2023-10-11T14:47:18.346 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VQMiXKmAr/eZTsFpIyFKPg+CTBg=', 'Content-Length': '218'} 2023-10-11T14:47:19.357 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d46769b45109d99-006526b576-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:19.358 DEBUG:boto:path=/dghqvd-29/ 2023-10-11T14:47:19.358 DEBUG:boto:auth_path=/dghqvd-29/ 2023-10-11T14:47:19.359 DEBUG:boto:path=/dghqvd-29/?cors 2023-10-11T14:47:19.359 DEBUG:boto:auth_path=/dghqvd-29/?cors 2023-10-11T14:47:19.359 DEBUG:boto:Method: GET 2023-10-11T14:47:19.360 DEBUG:boto:Path: /dghqvd-29/?cors 2023-10-11T14:47:19.360 DEBUG:boto:Data: 2023-10-11T14:47:19.361 DEBUG:boto:Headers: {} 2023-10-11T14:47:19.361 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:47:19.361 DEBUG:boto:Port: 8000 2023-10-11T14:47:19.362 DEBUG:boto:Params: {} 2023-10-11T14:47:19.363 DEBUG:boto:Token: None 2023-10-11T14:47:19.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:19 GMT /dghqvd-29/?cors 2023-10-11T14:47:19.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U3xVe8zPfciJU34J50zY/oD7YkQ= 2023-10-11T14:47:19.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U3xVe8zPfciJU34J50zY/oD7YkQ=', 'Content-Length': '0'} 2023-10-11T14:47:19.366 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041a078bb162be0f7-006526b577-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 11 Oct 2023 14:47:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:19.366 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-11T14:47:19.367 DEBUG:boto:path=/dghqvd-30/ 2023-10-11T14:47:19.367 DEBUG:boto:auth_path=/dghqvd-30/ 2023-10-11T14:47:19.368 DEBUG:boto:path=/dghqvd-30/?cors 2023-10-11T14:47:19.368 DEBUG:boto:auth_path=/dghqvd-30/?cors 2023-10-11T14:47:19.369 DEBUG:boto:Method: PUT 2023-10-11T14:47:19.369 DEBUG:boto:Path: /dghqvd-30/?cors 2023-10-11T14:47:19.369 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-10-11T14:47:19.370 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-10-11T14:47:19.370 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:19.371 DEBUG:boto:Port: 8001 2023-10-11T14:47:19.371 DEBUG:boto:Params: {} 2023-10-11T14:47:19.371 DEBUG:boto:Token: None 2023-10-11T14:47:19.372 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 11 Oct 2023 14:47:19 GMT /dghqvd-30/?cors 2023-10-11T14:47:19.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lZF8F1eeMdMz52pXAOprkFqSyI4= 2023-10-11T14:47:19.373 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lZF8F1eeMdMz52pXAOprkFqSyI4=', 'Content-Length': '218'} 2023-10-11T14:47:20.382 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000feee42519cd2f955-006526b577-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:20.383 DEBUG:boto:path=/dghqvd-30/ 2023-10-11T14:47:20.384 DEBUG:boto:auth_path=/dghqvd-30/ 2023-10-11T14:47:20.384 DEBUG:boto:path=/dghqvd-30/?cors 2023-10-11T14:47:20.385 DEBUG:boto:auth_path=/dghqvd-30/?cors 2023-10-11T14:47:20.385 DEBUG:boto:Method: GET 2023-10-11T14:47:20.385 DEBUG:boto:Path: /dghqvd-30/?cors 2023-10-11T14:47:20.386 DEBUG:boto:Data: 2023-10-11T14:47:20.386 DEBUG:boto:Headers: {} 2023-10-11T14:47:20.387 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:47:20.387 DEBUG:boto:Port: 8001 2023-10-11T14:47:20.388 DEBUG:boto:Params: {} 2023-10-11T14:47:20.388 DEBUG:boto:Token: None 2023-10-11T14:47:20.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:47:20 GMT /dghqvd-30/?cors 2023-10-11T14:47:20.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m+mW0CMhX1+n2ucRcq7kQBpCKBk= 2023-10-11T14:47:20.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m+mW0CMhX1+n2ucRcq7kQBpCKBk=', 'Content-Length': '0'} 2023-10-11T14:47:20.391 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003fe551c8d12443f0-006526b578-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 11 Oct 2023 14:47:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:20.392 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-10-11T14:47:20.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-10-11T14:47:20.393 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-31 2023-10-11T14:47:20.394 DEBUG:boto:path=/dghqvd-31/ 2023-10-11T14:47:20.394 DEBUG:boto:auth_path=/dghqvd-31/ 2023-10-11T14:47:20.395 DEBUG:boto:Method: PUT 2023-10-11T14:47:20.395 DEBUG:boto:Path: /dghqvd-31/ 2023-10-11T14:47:20.396 DEBUG:boto:Data: 2023-10-11T14:47:20.396 DEBUG:boto:Headers: {} 2023-10-11T14:47:20.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:20.397 DEBUG:boto:Port: 8000 2023-10-11T14:47:20.397 DEBUG:boto:Params: {} 2023-10-11T14:47:20.398 DEBUG:boto:Token: None 2023-10-11T14:47:20.398 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:20 GMT /dghqvd-31/ 2023-10-11T14:47:20.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5Oxr9pQiew5D0v9SktiSQy1Wjgs= 2023-10-11T14:47:20.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5Oxr9pQiew5D0v9SktiSQy1Wjgs=', 'Content-Length': '0'} 2023-10-11T14:47:20.415 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e060e44270d3bb0-006526b578-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:20.416 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-32 2023-10-11T14:47:20.416 DEBUG:boto:path=/dghqvd-32/ 2023-10-11T14:47:20.416 DEBUG:boto:auth_path=/dghqvd-32/ 2023-10-11T14:47:20.417 DEBUG:boto:Method: PUT 2023-10-11T14:47:20.417 DEBUG:boto:Path: /dghqvd-32/ 2023-10-11T14:47:20.418 DEBUG:boto:Data: 2023-10-11T14:47:20.418 DEBUG:boto:Headers: {} 2023-10-11T14:47:20.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:20.419 DEBUG:boto:Port: 8001 2023-10-11T14:47:20.421 DEBUG:boto:Params: {} 2023-10-11T14:47:20.422 DEBUG:boto:Token: None 2023-10-11T14:47:20.423 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:20 GMT /dghqvd-32/ 2023-10-11T14:47:20.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0ew3FL2GiAXlReJ9NvnLrxjwgE4= 2023-10-11T14:47:20.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0ew3FL2GiAXlReJ9NvnLrxjwgE4=', 'Content-Length': '0'} 2023-10-11T14:47:20.450 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052a4a98aa65c3ce6-006526b578-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:20.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:20.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:20.452 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:20.505 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:20.506 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:20.534 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:47:20.546 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:20.546 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.546 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:47:20.547 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:47:20.547 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.547 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.547 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:20.547 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:47:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:47:20.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035640.439010_182.1", 2023-10-11T14:47:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:20.439010Z" 2023-10-11T14:47:20.549 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.550 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:20.551 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035640.439010_182.1'} 2023-10-11T14:47:20.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:47:20.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:20.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:20.554 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:20.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:20.604 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:20.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:20.631+0000 7f6d60363780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:20.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:20.632+0000 7f6d60363780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:20.646 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:47:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:47:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:47:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:47:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.651 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.651 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:20.651 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:20.651 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:20.652 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:20.652 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:20.652 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:20.652 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:20.652 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:47:20.653 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:20.653 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.653 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.653 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:20.654 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:20.654 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:20.654 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:47:20.654 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:20.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:20.655 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:20.655 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:47:20.655 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:20.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.656 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:20.656 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:20.656 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:20.657 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:20.657 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035640.432870_179.1", 2023-10-11T14:47:20.657 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:20.657 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:20.657 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:20.658 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:20.432870Z", 2023-10-11T14:47:20.658 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:20.658 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.658 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:20.659 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:20.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.659 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:20.659 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:47:20.659 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:47:20.660 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:20.660 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:20Z" 2023-10-11T14:47:20.660 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:20.661 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035640.432870_179.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:20.432870Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:47:20Z" } 2023-10-11T14:47:20.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035640.432870_179.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:20.432870Z', 'realm_epoch': 2}}] 2023-10-11T14:47:20.662 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035640.439010_182.1'} 2023-10-11T14:47:20.663 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035640.432870_179.1'} 2023-10-11T14:47:20.663 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035640.439010_182.1 target=1_1697035640.432870_179.1 2023-10-11T14:47:25.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:25.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:25.665 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:25.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:25.723 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:25.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:25.751+0000 7fcaeaba0780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:25.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:25.752+0000 7fcaeaba0780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:25.767 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:47:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:47:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:47:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:47:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:25.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:25.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:25.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:25.774 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:25.774 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:47:25.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:25.775 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:25.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.775 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:25.775 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:25.775 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:25.776 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:25.776 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:47:25.776 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:25.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:25.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:25.777 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:47:25.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:25.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:25.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.778 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:25.778 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:25.778 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:25.778 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:25.778 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035640.439010_182.1", 2023-10-11T14:47:25.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:25.779 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:25.779 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:25.779 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:20.439010Z", 2023-10-11T14:47:25.779 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:25.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:25.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:25.780 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:25.780 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.781 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:25.781 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:47:25.781 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:47:25.781 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:25.781 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:25Z" 2023-10-11T14:47:25.782 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:25.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035640.439010_182.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:20.439010Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:47:25Z" } 2023-10-11T14:47:25.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035640.439010_182.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:20.439010Z', 'realm_epoch': 2}}] 2023-10-11T14:47:25.784 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035640.439010_182.1'} 2023-10-11T14:47:25.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035640.439010_182.1'} 2023-10-11T14:47:25.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:47:25.785 DEBUG:boto:path=/dghqvd-31/ 2023-10-11T14:47:25.786 DEBUG:boto:auth_path=/dghqvd-31/ 2023-10-11T14:47:25.786 DEBUG:boto:Method: HEAD 2023-10-11T14:47:25.787 DEBUG:boto:Path: /dghqvd-31/ 2023-10-11T14:47:25.787 DEBUG:boto:Data: 2023-10-11T14:47:25.788 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.788 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.789 DEBUG:boto:Params: {} 2023-10-11T14:47:25.789 DEBUG:boto:Token: None 2023-10-11T14:47:25.790 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-31/ 2023-10-11T14:47:25.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S/BHrpFIPB9aPAm3SWRdhSRKvZM= 2023-10-11T14:47:25.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S/BHrpFIPB9aPAm3SWRdhSRKvZM=', 'Content-Length': '0'} 2023-10-11T14:47:25.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcd68dcee099c051-006526b57d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.794 DEBUG:boto:path=/dghqvd-31/foo 2023-10-11T14:47:25.795 DEBUG:boto:auth_path=/dghqvd-31/foo 2023-10-11T14:47:25.795 DEBUG:boto:Method: PUT 2023-10-11T14:47:25.796 DEBUG:boto:Path: /dghqvd-31/foo 2023-10-11T14:47:25.796 DEBUG:boto:Data: 2023-10-11T14:47:25.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T14:47:25.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.798 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.798 DEBUG:boto:Params: {} 2023-10-11T14:47:25.798 DEBUG:boto:Token: None 2023-10-11T14:47:25.799 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-31/foo 2023-10-11T14:47:25.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WrEQ6RNvHtE7kK/gy4XfidBq46E= 2023-10-11T14:47:25.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WrEQ6RNvHtE7kK/gy4XfidBq46E='} 2023-10-11T14:47:25.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a55ad787c88f5a77-006526b57d-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.807 DEBUG:boto:path=/dghqvd-31/ 2023-10-11T14:47:25.807 DEBUG:boto:auth_path=/dghqvd-31/ 2023-10-11T14:47:25.808 DEBUG:boto:Method: DELETE 2023-10-11T14:47:25.808 DEBUG:boto:Path: /dghqvd-31/ 2023-10-11T14:47:25.809 DEBUG:boto:Data: 2023-10-11T14:47:25.809 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.810 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.810 DEBUG:boto:Params: {} 2023-10-11T14:47:25.811 DEBUG:boto:Token: None 2023-10-11T14:47:25.811 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-31/ 2023-10-11T14:47:25.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WUvB4Fa1YtGWgcqSfdLTPs80auE= 2023-10-11T14:47:25.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WUvB4Fa1YtGWgcqSfdLTPs80auE=', 'Content-Length': '0'} 2023-10-11T14:47:25.819 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000005561f4c14e91229c-006526b57d-1113-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.863 DEBUG:boto:path=/dghqvd-32/ 2023-10-11T14:47:25.863 DEBUG:boto:auth_path=/dghqvd-32/ 2023-10-11T14:47:25.864 DEBUG:boto:Method: HEAD 2023-10-11T14:47:25.864 DEBUG:boto:Path: /dghqvd-32/ 2023-10-11T14:47:25.865 DEBUG:boto:Data: 2023-10-11T14:47:25.865 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:25.866 DEBUG:boto:Port: 8001 2023-10-11T14:47:25.866 DEBUG:boto:Params: {} 2023-10-11T14:47:25.867 DEBUG:boto:Token: None 2023-10-11T14:47:25.868 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-32/ 2023-10-11T14:47:25.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PGLq+5Ih+A9bturMT2LbKi5C+4A= 2023-10-11T14:47:25.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PGLq+5Ih+A9bturMT2LbKi5C+4A=', 'Content-Length': '0'} 2023-10-11T14:47:25.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bd3ac1abc42a388-006526b57d-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.873 DEBUG:boto:path=/dghqvd-32/foo 2023-10-11T14:47:25.873 DEBUG:boto:auth_path=/dghqvd-32/foo 2023-10-11T14:47:25.874 DEBUG:boto:Method: PUT 2023-10-11T14:47:25.874 DEBUG:boto:Path: /dghqvd-32/foo 2023-10-11T14:47:25.874 DEBUG:boto:Data: 2023-10-11T14:47:25.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T14:47:25.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:25.876 DEBUG:boto:Port: 8001 2023-10-11T14:47:25.877 DEBUG:boto:Params: {} 2023-10-11T14:47:25.878 DEBUG:boto:Token: None 2023-10-11T14:47:25.879 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-32/foo 2023-10-11T14:47:25.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nBnvoFgiVpm+1xxKfo4col1rIEM= 2023-10-11T14:47:25.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nBnvoFgiVpm+1xxKfo4col1rIEM='} 2023-10-11T14:47:25.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ba1690cb56b79c1-006526b57d-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.887 DEBUG:boto:path=/dghqvd-32/ 2023-10-11T14:47:25.887 DEBUG:boto:auth_path=/dghqvd-32/ 2023-10-11T14:47:25.888 DEBUG:boto:Method: DELETE 2023-10-11T14:47:25.888 DEBUG:boto:Path: /dghqvd-32/ 2023-10-11T14:47:25.889 DEBUG:boto:Data: 2023-10-11T14:47:25.889 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:25.890 DEBUG:boto:Port: 8001 2023-10-11T14:47:25.891 DEBUG:boto:Params: {} 2023-10-11T14:47:25.891 DEBUG:boto:Token: None 2023-10-11T14:47:25.892 DEBUG:boto:StringToSign: DELETE Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-32/ 2023-10-11T14:47:25.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZEY512dVeij5IDvqCj6lsQfGWMM= 2023-10-11T14:47:25.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZEY512dVeij5IDvqCj6lsQfGWMM=', 'Content-Length': '0'} 2023-10-11T14:47:25.899 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000006d613a125ed3f949-006526b57d-110f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.942 DEBUG:boto:path=/dghqvd-31/ 2023-10-11T14:47:25.943 DEBUG:boto:auth_path=/dghqvd-31/ 2023-10-11T14:47:25.944 DEBUG:boto:Method: HEAD 2023-10-11T14:47:25.944 DEBUG:boto:Path: /dghqvd-31/ 2023-10-11T14:47:25.944 DEBUG:boto:Data: 2023-10-11T14:47:25.945 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.945 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.946 DEBUG:boto:Params: {} 2023-10-11T14:47:25.946 DEBUG:boto:Token: None 2023-10-11T14:47:25.947 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-31/ 2023-10-11T14:47:25.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S/BHrpFIPB9aPAm3SWRdhSRKvZM= 2023-10-11T14:47:25.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S/BHrpFIPB9aPAm3SWRdhSRKvZM=', 'Content-Length': '0'} 2023-10-11T14:47:25.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000078c51d2fc6ee962-006526b57d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.953 DEBUG:boto:path=/dghqvd-32/ 2023-10-11T14:47:25.954 DEBUG:boto:auth_path=/dghqvd-32/ 2023-10-11T14:47:25.955 DEBUG:boto:Method: HEAD 2023-10-11T14:47:25.956 DEBUG:boto:Path: /dghqvd-32/ 2023-10-11T14:47:25.956 DEBUG:boto:Data: 2023-10-11T14:47:25.957 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.959 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.962 DEBUG:boto:Params: {} 2023-10-11T14:47:25.964 DEBUG:boto:Token: None 2023-10-11T14:47:25.965 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-32/ 2023-10-11T14:47:25.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PGLq+5Ih+A9bturMT2LbKi5C+4A= 2023-10-11T14:47:25.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PGLq+5Ih+A9bturMT2LbKi5C+4A=', 'Content-Length': '0'} 2023-10-11T14:47:25.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acf0f6264b9f5790-006526b57d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-10-11T14:47:25.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-10-11T14:47:25.972 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-33 2023-10-11T14:47:25.972 DEBUG:boto:path=/dghqvd-33/ 2023-10-11T14:47:25.972 DEBUG:boto:auth_path=/dghqvd-33/ 2023-10-11T14:47:25.973 DEBUG:boto:Method: PUT 2023-10-11T14:47:25.974 DEBUG:boto:Path: /dghqvd-33/ 2023-10-11T14:47:25.974 DEBUG:boto:Data: 2023-10-11T14:47:25.975 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:25.976 DEBUG:boto:Port: 8000 2023-10-11T14:47:25.976 DEBUG:boto:Params: {} 2023-10-11T14:47:25.977 DEBUG:boto:Token: None 2023-10-11T14:47:25.978 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-33/ 2023-10-11T14:47:25.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZFv4mEBb9yEy2LKWAASHt5EPeCM= 2023-10-11T14:47:25.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZFv4mEBb9yEy2LKWAASHt5EPeCM=', 'Content-Length': '0'} 2023-10-11T14:47:25.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ffbecac78eeece47-006526b57d-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:25.993 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-34 2023-10-11T14:47:25.994 DEBUG:boto:path=/dghqvd-34/ 2023-10-11T14:47:25.994 DEBUG:boto:auth_path=/dghqvd-34/ 2023-10-11T14:47:25.994 DEBUG:boto:Method: PUT 2023-10-11T14:47:25.995 DEBUG:boto:Path: /dghqvd-34/ 2023-10-11T14:47:25.995 DEBUG:boto:Data: 2023-10-11T14:47:25.996 DEBUG:boto:Headers: {} 2023-10-11T14:47:25.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:25.997 DEBUG:boto:Port: 8001 2023-10-11T14:47:25.997 DEBUG:boto:Params: {} 2023-10-11T14:47:25.998 DEBUG:boto:Token: None 2023-10-11T14:47:25.998 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:47:25 GMT /dghqvd-34/ 2023-10-11T14:47:25.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xpi6jSzOeu8nR3eatJRJAUm8a10= 2023-10-11T14:47:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xpi6jSzOeu8nR3eatJRJAUm8a10=', 'Content-Length': '0'} 2023-10-11T14:47:26.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c588aebf0af19b51-006526b57d-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:26.021 DEBUG:boto:path=/dghqvd-33/ 2023-10-11T14:47:26.022 DEBUG:boto:auth_path=/dghqvd-33/ 2023-10-11T14:47:26.022 DEBUG:boto:Method: HEAD 2023-10-11T14:47:26.023 DEBUG:boto:Path: /dghqvd-33/ 2023-10-11T14:47:26.023 DEBUG:boto:Data: 2023-10-11T14:47:26.023 DEBUG:boto:Headers: {} 2023-10-11T14:47:26.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:26.024 DEBUG:boto:Port: 8000 2023-10-11T14:47:26.025 DEBUG:boto:Params: {} 2023-10-11T14:47:26.025 DEBUG:boto:Token: None 2023-10-11T14:47:26.026 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:26 GMT /dghqvd-33/ 2023-10-11T14:47:26.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bhXHw23VA6/DVgrinnyIAoUJJaU= 2023-10-11T14:47:26.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bhXHw23VA6/DVgrinnyIAoUJJaU=', 'Content-Length': '0'} 2023-10-11T14:47:26.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c169731f5e677ad6-006526b57e-1113-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:26.030 DEBUG:boto:path=/dghqvd-33/key 2023-10-11T14:47:26.030 DEBUG:boto:auth_path=/dghqvd-33/key 2023-10-11T14:47:26.031 DEBUG:boto:Method: PUT 2023-10-11T14:47:26.031 DEBUG:boto:Path: /dghqvd-33/key 2023-10-11T14:47:26.032 DEBUG:boto:Data: 2023-10-11T14:47:26.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-11T14:47:26.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:47:26.033 DEBUG:boto:Port: 8000 2023-10-11T14:47:26.033 DEBUG:boto:Params: {} 2023-10-11T14:47:26.034 DEBUG:boto:Token: None 2023-10-11T14:47:26.035 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 11 Oct 2023 14:47:26 GMT /dghqvd-33/key 2023-10-11T14:47:26.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ilSxJjleC8rIT3HaLW4Kfy4FQJI= 2023-10-11T14:47:26.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:47:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ilSxJjleC8rIT3HaLW4Kfy4FQJI='} 2023-10-11T14:47:26.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d2f79a7dafad435-006526b57e-1113-a1'), ('Date', 'Wed, 11 Oct 2023 14:47:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:26.042 DEBUG:boto:path=/dghqvd-34/ 2023-10-11T14:47:26.042 DEBUG:boto:auth_path=/dghqvd-34/ 2023-10-11T14:47:26.043 DEBUG:boto:Method: HEAD 2023-10-11T14:47:26.043 DEBUG:boto:Path: /dghqvd-34/ 2023-10-11T14:47:26.044 DEBUG:boto:Data: 2023-10-11T14:47:26.044 DEBUG:boto:Headers: {} 2023-10-11T14:47:26.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:26.045 DEBUG:boto:Port: 8001 2023-10-11T14:47:26.045 DEBUG:boto:Params: {} 2023-10-11T14:47:26.046 DEBUG:boto:Token: None 2023-10-11T14:47:26.046 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:47:26 GMT /dghqvd-34/ 2023-10-11T14:47:26.047 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gsk4fF6kGE1W6hlifWKF47K7Rtg= 2023-10-11T14:47:26.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:47:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gsk4fF6kGE1W6hlifWKF47K7Rtg=', 'Content-Length': '0'} 2023-10-11T14:47:26.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ffec349dd951cec-006526b57e-110f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:47:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:26.050 DEBUG:boto:path=/dghqvd-34/key 2023-10-11T14:47:26.051 DEBUG:boto:auth_path=/dghqvd-34/key 2023-10-11T14:47:26.051 DEBUG:boto:Method: PUT 2023-10-11T14:47:26.052 DEBUG:boto:Path: /dghqvd-34/key 2023-10-11T14:47:26.052 DEBUG:boto:Data: 2023-10-11T14:47:26.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-10-11T14:47:26.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:47:26.053 DEBUG:boto:Port: 8001 2023-10-11T14:47:26.054 DEBUG:boto:Params: {} 2023-10-11T14:47:26.054 DEBUG:boto:Token: None 2023-10-11T14:47:26.055 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 11 Oct 2023 14:47:26 GMT /dghqvd-34/key 2023-10-11T14:47:26.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xAcFAIDhVuceB0gfLollVxyr2vo= 2023-10-11T14:47:26.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:47:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xAcFAIDhVuceB0gfLollVxyr2vo='} 2023-10-11T14:47:26.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dc21541cafd4807-006526b57e-110f-a2'), ('Date', 'Wed, 11 Oct 2023 14:47:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:47:26.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.063 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:26.114 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:26.114 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:26.141 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:47:26.153 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:26.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:47:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:47:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:47:26.157 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:47:26.157 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.158 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.158 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:47:26.158 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:26.010273Z" 2023-10-11T14:47:26.158 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.159 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:26.160 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:47:26.160 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:47:26.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.162 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:26.212 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:26.212 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:26.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:26.239+0000 7f3a156fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:26.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:47:26.240+0000 7f3a156fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:47:26.255 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:26.255 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:26.256 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:26.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:26.256 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:26.256 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:47:26.257 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:26.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.257 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:26.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:26.258 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.258 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:26.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:47:26.258 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.259 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:47:26.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.259 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:47:26.259 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:26.260 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.260 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:26.260 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.261 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:26.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:26.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.261 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.261 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:47:26.262 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:26.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.264 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:26.264 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.264 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:26.264 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:47:26.265 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.265 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.265 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.265 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:47:26.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:26.266 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.266 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.266 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:26.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.267 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:47:26.267 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:47:26.267 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.268 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.268 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:47:26.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:26.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.269 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:26.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.269 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:26.270 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:47:26.270 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:47:26.270 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.270 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:26Z" 2023-10-11T14:47:26.270 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:26.272 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:47:26Z" } 2023-10-11T14:47:26.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:47:26.273 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:47:26.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:47:26.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:47:26.274 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T14:47:26.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.277 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:26.326 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:26.327 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:26.372 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:26.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.372 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:26.372 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:25.804258Z" 2023-10-11T14:47:26.373 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.373 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.373 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:26.373 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:26.374 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.374 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.374 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003183", 2023-10-11T14:47:26.374 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:26.039274Z" 2023-10-11T14:47:26.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.375 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:26.375 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:26.375 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.376 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:26.377 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003183', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:26.377 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:47:26.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:26.379 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:26.431 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:26.431 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:26.471 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:26.472 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:26.472 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:26.472 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:26.472 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:26.472 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:47:26.473 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.473 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:26.473 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.473 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:26.474 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.474 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:26.474 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-10-11T14:47:26.474 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.475 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.475 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.475 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:41.278145Z" 2023-10-11T14:47:26.475 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.475 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.476 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.477 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:26.478 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.478 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:26.478 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:26.478 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.479 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.479 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.479 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:26.479 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.479 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.480 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.480 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:26.480 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.480 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:26.480 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003042", 2023-10-11T14:47:26.481 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.481 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.481 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.481 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:46:35.293359Z" 2023-10-11T14:47:26.482 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:26.482 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:26.482 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:26.483 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:26.483 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:26.483 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:26.483 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:26.483 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:26.484 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:26.484 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.484 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:26.484 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:26.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:26.486 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:26Z" 2023-10-11T14:47:26.486 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:26.487 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:41.278145Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003042", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:46:35.293359Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:42:11.283392Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:26Z" } 2023-10-11T14:47:26.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:41.278145Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:46:35.293359Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:42:11.283392Z'}}] 2023-10-11T14:47:26.489 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003183', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:26.489 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000003042', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:26.489 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=2 master=00000000000000000000:00000000000000003183 target=00000000000000000000:00000000000000003042 2023-10-11T14:47:31.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:31.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:31.492 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:31.555 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:31.556 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:31.604 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:31.604 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:31.604 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:31.605 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:31.605 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:31.605 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:47:31.605 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.605 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:31.606 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:31.606 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:31.606 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:31.606 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:31.607 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:31.607 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:31.607 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:31.607 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:31.607 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:25.804258Z" 2023-10-11T14:47:31.608 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:31.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:31.608 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:31.608 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:31.609 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:31.609 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:31.609 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:31.609 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:31.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:31.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:31.610 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:31.610 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.610 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:31.611 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:31.611 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:31.611 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:31.611 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003042", 2023-10-11T14:47:31.611 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:31.612 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:31.612 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:31.612 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:46:35.293359Z" 2023-10-11T14:47:31.612 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:31.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.613 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:31.613 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:31.613 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:31.613 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:31.614 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:31.614 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:31.614 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:31.615 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:31.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.616 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:31.616 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:31.616 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:31.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:31.617 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:31Z" 2023-10-11T14:47:31.617 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:31.618 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:25.804258Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003042", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:46:35.293359Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:42:11.283392Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:31Z" } 2023-10-11T14:47:31.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:25.804258Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003042', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:46:35.293359Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:42:11.283392Z'}}] 2023-10-11T14:47:31.619 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003183', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:31.619 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003042', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:31.620 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=2 master=00000000000000000000:00000000000000003183 target=00000000000000000000:00000000000000003042 2023-10-11T14:47:36.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.624 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:36.684 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:36.685 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:36.729 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:36.730 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:36.730 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:36.730 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:36.731 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:36.731 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:47:36.731 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.731 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:36.731 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.732 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:36.732 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.732 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.732 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:36.733 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.733 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.733 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.733 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:25.804258Z" 2023-10-11T14:47:36.733 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.734 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.734 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:36.734 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.734 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.735 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:36.735 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.735 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.735 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.735 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:36.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.736 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.736 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.736 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:36.737 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.737 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.737 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-11T14:47:36.737 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.737 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.738 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.738 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:31.559331Z" 2023-10-11T14:47:36.738 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.738 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.739 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:36.739 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.739 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.739 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:36.739 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.740 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.740 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.740 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:36.740 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.741 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.741 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:36.741 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.742 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:36.742 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:36.742 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:36.742 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.742 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:36Z" 2023-10-11T14:47:36.743 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:36.744 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:25.804258Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:31.559331Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:42:11.283392Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:36Z" } 2023-10-11T14:47:36.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:25.804258Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:31.559331Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:42:11.283392Z'}}] 2023-10-11T14:47:36.745 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003183', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:36.745 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000003324', 3: '00000000000000000000:00000000000000002482'} 2023-10-11T14:47:36.746 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:47:36.746 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T14:47:36.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.748 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:36.802 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:36.845 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:47:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:30.480458Z" 2023-10-11T14:47:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T14:47:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:31.714080Z" 2023-10-11T14:47:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:47:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:47:36.849 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.849 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:36.850 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:36.851 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:47:36.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:36.852 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:36.904 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:36.904 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:36.946 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:47:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.949 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:36.949 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.949 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.949 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.950 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:25.883836Z" 2023-10-11T14:47:36.950 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.950 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.950 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.950 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:36.951 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.951 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.951 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:36.951 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.952 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.952 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.952 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:36.952 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.953 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.953 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.953 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:36.953 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-11T14:47:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.955 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.059007Z" 2023-10-11T14:47:36.955 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.955 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.955 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:36.956 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:36.956 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:36.956 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:36.956 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:47:36.956 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:36.957 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:36.957 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:36.957 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:47:36.957 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.957 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:36.958 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:36.958 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.958 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:36.958 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:36.959 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:36.959 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:36.959 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:36Z" 2023-10-11T14:47:36.959 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:36.960 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:25.883836Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.059007Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:36Z" } 2023-10-11T14:47:36.961 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:25.883836Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.059007Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:47:36.961 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:36.962 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:36.962 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-10-11T14:47:41.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:41.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:41.964 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:42.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:42.024 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:42.068 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:47:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:42.074 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:25.883836Z" 2023-10-11T14:47:42.076 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:42.076 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:42.076 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:42.076 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:42.077 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:42.077 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:42.077 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:42.077 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:42.078 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:42.078 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:42.078 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:42.078 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.078 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:42.079 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:42.079 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:42.079 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:42.079 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-11T14:47:42.080 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:42.080 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:42.080 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:42.080 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.059007Z" 2023-10-11T14:47:42.081 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:42.081 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.081 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:42.081 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:42.081 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:42.082 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:42.082 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:47:42.082 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:42.082 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:42.083 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:42.083 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:47:42.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:42.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:42.083 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:42.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.084 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:42.084 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:42.084 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:42.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:42.085 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:42Z" 2023-10-11T14:47:42.085 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:42.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:25.883836Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.059007Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:42Z" } 2023-10-11T14:47:42.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:25.883836Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.059007Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:47:42.087 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:42.088 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:42.088 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-10-11T14:47:47.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:47.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:47.091 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:47.150 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:47.198 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:47:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-10-11T14:47:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:25.883836Z" 2023-10-11T14:47:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:47.203 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:47.203 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:47.203 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:47.203 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:47.203 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:47.204 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:47.204 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:47.204 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.204 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:47.204 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:47.205 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:47.205 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:47.205 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-10-11T14:47:47.205 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:47.206 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:47.206 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:47.206 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.059007Z" 2023-10-11T14:47:47.206 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:47.206 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.207 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:47.207 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:47.207 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:47.207 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:47.207 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:47:47.208 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:47.208 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:47.208 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:47.208 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:47:47.208 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:47.209 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:47.209 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:47.209 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.209 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:47.210 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:47.210 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:47.210 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:47.210 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:47Z" 2023-10-11T14:47:47.210 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:47.211 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:25.883836Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.059007Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:47Z" } 2023-10-11T14:47:47.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:25.883836Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.059007Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:47:47.212 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:47.213 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:47.213 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004452 target=00000000000000000000:00000000000000004311 2023-10-11T14:47:52.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.216 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.276 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.277 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.321 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:47:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:47:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:47:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:47:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:47:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:47:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:47:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:47:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:30.480458Z" 2023-10-11T14:47:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:47:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:52.327 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:52.327 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:52.327 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:52.327 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:47:52.327 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.328 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.328 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.328 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:47:52.328 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:52.329 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:52.329 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T14:47:52.329 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:52.330 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:52.330 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:52.330 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:31.714080Z" 2023-10-11T14:47:52.331 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.331 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.331 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:47:52.332 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:52.332 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:47:52.332 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001777", 2023-10-11T14:47:52.332 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:47:52.333 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:47:52.333 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:47:52.333 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:41:50.517273Z" 2023-10-11T14:47:52.334 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.334 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.334 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:52.335 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.335 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:47:52.335 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:47:52.335 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:47:52.336 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.336 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:47:52Z" 2023-10-11T14:47:52.337 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:52.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:30.480458Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004452", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:31.714080Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:41:50.517273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:47:52Z" } 2023-10-11T14:47:52.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:30.480458Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004452', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:31.714080Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:41:50.517273Z'}}] 2023-10-11T14:47:52.339 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:52.340 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001777'} 2023-10-11T14:47:52.340 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:47:52.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.342 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.391 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.392 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.436 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:52.437 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.437 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:47:52.437 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:30.511368Z" 2023-10-11T14:47:52.437 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.438 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.438 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:52.438 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:52.438 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.438 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.439 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-11T14:47:52.439 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:31.559331Z" 2023-10-11T14:47:52.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.439 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:52.440 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:52.440 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.440 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:52.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:52.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:52.442 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:47:52.487 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.488 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.538 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.625 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:52.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.628 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.677 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.678 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.716 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:52.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.719 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.768 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.807 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:52.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.810 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:52.855 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.855 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.899 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:47:52.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.900 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:47:52.900 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:30.511368Z" 2023-10-11T14:47:52.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:47:52.901 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:47:52.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.901 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-11T14:47:52.902 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:31.559331Z" 2023-10-11T14:47:52.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:52.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:52.902 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-10-11T14:47:52.903 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:42:11.283392Z" 2023-10-11T14:47:52.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:52.903 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:47:52.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:52.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:52.905 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:47:52.950 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:52.951 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:52.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:52.999 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:53.089 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:53.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:53.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:53.092 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:53.142 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:53.181 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:53.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:53.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:47:53.184 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:47:53.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:53.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:53.273 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T14:47:53.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-10-11T14:47:53.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:53.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:53.277 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:47:53.327 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:53.328 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:53.346 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:53.347 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:47:53.347 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:47:53.347 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:47:53.347 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:47:53.348 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:47:53.348 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:47:53.348 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:47:53.348 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:47:53.348 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:47:53.349 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:47:53.349 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:47:53.349 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:47:53.349 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:47:53.349 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:47:53.350 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:47:53.350 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:47:53.350 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:47:53.350 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:47:53.351 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:47:53.351 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.351 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:47:53.352 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.352 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:47:53.352 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:47:53.352 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:47:53.353 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:47:53.353 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:47:53.353 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-11T14:47:53.353 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:47:53.353 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.354 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.354 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:47:53.354 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:47:53.354 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:47:53.354 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.355 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.355 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.355 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:47:53.355 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:47:53.356 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:53.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:53.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:47:53.358 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:47:53.408 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:47:53.408 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:47:53.449 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:47:53.472 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:47:53.472 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:47:53.473 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5, 2023-10-11T14:47:53.473 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:47:53.473 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:47:53.473 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:47:53.474 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:47:53.474 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:47:53.474 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.474 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:47:53.474 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:47:53.475 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:47:53.475 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:47:53.475 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.475 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:47:53.475 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.476 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:47:53.476 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:47:53.476 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:47:53.476 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:47:53.476 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.477 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:47:53.477 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:47:53.477 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.477 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:47:53.477 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.478 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:47:53.478 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:47:53.478 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:47:53.478 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:47:53.479 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:47:53.479 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": false, 2023-10-11T14:47:53.479 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:47:53.479 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:47:53.480 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:47:53.480 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:47:53.480 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.480 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:47:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:47:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.486 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:47:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:47:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:47:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:47:53.487 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:47:53.488 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:47:53.488 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:47:53.488 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:47:53.488 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:47:53.489 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:47:53.489 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:47:53.489 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.489 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.489 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.490 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.490 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:47:53.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.490 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:47:53.491 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:47:53.491 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:47:53.491 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:47:53.491 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.491 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.492 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.492 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:47:53.492 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:47:53.492 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:47:53.492 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:47:53.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.493 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:47:53.493 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.493 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.494 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.494 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:47:53.494 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:47:53.494 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:47:53.494 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:47:53.495 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.495 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:47:53.495 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.495 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:47:53.495 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:47:53.496 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:47:53.496 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:47:53.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.496 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:47:53.496 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:47:53.497 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.497 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:47:53.497 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.497 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:47:53.497 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:47:53.498 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:47:53.498 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:47:53.498 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:47:53.498 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:47:53.499 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:47:53.499 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:47:53.499 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:47:53.499 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:47:53.500 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.501 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.501 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.501 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.501 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:47:53.501 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:47:53.502 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:47:53.502 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:47:53.502 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.502 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:47:53.503 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:47:53.503 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:47:53.503 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:47:53.503 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:47:53.503 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:47:53.504 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:47:53.504 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:47:53.504 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:47:53.504 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:47:53.504 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.505 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.505 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.505 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.505 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:47:53.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.506 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:47:53.506 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:47:53.506 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:47:53.506 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:47:53.506 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.507 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.507 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:47:53.507 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:47:53.508 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:47:53.508 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:47:53.508 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.508 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:47:53.508 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:47:53.509 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.509 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.509 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:47:53.509 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:47:53.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.510 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:47:53.510 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:47:53.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.510 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.510 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:47:53.511 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:47:53.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.511 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.511 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:47:53.511 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:47:53.512 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.512 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:47:53.512 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:47:53.512 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:47:53.512 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.513 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:47:53.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.513 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:47:53.513 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:47:53.513 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:47:53.514 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:47:53.514 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:47:53.514 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:47:53.514 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:47:53.514 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:47:53.515 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:47:53.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.515 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:47:53.515 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:47:53.515 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:47:53.516 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:47:53.516 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:47:53.516 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:47:53.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.516 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:47:53.517 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:47:53.517 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:47:53.517 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:47:53.517 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:47:53.517 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:47:53.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.518 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:47:53.518 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:47:53.518 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:47:53.518 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:47:53.519 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:47:53.519 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:47:53.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.519 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:47:53.519 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:47:53.520 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:47:53.520 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:47:53.520 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:47:53.520 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:47:53.520 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:47:53.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:47:53.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:47:53.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:47:53.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:47:53.522 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:47:53.522 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-10-11T14:48:53.523 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-35 2023-10-11T14:48:53.524 DEBUG:boto:path=/dghqvd-35/ 2023-10-11T14:48:53.525 DEBUG:boto:auth_path=/dghqvd-35/ 2023-10-11T14:48:53.525 DEBUG:boto:Method: PUT 2023-10-11T14:48:53.526 DEBUG:boto:Path: /dghqvd-35/ 2023-10-11T14:48:53.526 DEBUG:boto:Data: 2023-10-11T14:48:53.526 DEBUG:boto:Headers: {} 2023-10-11T14:48:53.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:48:53.527 DEBUG:boto:Port: 8000 2023-10-11T14:48:53.528 DEBUG:boto:Params: {} 2023-10-11T14:48:53.529 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:48:53.529 DEBUG:boto:Token: None 2023-10-11T14:48:53.530 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:48:53 GMT /dghqvd-35/ 2023-10-11T14:48:53.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bhVEdhmH6TQbByLrfBvojO8Ujmg= 2023-10-11T14:48:53.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:48:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bhVEdhmH6TQbByLrfBvojO8Ujmg=', 'Content-Length': '0'} 2023-10-11T14:48:53.550 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bb8be444cc5d473-006526b5d5-152a-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:48:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:48:53.551 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:48:53.552 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:48:53.552 DEBUG:boto:Method: PUT 2023-10-11T14:48:53.552 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:48:53.553 DEBUG:boto:Data: 2023-10-11T14:48:53.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T14:48:53.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:48:53.554 DEBUG:boto:Port: 8000 2023-10-11T14:48:53.554 DEBUG:boto:Params: {} 2023-10-11T14:48:53.555 DEBUG:boto:Token: None 2023-10-11T14:48:53.555 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 11 Oct 2023 14:48:53 GMT /dghqvd-35/testredirect 2023-10-11T14:48:53.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ku9NHyvvu4lPPMZGM7wOikHaRlM= 2023-10-11T14:48:53.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:48:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ku9NHyvvu4lPPMZGM7wOikHaRlM='} 2023-10-11T14:48:53.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032b8af06aaf3a2b3-006526b5d5-152a-a1'), ('Date', 'Wed, 11 Oct 2023 14:48:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:48:53.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:53.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:53.567 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:48:53.619 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:48:53.620 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:48:53.649 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:48:53.661 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:48:53.661 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.661 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:48:53.661 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:48:53.662 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.662 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.662 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:48:53.662 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:48:53.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.663 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.663 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:48:53.663 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:48:53.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.666 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:48:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:48:53.545819Z" 2023-10-11T14:48:53.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.667 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:48:53.668 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:48:53.669 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:48:53.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:53.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:53.670 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:48:53.720 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:48:53.720 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:48:53.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:48:53.745+0000 7f8b5a9e0780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:48:53.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:48:53.746+0000 7f8b5a9e0780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:48:53.760 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:48:53.760 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:48:53.760 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:48:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:48:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:48:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:48:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:53.761 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.762 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:48:53.762 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.762 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:48:53.762 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:53.762 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:53.763 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:48:53.763 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:53.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:48:53.763 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:53.764 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:48:53.764 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:53.764 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.765 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:48:53.765 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:53.765 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:53.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:48:53.766 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:53.766 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:48:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:48:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:48:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:53.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:48:53.770 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:53.770 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:48:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:48:53.772 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:53.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:53.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:53.772 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:48:53.773 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:53.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:53.773 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:48:53.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.774 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:48:53.774 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:48:53.774 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:48:53.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:53.775 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:48:53Z" 2023-10-11T14:48:53.775 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:48:53.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:48:53Z" } 2023-10-11T14:48:53.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:48:53.777 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:48:53.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:48:53.778 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:48:58.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:58.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:48:58.784 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:48:58.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:48:58.842 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:48:58.872 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:48:58.869+0000 7fea2fbdc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:48:58.873 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:48:58.870+0000 7fea2fbdc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:48:58.885 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:48:58.886 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:48:58.886 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:48:58.887 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:48:58.887 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:48:58.887 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:48:58.888 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:58.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.889 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:48:58.889 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:58.890 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:48:58.890 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:58.890 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:58.891 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:48:58.891 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:58.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:48:58.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:58.892 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:48:58.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:58.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:58.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:58.894 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:48:58.896 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:58.896 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:58.896 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:48:58.897 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:58.897 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:58.897 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:58.897 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:48:58.898 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:58.898 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:58.898 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.898 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:58.898 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:48:58.899 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:58.899 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:58.899 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:48:58.899 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:58.899 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:58.900 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:58.900 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:48:58.900 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:58.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:58.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.901 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:48:58.901 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:48:58.901 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:48:58.901 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:48:58.902 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:48:58.902 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:48:58.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:48:58.902 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:48:58.902 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:48:58.903 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:48:58.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:58.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:48:58.903 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:48:58.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.904 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:48:58.904 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:48:58.904 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:48:58.904 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:48:58.904 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:48:58Z" 2023-10-11T14:48:58.905 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:48:58.906 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:48:58Z" } 2023-10-11T14:48:58.906 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:48:58.907 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:48:58.907 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:48:58.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:03.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:03.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:03.911 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:03.968 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:03.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:03.995+0000 7f8a64219780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:04.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:03.997+0000 7f8a64219780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:04.012 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:04.014 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:04.014 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:04.014 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:04.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.015 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:04.015 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:04.015 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:04.015 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:04.015 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:04.016 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:04.016 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:04.016 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:04.016 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:04.017 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:04.017 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:04.017 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:04.017 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.017 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:04.018 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:04.018 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:04.018 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:04.018 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:04.018 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:04.019 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:04.019 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:04.019 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:04.019 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:04.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:04.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:04.020 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:04.020 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:04.021 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:04.021 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:04.021 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:04.021 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:04.021 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:04.022 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:04.022 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:04.022 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:04.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:04.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:04.023 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:04.023 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:04.024 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:04.024 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:04.024 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:04.024 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:04.024 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:04.025 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:04.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:04.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:04.025 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:04.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.026 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:04.026 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:04.026 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:04.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:04.027 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:03Z" 2023-10-11T14:49:04.027 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:04.028 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:03Z" } 2023-10-11T14:49:04.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:04.029 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:04.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:04.030 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:09.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:09.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:09.032 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:09.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:09.091 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:09.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:09.118+0000 7f7bbb9fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:09.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:09.119+0000 7f7bbb9fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:09.136 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:09.136 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:09.136 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:09.136 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:09.137 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:09.137 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:09.137 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:09.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.138 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:09.138 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:09.138 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:09.138 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:09.139 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:09.139 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:09.139 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:09.139 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:09.139 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:09.140 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:09.140 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:09.140 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:09.140 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.140 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:09.141 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:09.141 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:09.141 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:09.141 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:09.142 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:09.142 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:09.142 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:09.142 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:09.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:09.143 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:09.143 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.143 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:09.143 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:09.144 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:09.144 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:09.144 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:09.144 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:09.144 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:09.145 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:09.145 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:09.145 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:09.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:09.145 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.146 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:09.146 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:09.146 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:09.146 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:09.147 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:09.147 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:09.147 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:09.147 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:09.147 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:09.148 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:09.148 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:09.148 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:09.148 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:09.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.149 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:09.149 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:09.149 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:09.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:09.149 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:09Z" 2023-10-11T14:49:09.150 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:09.151 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:09Z" } 2023-10-11T14:49:09.151 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:09.152 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:09.152 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:09.153 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:14.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:14.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:14.155 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:14.213 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:14.213 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:14.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:14.239+0000 7f3f3fdfd780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:14.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:14.240+0000 7f3f3fdfd780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:14.255 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:14.256 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:14.256 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:14.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:14.256 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:14.257 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:14.257 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:14.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.257 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:14.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:14.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:14.258 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:14.258 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:14.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:14.259 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:14.259 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:14.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:14.259 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:14.259 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:14.260 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:14.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:14.260 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:14.261 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:14.261 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:14.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:14.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:14.261 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:14.262 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:14.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:14.262 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:14.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:14.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:14.263 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:14.263 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:14.264 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:14.264 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:14.264 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:14.264 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:14.264 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:14.265 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:14.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:14.265 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:14.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.265 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:14.266 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:14.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:14.266 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:14.266 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:14.266 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:14.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:14.267 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:14.268 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:14.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:14.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:14.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:14.269 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:14.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.269 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:14.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:14.269 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:14.270 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:14.270 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:14Z" 2023-10-11T14:49:14.270 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:14.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:14Z" } 2023-10-11T14:49:14.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:14.272 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:14.273 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:14.273 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:19.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:19.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:19.275 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:19.332 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:19.333 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:19.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:19.359+0000 7fb766e46780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:19.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:19.360+0000 7fb766e46780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:19.376 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:19.376 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:19.376 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:19.377 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:19.377 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:19.377 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:19.377 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:19.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.378 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:19.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:19.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:19.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:19.379 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:19.380 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:19.380 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:19.380 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:19.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.380 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:19.381 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:19.381 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:19.381 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:19.381 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:19.382 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:19.382 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:19.382 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:19.382 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:19.382 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:19.383 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:19.383 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.383 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:19.383 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:19.383 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:19.384 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:19.384 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:19.384 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:19.384 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:19.384 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:19.385 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:19.385 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:19.385 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:19.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.386 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:19.386 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:19.386 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:19.386 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:19.386 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:19.387 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:19.387 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:19.387 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:19.387 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:19.387 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:19.388 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:19.388 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:19.388 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:19.388 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.388 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:19.389 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:19.389 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:19.389 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:19.389 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:19Z" 2023-10-11T14:49:19.390 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:19.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:19Z" } 2023-10-11T14:49:19.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:19.392 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:19.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:19.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:24.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:24.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:24.395 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:24.452 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:24.452 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:24.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:24.479+0000 7fd238392780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:24.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:24.481+0000 7fd238392780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:24.497 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:24.497 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:24.498 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:24.498 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:24.498 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:24.498 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:24.499 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:24.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.499 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:24.499 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:24.499 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:24.500 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:24.500 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:24.500 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:24.500 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:24.500 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:24.501 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:24.501 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:24.501 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:24.501 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:24.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.502 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:24.502 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:24.502 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:24.502 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:24.503 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:24.503 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:24.503 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:24.503 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:24.504 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:24.504 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:24.504 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:24.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:24.505 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:24.505 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:24.505 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:24.505 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:24.506 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:24.506 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:24.506 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:24.507 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:24.507 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:24.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:24.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:24.508 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:24.509 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:24.509 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:24.509 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:24.509 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:24.509 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:24.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:24.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:24.510 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:24.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.511 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:24.511 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:24.511 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:24.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:24.512 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:24Z" 2023-10-11T14:49:24.512 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:24.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:24Z" } 2023-10-11T14:49:24.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:24.515 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:24.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:24.516 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:29.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:29.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:29.519 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:29.576 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:29.577 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:29.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:29.604+0000 7f921866d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:29.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:29.605+0000 7f921866d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:29.621 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:29.621 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:29.621 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:29.622 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:29.622 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:29.622 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:29.622 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:29.623 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:29.624 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:29.626 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:29.627 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:29.628 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:29.629 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.630 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:29.631 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:29.632 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:29.633 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:29.634 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:29Z" 2023-10-11T14:49:29.634 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:29.635 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:29Z" } 2023-10-11T14:49:29.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:29.636 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:29.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:29.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:34.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:34.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:34.640 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:34.699 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:34.700 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:34.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:34.727+0000 7f2c4539d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:34.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:34.728+0000 7f2c4539d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:34.744 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:34.744 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:34.745 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:34.745 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:34.745 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:34.745 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:34.746 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:34.746 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.746 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:34.746 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:34.747 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:34.747 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:34.747 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:34.747 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:34.748 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:34.748 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:34.748 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:34.748 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:34.748 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:34.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:34.749 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.749 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:34.750 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:34.750 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:34.750 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:34.750 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:34.751 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:34.751 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:34.751 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:34.751 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:34.752 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:34.752 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:34.752 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.752 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:34.752 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:34.753 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:34.753 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:34.753 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:34.753 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:34.753 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:34.755 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:34.755 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:34.756 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:34.756 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:34.757 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.757 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:34.757 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:34.758 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:34.758 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:34.759 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:34.759 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:34.759 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:34.760 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:34.760 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:34.761 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:34.761 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:34.761 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:34.762 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:34.762 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.763 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:34.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:34.764 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:34.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:34.764 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:34Z" 2023-10-11T14:49:34.765 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:34.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:34Z" } 2023-10-11T14:49:34.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:34.768 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:34.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:34.769 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:39.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:39.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:39.771 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:39.829 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:39.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:39.856+0000 7f14188f3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:39.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:39.857+0000 7f14188f3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:39.873 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:39.874 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:39.874 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:39.874 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:39.874 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:39.874 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:39.875 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:39.875 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.875 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:39.875 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:39.876 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:39.876 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:39.876 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:39.876 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:39.876 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:39.877 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:39.877 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:39.877 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:39.877 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:39.878 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:39.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.878 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:39.878 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:39.878 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:39.879 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:39.879 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:39.879 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:39.879 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:39.879 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:39.880 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:39.880 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:39.880 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:39.880 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.881 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:39.881 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:39.881 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:39.881 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:39.881 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:39.882 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:39.882 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:39.882 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:39.883 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:39.883 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:39.883 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:39.883 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.884 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:39.884 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:39.884 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:39.884 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:39.885 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:39.885 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:39.885 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:39.885 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:39.885 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:39.886 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:39.886 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:39.886 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:39.886 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:39.886 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.887 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:39.887 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:39.887 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:39.887 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:39.888 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:39Z" 2023-10-11T14:49:39.888 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:39.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:39Z" } 2023-10-11T14:49:39.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:39.890 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:39.890 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:39.891 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:44.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:44.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:44.893 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:44.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:44.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:44.982 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:44.979+0000 7f57326f4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:44.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:44.980+0000 7f57326f4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:44.995 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:44.996 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:44.996 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:44.996 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:44.996 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:44.997 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:44.997 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:44.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:44.997 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:44.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:44.998 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:44.998 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:44.998 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:44.998 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:45.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:45.000 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:45.000 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:45.000 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:45.001 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:45.001 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:45.001 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:45.001 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:45.001 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:45.002 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:45.002 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:45.002 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:45.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:45.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:45.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:45.003 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:45.003 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:45.004 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:45.004 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:45.004 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:45.004 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:45.004 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:45.005 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:45.005 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:45.005 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:45.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:45.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:45.006 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:45.006 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:45.006 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:45.006 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:45.007 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:45.007 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:45.007 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:45.007 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:45.007 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:45.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:45.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:45.008 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:45.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:45.009 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:45.010 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:45.010 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:45.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:45.011 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:44Z" 2023-10-11T14:49:45.011 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:45.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:44Z" } 2023-10-11T14:49:45.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:45.013 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:45.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:45.014 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:50.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:50.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:50.016 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:50.074 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:50.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:50.102+0000 7f585766a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:50.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:50.104+0000 7f585766a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:50.119 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:50.120 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:50.120 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:50.120 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:50.120 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:50.121 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:50.121 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:50.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.121 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:50.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:50.122 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:50.122 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:50.122 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:50.122 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:50.123 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:50.123 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:50.123 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:50.123 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:50.123 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:50.124 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:50.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.124 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:50.124 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:50.124 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:50.125 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:50.125 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:50.125 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:50.125 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:50.126 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:50.126 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:50.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:50.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:50.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.127 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:50.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:50.127 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:50.127 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:50.127 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:50.128 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:50.128 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:50.128 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:50.128 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:50.129 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:50.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:50.129 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.129 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:50.129 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:50.130 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:50.130 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:50.130 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035646.010273_190.1", 2023-10-11T14:49:50.130 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:50.130 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:50.131 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:50.131 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:26.010273Z", 2023-10-11T14:49:50.131 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:50.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:50.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:50.132 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:50.132 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.132 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:50.132 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:50.132 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:50.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:50.133 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:50Z" 2023-10-11T14:49:50.133 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:50.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035646.010273_190.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:26.010273Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:50Z" } 2023-10-11T14:49:50.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035646.010273_190.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:26.010273Z', 'realm_epoch': 2}}] 2023-10-11T14:49:50.135 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:50.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035646.010273_190.1'} 2023-10-11T14:49:50.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035733.545819_204.1 target=1_1697035646.010273_190.1 2023-10-11T14:49:55.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:55.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:49:55.138 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:49:55.198 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:55.198 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:55.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:55.226+0000 7fea154c4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:55.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:49:55.227+0000 7fea154c4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:49:55.242 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:55.243 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:49:55.243 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:49:55.243 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:49:55.243 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:49:55.244 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:55.244 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.244 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.244 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:49:55.244 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.245 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:49:55.245 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:55.245 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:55.246 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:49:55.246 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:55.246 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:49:55.246 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:55.247 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:49:55.247 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.247 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.248 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:49:55.248 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:55.248 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:55.248 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:49:55.249 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:55.249 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:55.249 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:55.249 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:49:55.249 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.250 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.250 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.250 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.250 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:49:55.250 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:55.251 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:55.251 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:49:55.251 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:55.251 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:55.252 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:55.252 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:49:55.252 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.252 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.252 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.253 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.253 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:49:55.253 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:55.253 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:49:55.253 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:49:55.254 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:49:55.254 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:49:55.254 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:49:55.254 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:49:55.255 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.255 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.255 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.255 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.256 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.256 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:49:55.256 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:49:55.256 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:49:55.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.257 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:49:55Z" 2023-10-11T14:49:55.257 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:55.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:49:55Z" } 2023-10-11T14:49:55.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:49:55.260 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:55.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:49:55.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:49:55.262 DEBUG:boto:path=/dghqvd-35/ 2023-10-11T14:49:55.263 DEBUG:boto:auth_path=/dghqvd-35/ 2023-10-11T14:49:55.264 DEBUG:boto:Method: HEAD 2023-10-11T14:49:55.264 DEBUG:boto:Path: /dghqvd-35/ 2023-10-11T14:49:55.265 DEBUG:boto:Data: 2023-10-11T14:49:55.265 DEBUG:boto:Headers: {} 2023-10-11T14:49:55.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:49:55.266 DEBUG:boto:Port: 8001 2023-10-11T14:49:55.267 DEBUG:boto:Params: {} 2023-10-11T14:49:55.268 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:49:55.268 DEBUG:boto:Token: None 2023-10-11T14:49:55.269 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:49:55 GMT /dghqvd-35/ 2023-10-11T14:49:55.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DlX1zahsFdTcCtToVqRHsw44nR4= 2023-10-11T14:49:55.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:49:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DlX1zahsFdTcCtToVqRHsw44nR4=', 'Content-Length': '0'} 2023-10-11T14:49:55.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001fad6ee6e7e2e79-006526b613-1526-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:49:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:49:55.277 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:49:55.278 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:49:55.278 DEBUG:boto:Method: HEAD 2023-10-11T14:49:55.279 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:49:55.279 DEBUG:boto:Data: 2023-10-11T14:49:55.279 DEBUG:boto:Headers: {} 2023-10-11T14:49:55.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:49:55.280 DEBUG:boto:Port: 8001 2023-10-11T14:49:55.281 DEBUG:boto:Params: {} 2023-10-11T14:49:55.281 DEBUG:boto:Token: None 2023-10-11T14:49:55.281 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:49:55 GMT /dghqvd-35/testredirect 2023-10-11T14:49:55.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LH8ZlUdOScd4i0S72RJK6jpAwvo= 2023-10-11T14:49:55.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:49:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LH8ZlUdOScd4i0S72RJK6jpAwvo=', 'Content-Length': '0'} 2023-10-11T14:49:55.284 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000181e178bdebfccdb-006526b613-1526-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:49:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:49:55.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=daa316e9-869c-4074-a756-a8adfb4a3573', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:49:55.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=daa316e9-869c-4074-a756-a8adfb4a3573', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:49:55.286 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=daa316e9-869c-4074-a756-a8adfb4a3573 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:49:55.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:55.336 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:55.354 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:55.355 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:49:55.355 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:49:55.355 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:49:55.355 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:49:55.356 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:49:55.356 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:49:55.356 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:49:55.356 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:49:55.356 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:49:55.357 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:49:55.357 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:49:55.357 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:49:55.357 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:49:55.357 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:49:55.358 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:49:55.358 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:49:55.358 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:49:55.358 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:49:55.358 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:49:55.359 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.359 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:49:55.359 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.359 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:49:55.359 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:49:55.360 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:49:55.360 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:49:55.360 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:49:55.360 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-11T14:49:55.361 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:49:55.361 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.361 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.361 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:49:55.361 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:49:55.362 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:49:55.362 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.362 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.362 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.362 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:49:55.363 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:49:55.363 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:55.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:49:55.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:49:55.365 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:49:55.414 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:49:55.414 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:49:55.454 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:49:55.479 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:49:55.480 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:55.480 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6, 2023-10-11T14:49:55.480 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:49:55.481 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:49:55.481 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:49:55.481 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:49:55.482 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:49:55.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.483 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:49:55.483 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:49:55.483 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:49:55.484 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:49:55.484 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.484 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:49:55.485 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.485 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:49:55.485 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:49:55.486 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:49:55.486 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:49:55.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.487 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:49:55.487 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:49:55.487 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.488 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:49:55.488 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.488 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:49:55.489 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:49:55.489 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:49:55.489 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:49:55.490 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:49:55.490 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": false, 2023-10-11T14:49:55.491 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:49:55.491 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:49:55.492 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:49:55.492 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:49:55.492 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.493 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.493 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.494 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:49:55.494 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:49:55.495 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.495 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:49:55.495 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.496 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:49:55.496 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:49:55.497 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:49:55.497 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:49:55.497 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:49:55.498 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:49:55.498 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:49:55.498 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:49:55.498 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:49:55.499 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:49:55.499 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.499 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.499 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.500 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.500 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:49:55.500 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.500 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:49:55.501 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:49:55.501 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:49:55.501 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:49:55.501 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.501 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.502 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.502 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:49:55.502 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:49:55.502 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:49:55.502 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:49:55.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.503 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:49:55.503 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.503 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.504 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:49:55.504 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:49:55.504 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:49:55.505 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:49:55.505 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.505 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:49:55.505 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.505 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:49:55.506 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:49:55.506 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:49:55.506 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:49:55.506 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.506 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:49:55.507 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:49:55.507 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.507 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:49:55.507 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.508 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:49:55.508 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:49:55.508 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:49:55.508 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:49:55.509 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:49:55.509 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:49:55.509 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:49:55.510 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:49:55.510 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:49:55.511 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:49:55.511 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.512 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.512 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.512 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.513 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:49:55.513 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:49:55.513 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:49:55.513 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:49:55.514 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.514 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:49:55.514 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:49:55.514 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:49:55.515 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:49:55.515 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:49:55.515 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:49:55.515 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:49:55.515 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:49:55.516 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:49:55.516 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:49:55.516 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.516 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.517 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.517 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.517 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:49:55.517 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.517 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:49:55.518 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:49:55.518 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:49:55.518 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:49:55.518 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.519 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.519 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:49:55.519 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:49:55.519 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:49:55.520 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:49:55.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.520 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:49:55.521 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:49:55.521 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.522 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.522 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:49:55.522 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:49:55.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.523 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:49:55.523 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:49:55.523 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.524 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.524 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:49:55.524 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:49:55.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:49:55.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:49:55.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:49:55.526 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:49:55.526 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:49:55.526 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.526 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:49:55.526 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.527 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:49:55.527 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:49:55.527 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:49:55.527 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:49:55.528 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:49:55.528 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:49:55.528 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:49:55.528 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:49:55.529 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:49:55.529 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.529 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:49:55.529 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:49:55.529 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:49:55.530 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:49:55.530 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:49:55.530 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:49:55.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.531 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:49:55.531 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:49:55.531 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:49:55.532 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:49:55.532 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:49:55.532 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:49:55.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.533 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:49:55.533 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:49:55.533 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:49:55.533 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:49:55.533 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:49:55.534 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:49:55.534 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.534 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:49:55.535 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:49:55.535 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:49:55.535 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:49:55.535 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:49:55.536 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:49:55.536 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:49:55.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:49:55.536 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:49:55.536 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:49:55.537 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:49:55.537 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:49:55.537 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T14:50:55.539 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.540 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.541 DEBUG:boto:Method: HEAD 2023-10-11T14:50:55.541 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.542 DEBUG:boto:Data: 2023-10-11T14:50:55.542 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.543 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.544 DEBUG:boto:Params: {} 2023-10-11T14:50:55.544 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.545 DEBUG:boto:Token: None 2023-10-11T14:50:55.545 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/0bQrxMhjrl2XjyV6M1JNY5hDa0= 2023-10-11T14:50:55.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/0bQrxMhjrl2XjyV6M1JNY5hDa0=', 'Content-Length': '0'} 2023-10-11T14:50:55.553 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx00000e517f1e579b819f7-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.553 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.554 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-10-11T14:50:55.554 DEBUG:boto:Token: None 2023-10-11T14:50:55.555 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/0bQrxMhjrl2XjyV6M1JNY5hDa0= 2023-10-11T14:50:55.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/0bQrxMhjrl2XjyV6M1JNY5hDa0=', 'Content-Length': '0'} 2023-10-11T14:50:55.562 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:48:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000e412059da618f41a-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.564 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.564 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.565 DEBUG:boto:Method: GET 2023-10-11T14:50:55.565 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.565 DEBUG:boto:Data: 2023-10-11T14:50:55.566 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.567 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.567 DEBUG:boto:Params: {} 2023-10-11T14:50:55.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.568 DEBUG:boto:Token: None 2023-10-11T14:50:55.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.573 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx000008ba44da1713a6ff6-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.573 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.574 DEBUG:boto:Token: None 2023-10-11T14:50:55.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.575 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.578 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:48:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000e43cf8792cc97a39-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.579 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.579 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.580 DEBUG:boto:Method: PUT 2023-10-11T14:50:55.580 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.580 DEBUG:boto:Data: 2023-10-11T14:50:55.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T14:50:55.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:50:55.582 DEBUG:boto:Port: 8000 2023-10-11T14:50:55.582 DEBUG:boto:Params: {} 2023-10-11T14:50:55.583 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:50:55.583 DEBUG:boto:Token: None 2023-10-11T14:50:55.584 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H9ABmlo9QiZFcKCoQ/OTGc7FwpY= 2023-10-11T14:50:55.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H9ABmlo9QiZFcKCoQ/OTGc7FwpY='} 2023-10-11T14:50:55.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2c99e41fa46dd5e-006526b64f-15ae-a1'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.598 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.598 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.599 DEBUG:boto:Method: GET 2023-10-11T14:50:55.599 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.600 DEBUG:boto:Data: 2023-10-11T14:50:55.600 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.601 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.602 DEBUG:boto:Params: {} 2023-10-11T14:50:55.602 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.603 DEBUG:boto:Token: None 2023-10-11T14:50:55.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.608 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx000000d614b4fa7a9c2da-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.608 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.609 DEBUG:boto:Token: None 2023-10-11T14:50:55.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.612 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx000005b74adb5a49fbd04-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.613 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.614 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.614 DEBUG:boto:Method: PUT 2023-10-11T14:50:55.615 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.615 DEBUG:boto:Data: 2023-10-11T14:50:55.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T14:50:55.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:50:55.617 DEBUG:boto:Port: 8000 2023-10-11T14:50:55.617 DEBUG:boto:Params: {} 2023-10-11T14:50:55.618 DEBUG:boto:Token: None 2023-10-11T14:50:55.618 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.619 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WfzZAzW6UtngSLxrXg6PwSU1vhw= 2023-10-11T14:50:55.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WfzZAzW6UtngSLxrXg6PwSU1vhw='} 2023-10-11T14:50:55.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f3a8575f93657ca-006526b64f-15ae-a1'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.626 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.627 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.627 DEBUG:boto:Method: GET 2023-10-11T14:50:55.628 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.629 DEBUG:boto:Data: 2023-10-11T14:50:55.629 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.631 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.631 DEBUG:boto:Params: {} 2023-10-11T14:50:55.632 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.633 DEBUG:boto:Token: None 2023-10-11T14:50:55.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.638 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx000004b7fa0f6a243feca-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.639 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.639 DEBUG:boto:Token: None 2023-10-11T14:50:55.640 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.642 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx000005702a583017740b6-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.643 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.644 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.644 DEBUG:boto:Method: PUT 2023-10-11T14:50:55.644 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.645 DEBUG:boto:Data: 2023-10-11T14:50:55.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T14:50:55.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:50:55.646 DEBUG:boto:Port: 8000 2023-10-11T14:50:55.646 DEBUG:boto:Params: {} 2023-10-11T14:50:55.647 DEBUG:boto:Token: None 2023-10-11T14:50:55.647 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KWjC2P9pmbaz+JNv07ZXcpXanBU= 2023-10-11T14:50:55.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KWjC2P9pmbaz+JNv07ZXcpXanBU='} 2023-10-11T14:50:55.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b183973da67a708-006526b64f-15ae-a1'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.654 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.655 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.655 DEBUG:boto:Method: GET 2023-10-11T14:50:55.656 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.656 DEBUG:boto:Data: 2023-10-11T14:50:55.657 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.657 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.658 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.658 DEBUG:boto:Params: {} 2023-10-11T14:50:55.658 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.659 DEBUG:boto:Token: None 2023-10-11T14:50:55.659 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.663 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx00000ae2136dbd4af2f65-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.664 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.664 DEBUG:boto:Token: None 2023-10-11T14:50:55.665 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.668 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000051e01e055a271a5-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.668 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.669 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.669 DEBUG:boto:Method: PUT 2023-10-11T14:50:55.670 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.670 DEBUG:boto:Data: 2023-10-11T14:50:55.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T14:50:55.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:50:55.671 DEBUG:boto:Port: 8000 2023-10-11T14:50:55.671 DEBUG:boto:Params: {} 2023-10-11T14:50:55.672 DEBUG:boto:Token: None 2023-10-11T14:50:55.672 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w+JvUxIZXmB0g6tvhkbUAfhYbMo= 2023-10-11T14:50:55.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w+JvUxIZXmB0g6tvhkbUAfhYbMo='} 2023-10-11T14:50:55.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ce4f79064ded025-006526b64f-15ae-a1'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.719 DEBUG:boto:path=/dghqvd-35/testredirect 2023-10-11T14:50:55.720 DEBUG:boto:auth_path=/dghqvd-35/testredirect 2023-10-11T14:50:55.720 DEBUG:boto:Method: GET 2023-10-11T14:50:55.720 DEBUG:boto:Path: /dghqvd-35/testredirect 2023-10-11T14:50:55.721 DEBUG:boto:Data: 2023-10-11T14:50:55.721 DEBUG:boto:Headers: {} 2023-10-11T14:50:55.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:50:55.722 DEBUG:boto:Port: 8001 2023-10-11T14:50:55.723 DEBUG:boto:Params: {} 2023-10-11T14:50:55.723 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:50:55.723 DEBUG:boto:Token: None 2023-10-11T14:50:55.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.728 DEBUG:boto:Response headers: [('Location', 'http://smithi158.front.sepia.ceph.com:8000/dghqvd-35/testredirect?'), ('x-amz-request-id', 'tx00000dfa6c5627d39ebde-006526b64f-15b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.729 DEBUG:boto:Redirecting: http://smithi158.front.sepia.ceph.com/dghqvd-35/testredirect 2023-10-11T14:50:55.729 DEBUG:boto:Token: None 2023-10-11T14:50:55.730 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:50:55 GMT /dghqvd-35/testredirect 2023-10-11T14:50:55.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20= 2023-10-11T14:50:55.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:50:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:teR3Z7c1IDJn90piYPkPypFGZ20=', 'Content-Length': '0'} 2023-10-11T14:50:55.733 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000005ae57ea9b628fef9-006526b64f-15ae-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:50:55.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:50:55.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:50:55.735 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:50:55.791 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:50:55.791 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:50:55.812 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:50:55.812 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:50:55.812 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:50:55.813 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:50:55.813 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:50:55.813 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:50:55.813 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:50:55.814 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:50:55.814 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:50:55.814 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:50:55.814 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:50:55.814 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:50:55.815 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:50:55.815 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:50:55.815 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:50:55.815 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:50:55.816 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:50:55.816 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:50:55.816 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:50:55.816 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:50:55.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:50:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:50:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:50:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:50:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:50:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:50:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-11T14:50:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:50:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:50:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:50:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:50:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:50:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:50:55.821 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:50:55.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:50:55.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:50:55.824 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:50:55.874 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:50:55.874 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:50:55.914 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:50:55.937 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:50:55.938 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:50:55.938 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7, 2023-10-11T14:50:55.938 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:50:55.938 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:50:55.939 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:50:55.939 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:50:55.939 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:50:55.939 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.940 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:50:55.940 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:50:55.940 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:50:55.940 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:50:55.940 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.941 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:50:55.941 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.941 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:50:55.941 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:50:55.941 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:50:55.942 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:50:55.942 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.942 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:50:55.942 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:50:55.943 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.943 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:50:55.943 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.943 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:50:55.943 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:50:55.944 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:50:55.944 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:50:55.944 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:50:55.944 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:50:55.944 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:50:55.945 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:50:55.945 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:50:55.945 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:50:55.946 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.946 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.947 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.947 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.947 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:50:55.948 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:50:55.948 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.948 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:50:55.948 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.949 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:50:55.949 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:50:55.949 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:50:55.949 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:50:55.949 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:50:55.950 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:50:55.950 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:50:55.950 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:50:55.950 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:50:55.950 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:50:55.951 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.951 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.951 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.951 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.952 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:50:55.952 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.952 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:50:55.952 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:50:55.952 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:50:55.953 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:50:55.953 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.953 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.953 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.953 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:50:55.954 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:50:55.954 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:50:55.954 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:50:55.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.954 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:50:55.955 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.955 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.955 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.955 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.956 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:50:55.956 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:50:55.956 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:50:55.956 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:50:55.956 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.957 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:50:55.957 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.957 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:50:55.957 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:50:55.957 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:50:55.958 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:50:55.958 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.958 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:50:55.958 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:50:55.958 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.959 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:50:55.959 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.959 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:50:55.959 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:50:55.959 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:50:55.960 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:50:55.960 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:50:55.960 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:50:55.960 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:50:55.960 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:50:55.961 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:50:55.962 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:50:55.962 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.962 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.963 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.963 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.963 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:50:55.963 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:50:55.963 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:50:55.964 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:50:55.964 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.964 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:50:55.964 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:50:55.964 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:50:55.965 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:50:55.965 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:50:55.965 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:50:55.965 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:50:55.966 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:50:55.966 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:50:55.966 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:50:55.966 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.966 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.967 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.967 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.967 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:50:55.967 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.967 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:50:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:50:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:50:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:50:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.968 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.969 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.969 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:50:55.969 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:50:55.969 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:50:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:50:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:50:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:50:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.971 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.971 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:50:55.971 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:50:55.971 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.971 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:50:55.972 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:50:55.972 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.972 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.972 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:50:55.973 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:50:55.973 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.973 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.973 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:50:55.973 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:50:55.974 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.974 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:50:55.974 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:50:55.974 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:50:55.975 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.975 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:50:55.975 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.975 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:50:55.975 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:50:55.976 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:50:55.976 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:50:55.976 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:50:55.976 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:50:55.976 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:50:55.977 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:50:55.977 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:50:55.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.977 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:50:55.977 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:50:55.978 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:50:55.978 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:50:55.978 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:50:55.978 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:50:55.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.979 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:50:55.979 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:50:55.979 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:50:55.979 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:50:55.980 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:50:55.980 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:50:55.980 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.980 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:50:55.980 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:50:55.981 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:50:55.981 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:50:55.981 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:50:55.981 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:50:55.981 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.982 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:50:55.982 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:50:55.982 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:50:55.982 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:50:55.982 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:50:55.983 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:50:55.983 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:50:55.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:50:55.983 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:50:55.983 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:50:55.984 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:50:55.984 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:50:55.984 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-10-11T14:51:55.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:51:55.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:51:55.988 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:51:56.045 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:51:56.046 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:51:56.067 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:51:56.067 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:51:56.067 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:51:56.068 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "a2.rgw.meta:root", 2023-10-11T14:51:56.068 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "a2.rgw.control", 2023-10-11T14:51:56.068 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "a2.rgw.log:gc", 2023-10-11T14:51:56.068 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "a2.rgw.log:lc", 2023-10-11T14:51:56.068 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "a2.rgw.log", 2023-10-11T14:51:56.069 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-10-11T14:51:56.069 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-10-11T14:51:56.069 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-10-11T14:51:56.069 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-10-11T14:51:56.070 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-10-11T14:51:56.070 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-10-11T14:51:56.070 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-10-11T14:51:56.070 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-10-11T14:51:56.070 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "a2.rgw.otp", 2023-10-11T14:51:56.071 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:51:56.071 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:51:56.071 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:51:56.071 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.072 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:51:56.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:51:56.072 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:51:56.072 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "a2.rgw.buckets.index", 2023-10-11T14:51:56.073 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:51:56.073 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:51:56.073 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "a2.rgw.buckets.data", 2023-10-11T14:51:56.073 INFO:teuthology.orchestra.run.smithi158.stdout: "compression_type": "random" 2023-10-11T14:51:56.073 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.074 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.074 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-10-11T14:51:56.074 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:51:56.074 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "a2.rgw.log:notif" 2023-10-11T14:51:56.075 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:51:56.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:51:56.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:51:56.078 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:51:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:51:56.130 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:51:56.170 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:51:56.194 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:51:56.194 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:51:56.195 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8, 2023-10-11T14:51:56.195 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:51:56.195 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:51:56.195 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:51:56.195 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:51:56.196 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:51:56.196 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.196 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:51:56.196 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:51:56.196 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:51:56.197 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:51:56.197 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.197 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:51:56.197 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.197 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:51:56.198 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:51:56.198 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:51:56.198 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:51:56.198 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.199 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:51:56.199 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:51:56.199 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.199 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:51:56.199 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.200 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:51:56.200 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:51:56.200 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:51:56.200 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:51:56.201 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:51:56.201 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:51:56.201 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:51:56.201 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:51:56.201 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:51:56.202 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:51:56.202 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.202 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.202 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.202 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.203 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:51:56.203 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:51:56.203 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.203 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:51:56.204 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.204 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:51:56.204 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:51:56.204 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:51:56.204 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:51:56.205 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:51:56.205 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:51:56.205 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:51:56.205 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:51:56.205 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:51:56.206 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:51:56.206 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.206 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.206 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.207 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.207 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:51:56.207 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.207 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:51:56.207 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:51:56.208 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:51:56.208 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:51:56.208 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.208 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.208 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.209 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:51:56.209 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:51:56.209 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:51:56.209 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:51:56.209 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.210 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:51:56.210 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.210 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.210 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.210 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.211 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:51:56.211 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:51:56.211 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:51:56.211 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:51:56.211 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.212 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:51:56.212 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.212 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:51:56.212 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:51:56.212 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:51:56.213 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:51:56.213 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.213 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:51:56.213 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:51:56.214 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:51:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:51:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:51:56.215 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:51:56.216 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:51:56.216 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:51:56.216 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:51:56.216 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:51:56.216 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:51:56.217 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:51:56.217 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:51:56.218 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.218 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.219 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.219 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.219 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:51:56.219 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:51:56.220 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:51:56.220 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:51:56.220 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.220 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:51:56.220 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:51:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:51:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:51:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:51:56.221 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:51:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:51:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:51:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:51:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:51:56.222 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:51:56.223 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:51:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:51:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:51:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:51:56.224 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:51:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:51:56.225 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:51:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:51:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:51:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:51:56.226 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:51:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:51:56.227 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:51:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:51:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.228 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:51:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:51:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:51:56.229 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:51:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:51:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:51:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:51:56.230 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:51:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:51:56.231 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:51:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:51:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:51:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:51:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:51:56.232 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:51:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:51:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:51:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:51:56.233 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:51:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:51:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:51:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:51:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:51:56.234 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:51:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:51:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:51:56.235 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:51:56.236 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:51:56.236 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:51:56.236 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.236 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:51:56.236 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:51:56.237 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:51:56.237 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:51:56.237 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:51:56.237 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:51:56.237 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:51:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:51:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:51:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:51:56.238 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:51:56.239 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:51:56.239 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:51:56.239 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:51:56.239 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:51:56.239 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:51:56.240 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:51:56.240 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:51:56.240 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-10-11T14:51:56.408 INFO:tasks.rgw.c1.client.0.smithi158.stdout:2023-10-11T14:51:56.405+0000 7f46422bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-10-11T14:52:56.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-10-11T14:52:56.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:52:56.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:52:56.245 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:52:56.302 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:52:56.303 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:52:56.334 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:52:56.347 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:52:56.347 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.347 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:52:56.347 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:52:56.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.348 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.348 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:52:56.348 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:52:56.349 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.349 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.349 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:52:56.349 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:52:56.349 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.350 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.350 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:52:56.350 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:48:53.545819Z" 2023-10-11T14:52:56.350 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.351 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:52:56.352 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:52:56.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:52:56.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:52:56.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:52:56.354 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:52:56.407 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:52:56.408 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:52:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:52:56.432+0000 7fba2452e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:52:56.436 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:52:56.433+0000 7fba2452e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:52:56.448 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:52:56.448 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:52:56.448 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:52:56.448 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:52:56.449 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:52:56.449 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:52:56.449 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:52:56.449 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.450 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:52:56.450 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.450 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:52:56.450 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:52:56.450 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:52:56.451 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:52:56.451 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:52:56.451 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:52:56.451 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:52:56.452 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:52:56.452 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:52:56.452 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.452 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.452 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.453 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:52:56.453 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:52:56.453 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:52:56.453 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:52:56.453 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:52:56.454 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:52:56.454 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:52:56.454 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:52:56.454 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:52:56.455 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.455 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.455 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:52:56.455 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:52:56.456 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:52:56.456 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:52:56.456 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:52:56.456 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:52:56.456 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:52:56.457 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:52:56.457 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:52:56.457 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.457 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.458 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.458 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:52:56.458 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:52:56.458 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:52:56.458 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:52:56.459 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:52:56.459 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:52:56.459 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:52:56.459 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:52:56.459 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:52:56.460 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.460 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.460 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:52:56.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.460 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:52:56.461 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:52:56.461 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:52:56.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.461 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:52:56Z" 2023-10-11T14:52:56.462 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:52:56.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:52:56Z" } 2023-10-11T14:52:56.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:52:56.464 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:52:56.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:52:56.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:52:56.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:52:56.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '9S35SN7VRMRVPFGV', '--secret', '9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:52:56.466 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key 9S35SN7VRMRVPFGV --secret 9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:52:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:52:56.518 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:52:56.531 INFO:teuthology.orchestra.run.smithi158.stderr:NOTICE: set zone's realm_id=17c10fd9-3252-456e-81e5-d40bac926339 2023-10-11T14:52:56.546 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:52:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "11b39e78-5ea2-4a1c-b893-caed01f0d4f8", 2023-10-11T14:52:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove", 2023-10-11T14:52:56.546 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-11T14:52:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "remove.rgw.control", 2023-10-11T14:52:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-11T14:52:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-11T14:52:56.547 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "remove.rgw.log", 2023-10-11T14:52:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-11T14:52:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-11T14:52:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-11T14:52:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-11T14:52:56.548 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-11T14:52:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-11T14:52:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-11T14:52:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-11T14:52:56.549 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "remove.rgw.otp", 2023-10-11T14:52:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:52:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "9S35SN7VRMRVPFGV", 2023-10-11T14:52:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "9V6qiQ3BeSKFLNHtRHiEyykRoDSDVPxz" 2023-10-11T14:52:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.550 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:52:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:52:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:52:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:52:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-11T14:52:56.551 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:52:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:52:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-11T14:52:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:52:56.552 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-11T14:52:56.553 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:52:56.553 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:52:56.553 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.553 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:52:56.553 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:52:56.554 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:52:56.554 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-11T14:52:56.554 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:52:56.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:52:56.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:52:56.556 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:52:56.604 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:52:56.605 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:52:56.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:52:56.613+0000 7fbef293f780 0 period (064106cf-4523-427d-9a69-628884ec70af does not have zone 11b39e78-5ea2-4a1c-b893-caed01f0d4f8 configured 2023-10-11T14:53:03.432 INFO:teuthology.orchestra.run.smithi158.stderr:Sending period to new master zone daa316e9-869c-4074-a756-a8adfb4a3573 2023-10-11T14:53:03.457 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:03.458 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:03.458 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9, 2023-10-11T14:53:03.458 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:53:03.459 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:53:03.459 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:53:03.459 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:03.459 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:53:03.459 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.460 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:53:03.460 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:53:03.460 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:03.460 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:53:03.460 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.461 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.461 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.461 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:03.461 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:03.461 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.462 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:03.462 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.462 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "11b39e78-5ea2-4a1c-b893-caed01f0d4f8", 2023-10-11T14:53:03.462 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove", 2023-10-11T14:53:03.463 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [], 2023-10-11T14:53:03.463 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.463 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.463 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.463 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.464 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.464 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.464 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.464 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.464 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.465 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.465 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.465 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.465 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.465 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.466 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:03.466 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:53:03.466 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.466 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:53:03.467 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.467 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.467 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.467 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.467 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.468 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.468 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.468 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.468 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.469 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.469 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.469 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.469 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.469 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.470 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.470 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.470 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:53:03.470 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.470 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.471 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.471 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.471 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.471 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.472 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.472 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.472 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.472 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.472 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.473 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.473 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.473 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.473 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.474 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.474 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.474 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:03.474 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.474 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:03.475 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:03.475 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:03.475 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:03.475 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.475 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:03.477 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.477 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:03.477 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.477 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.477 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.478 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.478 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:53:03.478 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:53:03.478 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:03.478 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:03.479 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.479 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.479 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.479 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:03.479 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:03.480 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:03.480 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:03.480 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.480 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:03.480 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:53:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.481 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.482 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.482 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.482 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.482 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.483 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.483 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.483 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.483 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.483 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.484 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.484 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.484 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.484 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:03.484 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:53:03.485 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.485 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:53:03.485 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.485 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.485 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.486 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.486 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.486 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.486 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.487 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.487 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.487 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.487 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.487 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.488 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.488 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.488 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.488 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:03.488 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.489 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:03.489 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:03.489 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:03.489 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:03.489 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.490 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.490 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.490 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:03.490 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.490 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:03.491 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:03.491 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.491 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:03.491 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.491 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.492 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.492 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:53:03.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.493 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:03.493 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:53:03.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.493 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.493 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "11b39e78-5ea2-4a1c-b893-caed01f0d4f8", 2023-10-11T14:53:03.494 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3318155887 2023-10-11T14:53:03.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.494 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.494 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:03.494 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:53:03.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.495 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.495 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:03.495 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:53:03.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.496 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.496 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:53:03.496 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.496 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.497 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:53:03.497 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.497 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:53:03.497 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:53:03.498 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:03.498 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:03.498 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:03.498 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:03.498 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:53:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:03.499 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:03.500 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:03.500 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:03.500 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.500 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:53:03.500 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:03.501 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:03.501 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:03.501 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:03.501 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:03.501 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.502 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:53:03.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:03.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:03.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:03.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:03.503 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:03.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.503 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:53:03.503 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:03.503 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:03.504 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:03.504 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:03.504 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:03.504 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.505 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.506 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:53:03.506 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:03.506 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:03.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.508 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:03.564 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:03.564 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:03.584 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:03.584 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:53:03.584 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:53:03.585 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:03.585 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:53:03.585 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.585 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.585 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.586 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:03.586 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:03.586 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.586 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:03.587 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.587 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:03.587 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:53:03.587 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.587 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:53:03.588 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.588 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.588 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.588 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.588 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.589 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.589 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.589 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.589 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.589 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.590 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.590 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.590 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.590 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:53:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.591 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.592 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.592 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.592 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.592 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.592 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.593 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.593 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.593 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.593 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.593 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.594 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.594 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.594 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.594 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.594 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.595 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:03.595 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.595 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:03.595 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:03.595 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:03.596 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:03.596 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.596 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.596 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.597 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:03.597 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.597 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:03.597 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:03.597 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.598 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:03.598 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.598 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.598 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:03.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.601 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:03.659 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:03.659 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:03.694 INFO:teuthology.orchestra.run.smithi158.stderr:zone name remove not found in zonegroup a 2023-10-11T14:53:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-11T14:53:03.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.703 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:03.763 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:03.763 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:03.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:03.824 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:03.893 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:03.893 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:03.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:03.911+0000 7f70392cf780 0 WARNING: can't generate connection for zone 11b39e78-5ea2-4a1c-b893-caed01f0d4f8 id remove: no endpoints defined 2023-10-11T14:53:03.967 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:03.968 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:03.968 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10, 2023-10-11T14:53:03.968 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:53:03.969 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:53:03.969 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:53:03.969 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:03.969 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:53:03.970 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.970 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:53:03.970 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:53:03.970 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:03.970 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:53:03.971 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.971 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.971 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.971 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:03.972 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:53:03.973 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.973 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:53:03.973 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.973 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.974 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.974 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.974 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.974 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.975 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.975 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.975 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.975 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.975 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.976 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.976 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.976 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.976 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:53:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.977 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.978 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.978 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.978 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.978 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.978 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.979 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.979 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.979 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.979 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.979 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.980 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.980 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.980 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.980 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.980 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:03.981 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.981 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:03.981 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:03.981 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:03.982 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:03.982 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.982 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.982 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.982 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:03.983 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.983 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:03.983 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:03.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:53:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:53:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:03.985 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.986 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.986 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.986 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:03.986 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:03.987 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:03.987 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:03.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.987 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:03.987 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:53:03.988 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.988 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:03.988 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.988 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.988 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.989 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.989 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.989 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.989 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.990 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.990 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.990 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.990 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.991 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.992 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:03.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.992 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:03.993 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:53:03.993 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:03.993 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:53:03.993 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.994 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:03.994 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:03.994 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:03.994 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:03.994 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:03.995 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:03.995 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:03.995 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:03.995 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:03.996 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:03.996 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:03.996 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.996 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.996 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.997 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:03.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:03.997 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:03.997 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:03.997 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:03.998 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:03.998 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:03.998 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:03.998 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:03.999 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:03.999 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:03.999 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:03.999 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:03.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.000 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.000 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.000 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.000 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.001 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:53:04.001 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.001 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.001 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:53:04.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.002 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.002 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:04.002 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:53:04.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.003 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:04.003 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:53:04.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.004 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.004 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:53:04.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.004 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.004 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.005 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:53:04.005 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.005 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:53:04.005 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:53:04.005 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.006 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.006 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.006 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.006 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.006 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.007 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:53:04.007 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.007 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.007 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.007 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.008 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.008 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:53:04.008 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.008 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.009 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.009 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.009 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.009 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:53:04.010 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.010 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.010 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.010 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:53:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.011 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:53:04.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:04.013 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:04.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-10-11T14:53:04.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.018 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.074 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.075 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.102 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:04.103 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "267519f1-309f-4e3b-9b59-23282ba28e33", 2023-10-11T14:53:04.103 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove_zg", 2023-10-11T14:53:04.103 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "remove_zg", 2023-10-11T14:53:04.103 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:04.103 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [], 2023-10-11T14:53:04.104 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.104 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.104 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "", 2023-10-11T14:53:04.104 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [], 2023-10-11T14:53:04.104 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.105 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.105 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.106 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.106 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [] 2023-10-11T14:53:04.106 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.106 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.106 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.107 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.107 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.107 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.107 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.107 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.108 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.108 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.108 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:04.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.110 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.175 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.176 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:04.232+0000 7f546d68c780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-10-11T14:53:04.241 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:04.241 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "1142aae8-485e-42a1-8b10-859acff39c1b", 2023-10-11T14:53:04.241 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove", 2023-10-11T14:53:04.241 INFO:teuthology.orchestra.run.smithi158.stdout: "domain_root": "remove.rgw.meta:root", 2023-10-11T14:53:04.242 INFO:teuthology.orchestra.run.smithi158.stdout: "control_pool": "remove.rgw.control", 2023-10-11T14:53:04.242 INFO:teuthology.orchestra.run.smithi158.stdout: "gc_pool": "remove.rgw.log:gc", 2023-10-11T14:53:04.242 INFO:teuthology.orchestra.run.smithi158.stdout: "lc_pool": "remove.rgw.log:lc", 2023-10-11T14:53:04.242 INFO:teuthology.orchestra.run.smithi158.stdout: "log_pool": "remove.rgw.log", 2023-10-11T14:53:04.243 INFO:teuthology.orchestra.run.smithi158.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-10-11T14:53:04.243 INFO:teuthology.orchestra.run.smithi158.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-10-11T14:53:04.243 INFO:teuthology.orchestra.run.smithi158.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-10-11T14:53:04.243 INFO:teuthology.orchestra.run.smithi158.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-10-11T14:53:04.243 INFO:teuthology.orchestra.run.smithi158.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-10-11T14:53:04.244 INFO:teuthology.orchestra.run.smithi158.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-10-11T14:53:04.244 INFO:teuthology.orchestra.run.smithi158.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-10-11T14:53:04.244 INFO:teuthology.orchestra.run.smithi158.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-10-11T14:53:04.244 INFO:teuthology.orchestra.run.smithi158.stdout: "otp_pool": "remove.rgw.otp", 2023-10-11T14:53:04.245 INFO:teuthology.orchestra.run.smithi158.stdout: "system_key": { 2023-10-11T14:53:04.245 INFO:teuthology.orchestra.run.smithi158.stdout: "access_key": "", 2023-10-11T14:53:04.245 INFO:teuthology.orchestra.run.smithi158.stdout: "secret_key": "" 2023-10-11T14:53:04.245 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.245 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_pools": [ 2023-10-11T14:53:04.246 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.246 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "default-placement", 2023-10-11T14:53:04.246 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:04.246 INFO:teuthology.orchestra.run.smithi158.stdout: "index_pool": "remove.rgw.buckets.index", 2023-10-11T14:53:04.246 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": { 2023-10-11T14:53:04.247 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD": { 2023-10-11T14:53:04.247 INFO:teuthology.orchestra.run.smithi158.stdout: "data_pool": "remove.rgw.buckets.data" 2023-10-11T14:53:04.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.248 INFO:teuthology.orchestra.run.smithi158.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-10-11T14:53:04.248 INFO:teuthology.orchestra.run.smithi158.stdout: "index_type": 0, 2023-10-11T14:53:04.248 INFO:teuthology.orchestra.run.smithi158.stdout: "inline_data": true 2023-10-11T14:53:04.248 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.248 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.249 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.249 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.249 INFO:teuthology.orchestra.run.smithi158.stdout: "notif_pool": "remove.rgw.log:notif" 2023-10-11T14:53:04.249 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:04.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.252 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.321 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.410 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:04.410 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:04.410 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11, 2023-10-11T14:53:04.411 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:53:04.411 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:53:04.411 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:53:04.411 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:04.412 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:53:04.412 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.412 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "267519f1-309f-4e3b-9b59-23282ba28e33", 2023-10-11T14:53:04.412 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove_zg", 2023-10-11T14:53:04.412 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "remove_zg", 2023-10-11T14:53:04.413 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:04.413 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [], 2023-10-11T14:53:04.413 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.413 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.413 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "1142aae8-485e-42a1-8b10-859acff39c1b", 2023-10-11T14:53:04.414 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:04.414 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.414 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "1142aae8-485e-42a1-8b10-859acff39c1b", 2023-10-11T14:53:04.414 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "remove", 2023-10-11T14:53:04.415 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [], 2023-10-11T14:53:04.415 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.415 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": false, 2023-10-11T14:53:04.415 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.415 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.416 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.416 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.416 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.416 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.417 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.417 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.417 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.417 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.418 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.418 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.418 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.418 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.419 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.419 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:04.419 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:04.419 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.419 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.420 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.420 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.420 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.420 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.421 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.421 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.421 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.421 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.422 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.422 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.422 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:53:04.422 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:53:04.422 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:04.423 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:53:04.423 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.423 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.423 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.424 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.424 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.424 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.424 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:04.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.425 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:04.425 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:53:04.425 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.425 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:53:04.425 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.426 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.426 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.426 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.426 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.427 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.427 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.427 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.427 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.427 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.428 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.428 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.428 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.428 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.429 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.429 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:53:04.429 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.429 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.429 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.430 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.430 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.430 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.430 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.431 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.431 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.431 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.431 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.431 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.432 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.432 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.432 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.432 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.433 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.433 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.433 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.433 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.433 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.434 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:04.434 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:04.434 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.434 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.434 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.435 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.435 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.435 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.435 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.436 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.436 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.436 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.437 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.437 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:53:04.437 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:53:04.437 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:04.437 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:04.438 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.438 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.438 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.438 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.439 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.439 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.439 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:04.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.440 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.440 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:53:04.440 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.440 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.440 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.441 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.441 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.441 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.441 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.441 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.442 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.442 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.442 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.442 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.442 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.443 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.443 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.443 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.443 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.443 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:04.444 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:53:04.444 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.444 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:53:04.444 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.445 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.445 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.445 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.445 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.445 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.446 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.446 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.446 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.446 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.446 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.447 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.447 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.447 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.447 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.447 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.448 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.448 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.448 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.448 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:04.448 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:04.449 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.449 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.449 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.449 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.450 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.450 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.450 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.450 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.450 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.451 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:53:04.452 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.452 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.452 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:53:04.452 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.453 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.453 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "1142aae8-485e-42a1-8b10-859acff39c1b", 2023-10-11T14:53:04.453 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2987217095 2023-10-11T14:53:04.453 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.453 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.454 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:04.454 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:53:04.454 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.454 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.454 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:04.455 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:53:04.455 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.455 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.456 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:53:04.456 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.456 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.456 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.456 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:53:04.457 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.457 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:53:04.458 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:53:04.458 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.459 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.459 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.459 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.459 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.459 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:53:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:53:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:53:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:53:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:53:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:04.467 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:04.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.468 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.528 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.529 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:04.558+0000 7ff2072c8780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-10-11T14:53:04.561 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:04.559+0000 7ff2072c8780 0 Removed zone from zonegroup remove_zg 2023-10-11T14:53:04.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.579 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.641 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.641 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.686 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:04.684+0000 7f647303f780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-10-11T14:53:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-11T14:53:04.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.696 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.752 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.753 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:53:04.792 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:53:04.857 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:04.857 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:04.957 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:04.958 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:04.958 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 12, 2023-10-11T14:53:04.958 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T14:53:04.958 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T14:53:04.959 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T14:53:04.959 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:04.959 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T14:53:04.959 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.960 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T14:53:04.960 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T14:53:04.960 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:04.960 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T14:53:04.960 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.961 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.961 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.961 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.961 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.962 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.962 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:04.962 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.962 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:04.962 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T14:53:04.963 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.963 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T14:53:04.963 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.963 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.964 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.964 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.964 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.964 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.964 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.965 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.965 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.965 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.965 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.965 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.966 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.966 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.966 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.966 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.967 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T14:53:04.967 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.967 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.967 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.967 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.968 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.968 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.968 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.968 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.968 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.969 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.969 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.969 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.969 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.970 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.970 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.970 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.970 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.970 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.971 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.971 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.971 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.971 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:04.971 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:04.972 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.972 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.972 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.972 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.973 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.973 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.973 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.973 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.973 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.974 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.974 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.974 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.974 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.975 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T14:53:04.975 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T14:53:04.975 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T14:53:04.975 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T14:53:04.975 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.976 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.976 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.976 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T14:53:04.976 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T14:53:04.976 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T14:53:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T14:53:04.977 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.978 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T14:53:04.978 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.978 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.978 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.979 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.979 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.979 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.979 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.979 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.980 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.980 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.980 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.981 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.981 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.982 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.982 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.982 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:04.982 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T14:53:04.983 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T14:53:04.983 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T14:53:04.983 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.983 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T14:53:04.984 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T14:53:04.984 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T14:53:04.984 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T14:53:04.984 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T14:53:04.984 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T14:53:04.985 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T14:53:04.985 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T14:53:04.985 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T14:53:04.985 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T14:53:04.986 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.986 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.986 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.986 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.986 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T14:53:04.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.987 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T14:53:04.987 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T14:53:04.987 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T14:53:04.987 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T14:53:04.988 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.988 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.988 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.988 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T14:53:04.988 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:04.989 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T14:53:04.989 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T14:53:04.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.989 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T14:53:04.989 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T14:53:04.990 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.990 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.990 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T14:53:04.990 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T14:53:04.991 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.991 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T14:53:04.991 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T14:53:04.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.991 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.992 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:53:04.992 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T14:53:04.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T14:53:04.993 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T14:53:04.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.993 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:04.994 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.994 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T14:53:04.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:04.994 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:04.994 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.995 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T14:53:04.995 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:53:04.995 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T14:53:04.995 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T14:53:04.996 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.996 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.996 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.996 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.996 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.997 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T14:53:04.997 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T14:53:04.997 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T14:53:04.997 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T14:53:04.998 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T14:53:04.998 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T14:53:04.998 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:04.998 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T14:53:04.998 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:04.999 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:04.999 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:04.999 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:04.999 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:05.000 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:05.000 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T14:53:05.000 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:05.000 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:05.000 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:05.001 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:05.001 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:05.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:05.001 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T14:53:05.001 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T14:53:05.002 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T14:53:05.002 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T14:53:05.002 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T14:53:05.002 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T14:53:05.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:05.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:05.003 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T14:53:05.003 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T14:53:05.003 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:05.004 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:05.004 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-10-11T14:53:05.005 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-36 2023-10-11T14:53:05.006 DEBUG:boto:path=/dghqvd-36/ 2023-10-11T14:53:05.006 DEBUG:boto:auth_path=/dghqvd-36/ 2023-10-11T14:53:05.007 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.007 DEBUG:boto:Path: /dghqvd-36/ 2023-10-11T14:53:05.007 DEBUG:boto:Data: 2023-10-11T14:53:05.008 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.008 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:05.009 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.009 DEBUG:boto:Params: {} 2023-10-11T14:53:05.010 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:53:05.010 DEBUG:boto:Token: None 2023-10-11T14:53:05.011 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-36/ 2023-10-11T14:53:05.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wBGNI2BAgkA0ScOAKO3ueII31Hs= 2023-10-11T14:53:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wBGNI2BAgkA0ScOAKO3ueII31Hs=', 'Content-Length': '0'} 2023-10-11T14:53:05.166 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041d33d051f4a7c8f-006526b6d1-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.167 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-37 2023-10-11T14:53:05.167 DEBUG:boto:path=/dghqvd-37/ 2023-10-11T14:53:05.168 DEBUG:boto:auth_path=/dghqvd-37/ 2023-10-11T14:53:05.168 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.168 DEBUG:boto:Path: /dghqvd-37/ 2023-10-11T14:53:05.169 DEBUG:boto:Data: 2023-10-11T14:53:05.169 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:53:05.170 DEBUG:boto:Port: 8001 2023-10-11T14:53:05.170 DEBUG:boto:Params: {} 2023-10-11T14:53:05.171 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:53:05.172 DEBUG:boto:Token: None 2023-10-11T14:53:05.172 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-37/ 2023-10-11T14:53:05.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZMDxlrfq7c80FzyLC24zh+/v6Jw= 2023-10-11T14:53:05.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZMDxlrfq7c80FzyLC24zh+/v6Jw=', 'Content-Length': '0'} 2023-10-11T14:53:05.303 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064b33a48d2a3a1fd-006526b6d1-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.304 INFO:rgw_multi.tests:create bucket zone=b1 name=dghqvd-38 2023-10-11T14:53:05.304 DEBUG:boto:path=/dghqvd-38/ 2023-10-11T14:53:05.305 DEBUG:boto:auth_path=/dghqvd-38/ 2023-10-11T14:53:05.305 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.305 DEBUG:boto:Path: /dghqvd-38/ 2023-10-11T14:53:05.306 DEBUG:boto:Data: 2023-10-11T14:53:05.306 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.307 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:53:05.307 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.307 DEBUG:boto:Params: {} 2023-10-11T14:53:05.308 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:53:05.309 DEBUG:boto:Token: None 2023-10-11T14:53:05.309 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-38/ 2023-10-11T14:53:05.310 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x76pE6pZtxIu0bvL5aZbTb9ffZY= 2023-10-11T14:53:05.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x76pE6pZtxIu0bvL5aZbTb9ffZY=', 'Content-Length': '0'} 2023-10-11T14:53:05.395 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cfb9f7f1b5a5a5de-006526b6d1-1161-b1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.396 INFO:rgw_multi.tests:create bucket zone=b2 name=dghqvd-39 2023-10-11T14:53:05.397 DEBUG:boto:path=/dghqvd-39/ 2023-10-11T14:53:05.397 DEBUG:boto:auth_path=/dghqvd-39/ 2023-10-11T14:53:05.397 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.398 DEBUG:boto:Path: /dghqvd-39/ 2023-10-11T14:53:05.398 DEBUG:boto:Data: 2023-10-11T14:53:05.399 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.399 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:53:05.400 DEBUG:boto:Port: 8001 2023-10-11T14:53:05.400 DEBUG:boto:Params: {} 2023-10-11T14:53:05.404 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:53:05.405 DEBUG:boto:Token: None 2023-10-11T14:53:05.406 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-39/ 2023-10-11T14:53:05.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6HTtyyEPoprhDSOINJf7F+tfx4M= 2023-10-11T14:53:05.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6HTtyyEPoprhDSOINJf7F+tfx4M=', 'Content-Length': '0'} 2023-10-11T14:53:05.441 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf8e96765638f36e-006526b6d1-115f-b2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.442 DEBUG:boto:path=/dghqvd-36/ 2023-10-11T14:53:05.443 DEBUG:boto:auth_path=/dghqvd-36/ 2023-10-11T14:53:05.443 DEBUG:boto:path=/dghqvd-36/?website 2023-10-11T14:53:05.444 DEBUG:boto:auth_path=/dghqvd-36/?website 2023-10-11T14:53:05.444 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.444 DEBUG:boto:Path: /dghqvd-36/?website 2023-10-11T14:53:05.445 DEBUG:boto:Data: index.htmlerror.html 2023-10-11T14:53:05.445 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:05.446 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.447 DEBUG:boto:Params: {} 2023-10-11T14:53:05.447 DEBUG:boto:Token: None 2023-10-11T14:53:05.448 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-36/?website 2023-10-11T14:53:05.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WEk2UdE+bspOBPSjlhh47o1b+pA= 2023-10-11T14:53:05.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WEk2UdE+bspOBPSjlhh47o1b+pA=', 'Content-Length': '241'} 2023-10-11T14:53:05.450 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx000002ade1d8d0d546106-006526b6d1-1675-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-10-11T14:53:05.451 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-40 2023-10-11T14:53:05.452 DEBUG:boto:path=/dghqvd-40/ 2023-10-11T14:53:05.452 DEBUG:boto:auth_path=/dghqvd-40/ 2023-10-11T14:53:05.453 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.453 DEBUG:boto:Path: /dghqvd-40/ 2023-10-11T14:53:05.454 DEBUG:boto:Data: 2023-10-11T14:53:05.454 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:05.455 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.455 DEBUG:boto:Params: {} 2023-10-11T14:53:05.456 DEBUG:boto:Token: None 2023-10-11T14:53:05.456 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-40/ 2023-10-11T14:53:05.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i/QwiGPuZKolV4yddfZIel5XHa4= 2023-10-11T14:53:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i/QwiGPuZKolV4yddfZIel5XHa4=', 'Content-Length': '0'} 2023-10-11T14:53:05.473 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2d26aabfd6c320b-006526b6d1-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.474 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-41 2023-10-11T14:53:05.474 DEBUG:boto:path=/dghqvd-41/ 2023-10-11T14:53:05.475 DEBUG:boto:auth_path=/dghqvd-41/ 2023-10-11T14:53:05.475 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.476 DEBUG:boto:Path: /dghqvd-41/ 2023-10-11T14:53:05.476 DEBUG:boto:Data: 2023-10-11T14:53:05.477 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:53:05.477 DEBUG:boto:Port: 8001 2023-10-11T14:53:05.478 DEBUG:boto:Params: {} 2023-10-11T14:53:05.478 DEBUG:boto:Token: None 2023-10-11T14:53:05.479 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-41/ 2023-10-11T14:53:05.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Voz2UCNy4MQcXOylF4jiYxxrJqU= 2023-10-11T14:53:05.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Voz2UCNy4MQcXOylF4jiYxxrJqU=', 'Content-Length': '0'} 2023-10-11T14:53:05.506 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000233a5f6a0db37a12-006526b6d1-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.507 INFO:rgw_multi.tests:create bucket zone=b1 name=dghqvd-42 2023-10-11T14:53:05.508 DEBUG:boto:path=/dghqvd-42/ 2023-10-11T14:53:05.508 DEBUG:boto:auth_path=/dghqvd-42/ 2023-10-11T14:53:05.508 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.509 DEBUG:boto:Path: /dghqvd-42/ 2023-10-11T14:53:05.509 DEBUG:boto:Data: 2023-10-11T14:53:05.510 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.510 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:53:05.510 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.511 DEBUG:boto:Params: {} 2023-10-11T14:53:05.511 DEBUG:boto:Token: None 2023-10-11T14:53:05.512 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-42/ 2023-10-11T14:53:05.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DHQQEWDGNlyVhLAZrUzA+41OuFg= 2023-10-11T14:53:05.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DHQQEWDGNlyVhLAZrUzA+41OuFg=', 'Content-Length': '0'} 2023-10-11T14:53:05.541 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000043c511916ef8238-006526b6d1-1161-b1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.542 INFO:rgw_multi.tests:create bucket zone=b2 name=dghqvd-43 2023-10-11T14:53:05.543 DEBUG:boto:path=/dghqvd-43/ 2023-10-11T14:53:05.543 DEBUG:boto:auth_path=/dghqvd-43/ 2023-10-11T14:53:05.543 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.544 DEBUG:boto:Path: /dghqvd-43/ 2023-10-11T14:53:05.544 DEBUG:boto:Data: 2023-10-11T14:53:05.545 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.545 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:53:05.546 DEBUG:boto:Port: 8001 2023-10-11T14:53:05.546 DEBUG:boto:Params: {} 2023-10-11T14:53:05.547 DEBUG:boto:Token: None 2023-10-11T14:53:05.547 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-43/ 2023-10-11T14:53:05.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yqEc4D2eNI8FMr1gy8frc0v/21I= 2023-10-11T14:53:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yqEc4D2eNI8FMr1gy8frc0v/21I=', 'Content-Length': '0'} 2023-10-11T14:53:05.575 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b08f2017c6ce513b-006526b6d1-115f-b2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.575 DEBUG:boto:path=/dghqvd-40/ 2023-10-11T14:53:05.576 DEBUG:boto:auth_path=/dghqvd-40/ 2023-10-11T14:53:05.577 DEBUG:boto:path=/dghqvd-40/?policy 2023-10-11T14:53:05.577 DEBUG:boto:auth_path=/dghqvd-40/?policy 2023-10-11T14:53:05.577 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.578 DEBUG:boto:Path: /dghqvd-40/?policy 2023-10-11T14:53:05.579 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-11T14:53:05.579 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:05.580 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.580 DEBUG:boto:Params: {} 2023-10-11T14:53:05.581 DEBUG:boto:Token: None 2023-10-11T14:53:05.581 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-40/?policy 2023-10-11T14:53:05.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vkAuuVDWLb4zuMNTh7kxuvxvsAw= 2023-10-11T14:53:05.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vkAuuVDWLb4zuMNTh7kxuvxvsAw=', 'Content-Length': '97'} 2023-10-11T14:53:05.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093e253921032416f-006526b6d1-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.588 DEBUG:boto:path=/dghqvd-40/ 2023-10-11T14:53:05.588 DEBUG:boto:auth_path=/dghqvd-40/ 2023-10-11T14:53:05.589 DEBUG:boto:path=/dghqvd-40/?policy 2023-10-11T14:53:05.589 DEBUG:boto:auth_path=/dghqvd-40/?policy 2023-10-11T14:53:05.589 DEBUG:boto:Method: GET 2023-10-11T14:53:05.590 DEBUG:boto:Path: /dghqvd-40/?policy 2023-10-11T14:53:05.590 DEBUG:boto:Data: 2023-10-11T14:53:05.591 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:05.591 DEBUG:boto:Port: 8000 2023-10-11T14:53:05.592 DEBUG:boto:Params: {} 2023-10-11T14:53:05.592 DEBUG:boto:Token: None 2023-10-11T14:53:05.593 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-40/?policy 2023-10-11T14:53:05.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cf9Qi1L9g+m88L2vZpOVNneb0Ns= 2023-10-11T14:53:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cf9Qi1L9g+m88L2vZpOVNneb0Ns=', 'Content-Length': '0'} 2023-10-11T14:53:05.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075cc5b8b9fec99e5-006526b6d1-1675-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 11 Oct 2023 14:53:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:05.596 DEBUG:boto:path=/dghqvd-41/ 2023-10-11T14:53:05.596 DEBUG:boto:auth_path=/dghqvd-41/ 2023-10-11T14:53:05.597 DEBUG:boto:path=/dghqvd-41/?policy 2023-10-11T14:53:05.597 DEBUG:boto:auth_path=/dghqvd-41/?policy 2023-10-11T14:53:05.598 DEBUG:boto:Method: PUT 2023-10-11T14:53:05.598 DEBUG:boto:Path: /dghqvd-41/?policy 2023-10-11T14:53:05.598 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-11T14:53:05.599 DEBUG:boto:Headers: {} 2023-10-11T14:53:05.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:53:05.600 DEBUG:boto:Port: 8001 2023-10-11T14:53:05.600 DEBUG:boto:Params: {} 2023-10-11T14:53:05.601 DEBUG:boto:Token: None 2023-10-11T14:53:05.601 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:05 GMT /dghqvd-41/?policy 2023-10-11T14:53:05.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MVD1SJm4t/sg8S7L8Jlc05TR9ZA= 2023-10-11T14:53:05.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MVD1SJm4t/sg8S7L8Jlc05TR9ZA=', 'Content-Length': '97'} 2023-10-11T14:53:06.612 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000317a283fcfecafa-006526b6d1-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:53:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:06.613 DEBUG:boto:path=/dghqvd-41/ 2023-10-11T14:53:06.613 DEBUG:boto:auth_path=/dghqvd-41/ 2023-10-11T14:53:06.614 DEBUG:boto:path=/dghqvd-41/?policy 2023-10-11T14:53:06.614 DEBUG:boto:auth_path=/dghqvd-41/?policy 2023-10-11T14:53:06.614 DEBUG:boto:Method: GET 2023-10-11T14:53:06.615 DEBUG:boto:Path: /dghqvd-41/?policy 2023-10-11T14:53:06.615 DEBUG:boto:Data: 2023-10-11T14:53:06.616 DEBUG:boto:Headers: {} 2023-10-11T14:53:06.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:53:06.616 DEBUG:boto:Port: 8001 2023-10-11T14:53:06.617 DEBUG:boto:Params: {} 2023-10-11T14:53:06.617 DEBUG:boto:Token: None 2023-10-11T14:53:06.618 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:53:06 GMT /dghqvd-41/?policy 2023-10-11T14:53:06.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DxmNtRx89JW5aFU+NLMWWB9V22Y= 2023-10-11T14:53:06.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DxmNtRx89JW5aFU+NLMWWB9V22Y=', 'Content-Length': '0'} 2023-10-11T14:53:06.620 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f239efc1aa731d07-006526b6d2-1679-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 11 Oct 2023 14:53:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:06.621 DEBUG:boto:path=/dghqvd-42/ 2023-10-11T14:53:06.621 DEBUG:boto:auth_path=/dghqvd-42/ 2023-10-11T14:53:06.622 DEBUG:boto:path=/dghqvd-42/?policy 2023-10-11T14:53:06.622 DEBUG:boto:auth_path=/dghqvd-42/?policy 2023-10-11T14:53:06.622 DEBUG:boto:Method: PUT 2023-10-11T14:53:06.623 DEBUG:boto:Path: /dghqvd-42/?policy 2023-10-11T14:53:06.623 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-11T14:53:06.624 DEBUG:boto:Headers: {} 2023-10-11T14:53:06.624 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:53:06.624 DEBUG:boto:Port: 8000 2023-10-11T14:53:06.625 DEBUG:boto:Params: {} 2023-10-11T14:53:06.625 DEBUG:boto:Token: None 2023-10-11T14:53:06.626 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:06 GMT /dghqvd-42/?policy 2023-10-11T14:53:06.626 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7cuFCb/D+jheFx+xl5BBSxGzpn4= 2023-10-11T14:53:06.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7cuFCb/D+jheFx+xl5BBSxGzpn4=', 'Content-Length': '97'} 2023-10-11T14:53:07.638 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5fc18e45b9f8b88-006526b6d2-1161-b1'), ('Date', 'Wed, 11 Oct 2023 14:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:07.639 DEBUG:boto:path=/dghqvd-42/ 2023-10-11T14:53:07.639 DEBUG:boto:auth_path=/dghqvd-42/ 2023-10-11T14:53:07.640 DEBUG:boto:path=/dghqvd-42/?policy 2023-10-11T14:53:07.640 DEBUG:boto:auth_path=/dghqvd-42/?policy 2023-10-11T14:53:07.640 DEBUG:boto:Method: GET 2023-10-11T14:53:07.641 DEBUG:boto:Path: /dghqvd-42/?policy 2023-10-11T14:53:07.641 DEBUG:boto:Data: 2023-10-11T14:53:07.642 DEBUG:boto:Headers: {} 2023-10-11T14:53:07.642 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8000 2023-10-11T14:53:07.642 DEBUG:boto:Port: 8000 2023-10-11T14:53:07.643 DEBUG:boto:Params: {} 2023-10-11T14:53:07.643 DEBUG:boto:Token: None 2023-10-11T14:53:07.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:53:07 GMT /dghqvd-42/?policy 2023-10-11T14:53:07.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Eax3p4z30GSbWZEJjl3eUUtJORc= 2023-10-11T14:53:07.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Eax3p4z30GSbWZEJjl3eUUtJORc=', 'Content-Length': '0'} 2023-10-11T14:53:07.646 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3230aacd69ab402-006526b6d3-1161-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 11 Oct 2023 14:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:07.647 DEBUG:boto:path=/dghqvd-43/ 2023-10-11T14:53:07.647 DEBUG:boto:auth_path=/dghqvd-43/ 2023-10-11T14:53:07.647 DEBUG:boto:path=/dghqvd-43/?policy 2023-10-11T14:53:07.648 DEBUG:boto:auth_path=/dghqvd-43/?policy 2023-10-11T14:53:07.648 DEBUG:boto:Method: PUT 2023-10-11T14:53:07.648 DEBUG:boto:Path: /dghqvd-43/?policy 2023-10-11T14:53:07.649 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-10-11T14:53:07.649 DEBUG:boto:Headers: {} 2023-10-11T14:53:07.650 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:53:07.650 DEBUG:boto:Port: 8001 2023-10-11T14:53:07.650 DEBUG:boto:Params: {} 2023-10-11T14:53:07.651 DEBUG:boto:Token: None 2023-10-11T14:53:07.651 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:07 GMT /dghqvd-43/?policy 2023-10-11T14:53:07.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6nAcu52+WjU/OAE96sVnrmW+/PI= 2023-10-11T14:53:07.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6nAcu52+WjU/OAE96sVnrmW+/PI=', 'Content-Length': '97'} 2023-10-11T14:53:08.663 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004f234ef8f19f96bf-006526b6d3-115f-b2'), ('Date', 'Wed, 11 Oct 2023 14:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:08.664 DEBUG:boto:path=/dghqvd-43/ 2023-10-11T14:53:08.664 DEBUG:boto:auth_path=/dghqvd-43/ 2023-10-11T14:53:08.665 DEBUG:boto:path=/dghqvd-43/?policy 2023-10-11T14:53:08.665 DEBUG:boto:auth_path=/dghqvd-43/?policy 2023-10-11T14:53:08.666 DEBUG:boto:Method: GET 2023-10-11T14:53:08.666 DEBUG:boto:Path: /dghqvd-43/?policy 2023-10-11T14:53:08.667 DEBUG:boto:Data: 2023-10-11T14:53:08.667 DEBUG:boto:Headers: {} 2023-10-11T14:53:08.667 DEBUG:boto:Host: smithi172.front.sepia.ceph.com:8001 2023-10-11T14:53:08.668 DEBUG:boto:Port: 8001 2023-10-11T14:53:08.668 DEBUG:boto:Params: {} 2023-10-11T14:53:08.669 DEBUG:boto:Token: None 2023-10-11T14:53:08.669 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:53:08 GMT /dghqvd-43/?policy 2023-10-11T14:53:08.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WVIyHS+7gbPXgBTaYCw18T9MXpM= 2023-10-11T14:53:08.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WVIyHS+7gbPXgBTaYCw18T9MXpM=', 'Content-Length': '0'} 2023-10-11T14:53:08.672 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000053ebfd7b84eed4e0-006526b6d4-115f-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 11 Oct 2023 14:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:08.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-10-11T14:53:08.673 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-44 2023-10-11T14:53:08.674 DEBUG:boto:path=/dghqvd-44/ 2023-10-11T14:53:08.674 DEBUG:boto:auth_path=/dghqvd-44/ 2023-10-11T14:53:08.674 DEBUG:boto:Method: PUT 2023-10-11T14:53:08.675 DEBUG:boto:Path: /dghqvd-44/ 2023-10-11T14:53:08.675 DEBUG:boto:Data: 2023-10-11T14:53:08.676 DEBUG:boto:Headers: {} 2023-10-11T14:53:08.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:53:08.676 DEBUG:boto:Port: 8000 2023-10-11T14:53:08.677 DEBUG:boto:Params: {} 2023-10-11T14:53:08.677 DEBUG:boto:Token: None 2023-10-11T14:53:08.678 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:08 GMT /dghqvd-44/ 2023-10-11T14:53:08.678 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eloXxsdO3QxCYwwMv9QVoI6zXP8= 2023-10-11T14:53:08.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eloXxsdO3QxCYwwMv9QVoI6zXP8=', 'Content-Length': '0'} 2023-10-11T14:53:08.696 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e0488cc0f502e955-006526b6d4-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:08.696 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-45 2023-10-11T14:53:08.696 DEBUG:boto:path=/dghqvd-45/ 2023-10-11T14:53:08.697 DEBUG:boto:auth_path=/dghqvd-45/ 2023-10-11T14:53:08.697 DEBUG:boto:Method: PUT 2023-10-11T14:53:08.698 DEBUG:boto:Path: /dghqvd-45/ 2023-10-11T14:53:08.698 DEBUG:boto:Data: 2023-10-11T14:53:08.699 DEBUG:boto:Headers: {} 2023-10-11T14:53:08.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:53:08.699 DEBUG:boto:Port: 8001 2023-10-11T14:53:08.700 DEBUG:boto:Params: {} 2023-10-11T14:53:08.700 DEBUG:boto:Token: None 2023-10-11T14:53:08.701 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:53:08 GMT /dghqvd-45/ 2023-10-11T14:53:08.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0j+ag9XVvABdU8YyOOnxWZMmRAQ= 2023-10-11T14:53:08.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0j+ag9XVvABdU8YyOOnxWZMmRAQ=', 'Content-Length': '0'} 2023-10-11T14:53:08.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050320203c57dfa03-006526b6d4-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:53:08.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:08.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:08.730 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:08.786 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:08.786 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:08.818 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:53:08.830 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:53:08.831 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.831 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:08.831 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:53:08.831 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.832 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.832 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:08.832 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:53:08.832 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.832 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.833 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:08.833 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:53:08.833 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.833 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.834 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035988.716816_315.1", 2023-10-11T14:53:08.834 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:53:08.716816Z" 2023-10-11T14:53:08.834 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.834 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:53:08.835 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:08.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:53:08.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:08.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:08.837 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:08.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:08.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:08.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:08.912+0000 7f2a12a5f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:08.916 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:08.913+0000 7f2a12a5f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:08.927 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:08.927 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:08.928 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:08.928 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:08.928 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:08.929 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:08.929 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:08.929 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.929 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:08.929 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.930 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:08.930 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:08.930 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:08.931 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:08.931 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:08.931 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:08.931 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:08.931 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:08.932 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:08.932 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.932 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.933 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:08.933 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:08.933 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:08.933 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:08Z" 2023-10-11T14:53:08.944 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:08.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:08Z" } 2023-10-11T14:53:08.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:08.946 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:08.947 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:08.947 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:13.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:13.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:13.949 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:14.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:14.035+0000 7f43d0ba8780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:14.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:14.036+0000 7f43d0ba8780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:14.051 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:14.052 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:14.052 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:14.052 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:14.053 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:14.053 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:14.053 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:14.053 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.054 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:14.054 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:14.054 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:14.054 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:14.054 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:14.055 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:14.055 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:14.055 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:14.055 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:14.056 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:14.056 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:14.056 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:14.056 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.056 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:14.057 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:14.057 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:14.057 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:14.057 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:14.057 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:14.058 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:14.058 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:14.058 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:14.058 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:14.061 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:14.061 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.061 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:14Z" 2023-10-11T14:53:14.068 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:14.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:14Z" } 2023-10-11T14:53:14.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:14.070 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:14.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:14.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:19.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:19.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:19.073 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:19.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:19.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:19.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:19.159+0000 7fa763b12780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:19.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:19.160+0000 7fa763b12780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:19.175 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:19.176 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:19.176 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:19.176 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:19.177 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:19.177 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:19.177 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:19.177 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.177 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:19.178 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:19.178 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:19.178 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:19.178 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:19.179 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:19.179 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:19.179 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:19.179 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:19.179 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:19.180 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:19.180 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:19.180 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.180 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:19.181 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:19.181 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:19.181 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:19.181 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:19.181 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:19.182 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:19.182 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:19.182 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:19.182 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:19.183 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:19.183 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.183 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:19.183 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:19.184 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:19.184 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:19.184 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:19.184 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:19.184 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:19.185 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:19.185 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:19.185 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:19.185 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:19.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.186 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:19.186 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:19.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:19.187 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:19.187 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:19.187 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:19.187 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:19.190 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:19.190 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:19Z" 2023-10-11T14:53:19.190 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:19.191 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:19Z" } 2023-10-11T14:53:19.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:19.192 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:19.193 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:19.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:24.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:24.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:24.196 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:24.254 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:24.255 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:24.283+0000 7f8b1e701780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:24.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:24.284+0000 7f8b1e701780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:24.299 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:24.299 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:24.300 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:24.300 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:24.300 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:24.300 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:24.300 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:24.301 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.301 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:24.301 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:24.301 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:24.301 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:24.302 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:24.302 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:24.302 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:24.302 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:24.303 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:24.303 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:24.303 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:24.303 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:24.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.304 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:24.304 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:24.304 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:24.304 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:24.305 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:24.305 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:24.305 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:24.305 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:24.306 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:24.306 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:24.306 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:24.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:24.307 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:24.307 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:24.307 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:24.307 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:24.308 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:24.308 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:24.308 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:24.308 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:24.308 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:24.309 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:24.309 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.309 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:24.309 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:24.310 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:24.311 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:24.311 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:24.311 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:24.311 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:24.311 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:24.312 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:24.312 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.312 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:24.312 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:24.313 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:24.313 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:24.313 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:24Z" 2023-10-11T14:53:24.313 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:24.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:24Z" } 2023-10-11T14:53:24.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:24.315 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:24.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:24.316 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:29.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:29.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:29.320 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:29.375 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:29.376 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:29.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:29.402+0000 7fb0baaf4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:29.406 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:29.403+0000 7fb0baaf4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:29.418 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:29.418 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:29.419 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:29.419 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:29.419 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:29.419 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:29.419 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:29.420 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.420 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:29.420 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:29.420 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:29.420 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:29.421 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:29.421 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:29.421 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:29.421 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:29.422 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:29.422 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:29.422 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:29.422 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:29.422 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.423 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:29.423 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:29.423 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:29.423 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:29.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:29.424 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:29.424 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:29.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:29.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:29.425 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:29.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:29.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:29.426 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:29.426 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:29.426 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:29.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:29.427 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:29.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:29.427 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:29.427 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:29.428 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:29.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:29.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.428 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:29.428 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:29.429 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:29.429 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:29.429 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:29.429 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:29.429 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:29.430 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:29.430 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:29.430 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:29.430 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:29.430 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:29.431 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:29.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.431 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:29.431 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:29.432 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:29.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:29.432 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:29Z" 2023-10-11T14:53:29.432 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:29.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:29Z" } 2023-10-11T14:53:29.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:29.434 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:29.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:29.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:34.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:34.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:34.437 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:34.493 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:34.493 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:34.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:34.523+0000 7f22c035f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:34.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:34.524+0000 7f22c035f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:34.539 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:34.539 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:34.539 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:34.539 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:34.540 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:34.540 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:34.540 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:34.540 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.541 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:34.541 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:34.541 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:34.541 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:34.541 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:34.542 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:34.542 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:34.542 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:34.542 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:34.543 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:34.543 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:34.543 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:34.543 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.543 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:34.544 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:34.544 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:34.544 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:34.544 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:34.545 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:34.545 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:34.545 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:34.545 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:34.546 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:34.546 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:34.546 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.546 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:34.546 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:34.547 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:34.547 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:34.547 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:34.547 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:34.548 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:34.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:34.548 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:34.548 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:34.548 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:34.549 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.549 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:34.549 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:34.549 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:34.549 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:34.550 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:34.550 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:34.550 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:34.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:34.551 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:34.551 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:34.551 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:34.551 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:34.551 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:34.552 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.552 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:34.552 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:34.552 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:34.552 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:34.553 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:34Z" 2023-10-11T14:53:34.553 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:34.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:34Z" } 2023-10-11T14:53:34.555 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:34.555 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:34.556 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:34.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:39.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:39.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:39.559 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:39.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:39.648 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:39.645+0000 7f744199d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:39.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:39.646+0000 7f744199d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:39.661 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:39.662 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:39.662 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:39.662 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:39.662 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:39.662 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:39.663 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:39.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.663 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:39.663 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:39.664 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:39.664 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:39.664 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:39.664 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:39.664 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:39.665 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:39.665 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:39.665 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:39.665 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:39.665 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:39.666 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.666 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:39.666 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:39.666 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:39.667 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:39.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:39.667 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:39.667 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:39.668 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:39.668 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:39.668 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:39.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:39.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.669 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:39.669 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:39.669 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:39.669 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:39.670 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:39.670 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:39.670 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:39.670 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:39.671 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:39.671 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:39.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:39.671 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.672 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:39.672 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:39.672 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:39.672 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:39.673 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:39.673 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:39.673 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:39.673 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:39.673 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:39.674 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:39.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:39.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:39.674 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:39.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.675 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:39.675 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:39.675 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:39.675 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:39.676 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:39Z" 2023-10-11T14:53:39.676 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:39.677 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:39Z" } 2023-10-11T14:53:39.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:39.678 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:39.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:39.679 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:44.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:44.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:44.681 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:44.739 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:44.740 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:44.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:44.768+0000 7f102c0c1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:44.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:44.769+0000 7f102c0c1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:44.785 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:44.785 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:44.786 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:44.786 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:44.786 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:44.786 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:44.787 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:44.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.787 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:44.787 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:44.789 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:44.790 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.790 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:44.790 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:44.790 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:44.791 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:44.791 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:44.791 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:44.791 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:44.791 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:44.792 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:44.792 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:44.792 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:44.792 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.793 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:44.793 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:44.793 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:44.793 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:44.793 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:44.794 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:44.794 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:44.794 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:44.794 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:44.795 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:44.796 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:44.796 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:44.796 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:44.796 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:44.796 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:44.797 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:44.797 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:44.797 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:44.797 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:44.798 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:44.798 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:44.798 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.798 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:44.798 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:44.799 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:44.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:44.799 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:44Z" 2023-10-11T14:53:44.799 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:44.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:44Z" } 2023-10-11T14:53:44.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:44.802 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:44.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:44.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:49.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:49.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:49.805 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:49.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:49.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:49.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:49.889+0000 7f3dc0945780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:49.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:49.890+0000 7f3dc0945780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:49.907 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:49.907 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:49.907 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:49.907 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:49.908 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:49.908 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:49.908 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:49.908 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.909 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:49.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:49.909 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:49.910 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:49.910 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:49.910 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:49.910 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:49.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:49.911 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:49.911 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:49.911 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:49.912 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:49.912 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.912 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:49.912 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:49.912 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:49.913 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:49.913 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:49.913 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:49.913 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:49.914 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:49.914 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:49.914 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:49.914 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:49.915 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.915 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:49.916 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:49.916 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:49.916 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:49.916 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:49.917 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:49.917 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:49.917 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:49.917 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:49.918 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:49.918 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:49.918 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.918 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:49.918 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:49.919 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:49.919 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:49.919 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:49.919 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:49.920 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:49.920 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:49.920 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:49.920 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:49.921 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:49.921 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:49.921 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:49.921 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.921 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:49.922 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:49.922 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:49.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:49.922 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:49Z" 2023-10-11T14:53:49.923 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:49.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:49Z" } 2023-10-11T14:53:49.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:49.925 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:49.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:49.926 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:53:54.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:54.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:53:54.928 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:53:54.984 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:53:54.985 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:53:55.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:55.012+0000 7f66c7c85780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:55.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:53:55.013+0000 7f66c7c85780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:53:55.029 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:53:55.030 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:53:55.030 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:53:55.030 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:53:55.030 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:53:55.031 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:53:55.031 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:55.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.031 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:53:55.031 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:55.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:53:55.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:55.032 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:55.032 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:53:55.033 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:55.033 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:53:55.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:55.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:53:55.033 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:55.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:55.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:55.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:53:55.034 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:55.035 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:55.035 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:53:55.035 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:55.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:55.036 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:55.036 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:53:55.036 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:55.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:55.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.037 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:55.037 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:53:55.037 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:55.037 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:55.037 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:53:55.038 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:55.038 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:55.038 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:55.038 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:53:55.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:55.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:55.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:53:55.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:53:55.039 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:53:55.040 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:53:55.040 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:53:55.040 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:53:55.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:53:55.041 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:53:55.041 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:53:55.041 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:53:55.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:55.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:53:55.042 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:53:55.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.042 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:53:55.042 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:53:55.042 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:53:55.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:53:55.043 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:53:55Z" 2023-10-11T14:53:55.043 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:53:55.044 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:53:55Z" } 2023-10-11T14:53:55.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:53:55.045 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:53:55.045 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:53:55.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:00.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:00.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:00.048 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:00.106 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:00.107 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:00.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:00.135+0000 7f7e31e27780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:00.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:00.136+0000 7f7e31e27780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:00.151 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:00.152 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:00.152 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:00.152 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:00.152 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:00.153 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:00.153 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:00.153 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.153 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:00.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:00.154 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:00.154 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:00.154 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:00.154 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:00.155 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:00.156 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:00.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:00.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:00.156 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:00.157 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:00.157 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:00.157 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:00.157 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:00.158 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:00.158 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:00.158 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:00.158 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:00.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.159 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:00.159 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:00.159 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:00.159 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:00.160 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:00.160 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:00.160 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:00.160 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:00.160 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:00.161 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:00.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:00.161 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.161 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:00.161 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:00.162 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:00.162 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:00.162 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:00.162 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:00.162 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:00.163 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:00.163 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:00.163 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:00.163 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:00.164 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:00.164 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:00.164 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.164 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:00.164 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:00.165 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:00.165 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:00.165 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:00Z" 2023-10-11T14:54:00.165 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:00.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:00Z" } 2023-10-11T14:54:00.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:00.167 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:00.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:00.168 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:05.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:05.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:05.170 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:05.228 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:05.229 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:05.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:05.255+0000 7f469b24a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:05.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:05.256+0000 7f469b24a780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:05.272 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:05.273 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:05.273 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:05.273 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:05.273 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:05.274 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:05.274 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:05.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.274 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:05.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:05.275 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:05.275 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:05.275 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:05.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:05.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:05.276 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:05.276 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:05.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:05.277 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:05.277 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:05.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.277 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:05.278 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:05.278 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:05.278 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:05.278 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:05.278 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:05.279 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:05.279 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:05.279 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:05.279 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:05.280 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:05.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.280 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:05.280 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:05.280 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:05.281 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:05.281 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:05.281 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:05.281 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:05.282 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:05.282 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:05.282 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:05.282 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:05.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.283 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:05.283 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:05.283 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:05.283 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:05.284 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:05.284 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:05.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:05.284 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:05.285 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:05.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.286 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:05.286 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:05.286 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:05.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:05.287 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:05Z" 2023-10-11T14:54:05.287 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:05.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:05Z" } 2023-10-11T14:54:05.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:05.289 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:05.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:05.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:10.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:10.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:10.292 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:10.349 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:10.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:10.377+0000 7f8939899780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:10.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:10.379+0000 7f8939899780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:10.395 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:10.396 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:10.396 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:10.396 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:10.396 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:10.396 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:10.397 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:10.397 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.397 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:10.397 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:10.397 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:10.398 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:10.398 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:10.398 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:10.398 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:10.398 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:10.399 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:10.399 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:10.399 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:10.399 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:10.399 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.400 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:10.400 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:10.400 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:10.400 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:10.400 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:10.401 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:10.402 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.402 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:10.402 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:10.402 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:10.402 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:10.403 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:10.403 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:10.403 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:10.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:10.403 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:10.404 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:10.404 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:10.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:10.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:10.405 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:10.405 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:10.405 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:10.405 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:10.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:10.406 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:10.406 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:10.406 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:10.406 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:10.407 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:10.407 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:10.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.407 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:10.407 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:10.408 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:10.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:10.408 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:10Z" 2023-10-11T14:54:10.408 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:10.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:10Z" } 2023-10-11T14:54:10.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:10.410 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:10.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:10.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:15.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:15.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:15.413 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:15.470 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:15.471 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:15.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:15.498+0000 7fde2393f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:15.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:15.499+0000 7fde2393f780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:15.515 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:15.515 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:15.515 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:15.516 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:15.516 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:15.516 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:15.516 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:15.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.517 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:15.517 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:15.517 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:15.517 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:15.518 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:15.518 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:15.518 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:15.518 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:15.518 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:15.519 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:15.519 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:15.519 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:15.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:15.520 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:15.520 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:15.520 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:15.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:15.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:15.521 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:15.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:15.521 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:15.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:15.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:15.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:15.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:15.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:15.523 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:15.524 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:15.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:15.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.524 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:15.524 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:15.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:15.525 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:15.525 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:15.525 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:15.525 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:15.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:15.526 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:15.526 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:15.526 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:15.527 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:15.528 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:15.528 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:15.528 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:15Z" 2023-10-11T14:54:15.528 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:15.529 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:15Z" } 2023-10-11T14:54:15.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:15.530 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:15.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:15.531 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:20.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:20.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:20.533 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:20.590 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:20.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:20.620+0000 7fa358c43780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:20.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:20.621+0000 7fa358c43780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:20.636 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:20.637 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:20.637 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:20.637 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:20.637 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:20.637 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:20.638 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:20.638 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.638 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:20.638 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:20.639 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:20.639 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:20.639 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:20.639 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:20.639 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:20.640 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:20.640 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:20.640 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:20.640 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:20.640 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:20.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:20.641 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:20.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:20.641 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:20.642 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:20.642 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:20.642 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:20.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:20.642 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:20.643 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:20.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:20.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.644 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:20.644 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:20.644 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:20.644 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:20.645 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:20.645 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:20.645 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:20.645 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:20.645 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.646 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:20.647 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:20.648 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.649 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:20.650 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:20Z" 2023-10-11T14:54:20.651 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:20.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:20Z" } 2023-10-11T14:54:20.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:20.653 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:20.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:20.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:25.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:25.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:25.656 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:25.713 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:25.713 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:25.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:25.741+0000 7f72f06ab780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:25.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:25.742+0000 7f72f06ab780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:25.758 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:25.758 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:25.758 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:25.758 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:25.759 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:25.759 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:25.759 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:25.759 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.760 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:25.760 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:25.760 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:25.760 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:25.760 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:25.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:25.761 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:25.761 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:25.761 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:25.762 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:25.762 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:25.762 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:25.762 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.762 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:25.763 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:25.763 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:25.763 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:25.763 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:25.763 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:25.764 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:25.764 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:25.764 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:25.764 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:25.765 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:25.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.765 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:25.765 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:25.766 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:25.766 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:25.766 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:25.766 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:25.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:25.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:25.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:25.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:25.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:25.772 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:25Z" 2023-10-11T14:54:25.773 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:25.774 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:25Z" } 2023-10-11T14:54:25.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:25.775 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:25.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:25.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:30.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:30.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:30.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:30.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:30.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:30.863+0000 7f3e3d7d1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:30.867 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:30.864+0000 7f3e3d7d1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:30.880 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:30.881 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:30.881 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:30.881 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:30.882 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:30.882 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:30.882 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:30.882 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.883 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:30.883 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:30.883 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:30.883 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:30.883 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:30.884 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:30.884 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:30.884 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:30.884 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:30.884 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:30.885 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:30.885 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:30.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.885 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:30.886 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:30.886 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:30.886 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:30.886 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:30.886 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:30.887 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:30.887 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:30.887 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:30.887 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:30.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:30.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.888 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:30.888 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:30.888 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:30.888 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:30.889 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:30.889 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:30.889 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:30.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:30.890 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:30.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:30.890 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:30.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:30.891 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:30.891 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:30.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:30.891 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:30.891 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:30.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:30.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:30.892 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:30.892 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:30.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:30.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:30.893 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:30.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.893 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:30.894 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:30.894 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:30.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:30.894 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:30Z" 2023-10-11T14:54:30.894 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:30.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:30Z" } 2023-10-11T14:54:30.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:30.897 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:30.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:30.897 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:35.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:35.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:35.900 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:35.957 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:35.958 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:35.989 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:35.986+0000 7f984127c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:35.990 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:35.987+0000 7f984127c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:36.002 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:36.003 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:36.003 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:36.003 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:36.003 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:36.004 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:36.004 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:36.004 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.004 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:36.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:36.005 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:36.005 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:36.005 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:36.005 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:36.006 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:36.006 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:36.006 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:36.006 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:36.007 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:36.007 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:36.007 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.007 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:36.008 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:36.008 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:36.008 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:36.008 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:36.008 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:36.009 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:36.009 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:36.009 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:36.009 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:36.010 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:36.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.010 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:36.010 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:36.010 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:36.011 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:36.011 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:36.011 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:36.011 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:36.012 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:36.012 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:36.012 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:36.012 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:36.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.013 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:36.013 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:36.013 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:36.013 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:36.013 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:36.014 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:36.014 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:36.014 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:36.014 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:36.015 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:36.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:36.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:36.015 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:36.015 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.016 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:36.016 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:36.016 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:36.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:36.016 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:35Z" 2023-10-11T14:54:36.017 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:36.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:35Z" } 2023-10-11T14:54:36.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:36.019 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:36.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:36.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:41.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:41.022 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:41.079 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:41.080 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:41.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:41.108+0000 7fc909dfc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:41.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:41.109+0000 7fc909dfc780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:41.124 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:41.125 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:41.125 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:41.125 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:41.125 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:41.125 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:41.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:41.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.126 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:41.126 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:41.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:41.127 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:41.127 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:41.127 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:41.127 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:41.128 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:41.128 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:41.128 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:41.128 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:41.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:41.129 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.129 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:41.129 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:41.129 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:41.130 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:41.130 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:41.130 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:41.130 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:41.131 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:41.131 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:41.131 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:41.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:41.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:41.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:41.132 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:41.132 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:41.133 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:41.133 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:41.133 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:41.133 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:41.133 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:41.134 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:41.134 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:41.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:41.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:41.135 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:41.135 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:41.135 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:41.135 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:41.136 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:41.136 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:41.136 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:41.136 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:41.136 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:41.137 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:41.137 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:41.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.137 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:41.137 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:41.138 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:41.138 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:41.138 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:41Z" 2023-10-11T14:54:41.138 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:41.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:41Z" } 2023-10-11T14:54:41.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:41.140 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:41.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:41.141 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:46.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:46.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:46.144 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:46.202 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:46.203 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:46.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:46.232+0000 7fdcbafa5780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:46.236 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:46.234+0000 7fdcbafa5780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:46.249 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:46.250 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:46.250 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:46.250 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:46.251 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:46.251 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:46.251 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:46.251 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.251 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:46.252 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:46.252 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:46.252 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:46.252 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:46.253 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:46.253 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:46.253 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:46.253 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:46.253 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:46.254 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:46.254 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:46.254 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.254 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:46.255 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:46.255 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:46.255 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:46.255 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:46.255 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:46.256 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:46.256 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:46.256 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:46.256 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:46.257 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:46.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:46.257 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:46.258 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:46.258 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:46.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:46.258 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:46.258 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:46.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:46.259 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:46.259 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:46.259 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:46.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:46.260 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:46.260 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:46.260 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:46.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:46.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:46.261 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:46.261 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:46.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:46.262 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:46.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:46.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:46.262 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:46.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.263 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:46.263 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:46.263 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:46.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:46.264 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:46Z" 2023-10-11T14:54:46.264 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:46.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:46Z" } 2023-10-11T14:54:46.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:46.266 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:46.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:46.267 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:51.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:51.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:51.269 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:51.326 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:51.327 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:51.360 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:51.357+0000 7f48f52b4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:51.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:51.358+0000 7f48f52b4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:51.373 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:51.373 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:51.374 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:51.374 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:51.374 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:51.374 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:51.374 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:51.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.375 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:51.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:51.375 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:51.376 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:51.376 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:51.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:51.376 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:51.376 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:51.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:51.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:51.377 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:51.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:51.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:51.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:51.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:51.379 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:51.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:51.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:51.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:51.380 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:51.380 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:51.380 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:51.381 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:51.381 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.381 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:51.381 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:51.381 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:51.382 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:51.382 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:51.382 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:51.382 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:51.383 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:51.383 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:51.383 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:51.383 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:51.383 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.384 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:51.384 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:51.384 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:51.384 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:51.385 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:51.385 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:51.385 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:51.385 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:51.386 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:51.386 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:51.386 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:51.386 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:51.387 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:51.387 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.387 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:51.387 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:51.387 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:51.388 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:51.388 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:51Z" 2023-10-11T14:54:51.388 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:51.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:51Z" } 2023-10-11T14:54:51.390 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:51.391 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:51.391 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:51.391 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:54:56.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:56.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:54:56.394 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:54:56.452 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:54:56.452 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:54:56.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:56.481+0000 7fb99d7c4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:56.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:54:56.483+0000 7fb99d7c4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:54:56.497 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:54:56.498 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:54:56.498 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:54:56.498 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:54:56.499 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:54:56.499 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:54:56.499 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:56.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.499 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:54:56.500 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:56.500 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:54:56.500 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:56.500 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:56.501 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:54:56.501 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:56.501 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:54:56.501 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:56.501 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:54:56.502 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:56.502 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:56.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.502 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:56.503 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:54:56.503 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:56.503 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:56.503 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:54:56.503 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:56.504 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:56.504 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:56.504 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:54:56.504 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:56.504 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:56.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:56.505 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:54:56.505 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:56.506 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:56.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:54:56.506 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:56.506 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:56.506 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:56.507 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:54:56.507 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:56.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:56.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:54:56.508 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:54:56.508 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:54:56.508 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:54:56.508 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:54:56.509 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:54:56.509 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:54:56.509 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:54:56.509 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:54:56.509 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:54:56.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:56.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:54:56.510 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:54:56.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.511 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:54:56.511 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:54:56.511 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:54:56.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:54:56.511 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:54:56Z" 2023-10-11T14:54:56.512 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:54:56.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:54:56Z" } 2023-10-11T14:54:56.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:54:56.514 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:54:56.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:54:56.514 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:55:01.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:01.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:01.517 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:01.574 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:01.575 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:01.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:01.604+0000 7f5944828780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:01.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:01.605+0000 7f5944828780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:01.620 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:01.621 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:01.621 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:01.621 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:01.621 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:01.622 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:01.622 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:01.622 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.622 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:01.623 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:01.623 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:01.623 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:01.623 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:01.623 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:01.624 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:01.624 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:01.624 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:01.624 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:01.625 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:01.625 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:01.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:01.625 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:01.626 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:01.626 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:01.626 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:01.626 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:01.627 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:01.627 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:01.627 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:01.627 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:01.627 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:01.628 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:01.628 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:01.628 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:01.629 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:01.629 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:01.629 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:01.629 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:01.629 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:01.630 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:01.630 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:01.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:01.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.631 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:01.631 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:01.631 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:01.631 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:01.631 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:55:01.632 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:01.632 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:01.632 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:01.632 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:55:01.633 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:01.633 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:01.633 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:01.633 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:01.633 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.634 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:01.635 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:01.635 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:01.635 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:01.635 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:01Z" 2023-10-11T14:55:01.636 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:01.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:01Z" } 2023-10-11T14:55:01.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:55:01.638 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:01.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:55:01.639 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:55:06.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:06.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:06.642 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:06.697 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:06.697 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:06.724+0000 7f673d2b4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:06.725+0000 7f673d2b4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:06.740 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:06.740 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:06.740 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:06.741 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:06.741 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:06.741 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:06.741 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:06.742 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.742 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:06.742 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:06.742 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:06.743 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:06.743 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:06.743 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:06.743 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:06.743 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:06.744 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:06.744 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:06.744 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:06.744 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:06.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.745 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:06.745 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:06.745 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:06.745 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:06.746 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:06.746 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:06.746 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:06.746 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:06.747 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:06.747 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:06.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:06.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.748 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:06.748 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:06.748 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:06.748 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:06.748 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:06.749 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:06.749 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:06.749 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:06.749 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:06.749 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:06.750 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:06.750 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.750 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:06.750 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:06.751 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:06.751 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:06.751 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035733.545819_204.1", 2023-10-11T14:55:06.751 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:06.752 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:06.752 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:06.752 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:48:53.545819Z", 2023-10-11T14:55:06.752 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:06.752 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:06.753 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:06.753 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:06.753 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.753 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:06.753 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:06.754 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:06.754 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:06.754 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:06Z" 2023-10-11T14:55:06.754 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:06.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035733.545819_204.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:48:53.545819Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:06Z" } 2023-10-11T14:55:06.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035733.545819_204.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:48:53.545819Z', 'realm_epoch': 2}}] 2023-10-11T14:55:06.756 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:06.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035733.545819_204.1'} 2023-10-11T14:55:06.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035733.545819_204.1 2023-10-11T14:55:11.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:11.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:11.761 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:11.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:11.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:11.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:11.844+0000 7f38745e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:11.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:11.845+0000 7f38745e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:11.859 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:11.859 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:11.859 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:11.860 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:11.860 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:11.860 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:11.860 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:11.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.861 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:11.861 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:11.861 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:11.861 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:11.862 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:11.862 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:11.862 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:11.862 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:11.863 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:11.863 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:11.863 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:11.863 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:11.863 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.864 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:11.864 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:11.864 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:11.864 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:11.864 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:11.865 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:11.865 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:11.865 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:11.865 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:11.865 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:11.866 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:11.866 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:11.866 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:11.866 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:11.867 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:11.867 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:11.867 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:11.867 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:11.867 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:11.868 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:11.868 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:11.868 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:11.869 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.869 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:11.869 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:11.869 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:11.869 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:11.870 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035985.336196_277.1", 2023-10-11T14:55:11.870 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:11.870 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:11.870 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:11.870 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:05.336196Z", 2023-10-11T14:55:11.871 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:11.871 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:11.871 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:11.871 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:11.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.872 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:11.872 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:11.872 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:11.872 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:11.872 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:11Z" 2023-10-11T14:55:11.873 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:11.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035985.336196_277.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:05.336196Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:11Z" } 2023-10-11T14:55:11.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035985.336196_277.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:05.336196Z', 'realm_epoch': 2}}] 2023-10-11T14:55:11.875 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:11.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035985.336196_277.1'} 2023-10-11T14:55:11.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035985.336196_277.1 2023-10-11T14:55:16.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:16.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:16.878 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:16.936 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:16.936 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:16.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:16.963+0000 7f2964b1d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:16.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:16.964+0000 7f2964b1d780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:16.978 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:16.978 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:16.979 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:16.979 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:16.979 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:16.979 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:16.979 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:16.980 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.980 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:16.980 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:16.980 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:16.981 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:16.981 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:16.981 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:16.981 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:16.981 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:16.982 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:16.982 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:16.982 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:16.982 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:16.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.983 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:16.983 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:16.983 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:16.983 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:16.984 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:16.984 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:16.984 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:16.984 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:16.985 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:16.985 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:16.985 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:16.985 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.985 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:16.986 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:16.986 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:16.986 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:16.986 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:16.986 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:16.987 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:16.987 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:16.987 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:16.987 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:16.988 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:16.988 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.988 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:16.988 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:16.988 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:16.989 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:16.989 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035985.326708_274.1", 2023-10-11T14:55:16.989 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:16.989 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:16.989 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:16.990 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:05.326708Z", 2023-10-11T14:55:16.990 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:16.990 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:16.990 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:16.990 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:16.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.991 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:16.991 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:16.991 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:16.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:16.992 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:16Z" 2023-10-11T14:55:16.992 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:16.993 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035985.326708_274.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:05.326708Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:16Z" } 2023-10-11T14:55:16.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035985.326708_274.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:05.326708Z', 'realm_epoch': 2}}] 2023-10-11T14:55:16.994 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:16.994 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035985.326708_274.1'} 2023-10-11T14:55:16.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035985.326708_274.1 2023-10-11T14:55:21.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:21.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:21.997 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:22.049 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:22.050 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:22.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:22.076+0000 7f10e8491780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:22.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:22.077+0000 7f10e8491780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:22.091 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:22.092 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:22.092 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:22.092 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:22.092 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:22.092 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:22.093 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:22.093 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.093 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:22.093 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:22.093 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:22.094 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:22.094 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:22.094 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:22.094 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:22.095 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:22.095 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:22.095 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:22.095 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:22.095 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:22.096 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.096 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:22.096 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:22.096 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:22.097 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:22.097 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:22.097 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:22.097 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:22.097 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:22.098 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:22.098 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:22.098 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:22.098 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.098 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:22.099 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:22.099 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:22.099 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:22.099 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:22.100 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:22.100 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:22.100 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:22.100 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:22.100 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:22.101 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:22.101 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.101 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:22.101 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:22.101 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:22.102 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:22.102 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035985.418788_278.1", 2023-10-11T14:55:22.102 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:22.102 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:22.102 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:22.103 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:05.418788Z", 2023-10-11T14:55:22.103 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:22.103 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:22.103 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:22.103 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:22.104 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.104 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:22.104 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:22.104 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:22.104 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:22.105 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:22Z" 2023-10-11T14:55:22.105 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:22.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035985.418788_278.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:05.418788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:22Z" } 2023-10-11T14:55:22.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035985.418788_278.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:05.418788Z', 'realm_epoch': 2}}] 2023-10-11T14:55:22.107 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:22.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035985.418788_278.1'} 2023-10-11T14:55:22.108 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035985.418788_278.1 2023-10-11T14:55:27.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:27.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:27.110 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:27.168 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:27.169 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:27.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:27.196+0000 7f6494829780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:27.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:27.198+0000 7f6494829780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:27.214 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035985.418788_278.1", 2023-10-11T14:55:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:05.418788Z", 2023-10-11T14:55:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:27.228 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:27Z" 2023-10-11T14:55:27.228 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:27.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035985.418788_278.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:05.418788Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:27Z" } 2023-10-11T14:55:27.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035985.418788_278.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:05.418788Z', 'realm_epoch': 2}}] 2023-10-11T14:55:27.230 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:27.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035985.418788_278.1'} 2023-10-11T14:55:27.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035985.418788_278.1 2023-10-11T14:55:32.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:32.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:32.234 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:32.300 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:32.301 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:32.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:32.341+0000 7f721c87c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:32.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:32.342+0000 7f721c87c780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:32.358 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:32.359 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:32.359 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:32.359 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:32.359 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:32.360 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:32.360 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:32.360 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.360 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:32.360 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:32.361 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:32.361 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:32.361 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:32.361 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:32.362 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:32.362 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:32.362 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:32.362 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:32.363 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:32.363 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:32.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.363 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:32.363 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:32.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:32.364 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:32.364 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:32.364 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:32.364 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:32.365 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:32.365 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:32.365 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:32.365 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:32.365 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:32.366 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:32.366 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:32.366 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:32.367 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:32.367 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:32.367 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:32.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:32.367 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:32.368 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:32.368 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:32.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.368 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:32.368 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:32.369 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:32.369 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:32.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035985.332193_276.1", 2023-10-11T14:55:32.369 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:32.369 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:32.370 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:32.370 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:05.332193Z", 2023-10-11T14:55:32.370 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:32.370 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:32.370 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:32.371 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:32.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.371 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:32.371 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:32.372 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:32.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:32.372 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:32Z" 2023-10-11T14:55:32.372 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:32.373 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035985.332193_276.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:05.332193Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:32Z" } 2023-10-11T14:55:32.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035985.332193_276.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:05.332193Z', 'realm_epoch': 2}}] 2023-10-11T14:55:32.375 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:32.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035985.332193_276.1'} 2023-10-11T14:55:32.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697035988.716816_315.1 target=1_1697035985.332193_276.1 2023-10-11T14:55:37.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.379 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:37.436 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:37.436 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:37.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:37.465+0000 7f9ac8eb4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:37.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:37.466+0000 7f9ac8eb4780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:37.482 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:37.483 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.484 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:37.485 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:37.486 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.487 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:37.488 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:37.489 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:37.490 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:37.491 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697035988.716816_315.1", 2023-10-11T14:55:37.493 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:53:08.716816Z", 2023-10-11T14:55:37.494 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:37.495 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:37.495 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:37.495 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:37.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.495 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:37.496 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:37.496 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:37.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:37.496 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:37Z" 2023-10-11T14:55:37.496 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:37.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697035988.716816_315.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:53:08.716816Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:37Z" } 2023-10-11T14:55:37.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697035988.716816_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:53:08.716816Z', 'realm_epoch': 2}}] 2023-10-11T14:55:37.499 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:37.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697035988.716816_315.1'} 2023-10-11T14:55:37.500 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:55:37.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:55:37.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:55:37.502 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:55:37.555 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:37.555 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:37.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:55:37.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:55:37.620 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:55:37.671 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:37.672 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:37.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.735 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:37.782 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:37.782 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:37.823 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:55:37.823 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000078", 2023-10-11T14:55:37.823 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.history", 2023-10-11T14:55:37.823 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:6", 2023-10-11T14:55:37.824 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:10", 2023-10-11T14:55:37.824 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000070", 2023-10-11T14:55:37.824 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000104", 2023-10-11T14:55:37.824 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:7", 2023-10-11T14:55:37.825 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:5", 2023-10-11T14:55:37.825 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000026", 2023-10-11T14:55:37.825 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:2", 2023-10-11T14:55:37.825 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:2", 2023-10-11T14:55:37.825 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:3", 2023-10-11T14:55:37.826 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.8", 2023-10-11T14:55:37.826 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:2", 2023-10-11T14:55:37.826 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000028", 2023-10-11T14:55:37.826 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:8", 2023-10-11T14:55:37.826 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:5", 2023-10-11T14:55:37.827 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:9", 2023-10-11T14:55:37.827 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:0", 2023-10-11T14:55:37.827 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000040", 2023-10-11T14:55:37.827 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.6", 2023-10-11T14:55:37.828 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000015", 2023-10-11T14:55:37.828 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000069", 2023-10-11T14:55:37.828 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:1", 2023-10-11T14:55:37.828 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:2", 2023-10-11T14:55:37.828 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000095", 2023-10-11T14:55:37.829 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:9", 2023-10-11T14:55:37.829 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000003", 2023-10-11T14:55:37.829 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000047", 2023-10-11T14:55:37.829 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:8", 2023-10-11T14:55:37.829 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.1.VetCxASAdgK.0", 2023-10-11T14:55:37.830 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:6", 2023-10-11T14:55:37.830 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T14:55:37.830 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000035", 2023-10-11T14:55:37.830 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:8", 2023-10-11T14:55:37.831 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000037", 2023-10-11T14:55:37.831 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:2", 2023-10-11T14:55:37.831 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:4", 2023-10-11T14:55:37.831 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:4", 2023-10-11T14:55:37.831 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31", 2023-10-11T14:55:37.832 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000024", 2023-10-11T14:55:37.832 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:2", 2023-10-11T14:55:37.832 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:6", 2023-10-11T14:55:37.832 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000000", 2023-10-11T14:55:37.832 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000031", 2023-10-11T14:55:37.833 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.3", 2023-10-11T14:55:37.833 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9", 2023-10-11T14:55:37.833 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:10", 2023-10-11T14:55:37.833 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:5", 2023-10-11T14:55:37.833 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:4", 2023-10-11T14:55:37.834 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:10", 2023-10-11T14:55:37.834 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.3.C+0bo1qnUIy.0", 2023-10-11T14:55:37.834 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:0", 2023-10-11T14:55:37.834 INFO:teuthology.orchestra.run.smithi158.stdout: "data_loggenerations_metadata", 2023-10-11T14:55:37.835 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:0", 2023-10-11T14:55:37.835 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:0", 2023-10-11T14:55:37.835 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000076", 2023-10-11T14:55:37.835 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:5", 2023-10-11T14:55:37.835 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32", 2023-10-11T14:55:37.836 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000113", 2023-10-11T14:55:37.836 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000005", 2023-10-11T14:55:37.836 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000011", 2023-10-11T14:55:37.836 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:9", 2023-10-11T14:55:37.836 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000058", 2023-10-11T14:55:37.837 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000052", 2023-10-11T14:55:37.837 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:2", 2023-10-11T14:55:37.837 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:10", 2023-10-11T14:55:37.837 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:10", 2023-10-11T14:55:37.837 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:0", 2023-10-11T14:55:37.838 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:6", 2023-10-11T14:55:37.838 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:10", 2023-10-11T14:55:37.838 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000088", 2023-10-11T14:55:37.839 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000080", 2023-10-11T14:55:37.839 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000090", 2023-10-11T14:55:37.839 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:5", 2023-10-11T14:55:37.839 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000110", 2023-10-11T14:55:37.839 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:8", 2023-10-11T14:55:37.840 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:3", 2023-10-11T14:55:37.840 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.2.retry", 2023-10-11T14:55:37.840 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000096", 2023-10-11T14:55:37.840 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000087", 2023-10-11T14:55:37.840 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000008", 2023-10-11T14:55:37.841 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:2", 2023-10-11T14:55:37.841 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.0", 2023-10-11T14:55:37.841 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000006", 2023-10-11T14:55:37.841 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000029", 2023-10-11T14:55:37.842 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13", 2023-10-11T14:55:37.842 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000089", 2023-10-11T14:55:37.842 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000022", 2023-10-11T14:55:37.842 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:3", 2023-10-11T14:55:37.842 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:3", 2023-10-11T14:55:37.843 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog.trim", 2023-10-11T14:55:37.843 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:2", 2023-10-11T14:55:37.843 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000016", 2023-10-11T14:55:37.843 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.2", 2023-10-11T14:55:37.843 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000041", 2023-10-11T14:55:37.844 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:8", 2023-10-11T14:55:37.844 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:0", 2023-10-11T14:55:37.844 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000018", 2023-10-11T14:55:37.844 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000092", 2023-10-11T14:55:37.844 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:9", 2023-10-11T14:55:37.845 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:1", 2023-10-11T14:55:37.845 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.4", 2023-10-11T14:55:37.845 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:3", 2023-10-11T14:55:37.845 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000014", 2023-10-11T14:55:37.845 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000112", 2023-10-11T14:55:37.846 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000007", 2023-10-11T14:55:37.846 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000021", 2023-10-11T14:55:37.846 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12", 2023-10-11T14:55:37.846 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000064", 2023-10-11T14:55:37.846 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:2", 2023-10-11T14:55:37.847 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000071", 2023-10-11T14:55:37.847 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000074", 2023-10-11T14:55:37.847 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:10", 2023-10-11T14:55:37.847 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:7", 2023-10-11T14:55:37.847 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:8", 2023-10-11T14:55:37.848 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:0", 2023-10-11T14:55:37.848 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:10", 2023-10-11T14:55:37.848 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:8", 2023-10-11T14:55:37.848 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000081", 2023-10-11T14:55:37.849 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000009", 2023-10-11T14:55:37.849 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000121", 2023-10-11T14:55:37.849 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.1", 2023-10-11T14:55:37.849 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.0.retry", 2023-10-11T14:55:37.849 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8", 2023-10-11T14:55:37.850 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000125", 2023-10-11T14:55:37.850 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000082", 2023-10-11T14:55:37.850 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:10", 2023-10-11T14:55:37.850 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:9", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000105", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000059", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:1", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:5", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.0.v5nYCoJBjYQ.0", 2023-10-11T14:55:37.851 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:7", 2023-10-11T14:55:37.852 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000077", 2023-10-11T14:55:37.852 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.3.retry", 2023-10-11T14:55:37.852 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000032", 2023-10-11T14:55:37.852 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:3", 2023-10-11T14:55:37.852 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:7", 2023-10-11T14:55:37.853 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:5", 2023-10-11T14:55:37.853 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:4", 2023-10-11T14:55:37.853 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:8", 2023-10-11T14:55:37.853 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15", 2023-10-11T14:55:37.853 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:1", 2023-10-11T14:55:37.854 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1", 2023-10-11T14:55:37.854 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.1", 2023-10-11T14:55:37.854 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:3", 2023-10-11T14:55:37.854 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:4", 2023-10-11T14:55:37.855 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000053", 2023-10-11T14:55:37.855 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:2", 2023-10-11T14:55:37.855 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000091", 2023-10-11T14:55:37.855 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:9", 2023-10-11T14:55:37.856 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000065", 2023-10-11T14:55:37.856 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000083", 2023-10-11T14:55:37.856 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000010", 2023-10-11T14:55:37.856 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:0", 2023-10-11T14:55:37.856 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.7", 2023-10-11T14:55:37.857 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.3", 2023-10-11T14:55:37.857 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:1", 2023-10-11T14:55:37.857 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.0", 2023-10-11T14:55:37.857 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:0", 2023-10-11T14:55:37.857 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:4", 2023-10-11T14:55:37.858 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:4", 2023-10-11T14:55:37.858 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000045", 2023-10-11T14:55:37.858 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.2.d7LYAKVvoH2.0", 2023-10-11T14:55:37.858 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000002", 2023-10-11T14:55:37.858 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:1", 2023-10-11T14:55:37.859 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:5", 2023-10-11T14:55:37.859 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000116", 2023-10-11T14:55:37.859 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:7", 2023-10-11T14:55:37.859 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:1", 2023-10-11T14:55:37.859 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:2", 2023-10-11T14:55:37.860 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000034", 2023-10-11T14:55:37.860 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:5", 2023-10-11T14:55:37.860 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000101", 2023-10-11T14:55:37.860 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:0", 2023-10-11T14:55:37.860 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:4", 2023-10-11T14:55:37.861 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:7", 2023-10-11T14:55:37.861 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:4", 2023-10-11T14:55:37.861 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000079", 2023-10-11T14:55:37.861 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000049", 2023-10-11T14:55:37.861 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000117", 2023-10-11T14:55:37.862 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000044", 2023-10-11T14:55:37.862 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:4", 2023-10-11T14:55:37.862 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:1", 2023-10-11T14:55:37.862 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000066", 2023-10-11T14:55:37.862 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:1", 2023-10-11T14:55:37.863 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:9", 2023-10-11T14:55:37.863 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:7", 2023-10-11T14:55:37.863 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:3", 2023-10-11T14:55:37.863 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000068", 2023-10-11T14:55:37.864 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:7", 2023-10-11T14:55:37.864 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000085", 2023-10-11T14:55:37.864 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:6", 2023-10-11T14:55:37.864 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:6", 2023-10-11T14:55:37.864 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:6", 2023-10-11T14:55:37.865 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:5", 2023-10-11T14:55:37.865 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:9", 2023-10-11T14:55:37.865 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000073", 2023-10-11T14:55:37.865 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:6", 2023-10-11T14:55:37.865 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000038", 2023-10-11T14:55:37.866 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:0", 2023-10-11T14:55:37.866 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000118", 2023-10-11T14:55:37.866 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:8", 2023-10-11T14:55:37.866 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000036", 2023-10-11T14:55:37.866 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000103", 2023-10-11T14:55:37.867 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33", 2023-10-11T14:55:37.867 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17", 2023-10-11T14:55:37.867 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:7", 2023-10-11T14:55:37.867 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000119", 2023-10-11T14:55:37.867 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000098", 2023-10-11T14:55:37.868 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:4", 2023-10-11T14:55:37.868 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10", 2023-10-11T14:55:37.868 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000027", 2023-10-11T14:55:37.868 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:1", 2023-10-11T14:55:37.869 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:7", 2023-10-11T14:55:37.869 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000019", 2023-10-11T14:55:37.869 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:7", 2023-10-11T14:55:37.869 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:10", 2023-10-11T14:55:37.869 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000039", 2023-10-11T14:55:37.870 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34", 2023-10-11T14:55:37.870 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:4", 2023-10-11T14:55:37.870 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:4", 2023-10-11T14:55:37.870 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:6", 2023-10-11T14:55:37.870 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000100", 2023-10-11T14:55:37.871 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:9", 2023-10-11T14:55:37.871 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:5", 2023-10-11T14:55:37.871 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:3", 2023-10-11T14:55:37.871 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000093", 2023-10-11T14:55:37.871 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000004", 2023-10-11T14:55:37.872 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000063", 2023-10-11T14:55:37.872 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000122", 2023-10-11T14:55:37.872 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:6", 2023-10-11T14:55:37.872 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000057", 2023-10-11T14:55:37.872 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:0", 2023-10-11T14:55:37.873 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:8", 2023-10-11T14:55:37.873 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000054", 2023-10-11T14:55:37.873 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000114", 2023-10-11T14:55:37.873 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.2", 2023-10-11T14:55:37.874 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.0", 2023-10-11T14:55:37.874 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:4", 2023-10-11T14:55:37.874 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000012", 2023-10-11T14:55:37.874 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14", 2023-10-11T14:55:37.874 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11", 2023-10-11T14:55:37.875 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000084", 2023-10-11T14:55:37.875 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000043", 2023-10-11T14:55:37.875 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:1", 2023-10-11T14:55:37.875 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:9", 2023-10-11T14:55:37.876 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:8", 2023-10-11T14:55:37.876 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16", 2023-10-11T14:55:37.876 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:8", 2023-10-11T14:55:37.876 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000111", 2023-10-11T14:55:37.876 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:10", 2023-10-11T14:55:37.877 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:7", 2023-10-11T14:55:37.877 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:0", 2023-10-11T14:55:37.877 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000048", 2023-10-11T14:55:37.877 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000020", 2023-10-11T14:55:37.877 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000099", 2023-10-11T14:55:37.878 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000056", 2023-10-11T14:55:37.878 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:3", 2023-10-11T14:55:37.878 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:3", 2023-10-11T14:55:37.878 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:3", 2023-10-11T14:55:37.879 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000072", 2023-10-11T14:55:37.879 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:7", 2023-10-11T14:55:37.879 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:6", 2023-10-11T14:55:37.879 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000062", 2023-10-11T14:55:37.879 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:2", 2023-10-11T14:55:37.880 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:2", 2023-10-11T14:55:37.880 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:6", 2023-10-11T14:55:37.880 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000115", 2023-10-11T14:55:37.880 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.1", 2023-10-11T14:55:37.880 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000033", 2023-10-11T14:55:37.881 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.3", 2023-10-11T14:55:37.881 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:10", 2023-10-11T14:55:37.881 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18", 2023-10-11T14:55:37.881 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000025", 2023-10-11T14:55:37.881 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:9", 2023-10-11T14:55:37.882 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000123", 2023-10-11T14:55:37.882 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000108", 2023-10-11T14:55:37.882 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000094", 2023-10-11T14:55:37.882 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:8", 2023-10-11T14:55:37.883 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000060", 2023-10-11T14:55:37.883 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:1", 2023-10-11T14:55:37.883 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:5", 2023-10-11T14:55:37.883 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000109", 2023-10-11T14:55:37.883 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:1", 2023-10-11T14:55:37.884 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:2", 2023-10-11T14:55:37.884 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000013", 2023-10-11T14:55:37.884 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:6", 2023-10-11T14:55:37.884 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:1", 2023-10-11T14:55:37.884 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:4", 2023-10-11T14:55:37.885 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:6", 2023-10-11T14:55:37.885 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000051", 2023-10-11T14:55:37.885 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:8", 2023-10-11T14:55:37.885 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000106", 2023-10-11T14:55:37.885 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:9", 2023-10-11T14:55:37.886 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:0", 2023-10-11T14:55:37.886 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:10", 2023-10-11T14:55:37.886 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:9", 2023-10-11T14:55:37.886 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000017", 2023-10-11T14:55:37.887 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:3", 2023-10-11T14:55:37.887 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.3", 2023-10-11T14:55:37.887 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000023", 2023-10-11T14:55:37.887 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:9", 2023-10-11T14:55:37.887 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:3", 2023-10-11T14:55:37.888 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000107", 2023-10-11T14:55:37.888 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:9", 2023-10-11T14:55:37.888 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000046", 2023-10-11T14:55:37.888 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:0", 2023-10-11T14:55:37.888 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:0", 2023-10-11T14:55:37.889 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.2", 2023-10-11T14:55:37.889 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:1", 2023-10-11T14:55:37.889 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000067", 2023-10-11T14:55:37.889 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:4", 2023-10-11T14:55:37.889 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:7", 2023-10-11T14:55:37.890 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:0", 2023-10-11T14:55:37.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.9", 2023-10-11T14:55:37.890 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000086", 2023-10-11T14:55:37.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.5", 2023-10-11T14:55:37.890 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:1", 2023-10-11T14:55:37.891 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:5", 2023-10-11T14:55:37.891 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:2", 2023-10-11T14:55:37.891 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:10", 2023-10-11T14:55:37.891 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:9", 2023-10-11T14:55:37.892 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:10", 2023-10-11T14:55:37.892 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:6", 2023-10-11T14:55:37.892 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:5", 2023-10-11T14:55:37.892 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:6", 2023-10-11T14:55:37.892 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:7", 2023-10-11T14:55:37.893 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:1", 2023-10-11T14:55:37.893 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000050", 2023-10-11T14:55:37.893 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:8", 2023-10-11T14:55:37.893 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:5", 2023-10-11T14:55:37.893 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000102", 2023-10-11T14:55:37.894 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:9", 2023-10-11T14:55:37.894 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:5", 2023-10-11T14:55:37.894 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000124", 2023-10-11T14:55:37.894 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.37b32c30-76c0-479d-b458-e4824d081f13.2", 2023-10-11T14:55:37.894 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:3", 2023-10-11T14:55:37.895 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000001", 2023-10-11T14:55:37.895 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000126", 2023-10-11T14:55:37.895 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:8", 2023-10-11T14:55:37.895 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000055", 2023-10-11T14:55:37.896 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:3", 2023-10-11T14:55:37.896 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7", 2023-10-11T14:55:37.896 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000061", 2023-10-11T14:55:37.896 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:8", 2023-10-11T14:55:37.896 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:7", 2023-10-11T14:55:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000097", 2023-10-11T14:55:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:6", 2023-10-11T14:55:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:4", 2023-10-11T14:55:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000042", 2023-10-11T14:55:37.897 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:10", 2023-10-11T14:55:37.898 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:10", 2023-10-11T14:55:37.898 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:5", 2023-10-11T14:55:37.898 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:7", 2023-10-11T14:55:37.898 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000075", 2023-10-11T14:55:37.899 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000120", 2023-10-11T14:55:37.899 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000030", 2023-10-11T14:55:37.899 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:2", 2023-10-11T14:55:37.899 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:3" 2023-10-11T14:55:37.899 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:55:37.900 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:55:37.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:37.902 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:37.948 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:37.948 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:37.992 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:55:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:7", 2023-10-11T14:55:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:0", 2023-10-11T14:55:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000078", 2023-10-11T14:55:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.history", 2023-10-11T14:55:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:5", 2023-10-11T14:55:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:7", 2023-10-11T14:55:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:5", 2023-10-11T14:55:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000070", 2023-10-11T14:55:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000104", 2023-10-11T14:55:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:0", 2023-10-11T14:55:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000026", 2023-10-11T14:55:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.8", 2023-10-11T14:55:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000028", 2023-10-11T14:55:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:9", 2023-10-11T14:55:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:6", 2023-10-11T14:55:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:5", 2023-10-11T14:55:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000040", 2023-10-11T14:55:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.6", 2023-10-11T14:55:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.3.nDO3stFcP4A.0", 2023-10-11T14:55:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000015", 2023-10-11T14:55:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000069", 2023-10-11T14:55:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:9", 2023-10-11T14:55:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000095", 2023-10-11T14:55:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000003", 2023-10-11T14:55:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000047", 2023-10-11T14:55:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:2", 2023-10-11T14:55:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10", 2023-10-11T14:55:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000035", 2023-10-11T14:55:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:2", 2023-10-11T14:55:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:9", 2023-10-11T14:55:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000037", 2023-10-11T14:55:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:3", 2023-10-11T14:55:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:6", 2023-10-11T14:55:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:5", 2023-10-11T14:55:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000024", 2023-10-11T14:55:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000000", 2023-10-11T14:55:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000031", 2023-10-11T14:55:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.3", 2023-10-11T14:55:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:10", 2023-10-11T14:55:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:6", 2023-10-11T14:55:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:2", 2023-10-11T14:55:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:2", 2023-10-11T14:55:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:8", 2023-10-11T14:55:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.3.retry", 2023-10-11T14:55:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:0", 2023-10-11T14:55:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "data_loggenerations_metadata", 2023-10-11T14:55:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9", 2023-10-11T14:55:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:6", 2023-10-11T14:55:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:1", 2023-10-11T14:55:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:9", 2023-10-11T14:55:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:6", 2023-10-11T14:55:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000076", 2023-10-11T14:55:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:10", 2023-10-11T14:55:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000113", 2023-10-11T14:55:38.004 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000005", 2023-10-11T14:55:38.004 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000011", 2023-10-11T14:55:38.004 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000058", 2023-10-11T14:55:38.004 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:1", 2023-10-11T14:55:38.004 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000052", 2023-10-11T14:55:38.005 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:6", 2023-10-11T14:55:38.005 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:1", 2023-10-11T14:55:38.005 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13", 2023-10-11T14:55:38.005 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:4", 2023-10-11T14:55:38.005 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000088", 2023-10-11T14:55:38.006 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000080", 2023-10-11T14:55:38.006 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000090", 2023-10-11T14:55:38.006 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000110", 2023-10-11T14:55:38.006 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:3", 2023-10-11T14:55:38.007 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000096", 2023-10-11T14:55:38.007 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:8", 2023-10-11T14:55:38.007 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:9", 2023-10-11T14:55:38.007 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:7", 2023-10-11T14:55:38.007 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:3", 2023-10-11T14:55:38.008 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:10", 2023-10-11T14:55:38.008 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:0", 2023-10-11T14:55:38.008 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000087", 2023-10-11T14:55:38.008 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000008", 2023-10-11T14:55:38.008 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:7", 2023-10-11T14:55:38.009 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.0", 2023-10-11T14:55:38.009 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:8", 2023-10-11T14:55:38.009 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000006", 2023-10-11T14:55:38.009 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000029", 2023-10-11T14:55:38.009 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:5", 2023-10-11T14:55:38.010 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:1", 2023-10-11T14:55:38.010 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000089", 2023-10-11T14:55:38.010 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000022", 2023-10-11T14:55:38.010 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:0", 2023-10-11T14:55:38.010 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:1", 2023-10-11T14:55:38.011 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog.trim", 2023-10-11T14:55:38.011 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000016", 2023-10-11T14:55:38.011 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.2", 2023-10-11T14:55:38.011 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000041", 2023-10-11T14:55:38.011 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16", 2023-10-11T14:55:38.012 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:7", 2023-10-11T14:55:38.012 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000018", 2023-10-11T14:55:38.012 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000092", 2023-10-11T14:55:38.012 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:10", 2023-10-11T14:55:38.013 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:2", 2023-10-11T14:55:38.013 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:9", 2023-10-11T14:55:38.013 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:8", 2023-10-11T14:55:38.013 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.4", 2023-10-11T14:55:38.013 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000014", 2023-10-11T14:55:38.014 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000112", 2023-10-11T14:55:38.014 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000007", 2023-10-11T14:55:38.014 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000021", 2023-10-11T14:55:38.014 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.16", 2023-10-11T14:55:38.014 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:2", 2023-10-11T14:55:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:2", 2023-10-11T14:55:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000064", 2023-10-11T14:55:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000071", 2023-10-11T14:55:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:6", 2023-10-11T14:55:38.015 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000074", 2023-10-11T14:55:38.016 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:5", 2023-10-11T14:55:38.016 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000081", 2023-10-11T14:55:38.016 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:3", 2023-10-11T14:55:38.016 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000009", 2023-10-11T14:55:38.016 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000121", 2023-10-11T14:55:38.017 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.1", 2023-10-11T14:55:38.017 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:7", 2023-10-11T14:55:38.017 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.17", 2023-10-11T14:55:38.017 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:7", 2023-10-11T14:55:38.017 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000125", 2023-10-11T14:55:38.018 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000082", 2023-10-11T14:55:38.018 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:10", 2023-10-11T14:55:38.018 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000105", 2023-10-11T14:55:38.019 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000059", 2023-10-11T14:55:38.019 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:8", 2023-10-11T14:55:38.019 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:7", 2023-10-11T14:55:38.019 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:0", 2023-10-11T14:55:38.019 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:6", 2023-10-11T14:55:38.020 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:2", 2023-10-11T14:55:38.020 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32", 2023-10-11T14:55:38.020 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000077", 2023-10-11T14:55:38.020 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:1", 2023-10-11T14:55:38.021 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000032", 2023-10-11T14:55:38.021 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:4", 2023-10-11T14:55:38.021 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:9", 2023-10-11T14:55:38.021 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.1", 2023-10-11T14:55:38.021 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:7", 2023-10-11T14:55:38.022 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:8", 2023-10-11T14:55:38.022 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:8", 2023-10-11T14:55:38.022 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:8", 2023-10-11T14:55:38.022 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14", 2023-10-11T14:55:38.022 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000053", 2023-10-11T14:55:38.023 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:1", 2023-10-11T14:55:38.023 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000091", 2023-10-11T14:55:38.023 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000065", 2023-10-11T14:55:38.023 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000083", 2023-10-11T14:55:38.023 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000010", 2023-10-11T14:55:38.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:0", 2023-10-11T14:55:38.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:2", 2023-10-11T14:55:38.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:3", 2023-10-11T14:55:38.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:0", 2023-10-11T14:55:38.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:3", 2023-10-11T14:55:38.025 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.7", 2023-10-11T14:55:38.025 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.3", 2023-10-11T14:55:38.025 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:5", 2023-10-11T14:55:38.025 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:6", 2023-10-11T14:55:38.025 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:4", 2023-10-11T14:55:38.026 INFO:teuthology.orchestra.run.smithi158.stdout: "mdlog.sync-status.shard.2", 2023-10-11T14:55:38.026 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:7", 2023-10-11T14:55:38.026 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:7", 2023-10-11T14:55:38.026 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:9", 2023-10-11T14:55:38.026 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000045", 2023-10-11T14:55:38.027 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:7", 2023-10-11T14:55:38.027 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000002", 2023-10-11T14:55:38.027 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:3", 2023-10-11T14:55:38.027 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.12", 2023-10-11T14:55:38.027 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:4", 2023-10-11T14:55:38.028 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000116", 2023-10-11T14:55:38.028 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000034", 2023-10-11T14:55:38.028 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:7", 2023-10-11T14:55:38.028 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000101", 2023-10-11T14:55:38.029 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.13", 2023-10-11T14:55:38.029 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:2", 2023-10-11T14:55:38.029 INFO:teuthology.orchestra.run.smithi158.stdout: "mdlog.sync-status.shard.1", 2023-10-11T14:55:38.029 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:3", 2023-10-11T14:55:38.029 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000079", 2023-10-11T14:55:38.030 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000049", 2023-10-11T14:55:38.030 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:7", 2023-10-11T14:55:38.030 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000117", 2023-10-11T14:55:38.030 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000044", 2023-10-11T14:55:38.031 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:1", 2023-10-11T14:55:38.031 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:10", 2023-10-11T14:55:38.031 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:5", 2023-10-11T14:55:38.031 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:9", 2023-10-11T14:55:38.031 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:10", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:4", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.14", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:3", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:1", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000066", 2023-10-11T14:55:38.032 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:8", 2023-10-11T14:55:38.033 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000068", 2023-10-11T14:55:38.033 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:3", 2023-10-11T14:55:38.033 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000085", 2023-10-11T14:55:38.033 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:9", 2023-10-11T14:55:38.034 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:1", 2023-10-11T14:55:38.034 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:8", 2023-10-11T14:55:38.034 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33", 2023-10-11T14:55:38.034 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000073", 2023-10-11T14:55:38.035 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:10", 2023-10-11T14:55:38.035 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000038", 2023-10-11T14:55:38.035 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:0", 2023-10-11T14:55:38.035 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:0", 2023-10-11T14:55:38.035 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.10", 2023-10-11T14:55:38.036 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000118", 2023-10-11T14:55:38.036 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000036", 2023-10-11T14:55:38.036 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:4", 2023-10-11T14:55:38.036 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000103", 2023-10-11T14:55:38.037 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:9", 2023-10-11T14:55:38.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mdlog.sync-status.shard.3", 2023-10-11T14:55:38.037 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:7", 2023-10-11T14:55:38.037 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:1", 2023-10-11T14:55:38.037 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000119", 2023-10-11T14:55:38.038 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000098", 2023-10-11T14:55:38.038 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.2", 2023-10-11T14:55:38.038 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:6", 2023-10-11T14:55:38.038 INFO:teuthology.orchestra.run.smithi158.stdout: "mdlog.sync-status.shard.0", 2023-10-11T14:55:38.038 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:7", 2023-10-11T14:55:38.039 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000027", 2023-10-11T14:55:38.039 INFO:teuthology.orchestra.run.smithi158.stdout: "mdlog.sync-status", 2023-10-11T14:55:38.039 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000019", 2023-10-11T14:55:38.039 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8", 2023-10-11T14:55:38.039 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.0.ky1eFt+IBR7.0", 2023-10-11T14:55:38.040 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000039", 2023-10-11T14:55:38.040 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:0", 2023-10-11T14:55:38.040 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:0", 2023-10-11T14:55:38.040 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:3", 2023-10-11T14:55:38.040 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:1", 2023-10-11T14:55:38.041 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:5", 2023-10-11T14:55:38.041 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000100", 2023-10-11T14:55:38.041 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:8", 2023-10-11T14:55:38.041 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:4", 2023-10-11T14:55:38.042 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000093", 2023-10-11T14:55:38.042 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17", 2023-10-11T14:55:38.042 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:10", 2023-10-11T14:55:38.042 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:5", 2023-10-11T14:55:38.042 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000004", 2023-10-11T14:55:38.043 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000063", 2023-10-11T14:55:38.043 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:0", 2023-10-11T14:55:38.043 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000122", 2023-10-11T14:55:38.043 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000057", 2023-10-11T14:55:38.044 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:0", 2023-10-11T14:55:38.044 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000054", 2023-10-11T14:55:38.045 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000114", 2023-10-11T14:55:38.045 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.2", 2023-10-11T14:55:38.045 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.0", 2023-10-11T14:55:38.045 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:8", 2023-10-11T14:55:38.045 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:4", 2023-10-11T14:55:38.046 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000012", 2023-10-11T14:55:38.046 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.2.retry", 2023-10-11T14:55:38.046 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31", 2023-10-11T14:55:38.046 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:1", 2023-10-11T14:55:38.047 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:4", 2023-10-11T14:55:38.047 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:8", 2023-10-11T14:55:38.047 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.3", 2023-10-11T14:55:38.047 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.2.x0KF1b2GUd3.0", 2023-10-11T14:55:38.047 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000084", 2023-10-11T14:55:38.048 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000043", 2023-10-11T14:55:38.048 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:8", 2023-10-11T14:55:38.048 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.11", 2023-10-11T14:55:38.048 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:4", 2023-10-11T14:55:38.048 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:6", 2023-10-11T14:55:38.049 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000111", 2023-10-11T14:55:38.049 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000048", 2023-10-11T14:55:38.049 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000020", 2023-10-11T14:55:38.049 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:9", 2023-10-11T14:55:38.050 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:4", 2023-10-11T14:55:38.050 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000099", 2023-10-11T14:55:38.050 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000056", 2023-10-11T14:55:38.050 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:9", 2023-10-11T14:55:38.050 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.19", 2023-10-11T14:55:38.051 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000072", 2023-10-11T14:55:38.051 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:7", 2023-10-11T14:55:38.051 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000062", 2023-10-11T14:55:38.051 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000115", 2023-10-11T14:55:38.051 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11", 2023-10-11T14:55:38.052 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.1", 2023-10-11T14:55:38.052 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:6", 2023-10-11T14:55:38.052 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000033", 2023-10-11T14:55:38.052 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:4", 2023-10-11T14:55:38.052 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:6", 2023-10-11T14:55:38.053 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:9", 2023-10-11T14:55:38.053 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7", 2023-10-11T14:55:38.053 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:2", 2023-10-11T14:55:38.053 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:10", 2023-10-11T14:55:38.053 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000025", 2023-10-11T14:55:38.054 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:5", 2023-10-11T14:55:38.054 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000123", 2023-10-11T14:55:38.054 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:3", 2023-10-11T14:55:38.054 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:10", 2023-10-11T14:55:38.055 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.18", 2023-10-11T14:55:38.055 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:8", 2023-10-11T14:55:38.055 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000108", 2023-10-11T14:55:38.055 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000094", 2023-10-11T14:55:38.055 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:1", 2023-10-11T14:55:38.056 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:6", 2023-10-11T14:55:38.056 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:5", 2023-10-11T14:55:38.056 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.0", 2023-10-11T14:55:38.056 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000060", 2023-10-11T14:55:38.057 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000109", 2023-10-11T14:55:38.057 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.15", 2023-10-11T14:55:38.057 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:4", 2023-10-11T14:55:38.057 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:2", 2023-10-11T14:55:38.057 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000013", 2023-10-11T14:55:38.058 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:9", 2023-10-11T14:55:38.058 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:10", 2023-10-11T14:55:38.058 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:2", 2023-10-11T14:55:38.058 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:5", 2023-10-11T14:55:38.059 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000051", 2023-10-11T14:55:38.059 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000106", 2023-10-11T14:55:38.059 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:1", 2023-10-11T14:55:38.059 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000017", 2023-10-11T14:55:38.059 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-9:daa316e9-869c-4074-a756-a8adfb4a3573.4373.9:2", 2023-10-11T14:55:38.060 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:4", 2023-10-11T14:55:38.060 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:3", 2023-10-11T14:55:38.060 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:5", 2023-10-11T14:55:38.060 INFO:teuthology.orchestra.run.smithi158.stdout: "meta.log.064106cf-4523-427d-9a69-628884ec70af.3", 2023-10-11T14:55:38.061 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000023", 2023-10-11T14:55:38.061 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34", 2023-10-11T14:55:38.061 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:3", 2023-10-11T14:55:38.061 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:6", 2023-10-11T14:55:38.061 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000107", 2023-10-11T14:55:38.062 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000046", 2023-10-11T14:55:38.062 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-11:daa316e9-869c-4074-a756-a8adfb4a3573.4373.11:6", 2023-10-11T14:55:38.062 INFO:teuthology.orchestra.run.smithi158.stdout: "data_log.2", 2023-10-11T14:55:38.062 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:8", 2023-10-11T14:55:38.063 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000067", 2023-10-11T14:55:38.063 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:6", 2023-10-11T14:55:38.063 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:1", 2023-10-11T14:55:38.063 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:3", 2023-10-11T14:55:38.063 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.9", 2023-10-11T14:55:38.064 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000086", 2023-10-11T14:55:38.064 INFO:teuthology.orchestra.run.smithi158.stdout: "sync.error-log.5", 2023-10-11T14:55:38.064 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-34:daa316e9-869c-4074-a756-a8adfb4a3573.4373.34:2", 2023-10-11T14:55:38.064 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:0", 2023-10-11T14:55:38.064 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:0", 2023-10-11T14:55:38.065 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-31:daa316e9-869c-4074-a756-a8adfb4a3573.4373.31:9", 2023-10-11T14:55:38.065 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-16:daa316e9-869c-4074-a756-a8adfb4a3573.4373.16:10", 2023-10-11T14:55:38.065 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:3", 2023-10-11T14:55:38.065 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-33:daa316e9-869c-4074-a756-a8adfb4a3573.4373.33:8", 2023-10-11T14:55:38.066 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18", 2023-10-11T14:55:38.066 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-14:daa316e9-869c-4074-a756-a8adfb4a3573.4373.14:4", 2023-10-11T14:55:38.066 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:10", 2023-10-11T14:55:38.066 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-32:daa316e9-869c-4074-a756-a8adfb4a3573.4373.32:5", 2023-10-11T14:55:38.066 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T14:55:38.067 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000050", 2023-10-11T14:55:38.067 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000102", 2023-10-11T14:55:38.067 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-8:daa316e9-869c-4074-a756-a8adfb4a3573.4373.8:3", 2023-10-11T14:55:38.067 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-13:daa316e9-869c-4074-a756-a8adfb4a3573.4373.13:10", 2023-10-11T14:55:38.068 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:2", 2023-10-11T14:55:38.068 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-7:daa316e9-869c-4074-a756-a8adfb4a3573.4373.7:4", 2023-10-11T14:55:38.068 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000124", 2023-10-11T14:55:38.068 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:0", 2023-10-11T14:55:38.068 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000001", 2023-10-11T14:55:38.069 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1", 2023-10-11T14:55:38.069 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12", 2023-10-11T14:55:38.069 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000126", 2023-10-11T14:55:38.069 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000055", 2023-10-11T14:55:38.069 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000061", 2023-10-11T14:55:38.070 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15:2", 2023-10-11T14:55:38.070 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-18:daa316e9-869c-4074-a756-a8adfb4a3573.4373.18:9", 2023-10-11T14:55:38.070 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-15:daa316e9-869c-4074-a756-a8adfb4a3573.4373.15", 2023-10-11T14:55:38.070 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000097", 2023-10-11T14:55:38.071 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-12:daa316e9-869c-4074-a756-a8adfb4a3573.4373.12:5", 2023-10-11T14:55:38.071 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000042", 2023-10-11T14:55:38.071 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:5", 2023-10-11T14:55:38.071 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-10:daa316e9-869c-4074-a756-a8adfb4a3573.4373.10:10", 2023-10-11T14:55:38.072 INFO:teuthology.orchestra.run.smithi158.stdout: "datalog.sync-status.shard.daa316e9-869c-4074-a756-a8adfb4a3573.0.retry", 2023-10-11T14:55:38.072 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:1", 2023-10-11T14:55:38.072 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000075", 2023-10-11T14:55:38.072 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-17:daa316e9-869c-4074-a756-a8adfb4a3573.4373.17:10", 2023-10-11T14:55:38.073 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000120", 2023-10-11T14:55:38.073 INFO:teuthology.orchestra.run.smithi158.stdout: "obj_delete_at_hint.0000000030", 2023-10-11T14:55:38.073 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-35:daa316e9-869c-4074-a756-a8adfb4a3573.5420.1:4" 2023-10-11T14:55:38.073 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:55:38.073 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-11T14:55:38.075 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T14:55:38.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:38.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:38.076 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:38.126 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:38.126 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:38.170 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:55:38.171 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.171 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-11T14:55:38.171 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:37.716222Z" 2023-10-11T14:55:38.171 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.172 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.172 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-10-11T14:55:38.172 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:37.713471Z" 2023-10-11T14:55:38.172 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.173 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.173 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-11T14:55:38.173 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:37.714397Z" 2023-10-11T14:55:38.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.173 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.174 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003745", 2023-10-11T14:55:38.174 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:37.715273Z" 2023-10-11T14:55:38.174 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.174 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:55:38.176 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002055', 1: '00000000000000000000:00000000000000000933', 2: '00000000000000000000:00000000000000004167', 3: '00000000000000000000:00000000000000003745'} 2023-10-11T14:55:38.176 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:55:38.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:38.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:38.178 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:38.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:38.231 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:38.272 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:38.272 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:38.272 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:38.273 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:38.273 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:38.273 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:55:38.273 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.274 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:38.274 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.274 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:38.274 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:38.275 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:38.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:55:38.275 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:38.275 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:38.275 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:38.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:30.511368Z" 2023-10-11T14:55:38.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.276 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.277 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:38.277 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:38.277 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:38.277 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:38.277 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:38.278 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:38.278 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:38.278 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:55:38.278 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.279 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.279 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.279 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:38.279 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:38.279 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:31.559331Z" 2023-10-11T14:55:38.280 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.281 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.281 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:38.281 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:38.281 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:38.281 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:38.282 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-11T14:55:38.282 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:38.282 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:38.282 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:38.283 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:50:56.098201Z" 2023-10-11T14:55:38.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:38.283 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:38.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.284 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:38.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:55:38.284 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:55:38.284 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:38.284 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:38Z" 2023-10-11T14:55:38.285 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:38.286 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:30.511368Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:31.559331Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:50:56.098201Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:55:38Z" } 2023-10-11T14:55:38.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:30.511368Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:31.559331Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:50:56.098201Z'}}] 2023-10-11T14:55:38.287 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002055', 1: '00000000000000000000:00000000000000000933', 2: '00000000000000000000:00000000000000004167', 3: '00000000000000000000:00000000000000003745'} 2023-10-11T14:55:38.287 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000003324', 3: '00000000000000000000:00000000000000002903'} 2023-10-11T14:55:38.288 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=1 master=00000000000000000000:00000000000000000933 target=, shard=2 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=3 master=00000000000000000000:00000000000000003745 target=00000000000000000000:00000000000000002903 2023-10-11T14:55:43.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:43.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:43.291 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:43.351 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:43.352 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:43.398 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:43.398 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:43.398 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:43.399 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:43.399 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:43.399 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:55:43.399 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.399 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:43.400 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:43.400 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:43.400 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:43.400 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:43.400 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:55:43.401 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:43.401 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:43.401 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:43.401 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:30.511368Z" 2023-10-11T14:55:43.401 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:43.402 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.402 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:43.402 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:43.402 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:43.403 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:43.403 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:43.403 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:43.403 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:43.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:43.404 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:55:43.404 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:43.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:43.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:43.405 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:43.405 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:43.405 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-10-11T14:55:43.405 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:43.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:43.406 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:43.406 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:31.559331Z" 2023-10-11T14:55:43.406 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:43.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.407 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:43.407 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:43.407 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:43.407 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:43.408 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-10-11T14:55:43.408 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:43.408 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:43.408 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:43.408 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:50:56.098201Z" 2023-10-11T14:55:43.409 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:43.409 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:43.409 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:43.409 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.409 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:43.410 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:55:43.410 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:55:43.410 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:43.410 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:43Z" 2023-10-11T14:55:43.410 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:43.411 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:30.511368Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:31.559331Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:50:56.098201Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:55:43Z" } 2023-10-11T14:55:43.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:30.511368Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:31.559331Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:50:56.098201Z'}}] 2023-10-11T14:55:43.412 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002055', 1: '00000000000000000000:00000000000000000933', 2: '00000000000000000000:00000000000000004167', 3: '00000000000000000000:00000000000000003745'} 2023-10-11T14:55:43.413 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000003324', 3: '00000000000000000000:00000000000000002903'} 2023-10-11T14:55:43.413 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=1 master=00000000000000000000:00000000000000000933 target=, shard=2 master=00000000000000000000:00000000000000004167 target=00000000000000000000:00000000000000003324, shard=3 master=00000000000000000000:00000000000000003745 target=00000000000000000000:00000000000000002903 2023-10-11T14:55:48.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.415 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:48.474 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:48.474 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:48.518 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:48.518 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:48.519 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:48.519 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:48.519 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:48.519 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:55:48.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.520 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:48.520 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.520 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:48.520 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.520 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.521 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-10-11T14:55:48.521 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.521 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.521 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:37.716222Z" 2023-10-11T14:55:48.522 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:48.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.523 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.523 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-10-11T14:55:48.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.524 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.524 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:37.713471Z" 2023-10-11T14:55:48.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.524 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:48.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.525 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.525 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004167", 2023-10-11T14:55:48.525 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.526 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.526 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:37.714397Z" 2023-10-11T14:55:48.526 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.526 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.527 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.527 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:48.527 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.527 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.527 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003745", 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:37.715273Z" 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.528 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.529 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:48.529 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.529 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:48.529 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:55:48.529 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:55:48.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.530 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:48Z" 2023-10-11T14:55:48.530 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:48.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:37.716222Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000933", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:37.713471Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004167", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:37.714397Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003745", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:37.715273Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:55:48Z" } 2023-10-11T14:55:48.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:37.716222Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000933', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:37.713471Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004167', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:37.714397Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003745', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:37.715273Z'}}] 2023-10-11T14:55:48.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002055', 1: '00000000000000000000:00000000000000000933', 2: '00000000000000000000:00000000000000004167', 3: '00000000000000000000:00000000000000003745'} 2023-10-11T14:55:48.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002055', 1: '00000000000000000000:00000000000000000933', 2: '00000000000000000000:00000000000000004167', 3: '00000000000000000000:00000000000000003745'} 2023-10-11T14:55:48.533 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:55:48.533 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T14:55:48.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.534 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:48.583 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:48.583 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:48.629 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:55:48.629 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.629 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:55:48.629 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:30.480458Z" 2023-10-11T14:55:48.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.630 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.630 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:48.630 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:55:48.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.631 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.631 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T14:55:48.631 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:47:31.714080Z" 2023-10-11T14:55:48.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.632 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.632 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-10-11T14:55:48.632 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:50:56.327099Z" 2023-10-11T14:55:48.632 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.632 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:55:48.633 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001918'} 2023-10-11T14:55:48.634 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:55:48.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:48.636 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:48.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:48.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:48.732 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:48.732 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:48.732 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:48.732 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:48.733 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:48.733 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:55:48.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.733 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:48.734 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.734 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:48.734 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.734 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.735 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-10-11T14:55:48.735 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.735 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.735 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.735 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:30.480458Z" 2023-10-11T14:55:48.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.736 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.736 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.736 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:48.736 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.737 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.737 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:48.737 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.737 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.737 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.738 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000" 2023-10-11T14:55:48.738 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.738 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.739 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:48.739 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.739 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.739 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T14:55:48.739 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.740 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.740 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.740 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:47:31.714080Z" 2023-10-11T14:55:48.740 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.741 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:48.741 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:48.741 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:48.741 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:55:48.741 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-10-11T14:55:48.742 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:48.742 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:48.742 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:48.742 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:50:56.327099Z" 2023-10-11T14:55:48.743 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.743 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:48.743 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:48.743 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.743 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:48.744 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:55:48.744 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:55:48.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:48.744 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:48Z" 2023-10-11T14:55:48.744 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:48.746 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:30.480458Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004452", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:47:31.714080Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:50:56.327099Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:55:48Z" } 2023-10-11T14:55:48.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:30.480458Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004452', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:47:31.714080Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:50:56.327099Z'}}] 2023-10-11T14:55:48.747 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001918'} 2023-10-11T14:55:48.747 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004452', 3: '00000000000000000000:00000000000000001918'} 2023-10-11T14:55:48.747 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:55:48.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-10-11T14:55:48.749 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-46 2023-10-11T14:55:48.750 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:55:48.750 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:55:48.751 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.751 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:55:48.751 DEBUG:boto:Data: 2023-10-11T14:55:48.752 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.753 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.753 DEBUG:boto:Params: {} 2023-10-11T14:55:48.754 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:55:48.754 DEBUG:boto:Token: None 2023-10-11T14:55:48.755 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/ 2023-10-11T14:55:48.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wxyxxs1AQQJdQr/3PUsxK0BwCO0= 2023-10-11T14:55:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wxyxxs1AQQJdQr/3PUsxK0BwCO0=', 'Content-Length': '0'} 2023-10-11T14:55:48.771 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000043a07949aa74f7b5-006526b774-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.771 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-47 2023-10-11T14:55:48.772 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:55:48.772 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:55:48.772 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.773 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:55:48.773 DEBUG:boto:Data: 2023-10-11T14:55:48.774 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:48.774 DEBUG:boto:Port: 8001 2023-10-11T14:55:48.775 DEBUG:boto:Params: {} 2023-10-11T14:55:48.776 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:55:48.776 DEBUG:boto:Token: None 2023-10-11T14:55:48.776 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-47/ 2023-10-11T14:55:48.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HS7MbaTuZL/kAabGNA+hstq7IJE= 2023-10-11T14:55:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HS7MbaTuZL/kAabGNA+hstq7IJE=', 'Content-Length': '0'} 2023-10-11T14:55:48.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000487c87e0ad887957-006526b774-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.803 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:55:48.803 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:55:48.804 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.804 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:55:48.804 DEBUG:boto:Data: 2023-10-11T14:55:48.805 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.805 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.806 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.806 DEBUG:boto:Params: {} 2023-10-11T14:55:48.806 DEBUG:boto:Token: None 2023-10-11T14:55:48.807 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/ 2023-10-11T14:55:48.808 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk= 2023-10-11T14:55:48.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk=', 'Content-Length': '0'} 2023-10-11T14:55:48.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6bd34ece4ca0043-006526b774-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.811 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:55:48.811 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:55:48.812 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.812 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:55:48.812 DEBUG:boto:Data: 2023-10-11T14:55:48.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.813 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.813 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.814 DEBUG:boto:Params: {} 2023-10-11T14:55:48.814 DEBUG:boto:Token: None 2023-10-11T14:55:48.815 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/obj1 2023-10-11T14:55:48.815 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:50RgAQj0oGeg33oEc6M0BNMktbU= 2023-10-11T14:55:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:50RgAQj0oGeg33oEc6M0BNMktbU='} 2023-10-11T14:55:48.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a641fcd95378d973-006526b774-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.828 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:55:48.828 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:55:48.829 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.829 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:55:48.829 DEBUG:boto:Data: 2023-10-11T14:55:48.830 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.831 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.831 DEBUG:boto:Params: {} 2023-10-11T14:55:48.831 DEBUG:boto:Token: None 2023-10-11T14:55:48.832 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/ 2023-10-11T14:55:48.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk= 2023-10-11T14:55:48.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk=', 'Content-Length': '0'} 2023-10-11T14:55:48.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c3f8172099a15c2-006526b774-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.835 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:55:48.836 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:55:48.836 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.837 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:55:48.837 DEBUG:boto:Data: 2023-10-11T14:55:48.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.838 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.839 DEBUG:boto:Params: {} 2023-10-11T14:55:48.839 DEBUG:boto:Token: None 2023-10-11T14:55:48.840 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/obj2 2023-10-11T14:55:48.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y+yDl7Tpk/046GapWP6+bk8jbgA= 2023-10-11T14:55:48.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y+yDl7Tpk/046GapWP6+bk8jbgA='} 2023-10-11T14:55:48.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041d724619fcfb2d0-006526b774-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.884 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:55:48.884 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:55:48.885 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.885 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:55:48.886 DEBUG:boto:Data: 2023-10-11T14:55:48.886 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.887 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.887 DEBUG:boto:Params: {} 2023-10-11T14:55:48.888 DEBUG:boto:Token: None 2023-10-11T14:55:48.888 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/ 2023-10-11T14:55:48.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk= 2023-10-11T14:55:48.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk=', 'Content-Length': '0'} 2023-10-11T14:55:48.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b802e7a0ad63b1f-006526b774-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.892 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:55:48.892 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:55:48.893 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.893 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:55:48.894 DEBUG:boto:Data: 2023-10-11T14:55:48.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.895 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.896 DEBUG:boto:Params: {} 2023-10-11T14:55:48.896 DEBUG:boto:Token: None 2023-10-11T14:55:48.896 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/obj3 2023-10-11T14:55:48.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2fbl+aVQY472QctrK+wKgJ+cnw= 2023-10-11T14:55:48.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2fbl+aVQY472QctrK+wKgJ+cnw='} 2023-10-11T14:55:48.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b40c3ef4cfc457b1-006526b774-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.902 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:55:48.903 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:55:48.903 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.904 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:55:48.904 DEBUG:boto:Data: 2023-10-11T14:55:48.905 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.905 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.905 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.906 DEBUG:boto:Params: {} 2023-10-11T14:55:48.906 DEBUG:boto:Token: None 2023-10-11T14:55:48.907 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/ 2023-10-11T14:55:48.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk= 2023-10-11T14:55:48.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mX9Hj9Yyx4/rMLueuSgn53gsYqk=', 'Content-Length': '0'} 2023-10-11T14:55:48.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1320e9ab5f3fab1-006526b774-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.910 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:55:48.911 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:55:48.911 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.912 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:55:48.912 DEBUG:boto:Data: 2023-10-11T14:55:48.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:55:48.913 DEBUG:boto:Port: 8000 2023-10-11T14:55:48.914 DEBUG:boto:Params: {} 2023-10-11T14:55:48.914 DEBUG:boto:Token: None 2023-10-11T14:55:48.915 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-46/obj4 2023-10-11T14:55:48.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u1qDCj1ofm+XtxBB85ScCHfWs4E= 2023-10-11T14:55:48.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u1qDCj1ofm+XtxBB85ScCHfWs4E='} 2023-10-11T14:55:48.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db7c8e0eaeb62ef6-006526b774-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.959 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:55:48.959 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:55:48.960 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.960 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:55:48.960 DEBUG:boto:Data: 2023-10-11T14:55:48.961 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:48.962 DEBUG:boto:Port: 8001 2023-10-11T14:55:48.962 DEBUG:boto:Params: {} 2023-10-11T14:55:48.962 DEBUG:boto:Token: None 2023-10-11T14:55:48.963 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-47/ 2023-10-11T14:55:48.963 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nAfKgey3Wn2TEZ/qi/YyCrHfxF4= 2023-10-11T14:55:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nAfKgey3Wn2TEZ/qi/YyCrHfxF4=', 'Content-Length': '0'} 2023-10-11T14:55:48.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007420ba88d0ebc5f6-006526b774-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.967 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:55:48.967 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:55:48.967 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.968 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:55:48.968 DEBUG:boto:Data: 2023-10-11T14:55:48.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:48.969 DEBUG:boto:Port: 8001 2023-10-11T14:55:48.970 DEBUG:boto:Params: {} 2023-10-11T14:55:48.970 DEBUG:boto:Token: None 2023-10-11T14:55:48.971 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-47/obj1 2023-10-11T14:55:48.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u3bzlJtKtJ8omr08QzbNka1COBA= 2023-10-11T14:55:48.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u3bzlJtKtJ8omr08QzbNka1COBA='} 2023-10-11T14:55:48.977 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000562b3cf088b00e00-006526b774-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.978 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:55:48.978 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:55:48.979 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.979 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:55:48.980 DEBUG:boto:Data: 2023-10-11T14:55:48.980 DEBUG:boto:Headers: {} 2023-10-11T14:55:48.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:48.981 DEBUG:boto:Port: 8001 2023-10-11T14:55:48.981 DEBUG:boto:Params: {} 2023-10-11T14:55:48.982 DEBUG:boto:Token: None 2023-10-11T14:55:48.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-47/ 2023-10-11T14:55:48.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nAfKgey3Wn2TEZ/qi/YyCrHfxF4= 2023-10-11T14:55:48.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nAfKgey3Wn2TEZ/qi/YyCrHfxF4=', 'Content-Length': '0'} 2023-10-11T14:55:48.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc5d415dab6bd42b-006526b774-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.987 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:55:48.987 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:55:48.988 DEBUG:boto:Method: PUT 2023-10-11T14:55:48.989 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:55:48.989 DEBUG:boto:Data: 2023-10-11T14:55:48.989 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:48.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:48.990 DEBUG:boto:Port: 8001 2023-10-11T14:55:48.991 DEBUG:boto:Params: {} 2023-10-11T14:55:48.991 DEBUG:boto:Token: None 2023-10-11T14:55:48.991 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:48 GMT /dghqvd-47/obj2 2023-10-11T14:55:48.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bWFsaZsrIBMa7078I8HD0GIyYQA= 2023-10-11T14:55:48.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bWFsaZsrIBMa7078I8HD0GIyYQA='} 2023-10-11T14:55:48.997 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b47a08409d95661-006526b774-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:48.998 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:55:48.998 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:55:48.998 DEBUG:boto:Method: HEAD 2023-10-11T14:55:48.999 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:55:48.999 DEBUG:boto:Data: 2023-10-11T14:55:49.000 DEBUG:boto:Headers: {} 2023-10-11T14:55:49.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:49.001 DEBUG:boto:Port: 8001 2023-10-11T14:55:49.001 DEBUG:boto:Params: {} 2023-10-11T14:55:49.001 DEBUG:boto:Token: None 2023-10-11T14:55:49.002 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:49 GMT /dghqvd-47/ 2023-10-11T14:55:49.003 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UE0DwpZp8KlkyCTXdz9u1LUN4KI= 2023-10-11T14:55:49.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UE0DwpZp8KlkyCTXdz9u1LUN4KI=', 'Content-Length': '0'} 2023-10-11T14:55:49.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b98e3de96d3b23-006526b775-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:49.006 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:55:49.006 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:55:49.006 DEBUG:boto:Method: PUT 2023-10-11T14:55:49.007 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:55:49.007 DEBUG:boto:Data: 2023-10-11T14:55:49.008 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:49.008 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:49.008 DEBUG:boto:Port: 8001 2023-10-11T14:55:49.009 DEBUG:boto:Params: {} 2023-10-11T14:55:49.009 DEBUG:boto:Token: None 2023-10-11T14:55:49.010 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:49 GMT /dghqvd-47/obj3 2023-10-11T14:55:49.010 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7Gd7VqhtvvGjjP0N09mfVR5Bykc= 2023-10-11T14:55:49.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7Gd7VqhtvvGjjP0N09mfVR5Bykc='} 2023-10-11T14:55:49.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2caef7a67386207-006526b775-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:49.016 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:55:49.017 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:55:49.017 DEBUG:boto:Method: HEAD 2023-10-11T14:55:49.017 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:55:49.018 DEBUG:boto:Data: 2023-10-11T14:55:49.018 DEBUG:boto:Headers: {} 2023-10-11T14:55:49.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:49.019 DEBUG:boto:Port: 8001 2023-10-11T14:55:49.019 DEBUG:boto:Params: {} 2023-10-11T14:55:49.020 DEBUG:boto:Token: None 2023-10-11T14:55:49.020 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:55:49 GMT /dghqvd-47/ 2023-10-11T14:55:49.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UE0DwpZp8KlkyCTXdz9u1LUN4KI= 2023-10-11T14:55:49.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:55:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UE0DwpZp8KlkyCTXdz9u1LUN4KI=', 'Content-Length': '0'} 2023-10-11T14:55:49.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2618745186cc0d1-006526b775-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:49.024 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:55:49.024 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:55:49.025 DEBUG:boto:Method: PUT 2023-10-11T14:55:49.025 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:55:49.026 DEBUG:boto:Data: 2023-10-11T14:55:49.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:55:49.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:55:49.027 DEBUG:boto:Port: 8001 2023-10-11T14:55:49.027 DEBUG:boto:Params: {} 2023-10-11T14:55:49.028 DEBUG:boto:Token: None 2023-10-11T14:55:49.028 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:55:49 GMT /dghqvd-47/obj4 2023-10-11T14:55:49.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5ilRBBoAWPYIGEnfTyihJAw2qrA= 2023-10-11T14:55:49.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:55:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5ilRBBoAWPYIGEnfTyihJAw2qrA='} 2023-10-11T14:55:49.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a921b647242c2a4-006526b775-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:55:49.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:49.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:49.080 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:49.129 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:49.130 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:49.159 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:55:49.172 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:55:49.172 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.172 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:49.172 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:55:49.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.173 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.173 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:49.173 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:55:49.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.174 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.174 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:49.174 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:55:49.174 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.175 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.175 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036148.792675_327.1", 2023-10-11T14:55:49.175 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:48.792675Z" 2023-10-11T14:55:49.175 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.176 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:55:49.177 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036148.792675_327.1'} 2023-10-11T14:55:49.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:55:49.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:49.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:55:49.180 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:55:49.231 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:49.231 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:49.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:49.256+0000 7f6c7fe52780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:49.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:49.257+0000 7f6c7fe52780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:55:49.272 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:55:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:55:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:55:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:55:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:55:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:55:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:55:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:55:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:49.278 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:49.278 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:55:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:55:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:55:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:55:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:55:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:55:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:55:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:55:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:55:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:55:49.293 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:55:49.293 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:55:49.294 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:55:49.294 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036148.792675_327.1", 2023-10-11T14:55:49.294 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:55:49.295 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:55:49.295 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:55:49.296 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:48.792675Z", 2023-10-11T14:55:49.296 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:55:49.297 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.297 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:55:49.298 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:55:49.298 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.299 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:55:49.299 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:55:49.299 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:55:49.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:55:49.300 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:55:49Z" 2023-10-11T14:55:49.301 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:55:49.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036148.792675_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:48.792675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:55:49Z" } 2023-10-11T14:55:49.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036148.792675_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:48.792675Z', 'realm_epoch': 2}}] 2023-10-11T14:55:49.305 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036148.792675_327.1'} 2023-10-11T14:55:49.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036148.792675_327.1'} 2023-10-11T14:55:49.306 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:55:49.307 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-46 2023-10-11T14:55:49.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:55:49.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:55:49.309 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:55:49.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:55:49.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:55:49.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:49.396+0000 7feea3955780 1 waiting to reach incremental sync.. 2023-10-11T14:55:54.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:54.397+0000 7feea3955780 1 waiting to reach incremental sync.. 2023-10-11T14:55:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:55:59.397+0000 7feea3955780 1 waiting to reach incremental sync.. 2023-10-11T14:56:04.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:04.397+0000 7feea3955780 1 waiting to reach incremental sync.. 2023-10-11T14:56:09.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:09.400+0000 7feea3955780 1 bucket sync caught up with source: 2023-10-11T14:56:09.403 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.21.6, , 00000000001.26.6, , 00000000001.51.6, , , 00000000001.28.6, , , ] 2023-10-11T14:56:09.403 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.21.6, , 00000000001.26.6, , 00000000001.51.6, , , 00000000001.28.6, , , ] 2023-10-11T14:56:09.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:09.400+0000 7feea3955780 0 bucket checkpoint complete 2023-10-11T14:56:09.419 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-46 2023-10-11T14:56:09.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:56:09.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:56:09.421 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:56:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:56:09.484 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:56:09.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:09.517+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:14.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:14.517+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:19.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:19.518+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:24.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:24.518+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:29.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:29.519+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:34.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:34.519+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:39.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:39.520+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:44.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:44.521+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:49.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:49.521+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:54.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:54.522+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:56:59.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:56:59.521+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:04.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:04.522+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:09.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:09.523+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:14.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:14.523+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:19.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:19.524+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:24.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:24.525+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:29.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:29.525+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:34.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:34.526+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:39.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:39.527+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:44.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:44.528+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:49.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:49.529+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:54.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:54.529+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:57:59.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:57:59.530+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:04.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:04.530+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:09.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:09.531+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:14.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:14.531+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:19.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:19.532+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:24.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:24.533+0000 7f0e7da9d780 1 waiting to reach incremental sync.. 2023-10-11T14:58:29.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:29.533+0000 7f0e7da9d780 1 bucket sync caught up with source: 2023-10-11T14:58:29.537 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000001.236.6, , 00000000001.221.6, , , 00000000001.243.6, , , ] 2023-10-11T14:58:29.537 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000001.236.6, , 00000000001.221.6, , , 00000000001.243.6, , , ] 2023-10-11T14:58:29.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:29.533+0000 7f0e7da9d780 0 bucket checkpoint complete 2023-10-11T14:58:29.552 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-46 zones={a2, a1} 2023-10-11T14:58:29.553 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.554 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.554 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.554 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:29.555 DEBUG:boto:Data: 2023-10-11T14:58:29.555 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.556 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.556 DEBUG:boto:Params: {} 2023-10-11T14:58:29.557 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T14:58:29.557 DEBUG:boto:Token: None 2023-10-11T14:58:29.558 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwyEpqLG0tI3XWz6L/DaEOTp1AI= 2023-10-11T14:58:29.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwyEpqLG0tI3XWz6L/DaEOTp1AI=', 'Content-Length': '0'} 2023-10-11T14:58:29.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005919d6df7a6ddd18-006526b815-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.564 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.565 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.565 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.565 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:29.566 DEBUG:boto:Data: 2023-10-11T14:58:29.566 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.567 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.567 DEBUG:boto:Params: {} 2023-10-11T14:58:29.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T14:58:29.568 DEBUG:boto:Token: None 2023-10-11T14:58:29.569 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwyEpqLG0tI3XWz6L/DaEOTp1AI= 2023-10-11T14:58:29.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwyEpqLG0tI3XWz6L/DaEOTp1AI=', 'Content-Length': '0'} 2023-10-11T14:58:29.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aa5d6abd0cad12c-006526b815-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.574 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:58:29.574 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.575 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.575 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.576 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.576 DEBUG:boto:Method: GET 2023-10-11T14:58:29.576 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.577 DEBUG:boto:Data: 2023-10-11T14:58:29.577 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.578 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.578 DEBUG:boto:Params: {} 2023-10-11T14:58:29.579 DEBUG:boto:Token: None 2023-10-11T14:58:29.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/?versions 2023-10-11T14:58:29.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po= 2023-10-11T14:58:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po=', 'Content-Length': '0'} 2023-10-11T14:58:29.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1b6bb4340aec84-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.583 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:58:29.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:58:29.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:58:29.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:58:29.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:58:29.586 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.586 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.586 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.587 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.587 DEBUG:boto:Method: GET 2023-10-11T14:58:29.588 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.588 DEBUG:boto:Data: 2023-10-11T14:58:29.588 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.589 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.589 DEBUG:boto:Params: {} 2023-10-11T14:58:29.590 DEBUG:boto:Token: None 2023-10-11T14:58:29.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/?versions 2023-10-11T14:58:29.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po= 2023-10-11T14:58:29.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po=', 'Content-Length': '0'} 2023-10-11T14:58:29.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2caf387898639b-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.594 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:58:29.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:58:29.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:58:29.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:58:29.596 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.597 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.597 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.597 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.598 DEBUG:boto:Method: GET 2023-10-11T14:58:29.598 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.598 DEBUG:boto:Data: 2023-10-11T14:58:29.599 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.600 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.600 DEBUG:boto:Params: {} 2023-10-11T14:58:29.600 DEBUG:boto:Token: None 2023-10-11T14:58:29.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/?versions 2023-10-11T14:58:29.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po= 2023-10-11T14:58:29.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po=', 'Content-Length': '0'} 2023-10-11T14:58:29.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb5e5caf52baed4-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.605 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.605 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.606 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.606 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.607 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.607 DEBUG:boto:Method: GET 2023-10-11T14:58:29.607 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:58:29.608 DEBUG:boto:Data: 2023-10-11T14:58:29.608 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.609 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.609 DEBUG:boto:Params: {} 2023-10-11T14:58:29.610 DEBUG:boto:Token: None 2023-10-11T14:58:29.610 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/?versions 2023-10-11T14:58:29.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po= 2023-10-11T14:58:29.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PKu4YvC006Yj5kA4KV2n0ArG1Po=', 'Content-Length': '0'} 2023-10-11T14:58:29.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065627390af9a0981-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.614 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:29.616 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.616 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.616 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.617 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.617 DEBUG:boto:Method: GET 2023-10-11T14:58:29.618 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.618 DEBUG:boto:Data: 2023-10-11T14:58:29.619 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.619 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.620 DEBUG:boto:Params: {} 2023-10-11T14:58:29.620 DEBUG:boto:Token: None 2023-10-11T14:58:29.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE= 2023-10-11T14:58:29.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE=', 'Content-Length': '0'} 2023-10-11T14:58:29.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f222c071c6a0942-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.624 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.625 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.625 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.625 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.626 DEBUG:boto:Method: GET 2023-10-11T14:58:29.626 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.627 DEBUG:boto:Data: 2023-10-11T14:58:29.627 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.628 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.628 DEBUG:boto:Params: {} 2023-10-11T14:58:29.629 DEBUG:boto:Token: None 2023-10-11T14:58:29.629 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE= 2023-10-11T14:58:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE=', 'Content-Length': '0'} 2023-10-11T14:58:29.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000568e4e865752e5c2-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.633 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.634 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.634 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.634 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.635 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.635 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.636 DEBUG:boto:Data: 2023-10-11T14:58:29.636 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.636 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.637 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.637 DEBUG:boto:Params: {} 2023-10-11T14:58:29.638 DEBUG:boto:Token: None 2023-10-11T14:58:29.638 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P6XspI9Pnzc4CFLsYWX/OUMw2xU= 2023-10-11T14:58:29.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P6XspI9Pnzc4CFLsYWX/OUMw2xU=', 'Content-Length': '0'} 2023-10-11T14:58:29.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b67948154ef49f13-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.642 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.642 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.643 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.643 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.643 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.644 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.644 DEBUG:boto:Data: 2023-10-11T14:58:29.644 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.645 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.646 DEBUG:boto:Params: {} 2023-10-11T14:58:29.646 DEBUG:boto:Token: None 2023-10-11T14:58:29.647 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P6XspI9Pnzc4CFLsYWX/OUMw2xU= 2023-10-11T14:58:29.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P6XspI9Pnzc4CFLsYWX/OUMw2xU=', 'Content-Length': '0'} 2023-10-11T14:58:29.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034a70e4cb49c4290-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:29.650 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.651 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.651 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.652 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.652 DEBUG:boto:Method: GET 2023-10-11T14:58:29.652 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.653 DEBUG:boto:Data: 2023-10-11T14:58:29.653 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.654 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.655 DEBUG:boto:Params: {} 2023-10-11T14:58:29.655 DEBUG:boto:Token: None 2023-10-11T14:58:29.656 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.656 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE= 2023-10-11T14:58:29.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE=', 'Content-Length': '0'} 2023-10-11T14:58:29.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096bfd4243d046639-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.659 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.659 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.660 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.660 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.661 DEBUG:boto:Method: GET 2023-10-11T14:58:29.661 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.661 DEBUG:boto:Data: 2023-10-11T14:58:29.662 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.663 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.663 DEBUG:boto:Params: {} 2023-10-11T14:58:29.664 DEBUG:boto:Token: None 2023-10-11T14:58:29.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:58:29.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE= 2023-10-11T14:58:29.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFRVVN9uW5kcg6o2dPCk0DY/vcE=', 'Content-Length': '0'} 2023-10-11T14:58:29.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071bd4680bbcc2133-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.668 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.668 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.669 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.669 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.670 DEBUG:boto:Method: GET 2023-10-11T14:58:29.670 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.671 DEBUG:boto:Data: 2023-10-11T14:58:29.671 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.672 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.672 DEBUG:boto:Params: {} 2023-10-11T14:58:29.672 DEBUG:boto:Token: None 2023-10-11T14:58:29.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9f3f4663698371f-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.718 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.719 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.719 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.720 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.720 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.720 DEBUG:boto:Method: GET 2023-10-11T14:58:29.721 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.721 DEBUG:boto:Data: 2023-10-11T14:58:29.722 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.722 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.723 DEBUG:boto:Params: {} 2023-10-11T14:58:29.723 DEBUG:boto:Token: None 2023-10-11T14:58:29.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe6a6e4c013cf91-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.728 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.728 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.729 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.729 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.729 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:58:29.730 DEBUG:boto:Data: 2023-10-11T14:58:29.730 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.731 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.731 DEBUG:boto:Params: {} 2023-10-11T14:58:29.732 DEBUG:boto:Token: None 2023-10-11T14:58:29.732 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1 2023-10-11T14:58:29.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2rxloblkVd7w95I5JgZAHwsEXmk= 2023-10-11T14:58:29.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rxloblkVd7w95I5JgZAHwsEXmk=', 'Content-Length': '0'} 2023-10-11T14:58:29.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9b5a3e1819e8dd-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.736 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.736 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.737 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.737 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:58:29.737 DEBUG:boto:Data: 2023-10-11T14:58:29.738 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.739 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.739 DEBUG:boto:Params: {} 2023-10-11T14:58:29.739 DEBUG:boto:Token: None 2023-10-11T14:58:29.741 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1 2023-10-11T14:58:29.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2rxloblkVd7w95I5JgZAHwsEXmk= 2023-10-11T14:58:29.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rxloblkVd7w95I5JgZAHwsEXmk=', 'Content-Length': '0'} 2023-10-11T14:58:29.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000303da781fe8aa65b-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:29.746 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.746 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.747 DEBUG:boto:Method: GET 2023-10-11T14:58:29.747 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:58:29.747 DEBUG:boto:Data: 2023-10-11T14:58:29.748 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.748 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.749 DEBUG:boto:Params: {} 2023-10-11T14:58:29.749 DEBUG:boto:Token: None 2023-10-11T14:58:29.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1 2023-10-11T14:58:29.750 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrJJ6snjAz3DRxOCgP6yF/PAmu0= 2023-10-11T14:58:29.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrJJ6snjAz3DRxOCgP6yF/PAmu0=', 'Content-Length': '0'} 2023-10-11T14:58:29.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a63173db58a832c-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.753 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:58:29.753 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:58:29.754 DEBUG:boto:Method: GET 2023-10-11T14:58:29.754 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:58:29.755 DEBUG:boto:Data: 2023-10-11T14:58:29.755 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.756 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.756 DEBUG:boto:Params: {} 2023-10-11T14:58:29.757 DEBUG:boto:Token: None 2023-10-11T14:58:29.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj1 2023-10-11T14:58:29.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrJJ6snjAz3DRxOCgP6yF/PAmu0= 2023-10-11T14:58:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrJJ6snjAz3DRxOCgP6yF/PAmu0=', 'Content-Length': '0'} 2023-10-11T14:58:29.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4fade825064321d-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.761 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.761 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.762 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.762 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.763 DEBUG:boto:Method: GET 2023-10-11T14:58:29.763 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.763 DEBUG:boto:Data: 2023-10-11T14:58:29.764 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.765 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.765 DEBUG:boto:Params: {} 2023-10-11T14:58:29.765 DEBUG:boto:Token: None 2023-10-11T14:58:29.766 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c04e6a87b73342f-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.770 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.770 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.771 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.771 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.772 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.772 DEBUG:boto:Method: GET 2023-10-11T14:58:29.772 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:58:29.773 DEBUG:boto:Data: 2023-10-11T14:58:29.773 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.774 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.774 DEBUG:boto:Params: {} 2023-10-11T14:58:29.775 DEBUG:boto:Token: None 2023-10-11T14:58:29.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004031e0ed9485d598-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.779 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:29.780 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.781 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.781 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.782 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.782 DEBUG:boto:Method: GET 2023-10-11T14:58:29.782 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.783 DEBUG:boto:Data: 2023-10-11T14:58:29.783 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.784 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.785 DEBUG:boto:Params: {} 2023-10-11T14:58:29.785 DEBUG:boto:Token: None 2023-10-11T14:58:29.786 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI= 2023-10-11T14:58:29.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI=', 'Content-Length': '0'} 2023-10-11T14:58:29.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0aa084093632d23-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.789 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.790 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.790 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.790 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.791 DEBUG:boto:Method: GET 2023-10-11T14:58:29.791 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.792 DEBUG:boto:Data: 2023-10-11T14:58:29.792 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.793 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.793 DEBUG:boto:Params: {} 2023-10-11T14:58:29.794 DEBUG:boto:Token: None 2023-10-11T14:58:29.794 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI= 2023-10-11T14:58:29.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI=', 'Content-Length': '0'} 2023-10-11T14:58:29.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dedd8878c8e2d311-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.798 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.798 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.799 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.799 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.799 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.800 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.800 DEBUG:boto:Data: 2023-10-11T14:58:29.801 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.801 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.802 DEBUG:boto:Params: {} 2023-10-11T14:58:29.802 DEBUG:boto:Token: None 2023-10-11T14:58:29.803 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KEU/RgUsO/ItYKww6yQ+FkgDOu8= 2023-10-11T14:58:29.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KEU/RgUsO/ItYKww6yQ+FkgDOu8=', 'Content-Length': '0'} 2023-10-11T14:58:29.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4951264fb84cd35-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.806 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.807 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.807 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.807 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.808 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.808 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.809 DEBUG:boto:Data: 2023-10-11T14:58:29.809 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.810 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.810 DEBUG:boto:Params: {} 2023-10-11T14:58:29.811 DEBUG:boto:Token: None 2023-10-11T14:58:29.811 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KEU/RgUsO/ItYKww6yQ+FkgDOu8= 2023-10-11T14:58:29.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KEU/RgUsO/ItYKww6yQ+FkgDOu8=', 'Content-Length': '0'} 2023-10-11T14:58:29.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051b40cff40b3dac2-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:29.815 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.816 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.816 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.817 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.817 DEBUG:boto:Method: GET 2023-10-11T14:58:29.817 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.818 DEBUG:boto:Data: 2023-10-11T14:58:29.818 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.819 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.820 DEBUG:boto:Params: {} 2023-10-11T14:58:29.820 DEBUG:boto:Token: None 2023-10-11T14:58:29.820 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI= 2023-10-11T14:58:29.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI=', 'Content-Length': '0'} 2023-10-11T14:58:29.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d4b171f248e688-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.824 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.824 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.825 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.825 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.826 DEBUG:boto:Method: GET 2023-10-11T14:58:29.826 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.826 DEBUG:boto:Data: 2023-10-11T14:58:29.827 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.828 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.828 DEBUG:boto:Params: {} 2023-10-11T14:58:29.828 DEBUG:boto:Token: None 2023-10-11T14:58:29.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:58:29.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI= 2023-10-11T14:58:29.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:j5Z44x+ZYdji0NO8gE5664qPvAI=', 'Content-Length': '0'} 2023-10-11T14:58:29.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee4dbd652a6734c4-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.832 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.833 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.833 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.834 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.834 DEBUG:boto:Method: GET 2023-10-11T14:58:29.835 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.835 DEBUG:boto:Data: 2023-10-11T14:58:29.836 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.837 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.837 DEBUG:boto:Params: {} 2023-10-11T14:58:29.837 DEBUG:boto:Token: None 2023-10-11T14:58:29.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45779182443b33b-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.882 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.883 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.883 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.884 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.884 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.885 DEBUG:boto:Method: GET 2023-10-11T14:58:29.885 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.885 DEBUG:boto:Data: 2023-10-11T14:58:29.886 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.887 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.887 DEBUG:boto:Params: {} 2023-10-11T14:58:29.888 DEBUG:boto:Token: None 2023-10-11T14:58:29.888 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ae99efdb3f5ee0-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.892 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.893 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.893 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.894 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.894 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:58:29.895 DEBUG:boto:Data: 2023-10-11T14:58:29.895 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.896 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.896 DEBUG:boto:Params: {} 2023-10-11T14:58:29.897 DEBUG:boto:Token: None 2023-10-11T14:58:29.897 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2 2023-10-11T14:58:29.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yye/n92Gzh8MVw6h+fZOeRx6gj0= 2023-10-11T14:58:29.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yye/n92Gzh8MVw6h+fZOeRx6gj0=', 'Content-Length': '0'} 2023-10-11T14:58:29.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1e181a25088641f-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.900 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.901 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.901 DEBUG:boto:Method: HEAD 2023-10-11T14:58:29.902 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:58:29.902 DEBUG:boto:Data: 2023-10-11T14:58:29.902 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.903 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.904 DEBUG:boto:Params: {} 2023-10-11T14:58:29.904 DEBUG:boto:Token: None 2023-10-11T14:58:29.905 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2 2023-10-11T14:58:29.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yye/n92Gzh8MVw6h+fZOeRx6gj0= 2023-10-11T14:58:29.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yye/n92Gzh8MVw6h+fZOeRx6gj0=', 'Content-Length': '0'} 2023-10-11T14:58:29.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000676b9273f10b63db-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:29.908 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.909 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.909 DEBUG:boto:Method: GET 2023-10-11T14:58:29.910 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:58:29.910 DEBUG:boto:Data: 2023-10-11T14:58:29.910 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.911 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.912 DEBUG:boto:Params: {} 2023-10-11T14:58:29.912 DEBUG:boto:Token: None 2023-10-11T14:58:29.912 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2 2023-10-11T14:58:29.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sxHQai+wlRpyi/kTbr1prE3g+Sc= 2023-10-11T14:58:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sxHQai+wlRpyi/kTbr1prE3g+Sc=', 'Content-Length': '0'} 2023-10-11T14:58:29.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afe33a321e792a0e-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.916 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:58:29.917 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:58:29.917 DEBUG:boto:Method: GET 2023-10-11T14:58:29.917 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:58:29.918 DEBUG:boto:Data: 2023-10-11T14:58:29.918 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.919 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.919 DEBUG:boto:Params: {} 2023-10-11T14:58:29.920 DEBUG:boto:Token: None 2023-10-11T14:58:29.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj2 2023-10-11T14:58:29.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sxHQai+wlRpyi/kTbr1prE3g+Sc= 2023-10-11T14:58:29.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sxHQai+wlRpyi/kTbr1prE3g+Sc=', 'Content-Length': '0'} 2023-10-11T14:58:29.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8d035dc97c2c5b5-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.966 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.967 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.967 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.968 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.968 DEBUG:boto:Method: GET 2023-10-11T14:58:29.968 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.969 DEBUG:boto:Data: 2023-10-11T14:58:29.969 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.970 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.970 DEBUG:boto:Params: {} 2023-10-11T14:58:29.971 DEBUG:boto:Token: None 2023-10-11T14:58:29.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe1aef273c68233-006526b815-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.975 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.976 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:29.976 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:29.977 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.977 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.977 DEBUG:boto:Method: GET 2023-10-11T14:58:29.978 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:58:29.978 DEBUG:boto:Data: 2023-10-11T14:58:29.979 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.979 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.980 DEBUG:boto:Params: {} 2023-10-11T14:58:29.980 DEBUG:boto:Token: None 2023-10-11T14:58:29.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/ 2023-10-11T14:58:29.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g= 2023-10-11T14:58:29.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zE2FxCp00bz7uQOsJDwH8azVm5g=', 'Content-Length': '0'} 2023-10-11T14:58:29.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293ac317fcd25a86-006526b815-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.985 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:29.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:29.986 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:29.987 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:29.987 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.987 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.988 DEBUG:boto:Method: GET 2023-10-11T14:58:29.988 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.989 DEBUG:boto:Data: 2023-10-11T14:58:29.989 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:29.990 DEBUG:boto:Port: 8001 2023-10-11T14:58:29.990 DEBUG:boto:Params: {} 2023-10-11T14:58:29.991 DEBUG:boto:Token: None 2023-10-11T14:58:29.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h2WjCTuO6aAwQqPTPANu3QjNTh8= 2023-10-11T14:58:29.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h2WjCTuO6aAwQqPTPANu3QjNTh8=', 'Content-Length': '0'} 2023-10-11T14:58:29.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d303784dc594197b-006526b815-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:29.995 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:29.995 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:29.996 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.996 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.996 DEBUG:boto:Method: GET 2023-10-11T14:58:29.997 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:29.997 DEBUG:boto:Data: 2023-10-11T14:58:29.997 DEBUG:boto:Headers: {} 2023-10-11T14:58:29.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:29.998 DEBUG:boto:Port: 8000 2023-10-11T14:58:29.999 DEBUG:boto:Params: {} 2023-10-11T14:58:29.999 DEBUG:boto:Token: None 2023-10-11T14:58:30.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:29 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h2WjCTuO6aAwQqPTPANu3QjNTh8= 2023-10-11T14:58:30.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h2WjCTuO6aAwQqPTPANu3QjNTh8=', 'Content-Length': '0'} 2023-10-11T14:58:30.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000689dd54e01c84619-006526b815-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.003 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.004 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.004 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.004 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.005 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.005 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.006 DEBUG:boto:Data: 2023-10-11T14:58:30.006 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.007 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.007 DEBUG:boto:Params: {} 2023-10-11T14:58:30.008 DEBUG:boto:Token: None 2023-10-11T14:58:30.008 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SYtJbJ9Djo+uHtVBpU3TJclsf/U= 2023-10-11T14:58:30.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SYtJbJ9Djo+uHtVBpU3TJclsf/U=', 'Content-Length': '0'} 2023-10-11T14:58:30.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000206157222bba3a56-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.011 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.012 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.012 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.012 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.013 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.013 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.014 DEBUG:boto:Data: 2023-10-11T14:58:30.014 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.015 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.015 DEBUG:boto:Params: {} 2023-10-11T14:58:30.016 DEBUG:boto:Token: None 2023-10-11T14:58:30.016 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SYtJbJ9Djo+uHtVBpU3TJclsf/U= 2023-10-11T14:58:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SYtJbJ9Djo+uHtVBpU3TJclsf/U=', 'Content-Length': '0'} 2023-10-11T14:58:30.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005312c7c01fa6594e-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:30.020 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.020 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.021 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.021 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.022 DEBUG:boto:Method: GET 2023-10-11T14:58:30.022 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.023 DEBUG:boto:Data: 2023-10-11T14:58:30.023 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.024 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.025 DEBUG:boto:Params: {} 2023-10-11T14:58:30.025 DEBUG:boto:Token: None 2023-10-11T14:58:30.026 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rdvIKf0tujrQOHziQ+OWQyGtJu8= 2023-10-11T14:58:30.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rdvIKf0tujrQOHziQ+OWQyGtJu8=', 'Content-Length': '0'} 2023-10-11T14:58:30.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243ff754360e22c2-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.070 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.071 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.071 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.071 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.072 DEBUG:boto:Method: GET 2023-10-11T14:58:30.072 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.073 DEBUG:boto:Data: 2023-10-11T14:58:30.073 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.073 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.074 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.074 DEBUG:boto:Params: {} 2023-10-11T14:58:30.075 DEBUG:boto:Token: None 2023-10-11T14:58:30.075 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:58:30.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rdvIKf0tujrQOHziQ+OWQyGtJu8= 2023-10-11T14:58:30.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rdvIKf0tujrQOHziQ+OWQyGtJu8=', 'Content-Length': '0'} 2023-10-11T14:58:30.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000551a626b19fe91d1-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.079 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.079 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.080 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.080 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.080 DEBUG:boto:Method: GET 2023-10-11T14:58:30.081 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.081 DEBUG:boto:Data: 2023-10-11T14:58:30.082 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.082 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.083 DEBUG:boto:Params: {} 2023-10-11T14:58:30.083 DEBUG:boto:Token: None 2023-10-11T14:58:30.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2235ebef7b8da4e-006526b816-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.088 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.088 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.089 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.089 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.090 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.090 DEBUG:boto:Method: GET 2023-10-11T14:58:30.090 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.091 DEBUG:boto:Data: 2023-10-11T14:58:30.091 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.092 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.092 DEBUG:boto:Params: {} 2023-10-11T14:58:30.093 DEBUG:boto:Token: None 2023-10-11T14:58:30.093 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694ea3272e39d288-006526b816-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.138 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.139 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.139 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.140 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.140 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:58:30.140 DEBUG:boto:Data: 2023-10-11T14:58:30.141 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.142 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.142 DEBUG:boto:Params: {} 2023-10-11T14:58:30.143 DEBUG:boto:Token: None 2023-10-11T14:58:30.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3 2023-10-11T14:58:30.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aF1G6eqtZCxVkO4U2yTyV+0FFZs= 2023-10-11T14:58:30.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aF1G6eqtZCxVkO4U2yTyV+0FFZs=', 'Content-Length': '0'} 2023-10-11T14:58:30.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000882014f844034edb-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.149 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.149 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.150 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.150 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:58:30.151 DEBUG:boto:Data: 2023-10-11T14:58:30.151 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.152 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.152 DEBUG:boto:Params: {} 2023-10-11T14:58:30.152 DEBUG:boto:Token: None 2023-10-11T14:58:30.153 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3 2023-10-11T14:58:30.153 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aF1G6eqtZCxVkO4U2yTyV+0FFZs= 2023-10-11T14:58:30.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aF1G6eqtZCxVkO4U2yTyV+0FFZs=', 'Content-Length': '0'} 2023-10-11T14:58:30.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c86661aa519c420-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:30.157 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.157 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.158 DEBUG:boto:Method: GET 2023-10-11T14:58:30.158 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:58:30.159 DEBUG:boto:Data: 2023-10-11T14:58:30.159 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.160 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.160 DEBUG:boto:Params: {} 2023-10-11T14:58:30.161 DEBUG:boto:Token: None 2023-10-11T14:58:30.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3 2023-10-11T14:58:30.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N7eZ8mwjPS7vTSoThj89ypk7NDw= 2023-10-11T14:58:30.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N7eZ8mwjPS7vTSoThj89ypk7NDw=', 'Content-Length': '0'} 2023-10-11T14:58:30.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3d192a658a2af54-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.165 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:58:30.165 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:58:30.166 DEBUG:boto:Method: GET 2023-10-11T14:58:30.166 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:58:30.166 DEBUG:boto:Data: 2023-10-11T14:58:30.167 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.167 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.168 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.168 DEBUG:boto:Params: {} 2023-10-11T14:58:30.168 DEBUG:boto:Token: None 2023-10-11T14:58:30.169 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj3 2023-10-11T14:58:30.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N7eZ8mwjPS7vTSoThj89ypk7NDw= 2023-10-11T14:58:30.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N7eZ8mwjPS7vTSoThj89ypk7NDw=', 'Content-Length': '0'} 2023-10-11T14:58:30.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de4f4335d2518421-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.172 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.173 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.173 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.173 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.174 DEBUG:boto:Method: GET 2023-10-11T14:58:30.174 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.175 DEBUG:boto:Data: 2023-10-11T14:58:30.175 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.176 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.176 DEBUG:boto:Params: {} 2023-10-11T14:58:30.177 DEBUG:boto:Token: None 2023-10-11T14:58:30.177 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.178 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380e02e0df86b4d5-006526b816-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.181 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.182 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.182 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.183 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.183 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.183 DEBUG:boto:Method: GET 2023-10-11T14:58:30.184 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:58:30.184 DEBUG:boto:Data: 2023-10-11T14:58:30.184 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.185 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.186 DEBUG:boto:Params: {} 2023-10-11T14:58:30.186 DEBUG:boto:Token: None 2023-10-11T14:58:30.186 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9b8dd66c8bf870f-006526b816-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.190 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:30.191 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.192 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.192 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.192 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.193 DEBUG:boto:Method: GET 2023-10-11T14:58:30.193 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.194 DEBUG:boto:Data: 2023-10-11T14:58:30.194 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.195 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.195 DEBUG:boto:Params: {} 2023-10-11T14:58:30.196 DEBUG:boto:Token: None 2023-10-11T14:58:30.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg= 2023-10-11T14:58:30.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg=', 'Content-Length': '0'} 2023-10-11T14:58:30.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa101dbb07e31bb0-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.200 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.200 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.200 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.201 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.201 DEBUG:boto:Method: GET 2023-10-11T14:58:30.202 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.202 DEBUG:boto:Data: 2023-10-11T14:58:30.202 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.203 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.203 DEBUG:boto:Params: {} 2023-10-11T14:58:30.204 DEBUG:boto:Token: None 2023-10-11T14:58:30.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg= 2023-10-11T14:58:30.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg=', 'Content-Length': '0'} 2023-10-11T14:58:30.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dfc7eef1ff2029c-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.208 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.208 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.209 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.209 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.209 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.210 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.210 DEBUG:boto:Data: 2023-10-11T14:58:30.210 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.211 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.211 DEBUG:boto:Params: {} 2023-10-11T14:58:30.212 DEBUG:boto:Token: None 2023-10-11T14:58:30.212 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqBH/Ll+wEHeyEwNDpQJEvsC1D8= 2023-10-11T14:58:30.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqBH/Ll+wEHeyEwNDpQJEvsC1D8=', 'Content-Length': '0'} 2023-10-11T14:58:30.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033b3d066f59fab1a-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.216 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.216 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.216 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.217 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.217 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.218 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.218 DEBUG:boto:Data: 2023-10-11T14:58:30.218 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.219 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.220 DEBUG:boto:Params: {} 2023-10-11T14:58:30.220 DEBUG:boto:Token: None 2023-10-11T14:58:30.220 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqBH/Ll+wEHeyEwNDpQJEvsC1D8= 2023-10-11T14:58:30.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqBH/Ll+wEHeyEwNDpQJEvsC1D8=', 'Content-Length': '0'} 2023-10-11T14:58:30.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b80bbc9b34eb231-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:30.224 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.225 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.225 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.225 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.226 DEBUG:boto:Method: GET 2023-10-11T14:58:30.226 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.227 DEBUG:boto:Data: 2023-10-11T14:58:30.227 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.228 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.228 DEBUG:boto:Params: {} 2023-10-11T14:58:30.229 DEBUG:boto:Token: None 2023-10-11T14:58:30.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg= 2023-10-11T14:58:30.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg=', 'Content-Length': '0'} 2023-10-11T14:58:30.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f18bd25a5f18088-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.274 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.275 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.275 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.276 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.276 DEBUG:boto:Method: GET 2023-10-11T14:58:30.277 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.277 DEBUG:boto:Data: 2023-10-11T14:58:30.278 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.279 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.279 DEBUG:boto:Params: {} 2023-10-11T14:58:30.279 DEBUG:boto:Token: None 2023-10-11T14:58:30.280 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:58:30.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg= 2023-10-11T14:58:30.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ceeyUR5B/wCUqub1PnvwK9oUdmg=', 'Content-Length': '0'} 2023-10-11T14:58:30.283 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f031e2c1f5789a97-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.283 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.284 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.284 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.285 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.285 DEBUG:boto:Method: GET 2023-10-11T14:58:30.285 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.286 DEBUG:boto:Data: 2023-10-11T14:58:30.286 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.287 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.287 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.288 DEBUG:boto:Params: {} 2023-10-11T14:58:30.288 DEBUG:boto:Token: None 2023-10-11T14:58:30.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed1e49078a23b806-006526b816-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.292 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.293 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.293 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.294 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.294 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.295 DEBUG:boto:Method: GET 2023-10-11T14:58:30.295 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.295 DEBUG:boto:Data: 2023-10-11T14:58:30.296 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.297 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.297 DEBUG:boto:Params: {} 2023-10-11T14:58:30.297 DEBUG:boto:Token: None 2023-10-11T14:58:30.298 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580d4dcf42ab0d65-006526b816-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.342 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.343 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.343 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.344 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.344 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:58:30.345 DEBUG:boto:Data: 2023-10-11T14:58:30.345 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.346 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.346 DEBUG:boto:Params: {} 2023-10-11T14:58:30.349 DEBUG:boto:Token: None 2023-10-11T14:58:30.350 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4 2023-10-11T14:58:30.350 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wrxdGM66dpBhPCL6ih1qqhzSUuE= 2023-10-11T14:58:30.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wrxdGM66dpBhPCL6ih1qqhzSUuE=', 'Content-Length': '0'} 2023-10-11T14:58:30.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec6e4768466ea0b6-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.353 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.354 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.354 DEBUG:boto:Method: HEAD 2023-10-11T14:58:30.354 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:58:30.355 DEBUG:boto:Data: 2023-10-11T14:58:30.355 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.356 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.356 DEBUG:boto:Params: {} 2023-10-11T14:58:30.357 DEBUG:boto:Token: None 2023-10-11T14:58:30.357 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4 2023-10-11T14:58:30.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wrxdGM66dpBhPCL6ih1qqhzSUuE= 2023-10-11T14:58:30.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wrxdGM66dpBhPCL6ih1qqhzSUuE=', 'Content-Length': '0'} 2023-10-11T14:58:30.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080e7b6dfd5918ade-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:30.361 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.361 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.361 DEBUG:boto:Method: GET 2023-10-11T14:58:30.362 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:58:30.362 DEBUG:boto:Data: 2023-10-11T14:58:30.363 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.363 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.364 DEBUG:boto:Params: {} 2023-10-11T14:58:30.364 DEBUG:boto:Token: None 2023-10-11T14:58:30.365 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4 2023-10-11T14:58:30.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WG/2p7xEzbTqb9PVirS9OU9ev58= 2023-10-11T14:58:30.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WG/2p7xEzbTqb9PVirS9OU9ev58=', 'Content-Length': '0'} 2023-10-11T14:58:30.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000feea11be76c1742d-006526b816-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.368 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:58:30.369 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:58:30.369 DEBUG:boto:Method: GET 2023-10-11T14:58:30.370 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:58:30.370 DEBUG:boto:Data: 2023-10-11T14:58:30.371 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.371 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.372 DEBUG:boto:Params: {} 2023-10-11T14:58:30.372 DEBUG:boto:Token: None 2023-10-11T14:58:30.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/obj4 2023-10-11T14:58:30.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WG/2p7xEzbTqb9PVirS9OU9ev58= 2023-10-11T14:58:30.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WG/2p7xEzbTqb9PVirS9OU9ev58=', 'Content-Length': '0'} 2023-10-11T14:58:30.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0d348a09b3fd20c-006526b816-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.376 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.376 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.377 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.377 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.378 DEBUG:boto:Method: GET 2023-10-11T14:58:30.378 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.378 DEBUG:boto:Data: 2023-10-11T14:58:30.379 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:30.380 DEBUG:boto:Port: 8001 2023-10-11T14:58:30.380 DEBUG:boto:Params: {} 2023-10-11T14:58:30.380 DEBUG:boto:Token: None 2023-10-11T14:58:30.381 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4bb5e4b9f6caae5-006526b816-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.385 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.386 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:30.386 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:30.386 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.387 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.387 DEBUG:boto:Method: GET 2023-10-11T14:58:30.388 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:58:30.388 DEBUG:boto:Data: 2023-10-11T14:58:30.388 DEBUG:boto:Headers: {} 2023-10-11T14:58:30.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:30.389 DEBUG:boto:Port: 8000 2023-10-11T14:58:30.389 DEBUG:boto:Params: {} 2023-10-11T14:58:30.390 DEBUG:boto:Token: None 2023-10-11T14:58:30.390 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:30 GMT /dghqvd-46/ 2023-10-11T14:58:30.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A= 2023-10-11T14:58:30.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JPtvM4ny/8azByz8ARBCHP1ui2A=', 'Content-Length': '0'} 2023-10-11T14:58:30.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83b0128d2a9cf8c-006526b816-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:30.394 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:30.395 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-46 zones={a2, a1} 2023-10-11T14:58:30.395 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-47 2023-10-11T14:58:30.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:30.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:30.397 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:58:30.460 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:30.460 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:30.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:30.496+0000 7f115bec5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:30.499 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , , , , ] 2023-10-11T14:58:30.499 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:35.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:35.497+0000 7f115bec5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:35.501 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , , , , ] 2023-10-11T14:58:35.501 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:40.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:40.498+0000 7f115bec5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:40.502 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , , , , ] 2023-10-11T14:58:40.502 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:45.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:45.500+0000 7f115bec5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:45.503 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , , , , ] 2023-10-11T14:58:45.503 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:50.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:50.501+0000 7f115bec5780 1 bucket sync caught up with source: 2023-10-11T14:58:50.504 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:50.504 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.17.6, , 00000000001.16.6, , 00000000001.21.6, , , 00000000001.26.6, , , ] 2023-10-11T14:58:50.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:50.501+0000 7f115bec5780 0 bucket checkpoint complete 2023-10-11T14:58:50.520 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-47 2023-10-11T14:58:50.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:50.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:50.521 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:58:50.582 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:50.582 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:50.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:50.615+0000 7f029b369780 1 bucket sync caught up with source: 2023-10-11T14:58:50.619 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.134.6, , 00000000001.132.6, , 00000000001.126.6, , , 00000000001.204.6, , , ] 2023-10-11T14:58:50.619 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.134.6, , 00000000001.132.6, , 00000000001.126.6, , , 00000000001.204.6, , , ] 2023-10-11T14:58:50.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:50.615+0000 7f029b369780 0 bucket checkpoint complete 2023-10-11T14:58:50.631 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-47 zones={a2, a1} 2023-10-11T14:58:50.631 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.632 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.632 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.632 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:50.633 DEBUG:boto:Data: 2023-10-11T14:58:50.633 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.634 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.635 DEBUG:boto:Params: {} 2023-10-11T14:58:50.635 DEBUG:boto:Token: None 2023-10-11T14:58:50.636 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A/jAeS8/LSJcMaQmzmKTpbrvY7Q= 2023-10-11T14:58:50.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A/jAeS8/LSJcMaQmzmKTpbrvY7Q=', 'Content-Length': '0'} 2023-10-11T14:58:50.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6a4c8f49e79fcb5-006526b82a-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.640 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.640 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.640 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.641 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:50.641 DEBUG:boto:Data: 2023-10-11T14:58:50.641 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.642 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.643 DEBUG:boto:Params: {} 2023-10-11T14:58:50.643 DEBUG:boto:Token: None 2023-10-11T14:58:50.643 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A/jAeS8/LSJcMaQmzmKTpbrvY7Q= 2023-10-11T14:58:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A/jAeS8/LSJcMaQmzmKTpbrvY7Q=', 'Content-Length': '0'} 2023-10-11T14:58:50.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008953d8d11a31f7d1-006526b82a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.647 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:58:50.647 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.648 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.648 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.648 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.649 DEBUG:boto:Method: GET 2023-10-11T14:58:50.649 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.649 DEBUG:boto:Data: 2023-10-11T14:58:50.650 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.651 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.651 DEBUG:boto:Params: {} 2023-10-11T14:58:50.651 DEBUG:boto:Token: None 2023-10-11T14:58:50.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/?versions 2023-10-11T14:58:50.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA= 2023-10-11T14:58:50.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA=', 'Content-Length': '0'} 2023-10-11T14:58:50.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c60883b59c6961-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.655 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:58:50.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:58:50.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:58:50.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:58:50.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:58:50.658 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.658 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.659 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.659 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.660 DEBUG:boto:Method: GET 2023-10-11T14:58:50.660 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.660 DEBUG:boto:Data: 2023-10-11T14:58:50.661 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.662 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.662 DEBUG:boto:Params: {} 2023-10-11T14:58:50.662 DEBUG:boto:Token: None 2023-10-11T14:58:50.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/?versions 2023-10-11T14:58:50.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA= 2023-10-11T14:58:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA=', 'Content-Length': '0'} 2023-10-11T14:58:50.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4804c22b40f65fd-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.667 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:58:50.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:58:50.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:58:50.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:58:50.669 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.669 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.670 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.670 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.671 DEBUG:boto:Method: GET 2023-10-11T14:58:50.671 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.671 DEBUG:boto:Data: 2023-10-11T14:58:50.672 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.672 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.673 DEBUG:boto:Params: {} 2023-10-11T14:58:50.673 DEBUG:boto:Token: None 2023-10-11T14:58:50.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/?versions 2023-10-11T14:58:50.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA= 2023-10-11T14:58:50.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA=', 'Content-Length': '0'} 2023-10-11T14:58:50.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e52ebc7b2b4647-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.678 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.678 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.679 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.679 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.679 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.680 DEBUG:boto:Method: GET 2023-10-11T14:58:50.680 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:58:50.681 DEBUG:boto:Data: 2023-10-11T14:58:50.681 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.682 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.682 DEBUG:boto:Params: {} 2023-10-11T14:58:50.683 DEBUG:boto:Token: None 2023-10-11T14:58:50.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/?versions 2023-10-11T14:58:50.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA= 2023-10-11T14:58:50.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l+DVOTqnDiF/Teeai+JEvi64WeA=', 'Content-Length': '0'} 2023-10-11T14:58:50.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d72e2620d27bbd-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.687 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:50.688 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.688 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.689 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.689 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.690 DEBUG:boto:Method: GET 2023-10-11T14:58:50.690 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.690 DEBUG:boto:Data: 2023-10-11T14:58:50.691 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.692 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.692 DEBUG:boto:Params: {} 2023-10-11T14:58:50.692 DEBUG:boto:Token: None 2023-10-11T14:58:50.693 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA= 2023-10-11T14:58:50.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA=', 'Content-Length': '0'} 2023-10-11T14:58:50.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea62441e89470ede-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.738 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.739 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.739 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.740 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.740 DEBUG:boto:Method: GET 2023-10-11T14:58:50.740 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.741 DEBUG:boto:Data: 2023-10-11T14:58:50.741 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.742 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.742 DEBUG:boto:Params: {} 2023-10-11T14:58:50.743 DEBUG:boto:Token: None 2023-10-11T14:58:50.743 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA= 2023-10-11T14:58:50.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA=', 'Content-Length': '0'} 2023-10-11T14:58:50.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4187da45762e918-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.746 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.747 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.747 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.748 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.748 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.748 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.749 DEBUG:boto:Data: 2023-10-11T14:58:50.749 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.750 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.750 DEBUG:boto:Params: {} 2023-10-11T14:58:50.751 DEBUG:boto:Token: None 2023-10-11T14:58:50.751 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jGkKR24MkXC9qNIxDXoO2b/z554= 2023-10-11T14:58:50.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jGkKR24MkXC9qNIxDXoO2b/z554=', 'Content-Length': '0'} 2023-10-11T14:58:50.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006a038d067636637-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.754 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.754 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.755 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.755 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.756 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.756 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.756 DEBUG:boto:Data: 2023-10-11T14:58:50.757 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.757 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.758 DEBUG:boto:Params: {} 2023-10-11T14:58:50.758 DEBUG:boto:Token: None 2023-10-11T14:58:50.759 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.759 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jGkKR24MkXC9qNIxDXoO2b/z554= 2023-10-11T14:58:50.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jGkKR24MkXC9qNIxDXoO2b/z554=', 'Content-Length': '0'} 2023-10-11T14:58:50.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027afb710e37949eb-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:50.762 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.762 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.763 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.763 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.763 DEBUG:boto:Method: GET 2023-10-11T14:58:50.764 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.764 DEBUG:boto:Data: 2023-10-11T14:58:50.765 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.765 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.766 DEBUG:boto:Params: {} 2023-10-11T14:58:50.766 DEBUG:boto:Token: None 2023-10-11T14:58:50.767 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA= 2023-10-11T14:58:50.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA=', 'Content-Length': '0'} 2023-10-11T14:58:50.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000992d13a7d8190f39-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.770 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.770 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.770 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.771 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.771 DEBUG:boto:Method: GET 2023-10-11T14:58:50.772 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.772 DEBUG:boto:Data: 2023-10-11T14:58:50.772 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.773 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.774 DEBUG:boto:Params: {} 2023-10-11T14:58:50.774 DEBUG:boto:Token: None 2023-10-11T14:58:50.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:58:50.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA= 2023-10-11T14:58:50.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odm0r0I8cxTBqVKWNNs5t+TF+yA=', 'Content-Length': '0'} 2023-10-11T14:58:50.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d8fd38224cf4b32-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.778 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.778 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.778 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.779 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.779 DEBUG:boto:Method: GET 2023-10-11T14:58:50.780 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.780 DEBUG:boto:Data: 2023-10-11T14:58:50.780 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.781 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.781 DEBUG:boto:Params: {} 2023-10-11T14:58:50.782 DEBUG:boto:Token: None 2023-10-11T14:58:50.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5d9eb607496c7d-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.786 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.786 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.787 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.787 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.788 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.788 DEBUG:boto:Method: GET 2023-10-11T14:58:50.788 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.789 DEBUG:boto:Data: 2023-10-11T14:58:50.789 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.790 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.790 DEBUG:boto:Params: {} 2023-10-11T14:58:50.791 DEBUG:boto:Token: None 2023-10-11T14:58:50.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001650566c0f7297c1-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.795 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.795 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.795 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.796 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.796 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:58:50.797 DEBUG:boto:Data: 2023-10-11T14:58:50.797 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.798 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.798 DEBUG:boto:Params: {} 2023-10-11T14:58:50.798 DEBUG:boto:Token: None 2023-10-11T14:58:50.799 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1 2023-10-11T14:58:50.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsBAp3bYbDUFyKgRn2pb0EHMTBw= 2023-10-11T14:58:50.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsBAp3bYbDUFyKgRn2pb0EHMTBw=', 'Content-Length': '0'} 2023-10-11T14:58:50.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001ebef39c99dd6bd-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.802 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.802 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.803 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.803 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:58:50.803 DEBUG:boto:Data: 2023-10-11T14:58:50.804 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.804 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.804 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.805 DEBUG:boto:Params: {} 2023-10-11T14:58:50.805 DEBUG:boto:Token: None 2023-10-11T14:58:50.805 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1 2023-10-11T14:58:50.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsBAp3bYbDUFyKgRn2pb0EHMTBw= 2023-10-11T14:58:50.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsBAp3bYbDUFyKgRn2pb0EHMTBw=', 'Content-Length': '0'} 2023-10-11T14:58:50.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044af343b8260d8bf-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:58:50.809 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.809 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.810 DEBUG:boto:Method: GET 2023-10-11T14:58:50.810 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:58:50.810 DEBUG:boto:Data: 2023-10-11T14:58:50.811 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.811 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.813 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.813 DEBUG:boto:Params: {} 2023-10-11T14:58:50.814 DEBUG:boto:Token: None 2023-10-11T14:58:50.814 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1 2023-10-11T14:58:50.815 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kuT2hJ0/PGLn9r42XSwZQMyoDvA= 2023-10-11T14:58:50.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kuT2hJ0/PGLn9r42XSwZQMyoDvA=', 'Content-Length': '0'} 2023-10-11T14:58:50.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfd94b8c667c3236-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.818 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:58:50.818 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:58:50.819 DEBUG:boto:Method: GET 2023-10-11T14:58:50.819 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:58:50.819 DEBUG:boto:Data: 2023-10-11T14:58:50.820 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.820 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.821 DEBUG:boto:Params: {} 2023-10-11T14:58:50.821 DEBUG:boto:Token: None 2023-10-11T14:58:50.822 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj1 2023-10-11T14:58:50.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kuT2hJ0/PGLn9r42XSwZQMyoDvA= 2023-10-11T14:58:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kuT2hJ0/PGLn9r42XSwZQMyoDvA=', 'Content-Length': '0'} 2023-10-11T14:58:50.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fde6e2208a78171d-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.825 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.825 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.826 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.826 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.826 DEBUG:boto:Method: GET 2023-10-11T14:58:50.827 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.827 DEBUG:boto:Data: 2023-10-11T14:58:50.828 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.828 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.828 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.829 DEBUG:boto:Params: {} 2023-10-11T14:58:50.829 DEBUG:boto:Token: None 2023-10-11T14:58:50.830 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000621b8aa0f3e7e0c4-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.833 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.834 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.834 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.834 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.835 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.835 DEBUG:boto:Method: GET 2023-10-11T14:58:50.835 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:58:50.836 DEBUG:boto:Data: 2023-10-11T14:58:50.836 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.837 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.837 DEBUG:boto:Params: {} 2023-10-11T14:58:50.838 DEBUG:boto:Token: None 2023-10-11T14:58:50.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28a1fb68a2af6b3-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.842 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:50.843 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.843 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.843 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.844 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.844 DEBUG:boto:Method: GET 2023-10-11T14:58:50.844 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.845 DEBUG:boto:Data: 2023-10-11T14:58:50.845 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.846 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.846 DEBUG:boto:Params: {} 2023-10-11T14:58:50.847 DEBUG:boto:Token: None 2023-10-11T14:58:50.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k= 2023-10-11T14:58:50.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k=', 'Content-Length': '0'} 2023-10-11T14:58:50.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007dedfc2eaed8bc00-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.851 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.851 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.851 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.852 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.852 DEBUG:boto:Method: GET 2023-10-11T14:58:50.852 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.853 DEBUG:boto:Data: 2023-10-11T14:58:50.853 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.854 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.854 DEBUG:boto:Params: {} 2023-10-11T14:58:50.855 DEBUG:boto:Token: None 2023-10-11T14:58:50.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k= 2023-10-11T14:58:50.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k=', 'Content-Length': '0'} 2023-10-11T14:58:50.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c60737d3efaafae5-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.858 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.859 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.859 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.860 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.860 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.860 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.861 DEBUG:boto:Data: 2023-10-11T14:58:50.861 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.862 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.862 DEBUG:boto:Params: {} 2023-10-11T14:58:50.862 DEBUG:boto:Token: None 2023-10-11T14:58:50.863 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDT7OnWSKxExNL9SGzMpUgPRxUk= 2023-10-11T14:58:50.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDT7OnWSKxExNL9SGzMpUgPRxUk=', 'Content-Length': '0'} 2023-10-11T14:58:50.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002758ba4c46fb699f-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.866 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.866 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.866 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.867 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.867 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.867 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.868 DEBUG:boto:Data: 2023-10-11T14:58:50.868 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.869 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.869 DEBUG:boto:Params: {} 2023-10-11T14:58:50.870 DEBUG:boto:Token: None 2023-10-11T14:58:50.870 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDT7OnWSKxExNL9SGzMpUgPRxUk= 2023-10-11T14:58:50.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDT7OnWSKxExNL9SGzMpUgPRxUk=', 'Content-Length': '0'} 2023-10-11T14:58:50.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075d5105b96f74e0e-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:50.873 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.874 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.874 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.875 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.875 DEBUG:boto:Method: GET 2023-10-11T14:58:50.875 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.876 DEBUG:boto:Data: 2023-10-11T14:58:50.876 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.877 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.877 DEBUG:boto:Params: {} 2023-10-11T14:58:50.878 DEBUG:boto:Token: None 2023-10-11T14:58:50.878 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k= 2023-10-11T14:58:50.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k=', 'Content-Length': '0'} 2023-10-11T14:58:50.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048515000037270c6-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.881 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.882 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.882 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.882 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.883 DEBUG:boto:Method: GET 2023-10-11T14:58:50.883 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.884 DEBUG:boto:Data: 2023-10-11T14:58:50.884 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.885 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.885 DEBUG:boto:Params: {} 2023-10-11T14:58:50.886 DEBUG:boto:Token: None 2023-10-11T14:58:50.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:58:50.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k= 2023-10-11T14:58:50.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Me1mC9yIwKRO2gYUrkqfl67Mw/k=', 'Content-Length': '0'} 2023-10-11T14:58:50.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be55cf40eddefc36-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.889 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.890 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.890 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.890 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.891 DEBUG:boto:Method: GET 2023-10-11T14:58:50.891 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.892 DEBUG:boto:Data: 2023-10-11T14:58:50.892 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.893 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.893 DEBUG:boto:Params: {} 2023-10-11T14:58:50.893 DEBUG:boto:Token: None 2023-10-11T14:58:50.894 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea1931ea8fedc09-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.897 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.898 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.898 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.899 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.899 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.900 DEBUG:boto:Method: GET 2023-10-11T14:58:50.900 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.900 DEBUG:boto:Data: 2023-10-11T14:58:50.901 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.901 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.902 DEBUG:boto:Params: {} 2023-10-11T14:58:50.902 DEBUG:boto:Token: None 2023-10-11T14:58:50.903 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b3eea337220758-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.906 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.907 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.907 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.908 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.908 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:58:50.908 DEBUG:boto:Data: 2023-10-11T14:58:50.909 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.909 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.910 DEBUG:boto:Params: {} 2023-10-11T14:58:50.910 DEBUG:boto:Token: None 2023-10-11T14:58:50.911 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2 2023-10-11T14:58:50.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DW4danOgqvcFB3ZHByWrDUXZuyw= 2023-10-11T14:58:50.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DW4danOgqvcFB3ZHByWrDUXZuyw=', 'Content-Length': '0'} 2023-10-11T14:58:50.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008852030f5f48285a-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.914 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.914 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.914 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.915 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:58:50.915 DEBUG:boto:Data: 2023-10-11T14:58:50.916 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.916 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.917 DEBUG:boto:Params: {} 2023-10-11T14:58:50.917 DEBUG:boto:Token: None 2023-10-11T14:58:50.917 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2 2023-10-11T14:58:50.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DW4danOgqvcFB3ZHByWrDUXZuyw= 2023-10-11T14:58:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DW4danOgqvcFB3ZHByWrDUXZuyw=', 'Content-Length': '0'} 2023-10-11T14:58:50.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057f4a1031b1c74ae-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:58:50.922 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.922 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.923 DEBUG:boto:Method: GET 2023-10-11T14:58:50.923 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:58:50.923 DEBUG:boto:Data: 2023-10-11T14:58:50.924 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.924 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.925 DEBUG:boto:Params: {} 2023-10-11T14:58:50.925 DEBUG:boto:Token: None 2023-10-11T14:58:50.926 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2 2023-10-11T14:58:50.926 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QRf53zDSbXozbA3uHO41pzq/v94= 2023-10-11T14:58:50.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QRf53zDSbXozbA3uHO41pzq/v94=', 'Content-Length': '0'} 2023-10-11T14:58:50.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cdbf1e3645dc2b3-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.929 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:58:50.930 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:58:50.930 DEBUG:boto:Method: GET 2023-10-11T14:58:50.931 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:58:50.931 DEBUG:boto:Data: 2023-10-11T14:58:50.931 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.932 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.932 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.932 DEBUG:boto:Params: {} 2023-10-11T14:58:50.933 DEBUG:boto:Token: None 2023-10-11T14:58:50.933 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj2 2023-10-11T14:58:50.934 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QRf53zDSbXozbA3uHO41pzq/v94= 2023-10-11T14:58:50.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QRf53zDSbXozbA3uHO41pzq/v94=', 'Content-Length': '0'} 2023-10-11T14:58:50.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077a738c4a864faec-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.937 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.937 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.938 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.938 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.938 DEBUG:boto:Method: GET 2023-10-11T14:58:50.939 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.939 DEBUG:boto:Data: 2023-10-11T14:58:50.940 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.940 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.941 DEBUG:boto:Params: {} 2023-10-11T14:58:50.941 DEBUG:boto:Token: None 2023-10-11T14:58:50.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c068ecdfcf8e2cf4-006526b82a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.945 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.946 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:50.946 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:50.947 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.947 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.947 DEBUG:boto:Method: GET 2023-10-11T14:58:50.948 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:58:50.948 DEBUG:boto:Data: 2023-10-11T14:58:50.949 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.949 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.950 DEBUG:boto:Params: {} 2023-10-11T14:58:50.950 DEBUG:boto:Token: None 2023-10-11T14:58:50.951 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/ 2023-10-11T14:58:50.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y= 2023-10-11T14:58:50.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZN3jBep51dGAtyHwEy17UwcK+2Y=', 'Content-Length': '0'} 2023-10-11T14:58:50.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277137835622c09e-006526b82a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.954 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:50.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:50.955 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:50.956 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:50.956 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.956 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.957 DEBUG:boto:Method: GET 2023-10-11T14:58:50.957 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.957 DEBUG:boto:Data: 2023-10-11T14:58:50.958 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.959 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.959 DEBUG:boto:Params: {} 2023-10-11T14:58:50.959 DEBUG:boto:Token: None 2023-10-11T14:58:50.960 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk= 2023-10-11T14:58:50.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk=', 'Content-Length': '0'} 2023-10-11T14:58:50.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d47f2f3601efb210-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.963 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:50.964 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:50.964 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.965 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.965 DEBUG:boto:Method: GET 2023-10-11T14:58:50.966 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.966 DEBUG:boto:Data: 2023-10-11T14:58:50.966 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.967 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.968 DEBUG:boto:Params: {} 2023-10-11T14:58:50.968 DEBUG:boto:Token: None 2023-10-11T14:58:50.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk= 2023-10-11T14:58:50.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk=', 'Content-Length': '0'} 2023-10-11T14:58:50.971 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000311c649939b014e5-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.972 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:50.972 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:50.972 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.973 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.973 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.973 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.974 DEBUG:boto:Data: 2023-10-11T14:58:50.974 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.975 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.975 DEBUG:boto:Params: {} 2023-10-11T14:58:50.976 DEBUG:boto:Token: None 2023-10-11T14:58:50.976 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.977 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EXVMZo0cewtb+yKbBkTEn32hHNw= 2023-10-11T14:58:50.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EXVMZo0cewtb+yKbBkTEn32hHNw=', 'Content-Length': '0'} 2023-10-11T14:58:50.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008edce670d5c8cdbf-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.979 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:50.979 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:50.980 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.980 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.980 DEBUG:boto:Method: HEAD 2023-10-11T14:58:50.981 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.981 DEBUG:boto:Data: 2023-10-11T14:58:50.981 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:50.982 DEBUG:boto:Port: 8000 2023-10-11T14:58:50.983 DEBUG:boto:Params: {} 2023-10-11T14:58:50.983 DEBUG:boto:Token: None 2023-10-11T14:58:50.983 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EXVMZo0cewtb+yKbBkTEn32hHNw= 2023-10-11T14:58:50.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EXVMZo0cewtb+yKbBkTEn32hHNw=', 'Content-Length': '0'} 2023-10-11T14:58:50.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f420e5a6670a622d-006526b82a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:50.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:50.987 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:50.987 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:50.988 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.988 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.988 DEBUG:boto:Method: GET 2023-10-11T14:58:50.989 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.989 DEBUG:boto:Data: 2023-10-11T14:58:50.990 DEBUG:boto:Headers: {} 2023-10-11T14:58:50.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:50.990 DEBUG:boto:Port: 8001 2023-10-11T14:58:50.991 DEBUG:boto:Params: {} 2023-10-11T14:58:50.991 DEBUG:boto:Token: None 2023-10-11T14:58:50.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:50 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:50.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk= 2023-10-11T14:58:50.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5b1auJkEwUseGViJ0jo5psHhUUk=', 'Content-Length': '0'} 2023-10-11T14:58:50.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000784c7c58e9bf405d-006526b82a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.038 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:51.039 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:51.039 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:51.040 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:58:51.040 DEBUG:boto:Method: GET 2023-10-11T14:58:51.040 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:51.041 DEBUG:boto:Data: 2023-10-11T14:58:51.041 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.042 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.042 DEBUG:boto:Params: {} 2023-10-11T14:58:51.043 DEBUG:boto:Token: None 2023-10-11T14:58:51.043 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:58:51.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UuKX6cLq4nLDrv9XlfQdKDE2u3Q= 2023-10-11T14:58:51.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UuKX6cLq4nLDrv9XlfQdKDE2u3Q=', 'Content-Length': '0'} 2023-10-11T14:58:51.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3a154aa9af911f0-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.047 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.047 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.048 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.048 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.048 DEBUG:boto:Method: GET 2023-10-11T14:58:51.049 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.049 DEBUG:boto:Data: 2023-10-11T14:58:51.050 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.050 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.051 DEBUG:boto:Params: {} 2023-10-11T14:58:51.051 DEBUG:boto:Token: None 2023-10-11T14:58:51.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3fd19106aff2405-006526b82b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.055 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.056 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.056 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.056 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.057 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.057 DEBUG:boto:Method: GET 2023-10-11T14:58:51.057 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.058 DEBUG:boto:Data: 2023-10-11T14:58:51.058 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.059 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.059 DEBUG:boto:Params: {} 2023-10-11T14:58:51.060 DEBUG:boto:Token: None 2023-10-11T14:58:51.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b832cf473ed703a-006526b82b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.064 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.064 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:51.065 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:51.065 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.066 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:58:51.066 DEBUG:boto:Data: 2023-10-11T14:58:51.066 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.067 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.068 DEBUG:boto:Params: {} 2023-10-11T14:58:51.068 DEBUG:boto:Token: None 2023-10-11T14:58:51.068 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj3 2023-10-11T14:58:51.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OHBDbMv50SEJtn7EflIRBGkmy9c= 2023-10-11T14:58:51.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OHBDbMv50SEJtn7EflIRBGkmy9c=', 'Content-Length': '0'} 2023-10-11T14:58:51.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d34a9595be1e6894-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.071 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:51.072 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:51.072 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.073 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:58:51.073 DEBUG:boto:Data: 2023-10-11T14:58:51.074 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.074 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.075 DEBUG:boto:Params: {} 2023-10-11T14:58:51.075 DEBUG:boto:Token: None 2023-10-11T14:58:51.076 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj3 2023-10-11T14:58:51.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OHBDbMv50SEJtn7EflIRBGkmy9c= 2023-10-11T14:58:51.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OHBDbMv50SEJtn7EflIRBGkmy9c=', 'Content-Length': '0'} 2023-10-11T14:58:51.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cdab5e4a24dd6fa-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:58:51.079 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:51.080 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:51.080 DEBUG:boto:Method: GET 2023-10-11T14:58:51.080 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:58:51.081 DEBUG:boto:Data: 2023-10-11T14:58:51.081 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.082 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.082 DEBUG:boto:Params: {} 2023-10-11T14:58:51.083 DEBUG:boto:Token: None 2023-10-11T14:58:51.083 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj3 2023-10-11T14:58:51.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ehx3Qmy6CtgxncDPjtxIJjL1SR8= 2023-10-11T14:58:51.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ehx3Qmy6CtgxncDPjtxIJjL1SR8=', 'Content-Length': '0'} 2023-10-11T14:58:51.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ae94cf6c9d31210-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.126 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:58:51.127 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:58:51.127 DEBUG:boto:Method: GET 2023-10-11T14:58:51.127 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:58:51.128 DEBUG:boto:Data: 2023-10-11T14:58:51.128 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.129 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.129 DEBUG:boto:Params: {} 2023-10-11T14:58:51.130 DEBUG:boto:Token: None 2023-10-11T14:58:51.130 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj3 2023-10-11T14:58:51.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ehx3Qmy6CtgxncDPjtxIJjL1SR8= 2023-10-11T14:58:51.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ehx3Qmy6CtgxncDPjtxIJjL1SR8=', 'Content-Length': '0'} 2023-10-11T14:58:51.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000604f6831f9471dd6-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.134 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.134 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.134 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.135 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.135 DEBUG:boto:Method: GET 2023-10-11T14:58:51.136 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.136 DEBUG:boto:Data: 2023-10-11T14:58:51.136 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.137 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.138 DEBUG:boto:Params: {} 2023-10-11T14:58:51.138 DEBUG:boto:Token: None 2023-10-11T14:58:51.138 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.139 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882ebc9dc8097541-006526b82b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.142 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.143 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.143 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.144 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.144 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.144 DEBUG:boto:Method: GET 2023-10-11T14:58:51.145 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:58:51.145 DEBUG:boto:Data: 2023-10-11T14:58:51.146 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.147 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.147 DEBUG:boto:Params: {} 2023-10-11T14:58:51.147 DEBUG:boto:Token: None 2023-10-11T14:58:51.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc2a66b416d8c1c-006526b82b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.152 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:51.153 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.153 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.153 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.154 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.154 DEBUG:boto:Method: GET 2023-10-11T14:58:51.154 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.155 DEBUG:boto:Data: 2023-10-11T14:58:51.155 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.156 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.156 DEBUG:boto:Params: {} 2023-10-11T14:58:51.157 DEBUG:boto:Token: None 2023-10-11T14:58:51.157 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo= 2023-10-11T14:58:51.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo=', 'Content-Length': '0'} 2023-10-11T14:58:51.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052851702f5408b1b-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.160 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.161 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.161 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.162 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.162 DEBUG:boto:Method: GET 2023-10-11T14:58:51.162 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.163 DEBUG:boto:Data: 2023-10-11T14:58:51.163 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.164 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.165 DEBUG:boto:Params: {} 2023-10-11T14:58:51.165 DEBUG:boto:Token: None 2023-10-11T14:58:51.165 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.166 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo= 2023-10-11T14:58:51.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo=', 'Content-Length': '0'} 2023-10-11T14:58:51.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000808e1a38e080cd3c-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.210 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.211 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.211 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.211 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.212 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.212 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.212 DEBUG:boto:Data: 2023-10-11T14:58:51.213 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.213 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.214 DEBUG:boto:Params: {} 2023-10-11T14:58:51.214 DEBUG:boto:Token: None 2023-10-11T14:58:51.215 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2NGLapwqG9fU9E0Wy2uAtRmhZYg= 2023-10-11T14:58:51.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2NGLapwqG9fU9E0Wy2uAtRmhZYg=', 'Content-Length': '0'} 2023-10-11T14:58:51.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be17f67cb612074f-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.218 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.218 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.219 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.219 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.219 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.220 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.220 DEBUG:boto:Data: 2023-10-11T14:58:51.220 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.221 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.222 DEBUG:boto:Params: {} 2023-10-11T14:58:51.222 DEBUG:boto:Token: None 2023-10-11T14:58:51.222 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2NGLapwqG9fU9E0Wy2uAtRmhZYg= 2023-10-11T14:58:51.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2NGLapwqG9fU9E0Wy2uAtRmhZYg=', 'Content-Length': '0'} 2023-10-11T14:58:51.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000812e9fdf334ba382-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:51.226 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.226 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.227 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.227 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.227 DEBUG:boto:Method: GET 2023-10-11T14:58:51.228 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.228 DEBUG:boto:Data: 2023-10-11T14:58:51.229 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.229 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.230 DEBUG:boto:Params: {} 2023-10-11T14:58:51.230 DEBUG:boto:Token: None 2023-10-11T14:58:51.231 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo= 2023-10-11T14:58:51.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo=', 'Content-Length': '0'} 2023-10-11T14:58:51.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025cff58b58c3c542-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.234 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.234 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.235 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.235 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.235 DEBUG:boto:Method: GET 2023-10-11T14:58:51.236 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.236 DEBUG:boto:Data: 2023-10-11T14:58:51.237 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.237 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.238 DEBUG:boto:Params: {} 2023-10-11T14:58:51.238 DEBUG:boto:Token: None 2023-10-11T14:58:51.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:58:51.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo= 2023-10-11T14:58:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAnYKhBQlYi6repnOBX2FWwdAuo=', 'Content-Length': '0'} 2023-10-11T14:58:51.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002491b485a17f27e5-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.242 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.242 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.243 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.243 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.244 DEBUG:boto:Method: GET 2023-10-11T14:58:51.244 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.244 DEBUG:boto:Data: 2023-10-11T14:58:51.245 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.246 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.246 DEBUG:boto:Params: {} 2023-10-11T14:58:51.246 DEBUG:boto:Token: None 2023-10-11T14:58:51.247 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed8ea714aabf8f3b-006526b82b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.250 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.251 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.251 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.252 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.252 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.252 DEBUG:boto:Method: GET 2023-10-11T14:58:51.253 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.253 DEBUG:boto:Data: 2023-10-11T14:58:51.254 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.254 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.255 DEBUG:boto:Params: {} 2023-10-11T14:58:51.255 DEBUG:boto:Token: None 2023-10-11T14:58:51.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000defc84fb2d0e986a-006526b82b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.259 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.260 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.260 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.261 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.261 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:58:51.261 DEBUG:boto:Data: 2023-10-11T14:58:51.262 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.263 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.263 DEBUG:boto:Params: {} 2023-10-11T14:58:51.264 DEBUG:boto:Token: None 2023-10-11T14:58:51.264 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4 2023-10-11T14:58:51.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wPfV601c3zoZgPLEEWc3LnUx8VY= 2023-10-11T14:58:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wPfV601c3zoZgPLEEWc3LnUx8VY=', 'Content-Length': '0'} 2023-10-11T14:58:51.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a84c7c9fca07804-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.267 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.268 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.268 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.268 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:58:51.269 DEBUG:boto:Data: 2023-10-11T14:58:51.269 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.269 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.270 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.270 DEBUG:boto:Params: {} 2023-10-11T14:58:51.271 DEBUG:boto:Token: None 2023-10-11T14:58:51.271 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4 2023-10-11T14:58:51.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wPfV601c3zoZgPLEEWc3LnUx8VY= 2023-10-11T14:58:51.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wPfV601c3zoZgPLEEWc3LnUx8VY=', 'Content-Length': '0'} 2023-10-11T14:58:51.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c16aae9c0033a3fa-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:58:51.274 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.275 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.275 DEBUG:boto:Method: GET 2023-10-11T14:58:51.276 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:58:51.276 DEBUG:boto:Data: 2023-10-11T14:58:51.276 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.277 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.277 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.278 DEBUG:boto:Params: {} 2023-10-11T14:58:51.278 DEBUG:boto:Token: None 2023-10-11T14:58:51.278 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4 2023-10-11T14:58:51.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FWyQZ92NQLbK9ReoWtP2vXWIMLE= 2023-10-11T14:58:51.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FWyQZ92NQLbK9ReoWtP2vXWIMLE=', 'Content-Length': '0'} 2023-10-11T14:58:51.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4896e8f034fe8b5-006526b82b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.282 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:58:51.282 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:58:51.283 DEBUG:boto:Method: GET 2023-10-11T14:58:51.283 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:58:51.283 DEBUG:boto:Data: 2023-10-11T14:58:51.284 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.285 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.285 DEBUG:boto:Params: {} 2023-10-11T14:58:51.285 DEBUG:boto:Token: None 2023-10-11T14:58:51.286 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj4 2023-10-11T14:58:51.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FWyQZ92NQLbK9ReoWtP2vXWIMLE= 2023-10-11T14:58:51.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FWyQZ92NQLbK9ReoWtP2vXWIMLE=', 'Content-Length': '0'} 2023-10-11T14:58:51.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3b54fcc3a417c23-006526b82b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.289 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.289 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.290 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.290 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.291 DEBUG:boto:Method: GET 2023-10-11T14:58:51.291 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.291 DEBUG:boto:Data: 2023-10-11T14:58:51.292 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.292 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.293 DEBUG:boto:Params: {} 2023-10-11T14:58:51.293 DEBUG:boto:Token: None 2023-10-11T14:58:51.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad92c09bc4fdfead-006526b82b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.297 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.298 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.298 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.299 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.299 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.299 DEBUG:boto:Method: GET 2023-10-11T14:58:51.300 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:58:51.300 DEBUG:boto:Data: 2023-10-11T14:58:51.300 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.301 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.301 DEBUG:boto:Params: {} 2023-10-11T14:58:51.302 DEBUG:boto:Token: None 2023-10-11T14:58:51.302 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU= 2023-10-11T14:58:51.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQND7hvmNKTtHEGLpTqyCwxtGDU=', 'Content-Length': '0'} 2023-10-11T14:58:51.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83d470dfd167466-006526b82b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.306 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:58:51.307 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-47 zones={a2, a1} 2023-10-11T14:58:51.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.308 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:58:51.358 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:51.359 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:51.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dghqvd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dghqvd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.423 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dghqvd-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:58:51.473 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:51.474 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:51.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.538 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:58:51.589 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:51.589 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:51.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dghqvd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dghqvd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T14:58:51.654 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dghqvd-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T14:58:51.705 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:51.705 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:51.771 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:51.771 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:51.772 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.772 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:51.772 DEBUG:boto:Data: 2023-10-11T14:58:51.773 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.774 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.774 DEBUG:boto:Params: {} 2023-10-11T14:58:51.774 DEBUG:boto:Token: None 2023-10-11T14:58:51.775 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/ 2023-10-11T14:58:51.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q= 2023-10-11T14:58:51.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q=', 'Content-Length': '0'} 2023-10-11T14:58:51.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be0499cac80a25c7-006526b82b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.779 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:58:51.779 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:58:51.780 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.780 DEBUG:boto:Path: /dghqvd-46/obj5 2023-10-11T14:58:51.780 DEBUG:boto:Data: 2023-10-11T14:58:51.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.781 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.782 DEBUG:boto:Params: {} 2023-10-11T14:58:51.782 DEBUG:boto:Token: None 2023-10-11T14:58:51.783 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/obj5 2023-10-11T14:58:51.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9TRCfgoDWe/KOarXyiOmL59ef+w= 2023-10-11T14:58:51.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9TRCfgoDWe/KOarXyiOmL59ef+w='} 2023-10-11T14:58:51.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000441aee47f7b90544-006526b82b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.827 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:51.827 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:51.828 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.828 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:51.828 DEBUG:boto:Data: 2023-10-11T14:58:51.829 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.830 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.830 DEBUG:boto:Params: {} 2023-10-11T14:58:51.831 DEBUG:boto:Token: None 2023-10-11T14:58:51.831 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/ 2023-10-11T14:58:51.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q= 2023-10-11T14:58:51.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q=', 'Content-Length': '0'} 2023-10-11T14:58:51.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000868d119d94cd0595-006526b82b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.835 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:58:51.835 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:58:51.836 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.836 DEBUG:boto:Path: /dghqvd-46/obj6 2023-10-11T14:58:51.836 DEBUG:boto:Data: 2023-10-11T14:58:51.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.838 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.838 DEBUG:boto:Params: {} 2023-10-11T14:58:51.838 DEBUG:boto:Token: None 2023-10-11T14:58:51.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/obj6 2023-10-11T14:58:51.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQwuAq6r30b1sZwzNdBdYCCb2hs= 2023-10-11T14:58:51.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQwuAq6r30b1sZwzNdBdYCCb2hs='} 2023-10-11T14:58:51.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dd22e15ec585da9-006526b82b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.845 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:51.845 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:51.846 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.846 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:51.846 DEBUG:boto:Data: 2023-10-11T14:58:51.847 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.848 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.848 DEBUG:boto:Params: {} 2023-10-11T14:58:51.848 DEBUG:boto:Token: None 2023-10-11T14:58:51.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/ 2023-10-11T14:58:51.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q= 2023-10-11T14:58:51.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q=', 'Content-Length': '0'} 2023-10-11T14:58:51.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093933991ff06bd1e-006526b82b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.852 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:58:51.852 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:58:51.853 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.853 DEBUG:boto:Path: /dghqvd-46/obj7 2023-10-11T14:58:51.854 DEBUG:boto:Data: 2023-10-11T14:58:51.854 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.855 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.855 DEBUG:boto:Params: {} 2023-10-11T14:58:51.856 DEBUG:boto:Token: None 2023-10-11T14:58:51.856 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/obj7 2023-10-11T14:58:51.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t51jBW1fomVQjJT+EBkOA4JbzTQ= 2023-10-11T14:58:51.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t51jBW1fomVQjJT+EBkOA4JbzTQ='} 2023-10-11T14:58:51.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059dff6cd1609a10a-006526b82b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.862 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:58:51.862 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:58:51.863 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.863 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:58:51.863 DEBUG:boto:Data: 2023-10-11T14:58:51.864 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.865 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.865 DEBUG:boto:Params: {} 2023-10-11T14:58:51.865 DEBUG:boto:Token: None 2023-10-11T14:58:51.866 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/ 2023-10-11T14:58:51.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q= 2023-10-11T14:58:51.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w51ZeCFPf4ffGUfTgrQPB0qMk6Q=', 'Content-Length': '0'} 2023-10-11T14:58:51.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043ff85f26968da4b-006526b82b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.870 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:58:51.870 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:58:51.870 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.871 DEBUG:boto:Path: /dghqvd-46/obj8 2023-10-11T14:58:51.871 DEBUG:boto:Data: 2023-10-11T14:58:51.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:58:51.872 DEBUG:boto:Port: 8000 2023-10-11T14:58:51.873 DEBUG:boto:Params: {} 2023-10-11T14:58:51.873 DEBUG:boto:Token: None 2023-10-11T14:58:51.873 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-46/obj8 2023-10-11T14:58:51.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+M6SM09fP81dh29lOUh8X7efhtI= 2023-10-11T14:58:51.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+M6SM09fP81dh29lOUh8X7efhtI='} 2023-10-11T14:58:51.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac261979420f05be-006526b82b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.919 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.919 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.919 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.920 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:51.920 DEBUG:boto:Data: 2023-10-11T14:58:51.921 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.922 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.922 DEBUG:boto:Params: {} 2023-10-11T14:58:51.922 DEBUG:boto:Token: None 2023-10-11T14:58:51.923 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY= 2023-10-11T14:58:51.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY=', 'Content-Length': '0'} 2023-10-11T14:58:51.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3fd524388560e01-006526b82b-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.927 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:58:51.927 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:58:51.928 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.928 DEBUG:boto:Path: /dghqvd-47/obj5 2023-10-11T14:58:51.928 DEBUG:boto:Data: 2023-10-11T14:58:51.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.930 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.930 DEBUG:boto:Params: {} 2023-10-11T14:58:51.931 DEBUG:boto:Token: None 2023-10-11T14:58:51.931 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj5 2023-10-11T14:58:51.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/6REAQA4Ndc94ZbSNcdqVC+5nY= 2023-10-11T14:58:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/6REAQA4Ndc94ZbSNcdqVC+5nY='} 2023-10-11T14:58:51.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002aaf277077d0a719-006526b82b-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.937 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.938 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.938 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.938 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:51.939 DEBUG:boto:Data: 2023-10-11T14:58:51.939 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.940 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.941 DEBUG:boto:Params: {} 2023-10-11T14:58:51.941 DEBUG:boto:Token: None 2023-10-11T14:58:51.942 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY= 2023-10-11T14:58:51.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY=', 'Content-Length': '0'} 2023-10-11T14:58:51.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064146ea162860c6f-006526b82b-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.945 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:58:51.945 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:58:51.946 DEBUG:boto:Method: PUT 2023-10-11T14:58:51.946 DEBUG:boto:Path: /dghqvd-47/obj6 2023-10-11T14:58:51.947 DEBUG:boto:Data: 2023-10-11T14:58:51.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:51.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.948 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.948 DEBUG:boto:Params: {} 2023-10-11T14:58:51.949 DEBUG:boto:Token: None 2023-10-11T14:58:51.949 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/obj6 2023-10-11T14:58:51.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jLGZ8kYdH5y7yb4qC3kQs/A7GvY= 2023-10-11T14:58:51.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jLGZ8kYdH5y7yb4qC3kQs/A7GvY='} 2023-10-11T14:58:51.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4da9555419d03fa-006526b82b-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:51.995 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:51.995 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:51.996 DEBUG:boto:Method: HEAD 2023-10-11T14:58:51.996 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:51.996 DEBUG:boto:Data: 2023-10-11T14:58:51.997 DEBUG:boto:Headers: {} 2023-10-11T14:58:51.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:51.998 DEBUG:boto:Port: 8001 2023-10-11T14:58:51.998 DEBUG:boto:Params: {} 2023-10-11T14:58:51.998 DEBUG:boto:Token: None 2023-10-11T14:58:51.999 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:51 GMT /dghqvd-47/ 2023-10-11T14:58:51.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY= 2023-10-11T14:58:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3aHa2BdvjB9+nifPbW3s7gG0baY=', 'Content-Length': '0'} 2023-10-11T14:58:52.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c7a4a13cf4204a4-006526b82b-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:52.003 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:58:52.003 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:58:52.003 DEBUG:boto:Method: PUT 2023-10-11T14:58:52.004 DEBUG:boto:Path: /dghqvd-47/obj7 2023-10-11T14:58:52.004 DEBUG:boto:Data: 2023-10-11T14:58:52.005 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:52.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:52.005 DEBUG:boto:Port: 8001 2023-10-11T14:58:52.006 DEBUG:boto:Params: {} 2023-10-11T14:58:52.006 DEBUG:boto:Token: None 2023-10-11T14:58:52.007 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:52 GMT /dghqvd-47/obj7 2023-10-11T14:58:52.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gBnhLHoavRjYkwZwhWbMb75qBeo= 2023-10-11T14:58:52.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gBnhLHoavRjYkwZwhWbMb75qBeo='} 2023-10-11T14:58:52.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006336a04c21d2e7fa-006526b82c-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:52.013 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:58:52.014 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:58:52.014 DEBUG:boto:Method: HEAD 2023-10-11T14:58:52.015 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:58:52.015 DEBUG:boto:Data: 2023-10-11T14:58:52.015 DEBUG:boto:Headers: {} 2023-10-11T14:58:52.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:52.016 DEBUG:boto:Port: 8001 2023-10-11T14:58:52.017 DEBUG:boto:Params: {} 2023-10-11T14:58:52.017 DEBUG:boto:Token: None 2023-10-11T14:58:52.017 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:58:52 GMT /dghqvd-47/ 2023-10-11T14:58:52.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0VX4WQZFQjku68r+Jk94/rHTFAo= 2023-10-11T14:58:52.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:58:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0VX4WQZFQjku68r+Jk94/rHTFAo=', 'Content-Length': '0'} 2023-10-11T14:58:52.020 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e92f18b5014584a4-006526b82c-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:52.021 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:58:52.021 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:58:52.022 DEBUG:boto:Method: PUT 2023-10-11T14:58:52.022 DEBUG:boto:Path: /dghqvd-47/obj8 2023-10-11T14:58:52.023 DEBUG:boto:Data: 2023-10-11T14:58:52.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:58:52.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:58:52.024 DEBUG:boto:Port: 8001 2023-10-11T14:58:52.024 DEBUG:boto:Params: {} 2023-10-11T14:58:52.025 DEBUG:boto:Token: None 2023-10-11T14:58:52.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:58:52 GMT /dghqvd-47/obj8 2023-10-11T14:58:52.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8PWWIbCnUkw1FG8q/Jb1zJ9N5z8= 2023-10-11T14:58:52.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:58:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8PWWIbCnUkw1FG8q/Jb1zJ9N5z8='} 2023-10-11T14:58:52.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036ca50fbdded1857-006526b82c-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:58:52.031 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-46 2023-10-11T14:58:52.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:52.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:58:52.033 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:58:52.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:58:52.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:58:52.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:52.121+0000 7f718cee5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:52.124 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.21.6, , 00000000001.26.6, , 00000000001.51.6, , , 00000000001.28.6, , , ] 2023-10-11T14:58:52.124 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.25.6, 00000000002.34.1, 00000000004.30.6, 00000000002.91.1, 00000000004.55.6, 00000000002.22.1, 00000000002.35.1, 00000000004.32.6, 00000000002.30.1, 00000000002.23.1, 00000000002.36.1] 2023-10-11T14:58:57.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:58:57.122+0000 7f718cee5780 1 waiting for incremental sync to catch up: 2023-10-11T14:58:57.126 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.21.6, , 00000000001.26.6, , 00000000001.51.6, , , 00000000001.28.6, , , ] 2023-10-11T14:58:57.126 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.25.6, 00000000002.34.1, 00000000004.30.6, 00000000002.91.1, 00000000004.55.6, 00000000002.22.1, 00000000002.35.1, 00000000004.32.6, 00000000002.30.1, 00000000002.23.1, 00000000002.36.1] 2023-10-11T14:59:02.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:02.123+0000 7f718cee5780 1 waiting for incremental sync to catch up: 2023-10-11T14:59:02.127 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.21.6, , 00000000001.26.6, , 00000000001.51.6, , , 00000000001.28.6, , , ] 2023-10-11T14:59:02.128 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.25.6, 00000000002.34.1, 00000000004.30.6, 00000000002.91.1, 00000000004.55.6, 00000000002.22.1, 00000000002.35.1, 00000000004.32.6, 00000000002.30.1, 00000000002.23.1, 00000000002.36.1] 2023-10-11T14:59:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:07.124+0000 7f718cee5780 1 bucket sync caught up with source: 2023-10-11T14:59:07.128 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.25.6, 00000000002.34.1, 00000000004.30.6, 00000000002.91.1, 00000000004.55.6, 00000000002.22.1, 00000000002.35.1, 00000000004.32.6, 00000000002.30.1, 00000000002.23.1, 00000000002.36.1] 2023-10-11T14:59:07.128 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.25.6, 00000000002.34.1, 00000000004.30.6, 00000000002.91.1, 00000000004.55.6, 00000000002.22.1, 00000000002.35.1, 00000000004.32.6, 00000000002.30.1, 00000000002.23.1, 00000000002.36.1] 2023-10-11T14:59:07.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:07.124+0000 7f718cee5780 0 bucket checkpoint complete 2023-10-11T14:59:07.140 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-46 2023-10-11T14:59:07.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:07.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:07.142 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:59:07.202 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:07.203 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:07.238 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:07.235+0000 7fdcb5870780 1 waiting for incremental sync to catch up: 2023-10-11T14:59:07.239 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000001.236.6, , 00000000001.221.6, , , 00000000001.243.6, , , ] 2023-10-11T14:59:07.239 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:12.238+0000 7fdcb5870780 1 waiting for incremental sync to catch up: 2023-10-11T14:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000001.243.6, , , ] 2023-10-11T14:59:12.240 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:17.239+0000 7fdcb5870780 1 waiting for incremental sync to catch up: 2023-10-11T14:59:17.241 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000001.243.6, , , ] 2023-10-11T14:59:17.242 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:22.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:22.240+0000 7fdcb5870780 1 waiting for incremental sync to catch up: 2023-10-11T14:59:22.243 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000001.243.6, , , ] 2023-10-11T14:59:22.243 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:27.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:27.241+0000 7fdcb5870780 1 bucket sync caught up with source: 2023-10-11T14:59:27.244 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:27.244 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.78.6, , 00000000002.240.6, , 00000000002.225.6, , , 00000000002.247.6, , , ] 2023-10-11T14:59:27.244 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:27.241+0000 7fdcb5870780 0 bucket checkpoint complete 2023-10-11T14:59:27.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-46 zones={a2, a1} 2023-10-11T14:59:27.259 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.260 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.260 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.261 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:59:27.261 DEBUG:boto:Data: 2023-10-11T14:59:27.262 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.263 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.263 DEBUG:boto:Params: {} 2023-10-11T14:59:27.264 DEBUG:boto:Token: None 2023-10-11T14:59:27.264 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wSsO1B2bpNwB+4aWwO5tUpIgjLg= 2023-10-11T14:59:27.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wSsO1B2bpNwB+4aWwO5tUpIgjLg=', 'Content-Length': '0'} 2023-10-11T14:59:27.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3d30eca5b96599e-006526b84f-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.268 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.269 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.269 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.270 DEBUG:boto:Path: /dghqvd-46/ 2023-10-11T14:59:27.270 DEBUG:boto:Data: 2023-10-11T14:59:27.270 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.271 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.272 DEBUG:boto:Params: {} 2023-10-11T14:59:27.272 DEBUG:boto:Token: None 2023-10-11T14:59:27.273 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.273 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wSsO1B2bpNwB+4aWwO5tUpIgjLg= 2023-10-11T14:59:27.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wSsO1B2bpNwB+4aWwO5tUpIgjLg=', 'Content-Length': '0'} 2023-10-11T14:59:27.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081fc08c60b5a2c1b-006526b84f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:59:27.277 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.277 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.278 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.278 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.279 DEBUG:boto:Method: GET 2023-10-11T14:59:27.279 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.279 DEBUG:boto:Data: 2023-10-11T14:59:27.280 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.280 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.281 DEBUG:boto:Params: {} 2023-10-11T14:59:27.281 DEBUG:boto:Token: None 2023-10-11T14:59:27.282 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/?versions 2023-10-11T14:59:27.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg= 2023-10-11T14:59:27.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg=', 'Content-Length': '0'} 2023-10-11T14:59:27.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba05f7c553867b97-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.286 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:59:27.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:59:27.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:59:27.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:59:27.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T14:59:27.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T14:59:27.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T14:59:27.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T14:59:27.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:59:27.290 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.291 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.291 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.291 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.292 DEBUG:boto:Method: GET 2023-10-11T14:59:27.292 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.293 DEBUG:boto:Data: 2023-10-11T14:59:27.293 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.294 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.294 DEBUG:boto:Params: {} 2023-10-11T14:59:27.295 DEBUG:boto:Token: None 2023-10-11T14:59:27.295 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/?versions 2023-10-11T14:59:27.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg= 2023-10-11T14:59:27.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg=', 'Content-Length': '0'} 2023-10-11T14:59:27.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001504ca97870e6d8-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.300 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:59:27.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:59:27.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:59:27.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:59:27.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T14:59:27.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T14:59:27.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T14:59:27.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T14:59:27.304 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.304 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.305 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.305 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.305 DEBUG:boto:Method: GET 2023-10-11T14:59:27.306 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.306 DEBUG:boto:Data: 2023-10-11T14:59:27.307 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.308 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.308 DEBUG:boto:Params: {} 2023-10-11T14:59:27.308 DEBUG:boto:Token: None 2023-10-11T14:59:27.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/?versions 2023-10-11T14:59:27.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg= 2023-10-11T14:59:27.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg=', 'Content-Length': '0'} 2023-10-11T14:59:27.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf38d3649ad2968-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.313 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.314 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.314 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.315 DEBUG:boto:path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.315 DEBUG:boto:auth_path=/dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.315 DEBUG:boto:Method: GET 2023-10-11T14:59:27.316 DEBUG:boto:Path: /dghqvd-46/?versions&max-keys=999 2023-10-11T14:59:27.316 DEBUG:boto:Data: 2023-10-11T14:59:27.317 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.317 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.318 DEBUG:boto:Params: {} 2023-10-11T14:59:27.318 DEBUG:boto:Token: None 2023-10-11T14:59:27.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/?versions 2023-10-11T14:59:27.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg= 2023-10-11T14:59:27.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u6MnkU6AedPxzG+9WTlOwXg3ODg=', 'Content-Length': '0'} 2023-10-11T14:59:27.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f6e9ea9313eba5-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.322 DEBUG:boto:b'dghqvd-46999falseobj1nulltrue2023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:27.324 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.324 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.324 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.325 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.325 DEBUG:boto:Method: GET 2023-10-11T14:59:27.326 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.326 DEBUG:boto:Data: 2023-10-11T14:59:27.327 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.327 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.327 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.328 DEBUG:boto:Params: {} 2023-10-11T14:59:27.328 DEBUG:boto:Token: None 2023-10-11T14:59:27.329 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU= 2023-10-11T14:59:27.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU=', 'Content-Length': '0'} 2023-10-11T14:59:27.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006012ef5975f56583-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.375 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.375 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.376 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.376 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.377 DEBUG:boto:Method: GET 2023-10-11T14:59:27.377 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.377 DEBUG:boto:Data: 2023-10-11T14:59:27.378 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.379 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.379 DEBUG:boto:Params: {} 2023-10-11T14:59:27.379 DEBUG:boto:Token: None 2023-10-11T14:59:27.380 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU= 2023-10-11T14:59:27.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU=', 'Content-Length': '0'} 2023-10-11T14:59:27.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec014a16e166570-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.383 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.384 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.384 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.385 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.385 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.385 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.386 DEBUG:boto:Data: 2023-10-11T14:59:27.386 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.387 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.387 DEBUG:boto:Params: {} 2023-10-11T14:59:27.388 DEBUG:boto:Token: None 2023-10-11T14:59:27.388 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/cQuMq5+nRCf4a5DidBmcdwVca0= 2023-10-11T14:59:27.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/cQuMq5+nRCf4a5DidBmcdwVca0=', 'Content-Length': '0'} 2023-10-11T14:59:27.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015262a17a49288a2-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.391 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.392 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.392 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.393 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.393 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.393 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.394 DEBUG:boto:Data: 2023-10-11T14:59:27.394 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.395 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.396 DEBUG:boto:Params: {} 2023-10-11T14:59:27.396 DEBUG:boto:Token: None 2023-10-11T14:59:27.396 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.397 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/cQuMq5+nRCf4a5DidBmcdwVca0= 2023-10-11T14:59:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/cQuMq5+nRCf4a5DidBmcdwVca0=', 'Content-Length': '0'} 2023-10-11T14:59:27.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbd5da755a9ea809-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:27.400 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.400 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.401 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.401 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.402 DEBUG:boto:Method: GET 2023-10-11T14:59:27.402 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.403 DEBUG:boto:Data: 2023-10-11T14:59:27.403 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.404 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.404 DEBUG:boto:Params: {} 2023-10-11T14:59:27.405 DEBUG:boto:Token: None 2023-10-11T14:59:27.405 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU= 2023-10-11T14:59:27.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU=', 'Content-Length': '0'} 2023-10-11T14:59:27.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f0ca95db455d9bc-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.409 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.409 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.410 DEBUG:boto:path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.410 DEBUG:boto:auth_path=/dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.411 DEBUG:boto:Method: GET 2023-10-11T14:59:27.411 DEBUG:boto:Path: /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.412 DEBUG:boto:Data: 2023-10-11T14:59:27.412 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.413 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.413 DEBUG:boto:Params: {} 2023-10-11T14:59:27.414 DEBUG:boto:Token: None 2023-10-11T14:59:27.414 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1?versionId=null 2023-10-11T14:59:27.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU= 2023-10-11T14:59:27.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MXrPOOn0IcFG4tMRduVjIrKUIgU=', 'Content-Length': '0'} 2023-10-11T14:59:27.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029451a88ca8e3295-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.462 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.463 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.463 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.463 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.464 DEBUG:boto:Method: GET 2023-10-11T14:59:27.464 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.465 DEBUG:boto:Data: 2023-10-11T14:59:27.465 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.466 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.467 DEBUG:boto:Params: {} 2023-10-11T14:59:27.467 DEBUG:boto:Token: None 2023-10-11T14:59:27.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007976b7e7dc6ea5e5-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.471 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.472 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.472 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.473 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.473 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.473 DEBUG:boto:Method: GET 2023-10-11T14:59:27.474 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.474 DEBUG:boto:Data: 2023-10-11T14:59:27.475 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.476 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.476 DEBUG:boto:Params: {} 2023-10-11T14:59:27.476 DEBUG:boto:Token: None 2023-10-11T14:59:27.477 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000093224ad51436c16-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.481 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.482 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.482 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.483 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.483 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:59:27.484 DEBUG:boto:Data: 2023-10-11T14:59:27.484 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.485 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.485 DEBUG:boto:Params: {} 2023-10-11T14:59:27.486 DEBUG:boto:Token: None 2023-10-11T14:59:27.486 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1 2023-10-11T14:59:27.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/lFyCwXvqHcIoc0gzDWLhN+tEYQ= 2023-10-11T14:59:27.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/lFyCwXvqHcIoc0gzDWLhN+tEYQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe7dfd6b091d1ac1-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.489 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.490 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.490 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.491 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:59:27.491 DEBUG:boto:Data: 2023-10-11T14:59:27.492 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.492 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.493 DEBUG:boto:Params: {} 2023-10-11T14:59:27.493 DEBUG:boto:Token: None 2023-10-11T14:59:27.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1 2023-10-11T14:59:27.494 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/lFyCwXvqHcIoc0gzDWLhN+tEYQ= 2023-10-11T14:59:27.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/lFyCwXvqHcIoc0gzDWLhN+tEYQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e157a542fd1079f-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:27.497 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.497 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.498 DEBUG:boto:Method: GET 2023-10-11T14:59:27.498 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:59:27.499 DEBUG:boto:Data: 2023-10-11T14:59:27.499 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.500 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.501 DEBUG:boto:Params: {} 2023-10-11T14:59:27.501 DEBUG:boto:Token: None 2023-10-11T14:59:27.501 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1 2023-10-11T14:59:27.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TzlRoEglsXgtDHwbSdMaL8dv71s= 2023-10-11T14:59:27.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TzlRoEglsXgtDHwbSdMaL8dv71s=', 'Content-Length': '0'} 2023-10-11T14:59:27.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dff49ae4e80fef4-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.546 DEBUG:boto:path=/dghqvd-46/obj1 2023-10-11T14:59:27.547 DEBUG:boto:auth_path=/dghqvd-46/obj1 2023-10-11T14:59:27.547 DEBUG:boto:Method: GET 2023-10-11T14:59:27.548 DEBUG:boto:Path: /dghqvd-46/obj1 2023-10-11T14:59:27.548 DEBUG:boto:Data: 2023-10-11T14:59:27.548 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.549 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.549 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.550 DEBUG:boto:Params: {} 2023-10-11T14:59:27.550 DEBUG:boto:Token: None 2023-10-11T14:59:27.551 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj1 2023-10-11T14:59:27.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TzlRoEglsXgtDHwbSdMaL8dv71s= 2023-10-11T14:59:27.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TzlRoEglsXgtDHwbSdMaL8dv71s=', 'Content-Length': '0'} 2023-10-11T14:59:27.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000803bce83abf9cee-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.554 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.555 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.555 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.556 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.556 DEBUG:boto:Method: GET 2023-10-11T14:59:27.557 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.557 DEBUG:boto:Data: 2023-10-11T14:59:27.557 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.558 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.559 DEBUG:boto:Params: {} 2023-10-11T14:59:27.559 DEBUG:boto:Token: None 2023-10-11T14:59:27.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa66f94ef35a39d0-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.564 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.565 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.565 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.566 DEBUG:boto:path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.566 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.566 DEBUG:boto:Method: GET 2023-10-11T14:59:27.567 DEBUG:boto:Path: /dghqvd-46/?prefix=obj1 2023-10-11T14:59:27.567 DEBUG:boto:Data: 2023-10-11T14:59:27.568 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.568 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.569 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.569 DEBUG:boto:Params: {} 2023-10-11T14:59:27.570 DEBUG:boto:Token: None 2023-10-11T14:59:27.570 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000116d02c7d1ef7956-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.574 DEBUG:boto:b'dghqvd-46obj11000falseobj12023-10-11T14:55:48.818Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:27.575 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.576 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.576 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.577 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.577 DEBUG:boto:Method: GET 2023-10-11T14:59:27.578 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.578 DEBUG:boto:Data: 2023-10-11T14:59:27.578 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.579 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.580 DEBUG:boto:Params: {} 2023-10-11T14:59:27.580 DEBUG:boto:Token: None 2023-10-11T14:59:27.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U= 2023-10-11T14:59:27.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U=', 'Content-Length': '0'} 2023-10-11T14:59:27.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f585cf6c71d648d7-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.584 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.585 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.585 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.586 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.586 DEBUG:boto:Method: GET 2023-10-11T14:59:27.587 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.587 DEBUG:boto:Data: 2023-10-11T14:59:27.587 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.588 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.589 DEBUG:boto:Params: {} 2023-10-11T14:59:27.589 DEBUG:boto:Token: None 2023-10-11T14:59:27.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U= 2023-10-11T14:59:27.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U=', 'Content-Length': '0'} 2023-10-11T14:59:27.592 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d30e340d490eeb-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.593 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.594 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.594 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.594 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.595 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.595 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.596 DEBUG:boto:Data: 2023-10-11T14:59:27.596 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.597 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.597 DEBUG:boto:Params: {} 2023-10-11T14:59:27.598 DEBUG:boto:Token: None 2023-10-11T14:59:27.598 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NenmvL/fx8s6FvVnrGsW0vKNJmQ= 2023-10-11T14:59:27.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NenmvL/fx8s6FvVnrGsW0vKNJmQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074fadb4caebd8b7e-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.602 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.602 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.602 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.603 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.603 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.604 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.604 DEBUG:boto:Data: 2023-10-11T14:59:27.604 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.605 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.605 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.605 DEBUG:boto:Params: {} 2023-10-11T14:59:27.606 DEBUG:boto:Token: None 2023-10-11T14:59:27.606 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NenmvL/fx8s6FvVnrGsW0vKNJmQ= 2023-10-11T14:59:27.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NenmvL/fx8s6FvVnrGsW0vKNJmQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eef720af092cc6d6-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:27.610 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.611 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.611 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.612 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.612 DEBUG:boto:Method: GET 2023-10-11T14:59:27.613 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.613 DEBUG:boto:Data: 2023-10-11T14:59:27.613 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.614 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.614 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.615 DEBUG:boto:Params: {} 2023-10-11T14:59:27.615 DEBUG:boto:Token: None 2023-10-11T14:59:27.616 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U= 2023-10-11T14:59:27.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U=', 'Content-Length': '0'} 2023-10-11T14:59:27.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007386be5f5862b8a-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.619 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.620 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.620 DEBUG:boto:path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.620 DEBUG:boto:auth_path=/dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.621 DEBUG:boto:Method: GET 2023-10-11T14:59:27.621 DEBUG:boto:Path: /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.622 DEBUG:boto:Data: 2023-10-11T14:59:27.622 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.623 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.623 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.624 DEBUG:boto:Params: {} 2023-10-11T14:59:27.624 DEBUG:boto:Token: None 2023-10-11T14:59:27.624 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2?versionId=null 2023-10-11T14:59:27.625 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U= 2023-10-11T14:59:27.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jd4mz+/6dSdd83NpAvWbPFv+p+U=', 'Content-Length': '0'} 2023-10-11T14:59:27.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000928c5e0730b324e2-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.628 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.629 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.629 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.630 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.630 DEBUG:boto:Method: GET 2023-10-11T14:59:27.630 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.631 DEBUG:boto:Data: 2023-10-11T14:59:27.631 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.632 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.633 DEBUG:boto:Params: {} 2023-10-11T14:59:27.633 DEBUG:boto:Token: None 2023-10-11T14:59:27.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095ec9c056a3e4f6a-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.682 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.683 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.683 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.684 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.684 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.685 DEBUG:boto:Method: GET 2023-10-11T14:59:27.685 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.685 DEBUG:boto:Data: 2023-10-11T14:59:27.686 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.687 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.687 DEBUG:boto:Params: {} 2023-10-11T14:59:27.688 DEBUG:boto:Token: None 2023-10-11T14:59:27.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483ca4afe9cefb91-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.692 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.693 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.693 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.694 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.694 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:59:27.695 DEBUG:boto:Data: 2023-10-11T14:59:27.695 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.696 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.696 DEBUG:boto:Params: {} 2023-10-11T14:59:27.697 DEBUG:boto:Token: None 2023-10-11T14:59:27.697 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2 2023-10-11T14:59:27.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L9GHr5b52aNoZBfAixlHzsaiofw= 2023-10-11T14:59:27.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L9GHr5b52aNoZBfAixlHzsaiofw=', 'Content-Length': '0'} 2023-10-11T14:59:27.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9e2c163a485934d-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.701 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.701 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.701 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.702 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:59:27.702 DEBUG:boto:Data: 2023-10-11T14:59:27.703 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.703 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.704 DEBUG:boto:Params: {} 2023-10-11T14:59:27.704 DEBUG:boto:Token: None 2023-10-11T14:59:27.705 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2 2023-10-11T14:59:27.705 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L9GHr5b52aNoZBfAixlHzsaiofw= 2023-10-11T14:59:27.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L9GHr5b52aNoZBfAixlHzsaiofw=', 'Content-Length': '0'} 2023-10-11T14:59:27.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd5afd330be1feee-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:27.708 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.709 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.709 DEBUG:boto:Method: GET 2023-10-11T14:59:27.710 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:59:27.710 DEBUG:boto:Data: 2023-10-11T14:59:27.710 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.711 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.712 DEBUG:boto:Params: {} 2023-10-11T14:59:27.712 DEBUG:boto:Token: None 2023-10-11T14:59:27.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2 2023-10-11T14:59:27.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:njhXiLRkzC/GbrXhM1aLAQGQUCI= 2023-10-11T14:59:27.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:njhXiLRkzC/GbrXhM1aLAQGQUCI=', 'Content-Length': '0'} 2023-10-11T14:59:27.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073a9108b7257e9b4-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.716 DEBUG:boto:path=/dghqvd-46/obj2 2023-10-11T14:59:27.716 DEBUG:boto:auth_path=/dghqvd-46/obj2 2023-10-11T14:59:27.717 DEBUG:boto:Method: GET 2023-10-11T14:59:27.717 DEBUG:boto:Path: /dghqvd-46/obj2 2023-10-11T14:59:27.718 DEBUG:boto:Data: 2023-10-11T14:59:27.718 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.719 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.719 DEBUG:boto:Params: {} 2023-10-11T14:59:27.720 DEBUG:boto:Token: None 2023-10-11T14:59:27.720 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj2 2023-10-11T14:59:27.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:njhXiLRkzC/GbrXhM1aLAQGQUCI= 2023-10-11T14:59:27.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:njhXiLRkzC/GbrXhM1aLAQGQUCI=', 'Content-Length': '0'} 2023-10-11T14:59:27.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf782db7f825c47c-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.724 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.724 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.725 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.725 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.725 DEBUG:boto:Method: GET 2023-10-11T14:59:27.726 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.726 DEBUG:boto:Data: 2023-10-11T14:59:27.727 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.728 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.728 DEBUG:boto:Params: {} 2023-10-11T14:59:27.729 DEBUG:boto:Token: None 2023-10-11T14:59:27.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d023154f1e0efbd1-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.733 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.734 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.734 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.735 DEBUG:boto:path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.735 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.736 DEBUG:boto:Method: GET 2023-10-11T14:59:27.736 DEBUG:boto:Path: /dghqvd-46/?prefix=obj2 2023-10-11T14:59:27.736 DEBUG:boto:Data: 2023-10-11T14:59:27.737 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.738 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.738 DEBUG:boto:Params: {} 2023-10-11T14:59:27.739 DEBUG:boto:Token: None 2023-10-11T14:59:27.739 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002953dd5e5c860681-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.786 DEBUG:boto:b'dghqvd-46obj21000falseobj22023-10-11T14:55:48.841Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:27.787 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.788 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.788 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.789 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.789 DEBUG:boto:Method: GET 2023-10-11T14:59:27.789 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.790 DEBUG:boto:Data: 2023-10-11T14:59:27.790 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.791 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.791 DEBUG:boto:Params: {} 2023-10-11T14:59:27.792 DEBUG:boto:Token: None 2023-10-11T14:59:27.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA= 2023-10-11T14:59:27.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA=', 'Content-Length': '0'} 2023-10-11T14:59:27.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a67db15e7234bbf-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.796 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.797 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.797 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.798 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.798 DEBUG:boto:Method: GET 2023-10-11T14:59:27.798 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.799 DEBUG:boto:Data: 2023-10-11T14:59:27.799 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.800 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.801 DEBUG:boto:Params: {} 2023-10-11T14:59:27.801 DEBUG:boto:Token: None 2023-10-11T14:59:27.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA= 2023-10-11T14:59:27.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA=', 'Content-Length': '0'} 2023-10-11T14:59:27.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eb8d6c497565c1d-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.805 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.806 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.806 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.807 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.807 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.808 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.808 DEBUG:boto:Data: 2023-10-11T14:59:27.808 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.809 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.809 DEBUG:boto:Params: {} 2023-10-11T14:59:27.810 DEBUG:boto:Token: None 2023-10-11T14:59:27.810 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JAAiRFcZBjARTo5fiZIpiwkUXJg= 2023-10-11T14:59:27.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JAAiRFcZBjARTo5fiZIpiwkUXJg=', 'Content-Length': '0'} 2023-10-11T14:59:27.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a39047a51c60d9a8-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.814 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.814 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.815 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.815 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.816 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.816 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.816 DEBUG:boto:Data: 2023-10-11T14:59:27.817 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.818 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.818 DEBUG:boto:Params: {} 2023-10-11T14:59:27.819 DEBUG:boto:Token: None 2023-10-11T14:59:27.819 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JAAiRFcZBjARTo5fiZIpiwkUXJg= 2023-10-11T14:59:27.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JAAiRFcZBjARTo5fiZIpiwkUXJg=', 'Content-Length': '0'} 2023-10-11T14:59:27.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d10fbec77d42d82-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:27.823 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.823 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.824 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.824 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.825 DEBUG:boto:Method: GET 2023-10-11T14:59:27.825 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.825 DEBUG:boto:Data: 2023-10-11T14:59:27.826 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.827 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.827 DEBUG:boto:Params: {} 2023-10-11T14:59:27.828 DEBUG:boto:Token: None 2023-10-11T14:59:27.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA= 2023-10-11T14:59:27.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA=', 'Content-Length': '0'} 2023-10-11T14:59:27.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003717059db488357a-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.832 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.832 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.833 DEBUG:boto:path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.833 DEBUG:boto:auth_path=/dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.833 DEBUG:boto:Method: GET 2023-10-11T14:59:27.834 DEBUG:boto:Path: /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.834 DEBUG:boto:Data: 2023-10-11T14:59:27.835 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.836 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.836 DEBUG:boto:Params: {} 2023-10-11T14:59:27.837 DEBUG:boto:Token: None 2023-10-11T14:59:27.837 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3?versionId=null 2023-10-11T14:59:27.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA= 2023-10-11T14:59:27.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0hQHrDtgo0zTIsuTwjqU8JxquA=', 'Content-Length': '0'} 2023-10-11T14:59:27.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b136be513d397b06-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.841 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.841 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.842 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.842 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.843 DEBUG:boto:Method: GET 2023-10-11T14:59:27.843 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.843 DEBUG:boto:Data: 2023-10-11T14:59:27.844 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.845 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.845 DEBUG:boto:Params: {} 2023-10-11T14:59:27.846 DEBUG:boto:Token: None 2023-10-11T14:59:27.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd61c50386173114-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.850 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.851 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.851 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.852 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.852 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.853 DEBUG:boto:Method: GET 2023-10-11T14:59:27.853 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.854 DEBUG:boto:Data: 2023-10-11T14:59:27.854 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.855 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.856 DEBUG:boto:Params: {} 2023-10-11T14:59:27.856 DEBUG:boto:Token: None 2023-10-11T14:59:27.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b7fdd038126f51-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.861 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.861 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.862 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.862 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.863 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:59:27.863 DEBUG:boto:Data: 2023-10-11T14:59:27.864 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.864 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.865 DEBUG:boto:Params: {} 2023-10-11T14:59:27.865 DEBUG:boto:Token: None 2023-10-11T14:59:27.866 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3 2023-10-11T14:59:27.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TCbRn2hlFzGtOtzfLtUEJS8tu7o= 2023-10-11T14:59:27.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TCbRn2hlFzGtOtzfLtUEJS8tu7o=', 'Content-Length': '0'} 2023-10-11T14:59:27.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009af3a9657a205252-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.869 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.869 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.870 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.870 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:59:27.871 DEBUG:boto:Data: 2023-10-11T14:59:27.871 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.872 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.872 DEBUG:boto:Params: {} 2023-10-11T14:59:27.873 DEBUG:boto:Token: None 2023-10-11T14:59:27.873 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3 2023-10-11T14:59:27.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TCbRn2hlFzGtOtzfLtUEJS8tu7o= 2023-10-11T14:59:27.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TCbRn2hlFzGtOtzfLtUEJS8tu7o=', 'Content-Length': '0'} 2023-10-11T14:59:27.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7827f4472dc346f-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:27.877 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.877 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.877 DEBUG:boto:Method: GET 2023-10-11T14:59:27.878 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:59:27.878 DEBUG:boto:Data: 2023-10-11T14:59:27.879 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.879 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.879 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.880 DEBUG:boto:Params: {} 2023-10-11T14:59:27.880 DEBUG:boto:Token: None 2023-10-11T14:59:27.881 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3 2023-10-11T14:59:27.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LhUb+4weQEzz8gzp5Vlc9ahQRGY= 2023-10-11T14:59:27.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LhUb+4weQEzz8gzp5Vlc9ahQRGY=', 'Content-Length': '0'} 2023-10-11T14:59:27.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9eba60b7419416e-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.885 DEBUG:boto:path=/dghqvd-46/obj3 2023-10-11T14:59:27.885 DEBUG:boto:auth_path=/dghqvd-46/obj3 2023-10-11T14:59:27.886 DEBUG:boto:Method: GET 2023-10-11T14:59:27.886 DEBUG:boto:Path: /dghqvd-46/obj3 2023-10-11T14:59:27.886 DEBUG:boto:Data: 2023-10-11T14:59:27.887 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.888 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.888 DEBUG:boto:Params: {} 2023-10-11T14:59:27.888 DEBUG:boto:Token: None 2023-10-11T14:59:27.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj3 2023-10-11T14:59:27.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LhUb+4weQEzz8gzp5Vlc9ahQRGY= 2023-10-11T14:59:27.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LhUb+4weQEzz8gzp5Vlc9ahQRGY=', 'Content-Length': '0'} 2023-10-11T14:59:27.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e39deb5d9b1a9d23-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.893 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.893 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.893 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.894 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.894 DEBUG:boto:Method: GET 2023-10-11T14:59:27.895 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.895 DEBUG:boto:Data: 2023-10-11T14:59:27.895 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.896 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.897 DEBUG:boto:Params: {} 2023-10-11T14:59:27.897 DEBUG:boto:Token: None 2023-10-11T14:59:27.898 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096cc44a09e2c5e1d-006526b84f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.902 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.902 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:27.903 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:27.903 DEBUG:boto:path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.904 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.904 DEBUG:boto:Method: GET 2023-10-11T14:59:27.905 DEBUG:boto:Path: /dghqvd-46/?prefix=obj3 2023-10-11T14:59:27.905 DEBUG:boto:Data: 2023-10-11T14:59:27.906 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.907 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.907 DEBUG:boto:Params: {} 2023-10-11T14:59:27.907 DEBUG:boto:Token: None 2023-10-11T14:59:27.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/ 2023-10-11T14:59:27.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc= 2023-10-11T14:59:27.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4g0af01t9vx99icxnBUIm4zhLmc=', 'Content-Length': '0'} 2023-10-11T14:59:27.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fbddd567526f95d-006526b84f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.912 DEBUG:boto:b'dghqvd-46obj31000falseobj32023-10-11T14:55:48.897Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:27.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:27.913 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.913 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.913 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.914 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.914 DEBUG:boto:Method: GET 2023-10-11T14:59:27.915 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.915 DEBUG:boto:Data: 2023-10-11T14:59:27.916 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.917 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.917 DEBUG:boto:Params: {} 2023-10-11T14:59:27.918 DEBUG:boto:Token: None 2023-10-11T14:59:27.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU= 2023-10-11T14:59:27.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU=', 'Content-Length': '0'} 2023-10-11T14:59:27.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfc43f91fb457bce-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.922 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.922 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.923 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.923 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.923 DEBUG:boto:Method: GET 2023-10-11T14:59:27.924 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.924 DEBUG:boto:Data: 2023-10-11T14:59:27.925 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.926 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.926 DEBUG:boto:Params: {} 2023-10-11T14:59:27.927 DEBUG:boto:Token: None 2023-10-11T14:59:27.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU= 2023-10-11T14:59:27.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU=', 'Content-Length': '0'} 2023-10-11T14:59:27.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003dd97fc82d434c32-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.931 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.931 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.932 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.932 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.933 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.933 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.934 DEBUG:boto:Data: 2023-10-11T14:59:27.934 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.935 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.935 DEBUG:boto:Params: {} 2023-10-11T14:59:27.936 DEBUG:boto:Token: None 2023-10-11T14:59:27.936 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WiDXNceiKggLzGP/xyqIzKsXflQ= 2023-10-11T14:59:27.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WiDXNceiKggLzGP/xyqIzKsXflQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016d3a576e6da1276-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.940 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.940 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.941 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.941 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.941 DEBUG:boto:Method: HEAD 2023-10-11T14:59:27.942 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.942 DEBUG:boto:Data: 2023-10-11T14:59:27.943 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:27.943 DEBUG:boto:Port: 8000 2023-10-11T14:59:27.944 DEBUG:boto:Params: {} 2023-10-11T14:59:27.944 DEBUG:boto:Token: None 2023-10-11T14:59:27.945 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WiDXNceiKggLzGP/xyqIzKsXflQ= 2023-10-11T14:59:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WiDXNceiKggLzGP/xyqIzKsXflQ=', 'Content-Length': '0'} 2023-10-11T14:59:27.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a98c193bb36b78cc-006526b84f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:27.948 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.949 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.949 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.950 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.950 DEBUG:boto:Method: GET 2023-10-11T14:59:27.951 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.951 DEBUG:boto:Data: 2023-10-11T14:59:27.952 DEBUG:boto:Headers: {} 2023-10-11T14:59:27.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:27.952 DEBUG:boto:Port: 8001 2023-10-11T14:59:27.953 DEBUG:boto:Params: {} 2023-10-11T14:59:27.953 DEBUG:boto:Token: None 2023-10-11T14:59:27.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:27 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:27.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU= 2023-10-11T14:59:27.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/wL07s6vi9UOhPW2EIg1i1SGKeU=', 'Content-Length': '0'} 2023-10-11T14:59:27.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000099c80bb5d1d7435-006526b84f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:27.998 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:27.999 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:27.999 DEBUG:boto:path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:28.000 DEBUG:boto:auth_path=/dghqvd-46/obj4?versionId=null 2023-10-11T14:59:28.000 DEBUG:boto:Method: GET 2023-10-11T14:59:28.001 DEBUG:boto:Path: /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:28.001 DEBUG:boto:Data: 2023-10-11T14:59:28.001 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.002 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.003 DEBUG:boto:Params: {} 2023-10-11T14:59:28.003 DEBUG:boto:Token: None 2023-10-11T14:59:28.004 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj4?versionId=null 2023-10-11T14:59:28.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HWwKVWrg0TRcqPsMIedCyFo3FWY= 2023-10-11T14:59:28.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HWwKVWrg0TRcqPsMIedCyFo3FWY=', 'Content-Length': '0'} 2023-10-11T14:59:28.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6a85abdf31c3d78-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.007 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.008 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.008 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.009 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.009 DEBUG:boto:Method: GET 2023-10-11T14:59:28.010 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.010 DEBUG:boto:Data: 2023-10-11T14:59:28.011 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.011 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.011 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.012 DEBUG:boto:Params: {} 2023-10-11T14:59:28.012 DEBUG:boto:Token: None 2023-10-11T14:59:28.013 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011a83cf3b1d28ca5-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.017 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.018 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.018 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.019 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.019 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.020 DEBUG:boto:Method: GET 2023-10-11T14:59:28.020 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.021 DEBUG:boto:Data: 2023-10-11T14:59:28.021 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.022 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.022 DEBUG:boto:Params: {} 2023-10-11T14:59:28.023 DEBUG:boto:Token: None 2023-10-11T14:59:28.023 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004383bed7eebebca3-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.027 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.028 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:28.028 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:28.029 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.029 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:59:28.030 DEBUG:boto:Data: 2023-10-11T14:59:28.030 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.031 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.031 DEBUG:boto:Params: {} 2023-10-11T14:59:28.032 DEBUG:boto:Token: None 2023-10-11T14:59:28.032 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj4 2023-10-11T14:59:28.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KHhW98i1VLHehKNyVPKY/yipZyI= 2023-10-11T14:59:28.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KHhW98i1VLHehKNyVPKY/yipZyI=', 'Content-Length': '0'} 2023-10-11T14:59:28.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8ad9bc538e7e236-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.036 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:28.036 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:28.037 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.037 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:59:28.037 DEBUG:boto:Data: 2023-10-11T14:59:28.038 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.039 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.039 DEBUG:boto:Params: {} 2023-10-11T14:59:28.039 DEBUG:boto:Token: None 2023-10-11T14:59:28.040 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj4 2023-10-11T14:59:28.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KHhW98i1VLHehKNyVPKY/yipZyI= 2023-10-11T14:59:28.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KHhW98i1VLHehKNyVPKY/yipZyI=', 'Content-Length': '0'} 2023-10-11T14:59:28.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7f4eb10a118c1dc-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:28.044 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:28.044 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:28.045 DEBUG:boto:Method: GET 2023-10-11T14:59:28.045 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:59:28.046 DEBUG:boto:Data: 2023-10-11T14:59:28.046 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.046 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.047 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.047 DEBUG:boto:Params: {} 2023-10-11T14:59:28.048 DEBUG:boto:Token: None 2023-10-11T14:59:28.048 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj4 2023-10-11T14:59:28.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MLcTLHHezmw5BnbGMrnZ3WJQLyE= 2023-10-11T14:59:28.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MLcTLHHezmw5BnbGMrnZ3WJQLyE=', 'Content-Length': '0'} 2023-10-11T14:59:28.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005eaec275cc07ffd3-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.052 DEBUG:boto:path=/dghqvd-46/obj4 2023-10-11T14:59:28.052 DEBUG:boto:auth_path=/dghqvd-46/obj4 2023-10-11T14:59:28.053 DEBUG:boto:Method: GET 2023-10-11T14:59:28.053 DEBUG:boto:Path: /dghqvd-46/obj4 2023-10-11T14:59:28.053 DEBUG:boto:Data: 2023-10-11T14:59:28.054 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.055 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.055 DEBUG:boto:Params: {} 2023-10-11T14:59:28.056 DEBUG:boto:Token: None 2023-10-11T14:59:28.056 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj4 2023-10-11T14:59:28.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MLcTLHHezmw5BnbGMrnZ3WJQLyE= 2023-10-11T14:59:28.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MLcTLHHezmw5BnbGMrnZ3WJQLyE=', 'Content-Length': '0'} 2023-10-11T14:59:28.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3b02003f14e023f-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.106 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.107 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.107 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.108 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.108 DEBUG:boto:Method: GET 2023-10-11T14:59:28.109 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.109 DEBUG:boto:Data: 2023-10-11T14:59:28.110 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.110 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.111 DEBUG:boto:Params: {} 2023-10-11T14:59:28.111 DEBUG:boto:Token: None 2023-10-11T14:59:28.112 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45a870335453bd2-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.116 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.117 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.117 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.118 DEBUG:boto:path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.118 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.118 DEBUG:boto:Method: GET 2023-10-11T14:59:28.119 DEBUG:boto:Path: /dghqvd-46/?prefix=obj4 2023-10-11T14:59:28.119 DEBUG:boto:Data: 2023-10-11T14:59:28.120 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.121 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.121 DEBUG:boto:Params: {} 2023-10-11T14:59:28.121 DEBUG:boto:Token: None 2023-10-11T14:59:28.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000190d9b20b45115e3-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.127 DEBUG:boto:b'dghqvd-46obj41000falseobj42023-10-11T14:55:48.916Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:28.128 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.128 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.129 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.129 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.130 DEBUG:boto:Method: GET 2023-10-11T14:59:28.130 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.131 DEBUG:boto:Data: 2023-10-11T14:59:28.131 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.132 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.132 DEBUG:boto:Params: {} 2023-10-11T14:59:28.133 DEBUG:boto:Token: None 2023-10-11T14:59:28.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo= 2023-10-11T14:59:28.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo=', 'Content-Length': '0'} 2023-10-11T14:59:28.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070d49f2f6aed0e35-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.182 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.183 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.183 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.184 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.184 DEBUG:boto:Method: GET 2023-10-11T14:59:28.185 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.185 DEBUG:boto:Data: 2023-10-11T14:59:28.186 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.187 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.187 DEBUG:boto:Params: {} 2023-10-11T14:59:28.187 DEBUG:boto:Token: None 2023-10-11T14:59:28.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo= 2023-10-11T14:59:28.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo=', 'Content-Length': '0'} 2023-10-11T14:59:28.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003803e643aff61869-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.192 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.192 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.192 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.193 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.193 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.194 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.194 DEBUG:boto:Data: 2023-10-11T14:59:28.194 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.195 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.196 DEBUG:boto:Params: {} 2023-10-11T14:59:28.196 DEBUG:boto:Token: None 2023-10-11T14:59:28.197 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+Djt+bs5GaA0sJoIa2CpI0E7UTc= 2023-10-11T14:59:28.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+Djt+bs5GaA0sJoIa2CpI0E7UTc=', 'Content-Length': '0'} 2023-10-11T14:59:28.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069443add57d9cf19-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.200 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.201 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.201 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.201 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.202 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.202 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.203 DEBUG:boto:Data: 2023-10-11T14:59:28.203 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.204 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.204 DEBUG:boto:Params: {} 2023-10-11T14:59:28.205 DEBUG:boto:Token: None 2023-10-11T14:59:28.205 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+Djt+bs5GaA0sJoIa2CpI0E7UTc= 2023-10-11T14:59:28.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+Djt+bs5GaA0sJoIa2CpI0E7UTc=', 'Content-Length': '0'} 2023-10-11T14:59:28.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000426cbb9ad76f94e3-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:28.209 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.209 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.210 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.210 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.211 DEBUG:boto:Method: GET 2023-10-11T14:59:28.211 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.212 DEBUG:boto:Data: 2023-10-11T14:59:28.212 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.212 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.213 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.213 DEBUG:boto:Params: {} 2023-10-11T14:59:28.214 DEBUG:boto:Token: None 2023-10-11T14:59:28.214 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo= 2023-10-11T14:59:28.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo=', 'Content-Length': '0'} 2023-10-11T14:59:28.217 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d92e99f3a048697-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.218 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.218 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.219 DEBUG:boto:path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.219 DEBUG:boto:auth_path=/dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.219 DEBUG:boto:Method: GET 2023-10-11T14:59:28.220 DEBUG:boto:Path: /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.220 DEBUG:boto:Data: 2023-10-11T14:59:28.221 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.221 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.222 DEBUG:boto:Params: {} 2023-10-11T14:59:28.222 DEBUG:boto:Token: None 2023-10-11T14:59:28.223 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5?versionId=null 2023-10-11T14:59:28.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo= 2023-10-11T14:59:28.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mAwNpi21ljVKTt3PA7RxHVQUFo=', 'Content-Length': '0'} 2023-10-11T14:59:28.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dc8fb9ed4dc6f58-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.226 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.227 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.227 DEBUG:boto:path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.227 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.228 DEBUG:boto:Method: GET 2023-10-11T14:59:28.228 DEBUG:boto:Path: /dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.229 DEBUG:boto:Data: 2023-10-11T14:59:28.229 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.230 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.231 DEBUG:boto:Params: {} 2023-10-11T14:59:28.231 DEBUG:boto:Token: None 2023-10-11T14:59:28.231 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a5cb5c20ff3553-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.236 DEBUG:boto:b'dghqvd-46obj51000falseobj52023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.236 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.237 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.237 DEBUG:boto:path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.238 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.238 DEBUG:boto:Method: GET 2023-10-11T14:59:28.239 DEBUG:boto:Path: /dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.239 DEBUG:boto:Data: 2023-10-11T14:59:28.239 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.240 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.241 DEBUG:boto:Params: {} 2023-10-11T14:59:28.241 DEBUG:boto:Token: None 2023-10-11T14:59:28.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f209d78b00283995-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.246 DEBUG:boto:b'dghqvd-46obj51000falseobj52023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.246 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.247 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.247 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.248 DEBUG:boto:Path: /dghqvd-46/obj5 2023-10-11T14:59:28.248 DEBUG:boto:Data: 2023-10-11T14:59:28.248 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.249 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.250 DEBUG:boto:Params: {} 2023-10-11T14:59:28.250 DEBUG:boto:Token: None 2023-10-11T14:59:28.251 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5 2023-10-11T14:59:28.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZueX2uHty5PRBV54V2C7bvLUxFs= 2023-10-11T14:59:28.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZueX2uHty5PRBV54V2C7bvLUxFs=', 'Content-Length': '0'} 2023-10-11T14:59:28.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007b5db7cc9e2710b-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.254 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.255 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.255 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.255 DEBUG:boto:Path: /dghqvd-46/obj5 2023-10-11T14:59:28.256 DEBUG:boto:Data: 2023-10-11T14:59:28.256 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.257 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.257 DEBUG:boto:Params: {} 2023-10-11T14:59:28.258 DEBUG:boto:Token: None 2023-10-11T14:59:28.258 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5 2023-10-11T14:59:28.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZueX2uHty5PRBV54V2C7bvLUxFs= 2023-10-11T14:59:28.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZueX2uHty5PRBV54V2C7bvLUxFs=', 'Content-Length': '0'} 2023-10-11T14:59:28.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ccc02a33854da23-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:28.262 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.263 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.263 DEBUG:boto:Method: GET 2023-10-11T14:59:28.264 DEBUG:boto:Path: /dghqvd-46/obj5 2023-10-11T14:59:28.264 DEBUG:boto:Data: 2023-10-11T14:59:28.264 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.265 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.266 DEBUG:boto:Params: {} 2023-10-11T14:59:28.266 DEBUG:boto:Token: None 2023-10-11T14:59:28.267 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5 2023-10-11T14:59:28.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EEKkXIoqlQrFgKYJZ6uIiqtXkf0= 2023-10-11T14:59:28.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EEKkXIoqlQrFgKYJZ6uIiqtXkf0=', 'Content-Length': '0'} 2023-10-11T14:59:28.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdf4649f98261859-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.270 DEBUG:boto:path=/dghqvd-46/obj5 2023-10-11T14:59:28.271 DEBUG:boto:auth_path=/dghqvd-46/obj5 2023-10-11T14:59:28.271 DEBUG:boto:Method: GET 2023-10-11T14:59:28.272 DEBUG:boto:Path: /dghqvd-46/obj5 2023-10-11T14:59:28.272 DEBUG:boto:Data: 2023-10-11T14:59:28.273 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.273 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.274 DEBUG:boto:Params: {} 2023-10-11T14:59:28.274 DEBUG:boto:Token: None 2023-10-11T14:59:28.275 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj5 2023-10-11T14:59:28.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EEKkXIoqlQrFgKYJZ6uIiqtXkf0= 2023-10-11T14:59:28.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EEKkXIoqlQrFgKYJZ6uIiqtXkf0=', 'Content-Length': '0'} 2023-10-11T14:59:28.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000820e8f728b9f5f8b-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.279 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.279 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.279 DEBUG:boto:path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.280 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.280 DEBUG:boto:Method: GET 2023-10-11T14:59:28.281 DEBUG:boto:Path: /dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.281 DEBUG:boto:Data: 2023-10-11T14:59:28.281 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.282 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.283 DEBUG:boto:Params: {} 2023-10-11T14:59:28.283 DEBUG:boto:Token: None 2023-10-11T14:59:28.284 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d19500c3d5f2e3-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.288 DEBUG:boto:b'dghqvd-46obj51000falseobj52023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.288 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.289 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.289 DEBUG:boto:path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.289 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.290 DEBUG:boto:Method: GET 2023-10-11T14:59:28.290 DEBUG:boto:Path: /dghqvd-46/?prefix=obj5 2023-10-11T14:59:28.291 DEBUG:boto:Data: 2023-10-11T14:59:28.291 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.292 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.292 DEBUG:boto:Params: {} 2023-10-11T14:59:28.293 DEBUG:boto:Token: None 2023-10-11T14:59:28.293 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002fdae22ad704230-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.297 DEBUG:boto:b'dghqvd-46obj51000falseobj52023-10-11T14:58:51.784Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:28.298 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.299 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.299 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.300 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.300 DEBUG:boto:Method: GET 2023-10-11T14:59:28.301 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.301 DEBUG:boto:Data: 2023-10-11T14:59:28.302 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.302 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.303 DEBUG:boto:Params: {} 2023-10-11T14:59:28.303 DEBUG:boto:Token: None 2023-10-11T14:59:28.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.304 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs= 2023-10-11T14:59:28.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs=', 'Content-Length': '0'} 2023-10-11T14:59:28.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083eda738bf4692b2-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.350 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.351 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.351 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.352 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.352 DEBUG:boto:Method: GET 2023-10-11T14:59:28.352 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.353 DEBUG:boto:Data: 2023-10-11T14:59:28.353 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.354 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.355 DEBUG:boto:Params: {} 2023-10-11T14:59:28.355 DEBUG:boto:Token: None 2023-10-11T14:59:28.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs= 2023-10-11T14:59:28.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs=', 'Content-Length': '0'} 2023-10-11T14:59:28.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbcdffb67369c034-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.359 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.360 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.360 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.361 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.361 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.362 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.362 DEBUG:boto:Data: 2023-10-11T14:59:28.362 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.363 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.364 DEBUG:boto:Params: {} 2023-10-11T14:59:28.364 DEBUG:boto:Token: None 2023-10-11T14:59:28.365 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SdOUTXcmPQYKoMKFQqbfexqByzw= 2023-10-11T14:59:28.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SdOUTXcmPQYKoMKFQqbfexqByzw=', 'Content-Length': '0'} 2023-10-11T14:59:28.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000817606fcd1b38ad-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.368 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.369 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.369 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.370 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.370 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.371 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.371 DEBUG:boto:Data: 2023-10-11T14:59:28.371 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.372 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.373 DEBUG:boto:Params: {} 2023-10-11T14:59:28.373 DEBUG:boto:Token: None 2023-10-11T14:59:28.373 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SdOUTXcmPQYKoMKFQqbfexqByzw= 2023-10-11T14:59:28.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SdOUTXcmPQYKoMKFQqbfexqByzw=', 'Content-Length': '0'} 2023-10-11T14:59:28.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004db1a31bdbd1d062-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:28.377 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.378 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.378 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.379 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.379 DEBUG:boto:Method: GET 2023-10-11T14:59:28.379 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.380 DEBUG:boto:Data: 2023-10-11T14:59:28.380 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.381 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.381 DEBUG:boto:Params: {} 2023-10-11T14:59:28.382 DEBUG:boto:Token: None 2023-10-11T14:59:28.382 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs= 2023-10-11T14:59:28.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs=', 'Content-Length': '0'} 2023-10-11T14:59:28.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1351a2f65c3a9c8-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.386 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.386 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.387 DEBUG:boto:path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.387 DEBUG:boto:auth_path=/dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.388 DEBUG:boto:Method: GET 2023-10-11T14:59:28.388 DEBUG:boto:Path: /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.388 DEBUG:boto:Data: 2023-10-11T14:59:28.389 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.390 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.390 DEBUG:boto:Params: {} 2023-10-11T14:59:28.391 DEBUG:boto:Token: None 2023-10-11T14:59:28.391 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6?versionId=null 2023-10-11T14:59:28.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs= 2023-10-11T14:59:28.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:42o9aTDvgbWg/S1HXHi6jDQstbs=', 'Content-Length': '0'} 2023-10-11T14:59:28.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039dff7a756790b94-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.434 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.435 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.435 DEBUG:boto:path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.436 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.436 DEBUG:boto:Method: GET 2023-10-11T14:59:28.437 DEBUG:boto:Path: /dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.437 DEBUG:boto:Data: 2023-10-11T14:59:28.438 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.438 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.439 DEBUG:boto:Params: {} 2023-10-11T14:59:28.439 DEBUG:boto:Token: None 2023-10-11T14:59:28.440 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bce554ac74a18e4f-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.444 DEBUG:boto:b'dghqvd-46obj61000falseobj62023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.444 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.445 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.445 DEBUG:boto:path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.446 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.446 DEBUG:boto:Method: GET 2023-10-11T14:59:28.447 DEBUG:boto:Path: /dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.447 DEBUG:boto:Data: 2023-10-11T14:59:28.447 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.448 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.448 DEBUG:boto:Params: {} 2023-10-11T14:59:28.449 DEBUG:boto:Token: None 2023-10-11T14:59:28.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bffeb594d68e8e32-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.453 DEBUG:boto:b'dghqvd-46obj61000falseobj62023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.454 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.454 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.455 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.455 DEBUG:boto:Path: /dghqvd-46/obj6 2023-10-11T14:59:28.456 DEBUG:boto:Data: 2023-10-11T14:59:28.456 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.456 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.457 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.457 DEBUG:boto:Params: {} 2023-10-11T14:59:28.458 DEBUG:boto:Token: None 2023-10-11T14:59:28.458 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6 2023-10-11T14:59:28.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXxFB6i/wmZD3jQ2zTVyoIGJqjc= 2023-10-11T14:59:28.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXxFB6i/wmZD3jQ2zTVyoIGJqjc=', 'Content-Length': '0'} 2023-10-11T14:59:28.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005671da25bf8be040-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.461 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.462 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.462 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.463 DEBUG:boto:Path: /dghqvd-46/obj6 2023-10-11T14:59:28.463 DEBUG:boto:Data: 2023-10-11T14:59:28.464 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.465 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.465 DEBUG:boto:Params: {} 2023-10-11T14:59:28.465 DEBUG:boto:Token: None 2023-10-11T14:59:28.466 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6 2023-10-11T14:59:28.466 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXxFB6i/wmZD3jQ2zTVyoIGJqjc= 2023-10-11T14:59:28.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXxFB6i/wmZD3jQ2zTVyoIGJqjc=', 'Content-Length': '0'} 2023-10-11T14:59:28.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010a9511d7573ddc4-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:28.470 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.470 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.471 DEBUG:boto:Method: GET 2023-10-11T14:59:28.471 DEBUG:boto:Path: /dghqvd-46/obj6 2023-10-11T14:59:28.472 DEBUG:boto:Data: 2023-10-11T14:59:28.472 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.473 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.473 DEBUG:boto:Params: {} 2023-10-11T14:59:28.474 DEBUG:boto:Token: None 2023-10-11T14:59:28.474 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6 2023-10-11T14:59:28.475 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hzEoUy6eQb+3I+e5Dy9dfgHvCUQ= 2023-10-11T14:59:28.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hzEoUy6eQb+3I+e5Dy9dfgHvCUQ=', 'Content-Length': '0'} 2023-10-11T14:59:28.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df9c2d540e5acdc8-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.478 DEBUG:boto:path=/dghqvd-46/obj6 2023-10-11T14:59:28.479 DEBUG:boto:auth_path=/dghqvd-46/obj6 2023-10-11T14:59:28.479 DEBUG:boto:Method: GET 2023-10-11T14:59:28.480 DEBUG:boto:Path: /dghqvd-46/obj6 2023-10-11T14:59:28.480 DEBUG:boto:Data: 2023-10-11T14:59:28.481 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.482 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.482 DEBUG:boto:Params: {} 2023-10-11T14:59:28.482 DEBUG:boto:Token: None 2023-10-11T14:59:28.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj6 2023-10-11T14:59:28.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hzEoUy6eQb+3I+e5Dy9dfgHvCUQ= 2023-10-11T14:59:28.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hzEoUy6eQb+3I+e5Dy9dfgHvCUQ=', 'Content-Length': '0'} 2023-10-11T14:59:28.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce456f0da7e96678-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.487 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.487 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.488 DEBUG:boto:path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.488 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.489 DEBUG:boto:Method: GET 2023-10-11T14:59:28.489 DEBUG:boto:Path: /dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.489 DEBUG:boto:Data: 2023-10-11T14:59:28.490 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.491 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.491 DEBUG:boto:Params: {} 2023-10-11T14:59:28.492 DEBUG:boto:Token: None 2023-10-11T14:59:28.492 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8d55cd13c354d7-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.496 DEBUG:boto:b'dghqvd-46obj61000falseobj62023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.497 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.497 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.498 DEBUG:boto:path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.498 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.499 DEBUG:boto:Method: GET 2023-10-11T14:59:28.499 DEBUG:boto:Path: /dghqvd-46/?prefix=obj6 2023-10-11T14:59:28.500 DEBUG:boto:Data: 2023-10-11T14:59:28.500 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.501 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.501 DEBUG:boto:Params: {} 2023-10-11T14:59:28.502 DEBUG:boto:Token: None 2023-10-11T14:59:28.503 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.503 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5dd99f017d7aad2-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.507 DEBUG:boto:b'dghqvd-46obj61000falseobj62023-10-11T14:58:51.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:28.508 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.508 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.509 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.509 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.510 DEBUG:boto:Method: GET 2023-10-11T14:59:28.510 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.511 DEBUG:boto:Data: 2023-10-11T14:59:28.511 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.512 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.512 DEBUG:boto:Params: {} 2023-10-11T14:59:28.513 DEBUG:boto:Token: None 2023-10-11T14:59:28.513 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.514 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE= 2023-10-11T14:59:28.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE=', 'Content-Length': '0'} 2023-10-11T14:59:28.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e758d6c8559a2d3f-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.517 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.517 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.518 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.518 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.519 DEBUG:boto:Method: GET 2023-10-11T14:59:28.519 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.519 DEBUG:boto:Data: 2023-10-11T14:59:28.520 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.521 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.521 DEBUG:boto:Params: {} 2023-10-11T14:59:28.522 DEBUG:boto:Token: None 2023-10-11T14:59:28.522 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.523 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE= 2023-10-11T14:59:28.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE=', 'Content-Length': '0'} 2023-10-11T14:59:28.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8cc3c1badcf22f6-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.526 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.526 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.526 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.527 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.527 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.528 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.528 DEBUG:boto:Data: 2023-10-11T14:59:28.528 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.529 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.530 DEBUG:boto:Params: {} 2023-10-11T14:59:28.530 DEBUG:boto:Token: None 2023-10-11T14:59:28.531 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeeG2jlFYDaL8flEHZdWTBMzQK4= 2023-10-11T14:59:28.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeeG2jlFYDaL8flEHZdWTBMzQK4=', 'Content-Length': '0'} 2023-10-11T14:59:28.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c4c7081abb5dd6-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.534 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.535 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.535 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.536 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.536 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.537 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.537 DEBUG:boto:Data: 2023-10-11T14:59:28.537 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.538 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.539 DEBUG:boto:Params: {} 2023-10-11T14:59:28.539 DEBUG:boto:Token: None 2023-10-11T14:59:28.540 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeeG2jlFYDaL8flEHZdWTBMzQK4= 2023-10-11T14:59:28.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeeG2jlFYDaL8flEHZdWTBMzQK4=', 'Content-Length': '0'} 2023-10-11T14:59:28.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000906b002c8007320a-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:28.543 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.544 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.544 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.545 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.545 DEBUG:boto:Method: GET 2023-10-11T14:59:28.546 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.546 DEBUG:boto:Data: 2023-10-11T14:59:28.547 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.547 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.548 DEBUG:boto:Params: {} 2023-10-11T14:59:28.548 DEBUG:boto:Token: None 2023-10-11T14:59:28.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE= 2023-10-11T14:59:28.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE=', 'Content-Length': '0'} 2023-10-11T14:59:28.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f07d9d86cbef66d7-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.553 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.553 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.554 DEBUG:boto:path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.554 DEBUG:boto:auth_path=/dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.555 DEBUG:boto:Method: GET 2023-10-11T14:59:28.555 DEBUG:boto:Path: /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.556 DEBUG:boto:Data: 2023-10-11T14:59:28.556 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.557 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.557 DEBUG:boto:Params: {} 2023-10-11T14:59:28.558 DEBUG:boto:Token: None 2023-10-11T14:59:28.558 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7?versionId=null 2023-10-11T14:59:28.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE= 2023-10-11T14:59:28.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7mKSiwHhn5TfiNTFbI99Wwfi2jE=', 'Content-Length': '0'} 2023-10-11T14:59:28.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e3ec5fedd771962-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.562 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.563 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.563 DEBUG:boto:path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.563 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.564 DEBUG:boto:Method: GET 2023-10-11T14:59:28.564 DEBUG:boto:Path: /dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.565 DEBUG:boto:Data: 2023-10-11T14:59:28.565 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.566 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.567 DEBUG:boto:Params: {} 2023-10-11T14:59:28.567 DEBUG:boto:Token: None 2023-10-11T14:59:28.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe4c46e08a71edf-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.572 DEBUG:boto:b'dghqvd-46obj71000falseobj72023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.573 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.573 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.574 DEBUG:boto:path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.574 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.575 DEBUG:boto:Method: GET 2023-10-11T14:59:28.575 DEBUG:boto:Path: /dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.575 DEBUG:boto:Data: 2023-10-11T14:59:28.576 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.576 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.577 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.577 DEBUG:boto:Params: {} 2023-10-11T14:59:28.578 DEBUG:boto:Token: None 2023-10-11T14:59:28.578 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45dc9490120467b-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.582 DEBUG:boto:b'dghqvd-46obj71000falseobj72023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.583 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.583 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.584 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.584 DEBUG:boto:Path: /dghqvd-46/obj7 2023-10-11T14:59:28.584 DEBUG:boto:Data: 2023-10-11T14:59:28.585 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.585 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.586 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.586 DEBUG:boto:Params: {} 2023-10-11T14:59:28.587 DEBUG:boto:Token: None 2023-10-11T14:59:28.587 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7 2023-10-11T14:59:28.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/tzJtZDwAu+L1++R4Fts13HJjdk= 2023-10-11T14:59:28.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/tzJtZDwAu+L1++R4Fts13HJjdk=', 'Content-Length': '0'} 2023-10-11T14:59:28.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c90d0f39ce21ee-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.590 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.591 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.591 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.592 DEBUG:boto:Path: /dghqvd-46/obj7 2023-10-11T14:59:28.592 DEBUG:boto:Data: 2023-10-11T14:59:28.593 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.593 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.594 DEBUG:boto:Params: {} 2023-10-11T14:59:28.594 DEBUG:boto:Token: None 2023-10-11T14:59:28.595 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7 2023-10-11T14:59:28.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/tzJtZDwAu+L1++R4Fts13HJjdk= 2023-10-11T14:59:28.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/tzJtZDwAu+L1++R4Fts13HJjdk=', 'Content-Length': '0'} 2023-10-11T14:59:28.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006f78b5bafde63be-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:28.599 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.599 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.600 DEBUG:boto:Method: GET 2023-10-11T14:59:28.600 DEBUG:boto:Path: /dghqvd-46/obj7 2023-10-11T14:59:28.601 DEBUG:boto:Data: 2023-10-11T14:59:28.601 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.602 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.603 DEBUG:boto:Params: {} 2023-10-11T14:59:28.605 DEBUG:boto:Token: None 2023-10-11T14:59:28.606 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7 2023-10-11T14:59:28.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LveB2vn5vholsY1evrXY5HO6k1w= 2023-10-11T14:59:28.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LveB2vn5vholsY1evrXY5HO6k1w=', 'Content-Length': '0'} 2023-10-11T14:59:28.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d8d5bcac0d67e2d-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.650 DEBUG:boto:path=/dghqvd-46/obj7 2023-10-11T14:59:28.651 DEBUG:boto:auth_path=/dghqvd-46/obj7 2023-10-11T14:59:28.651 DEBUG:boto:Method: GET 2023-10-11T14:59:28.652 DEBUG:boto:Path: /dghqvd-46/obj7 2023-10-11T14:59:28.652 DEBUG:boto:Data: 2023-10-11T14:59:28.653 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.653 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.654 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.654 DEBUG:boto:Params: {} 2023-10-11T14:59:28.655 DEBUG:boto:Token: None 2023-10-11T14:59:28.655 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj7 2023-10-11T14:59:28.656 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LveB2vn5vholsY1evrXY5HO6k1w= 2023-10-11T14:59:28.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LveB2vn5vholsY1evrXY5HO6k1w=', 'Content-Length': '0'} 2023-10-11T14:59:28.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f55cdbfb718845b-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.659 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.659 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.660 DEBUG:boto:path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.660 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.661 DEBUG:boto:Method: GET 2023-10-11T14:59:28.661 DEBUG:boto:Path: /dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.662 DEBUG:boto:Data: 2023-10-11T14:59:28.662 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.663 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.663 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.663 DEBUG:boto:Params: {} 2023-10-11T14:59:28.664 DEBUG:boto:Token: None 2023-10-11T14:59:28.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc6ce338afc3ae89-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.668 DEBUG:boto:b'dghqvd-46obj71000falseobj72023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.669 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.669 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.670 DEBUG:boto:path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.670 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.671 DEBUG:boto:Method: GET 2023-10-11T14:59:28.671 DEBUG:boto:Path: /dghqvd-46/?prefix=obj7 2023-10-11T14:59:28.672 DEBUG:boto:Data: 2023-10-11T14:59:28.672 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.673 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.674 DEBUG:boto:Params: {} 2023-10-11T14:59:28.674 DEBUG:boto:Token: None 2023-10-11T14:59:28.675 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc636804d2fa7577-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.722 DEBUG:boto:b'dghqvd-46obj71000falseobj72023-10-11T14:58:51.857Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:28.724 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.724 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.724 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.725 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.725 DEBUG:boto:Method: GET 2023-10-11T14:59:28.726 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.726 DEBUG:boto:Data: 2023-10-11T14:59:28.727 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.728 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.728 DEBUG:boto:Params: {} 2023-10-11T14:59:28.728 DEBUG:boto:Token: None 2023-10-11T14:59:28.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE= 2023-10-11T14:59:28.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE=', 'Content-Length': '0'} 2023-10-11T14:59:28.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4fb77345eb3ce0a-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.733 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.733 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.734 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.734 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.735 DEBUG:boto:Method: GET 2023-10-11T14:59:28.735 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.736 DEBUG:boto:Data: 2023-10-11T14:59:28.736 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.737 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.737 DEBUG:boto:Params: {} 2023-10-11T14:59:28.738 DEBUG:boto:Token: None 2023-10-11T14:59:28.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE= 2023-10-11T14:59:28.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE=', 'Content-Length': '0'} 2023-10-11T14:59:28.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000178ecd14e27000-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.742 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.743 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.743 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.743 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.744 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.744 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.745 DEBUG:boto:Data: 2023-10-11T14:59:28.745 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.745 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.746 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.746 DEBUG:boto:Params: {} 2023-10-11T14:59:28.747 DEBUG:boto:Token: None 2023-10-11T14:59:28.747 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XoKqvJDcq4gU6TtsAhm1DlO+1rc= 2023-10-11T14:59:28.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XoKqvJDcq4gU6TtsAhm1DlO+1rc=', 'Content-Length': '0'} 2023-10-11T14:59:28.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091aadb7789b3f800-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.750 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.751 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.751 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.752 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.752 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.752 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.753 DEBUG:boto:Data: 2023-10-11T14:59:28.753 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.754 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.754 DEBUG:boto:Params: {} 2023-10-11T14:59:28.755 DEBUG:boto:Token: None 2023-10-11T14:59:28.755 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XoKqvJDcq4gU6TtsAhm1DlO+1rc= 2023-10-11T14:59:28.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XoKqvJDcq4gU6TtsAhm1DlO+1rc=', 'Content-Length': '0'} 2023-10-11T14:59:28.758 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041c1e7ba3ba273cf-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:28.759 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.760 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.760 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.760 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.761 DEBUG:boto:Method: GET 2023-10-11T14:59:28.761 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.762 DEBUG:boto:Data: 2023-10-11T14:59:28.762 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.763 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.763 DEBUG:boto:Params: {} 2023-10-11T14:59:28.764 DEBUG:boto:Token: None 2023-10-11T14:59:28.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE= 2023-10-11T14:59:28.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE=', 'Content-Length': '0'} 2023-10-11T14:59:28.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051cfafe36d0b9cc6-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.768 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.768 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.769 DEBUG:boto:path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.769 DEBUG:boto:auth_path=/dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.770 DEBUG:boto:Method: GET 2023-10-11T14:59:28.770 DEBUG:boto:Path: /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.770 DEBUG:boto:Data: 2023-10-11T14:59:28.771 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.772 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.772 DEBUG:boto:Params: {} 2023-10-11T14:59:28.773 DEBUG:boto:Token: None 2023-10-11T14:59:28.773 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8?versionId=null 2023-10-11T14:59:28.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE= 2023-10-11T14:59:28.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:urH6c+vrsc93L3v+CHxgM/aHaYE=', 'Content-Length': '0'} 2023-10-11T14:59:28.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a0b48344d36190a-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.777 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.777 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.778 DEBUG:boto:path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.778 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.779 DEBUG:boto:Method: GET 2023-10-11T14:59:28.779 DEBUG:boto:Path: /dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.780 DEBUG:boto:Data: 2023-10-11T14:59:28.780 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.781 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.781 DEBUG:boto:Params: {} 2023-10-11T14:59:28.782 DEBUG:boto:Token: None 2023-10-11T14:59:28.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ffc7542cf2f0bc-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.786 DEBUG:boto:b'dghqvd-46obj81000falseobj82023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.787 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.788 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.788 DEBUG:boto:path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.788 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.789 DEBUG:boto:Method: GET 2023-10-11T14:59:28.789 DEBUG:boto:Path: /dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.789 DEBUG:boto:Data: 2023-10-11T14:59:28.790 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.791 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.791 DEBUG:boto:Params: {} 2023-10-11T14:59:28.792 DEBUG:boto:Token: None 2023-10-11T14:59:28.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151397761e2b8160-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.796 DEBUG:boto:b'dghqvd-46obj81000falseobj82023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.797 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.797 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.797 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.798 DEBUG:boto:Path: /dghqvd-46/obj8 2023-10-11T14:59:28.798 DEBUG:boto:Data: 2023-10-11T14:59:28.799 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.799 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.800 DEBUG:boto:Params: {} 2023-10-11T14:59:28.800 DEBUG:boto:Token: None 2023-10-11T14:59:28.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8 2023-10-11T14:59:28.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZlB9qJPZWVE/9+T+Fh0Y+DxwGnM= 2023-10-11T14:59:28.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZlB9qJPZWVE/9+T+Fh0Y+DxwGnM=', 'Content-Length': '0'} 2023-10-11T14:59:28.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037a14ba79e575146-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.804 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.805 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.805 DEBUG:boto:Method: HEAD 2023-10-11T14:59:28.805 DEBUG:boto:Path: /dghqvd-46/obj8 2023-10-11T14:59:28.806 DEBUG:boto:Data: 2023-10-11T14:59:28.806 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.807 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.807 DEBUG:boto:Params: {} 2023-10-11T14:59:28.808 DEBUG:boto:Token: None 2023-10-11T14:59:28.808 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8 2023-10-11T14:59:28.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZlB9qJPZWVE/9+T+Fh0Y+DxwGnM= 2023-10-11T14:59:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZlB9qJPZWVE/9+T+Fh0Y+DxwGnM=', 'Content-Length': '0'} 2023-10-11T14:59:28.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068598e507db8080c-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:28.814 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.814 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.815 DEBUG:boto:Method: GET 2023-10-11T14:59:28.815 DEBUG:boto:Path: /dghqvd-46/obj8 2023-10-11T14:59:28.816 DEBUG:boto:Data: 2023-10-11T14:59:28.816 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.817 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.817 DEBUG:boto:Params: {} 2023-10-11T14:59:28.818 DEBUG:boto:Token: None 2023-10-11T14:59:28.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8 2023-10-11T14:59:28.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HTjlw2nRfZmCzzE15+IPsoNnr28= 2023-10-11T14:59:28.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HTjlw2nRfZmCzzE15+IPsoNnr28=', 'Content-Length': '0'} 2023-10-11T14:59:28.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-46:daa316e9-869c-4074-a756-a8adfb4a3573.5751.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de8c81b2baff8b82-006526b850-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.822 DEBUG:boto:path=/dghqvd-46/obj8 2023-10-11T14:59:28.822 DEBUG:boto:auth_path=/dghqvd-46/obj8 2023-10-11T14:59:28.823 DEBUG:boto:Method: GET 2023-10-11T14:59:28.823 DEBUG:boto:Path: /dghqvd-46/obj8 2023-10-11T14:59:28.824 DEBUG:boto:Data: 2023-10-11T14:59:28.824 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.824 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.825 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.825 DEBUG:boto:Params: {} 2023-10-11T14:59:28.826 DEBUG:boto:Token: None 2023-10-11T14:59:28.826 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/obj8 2023-10-11T14:59:28.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HTjlw2nRfZmCzzE15+IPsoNnr28= 2023-10-11T14:59:28.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HTjlw2nRfZmCzzE15+IPsoNnr28=', 'Content-Length': '0'} 2023-10-11T14:59:28.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c72335ea97e537-006526b850-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.870 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.871 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.871 DEBUG:boto:path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.872 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.872 DEBUG:boto:Method: GET 2023-10-11T14:59:28.873 DEBUG:boto:Path: /dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.873 DEBUG:boto:Data: 2023-10-11T14:59:28.873 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:28.874 DEBUG:boto:Port: 8001 2023-10-11T14:59:28.875 DEBUG:boto:Params: {} 2023-10-11T14:59:28.875 DEBUG:boto:Token: None 2023-10-11T14:59:28.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df53a8d4dd974222-006526b850-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.880 DEBUG:boto:b'dghqvd-46obj81000falseobj82023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.881 DEBUG:boto:path=/dghqvd-46/ 2023-10-11T14:59:28.881 DEBUG:boto:auth_path=/dghqvd-46/ 2023-10-11T14:59:28.881 DEBUG:boto:path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.882 DEBUG:boto:auth_path=/dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.882 DEBUG:boto:Method: GET 2023-10-11T14:59:28.883 DEBUG:boto:Path: /dghqvd-46/?prefix=obj8 2023-10-11T14:59:28.883 DEBUG:boto:Data: 2023-10-11T14:59:28.884 DEBUG:boto:Headers: {} 2023-10-11T14:59:28.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:28.884 DEBUG:boto:Port: 8000 2023-10-11T14:59:28.885 DEBUG:boto:Params: {} 2023-10-11T14:59:28.885 DEBUG:boto:Token: None 2023-10-11T14:59:28.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:28 GMT /dghqvd-46/ 2023-10-11T14:59:28.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0= 2023-10-11T14:59:28.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajekMi2DrXzMOTXcT5nHj1vdLB0=', 'Content-Length': '0'} 2023-10-11T14:59:28.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009233b7c1b9fdc596-006526b850-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:28.890 DEBUG:boto:b'dghqvd-46obj81000falseobj82023-10-11T14:58:51.874Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:28.891 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-46 zones={a2, a1} 2023-10-11T14:59:28.891 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-47 2023-10-11T14:59:28.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:28.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:28.893 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:59:28.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:28.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:28.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:28.986+0000 7fcf404c2780 1 bucket sync caught up with source: 2023-10-11T14:59:28.988 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.23.6, 00000000002.23.1, 00000000004.25.6, 00000000002.15.1, 00000000004.27.6, 00000000002.25.1, 00000000002.39.1, 00000000004.30.6, 00000000002.40.1, 00000000002.15.1, 00000000002.14.1] 2023-10-11T14:59:28.988 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.23.6, 00000000002.23.1, 00000000004.25.6, 00000000002.15.1, 00000000004.27.6, 00000000002.25.1, 00000000002.39.1, 00000000004.30.6, 00000000002.40.1, 00000000002.15.1, 00000000002.14.1] 2023-10-11T14:59:28.988 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:28.986+0000 7fcf404c2780 0 bucket checkpoint complete 2023-10-11T14:59:29.001 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-47 2023-10-11T14:59:29.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:29.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:29.002 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:59:29.058 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:29.058 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:29.090+0000 7f019a3c5780 1 bucket sync caught up with source: 2023-10-11T14:59:29.092 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.138.6, , 00000000002.136.6, , 00000000002.128.6, , , 00000000002.206.6, , , ] 2023-10-11T14:59:29.092 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.138.6, , 00000000002.136.6, , 00000000002.128.6, , , 00000000002.206.6, , , ] 2023-10-11T14:59:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:29.090+0000 7f019a3c5780 0 bucket checkpoint complete 2023-10-11T14:59:29.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-47 zones={a2, a1} 2023-10-11T14:59:29.105 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.105 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.106 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.106 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:59:29.106 DEBUG:boto:Data: 2023-10-11T14:59:29.107 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.108 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.108 DEBUG:boto:Params: {} 2023-10-11T14:59:29.109 DEBUG:boto:Token: None 2023-10-11T14:59:29.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOvSiN5XUxtzrbPIsgcuWxOdfz0= 2023-10-11T14:59:29.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOvSiN5XUxtzrbPIsgcuWxOdfz0=', 'Content-Length': '0'} 2023-10-11T14:59:29.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2e9667746e9dbd0-006526b851-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.113 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.113 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.114 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.114 DEBUG:boto:Path: /dghqvd-47/ 2023-10-11T14:59:29.115 DEBUG:boto:Data: 2023-10-11T14:59:29.115 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.116 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.116 DEBUG:boto:Params: {} 2023-10-11T14:59:29.117 DEBUG:boto:Token: None 2023-10-11T14:59:29.117 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOvSiN5XUxtzrbPIsgcuWxOdfz0= 2023-10-11T14:59:29.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOvSiN5XUxtzrbPIsgcuWxOdfz0=', 'Content-Length': '0'} 2023-10-11T14:59:29.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000867149475540d8f7-006526b851-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T14:59:29.122 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.122 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.123 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.123 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.123 DEBUG:boto:Method: GET 2023-10-11T14:59:29.124 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.124 DEBUG:boto:Data: 2023-10-11T14:59:29.125 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.125 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.126 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.126 DEBUG:boto:Params: {} 2023-10-11T14:59:29.126 DEBUG:boto:Token: None 2023-10-11T14:59:29.127 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/?versions 2023-10-11T14:59:29.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk= 2023-10-11T14:59:29.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk=', 'Content-Length': '0'} 2023-10-11T14:59:29.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065870236ef2a7948-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.131 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:59:29.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:59:29.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:59:29.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:59:29.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T14:59:29.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T14:59:29.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T14:59:29.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T14:59:29.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T14:59:29.135 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.136 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.136 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.136 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.137 DEBUG:boto:Method: GET 2023-10-11T14:59:29.137 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.138 DEBUG:boto:Data: 2023-10-11T14:59:29.138 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.139 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.139 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.139 DEBUG:boto:Params: {} 2023-10-11T14:59:29.140 DEBUG:boto:Token: None 2023-10-11T14:59:29.140 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/?versions 2023-10-11T14:59:29.141 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk= 2023-10-11T14:59:29.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk=', 'Content-Length': '0'} 2023-10-11T14:59:29.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db59e9dc6aace0b9-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.144 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T14:59:29.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T14:59:29.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T14:59:29.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T14:59:29.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T14:59:29.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T14:59:29.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T14:59:29.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T14:59:29.148 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.149 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.149 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.149 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.150 DEBUG:boto:Method: GET 2023-10-11T14:59:29.150 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.151 DEBUG:boto:Data: 2023-10-11T14:59:29.151 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.152 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.152 DEBUG:boto:Params: {} 2023-10-11T14:59:29.153 DEBUG:boto:Token: None 2023-10-11T14:59:29.153 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/?versions 2023-10-11T14:59:29.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk= 2023-10-11T14:59:29.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk=', 'Content-Length': '0'} 2023-10-11T14:59:29.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b494623ca12c2167-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.157 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.158 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.158 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.159 DEBUG:boto:path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.159 DEBUG:boto:auth_path=/dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.159 DEBUG:boto:Method: GET 2023-10-11T14:59:29.160 DEBUG:boto:Path: /dghqvd-47/?versions&max-keys=999 2023-10-11T14:59:29.160 DEBUG:boto:Data: 2023-10-11T14:59:29.161 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.161 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.161 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.162 DEBUG:boto:Params: {} 2023-10-11T14:59:29.162 DEBUG:boto:Token: None 2023-10-11T14:59:29.163 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/?versions 2023-10-11T14:59:29.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk= 2023-10-11T14:59:29.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bazr6aeorbLypSRJ0xjdBurtBgk=', 'Content-Length': '0'} 2023-10-11T14:59:29.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeba4b0fd646a9a2-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.167 DEBUG:boto:b'dghqvd-47999falseobj1nulltrue2023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:29.168 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.169 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.169 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.169 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.170 DEBUG:boto:Method: GET 2023-10-11T14:59:29.170 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.171 DEBUG:boto:Data: 2023-10-11T14:59:29.171 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.172 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.172 DEBUG:boto:Params: {} 2023-10-11T14:59:29.173 DEBUG:boto:Token: None 2023-10-11T14:59:29.173 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8= 2023-10-11T14:59:29.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8=', 'Content-Length': '0'} 2023-10-11T14:59:29.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088aeb04de6b9384a-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.222 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.223 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.223 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.224 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.224 DEBUG:boto:Method: GET 2023-10-11T14:59:29.225 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.225 DEBUG:boto:Data: 2023-10-11T14:59:29.226 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.227 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.227 DEBUG:boto:Params: {} 2023-10-11T14:59:29.227 DEBUG:boto:Token: None 2023-10-11T14:59:29.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8= 2023-10-11T14:59:29.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8=', 'Content-Length': '0'} 2023-10-11T14:59:29.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1568c226cbc9a81-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.232 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.232 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.232 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.233 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.233 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.234 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.234 DEBUG:boto:Data: 2023-10-11T14:59:29.235 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.235 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.236 DEBUG:boto:Params: {} 2023-10-11T14:59:29.236 DEBUG:boto:Token: None 2023-10-11T14:59:29.237 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F+UJZOMENOLIB+bktUKWKz2FtMU= 2023-10-11T14:59:29.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F+UJZOMENOLIB+bktUKWKz2FtMU=', 'Content-Length': '0'} 2023-10-11T14:59:29.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c1fdd340f683d96-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.240 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.240 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.240 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.241 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.241 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.242 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.242 DEBUG:boto:Data: 2023-10-11T14:59:29.242 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.243 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.244 DEBUG:boto:Params: {} 2023-10-11T14:59:29.244 DEBUG:boto:Token: None 2023-10-11T14:59:29.245 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F+UJZOMENOLIB+bktUKWKz2FtMU= 2023-10-11T14:59:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F+UJZOMENOLIB+bktUKWKz2FtMU=', 'Content-Length': '0'} 2023-10-11T14:59:29.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095fe7d3895a6813a-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:29.248 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.249 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.249 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.250 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.250 DEBUG:boto:Method: GET 2023-10-11T14:59:29.250 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.251 DEBUG:boto:Data: 2023-10-11T14:59:29.251 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.252 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.252 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.253 DEBUG:boto:Params: {} 2023-10-11T14:59:29.253 DEBUG:boto:Token: None 2023-10-11T14:59:29.253 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.254 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8= 2023-10-11T14:59:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8=', 'Content-Length': '0'} 2023-10-11T14:59:29.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0275a668ea55701-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.257 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.257 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.258 DEBUG:boto:path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.258 DEBUG:boto:auth_path=/dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.259 DEBUG:boto:Method: GET 2023-10-11T14:59:29.259 DEBUG:boto:Path: /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.259 DEBUG:boto:Data: 2023-10-11T14:59:29.260 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.260 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.261 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.261 DEBUG:boto:Params: {} 2023-10-11T14:59:29.262 DEBUG:boto:Token: None 2023-10-11T14:59:29.262 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1?versionId=null 2023-10-11T14:59:29.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8= 2023-10-11T14:59:29.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wn0KCNIi9wjErv6TlkEuDH2ijq8=', 'Content-Length': '0'} 2023-10-11T14:59:29.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cca0d933f5a7c76a-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.306 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.307 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.307 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.308 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.308 DEBUG:boto:Method: GET 2023-10-11T14:59:29.309 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.309 DEBUG:boto:Data: 2023-10-11T14:59:29.309 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.310 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.311 DEBUG:boto:Params: {} 2023-10-11T14:59:29.311 DEBUG:boto:Token: None 2023-10-11T14:59:29.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090cd906d8c562839-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.315 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.316 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.317 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.317 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.317 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.318 DEBUG:boto:Method: GET 2023-10-11T14:59:29.318 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.319 DEBUG:boto:Data: 2023-10-11T14:59:29.319 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.320 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.320 DEBUG:boto:Params: {} 2023-10-11T14:59:29.321 DEBUG:boto:Token: None 2023-10-11T14:59:29.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014cae96058142e5f-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.325 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.326 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.326 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.327 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.327 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:59:29.328 DEBUG:boto:Data: 2023-10-11T14:59:29.328 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.329 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.329 DEBUG:boto:Params: {} 2023-10-11T14:59:29.330 DEBUG:boto:Token: None 2023-10-11T14:59:29.330 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1 2023-10-11T14:59:29.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6wCpGizw4Cms8b/1E4ueFCy0Kuk= 2023-10-11T14:59:29.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6wCpGizw4Cms8b/1E4ueFCy0Kuk=', 'Content-Length': '0'} 2023-10-11T14:59:29.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8eb0d593bca79d5-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.333 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.334 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.334 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.335 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:59:29.335 DEBUG:boto:Data: 2023-10-11T14:59:29.335 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.336 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.337 DEBUG:boto:Params: {} 2023-10-11T14:59:29.337 DEBUG:boto:Token: None 2023-10-11T14:59:29.338 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1 2023-10-11T14:59:29.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6wCpGizw4Cms8b/1E4ueFCy0Kuk= 2023-10-11T14:59:29.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6wCpGizw4Cms8b/1E4ueFCy0Kuk=', 'Content-Length': '0'} 2023-10-11T14:59:29.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4eef3063371ac39-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T14:59:29.342 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.342 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.343 DEBUG:boto:Method: GET 2023-10-11T14:59:29.343 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:59:29.343 DEBUG:boto:Data: 2023-10-11T14:59:29.344 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.345 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.345 DEBUG:boto:Params: {} 2023-10-11T14:59:29.346 DEBUG:boto:Token: None 2023-10-11T14:59:29.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1 2023-10-11T14:59:29.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:66UOv9PeTjqeiHkY1NKAoQecCZs= 2023-10-11T14:59:29.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:66UOv9PeTjqeiHkY1NKAoQecCZs=', 'Content-Length': '0'} 2023-10-11T14:59:29.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027733c5e5f3438ee-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.350 DEBUG:boto:path=/dghqvd-47/obj1 2023-10-11T14:59:29.350 DEBUG:boto:auth_path=/dghqvd-47/obj1 2023-10-11T14:59:29.350 DEBUG:boto:Method: GET 2023-10-11T14:59:29.351 DEBUG:boto:Path: /dghqvd-47/obj1 2023-10-11T14:59:29.351 DEBUG:boto:Data: 2023-10-11T14:59:29.352 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.353 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.353 DEBUG:boto:Params: {} 2023-10-11T14:59:29.353 DEBUG:boto:Token: None 2023-10-11T14:59:29.354 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj1 2023-10-11T14:59:29.354 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:66UOv9PeTjqeiHkY1NKAoQecCZs= 2023-10-11T14:59:29.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:66UOv9PeTjqeiHkY1NKAoQecCZs=', 'Content-Length': '0'} 2023-10-11T14:59:29.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018ef300a9ab99dc2-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.357 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.358 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.358 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.359 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.359 DEBUG:boto:Method: GET 2023-10-11T14:59:29.360 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.360 DEBUG:boto:Data: 2023-10-11T14:59:29.361 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.361 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.362 DEBUG:boto:Params: {} 2023-10-11T14:59:29.362 DEBUG:boto:Token: None 2023-10-11T14:59:29.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be518662bb693ed-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.367 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.368 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.368 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.368 DEBUG:boto:path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.369 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.369 DEBUG:boto:Method: GET 2023-10-11T14:59:29.370 DEBUG:boto:Path: /dghqvd-47/?prefix=obj1 2023-10-11T14:59:29.370 DEBUG:boto:Data: 2023-10-11T14:59:29.371 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.371 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.372 DEBUG:boto:Params: {} 2023-10-11T14:59:29.372 DEBUG:boto:Token: None 2023-10-11T14:59:29.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7de527bea9ce8a-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.376 DEBUG:boto:b'dghqvd-47obj11000falseobj12023-10-11T14:55:48.972Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:29.377 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.378 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.378 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.379 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.379 DEBUG:boto:Method: GET 2023-10-11T14:59:29.379 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.380 DEBUG:boto:Data: 2023-10-11T14:59:29.380 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.381 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.382 DEBUG:boto:Params: {} 2023-10-11T14:59:29.382 DEBUG:boto:Token: None 2023-10-11T14:59:29.383 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8= 2023-10-11T14:59:29.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8=', 'Content-Length': '0'} 2023-10-11T14:59:29.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099ce9a9eaeda7845-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.426 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.427 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.427 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.428 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.428 DEBUG:boto:Method: GET 2023-10-11T14:59:29.429 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.429 DEBUG:boto:Data: 2023-10-11T14:59:29.429 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.430 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.431 DEBUG:boto:Params: {} 2023-10-11T14:59:29.431 DEBUG:boto:Token: None 2023-10-11T14:59:29.432 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8= 2023-10-11T14:59:29.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8=', 'Content-Length': '0'} 2023-10-11T14:59:29.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b16b20dd1b8fc8-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.436 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.437 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.437 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.437 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.438 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.438 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.439 DEBUG:boto:Data: 2023-10-11T14:59:29.439 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.440 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.440 DEBUG:boto:Params: {} 2023-10-11T14:59:29.441 DEBUG:boto:Token: None 2023-10-11T14:59:29.441 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bKPF2RQTSuG1xrIhoEJuMEQuOBE= 2023-10-11T14:59:29.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bKPF2RQTSuG1xrIhoEJuMEQuOBE=', 'Content-Length': '0'} 2023-10-11T14:59:29.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000424d1a799ed71e39-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.444 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.445 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.445 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.446 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.446 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.446 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.447 DEBUG:boto:Data: 2023-10-11T14:59:29.447 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.448 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.449 DEBUG:boto:Params: {} 2023-10-11T14:59:29.449 DEBUG:boto:Token: None 2023-10-11T14:59:29.450 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bKPF2RQTSuG1xrIhoEJuMEQuOBE= 2023-10-11T14:59:29.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bKPF2RQTSuG1xrIhoEJuMEQuOBE=', 'Content-Length': '0'} 2023-10-11T14:59:29.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004df307fc6557db0a-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:29.454 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.454 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.455 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.455 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.456 DEBUG:boto:Method: GET 2023-10-11T14:59:29.456 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.456 DEBUG:boto:Data: 2023-10-11T14:59:29.457 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.458 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.458 DEBUG:boto:Params: {} 2023-10-11T14:59:29.459 DEBUG:boto:Token: None 2023-10-11T14:59:29.459 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8= 2023-10-11T14:59:29.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8=', 'Content-Length': '0'} 2023-10-11T14:59:29.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ac1109ec750ea86-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.463 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.464 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.464 DEBUG:boto:path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.465 DEBUG:boto:auth_path=/dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.465 DEBUG:boto:Method: GET 2023-10-11T14:59:29.466 DEBUG:boto:Path: /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.466 DEBUG:boto:Data: 2023-10-11T14:59:29.467 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.467 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.468 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.468 DEBUG:boto:Params: {} 2023-10-11T14:59:29.469 DEBUG:boto:Token: None 2023-10-11T14:59:29.469 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2?versionId=null 2023-10-11T14:59:29.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8= 2023-10-11T14:59:29.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XwygpSV8odFxtOx/kHMvL2v8EZ8=', 'Content-Length': '0'} 2023-10-11T14:59:29.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024942453e169e373-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.473 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.473 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.474 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.474 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.474 DEBUG:boto:Method: GET 2023-10-11T14:59:29.475 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.475 DEBUG:boto:Data: 2023-10-11T14:59:29.476 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.476 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.477 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.477 DEBUG:boto:Params: {} 2023-10-11T14:59:29.478 DEBUG:boto:Token: None 2023-10-11T14:59:29.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c204237234aae9-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.482 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.483 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.483 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.483 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.484 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.484 DEBUG:boto:Method: GET 2023-10-11T14:59:29.485 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.485 DEBUG:boto:Data: 2023-10-11T14:59:29.485 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.486 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.487 DEBUG:boto:Params: {} 2023-10-11T14:59:29.487 DEBUG:boto:Token: None 2023-10-11T14:59:29.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e8a6459255b2f59-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.491 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.492 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.492 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.493 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.493 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:59:29.494 DEBUG:boto:Data: 2023-10-11T14:59:29.494 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.495 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.495 DEBUG:boto:Params: {} 2023-10-11T14:59:29.496 DEBUG:boto:Token: None 2023-10-11T14:59:29.496 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2 2023-10-11T14:59:29.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHQcD1NPjCRrOwSOIk9iyGch6ho= 2023-10-11T14:59:29.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHQcD1NPjCRrOwSOIk9iyGch6ho=', 'Content-Length': '0'} 2023-10-11T14:59:29.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e12fe9199c2526fd-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.499 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.500 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.500 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.500 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:59:29.501 DEBUG:boto:Data: 2023-10-11T14:59:29.501 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.502 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.502 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.503 DEBUG:boto:Params: {} 2023-10-11T14:59:29.503 DEBUG:boto:Token: None 2023-10-11T14:59:29.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2 2023-10-11T14:59:29.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHQcD1NPjCRrOwSOIk9iyGch6ho= 2023-10-11T14:59:29.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHQcD1NPjCRrOwSOIk9iyGch6ho=', 'Content-Length': '0'} 2023-10-11T14:59:29.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cc1bd62d3eab65a-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T14:59:29.507 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.507 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.508 DEBUG:boto:Method: GET 2023-10-11T14:59:29.508 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:59:29.509 DEBUG:boto:Data: 2023-10-11T14:59:29.509 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.510 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.510 DEBUG:boto:Params: {} 2023-10-11T14:59:29.511 DEBUG:boto:Token: None 2023-10-11T14:59:29.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2 2023-10-11T14:59:29.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S6dcDowBAco7SjGYxCPeJsZ+yqU= 2023-10-11T14:59:29.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S6dcDowBAco7SjGYxCPeJsZ+yqU=', 'Content-Length': '0'} 2023-10-11T14:59:29.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006589de78e6b3a8b9-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.558 DEBUG:boto:path=/dghqvd-47/obj2 2023-10-11T14:59:29.559 DEBUG:boto:auth_path=/dghqvd-47/obj2 2023-10-11T14:59:29.559 DEBUG:boto:Method: GET 2023-10-11T14:59:29.560 DEBUG:boto:Path: /dghqvd-47/obj2 2023-10-11T14:59:29.560 DEBUG:boto:Data: 2023-10-11T14:59:29.561 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.561 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.562 DEBUG:boto:Params: {} 2023-10-11T14:59:29.562 DEBUG:boto:Token: None 2023-10-11T14:59:29.563 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj2 2023-10-11T14:59:29.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S6dcDowBAco7SjGYxCPeJsZ+yqU= 2023-10-11T14:59:29.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S6dcDowBAco7SjGYxCPeJsZ+yqU=', 'Content-Length': '0'} 2023-10-11T14:59:29.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f3871b4a882480f-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.567 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.567 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.568 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.568 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.568 DEBUG:boto:Method: GET 2023-10-11T14:59:29.569 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.569 DEBUG:boto:Data: 2023-10-11T14:59:29.570 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.571 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.571 DEBUG:boto:Params: {} 2023-10-11T14:59:29.571 DEBUG:boto:Token: None 2023-10-11T14:59:29.572 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004704fd7a152802b5-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.576 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.576 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.577 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.577 DEBUG:boto:path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.578 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.578 DEBUG:boto:Method: GET 2023-10-11T14:59:29.579 DEBUG:boto:Path: /dghqvd-47/?prefix=obj2 2023-10-11T14:59:29.579 DEBUG:boto:Data: 2023-10-11T14:59:29.579 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.580 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.580 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.581 DEBUG:boto:Params: {} 2023-10-11T14:59:29.581 DEBUG:boto:Token: None 2023-10-11T14:59:29.582 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798a94928c17aec2-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.585 DEBUG:boto:b'dghqvd-47obj21000falseobj22023-10-11T14:55:48.992Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:29.586 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.587 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.587 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.588 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.588 DEBUG:boto:Method: GET 2023-10-11T14:59:29.588 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.589 DEBUG:boto:Data: 2023-10-11T14:59:29.589 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.590 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.591 DEBUG:boto:Params: {} 2023-10-11T14:59:29.591 DEBUG:boto:Token: None 2023-10-11T14:59:29.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M= 2023-10-11T14:59:29.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M=', 'Content-Length': '0'} 2023-10-11T14:59:29.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e2f0d53cd6d3d7c-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.595 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.595 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.596 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.596 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.597 DEBUG:boto:Method: GET 2023-10-11T14:59:29.597 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.598 DEBUG:boto:Data: 2023-10-11T14:59:29.598 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.599 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.599 DEBUG:boto:Params: {} 2023-10-11T14:59:29.600 DEBUG:boto:Token: None 2023-10-11T14:59:29.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M= 2023-10-11T14:59:29.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M=', 'Content-Length': '0'} 2023-10-11T14:59:29.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000147c2e6e77a84e1a-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.605 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.605 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.605 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.606 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.606 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.607 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.607 DEBUG:boto:Data: 2023-10-11T14:59:29.608 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.608 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.608 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.609 DEBUG:boto:Params: {} 2023-10-11T14:59:29.609 DEBUG:boto:Token: None 2023-10-11T14:59:29.610 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sj8/MzInnBJoLznCkZeqkWCfztY= 2023-10-11T14:59:29.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sj8/MzInnBJoLznCkZeqkWCfztY=', 'Content-Length': '0'} 2023-10-11T14:59:29.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000029a144498e92d9e-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.613 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.613 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.613 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.614 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.614 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.615 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.615 DEBUG:boto:Data: 2023-10-11T14:59:29.616 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.616 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.617 DEBUG:boto:Params: {} 2023-10-11T14:59:29.617 DEBUG:boto:Token: None 2023-10-11T14:59:29.618 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sj8/MzInnBJoLznCkZeqkWCfztY= 2023-10-11T14:59:29.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sj8/MzInnBJoLznCkZeqkWCfztY=', 'Content-Length': '0'} 2023-10-11T14:59:29.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cc458f1aa45990d-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:29.621 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.622 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.622 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.625 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.626 DEBUG:boto:Method: GET 2023-10-11T14:59:29.626 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.626 DEBUG:boto:Data: 2023-10-11T14:59:29.627 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.628 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.628 DEBUG:boto:Params: {} 2023-10-11T14:59:29.629 DEBUG:boto:Token: None 2023-10-11T14:59:29.629 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M= 2023-10-11T14:59:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M=', 'Content-Length': '0'} 2023-10-11T14:59:29.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc9a0b7d186bfd8c-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.674 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.675 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.675 DEBUG:boto:path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.676 DEBUG:boto:auth_path=/dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.676 DEBUG:boto:Method: GET 2023-10-11T14:59:29.677 DEBUG:boto:Path: /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.677 DEBUG:boto:Data: 2023-10-11T14:59:29.678 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.678 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.679 DEBUG:boto:Params: {} 2023-10-11T14:59:29.679 DEBUG:boto:Token: None 2023-10-11T14:59:29.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3?versionId=null 2023-10-11T14:59:29.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M= 2023-10-11T14:59:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jfCEn3C54OVTT3uorSzMeT57o5M=', 'Content-Length': '0'} 2023-10-11T14:59:29.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2b1046dfc72fa66-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.684 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.684 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.685 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.685 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.686 DEBUG:boto:Method: GET 2023-10-11T14:59:29.686 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.686 DEBUG:boto:Data: 2023-10-11T14:59:29.687 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.688 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.688 DEBUG:boto:Params: {} 2023-10-11T14:59:29.689 DEBUG:boto:Token: None 2023-10-11T14:59:29.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003798d60e47b80804-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.693 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.694 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.694 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.694 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.695 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.695 DEBUG:boto:Method: GET 2023-10-11T14:59:29.696 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.696 DEBUG:boto:Data: 2023-10-11T14:59:29.697 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.697 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.698 DEBUG:boto:Params: {} 2023-10-11T14:59:29.698 DEBUG:boto:Token: None 2023-10-11T14:59:29.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df84afaf824e06a-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.703 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.703 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.704 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.704 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.704 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:59:29.705 DEBUG:boto:Data: 2023-10-11T14:59:29.705 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.706 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.707 DEBUG:boto:Params: {} 2023-10-11T14:59:29.707 DEBUG:boto:Token: None 2023-10-11T14:59:29.708 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3 2023-10-11T14:59:29.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XUqsUOo7a+Ik4lZljz55m9EDeiU= 2023-10-11T14:59:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XUqsUOo7a+Ik4lZljz55m9EDeiU=', 'Content-Length': '0'} 2023-10-11T14:59:29.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a063a4eeda366a25-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.711 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.711 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.711 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.712 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:59:29.712 DEBUG:boto:Data: 2023-10-11T14:59:29.713 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.714 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.714 DEBUG:boto:Params: {} 2023-10-11T14:59:29.715 DEBUG:boto:Token: None 2023-10-11T14:59:29.715 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3 2023-10-11T14:59:29.715 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XUqsUOo7a+Ik4lZljz55m9EDeiU= 2023-10-11T14:59:29.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XUqsUOo7a+Ik4lZljz55m9EDeiU=', 'Content-Length': '0'} 2023-10-11T14:59:29.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000187d8fe554ee639c-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T14:59:29.719 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.719 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.719 DEBUG:boto:Method: GET 2023-10-11T14:59:29.720 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:59:29.720 DEBUG:boto:Data: 2023-10-11T14:59:29.721 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.722 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.722 DEBUG:boto:Params: {} 2023-10-11T14:59:29.723 DEBUG:boto:Token: None 2023-10-11T14:59:29.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3 2023-10-11T14:59:29.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UB0+XtUPsPg0VfnPtXDZXFWS6u8= 2023-10-11T14:59:29.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UB0+XtUPsPg0VfnPtXDZXFWS6u8=', 'Content-Length': '0'} 2023-10-11T14:59:29.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf0d78fd1ed182f0-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.727 DEBUG:boto:path=/dghqvd-47/obj3 2023-10-11T14:59:29.727 DEBUG:boto:auth_path=/dghqvd-47/obj3 2023-10-11T14:59:29.727 DEBUG:boto:Method: GET 2023-10-11T14:59:29.728 DEBUG:boto:Path: /dghqvd-47/obj3 2023-10-11T14:59:29.728 DEBUG:boto:Data: 2023-10-11T14:59:29.729 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.730 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.730 DEBUG:boto:Params: {} 2023-10-11T14:59:29.731 DEBUG:boto:Token: None 2023-10-11T14:59:29.731 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj3 2023-10-11T14:59:29.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UB0+XtUPsPg0VfnPtXDZXFWS6u8= 2023-10-11T14:59:29.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UB0+XtUPsPg0VfnPtXDZXFWS6u8=', 'Content-Length': '0'} 2023-10-11T14:59:29.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e4052c56920e2d-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.735 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.736 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.736 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.737 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.737 DEBUG:boto:Method: GET 2023-10-11T14:59:29.738 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.738 DEBUG:boto:Data: 2023-10-11T14:59:29.739 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.740 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.740 DEBUG:boto:Params: {} 2023-10-11T14:59:29.741 DEBUG:boto:Token: None 2023-10-11T14:59:29.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8e52f3eb978681-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.745 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.746 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.746 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.747 DEBUG:boto:path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.747 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.747 DEBUG:boto:Method: GET 2023-10-11T14:59:29.748 DEBUG:boto:Path: /dghqvd-47/?prefix=obj3 2023-10-11T14:59:29.748 DEBUG:boto:Data: 2023-10-11T14:59:29.749 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.750 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.750 DEBUG:boto:Params: {} 2023-10-11T14:59:29.751 DEBUG:boto:Token: None 2023-10-11T14:59:29.751 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c823eab3bb7f102a-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.794 DEBUG:boto:b'dghqvd-47obj31000falseobj32023-10-11T14:55:49.011Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:29.795 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.796 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.796 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.797 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.797 DEBUG:boto:Method: GET 2023-10-11T14:59:29.797 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.798 DEBUG:boto:Data: 2023-10-11T14:59:29.798 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.799 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.800 DEBUG:boto:Params: {} 2023-10-11T14:59:29.800 DEBUG:boto:Token: None 2023-10-11T14:59:29.801 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo= 2023-10-11T14:59:29.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo=', 'Content-Length': '0'} 2023-10-11T14:59:29.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a10fc4bfa05976b-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.805 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.806 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.806 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.807 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.807 DEBUG:boto:Method: GET 2023-10-11T14:59:29.808 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.808 DEBUG:boto:Data: 2023-10-11T14:59:29.809 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.809 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.810 DEBUG:boto:Params: {} 2023-10-11T14:59:29.810 DEBUG:boto:Token: None 2023-10-11T14:59:29.811 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo= 2023-10-11T14:59:29.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo=', 'Content-Length': '0'} 2023-10-11T14:59:29.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c27f616867adb80d-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.814 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.815 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.815 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.815 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.816 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.816 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.817 DEBUG:boto:Data: 2023-10-11T14:59:29.817 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.818 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.818 DEBUG:boto:Params: {} 2023-10-11T14:59:29.819 DEBUG:boto:Token: None 2023-10-11T14:59:29.819 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r/A1WEkuCofR2zpArwTW5zKAcbQ= 2023-10-11T14:59:29.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r/A1WEkuCofR2zpArwTW5zKAcbQ=', 'Content-Length': '0'} 2023-10-11T14:59:29.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d9a1953a98210c4-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.823 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.823 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.823 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.824 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.824 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.825 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.825 DEBUG:boto:Data: 2023-10-11T14:59:29.825 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.826 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.827 DEBUG:boto:Params: {} 2023-10-11T14:59:29.827 DEBUG:boto:Token: None 2023-10-11T14:59:29.828 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r/A1WEkuCofR2zpArwTW5zKAcbQ= 2023-10-11T14:59:29.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r/A1WEkuCofR2zpArwTW5zKAcbQ=', 'Content-Length': '0'} 2023-10-11T14:59:29.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066facb391a7e9b59-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:29.831 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.833 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.833 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.834 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.834 DEBUG:boto:Method: GET 2023-10-11T14:59:29.835 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.835 DEBUG:boto:Data: 2023-10-11T14:59:29.835 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.837 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.837 DEBUG:boto:Params: {} 2023-10-11T14:59:29.838 DEBUG:boto:Token: None 2023-10-11T14:59:29.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo= 2023-10-11T14:59:29.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo=', 'Content-Length': '0'} 2023-10-11T14:59:29.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cfde1952bd2c30b-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.841 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.842 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.842 DEBUG:boto:path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.843 DEBUG:boto:auth_path=/dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.843 DEBUG:boto:Method: GET 2023-10-11T14:59:29.844 DEBUG:boto:Path: /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.844 DEBUG:boto:Data: 2023-10-11T14:59:29.845 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.845 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.846 DEBUG:boto:Params: {} 2023-10-11T14:59:29.846 DEBUG:boto:Token: None 2023-10-11T14:59:29.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4?versionId=null 2023-10-11T14:59:29.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo= 2023-10-11T14:59:29.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p1wWDU7/a93CyuyG80aPjoRRzeo=', 'Content-Length': '0'} 2023-10-11T14:59:29.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005be18557766db6a2-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.851 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.851 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.852 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.852 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.852 DEBUG:boto:Method: GET 2023-10-11T14:59:29.853 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.853 DEBUG:boto:Data: 2023-10-11T14:59:29.854 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.855 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.855 DEBUG:boto:Params: {} 2023-10-11T14:59:29.856 DEBUG:boto:Token: None 2023-10-11T14:59:29.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003394594752921d5f-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.860 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.861 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.861 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.862 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.862 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.863 DEBUG:boto:Method: GET 2023-10-11T14:59:29.863 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.864 DEBUG:boto:Data: 2023-10-11T14:59:29.864 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.865 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.865 DEBUG:boto:Params: {} 2023-10-11T14:59:29.866 DEBUG:boto:Token: None 2023-10-11T14:59:29.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec4ce4ff276c733a-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.870 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.871 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.871 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.872 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.872 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:59:29.873 DEBUG:boto:Data: 2023-10-11T14:59:29.873 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.874 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.874 DEBUG:boto:Params: {} 2023-10-11T14:59:29.875 DEBUG:boto:Token: None 2023-10-11T14:59:29.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4 2023-10-11T14:59:29.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/fZaGMZoB0fea6fdgfk+gcyoZo4= 2023-10-11T14:59:29.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/fZaGMZoB0fea6fdgfk+gcyoZo4=', 'Content-Length': '0'} 2023-10-11T14:59:29.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bdeeee022af533a-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.879 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.879 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.879 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.880 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:59:29.880 DEBUG:boto:Data: 2023-10-11T14:59:29.881 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.881 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.882 DEBUG:boto:Params: {} 2023-10-11T14:59:29.882 DEBUG:boto:Token: None 2023-10-11T14:59:29.883 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4 2023-10-11T14:59:29.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/fZaGMZoB0fea6fdgfk+gcyoZo4= 2023-10-11T14:59:29.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/fZaGMZoB0fea6fdgfk+gcyoZo4=', 'Content-Length': '0'} 2023-10-11T14:59:29.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000757715c86493a630-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T14:59:29.886 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.887 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.887 DEBUG:boto:Method: GET 2023-10-11T14:59:29.888 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:59:29.888 DEBUG:boto:Data: 2023-10-11T14:59:29.889 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.889 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.890 DEBUG:boto:Params: {} 2023-10-11T14:59:29.890 DEBUG:boto:Token: None 2023-10-11T14:59:29.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4 2023-10-11T14:59:29.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tdTMqSJUyzyHdZIrfQ+1JzX+l+M= 2023-10-11T14:59:29.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tdTMqSJUyzyHdZIrfQ+1JzX+l+M=', 'Content-Length': '0'} 2023-10-11T14:59:29.893 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65540b114c2b280-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.894 DEBUG:boto:path=/dghqvd-47/obj4 2023-10-11T14:59:29.895 DEBUG:boto:auth_path=/dghqvd-47/obj4 2023-10-11T14:59:29.895 DEBUG:boto:Method: GET 2023-10-11T14:59:29.895 DEBUG:boto:Path: /dghqvd-47/obj4 2023-10-11T14:59:29.896 DEBUG:boto:Data: 2023-10-11T14:59:29.896 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.897 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.898 DEBUG:boto:Params: {} 2023-10-11T14:59:29.898 DEBUG:boto:Token: None 2023-10-11T14:59:29.899 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj4 2023-10-11T14:59:29.899 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tdTMqSJUyzyHdZIrfQ+1JzX+l+M= 2023-10-11T14:59:29.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tdTMqSJUyzyHdZIrfQ+1JzX+l+M=', 'Content-Length': '0'} 2023-10-11T14:59:29.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:55:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082a2dc1999b6fd69-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.902 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.902 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.903 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.903 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.904 DEBUG:boto:Method: GET 2023-10-11T14:59:29.904 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.905 DEBUG:boto:Data: 2023-10-11T14:59:29.905 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.905 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.906 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.906 DEBUG:boto:Params: {} 2023-10-11T14:59:29.907 DEBUG:boto:Token: None 2023-10-11T14:59:29.907 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92c17ff570a1d9b-006526b851-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.911 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.912 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:29.912 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:29.913 DEBUG:boto:path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.913 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.913 DEBUG:boto:Method: GET 2023-10-11T14:59:29.914 DEBUG:boto:Path: /dghqvd-47/?prefix=obj4 2023-10-11T14:59:29.914 DEBUG:boto:Data: 2023-10-11T14:59:29.915 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.915 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.916 DEBUG:boto:Params: {} 2023-10-11T14:59:29.916 DEBUG:boto:Token: None 2023-10-11T14:59:29.917 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/ 2023-10-11T14:59:29.917 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y= 2023-10-11T14:59:29.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ck5oNIjFtHIRp54R2vYUoy2UF6Y=', 'Content-Length': '0'} 2023-10-11T14:59:29.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7bb2943e260774-006526b851-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.920 DEBUG:boto:b'dghqvd-47obj41000falseobj42023-10-11T14:55:49.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:29.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:29.921 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:29.922 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:29.922 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.923 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.923 DEBUG:boto:Method: GET 2023-10-11T14:59:29.923 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.924 DEBUG:boto:Data: 2023-10-11T14:59:29.924 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.925 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.926 DEBUG:boto:Params: {} 2023-10-11T14:59:29.926 DEBUG:boto:Token: None 2023-10-11T14:59:29.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yij0dB7wT2pTJ87MfNQtBP3UaXk= 2023-10-11T14:59:29.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yij0dB7wT2pTJ87MfNQtBP3UaXk=', 'Content-Length': '0'} 2023-10-11T14:59:29.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000816fb233927c2db2-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.974 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:29.975 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:29.975 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.976 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.976 DEBUG:boto:Method: GET 2023-10-11T14:59:29.977 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.977 DEBUG:boto:Data: 2023-10-11T14:59:29.977 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.978 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.979 DEBUG:boto:Params: {} 2023-10-11T14:59:29.979 DEBUG:boto:Token: None 2023-10-11T14:59:29.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yij0dB7wT2pTJ87MfNQtBP3UaXk= 2023-10-11T14:59:29.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yij0dB7wT2pTJ87MfNQtBP3UaXk=', 'Content-Length': '0'} 2023-10-11T14:59:29.982 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9528b092f058d10-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.983 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:29.984 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:29.984 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.984 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.985 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.985 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.986 DEBUG:boto:Data: 2023-10-11T14:59:29.986 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:29.987 DEBUG:boto:Port: 8001 2023-10-11T14:59:29.987 DEBUG:boto:Params: {} 2023-10-11T14:59:29.988 DEBUG:boto:Token: None 2023-10-11T14:59:29.988 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gasb8ZW7NxtlleIzEOJO6rWjAjw= 2023-10-11T14:59:29.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gasb8ZW7NxtlleIzEOJO6rWjAjw=', 'Content-Length': '0'} 2023-10-11T14:59:29.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006261ed808cd915cd-006526b851-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:29.991 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:29.992 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:29.992 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.993 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.993 DEBUG:boto:Method: HEAD 2023-10-11T14:59:29.993 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.994 DEBUG:boto:Data: 2023-10-11T14:59:29.994 DEBUG:boto:Headers: {} 2023-10-11T14:59:29.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:29.995 DEBUG:boto:Port: 8000 2023-10-11T14:59:29.995 DEBUG:boto:Params: {} 2023-10-11T14:59:29.996 DEBUG:boto:Token: None 2023-10-11T14:59:29.996 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:29 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:29.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gasb8ZW7NxtlleIzEOJO6rWjAjw= 2023-10-11T14:59:29.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gasb8ZW7NxtlleIzEOJO6rWjAjw=', 'Content-Length': '0'} 2023-10-11T14:59:29.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005632ea7ef8b824b9-006526b851-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:30.000 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.001 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.001 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.001 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.002 DEBUG:boto:Method: GET 2023-10-11T14:59:30.002 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.003 DEBUG:boto:Data: 2023-10-11T14:59:30.003 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.004 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.004 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.004 DEBUG:boto:Params: {} 2023-10-11T14:59:30.005 DEBUG:boto:Token: None 2023-10-11T14:59:30.005 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.006 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miFOyeU4/PeFSib/ALTFvU7TJEU= 2023-10-11T14:59:30.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miFOyeU4/PeFSib/ALTFvU7TJEU=', 'Content-Length': '0'} 2023-10-11T14:59:30.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f731a879dd94f0f5-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.009 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.009 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.009 DEBUG:boto:path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.010 DEBUG:boto:auth_path=/dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.010 DEBUG:boto:Method: GET 2023-10-11T14:59:30.011 DEBUG:boto:Path: /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.011 DEBUG:boto:Data: 2023-10-11T14:59:30.012 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.012 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.013 DEBUG:boto:Params: {} 2023-10-11T14:59:30.013 DEBUG:boto:Token: None 2023-10-11T14:59:30.014 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5?versionId=null 2023-10-11T14:59:30.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miFOyeU4/PeFSib/ALTFvU7TJEU= 2023-10-11T14:59:30.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miFOyeU4/PeFSib/ALTFvU7TJEU=', 'Content-Length': '0'} 2023-10-11T14:59:30.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005acd454e49ce97ed-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.062 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.063 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.063 DEBUG:boto:path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.064 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.064 DEBUG:boto:Method: GET 2023-10-11T14:59:30.065 DEBUG:boto:Path: /dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.065 DEBUG:boto:Data: 2023-10-11T14:59:30.066 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.066 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.067 DEBUG:boto:Params: {} 2023-10-11T14:59:30.067 DEBUG:boto:Token: None 2023-10-11T14:59:30.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336266cdb50dd98d-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.072 DEBUG:boto:b'dghqvd-47obj51000falseobj52023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.073 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.073 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.073 DEBUG:boto:path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.074 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.074 DEBUG:boto:Method: GET 2023-10-11T14:59:30.075 DEBUG:boto:Path: /dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.075 DEBUG:boto:Data: 2023-10-11T14:59:30.076 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.077 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.077 DEBUG:boto:Params: {} 2023-10-11T14:59:30.077 DEBUG:boto:Token: None 2023-10-11T14:59:30.078 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006ae3693144947fe-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.082 DEBUG:boto:b'dghqvd-47obj51000falseobj52023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.083 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.083 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.083 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.084 DEBUG:boto:Path: /dghqvd-47/obj5 2023-10-11T14:59:30.084 DEBUG:boto:Data: 2023-10-11T14:59:30.085 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.086 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.086 DEBUG:boto:Params: {} 2023-10-11T14:59:30.087 DEBUG:boto:Token: None 2023-10-11T14:59:30.087 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5 2023-10-11T14:59:30.088 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkfCcTglap4/5Eex7CpO3LGbF1I= 2023-10-11T14:59:30.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkfCcTglap4/5Eex7CpO3LGbF1I=', 'Content-Length': '0'} 2023-10-11T14:59:30.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029e4e97c3fa6a01c-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.090 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.091 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.091 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.092 DEBUG:boto:Path: /dghqvd-47/obj5 2023-10-11T14:59:30.092 DEBUG:boto:Data: 2023-10-11T14:59:30.093 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.093 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.094 DEBUG:boto:Params: {} 2023-10-11T14:59:30.094 DEBUG:boto:Token: None 2023-10-11T14:59:30.095 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5 2023-10-11T14:59:30.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NkfCcTglap4/5Eex7CpO3LGbF1I= 2023-10-11T14:59:30.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NkfCcTglap4/5Eex7CpO3LGbF1I=', 'Content-Length': '0'} 2023-10-11T14:59:30.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5048a40e612a4c7-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T14:59:30.098 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.099 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.099 DEBUG:boto:Method: GET 2023-10-11T14:59:30.099 DEBUG:boto:Path: /dghqvd-47/obj5 2023-10-11T14:59:30.100 DEBUG:boto:Data: 2023-10-11T14:59:30.100 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.101 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.102 DEBUG:boto:Params: {} 2023-10-11T14:59:30.102 DEBUG:boto:Token: None 2023-10-11T14:59:30.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5 2023-10-11T14:59:30.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFw62M11pPaW6DTgG/4/YN4zy4I= 2023-10-11T14:59:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFw62M11pPaW6DTgG/4/YN4zy4I=', 'Content-Length': '0'} 2023-10-11T14:59:30.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfc729d8f9ff8957-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.146 DEBUG:boto:path=/dghqvd-47/obj5 2023-10-11T14:59:30.147 DEBUG:boto:auth_path=/dghqvd-47/obj5 2023-10-11T14:59:30.147 DEBUG:boto:Method: GET 2023-10-11T14:59:30.148 DEBUG:boto:Path: /dghqvd-47/obj5 2023-10-11T14:59:30.148 DEBUG:boto:Data: 2023-10-11T14:59:30.149 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.149 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.150 DEBUG:boto:Params: {} 2023-10-11T14:59:30.150 DEBUG:boto:Token: None 2023-10-11T14:59:30.151 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj5 2023-10-11T14:59:30.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFw62M11pPaW6DTgG/4/YN4zy4I= 2023-10-11T14:59:30.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFw62M11pPaW6DTgG/4/YN4zy4I=', 'Content-Length': '0'} 2023-10-11T14:59:30.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054cac9b147d57f61-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.155 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.155 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.155 DEBUG:boto:path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.156 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.157 DEBUG:boto:Method: GET 2023-10-11T14:59:30.157 DEBUG:boto:Path: /dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.157 DEBUG:boto:Data: 2023-10-11T14:59:30.158 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.159 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.159 DEBUG:boto:Params: {} 2023-10-11T14:59:30.160 DEBUG:boto:Token: None 2023-10-11T14:59:30.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e486c7a487229f98-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.164 DEBUG:boto:b'dghqvd-47obj51000falseobj52023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.165 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.165 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.165 DEBUG:boto:path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.166 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.166 DEBUG:boto:Method: GET 2023-10-11T14:59:30.167 DEBUG:boto:Path: /dghqvd-47/?prefix=obj5 2023-10-11T14:59:30.167 DEBUG:boto:Data: 2023-10-11T14:59:30.168 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.169 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.169 DEBUG:boto:Params: {} 2023-10-11T14:59:30.170 DEBUG:boto:Token: None 2023-10-11T14:59:30.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73875b97a9350f5-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.174 DEBUG:boto:b'dghqvd-47obj51000falseobj52023-10-11T14:58:51.932Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:30.175 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.176 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.176 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.177 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.177 DEBUG:boto:Method: GET 2023-10-11T14:59:30.177 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.178 DEBUG:boto:Data: 2023-10-11T14:59:30.178 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.179 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.179 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.180 DEBUG:boto:Params: {} 2023-10-11T14:59:30.180 DEBUG:boto:Token: None 2023-10-11T14:59:30.181 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg= 2023-10-11T14:59:30.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg=', 'Content-Length': '0'} 2023-10-11T14:59:30.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc72f59fccefb555-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.184 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.185 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.185 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.185 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.186 DEBUG:boto:Method: GET 2023-10-11T14:59:30.186 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.187 DEBUG:boto:Data: 2023-10-11T14:59:30.187 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.188 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.188 DEBUG:boto:Params: {} 2023-10-11T14:59:30.189 DEBUG:boto:Token: None 2023-10-11T14:59:30.189 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.190 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg= 2023-10-11T14:59:30.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg=', 'Content-Length': '0'} 2023-10-11T14:59:30.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a989f5f71ac1397d-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.193 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.193 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.194 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.194 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.195 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.195 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.195 DEBUG:boto:Data: 2023-10-11T14:59:30.196 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.196 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.197 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.197 DEBUG:boto:Params: {} 2023-10-11T14:59:30.197 DEBUG:boto:Token: None 2023-10-11T14:59:30.198 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.198 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5MGoiPhzZ4bT2AJ4v3Sg/XRUQmQ= 2023-10-11T14:59:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5MGoiPhzZ4bT2AJ4v3Sg/XRUQmQ=', 'Content-Length': '0'} 2023-10-11T14:59:30.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5338e3ba2b34cef-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.201 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.202 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.202 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.202 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.203 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.203 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.204 DEBUG:boto:Data: 2023-10-11T14:59:30.204 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.205 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.205 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.205 DEBUG:boto:Params: {} 2023-10-11T14:59:30.206 DEBUG:boto:Token: None 2023-10-11T14:59:30.206 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5MGoiPhzZ4bT2AJ4v3Sg/XRUQmQ= 2023-10-11T14:59:30.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5MGoiPhzZ4bT2AJ4v3Sg/XRUQmQ=', 'Content-Length': '0'} 2023-10-11T14:59:30.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ac969d4cf830ef-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:30.210 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.211 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.211 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.211 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.212 DEBUG:boto:Method: GET 2023-10-11T14:59:30.212 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.213 DEBUG:boto:Data: 2023-10-11T14:59:30.213 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.214 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.214 DEBUG:boto:Params: {} 2023-10-11T14:59:30.215 DEBUG:boto:Token: None 2023-10-11T14:59:30.215 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg= 2023-10-11T14:59:30.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg=', 'Content-Length': '0'} 2023-10-11T14:59:30.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059fe052b801423fa-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.219 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.219 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.220 DEBUG:boto:path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.220 DEBUG:boto:auth_path=/dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.220 DEBUG:boto:Method: GET 2023-10-11T14:59:30.221 DEBUG:boto:Path: /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.221 DEBUG:boto:Data: 2023-10-11T14:59:30.222 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.222 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.223 DEBUG:boto:Params: {} 2023-10-11T14:59:30.223 DEBUG:boto:Token: None 2023-10-11T14:59:30.224 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6?versionId=null 2023-10-11T14:59:30.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg= 2023-10-11T14:59:30.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tAZjc04XM355xh+CFX3yr2xvCpg=', 'Content-Length': '0'} 2023-10-11T14:59:30.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f977d4ea6ea07e4-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.227 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.228 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.228 DEBUG:boto:path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.229 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.229 DEBUG:boto:Method: GET 2023-10-11T14:59:30.230 DEBUG:boto:Path: /dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.230 DEBUG:boto:Data: 2023-10-11T14:59:30.230 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.231 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.232 DEBUG:boto:Params: {} 2023-10-11T14:59:30.232 DEBUG:boto:Token: None 2023-10-11T14:59:30.233 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001925720d6e303bd8-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.282 DEBUG:boto:b'dghqvd-47obj61000falseobj62023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.283 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.284 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.285 DEBUG:boto:path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.285 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.286 DEBUG:boto:Method: GET 2023-10-11T14:59:30.286 DEBUG:boto:Path: /dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.287 DEBUG:boto:Data: 2023-10-11T14:59:30.287 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.287 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.288 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.288 DEBUG:boto:Params: {} 2023-10-11T14:59:30.289 DEBUG:boto:Token: None 2023-10-11T14:59:30.289 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.290 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a776b23425a8cbf-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.293 DEBUG:boto:b'dghqvd-47obj61000falseobj62023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.294 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.294 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.295 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.295 DEBUG:boto:Path: /dghqvd-47/obj6 2023-10-11T14:59:30.296 DEBUG:boto:Data: 2023-10-11T14:59:30.296 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.297 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.297 DEBUG:boto:Params: {} 2023-10-11T14:59:30.298 DEBUG:boto:Token: None 2023-10-11T14:59:30.298 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6 2023-10-11T14:59:30.299 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:670HaDccDetOW9eeIWk1e1klI/w= 2023-10-11T14:59:30.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:670HaDccDetOW9eeIWk1e1klI/w=', 'Content-Length': '0'} 2023-10-11T14:59:30.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f381b997c59519b-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.302 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.302 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.303 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.303 DEBUG:boto:Path: /dghqvd-47/obj6 2023-10-11T14:59:30.304 DEBUG:boto:Data: 2023-10-11T14:59:30.305 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.305 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.306 DEBUG:boto:Params: {} 2023-10-11T14:59:30.306 DEBUG:boto:Token: None 2023-10-11T14:59:30.307 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6 2023-10-11T14:59:30.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:670HaDccDetOW9eeIWk1e1klI/w= 2023-10-11T14:59:30.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:670HaDccDetOW9eeIWk1e1klI/w=', 'Content-Length': '0'} 2023-10-11T14:59:30.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ed9cbe17e4b0da1-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T14:59:30.311 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.311 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.312 DEBUG:boto:Method: GET 2023-10-11T14:59:30.312 DEBUG:boto:Path: /dghqvd-47/obj6 2023-10-11T14:59:30.313 DEBUG:boto:Data: 2023-10-11T14:59:30.313 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.314 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.314 DEBUG:boto:Params: {} 2023-10-11T14:59:30.315 DEBUG:boto:Token: None 2023-10-11T14:59:30.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6 2023-10-11T14:59:30.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sgcEBLwhQwGUCSQ4eb16VZ4NH0E= 2023-10-11T14:59:30.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sgcEBLwhQwGUCSQ4eb16VZ4NH0E=', 'Content-Length': '0'} 2023-10-11T14:59:30.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006fb4b6f9c3e258a-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.319 DEBUG:boto:path=/dghqvd-47/obj6 2023-10-11T14:59:30.320 DEBUG:boto:auth_path=/dghqvd-47/obj6 2023-10-11T14:59:30.320 DEBUG:boto:Method: GET 2023-10-11T14:59:30.321 DEBUG:boto:Path: /dghqvd-47/obj6 2023-10-11T14:59:30.321 DEBUG:boto:Data: 2023-10-11T14:59:30.322 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.323 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.323 DEBUG:boto:Params: {} 2023-10-11T14:59:30.324 DEBUG:boto:Token: None 2023-10-11T14:59:30.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj6 2023-10-11T14:59:30.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sgcEBLwhQwGUCSQ4eb16VZ4NH0E= 2023-10-11T14:59:30.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sgcEBLwhQwGUCSQ4eb16VZ4NH0E=', 'Content-Length': '0'} 2023-10-11T14:59:30.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092d5b89ecf431d1e-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.328 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.328 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.329 DEBUG:boto:path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.329 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.329 DEBUG:boto:Method: GET 2023-10-11T14:59:30.330 DEBUG:boto:Path: /dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.330 DEBUG:boto:Data: 2023-10-11T14:59:30.331 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.332 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.332 DEBUG:boto:Params: {} 2023-10-11T14:59:30.333 DEBUG:boto:Token: None 2023-10-11T14:59:30.333 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e0314596089df4-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.337 DEBUG:boto:b'dghqvd-47obj61000falseobj62023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.338 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.339 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.339 DEBUG:boto:path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.340 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.340 DEBUG:boto:Method: GET 2023-10-11T14:59:30.340 DEBUG:boto:Path: /dghqvd-47/?prefix=obj6 2023-10-11T14:59:30.341 DEBUG:boto:Data: 2023-10-11T14:59:30.341 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.342 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.343 DEBUG:boto:Params: {} 2023-10-11T14:59:30.343 DEBUG:boto:Token: None 2023-10-11T14:59:30.344 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ef1dc3b55cfd73-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.348 DEBUG:boto:b'dghqvd-47obj61000falseobj62023-10-11T14:58:51.950Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:30.349 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.349 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.350 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.350 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.351 DEBUG:boto:Method: GET 2023-10-11T14:59:30.351 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.352 DEBUG:boto:Data: 2023-10-11T14:59:30.352 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.353 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.353 DEBUG:boto:Params: {} 2023-10-11T14:59:30.354 DEBUG:boto:Token: None 2023-10-11T14:59:30.354 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8= 2023-10-11T14:59:30.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8=', 'Content-Length': '0'} 2023-10-11T14:59:30.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c5f941d9c6d9005-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.358 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.358 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.359 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.359 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.360 DEBUG:boto:Method: GET 2023-10-11T14:59:30.360 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.361 DEBUG:boto:Data: 2023-10-11T14:59:30.361 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.362 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.362 DEBUG:boto:Params: {} 2023-10-11T14:59:30.363 DEBUG:boto:Token: None 2023-10-11T14:59:30.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8= 2023-10-11T14:59:30.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8=', 'Content-Length': '0'} 2023-10-11T14:59:30.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025f5433cfa4e50dd-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.414 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.415 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.415 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.416 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.416 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.417 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.417 DEBUG:boto:Data: 2023-10-11T14:59:30.417 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.418 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.419 DEBUG:boto:Params: {} 2023-10-11T14:59:30.419 DEBUG:boto:Token: None 2023-10-11T14:59:30.420 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qNNfMD/m39oWV97Gl7zHsbfCNqs= 2023-10-11T14:59:30.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qNNfMD/m39oWV97Gl7zHsbfCNqs=', 'Content-Length': '0'} 2023-10-11T14:59:30.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df097d42d3daf3a2-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.423 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.423 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.424 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.424 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.425 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.425 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.425 DEBUG:boto:Data: 2023-10-11T14:59:30.426 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.427 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.427 DEBUG:boto:Params: {} 2023-10-11T14:59:30.428 DEBUG:boto:Token: None 2023-10-11T14:59:30.428 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qNNfMD/m39oWV97Gl7zHsbfCNqs= 2023-10-11T14:59:30.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qNNfMD/m39oWV97Gl7zHsbfCNqs=', 'Content-Length': '0'} 2023-10-11T14:59:30.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c04058777fe20ebb-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:30.432 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.432 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.433 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.433 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.433 DEBUG:boto:Method: GET 2023-10-11T14:59:30.434 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.434 DEBUG:boto:Data: 2023-10-11T14:59:30.435 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.436 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.436 DEBUG:boto:Params: {} 2023-10-11T14:59:30.437 DEBUG:boto:Token: None 2023-10-11T14:59:30.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8= 2023-10-11T14:59:30.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8=', 'Content-Length': '0'} 2023-10-11T14:59:30.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbd7e85ec8559cec-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.482 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.483 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.484 DEBUG:boto:path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.484 DEBUG:boto:auth_path=/dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.484 DEBUG:boto:Method: GET 2023-10-11T14:59:30.485 DEBUG:boto:Path: /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.485 DEBUG:boto:Data: 2023-10-11T14:59:30.486 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.487 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.487 DEBUG:boto:Params: {} 2023-10-11T14:59:30.487 DEBUG:boto:Token: None 2023-10-11T14:59:30.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7?versionId=null 2023-10-11T14:59:30.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8= 2023-10-11T14:59:30.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Uu7GE0S4rfAAyfgcgmRlf01UT8=', 'Content-Length': '0'} 2023-10-11T14:59:30.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1471b44018fa8c-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.492 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.492 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.493 DEBUG:boto:path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.493 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.494 DEBUG:boto:Method: GET 2023-10-11T14:59:30.494 DEBUG:boto:Path: /dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.495 DEBUG:boto:Data: 2023-10-11T14:59:30.495 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.496 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.496 DEBUG:boto:Params: {} 2023-10-11T14:59:30.497 DEBUG:boto:Token: None 2023-10-11T14:59:30.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009662a522f3070aea-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.501 DEBUG:boto:b'dghqvd-47obj71000falseobj72023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.502 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.502 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.503 DEBUG:boto:path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.503 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.504 DEBUG:boto:Method: GET 2023-10-11T14:59:30.504 DEBUG:boto:Path: /dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.505 DEBUG:boto:Data: 2023-10-11T14:59:30.505 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.506 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.506 DEBUG:boto:Params: {} 2023-10-11T14:59:30.507 DEBUG:boto:Token: None 2023-10-11T14:59:30.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d39062af8a3fbeba-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.511 DEBUG:boto:b'dghqvd-47obj71000falseobj72023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.512 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.512 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.512 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.513 DEBUG:boto:Path: /dghqvd-47/obj7 2023-10-11T14:59:30.513 DEBUG:boto:Data: 2023-10-11T14:59:30.514 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.514 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.515 DEBUG:boto:Params: {} 2023-10-11T14:59:30.515 DEBUG:boto:Token: None 2023-10-11T14:59:30.516 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7 2023-10-11T14:59:30.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XHg1VFbVUJOmnBhFw847bI4krS0= 2023-10-11T14:59:30.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XHg1VFbVUJOmnBhFw847bI4krS0=', 'Content-Length': '0'} 2023-10-11T14:59:30.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086635223abc03e26-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.519 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.520 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.520 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.521 DEBUG:boto:Path: /dghqvd-47/obj7 2023-10-11T14:59:30.521 DEBUG:boto:Data: 2023-10-11T14:59:30.521 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.522 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.522 DEBUG:boto:Params: {} 2023-10-11T14:59:30.523 DEBUG:boto:Token: None 2023-10-11T14:59:30.523 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7 2023-10-11T14:59:30.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XHg1VFbVUJOmnBhFw847bI4krS0= 2023-10-11T14:59:30.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XHg1VFbVUJOmnBhFw847bI4krS0=', 'Content-Length': '0'} 2023-10-11T14:59:30.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e4b744d1e5510b1-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T14:59:30.527 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.528 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.528 DEBUG:boto:Method: GET 2023-10-11T14:59:30.529 DEBUG:boto:Path: /dghqvd-47/obj7 2023-10-11T14:59:30.529 DEBUG:boto:Data: 2023-10-11T14:59:30.529 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.530 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.531 DEBUG:boto:Params: {} 2023-10-11T14:59:30.531 DEBUG:boto:Token: None 2023-10-11T14:59:30.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7 2023-10-11T14:59:30.532 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bkhpGKUdki8UiE9veOo1u7KqVQo= 2023-10-11T14:59:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bkhpGKUdki8UiE9veOo1u7KqVQo=', 'Content-Length': '0'} 2023-10-11T14:59:30.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cda08cdc1880af00-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.535 DEBUG:boto:path=/dghqvd-47/obj7 2023-10-11T14:59:30.535 DEBUG:boto:auth_path=/dghqvd-47/obj7 2023-10-11T14:59:30.536 DEBUG:boto:Method: GET 2023-10-11T14:59:30.536 DEBUG:boto:Path: /dghqvd-47/obj7 2023-10-11T14:59:30.537 DEBUG:boto:Data: 2023-10-11T14:59:30.537 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.538 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.539 DEBUG:boto:Params: {} 2023-10-11T14:59:30.539 DEBUG:boto:Token: None 2023-10-11T14:59:30.540 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj7 2023-10-11T14:59:30.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bkhpGKUdki8UiE9veOo1u7KqVQo= 2023-10-11T14:59:30.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bkhpGKUdki8UiE9veOo1u7KqVQo=', 'Content-Length': '0'} 2023-10-11T14:59:30.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003398bdc7134468be-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.586 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.587 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.587 DEBUG:boto:path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.588 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.588 DEBUG:boto:Method: GET 2023-10-11T14:59:30.589 DEBUG:boto:Path: /dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.589 DEBUG:boto:Data: 2023-10-11T14:59:30.590 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.591 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.591 DEBUG:boto:Params: {} 2023-10-11T14:59:30.592 DEBUG:boto:Token: None 2023-10-11T14:59:30.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306860c067af2e18-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.596 DEBUG:boto:b'dghqvd-47obj71000falseobj72023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.597 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.597 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.598 DEBUG:boto:path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.598 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.599 DEBUG:boto:Method: GET 2023-10-11T14:59:30.599 DEBUG:boto:Path: /dghqvd-47/?prefix=obj7 2023-10-11T14:59:30.600 DEBUG:boto:Data: 2023-10-11T14:59:30.600 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.601 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.602 DEBUG:boto:Params: {} 2023-10-11T14:59:30.602 DEBUG:boto:Token: None 2023-10-11T14:59:30.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae12993fd5f2b30-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.607 DEBUG:boto:b'dghqvd-47obj71000falseobj72023-10-11T14:58:52.008Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:30.608 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.608 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.609 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.609 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.610 DEBUG:boto:Method: GET 2023-10-11T14:59:30.610 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.611 DEBUG:boto:Data: 2023-10-11T14:59:30.611 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.612 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.613 DEBUG:boto:Params: {} 2023-10-11T14:59:30.613 DEBUG:boto:Token: None 2023-10-11T14:59:30.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4= 2023-10-11T14:59:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4=', 'Content-Length': '0'} 2023-10-11T14:59:30.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d15fb43f7eea001b-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.658 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.659 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.660 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.660 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.660 DEBUG:boto:Method: GET 2023-10-11T14:59:30.661 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.661 DEBUG:boto:Data: 2023-10-11T14:59:30.662 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.663 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.663 DEBUG:boto:Params: {} 2023-10-11T14:59:30.664 DEBUG:boto:Token: None 2023-10-11T14:59:30.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4= 2023-10-11T14:59:30.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4=', 'Content-Length': '0'} 2023-10-11T14:59:30.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c18397b046bf39d-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.668 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.668 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.669 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.669 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.670 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.670 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.671 DEBUG:boto:Data: 2023-10-11T14:59:30.671 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.672 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.673 DEBUG:boto:Params: {} 2023-10-11T14:59:30.673 DEBUG:boto:Token: None 2023-10-11T14:59:30.674 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c9x4BTtTkWsR8URNPSxXoYimEVI= 2023-10-11T14:59:30.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c9x4BTtTkWsR8URNPSxXoYimEVI=', 'Content-Length': '0'} 2023-10-11T14:59:30.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9f3398cb6c59c47-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.677 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.678 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.678 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.679 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.679 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.680 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.680 DEBUG:boto:Data: 2023-10-11T14:59:30.680 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.681 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.682 DEBUG:boto:Params: {} 2023-10-11T14:59:30.682 DEBUG:boto:Token: None 2023-10-11T14:59:30.683 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c9x4BTtTkWsR8URNPSxXoYimEVI= 2023-10-11T14:59:30.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c9x4BTtTkWsR8URNPSxXoYimEVI=', 'Content-Length': '0'} 2023-10-11T14:59:30.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c12b588f04f5c1c-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:30.687 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.687 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.688 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.688 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.689 DEBUG:boto:Method: GET 2023-10-11T14:59:30.689 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.690 DEBUG:boto:Data: 2023-10-11T14:59:30.690 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.691 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.691 DEBUG:boto:Params: {} 2023-10-11T14:59:30.692 DEBUG:boto:Token: None 2023-10-11T14:59:30.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4= 2023-10-11T14:59:30.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4=', 'Content-Length': '0'} 2023-10-11T14:59:30.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045a5e6d707160d2f-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.696 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.696 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.697 DEBUG:boto:path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.697 DEBUG:boto:auth_path=/dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.698 DEBUG:boto:Method: GET 2023-10-11T14:59:30.698 DEBUG:boto:Path: /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.699 DEBUG:boto:Data: 2023-10-11T14:59:30.699 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.700 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.700 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.701 DEBUG:boto:Params: {} 2023-10-11T14:59:30.701 DEBUG:boto:Token: None 2023-10-11T14:59:30.702 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8?versionId=null 2023-10-11T14:59:30.702 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4= 2023-10-11T14:59:30.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jTYbDUAeBDCzic27BEBQ2IDsAo4=', 'Content-Length': '0'} 2023-10-11T14:59:30.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbc18fbc5283cfc4-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.705 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.706 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.706 DEBUG:boto:path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.707 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.707 DEBUG:boto:Method: GET 2023-10-11T14:59:30.708 DEBUG:boto:Path: /dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.708 DEBUG:boto:Data: 2023-10-11T14:59:30.709 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.709 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.710 DEBUG:boto:Params: {} 2023-10-11T14:59:30.710 DEBUG:boto:Token: None 2023-10-11T14:59:30.711 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001827769e2d1bbfbe-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.715 DEBUG:boto:b'dghqvd-47obj81000falseobj82023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.716 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.716 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.717 DEBUG:boto:path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.717 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.718 DEBUG:boto:Method: GET 2023-10-11T14:59:30.718 DEBUG:boto:Path: /dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.719 DEBUG:boto:Data: 2023-10-11T14:59:30.719 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.720 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.720 DEBUG:boto:Params: {} 2023-10-11T14:59:30.721 DEBUG:boto:Token: None 2023-10-11T14:59:30.722 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.722 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000268051afcae51604-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.726 DEBUG:boto:b'dghqvd-47obj81000falseobj82023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.727 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.727 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.728 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.728 DEBUG:boto:Path: /dghqvd-47/obj8 2023-10-11T14:59:30.728 DEBUG:boto:Data: 2023-10-11T14:59:30.729 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.730 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.730 DEBUG:boto:Params: {} 2023-10-11T14:59:30.731 DEBUG:boto:Token: None 2023-10-11T14:59:30.731 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8 2023-10-11T14:59:30.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ivoQyHLe6Rj3rSL+zGlRZzLAn2A= 2023-10-11T14:59:30.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ivoQyHLe6Rj3rSL+zGlRZzLAn2A=', 'Content-Length': '0'} 2023-10-11T14:59:30.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000157e0e02b79bed90-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.735 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.735 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.736 DEBUG:boto:Method: HEAD 2023-10-11T14:59:30.736 DEBUG:boto:Path: /dghqvd-47/obj8 2023-10-11T14:59:30.736 DEBUG:boto:Data: 2023-10-11T14:59:30.737 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.738 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.738 DEBUG:boto:Params: {} 2023-10-11T14:59:30.739 DEBUG:boto:Token: None 2023-10-11T14:59:30.739 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8 2023-10-11T14:59:30.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ivoQyHLe6Rj3rSL+zGlRZzLAn2A= 2023-10-11T14:59:30.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ivoQyHLe6Rj3rSL+zGlRZzLAn2A=', 'Content-Length': '0'} 2023-10-11T14:59:30.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000371309d3a9890f12-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T14:59:30.743 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.744 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.744 DEBUG:boto:Method: GET 2023-10-11T14:59:30.744 DEBUG:boto:Path: /dghqvd-47/obj8 2023-10-11T14:59:30.745 DEBUG:boto:Data: 2023-10-11T14:59:30.745 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.746 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.747 DEBUG:boto:Params: {} 2023-10-11T14:59:30.747 DEBUG:boto:Token: None 2023-10-11T14:59:30.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8 2023-10-11T14:59:30.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8Nj7nXeXhKpWytLb+J3CIhXqytE= 2023-10-11T14:59:30.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8Nj7nXeXhKpWytLb+J3CIhXqytE=', 'Content-Length': '0'} 2023-10-11T14:59:30.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a6cf845e68cf995-006526b852-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.751 DEBUG:boto:path=/dghqvd-47/obj8 2023-10-11T14:59:30.752 DEBUG:boto:auth_path=/dghqvd-47/obj8 2023-10-11T14:59:30.752 DEBUG:boto:Method: GET 2023-10-11T14:59:30.753 DEBUG:boto:Path: /dghqvd-47/obj8 2023-10-11T14:59:30.753 DEBUG:boto:Data: 2023-10-11T14:59:30.754 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.755 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.755 DEBUG:boto:Params: {} 2023-10-11T14:59:30.756 DEBUG:boto:Token: None 2023-10-11T14:59:30.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/obj8 2023-10-11T14:59:30.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8Nj7nXeXhKpWytLb+J3CIhXqytE= 2023-10-11T14:59:30.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8Nj7nXeXhKpWytLb+J3CIhXqytE=', 'Content-Length': '0'} 2023-10-11T14:59:30.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:58:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-47:daa316e9-869c-4074-a756-a8adfb4a3573.5751.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fa61669237ad91c-006526b852-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.760 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.760 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.761 DEBUG:boto:path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.761 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.762 DEBUG:boto:Method: GET 2023-10-11T14:59:30.763 DEBUG:boto:Path: /dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.763 DEBUG:boto:Data: 2023-10-11T14:59:30.764 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:30.764 DEBUG:boto:Port: 8001 2023-10-11T14:59:30.765 DEBUG:boto:Params: {} 2023-10-11T14:59:30.766 DEBUG:boto:Token: None 2023-10-11T14:59:30.766 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eafe49a38d33b6f0-006526b852-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.810 DEBUG:boto:b'dghqvd-47obj81000falseobj82023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.811 DEBUG:boto:path=/dghqvd-47/ 2023-10-11T14:59:30.812 DEBUG:boto:auth_path=/dghqvd-47/ 2023-10-11T14:59:30.812 DEBUG:boto:path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.813 DEBUG:boto:auth_path=/dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.813 DEBUG:boto:Method: GET 2023-10-11T14:59:30.814 DEBUG:boto:Path: /dghqvd-47/?prefix=obj8 2023-10-11T14:59:30.814 DEBUG:boto:Data: 2023-10-11T14:59:30.815 DEBUG:boto:Headers: {} 2023-10-11T14:59:30.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:30.815 DEBUG:boto:Port: 8000 2023-10-11T14:59:30.816 DEBUG:boto:Params: {} 2023-10-11T14:59:30.816 DEBUG:boto:Token: None 2023-10-11T14:59:30.817 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 14:59:30 GMT /dghqvd-47/ 2023-10-11T14:59:30.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4= 2023-10-11T14:59:30.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:30 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7yIg+Kkpn3+uVgSyyVlUro8ngD4=', 'Content-Length': '0'} 2023-10-11T14:59:30.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed4978172fb4412d-006526b852-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 14:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:30.821 DEBUG:boto:b'dghqvd-47obj81000falseobj82023-10-11T14:58:52.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T14:59:30.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-47 zones={a2, a1} 2023-10-11T14:59:30.822 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T14:59:30.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:30.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:30.823 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:30.872 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:30.872 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:30.918 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:59:30.919 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:30.919 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-11T14:59:30.919 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:06.505557Z" 2023-10-11T14:59:30.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:30.920 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:30.920 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-11T14:59:30.920 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:06.342311Z" 2023-10-11T14:59:30.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:30.921 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:30.921 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-11T14:59:30.921 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:06.429569Z" 2023-10-11T14:59:30.921 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:30.922 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:30.922 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005437", 2023-10-11T14:59:30.922 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:58:51.751940Z" 2023-10-11T14:59:30.922 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:30.923 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:59:30.924 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005437'} 2023-10-11T14:59:30.924 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:59:30.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:30.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:30.926 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:30.977 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:30.977 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:31.019 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:59:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:59:31.020 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:59:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:59:31.021 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:59:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T14:59:31.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:59:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:59:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.024 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-10-11T14:59:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.025 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.026 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:06.505557Z" 2023-10-11T14:59:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:59:31.028 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.029 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-10-11T14:59:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.030 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:06.342311Z" 2023-10-11T14:59:31.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:59:31.033 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006987", 2023-10-11T14:59:31.034 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.035 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:06.429569Z" 2023-10-11T14:59:31.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.037 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:59:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.038 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005437", 2023-10-11T14:59:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:58:51.751940Z" 2023-10-11T14:59:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:59:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:59:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:59:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:59:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:59:31Z" 2023-10-11T14:59:31.044 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:59:31.046 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:06.505557Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:06.342311Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006987", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:06.429569Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005437", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:58:51.751940Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:59:31Z" } 2023-10-11T14:59:31.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:06.505557Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:06.342311Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006987', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:06.429569Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005437', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:58:51.751940Z'}}] 2023-10-11T14:59:31.048 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005437'} 2023-10-11T14:59:31.048 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005437'} 2023-10-11T14:59:31.049 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T14:59:31.050 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T14:59:31.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.052 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:31.101 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:31.102 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:31.147 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:59:31.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.148 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-11T14:59:31.148 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:06.443027Z" 2023-10-11T14:59:31.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.149 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.150 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-10-11T14:59:31.150 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:06.485205Z" 2023-10-11T14:59:31.151 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.151 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.152 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006423", 2023-10-11T14:59:31.152 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:07.567651Z" 2023-10-11T14:59:31.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.153 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002761", 2023-10-11T14:59:31.154 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:55:48.871800Z" 2023-10-11T14:59:31.154 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.155 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:59:31.156 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002620', 1: '00000000000000000000:00000000000000001497', 2: '00000000000000000000:00000000000000006423', 3: '00000000000000000000:00000000000000002761'} 2023-10-11T14:59:31.157 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:59:31.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.160 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:31.213 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:31.214 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:31.254 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:59:31.254 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:59:31.255 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:59:31.255 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:59:31.255 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:59:31.255 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T14:59:31.255 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.256 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:59:31.256 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.256 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:59:31.256 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.257 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-10-11T14:59:31.257 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.257 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.257 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.258 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:06.443027Z" 2023-10-11T14:59:31.258 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.258 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:59:31.259 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.259 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.259 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-10-11T14:59:31.259 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.260 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.261 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.261 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:06.485205Z" 2023-10-11T14:59:31.261 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.262 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.262 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.262 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:59:31.262 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.263 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.263 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006423", 2023-10-11T14:59:31.263 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.263 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.263 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.264 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:07.567651Z" 2023-10-11T14:59:31.264 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.264 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.264 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:59:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T14:59:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000002761", 2023-10-11T14:59:31.265 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.266 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.266 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.266 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:55:48.871800Z" 2023-10-11T14:59:31.266 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.266 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.267 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:59:31.267 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.267 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:59:31.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T14:59:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T14:59:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:59:31Z" 2023-10-11T14:59:31.268 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:59:31.269 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:06.443027Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:06.485205Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006423", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:07.567651Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:55:48.871800Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T14:59:31Z" } 2023-10-11T14:59:31.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:06.443027Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:06.485205Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006423', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:07.567651Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:55:48.871800Z'}}] 2023-10-11T14:59:31.270 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002620', 1: '00000000000000000000:00000000000000001497', 2: '00000000000000000000:00000000000000006423', 3: '00000000000000000000:00000000000000002761'} 2023-10-11T14:59:31.271 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002620', 1: '00000000000000000000:00000000000000001497', 2: '00000000000000000000:00000000000000006423', 3: '00000000000000000000:00000000000000002761'} 2023-10-11T14:59:31.271 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T14:59:31.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-10-11T14:59:31.273 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-48 2023-10-11T14:59:31.274 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T14:59:31.274 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T14:59:31.274 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.275 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T14:59:31.275 DEBUG:boto:Data: 2023-10-11T14:59:31.276 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.276 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.277 DEBUG:boto:Params: {} 2023-10-11T14:59:31.277 DEBUG:boto:Token: None 2023-10-11T14:59:31.278 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/ 2023-10-11T14:59:31.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w5llBVNC9Aj3xZPqp19aGfxEMvY= 2023-10-11T14:59:31.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w5llBVNC9Aj3xZPqp19aGfxEMvY=', 'Content-Length': '0'} 2023-10-11T14:59:31.292 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c65b020d685edc5a-006526b853-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.293 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-49 2023-10-11T14:59:31.293 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T14:59:31.294 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T14:59:31.294 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.294 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T14:59:31.295 DEBUG:boto:Data: 2023-10-11T14:59:31.295 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.296 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.297 DEBUG:boto:Params: {} 2023-10-11T14:59:31.297 DEBUG:boto:Token: None 2023-10-11T14:59:31.298 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/ 2023-10-11T14:59:31.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AWUI6ZekbHM2HnETXQlS19vQOrA= 2023-10-11T14:59:31.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AWUI6ZekbHM2HnETXQlS19vQOrA=', 'Content-Length': '0'} 2023-10-11T14:59:31.320 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000499b14e1a284bfc7-006526b853-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.321 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T14:59:31.321 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T14:59:31.322 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.323 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T14:59:31.323 DEBUG:boto:Data: 2023-10-11T14:59:31.323 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.324 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.325 DEBUG:boto:Params: {} 2023-10-11T14:59:31.325 DEBUG:boto:Token: None 2023-10-11T14:59:31.326 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/ 2023-10-11T14:59:31.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg= 2023-10-11T14:59:31.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg=', 'Content-Length': '0'} 2023-10-11T14:59:31.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e2cc3c821229f3f-006526b853-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.330 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T14:59:31.331 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T14:59:31.331 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.332 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T14:59:31.332 DEBUG:boto:Data: 2023-10-11T14:59:31.332 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.333 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.334 DEBUG:boto:Params: {} 2023-10-11T14:59:31.334 DEBUG:boto:Token: None 2023-10-11T14:59:31.335 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/obj1 2023-10-11T14:59:31.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gjcxq4ewohJ5+cilfc/ZhH0C5ek= 2023-10-11T14:59:31.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gjcxq4ewohJ5+cilfc/ZhH0C5ek='} 2023-10-11T14:59:31.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000087aa7f4b766f485-006526b853-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.342 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T14:59:31.342 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T14:59:31.343 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.343 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T14:59:31.344 DEBUG:boto:Data: 2023-10-11T14:59:31.344 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.345 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.345 DEBUG:boto:Params: {} 2023-10-11T14:59:31.346 DEBUG:boto:Token: None 2023-10-11T14:59:31.346 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/ 2023-10-11T14:59:31.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg= 2023-10-11T14:59:31.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg=', 'Content-Length': '0'} 2023-10-11T14:59:31.349 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b4169c53a08af23-006526b853-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.350 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T14:59:31.350 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T14:59:31.351 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.351 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T14:59:31.352 DEBUG:boto:Data: 2023-10-11T14:59:31.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.353 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.353 DEBUG:boto:Params: {} 2023-10-11T14:59:31.354 DEBUG:boto:Token: None 2023-10-11T14:59:31.354 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/obj2 2023-10-11T14:59:31.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BAD/BlGmxjH3+ycheW7/uPNg0FE= 2023-10-11T14:59:31.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BAD/BlGmxjH3+ycheW7/uPNg0FE='} 2023-10-11T14:59:31.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cf05cb35ceb05b8-006526b853-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.399 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T14:59:31.399 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T14:59:31.400 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.400 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T14:59:31.400 DEBUG:boto:Data: 2023-10-11T14:59:31.401 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.402 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.402 DEBUG:boto:Params: {} 2023-10-11T14:59:31.402 DEBUG:boto:Token: None 2023-10-11T14:59:31.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/ 2023-10-11T14:59:31.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg= 2023-10-11T14:59:31.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg=', 'Content-Length': '0'} 2023-10-11T14:59:31.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000681e83b4b63922b7-006526b853-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.406 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T14:59:31.407 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T14:59:31.407 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.408 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T14:59:31.408 DEBUG:boto:Data: 2023-10-11T14:59:31.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.409 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.410 DEBUG:boto:Params: {} 2023-10-11T14:59:31.410 DEBUG:boto:Token: None 2023-10-11T14:59:31.411 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/obj3 2023-10-11T14:59:31.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:abiGP42DTaNUciL9BobwCg08rtU= 2023-10-11T14:59:31.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:abiGP42DTaNUciL9BobwCg08rtU='} 2023-10-11T14:59:31.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065532a8cd8d1e249-006526b853-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.417 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T14:59:31.417 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T14:59:31.418 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.418 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T14:59:31.419 DEBUG:boto:Data: 2023-10-11T14:59:31.419 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.420 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.420 DEBUG:boto:Params: {} 2023-10-11T14:59:31.420 DEBUG:boto:Token: None 2023-10-11T14:59:31.421 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/ 2023-10-11T14:59:31.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg= 2023-10-11T14:59:31.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L11Hyx5EpYwcR+iHR9JmDDNUdkg=', 'Content-Length': '0'} 2023-10-11T14:59:31.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a25b024d807f06a0-006526b853-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.424 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T14:59:31.425 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T14:59:31.425 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.426 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T14:59:31.426 DEBUG:boto:Data: 2023-10-11T14:59:31.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T14:59:31.427 DEBUG:boto:Port: 8000 2023-10-11T14:59:31.428 DEBUG:boto:Params: {} 2023-10-11T14:59:31.428 DEBUG:boto:Token: None 2023-10-11T14:59:31.428 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-48/obj4 2023-10-11T14:59:31.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AZGO8OagVihE4szDXNjStZ8X7hk= 2023-10-11T14:59:31.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AZGO8OagVihE4szDXNjStZ8X7hk='} 2023-10-11T14:59:31.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005405f486d5db20ce-006526b853-1675-a1'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.435 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T14:59:31.435 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T14:59:31.435 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.436 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T14:59:31.436 DEBUG:boto:Data: 2023-10-11T14:59:31.437 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.437 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.438 DEBUG:boto:Params: {} 2023-10-11T14:59:31.438 DEBUG:boto:Token: None 2023-10-11T14:59:31.439 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/ 2023-10-11T14:59:31.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w= 2023-10-11T14:59:31.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w=', 'Content-Length': '0'} 2023-10-11T14:59:31.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ee9f7a5e95d5268-006526b853-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.443 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T14:59:31.443 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T14:59:31.444 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.444 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T14:59:31.444 DEBUG:boto:Data: 2023-10-11T14:59:31.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.446 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.446 DEBUG:boto:Params: {} 2023-10-11T14:59:31.447 DEBUG:boto:Token: None 2023-10-11T14:59:31.447 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/obj1 2023-10-11T14:59:31.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kei0Ex0MhJsDsFw1gKXGV/ZX8hI= 2023-10-11T14:59:31.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kei0Ex0MhJsDsFw1gKXGV/ZX8hI='} 2023-10-11T14:59:31.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097eae71e1e2d900a-006526b853-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.454 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T14:59:31.455 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T14:59:31.455 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.455 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T14:59:31.456 DEBUG:boto:Data: 2023-10-11T14:59:31.456 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.457 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.457 DEBUG:boto:Params: {} 2023-10-11T14:59:31.458 DEBUG:boto:Token: None 2023-10-11T14:59:31.458 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/ 2023-10-11T14:59:31.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w= 2023-10-11T14:59:31.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w=', 'Content-Length': '0'} 2023-10-11T14:59:31.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cbf195f1939933e-006526b853-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.462 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T14:59:31.466 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T14:59:31.466 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.466 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T14:59:31.467 DEBUG:boto:Data: 2023-10-11T14:59:31.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.468 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.469 DEBUG:boto:Params: {} 2023-10-11T14:59:31.469 DEBUG:boto:Token: None 2023-10-11T14:59:31.469 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/obj2 2023-10-11T14:59:31.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FgqDsQYFyP/Gq8S+6LLmcSV3BpA= 2023-10-11T14:59:31.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FgqDsQYFyP/Gq8S+6LLmcSV3BpA='} 2023-10-11T14:59:31.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000203753541fa2b69f-006526b853-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.476 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T14:59:31.476 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T14:59:31.476 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.477 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T14:59:31.477 DEBUG:boto:Data: 2023-10-11T14:59:31.478 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.478 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.479 DEBUG:boto:Params: {} 2023-10-11T14:59:31.479 DEBUG:boto:Token: None 2023-10-11T14:59:31.480 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/ 2023-10-11T14:59:31.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w= 2023-10-11T14:59:31.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w=', 'Content-Length': '0'} 2023-10-11T14:59:31.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc7c619de1eb8452-006526b853-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.483 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T14:59:31.484 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T14:59:31.484 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.485 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T14:59:31.485 DEBUG:boto:Data: 2023-10-11T14:59:31.486 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.486 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.487 DEBUG:boto:Params: {} 2023-10-11T14:59:31.487 DEBUG:boto:Token: None 2023-10-11T14:59:31.488 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/obj3 2023-10-11T14:59:31.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VSvXY87RW5EPqPdJjtv1jhwxzrM= 2023-10-11T14:59:31.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VSvXY87RW5EPqPdJjtv1jhwxzrM='} 2023-10-11T14:59:31.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a4402e04523ce1d-006526b853-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.531 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T14:59:31.531 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T14:59:31.532 DEBUG:boto:Method: HEAD 2023-10-11T14:59:31.532 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T14:59:31.533 DEBUG:boto:Data: 2023-10-11T14:59:31.533 DEBUG:boto:Headers: {} 2023-10-11T14:59:31.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.534 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.534 DEBUG:boto:Params: {} 2023-10-11T14:59:31.535 DEBUG:boto:Token: None 2023-10-11T14:59:31.535 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/ 2023-10-11T14:59:31.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w= 2023-10-11T14:59:31.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O8mj9QtgKLqppLOlz5uRSRwlE1w=', 'Content-Length': '0'} 2023-10-11T14:59:31.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d86ae0b2bfe3fb38-006526b853-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.539 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T14:59:31.540 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T14:59:31.540 DEBUG:boto:Method: PUT 2023-10-11T14:59:31.540 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T14:59:31.541 DEBUG:boto:Data: 2023-10-11T14:59:31.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T14:59:31.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T14:59:31.542 DEBUG:boto:Port: 8001 2023-10-11T14:59:31.542 DEBUG:boto:Params: {} 2023-10-11T14:59:31.543 DEBUG:boto:Token: None 2023-10-11T14:59:31.543 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 14:59:31 GMT /dghqvd-49/obj4 2023-10-11T14:59:31.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KDGL+kmx1YqAVHiVzWUhy7KWthY= 2023-10-11T14:59:31.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 14:59:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KDGL+kmx1YqAVHiVzWUhy7KWthY='} 2023-10-11T14:59:31.549 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f27437660df50050-006526b853-1679-a2'), ('Date', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T14:59:31.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.550 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:31.598 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:31.599 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:31.627 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T14:59:31.639 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T14:59:31.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.640 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:59:31.640 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T14:59:31.640 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:59:31.641 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T14:59:31.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.642 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:59:31.642 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T14:59:31.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.642 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.642 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036371.310891_346.1", 2023-10-11T14:59:31.643 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:59:31.310891Z" 2023-10-11T14:59:31.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.643 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T14:59:31.644 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T14:59:31.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T14:59:31.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:31.646 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:31.695 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:31.696 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:31.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:31.721+0000 7fbf5f47e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:31.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:31.722+0000 7fbf5f47e780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:31.737 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:59:31.737 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:59:31.737 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:59:31.737 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:59:31.738 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:59:31.738 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:59:31.738 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:31.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:59:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:59:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:59:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:59:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:59:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:59:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:59:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:59:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:59:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:59:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:59:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:59:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036371.284367_339.1", 2023-10-11T14:59:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:31.284367Z", 2023-10-11T14:59:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:59:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:59:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:59:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:59:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:59:31Z" 2023-10-11T14:59:31.751 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:59:31.752 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036371.284367_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:31.284367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:59:31Z" } 2023-10-11T14:59:31.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036371.284367_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:31.284367Z', 'realm_epoch': 2}}] 2023-10-11T14:59:31.753 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T14:59:31.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.284367_339.1'} 2023-10-11T14:59:31.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697036371.310891_346.1 target=1_1697036371.284367_339.1 2023-10-11T14:59:36.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:36.757 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:36.814 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:36.815 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:36.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:36.844+0000 7f0dd78d3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:36.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:36.845+0000 7f0dd78d3780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:36.860 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:59:36.860 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:59:36.860 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:59:36.861 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:59:36.861 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:59:36.861 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:59:36.861 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:36.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.862 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:59:36.862 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:36.862 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:59:36.862 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:36.863 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:36.863 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:59:36.863 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:36.863 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:59:36.863 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:36.864 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:59:36.864 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:36.864 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:36.864 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:59:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:59:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:59:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:59:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:59:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:59:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:59:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036371.284367_339.1", 2023-10-11T14:59:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:31.284367Z", 2023-10-11T14:59:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:59:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:59:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:59:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:59:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:59:36Z" 2023-10-11T14:59:36.874 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:59:36.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036371.284367_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:31.284367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:59:36Z" } 2023-10-11T14:59:36.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036371.284367_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:31.284367Z', 'realm_epoch': 2}}] 2023-10-11T14:59:36.876 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T14:59:36.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.284367_339.1'} 2023-10-11T14:59:36.877 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697036371.310891_346.1 target=1_1697036371.284367_339.1 2023-10-11T14:59:41.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:41.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T14:59:41.880 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T14:59:41.938 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:41.938 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:41.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:41.967+0000 7f343f500780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:41.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:41.968+0000 7f343f500780 20 RGW-SYNC:meta: read sync status 2023-10-11T14:59:41.982 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T14:59:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T14:59:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T14:59:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T14:59:41.984 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T14:59:41.984 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T14:59:41.984 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:41.984 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.984 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T14:59:41.985 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:41.985 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T14:59:41.985 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:41.985 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:41.985 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T14:59:41.986 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:41.986 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T14:59:41.986 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:41.986 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T14:59:41.987 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:41.987 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:41.987 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:41.987 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T14:59:41.988 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:41.988 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:41.988 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T14:59:41.988 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:41.988 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T14:59:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T14:59:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T14:59:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T14:59:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T14:59:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T14:59:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T14:59:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036371.310891_346.1", 2023-10-11T14:59:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T14:59:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T14:59:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T14:59:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:31.310891Z", 2023-10-11T14:59:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T14:59:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T14:59:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T14:59:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T14:59:41Z" 2023-10-11T14:59:41.996 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T14:59:41.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036371.310891_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:31.310891Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T14:59:41Z" } 2023-10-11T14:59:41.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036371.310891_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:31.310891Z', 'realm_epoch': 2}}] 2023-10-11T14:59:41.998 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T14:59:41.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T14:59:41.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T14:59:41.999 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-48 2023-10-11T14:59:41.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:42.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:42.000 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:59:42.058 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:42.059 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:42.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:42.090+0000 7f3ad02c6780 1 waiting to reach incremental sync.. 2023-10-11T14:59:47.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:47.091+0000 7f3ad02c6780 1 waiting to reach incremental sync.. 2023-10-11T14:59:52.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:52.092+0000 7f3ad02c6780 1 bucket sync caught up with source: 2023-10-11T14:59:52.098 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.28.6, , 00000000001.30.6, , 00000000001.26.6, , , 00000000001.48.6, , , ] 2023-10-11T14:59:52.098 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.28.6, , 00000000001.30.6, , 00000000001.26.6, , , 00000000001.48.6, , , ] 2023-10-11T14:59:52.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:52.092+0000 7f3ad02c6780 0 bucket checkpoint complete 2023-10-11T14:59:52.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-48 2023-10-11T14:59:52.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:52.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T14:59:52.112 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T14:59:52.176 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T14:59:52.176 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T14:59:52.212 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:52.210+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T14:59:57.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T14:59:57.211+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:02.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:02.211+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:07.214 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:07.211+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:12.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:12.212+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:17.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:17.213+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:22.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:22.213+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:27.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:27.214+0000 7fc8e8e55780 1 waiting to reach incremental sync.. 2023-10-11T15:00:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:32.216+0000 7fc8e8e55780 1 bucket sync caught up with source: 2023-10-11T15:00:32.219 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.230.6, , 00000000001.164.6, , 00000000001.167.6, , , 00000000001.276.6, , , ] 2023-10-11T15:00:32.219 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.230.6, , 00000000001.164.6, , 00000000001.167.6, , , 00000000001.276.6, , , ] 2023-10-11T15:00:32.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:32.216+0000 7fc8e8e55780 0 bucket checkpoint complete 2023-10-11T15:00:32.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-48 zones={a2, a1} 2023-10-11T15:00:32.237 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.238 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.238 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.239 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:32.239 DEBUG:boto:Data: 2023-10-11T15:00:32.240 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.240 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.241 DEBUG:boto:Params: {} 2023-10-11T15:00:32.241 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:00:32.242 DEBUG:boto:Token: None 2023-10-11T15:00:32.242 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDUXUr7T24/MrPfErrcpQeZjKmI= 2023-10-11T15:00:32.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDUXUr7T24/MrPfErrcpQeZjKmI=', 'Content-Length': '0'} 2023-10-11T15:00:32.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c006cba3fc2edd37-006526b890-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.248 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.248 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.249 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.249 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:32.250 DEBUG:boto:Data: 2023-10-11T15:00:32.250 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.251 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.251 DEBUG:boto:Params: {} 2023-10-11T15:00:32.252 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:00:32.252 DEBUG:boto:Token: None 2023-10-11T15:00:32.253 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDUXUr7T24/MrPfErrcpQeZjKmI= 2023-10-11T15:00:32.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDUXUr7T24/MrPfErrcpQeZjKmI=', 'Content-Length': '0'} 2023-10-11T15:00:32.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001234fbed603baf7a-006526b890-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.258 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:00:32.258 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.258 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.259 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.259 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.260 DEBUG:boto:Method: GET 2023-10-11T15:00:32.260 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.260 DEBUG:boto:Data: 2023-10-11T15:00:32.261 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.261 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.262 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.262 DEBUG:boto:Params: {} 2023-10-11T15:00:32.263 DEBUG:boto:Token: None 2023-10-11T15:00:32.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/?versions 2023-10-11T15:00:32.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo= 2023-10-11T15:00:32.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo=', 'Content-Length': '0'} 2023-10-11T15:00:32.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d06ffbf98e9761-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.267 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:00:32.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:00:32.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:00:32.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:00:32.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:00:32.269 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.270 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.270 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.271 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.271 DEBUG:boto:Method: GET 2023-10-11T15:00:32.272 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.272 DEBUG:boto:Data: 2023-10-11T15:00:32.272 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.273 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.274 DEBUG:boto:Params: {} 2023-10-11T15:00:32.274 DEBUG:boto:Token: None 2023-10-11T15:00:32.275 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/?versions 2023-10-11T15:00:32.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo= 2023-10-11T15:00:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo=', 'Content-Length': '0'} 2023-10-11T15:00:32.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa048715f67209f1-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.278 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:00:32.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:00:32.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:00:32.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:00:32.281 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.281 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.282 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.282 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.282 DEBUG:boto:Method: GET 2023-10-11T15:00:32.283 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.283 DEBUG:boto:Data: 2023-10-11T15:00:32.284 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.284 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.285 DEBUG:boto:Params: {} 2023-10-11T15:00:32.285 DEBUG:boto:Token: None 2023-10-11T15:00:32.286 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/?versions 2023-10-11T15:00:32.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo= 2023-10-11T15:00:32.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo=', 'Content-Length': '0'} 2023-10-11T15:00:32.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba31012ce5e779e-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.290 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.290 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.291 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.291 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.292 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.292 DEBUG:boto:Method: GET 2023-10-11T15:00:32.292 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:00:32.293 DEBUG:boto:Data: 2023-10-11T15:00:32.293 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.294 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.294 DEBUG:boto:Params: {} 2023-10-11T15:00:32.295 DEBUG:boto:Token: None 2023-10-11T15:00:32.295 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/?versions 2023-10-11T15:00:32.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo= 2023-10-11T15:00:32.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4rhuTJfQFFkHz7OS4Yw5rqLTiwo=', 'Content-Length': '0'} 2023-10-11T15:00:32.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2d0c31fe4354064-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.300 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:32.301 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.302 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.302 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.303 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.303 DEBUG:boto:Method: GET 2023-10-11T15:00:32.304 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.304 DEBUG:boto:Data: 2023-10-11T15:00:32.304 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.305 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.306 DEBUG:boto:Params: {} 2023-10-11T15:00:32.306 DEBUG:boto:Token: None 2023-10-11T15:00:32.307 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4= 2023-10-11T15:00:32.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4=', 'Content-Length': '0'} 2023-10-11T15:00:32.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006391f706799b69c1-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.310 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.311 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.311 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.312 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.312 DEBUG:boto:Method: GET 2023-10-11T15:00:32.312 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.313 DEBUG:boto:Data: 2023-10-11T15:00:32.313 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.314 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.315 DEBUG:boto:Params: {} 2023-10-11T15:00:32.315 DEBUG:boto:Token: None 2023-10-11T15:00:32.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4= 2023-10-11T15:00:32.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4=', 'Content-Length': '0'} 2023-10-11T15:00:32.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005ee6f7539f55e8d-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.319 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.319 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.320 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.320 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.321 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.321 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.321 DEBUG:boto:Data: 2023-10-11T15:00:32.322 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.323 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.323 DEBUG:boto:Params: {} 2023-10-11T15:00:32.324 DEBUG:boto:Token: None 2023-10-11T15:00:32.324 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qr+EFlfqVme58Uul1+siw0xffss= 2023-10-11T15:00:32.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qr+EFlfqVme58Uul1+siw0xffss=', 'Content-Length': '0'} 2023-10-11T15:00:32.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017f7e4e9c7cc43e0-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.327 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.328 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.328 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.328 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.329 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.329 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.330 DEBUG:boto:Data: 2023-10-11T15:00:32.330 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.331 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.331 DEBUG:boto:Params: {} 2023-10-11T15:00:32.332 DEBUG:boto:Token: None 2023-10-11T15:00:32.332 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.333 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qr+EFlfqVme58Uul1+siw0xffss= 2023-10-11T15:00:32.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qr+EFlfqVme58Uul1+siw0xffss=', 'Content-Length': '0'} 2023-10-11T15:00:32.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac7333e6ccdeb234-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:32.336 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.336 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.337 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.337 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.338 DEBUG:boto:Method: GET 2023-10-11T15:00:32.338 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.339 DEBUG:boto:Data: 2023-10-11T15:00:32.339 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.340 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.340 DEBUG:boto:Params: {} 2023-10-11T15:00:32.341 DEBUG:boto:Token: None 2023-10-11T15:00:32.341 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4= 2023-10-11T15:00:32.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4=', 'Content-Length': '0'} 2023-10-11T15:00:32.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ed09325839810c7-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.386 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.387 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.387 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.388 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.388 DEBUG:boto:Method: GET 2023-10-11T15:00:32.389 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.389 DEBUG:boto:Data: 2023-10-11T15:00:32.389 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.390 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.390 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.391 DEBUG:boto:Params: {} 2023-10-11T15:00:32.391 DEBUG:boto:Token: None 2023-10-11T15:00:32.392 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:00:32.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4= 2023-10-11T15:00:32.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAzc2m8fpXKPLB/0JMSK1HjXjn4=', 'Content-Length': '0'} 2023-10-11T15:00:32.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdefb69291bacf82-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.395 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.396 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.396 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.397 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.397 DEBUG:boto:Method: GET 2023-10-11T15:00:32.397 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.398 DEBUG:boto:Data: 2023-10-11T15:00:32.398 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.399 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.400 DEBUG:boto:Params: {} 2023-10-11T15:00:32.400 DEBUG:boto:Token: None 2023-10-11T15:00:32.401 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004082a8ef02386ada-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.405 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.405 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.406 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.406 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.407 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.407 DEBUG:boto:Method: GET 2023-10-11T15:00:32.408 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.408 DEBUG:boto:Data: 2023-10-11T15:00:32.408 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.409 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.410 DEBUG:boto:Params: {} 2023-10-11T15:00:32.410 DEBUG:boto:Token: None 2023-10-11T15:00:32.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079ec6b0db920e15a-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.454 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.455 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.456 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.456 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.456 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:00:32.457 DEBUG:boto:Data: 2023-10-11T15:00:32.457 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.458 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.458 DEBUG:boto:Params: {} 2023-10-11T15:00:32.459 DEBUG:boto:Token: None 2023-10-11T15:00:32.459 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1 2023-10-11T15:00:32.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jJk+OR9JT4IQhGzlaC82tDGjUow= 2023-10-11T15:00:32.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jJk+OR9JT4IQhGzlaC82tDGjUow=', 'Content-Length': '0'} 2023-10-11T15:00:32.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd9fbf300f06ebcf-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.463 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.463 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.464 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.464 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:00:32.465 DEBUG:boto:Data: 2023-10-11T15:00:32.465 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.466 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.466 DEBUG:boto:Params: {} 2023-10-11T15:00:32.467 DEBUG:boto:Token: None 2023-10-11T15:00:32.467 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1 2023-10-11T15:00:32.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jJk+OR9JT4IQhGzlaC82tDGjUow= 2023-10-11T15:00:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jJk+OR9JT4IQhGzlaC82tDGjUow=', 'Content-Length': '0'} 2023-10-11T15:00:32.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000071ab2fcb750c55-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:32.471 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.471 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.472 DEBUG:boto:Method: GET 2023-10-11T15:00:32.472 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:00:32.473 DEBUG:boto:Data: 2023-10-11T15:00:32.473 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.474 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.474 DEBUG:boto:Params: {} 2023-10-11T15:00:32.475 DEBUG:boto:Token: None 2023-10-11T15:00:32.475 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1 2023-10-11T15:00:32.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82kNgDEnpI4aajHHmIfm/usAvtM= 2023-10-11T15:00:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82kNgDEnpI4aajHHmIfm/usAvtM=', 'Content-Length': '0'} 2023-10-11T15:00:32.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428502844abd90fb-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.479 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:00:32.479 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:00:32.480 DEBUG:boto:Method: GET 2023-10-11T15:00:32.480 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:00:32.481 DEBUG:boto:Data: 2023-10-11T15:00:32.481 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.482 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.482 DEBUG:boto:Params: {} 2023-10-11T15:00:32.483 DEBUG:boto:Token: None 2023-10-11T15:00:32.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj1 2023-10-11T15:00:32.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82kNgDEnpI4aajHHmIfm/usAvtM= 2023-10-11T15:00:32.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82kNgDEnpI4aajHHmIfm/usAvtM=', 'Content-Length': '0'} 2023-10-11T15:00:32.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae8d06e09f4b9ea0-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.487 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.487 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.488 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.488 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.488 DEBUG:boto:Method: GET 2023-10-11T15:00:32.489 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.489 DEBUG:boto:Data: 2023-10-11T15:00:32.490 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.490 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.491 DEBUG:boto:Params: {} 2023-10-11T15:00:32.491 DEBUG:boto:Token: None 2023-10-11T15:00:32.492 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aac02c63b25c47b-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.538 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.539 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.540 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.540 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.540 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.541 DEBUG:boto:Method: GET 2023-10-11T15:00:32.541 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:00:32.542 DEBUG:boto:Data: 2023-10-11T15:00:32.542 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.543 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.544 DEBUG:boto:Params: {} 2023-10-11T15:00:32.544 DEBUG:boto:Token: None 2023-10-11T15:00:32.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008303a2b73b52ffd3-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.549 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:32.551 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.551 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.552 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.552 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.553 DEBUG:boto:Method: GET 2023-10-11T15:00:32.553 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.554 DEBUG:boto:Data: 2023-10-11T15:00:32.554 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.555 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.555 DEBUG:boto:Params: {} 2023-10-11T15:00:32.556 DEBUG:boto:Token: None 2023-10-11T15:00:32.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94= 2023-10-11T15:00:32.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94=', 'Content-Length': '0'} 2023-10-11T15:00:32.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001805827f435f51a5-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.560 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.561 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.561 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.561 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.562 DEBUG:boto:Method: GET 2023-10-11T15:00:32.562 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.563 DEBUG:boto:Data: 2023-10-11T15:00:32.563 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.564 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.564 DEBUG:boto:Params: {} 2023-10-11T15:00:32.565 DEBUG:boto:Token: None 2023-10-11T15:00:32.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94= 2023-10-11T15:00:32.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94=', 'Content-Length': '0'} 2023-10-11T15:00:32.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009922f5162787c4a0-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.569 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.569 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.570 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.570 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.571 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.571 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.571 DEBUG:boto:Data: 2023-10-11T15:00:32.572 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.573 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.573 DEBUG:boto:Params: {} 2023-10-11T15:00:32.573 DEBUG:boto:Token: None 2023-10-11T15:00:32.574 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UUBfXZY2kmlEEvRmutq1uyf90gg= 2023-10-11T15:00:32.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UUBfXZY2kmlEEvRmutq1uyf90gg=', 'Content-Length': '0'} 2023-10-11T15:00:32.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ee722ff35b93db4-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.578 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.578 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.578 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.579 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.579 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.580 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.580 DEBUG:boto:Data: 2023-10-11T15:00:32.581 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.581 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.582 DEBUG:boto:Params: {} 2023-10-11T15:00:32.582 DEBUG:boto:Token: None 2023-10-11T15:00:32.583 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UUBfXZY2kmlEEvRmutq1uyf90gg= 2023-10-11T15:00:32.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UUBfXZY2kmlEEvRmutq1uyf90gg=', 'Content-Length': '0'} 2023-10-11T15:00:32.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a025de3dabae67bc-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:32.589 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.589 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.590 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.590 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.591 DEBUG:boto:Method: GET 2023-10-11T15:00:32.591 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.591 DEBUG:boto:Data: 2023-10-11T15:00:32.592 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.593 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.593 DEBUG:boto:Params: {} 2023-10-11T15:00:32.593 DEBUG:boto:Token: None 2023-10-11T15:00:32.594 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94= 2023-10-11T15:00:32.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94=', 'Content-Length': '0'} 2023-10-11T15:00:32.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094879dac54e53d42-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.598 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.598 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.599 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.599 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.600 DEBUG:boto:Method: GET 2023-10-11T15:00:32.600 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.600 DEBUG:boto:Data: 2023-10-11T15:00:32.601 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.602 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.602 DEBUG:boto:Params: {} 2023-10-11T15:00:32.603 DEBUG:boto:Token: None 2023-10-11T15:00:32.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:00:32.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94= 2023-10-11T15:00:32.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xdt6cArM03kT7Zs2qrDdX4yRh94=', 'Content-Length': '0'} 2023-10-11T15:00:32.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ec50529b69963f2-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.607 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.607 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.607 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.608 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.608 DEBUG:boto:Method: GET 2023-10-11T15:00:32.609 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.609 DEBUG:boto:Data: 2023-10-11T15:00:32.609 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.610 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.611 DEBUG:boto:Params: {} 2023-10-11T15:00:32.611 DEBUG:boto:Token: None 2023-10-11T15:00:32.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec3f7d26c771e06-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.616 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.616 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.617 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.617 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.618 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.618 DEBUG:boto:Method: GET 2023-10-11T15:00:32.618 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.619 DEBUG:boto:Data: 2023-10-11T15:00:32.619 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.620 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.620 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.620 DEBUG:boto:Params: {} 2023-10-11T15:00:32.621 DEBUG:boto:Token: None 2023-10-11T15:00:32.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.622 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008833c5b45d249947-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.625 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.626 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.626 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.627 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.627 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:00:32.628 DEBUG:boto:Data: 2023-10-11T15:00:32.628 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.629 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.629 DEBUG:boto:Params: {} 2023-10-11T15:00:32.630 DEBUG:boto:Token: None 2023-10-11T15:00:32.630 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2 2023-10-11T15:00:32.631 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ota3Q/nsWGwXpZPQ7lnL4fP8OWU= 2023-10-11T15:00:32.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ota3Q/nsWGwXpZPQ7lnL4fP8OWU=', 'Content-Length': '0'} 2023-10-11T15:00:32.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4a55fa377c1a9f5-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.634 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.634 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.634 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.635 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:00:32.635 DEBUG:boto:Data: 2023-10-11T15:00:32.636 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.636 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.636 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.637 DEBUG:boto:Params: {} 2023-10-11T15:00:32.637 DEBUG:boto:Token: None 2023-10-11T15:00:32.638 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2 2023-10-11T15:00:32.638 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ota3Q/nsWGwXpZPQ7lnL4fP8OWU= 2023-10-11T15:00:32.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ota3Q/nsWGwXpZPQ7lnL4fP8OWU=', 'Content-Length': '0'} 2023-10-11T15:00:32.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec032d7d59362300-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:32.642 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.642 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.643 DEBUG:boto:Method: GET 2023-10-11T15:00:32.643 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:00:32.643 DEBUG:boto:Data: 2023-10-11T15:00:32.644 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.644 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.645 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.645 DEBUG:boto:Params: {} 2023-10-11T15:00:32.646 DEBUG:boto:Token: None 2023-10-11T15:00:32.646 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2 2023-10-11T15:00:32.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n8ankFgC/Ew6PcUsgLkZx3ksiuI= 2023-10-11T15:00:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n8ankFgC/Ew6PcUsgLkZx3ksiuI=', 'Content-Length': '0'} 2023-10-11T15:00:32.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8c0f68025eed3f0-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.650 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:00:32.650 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:00:32.651 DEBUG:boto:Method: GET 2023-10-11T15:00:32.651 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:00:32.651 DEBUG:boto:Data: 2023-10-11T15:00:32.652 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.653 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.653 DEBUG:boto:Params: {} 2023-10-11T15:00:32.654 DEBUG:boto:Token: None 2023-10-11T15:00:32.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj2 2023-10-11T15:00:32.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n8ankFgC/Ew6PcUsgLkZx3ksiuI= 2023-10-11T15:00:32.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n8ankFgC/Ew6PcUsgLkZx3ksiuI=', 'Content-Length': '0'} 2023-10-11T15:00:32.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b00457b61a263e58-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.657 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.658 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.658 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.659 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.659 DEBUG:boto:Method: GET 2023-10-11T15:00:32.660 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.660 DEBUG:boto:Data: 2023-10-11T15:00:32.661 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.661 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.662 DEBUG:boto:Params: {} 2023-10-11T15:00:32.662 DEBUG:boto:Token: None 2023-10-11T15:00:32.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7e98a802b64947b-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.667 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.668 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.668 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.669 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.669 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.670 DEBUG:boto:Method: GET 2023-10-11T15:00:32.670 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:00:32.670 DEBUG:boto:Data: 2023-10-11T15:00:32.671 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.672 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.672 DEBUG:boto:Params: {} 2023-10-11T15:00:32.673 DEBUG:boto:Token: None 2023-10-11T15:00:32.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd1b2f9a135238f-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.677 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:32.679 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.679 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.679 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.680 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.680 DEBUG:boto:Method: GET 2023-10-11T15:00:32.681 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.681 DEBUG:boto:Data: 2023-10-11T15:00:32.682 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.682 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.683 DEBUG:boto:Params: {} 2023-10-11T15:00:32.683 DEBUG:boto:Token: None 2023-10-11T15:00:32.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s= 2023-10-11T15:00:32.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s=', 'Content-Length': '0'} 2023-10-11T15:00:32.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057d77f2d2a606c5a-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.730 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.731 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.731 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.732 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.732 DEBUG:boto:Method: GET 2023-10-11T15:00:32.733 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.733 DEBUG:boto:Data: 2023-10-11T15:00:32.733 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.734 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.735 DEBUG:boto:Params: {} 2023-10-11T15:00:32.735 DEBUG:boto:Token: None 2023-10-11T15:00:32.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s= 2023-10-11T15:00:32.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s=', 'Content-Length': '0'} 2023-10-11T15:00:32.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5620287f5a410bc-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.739 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.740 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.740 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.741 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.741 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.741 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.742 DEBUG:boto:Data: 2023-10-11T15:00:32.742 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.743 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.743 DEBUG:boto:Params: {} 2023-10-11T15:00:32.744 DEBUG:boto:Token: None 2023-10-11T15:00:32.744 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xi+6gpRlvdHYM4I4ZHSzZFEwAx0= 2023-10-11T15:00:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xi+6gpRlvdHYM4I4ZHSzZFEwAx0=', 'Content-Length': '0'} 2023-10-11T15:00:32.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcc53cd2bf2700ef-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.748 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.748 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.748 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.749 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.749 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.750 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.750 DEBUG:boto:Data: 2023-10-11T15:00:32.751 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.751 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.752 DEBUG:boto:Params: {} 2023-10-11T15:00:32.752 DEBUG:boto:Token: None 2023-10-11T15:00:32.753 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xi+6gpRlvdHYM4I4ZHSzZFEwAx0= 2023-10-11T15:00:32.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xi+6gpRlvdHYM4I4ZHSzZFEwAx0=', 'Content-Length': '0'} 2023-10-11T15:00:32.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006894bbbad71066ec-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:32.757 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.757 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.758 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.758 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.758 DEBUG:boto:Method: GET 2023-10-11T15:00:32.759 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.759 DEBUG:boto:Data: 2023-10-11T15:00:32.760 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.761 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.761 DEBUG:boto:Params: {} 2023-10-11T15:00:32.761 DEBUG:boto:Token: None 2023-10-11T15:00:32.762 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s= 2023-10-11T15:00:32.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s=', 'Content-Length': '0'} 2023-10-11T15:00:32.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067a1be13b9069ead-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.766 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.766 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.766 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.767 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.767 DEBUG:boto:Method: GET 2023-10-11T15:00:32.768 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.768 DEBUG:boto:Data: 2023-10-11T15:00:32.769 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.769 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.770 DEBUG:boto:Params: {} 2023-10-11T15:00:32.770 DEBUG:boto:Token: None 2023-10-11T15:00:32.771 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:00:32.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s= 2023-10-11T15:00:32.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A7RJdUGTLLOS+z0AVz0ldh18e6s=', 'Content-Length': '0'} 2023-10-11T15:00:32.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f63efd876a0e8fe-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.774 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.775 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.775 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.776 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.776 DEBUG:boto:Method: GET 2023-10-11T15:00:32.777 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.777 DEBUG:boto:Data: 2023-10-11T15:00:32.778 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.778 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.779 DEBUG:boto:Params: {} 2023-10-11T15:00:32.779 DEBUG:boto:Token: None 2023-10-11T15:00:32.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc9884cdd017f85-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.784 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.784 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.785 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.785 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.786 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.786 DEBUG:boto:Method: GET 2023-10-11T15:00:32.787 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.789 DEBUG:boto:Data: 2023-10-11T15:00:32.789 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.790 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.791 DEBUG:boto:Params: {} 2023-10-11T15:00:32.791 DEBUG:boto:Token: None 2023-10-11T15:00:32.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a12a0b471c69b8-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.796 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.796 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.797 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.797 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.798 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:00:32.798 DEBUG:boto:Data: 2023-10-11T15:00:32.799 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.799 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.800 DEBUG:boto:Params: {} 2023-10-11T15:00:32.800 DEBUG:boto:Token: None 2023-10-11T15:00:32.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3 2023-10-11T15:00:32.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jNAIM1pmK1Acppb+PjUyWZYkVJc= 2023-10-11T15:00:32.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jNAIM1pmK1Acppb+PjUyWZYkVJc=', 'Content-Length': '0'} 2023-10-11T15:00:32.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5671a2392317634-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.804 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.804 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.805 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.805 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:00:32.806 DEBUG:boto:Data: 2023-10-11T15:00:32.806 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.807 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.807 DEBUG:boto:Params: {} 2023-10-11T15:00:32.808 DEBUG:boto:Token: None 2023-10-11T15:00:32.808 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3 2023-10-11T15:00:32.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jNAIM1pmK1Acppb+PjUyWZYkVJc= 2023-10-11T15:00:32.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jNAIM1pmK1Acppb+PjUyWZYkVJc=', 'Content-Length': '0'} 2023-10-11T15:00:32.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6a31f042409c1dd-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:32.812 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.813 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.813 DEBUG:boto:Method: GET 2023-10-11T15:00:32.813 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:00:32.814 DEBUG:boto:Data: 2023-10-11T15:00:32.814 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.815 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.816 DEBUG:boto:Params: {} 2023-10-11T15:00:32.816 DEBUG:boto:Token: None 2023-10-11T15:00:32.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3 2023-10-11T15:00:32.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BvHk4de1Coo0QzkzTXOS0g2GqbA= 2023-10-11T15:00:32.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BvHk4de1Coo0QzkzTXOS0g2GqbA=', 'Content-Length': '0'} 2023-10-11T15:00:32.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009987e313ea7d48ce-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.862 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:00:32.863 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:00:32.863 DEBUG:boto:Method: GET 2023-10-11T15:00:32.864 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:00:32.864 DEBUG:boto:Data: 2023-10-11T15:00:32.865 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.866 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.866 DEBUG:boto:Params: {} 2023-10-11T15:00:32.866 DEBUG:boto:Token: None 2023-10-11T15:00:32.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj3 2023-10-11T15:00:32.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BvHk4de1Coo0QzkzTXOS0g2GqbA= 2023-10-11T15:00:32.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BvHk4de1Coo0QzkzTXOS0g2GqbA=', 'Content-Length': '0'} 2023-10-11T15:00:32.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d2626c0d2dfdd0-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.871 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.871 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.872 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.872 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.873 DEBUG:boto:Method: GET 2023-10-11T15:00:32.873 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.873 DEBUG:boto:Data: 2023-10-11T15:00:32.874 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.875 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.875 DEBUG:boto:Params: {} 2023-10-11T15:00:32.876 DEBUG:boto:Token: None 2023-10-11T15:00:32.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cfa571bfaf2eea7-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.880 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.881 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.882 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.882 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.882 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.883 DEBUG:boto:Method: GET 2023-10-11T15:00:32.883 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:00:32.884 DEBUG:boto:Data: 2023-10-11T15:00:32.884 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.885 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.885 DEBUG:boto:Params: {} 2023-10-11T15:00:32.886 DEBUG:boto:Token: None 2023-10-11T15:00:32.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a88e5ecd9523f2-006526b890-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.890 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:32.891 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.892 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.892 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.893 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.893 DEBUG:boto:Method: GET 2023-10-11T15:00:32.894 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.894 DEBUG:boto:Data: 2023-10-11T15:00:32.894 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.895 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.896 DEBUG:boto:Params: {} 2023-10-11T15:00:32.896 DEBUG:boto:Token: None 2023-10-11T15:00:32.897 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ= 2023-10-11T15:00:32.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ=', 'Content-Length': '0'} 2023-10-11T15:00:32.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9fa89dbf6fa398c-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.900 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.901 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.901 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.901 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.902 DEBUG:boto:Method: GET 2023-10-11T15:00:32.902 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.903 DEBUG:boto:Data: 2023-10-11T15:00:32.903 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.904 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.904 DEBUG:boto:Params: {} 2023-10-11T15:00:32.905 DEBUG:boto:Token: None 2023-10-11T15:00:32.905 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.906 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ= 2023-10-11T15:00:32.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ=', 'Content-Length': '0'} 2023-10-11T15:00:32.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c3fa56d9c0d285d-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.909 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.910 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.910 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.911 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.911 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.911 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.912 DEBUG:boto:Data: 2023-10-11T15:00:32.912 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.913 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.913 DEBUG:boto:Params: {} 2023-10-11T15:00:32.914 DEBUG:boto:Token: None 2023-10-11T15:00:32.914 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xTkz2SokCEV1aaB2qVfUaqp9V/Q= 2023-10-11T15:00:32.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xTkz2SokCEV1aaB2qVfUaqp9V/Q=', 'Content-Length': '0'} 2023-10-11T15:00:32.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e380365edee5ce4e-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.918 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.918 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.919 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.919 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.919 DEBUG:boto:Method: HEAD 2023-10-11T15:00:32.920 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.920 DEBUG:boto:Data: 2023-10-11T15:00:32.921 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.921 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.922 DEBUG:boto:Params: {} 2023-10-11T15:00:32.922 DEBUG:boto:Token: None 2023-10-11T15:00:32.923 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xTkz2SokCEV1aaB2qVfUaqp9V/Q= 2023-10-11T15:00:32.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xTkz2SokCEV1aaB2qVfUaqp9V/Q=', 'Content-Length': '0'} 2023-10-11T15:00:32.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2b0b80a15461bf6-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:32.927 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.927 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.928 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.928 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.928 DEBUG:boto:Method: GET 2023-10-11T15:00:32.929 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.929 DEBUG:boto:Data: 2023-10-11T15:00:32.930 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.931 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.931 DEBUG:boto:Params: {} 2023-10-11T15:00:32.932 DEBUG:boto:Token: None 2023-10-11T15:00:32.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ= 2023-10-11T15:00:32.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ=', 'Content-Length': '0'} 2023-10-11T15:00:32.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9bbc2ca64f74d73-006526b890-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.936 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:32.936 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:32.936 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.937 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.937 DEBUG:boto:Method: GET 2023-10-11T15:00:32.938 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.938 DEBUG:boto:Data: 2023-10-11T15:00:32.939 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:32.939 DEBUG:boto:Port: 8000 2023-10-11T15:00:32.940 DEBUG:boto:Params: {} 2023-10-11T15:00:32.940 DEBUG:boto:Token: None 2023-10-11T15:00:32.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:00:32.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ= 2023-10-11T15:00:32.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ennEqUCyI0o4VQ97CRpATJUGPZQ=', 'Content-Length': '0'} 2023-10-11T15:00:32.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d09dbb398469f198-006526b890-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.945 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.945 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.945 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.946 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.946 DEBUG:boto:Method: GET 2023-10-11T15:00:32.947 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.947 DEBUG:boto:Data: 2023-10-11T15:00:32.948 DEBUG:boto:Headers: {} 2023-10-11T15:00:32.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:32.948 DEBUG:boto:Port: 8001 2023-10-11T15:00:32.949 DEBUG:boto:Params: {} 2023-10-11T15:00:32.949 DEBUG:boto:Token: None 2023-10-11T15:00:32.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:32 GMT /dghqvd-48/ 2023-10-11T15:00:32.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI= 2023-10-11T15:00:32.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pUiO+jtlsFj6hHwIVykiQLdEKRI=', 'Content-Length': '0'} 2023-10-11T15:00:32.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0abbcb41337fe26-006526b890-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:32.994 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:32.995 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:32.997 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:32.998 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.998 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.999 DEBUG:boto:Method: GET 2023-10-11T15:00:32.999 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:00:32.999 DEBUG:boto:Data: 2023-10-11T15:00:33.000 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.001 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.001 DEBUG:boto:Params: {} 2023-10-11T15:00:33.002 DEBUG:boto:Token: None 2023-10-11T15:00:33.002 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/ 2023-10-11T15:00:33.003 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc= 2023-10-11T15:00:33.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc=', 'Content-Length': '0'} 2023-10-11T15:00:33.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a21433b6b05009b1-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.006 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.007 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:33.007 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:33.008 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.008 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:00:33.009 DEBUG:boto:Data: 2023-10-11T15:00:33.009 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.010 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.010 DEBUG:boto:Params: {} 2023-10-11T15:00:33.011 DEBUG:boto:Token: None 2023-10-11T15:00:33.011 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/obj4 2023-10-11T15:00:33.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qu9JI4tz8EvG0dK5+WT41UTUdd4= 2023-10-11T15:00:33.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qu9JI4tz8EvG0dK5+WT41UTUdd4=', 'Content-Length': '0'} 2023-10-11T15:00:33.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcfa511e77e18178-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.015 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:33.015 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:33.015 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.016 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:00:33.016 DEBUG:boto:Data: 2023-10-11T15:00:33.017 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.017 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.018 DEBUG:boto:Params: {} 2023-10-11T15:00:33.018 DEBUG:boto:Token: None 2023-10-11T15:00:33.019 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/obj4 2023-10-11T15:00:33.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qu9JI4tz8EvG0dK5+WT41UTUdd4= 2023-10-11T15:00:33.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qu9JI4tz8EvG0dK5+WT41UTUdd4=', 'Content-Length': '0'} 2023-10-11T15:00:33.022 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000503a2e9576a9509a-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:33.023 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:33.023 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:33.023 DEBUG:boto:Method: GET 2023-10-11T15:00:33.024 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:00:33.024 DEBUG:boto:Data: 2023-10-11T15:00:33.025 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.026 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.026 DEBUG:boto:Params: {} 2023-10-11T15:00:33.026 DEBUG:boto:Token: None 2023-10-11T15:00:33.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/obj4 2023-10-11T15:00:33.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lp7arxQG8k47m70bj4x0y5ewu/A= 2023-10-11T15:00:33.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lp7arxQG8k47m70bj4x0y5ewu/A=', 'Content-Length': '0'} 2023-10-11T15:00:33.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d1e3942a20a985b-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.031 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:00:33.031 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:00:33.031 DEBUG:boto:Method: GET 2023-10-11T15:00:33.032 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:00:33.032 DEBUG:boto:Data: 2023-10-11T15:00:33.033 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.034 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.034 DEBUG:boto:Params: {} 2023-10-11T15:00:33.034 DEBUG:boto:Token: None 2023-10-11T15:00:33.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/obj4 2023-10-11T15:00:33.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Lp7arxQG8k47m70bj4x0y5ewu/A= 2023-10-11T15:00:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Lp7arxQG8k47m70bj4x0y5ewu/A=', 'Content-Length': '0'} 2023-10-11T15:00:33.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f21f1da5e9420d6-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.039 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:33.039 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:33.039 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.040 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.040 DEBUG:boto:Method: GET 2023-10-11T15:00:33.041 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.041 DEBUG:boto:Data: 2023-10-11T15:00:33.042 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.042 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.043 DEBUG:boto:Params: {} 2023-10-11T15:00:33.043 DEBUG:boto:Token: None 2023-10-11T15:00:33.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/ 2023-10-11T15:00:33.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc= 2023-10-11T15:00:33.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc=', 'Content-Length': '0'} 2023-10-11T15:00:33.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5af0a4596dd5023-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.048 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.048 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:33.049 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:33.049 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.050 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.050 DEBUG:boto:Method: GET 2023-10-11T15:00:33.051 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:00:33.051 DEBUG:boto:Data: 2023-10-11T15:00:33.051 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.052 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.053 DEBUG:boto:Params: {} 2023-10-11T15:00:33.053 DEBUG:boto:Token: None 2023-10-11T15:00:33.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-48/ 2023-10-11T15:00:33.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc= 2023-10-11T15:00:33.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2JjWWvcGkd+TDJEif6XwUA4/UOc=', 'Content-Length': '0'} 2023-10-11T15:00:33.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0b9258853e57d2-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.058 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-48 zones={a2, a1} 2023-10-11T15:00:33.059 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-49 2023-10-11T15:00:33.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:33.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:33.061 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:00:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:33.127 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:33.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:33.163+0000 7f329f13c780 1 bucket sync caught up with source: 2023-10-11T15:00:33.166 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.32.6, , 00000000001.31.6, , 00000000001.42.6, , , 00000000001.19.6, , , ] 2023-10-11T15:00:33.166 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.32.6, , 00000000001.31.6, , 00000000001.42.6, , , 00000000001.19.6, , , ] 2023-10-11T15:00:33.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:33.163+0000 7f329f13c780 0 bucket checkpoint complete 2023-10-11T15:00:33.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-49 2023-10-11T15:00:33.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:33.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:33.181 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:00:33.237 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:33.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:33.268+0000 7f9ad059d780 1 bucket sync caught up with source: 2023-10-11T15:00:33.272 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.147.6, , 00000000001.137.6, , 00000000001.249.6, , , 00000000001.168.6, , , ] 2023-10-11T15:00:33.272 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.147.6, , 00000000001.137.6, , 00000000001.249.6, , , 00000000001.168.6, , , ] 2023-10-11T15:00:33.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:33.268+0000 7f9ad059d780 0 bucket checkpoint complete 2023-10-11T15:00:33.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-49 zones={a2, a1} 2023-10-11T15:00:33.284 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.284 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.285 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.285 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:33.286 DEBUG:boto:Data: 2023-10-11T15:00:33.286 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.287 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.287 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.288 DEBUG:boto:Params: {} 2023-10-11T15:00:33.288 DEBUG:boto:Token: None 2023-10-11T15:00:33.289 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CehThhqQDc8VofXUgKkBgFiUYTQ= 2023-10-11T15:00:33.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CehThhqQDc8VofXUgKkBgFiUYTQ=', 'Content-Length': '0'} 2023-10-11T15:00:33.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b03d2034393e377-006526b891-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.292 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.293 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.293 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.294 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:33.294 DEBUG:boto:Data: 2023-10-11T15:00:33.295 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.295 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.296 DEBUG:boto:Params: {} 2023-10-11T15:00:33.296 DEBUG:boto:Token: None 2023-10-11T15:00:33.297 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CehThhqQDc8VofXUgKkBgFiUYTQ= 2023-10-11T15:00:33.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CehThhqQDc8VofXUgKkBgFiUYTQ=', 'Content-Length': '0'} 2023-10-11T15:00:33.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aefcbcc3e9b4e5d-006526b891-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:00:33.301 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.301 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.302 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.302 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.302 DEBUG:boto:Method: GET 2023-10-11T15:00:33.303 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.303 DEBUG:boto:Data: 2023-10-11T15:00:33.304 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.304 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.305 DEBUG:boto:Params: {} 2023-10-11T15:00:33.305 DEBUG:boto:Token: None 2023-10-11T15:00:33.306 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/?versions 2023-10-11T15:00:33.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8= 2023-10-11T15:00:33.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8=', 'Content-Length': '0'} 2023-10-11T15:00:33.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097b27433ab0e9a2d-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.310 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:00:33.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:00:33.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:00:33.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:00:33.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:00:33.312 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.313 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.313 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.314 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.314 DEBUG:boto:Method: GET 2023-10-11T15:00:33.314 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.315 DEBUG:boto:Data: 2023-10-11T15:00:33.315 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.316 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.317 DEBUG:boto:Params: {} 2023-10-11T15:00:33.317 DEBUG:boto:Token: None 2023-10-11T15:00:33.317 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/?versions 2023-10-11T15:00:33.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8= 2023-10-11T15:00:33.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8=', 'Content-Length': '0'} 2023-10-11T15:00:33.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007366a89163f0ed2c-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.321 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:00:33.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:00:33.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:00:33.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:00:33.323 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.324 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.324 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.325 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.325 DEBUG:boto:Method: GET 2023-10-11T15:00:33.326 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.326 DEBUG:boto:Data: 2023-10-11T15:00:33.326 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.327 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.327 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.328 DEBUG:boto:Params: {} 2023-10-11T15:00:33.328 DEBUG:boto:Token: None 2023-10-11T15:00:33.329 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/?versions 2023-10-11T15:00:33.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8= 2023-10-11T15:00:33.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8=', 'Content-Length': '0'} 2023-10-11T15:00:33.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042faf7b1494c7776-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.333 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.333 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.334 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.334 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.335 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.335 DEBUG:boto:Method: GET 2023-10-11T15:00:33.336 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:00:33.336 DEBUG:boto:Data: 2023-10-11T15:00:33.336 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.337 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.337 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.338 DEBUG:boto:Params: {} 2023-10-11T15:00:33.338 DEBUG:boto:Token: None 2023-10-11T15:00:33.339 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/?versions 2023-10-11T15:00:33.339 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8= 2023-10-11T15:00:33.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJWz7pTtcwc4A/NkQP2Zoqq6Mv8=', 'Content-Length': '0'} 2023-10-11T15:00:33.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002949fc13cb6185c8-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.343 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:33.344 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.345 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.345 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.345 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.346 DEBUG:boto:Method: GET 2023-10-11T15:00:33.346 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.347 DEBUG:boto:Data: 2023-10-11T15:00:33.347 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.348 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.348 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.349 DEBUG:boto:Params: {} 2023-10-11T15:00:33.349 DEBUG:boto:Token: None 2023-10-11T15:00:33.349 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.350 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8= 2023-10-11T15:00:33.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8=', 'Content-Length': '0'} 2023-10-11T15:00:33.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f6e7eef00e54f90-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.353 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.354 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.354 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.355 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.355 DEBUG:boto:Method: GET 2023-10-11T15:00:33.356 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.356 DEBUG:boto:Data: 2023-10-11T15:00:33.357 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.357 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.357 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.358 DEBUG:boto:Params: {} 2023-10-11T15:00:33.358 DEBUG:boto:Token: None 2023-10-11T15:00:33.359 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.359 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8= 2023-10-11T15:00:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8=', 'Content-Length': '0'} 2023-10-11T15:00:33.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a95aa223552d8204-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.362 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.363 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.363 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.364 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.364 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.365 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.365 DEBUG:boto:Data: 2023-10-11T15:00:33.365 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.366 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.367 DEBUG:boto:Params: {} 2023-10-11T15:00:33.367 DEBUG:boto:Token: None 2023-10-11T15:00:33.367 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dythHa3xc3F2NfKM9y1h22lmao= 2023-10-11T15:00:33.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dythHa3xc3F2NfKM9y1h22lmao=', 'Content-Length': '0'} 2023-10-11T15:00:33.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007720c137325902fb-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.371 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.371 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.371 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.372 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.372 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.373 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.373 DEBUG:boto:Data: 2023-10-11T15:00:33.374 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.374 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.375 DEBUG:boto:Params: {} 2023-10-11T15:00:33.375 DEBUG:boto:Token: None 2023-10-11T15:00:33.376 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dythHa3xc3F2NfKM9y1h22lmao= 2023-10-11T15:00:33.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dythHa3xc3F2NfKM9y1h22lmao=', 'Content-Length': '0'} 2023-10-11T15:00:33.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007374d606294c6cc-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:33.380 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.380 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.381 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.381 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.381 DEBUG:boto:Method: GET 2023-10-11T15:00:33.382 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.382 DEBUG:boto:Data: 2023-10-11T15:00:33.383 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.383 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.384 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.384 DEBUG:boto:Params: {} 2023-10-11T15:00:33.384 DEBUG:boto:Token: None 2023-10-11T15:00:33.385 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8= 2023-10-11T15:00:33.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8=', 'Content-Length': '0'} 2023-10-11T15:00:33.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000277fc2d6cba2c150-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.388 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.389 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.389 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.390 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.390 DEBUG:boto:Method: GET 2023-10-11T15:00:33.391 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.391 DEBUG:boto:Data: 2023-10-11T15:00:33.391 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.392 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.393 DEBUG:boto:Params: {} 2023-10-11T15:00:33.393 DEBUG:boto:Token: None 2023-10-11T15:00:33.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:00:33.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8= 2023-10-11T15:00:33.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fNqYIGwILEhKit1rAdCSTsy5cT8=', 'Content-Length': '0'} 2023-10-11T15:00:33.397 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000433b2560b33a7af3-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.438 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.439 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.439 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.440 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.440 DEBUG:boto:Method: GET 2023-10-11T15:00:33.441 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.441 DEBUG:boto:Data: 2023-10-11T15:00:33.441 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.442 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.443 DEBUG:boto:Params: {} 2023-10-11T15:00:33.443 DEBUG:boto:Token: None 2023-10-11T15:00:33.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047999ee94eba7a63-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.448 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.448 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.449 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.449 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.450 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.450 DEBUG:boto:Method: GET 2023-10-11T15:00:33.451 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.451 DEBUG:boto:Data: 2023-10-11T15:00:33.452 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.453 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.453 DEBUG:boto:Params: {} 2023-10-11T15:00:33.454 DEBUG:boto:Token: None 2023-10-11T15:00:33.454 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000540f05d77c56539c-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.458 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.459 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.460 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.460 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.461 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:00:33.461 DEBUG:boto:Data: 2023-10-11T15:00:33.461 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.462 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.463 DEBUG:boto:Params: {} 2023-10-11T15:00:33.463 DEBUG:boto:Token: None 2023-10-11T15:00:33.464 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1 2023-10-11T15:00:33.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XLMOF6fc7qjlXDewRRJ7UKCFc+4= 2023-10-11T15:00:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XLMOF6fc7qjlXDewRRJ7UKCFc+4=', 'Content-Length': '0'} 2023-10-11T15:00:33.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c926f3f09419f3d6-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.467 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.467 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.468 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.468 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:00:33.468 DEBUG:boto:Data: 2023-10-11T15:00:33.469 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.470 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.470 DEBUG:boto:Params: {} 2023-10-11T15:00:33.471 DEBUG:boto:Token: None 2023-10-11T15:00:33.471 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1 2023-10-11T15:00:33.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XLMOF6fc7qjlXDewRRJ7UKCFc+4= 2023-10-11T15:00:33.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XLMOF6fc7qjlXDewRRJ7UKCFc+4=', 'Content-Length': '0'} 2023-10-11T15:00:33.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061aa917883c062c1-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:00:33.475 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.475 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.476 DEBUG:boto:Method: GET 2023-10-11T15:00:33.476 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:00:33.476 DEBUG:boto:Data: 2023-10-11T15:00:33.477 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.478 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.478 DEBUG:boto:Params: {} 2023-10-11T15:00:33.479 DEBUG:boto:Token: None 2023-10-11T15:00:33.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1 2023-10-11T15:00:33.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CWprrmbDFvyKacJ0a96ifMBYY1Y= 2023-10-11T15:00:33.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CWprrmbDFvyKacJ0a96ifMBYY1Y=', 'Content-Length': '0'} 2023-10-11T15:00:33.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000700ec622f3aaea4d-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.522 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:00:33.523 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:00:33.523 DEBUG:boto:Method: GET 2023-10-11T15:00:33.524 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:00:33.524 DEBUG:boto:Data: 2023-10-11T15:00:33.525 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.525 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.526 DEBUG:boto:Params: {} 2023-10-11T15:00:33.526 DEBUG:boto:Token: None 2023-10-11T15:00:33.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj1 2023-10-11T15:00:33.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CWprrmbDFvyKacJ0a96ifMBYY1Y= 2023-10-11T15:00:33.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CWprrmbDFvyKacJ0a96ifMBYY1Y=', 'Content-Length': '0'} 2023-10-11T15:00:33.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2f95156b0b48174-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.530 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.531 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.531 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.532 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.532 DEBUG:boto:Method: GET 2023-10-11T15:00:33.532 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.533 DEBUG:boto:Data: 2023-10-11T15:00:33.533 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.534 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.534 DEBUG:boto:Params: {} 2023-10-11T15:00:33.535 DEBUG:boto:Token: None 2023-10-11T15:00:33.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000879bb75561d615db-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.539 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.540 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.540 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.541 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.541 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.541 DEBUG:boto:Method: GET 2023-10-11T15:00:33.542 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:00:33.542 DEBUG:boto:Data: 2023-10-11T15:00:33.543 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.544 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.544 DEBUG:boto:Params: {} 2023-10-11T15:00:33.544 DEBUG:boto:Token: None 2023-10-11T15:00:33.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a4503120889a09-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.549 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:33.550 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.550 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.551 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.551 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.552 DEBUG:boto:Method: GET 2023-10-11T15:00:33.552 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.553 DEBUG:boto:Data: 2023-10-11T15:00:33.553 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.554 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.554 DEBUG:boto:Params: {} 2023-10-11T15:00:33.555 DEBUG:boto:Token: None 2023-10-11T15:00:33.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0= 2023-10-11T15:00:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0=', 'Content-Length': '0'} 2023-10-11T15:00:33.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008685ecd3eae946cf-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.559 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.559 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.560 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.560 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.560 DEBUG:boto:Method: GET 2023-10-11T15:00:33.561 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.561 DEBUG:boto:Data: 2023-10-11T15:00:33.562 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.563 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.563 DEBUG:boto:Params: {} 2023-10-11T15:00:33.563 DEBUG:boto:Token: None 2023-10-11T15:00:33.564 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0= 2023-10-11T15:00:33.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0=', 'Content-Length': '0'} 2023-10-11T15:00:33.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe1fa4bf26b1278-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.567 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.568 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.568 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.569 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.569 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.569 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.570 DEBUG:boto:Data: 2023-10-11T15:00:33.570 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.571 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.571 DEBUG:boto:Params: {} 2023-10-11T15:00:33.572 DEBUG:boto:Token: None 2023-10-11T15:00:33.572 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qv7wwOkjiGJ4IOE0UG8KMXJEcAc= 2023-10-11T15:00:33.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qv7wwOkjiGJ4IOE0UG8KMXJEcAc=', 'Content-Length': '0'} 2023-10-11T15:00:33.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b016d234942c627c-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.575 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.576 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.576 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.577 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.577 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.578 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.578 DEBUG:boto:Data: 2023-10-11T15:00:33.578 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.579 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.580 DEBUG:boto:Params: {} 2023-10-11T15:00:33.580 DEBUG:boto:Token: None 2023-10-11T15:00:33.581 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qv7wwOkjiGJ4IOE0UG8KMXJEcAc= 2023-10-11T15:00:33.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qv7wwOkjiGJ4IOE0UG8KMXJEcAc=', 'Content-Length': '0'} 2023-10-11T15:00:33.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006004d42e39fa486-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:33.585 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.585 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.586 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.586 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.587 DEBUG:boto:Method: GET 2023-10-11T15:00:33.587 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.588 DEBUG:boto:Data: 2023-10-11T15:00:33.588 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.589 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.589 DEBUG:boto:Params: {} 2023-10-11T15:00:33.590 DEBUG:boto:Token: None 2023-10-11T15:00:33.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0= 2023-10-11T15:00:33.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0=', 'Content-Length': '0'} 2023-10-11T15:00:33.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073274e0ee64ffbb2-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.634 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.635 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.635 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.636 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.636 DEBUG:boto:Method: GET 2023-10-11T15:00:33.637 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.637 DEBUG:boto:Data: 2023-10-11T15:00:33.638 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.639 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.639 DEBUG:boto:Params: {} 2023-10-11T15:00:33.639 DEBUG:boto:Token: None 2023-10-11T15:00:33.640 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:00:33.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0= 2023-10-11T15:00:33.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FB9ZHxPWdc3UXcrGS1q6nSnzcV0=', 'Content-Length': '0'} 2023-10-11T15:00:33.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f313aac5541ef11-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.643 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.644 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.644 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.645 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.645 DEBUG:boto:Method: GET 2023-10-11T15:00:33.646 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.646 DEBUG:boto:Data: 2023-10-11T15:00:33.647 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.647 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.648 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.648 DEBUG:boto:Params: {} 2023-10-11T15:00:33.649 DEBUG:boto:Token: None 2023-10-11T15:00:33.649 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.650 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1eec8471f84fd39-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.655 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.656 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.656 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.657 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.657 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.658 DEBUG:boto:Method: GET 2023-10-11T15:00:33.658 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.659 DEBUG:boto:Data: 2023-10-11T15:00:33.659 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.660 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.660 DEBUG:boto:Params: {} 2023-10-11T15:00:33.661 DEBUG:boto:Token: None 2023-10-11T15:00:33.661 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009648c93830047987-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.706 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.707 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.708 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.708 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.708 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:00:33.709 DEBUG:boto:Data: 2023-10-11T15:00:33.709 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.710 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.710 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.711 DEBUG:boto:Params: {} 2023-10-11T15:00:33.711 DEBUG:boto:Token: None 2023-10-11T15:00:33.711 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2 2023-10-11T15:00:33.712 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:71FPFqu9oVSKoAZE270085UsHLw= 2023-10-11T15:00:33.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:71FPFqu9oVSKoAZE270085UsHLw=', 'Content-Length': '0'} 2023-10-11T15:00:33.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0986cc06bb8616a-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.715 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.715 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.715 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.716 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:00:33.716 DEBUG:boto:Data: 2023-10-11T15:00:33.717 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.718 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.718 DEBUG:boto:Params: {} 2023-10-11T15:00:33.718 DEBUG:boto:Token: None 2023-10-11T15:00:33.719 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2 2023-10-11T15:00:33.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:71FPFqu9oVSKoAZE270085UsHLw= 2023-10-11T15:00:33.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:71FPFqu9oVSKoAZE270085UsHLw=', 'Content-Length': '0'} 2023-10-11T15:00:33.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003006aeebcd7208c7-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:00:33.722 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.723 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.723 DEBUG:boto:Method: GET 2023-10-11T15:00:33.724 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:00:33.724 DEBUG:boto:Data: 2023-10-11T15:00:33.725 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.725 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.725 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.726 DEBUG:boto:Params: {} 2023-10-11T15:00:33.726 DEBUG:boto:Token: None 2023-10-11T15:00:33.727 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2 2023-10-11T15:00:33.727 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8zyFOrT7AufrX4Op/o2qUyS+BN0= 2023-10-11T15:00:33.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8zyFOrT7AufrX4Op/o2qUyS+BN0=', 'Content-Length': '0'} 2023-10-11T15:00:33.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000695206b16b247815-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.730 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:00:33.731 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:00:33.731 DEBUG:boto:Method: GET 2023-10-11T15:00:33.731 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:00:33.732 DEBUG:boto:Data: 2023-10-11T15:00:33.732 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.733 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.733 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.734 DEBUG:boto:Params: {} 2023-10-11T15:00:33.734 DEBUG:boto:Token: None 2023-10-11T15:00:33.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj2 2023-10-11T15:00:33.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8zyFOrT7AufrX4Op/o2qUyS+BN0= 2023-10-11T15:00:33.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8zyFOrT7AufrX4Op/o2qUyS+BN0=', 'Content-Length': '0'} 2023-10-11T15:00:33.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac6e45eb45aa34fc-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.738 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.738 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.739 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.739 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.740 DEBUG:boto:Method: GET 2023-10-11T15:00:33.740 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.740 DEBUG:boto:Data: 2023-10-11T15:00:33.741 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.742 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.742 DEBUG:boto:Params: {} 2023-10-11T15:00:33.743 DEBUG:boto:Token: None 2023-10-11T15:00:33.743 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b50b51189fc7c179-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.747 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.748 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.748 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.748 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.749 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.749 DEBUG:boto:Method: GET 2023-10-11T15:00:33.750 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:00:33.750 DEBUG:boto:Data: 2023-10-11T15:00:33.751 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.751 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.752 DEBUG:boto:Params: {} 2023-10-11T15:00:33.752 DEBUG:boto:Token: None 2023-10-11T15:00:33.753 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000730a8d3aa75380a7-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.757 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:33.758 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.758 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.758 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.759 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.759 DEBUG:boto:Method: GET 2023-10-11T15:00:33.760 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.760 DEBUG:boto:Data: 2023-10-11T15:00:33.761 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.761 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.762 DEBUG:boto:Params: {} 2023-10-11T15:00:33.762 DEBUG:boto:Token: None 2023-10-11T15:00:33.763 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0= 2023-10-11T15:00:33.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0=', 'Content-Length': '0'} 2023-10-11T15:00:33.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004ac693d13b0e913-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.766 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.767 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.767 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.768 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.768 DEBUG:boto:Method: GET 2023-10-11T15:00:33.769 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.769 DEBUG:boto:Data: 2023-10-11T15:00:33.770 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.770 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.771 DEBUG:boto:Params: {} 2023-10-11T15:00:33.771 DEBUG:boto:Token: None 2023-10-11T15:00:33.772 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0= 2023-10-11T15:00:33.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0=', 'Content-Length': '0'} 2023-10-11T15:00:33.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004184e153efb8266f-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.775 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.775 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.776 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.776 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.777 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.777 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.777 DEBUG:boto:Data: 2023-10-11T15:00:33.778 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.779 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.779 DEBUG:boto:Params: {} 2023-10-11T15:00:33.780 DEBUG:boto:Token: None 2023-10-11T15:00:33.780 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rYyXkLz92aZilacdhysP7AJNrig= 2023-10-11T15:00:33.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rYyXkLz92aZilacdhysP7AJNrig=', 'Content-Length': '0'} 2023-10-11T15:00:33.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e0de90a4e6086c-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.783 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.784 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.784 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.784 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.785 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.785 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.786 DEBUG:boto:Data: 2023-10-11T15:00:33.786 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.787 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.787 DEBUG:boto:Params: {} 2023-10-11T15:00:33.788 DEBUG:boto:Token: None 2023-10-11T15:00:33.788 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rYyXkLz92aZilacdhysP7AJNrig= 2023-10-11T15:00:33.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rYyXkLz92aZilacdhysP7AJNrig=', 'Content-Length': '0'} 2023-10-11T15:00:33.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad7c7f43db1f54cc-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:33.792 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.792 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.793 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.793 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.793 DEBUG:boto:Method: GET 2023-10-11T15:00:33.794 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.794 DEBUG:boto:Data: 2023-10-11T15:00:33.795 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.796 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.796 DEBUG:boto:Params: {} 2023-10-11T15:00:33.796 DEBUG:boto:Token: None 2023-10-11T15:00:33.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0= 2023-10-11T15:00:33.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0=', 'Content-Length': '0'} 2023-10-11T15:00:33.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000634c5f1fc8c24bb3-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.842 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.843 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.843 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.844 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.844 DEBUG:boto:Method: GET 2023-10-11T15:00:33.845 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.845 DEBUG:boto:Data: 2023-10-11T15:00:33.845 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.846 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.847 DEBUG:boto:Params: {} 2023-10-11T15:00:33.847 DEBUG:boto:Token: None 2023-10-11T15:00:33.848 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:00:33.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0= 2023-10-11T15:00:33.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW68trB3nmUU3nay9PO/83N0Mn0=', 'Content-Length': '0'} 2023-10-11T15:00:33.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007775229df8949e72-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.851 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.852 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.852 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.853 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.853 DEBUG:boto:Method: GET 2023-10-11T15:00:33.854 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.854 DEBUG:boto:Data: 2023-10-11T15:00:33.854 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.855 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.856 DEBUG:boto:Params: {} 2023-10-11T15:00:33.856 DEBUG:boto:Token: None 2023-10-11T15:00:33.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401f1d0df4dd96c4-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.860 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.861 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.861 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.862 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.862 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.863 DEBUG:boto:Method: GET 2023-10-11T15:00:33.863 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.863 DEBUG:boto:Data: 2023-10-11T15:00:33.864 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.865 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.865 DEBUG:boto:Params: {} 2023-10-11T15:00:33.866 DEBUG:boto:Token: None 2023-10-11T15:00:33.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e97ef97b7867b61-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.910 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.911 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.912 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.912 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.912 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:00:33.913 DEBUG:boto:Data: 2023-10-11T15:00:33.913 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.914 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.914 DEBUG:boto:Params: {} 2023-10-11T15:00:33.915 DEBUG:boto:Token: None 2023-10-11T15:00:33.915 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3 2023-10-11T15:00:33.916 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3C7hDiiUmWUS7oaxDzqEZktYXqg= 2023-10-11T15:00:33.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3C7hDiiUmWUS7oaxDzqEZktYXqg=', 'Content-Length': '0'} 2023-10-11T15:00:33.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edf12d0fb1c9bb2e-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.919 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.919 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.919 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.920 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:00:33.920 DEBUG:boto:Data: 2023-10-11T15:00:33.921 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.921 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.922 DEBUG:boto:Params: {} 2023-10-11T15:00:33.922 DEBUG:boto:Token: None 2023-10-11T15:00:33.923 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3 2023-10-11T15:00:33.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3C7hDiiUmWUS7oaxDzqEZktYXqg= 2023-10-11T15:00:33.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3C7hDiiUmWUS7oaxDzqEZktYXqg=', 'Content-Length': '0'} 2023-10-11T15:00:33.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000855818049fc3931e-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:00:33.926 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.927 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.927 DEBUG:boto:Method: GET 2023-10-11T15:00:33.928 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:00:33.928 DEBUG:boto:Data: 2023-10-11T15:00:33.929 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.929 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.930 DEBUG:boto:Params: {} 2023-10-11T15:00:33.930 DEBUG:boto:Token: None 2023-10-11T15:00:33.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3 2023-10-11T15:00:33.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XyGp1mwihBMCGG4ac+m1gaoT1Ic= 2023-10-11T15:00:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XyGp1mwihBMCGG4ac+m1gaoT1Ic=', 'Content-Length': '0'} 2023-10-11T15:00:33.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ae53130ae6dbf68-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.934 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:00:33.935 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:00:33.935 DEBUG:boto:Method: GET 2023-10-11T15:00:33.935 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:00:33.936 DEBUG:boto:Data: 2023-10-11T15:00:33.936 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.937 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.938 DEBUG:boto:Params: {} 2023-10-11T15:00:33.938 DEBUG:boto:Token: None 2023-10-11T15:00:33.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj3 2023-10-11T15:00:33.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XyGp1mwihBMCGG4ac+m1gaoT1Ic= 2023-10-11T15:00:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XyGp1mwihBMCGG4ac+m1gaoT1Ic=', 'Content-Length': '0'} 2023-10-11T15:00:33.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003011ee6e1231ea66-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.942 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.942 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.943 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.943 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.943 DEBUG:boto:Method: GET 2023-10-11T15:00:33.944 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.944 DEBUG:boto:Data: 2023-10-11T15:00:33.945 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.945 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.946 DEBUG:boto:Params: {} 2023-10-11T15:00:33.946 DEBUG:boto:Token: None 2023-10-11T15:00:33.947 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a6206dee9dac83b-006526b891-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.951 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.951 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:33.952 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:33.952 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.952 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.953 DEBUG:boto:Method: GET 2023-10-11T15:00:33.953 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:00:33.954 DEBUG:boto:Data: 2023-10-11T15:00:33.954 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.955 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.956 DEBUG:boto:Params: {} 2023-10-11T15:00:33.956 DEBUG:boto:Token: None 2023-10-11T15:00:33.957 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/ 2023-10-11T15:00:33.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk= 2023-10-11T15:00:33.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oo6idFEhdCpSvHH3jYbNXp0vYIk=', 'Content-Length': '0'} 2023-10-11T15:00:33.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037808535cf51dfd6-006526b891-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.960 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:33.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:33.961 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:33.962 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:33.962 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.963 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.963 DEBUG:boto:Method: GET 2023-10-11T15:00:33.964 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.964 DEBUG:boto:Data: 2023-10-11T15:00:33.964 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.965 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.966 DEBUG:boto:Params: {} 2023-10-11T15:00:33.966 DEBUG:boto:Token: None 2023-10-11T15:00:33.967 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.967 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Gl0GHq3jK50WbUXgHajOK9mJMA= 2023-10-11T15:00:33.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Gl0GHq3jK50WbUXgHajOK9mJMA=', 'Content-Length': '0'} 2023-10-11T15:00:33.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071570a7acd87b50e-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.970 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:33.970 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:33.971 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.971 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.972 DEBUG:boto:Method: GET 2023-10-11T15:00:33.972 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.972 DEBUG:boto:Data: 2023-10-11T15:00:33.973 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.974 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.974 DEBUG:boto:Params: {} 2023-10-11T15:00:33.975 DEBUG:boto:Token: None 2023-10-11T15:00:33.975 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Gl0GHq3jK50WbUXgHajOK9mJMA= 2023-10-11T15:00:33.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Gl0GHq3jK50WbUXgHajOK9mJMA=', 'Content-Length': '0'} 2023-10-11T15:00:33.978 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000152ed4249618576d-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.979 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:33.979 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:33.980 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.980 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.980 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.981 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.981 DEBUG:boto:Data: 2023-10-11T15:00:33.982 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.982 DEBUG:boto:Port: 8001 2023-10-11T15:00:33.983 DEBUG:boto:Params: {} 2023-10-11T15:00:33.983 DEBUG:boto:Token: None 2023-10-11T15:00:33.984 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1JeLKt6xZwanRZIL2vBG1e5L8Ug= 2023-10-11T15:00:33.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1JeLKt6xZwanRZIL2vBG1e5L8Ug=', 'Content-Length': '0'} 2023-10-11T15:00:33.986 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a58e4753a9a978d-006526b891-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.987 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:33.987 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:33.988 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.988 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.988 DEBUG:boto:Method: HEAD 2023-10-11T15:00:33.989 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.989 DEBUG:boto:Data: 2023-10-11T15:00:33.990 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:33.990 DEBUG:boto:Port: 8000 2023-10-11T15:00:33.991 DEBUG:boto:Params: {} 2023-10-11T15:00:33.991 DEBUG:boto:Token: None 2023-10-11T15:00:33.992 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:33 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1JeLKt6xZwanRZIL2vBG1e5L8Ug= 2023-10-11T15:00:33.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1JeLKt6xZwanRZIL2vBG1e5L8Ug=', 'Content-Length': '0'} 2023-10-11T15:00:33.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fc59ce34e682680-006526b891-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:33.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:33.995 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:33.996 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:33.996 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.997 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.997 DEBUG:boto:Method: GET 2023-10-11T15:00:33.998 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:33.998 DEBUG:boto:Data: 2023-10-11T15:00:33.998 DEBUG:boto:Headers: {} 2023-10-11T15:00:33.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:33.999 DEBUG:boto:Port: 8001 2023-10-11T15:00:34.000 DEBUG:boto:Params: {} 2023-10-11T15:00:34.000 DEBUG:boto:Token: None 2023-10-11T15:00:34.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:34.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k+V2jjr7c+BObS92REUeULYNwdA= 2023-10-11T15:00:34.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k+V2jjr7c+BObS92REUeULYNwdA=', 'Content-Length': '0'} 2023-10-11T15:00:34.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f75e766057e56944-006526b892-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.046 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:34.047 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:34.047 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:34.048 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:00:34.048 DEBUG:boto:Method: GET 2023-10-11T15:00:34.049 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:34.049 DEBUG:boto:Data: 2023-10-11T15:00:34.049 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:34.050 DEBUG:boto:Port: 8000 2023-10-11T15:00:34.051 DEBUG:boto:Params: {} 2023-10-11T15:00:34.051 DEBUG:boto:Token: None 2023-10-11T15:00:34.052 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:00:34.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k+V2jjr7c+BObS92REUeULYNwdA= 2023-10-11T15:00:34.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k+V2jjr7c+BObS92REUeULYNwdA=', 'Content-Length': '0'} 2023-10-11T15:00:34.054 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3939a7250b6b685-006526b892-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.055 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:34.055 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:34.056 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.056 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.057 DEBUG:boto:Method: GET 2023-10-11T15:00:34.057 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.058 DEBUG:boto:Data: 2023-10-11T15:00:34.058 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:34.059 DEBUG:boto:Port: 8001 2023-10-11T15:00:34.059 DEBUG:boto:Params: {} 2023-10-11T15:00:34.060 DEBUG:boto:Token: None 2023-10-11T15:00:34.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/ 2023-10-11T15:00:34.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo= 2023-10-11T15:00:34.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo=', 'Content-Length': '0'} 2023-10-11T15:00:34.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f27d01870fee50e3-006526b892-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.064 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:34.064 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:34.065 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:34.065 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.066 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.066 DEBUG:boto:Method: GET 2023-10-11T15:00:34.067 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.067 DEBUG:boto:Data: 2023-10-11T15:00:34.067 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.068 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:34.068 DEBUG:boto:Port: 8000 2023-10-11T15:00:34.069 DEBUG:boto:Params: {} 2023-10-11T15:00:34.069 DEBUG:boto:Token: None 2023-10-11T15:00:34.070 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/ 2023-10-11T15:00:34.070 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo= 2023-10-11T15:00:34.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo=', 'Content-Length': '0'} 2023-10-11T15:00:34.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007499010affc4ae05-006526b892-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.114 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:34.115 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:34.115 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:34.116 DEBUG:boto:Method: HEAD 2023-10-11T15:00:34.116 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:00:34.117 DEBUG:boto:Data: 2023-10-11T15:00:34.117 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:34.118 DEBUG:boto:Port: 8001 2023-10-11T15:00:34.118 DEBUG:boto:Params: {} 2023-10-11T15:00:34.119 DEBUG:boto:Token: None 2023-10-11T15:00:34.119 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4 2023-10-11T15:00:34.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x3EYqyqUhLEgoWfVU3yaDBKJ4d4= 2023-10-11T15:00:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x3EYqyqUhLEgoWfVU3yaDBKJ4d4=', 'Content-Length': '0'} 2023-10-11T15:00:34.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001492cb95909cb077-006526b892-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.123 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:34.123 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:34.124 DEBUG:boto:Method: HEAD 2023-10-11T15:00:34.124 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:00:34.124 DEBUG:boto:Data: 2023-10-11T15:00:34.125 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.125 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:34.126 DEBUG:boto:Port: 8000 2023-10-11T15:00:34.126 DEBUG:boto:Params: {} 2023-10-11T15:00:34.127 DEBUG:boto:Token: None 2023-10-11T15:00:34.127 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4 2023-10-11T15:00:34.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x3EYqyqUhLEgoWfVU3yaDBKJ4d4= 2023-10-11T15:00:34.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x3EYqyqUhLEgoWfVU3yaDBKJ4d4=', 'Content-Length': '0'} 2023-10-11T15:00:34.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026a8a0ec771834d0-006526b892-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:00:34.131 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:34.131 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:34.132 DEBUG:boto:Method: GET 2023-10-11T15:00:34.132 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:00:34.133 DEBUG:boto:Data: 2023-10-11T15:00:34.133 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:34.134 DEBUG:boto:Port: 8001 2023-10-11T15:00:34.134 DEBUG:boto:Params: {} 2023-10-11T15:00:34.135 DEBUG:boto:Token: None 2023-10-11T15:00:34.135 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4 2023-10-11T15:00:34.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wTOwazWALOYP3+PJonW76qxxKks= 2023-10-11T15:00:34.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wTOwazWALOYP3+PJonW76qxxKks=', 'Content-Length': '0'} 2023-10-11T15:00:34.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005956fc375636b0c7-006526b892-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.139 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:00:34.139 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:00:34.140 DEBUG:boto:Method: GET 2023-10-11T15:00:34.140 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:00:34.140 DEBUG:boto:Data: 2023-10-11T15:00:34.141 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:34.142 DEBUG:boto:Port: 8000 2023-10-11T15:00:34.142 DEBUG:boto:Params: {} 2023-10-11T15:00:34.143 DEBUG:boto:Token: None 2023-10-11T15:00:34.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/obj4 2023-10-11T15:00:34.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wTOwazWALOYP3+PJonW76qxxKks= 2023-10-11T15:00:34.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wTOwazWALOYP3+PJonW76qxxKks=', 'Content-Length': '0'} 2023-10-11T15:00:34.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fdb9c6af6885708-006526b892-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.147 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:34.147 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:34.148 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.148 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.148 DEBUG:boto:Method: GET 2023-10-11T15:00:34.149 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.149 DEBUG:boto:Data: 2023-10-11T15:00:34.150 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:34.151 DEBUG:boto:Port: 8001 2023-10-11T15:00:34.151 DEBUG:boto:Params: {} 2023-10-11T15:00:34.151 DEBUG:boto:Token: None 2023-10-11T15:00:34.152 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/ 2023-10-11T15:00:34.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo= 2023-10-11T15:00:34.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo=', 'Content-Length': '0'} 2023-10-11T15:00:34.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc72e55844956392-006526b892-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.156 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:34.156 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:34.157 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:34.157 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.158 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.158 DEBUG:boto:Method: GET 2023-10-11T15:00:34.158 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:00:34.159 DEBUG:boto:Data: 2023-10-11T15:00:34.159 DEBUG:boto:Headers: {} 2023-10-11T15:00:34.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:34.160 DEBUG:boto:Port: 8000 2023-10-11T15:00:34.160 DEBUG:boto:Params: {} 2023-10-11T15:00:34.161 DEBUG:boto:Token: None 2023-10-11T15:00:34.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:00:34 GMT /dghqvd-49/ 2023-10-11T15:00:34.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo= 2023-10-11T15:00:34.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ah2W5Hd9XjXbsGcHCynR7UCILMo=', 'Content-Length': '0'} 2023-10-11T15:00:34.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7681c2b8a85d483-006526b892-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:00:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:34.165 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:00:34.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-49 zones={a2, a1} 2023-10-11T15:00:34.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:34.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:34.168 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:00:34.219 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:34.220 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:34.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dghqvd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:34.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dghqvd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:34.284 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dghqvd-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:00:34.334 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:34.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:34.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:34.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:34.400 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:00:34.448 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:34.449 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:34.477 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:00:34.488 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:00:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:00:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:00:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:00:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:00:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.491 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:00:34.491 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:00:34.491 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.491 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.492 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036434.383789_351.1", 2023-10-11T15:00:34.492 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:00:34.383789Z" 2023-10-11T15:00:34.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.492 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:00:34.493 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036434.383789_351.1'} 2023-10-11T15:00:34.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:00:34.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:34.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:34.495 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:00:34.544 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:34.545 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:34.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:34.569+0000 7f2382a07780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:00:34.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:34.570+0000 7f2382a07780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:00:34.585 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:00:34.585 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:00:34.586 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:00:34.586 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:00:34.586 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:00:34.586 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:00:34.586 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:34.587 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.587 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:00:34.587 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.587 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:00:34.587 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:34.588 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:34.588 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:00:34.588 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:34.588 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:00:34.589 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:34.589 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:00:34.589 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:34.589 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.589 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.590 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.590 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:00:34.590 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:34.590 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:34.590 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:00:34.591 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:34.591 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:34.591 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:34.591 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:00:34.591 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:34.592 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.592 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.592 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.592 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:00:34.592 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:34.593 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:34.593 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:00:34.593 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:34.593 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:34.593 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:34.594 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:00:34.594 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:34.594 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.594 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.595 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:34.595 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:00:34.595 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:34.595 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:34.595 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036371.310891_346.1", 2023-10-11T15:00:34.596 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:34.596 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:34.596 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:34.596 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:59:31.310891Z", 2023-10-11T15:00:34.596 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:34.597 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.597 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:34.597 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:00:34.597 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.597 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:00:34.598 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:00:34.598 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:00:34.598 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:34.598 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:00:34Z" 2023-10-11T15:00:34.598 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:00:34.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036371.310891_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:59:31.310891Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:00:34Z" } 2023-10-11T15:00:34.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036371.310891_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:59:31.310891Z', 'realm_epoch': 2}}] 2023-10-11T15:00:34.600 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036434.383789_351.1'} 2023-10-11T15:00:34.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036371.310891_346.1'} 2023-10-11T15:00:34.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697036434.383789_351.1 target=1_1697036371.310891_346.1 2023-10-11T15:00:39.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:39.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:00:39.603 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:00:39.660 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:39.660 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:39.693 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:39.690+0000 7fd7fe641780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:00:39.694 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:39.691+0000 7fd7fe641780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:00:39.708 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:00:39.708 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:00:39.708 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:00:39.708 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:00:39.709 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:00:39.709 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:00:39.709 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:39.709 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.709 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:00:39.710 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:39.710 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:00:39.710 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:39.710 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:39.710 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:00:39.711 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:39.711 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:00:39.711 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:39.711 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:00:39.711 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:39.712 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:39.712 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.712 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:39.712 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:00:39.712 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:39.713 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:00:39.714 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:39.714 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:39.714 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.714 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:39.714 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:00:39.715 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:39.715 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:39.715 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:00:39.715 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:39.715 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:39.716 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:39.716 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:00:39.716 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:39.716 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:39.716 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.717 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:00:39.717 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:00:39.717 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:00:39.717 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:00:39.717 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036434.383789_351.1", 2023-10-11T15:00:39.718 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:00:39.718 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:00:39.718 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:00:39.718 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:00:34.383789Z", 2023-10-11T15:00:39.718 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:00:39.719 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:00:39.720 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:00:39.720 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:00:39.720 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:00:39Z" 2023-10-11T15:00:39.720 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:00:39.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036434.383789_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:00:34.383789Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:00:39Z" } 2023-10-11T15:00:39.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036434.383789_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:00:34.383789Z', 'realm_epoch': 2}}] 2023-10-11T15:00:39.722 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036434.383789_351.1'} 2023-10-11T15:00:39.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036434.383789_351.1'} 2023-10-11T15:00:39.723 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:00:39.723 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:39.724 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:39.724 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.724 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:39.725 DEBUG:boto:Data: 2023-10-11T15:00:39.725 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.725 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.726 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.726 DEBUG:boto:Params: {} 2023-10-11T15:00:39.727 DEBUG:boto:Token: None 2023-10-11T15:00:39.727 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/ 2023-10-11T15:00:39.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y= 2023-10-11T15:00:39.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y=', 'Content-Length': '0'} 2023-10-11T15:00:39.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009d91fde74125d91-006526b897-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.731 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:00:39.731 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:00:39.732 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.732 DEBUG:boto:Path: /dghqvd-48/obj5 2023-10-11T15:00:39.732 DEBUG:boto:Data: 2023-10-11T15:00:39.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.733 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.733 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.734 DEBUG:boto:Params: {} 2023-10-11T15:00:39.734 DEBUG:boto:Token: None 2023-10-11T15:00:39.735 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/obj5 2023-10-11T15:00:39.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i50KTSKf2dXEBHL8urqugfgQqRc= 2023-10-11T15:00:39.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i50KTSKf2dXEBHL8urqugfgQqRc='} 2023-10-11T15:00:39.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b217e023b2a742c-006526b897-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.779 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:39.779 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:39.780 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.780 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:39.780 DEBUG:boto:Data: 2023-10-11T15:00:39.781 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.782 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.782 DEBUG:boto:Params: {} 2023-10-11T15:00:39.782 DEBUG:boto:Token: None 2023-10-11T15:00:39.783 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/ 2023-10-11T15:00:39.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y= 2023-10-11T15:00:39.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y=', 'Content-Length': '0'} 2023-10-11T15:00:39.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008974b7be75c904e-006526b897-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.786 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:00:39.787 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:00:39.787 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.788 DEBUG:boto:Path: /dghqvd-48/obj6 2023-10-11T15:00:39.788 DEBUG:boto:Data: 2023-10-11T15:00:39.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.789 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.790 DEBUG:boto:Params: {} 2023-10-11T15:00:39.790 DEBUG:boto:Token: None 2023-10-11T15:00:39.791 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/obj6 2023-10-11T15:00:39.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xCoXqQ3rPV8BJEISM5z1zcnhdpY= 2023-10-11T15:00:39.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xCoXqQ3rPV8BJEISM5z1zcnhdpY='} 2023-10-11T15:00:39.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000529e80037d5426da-006526b897-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.797 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:39.797 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:39.798 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.798 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:39.798 DEBUG:boto:Data: 2023-10-11T15:00:39.799 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.800 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.800 DEBUG:boto:Params: {} 2023-10-11T15:00:39.800 DEBUG:boto:Token: None 2023-10-11T15:00:39.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/ 2023-10-11T15:00:39.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y= 2023-10-11T15:00:39.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y=', 'Content-Length': '0'} 2023-10-11T15:00:39.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0ba8d808c586b1c-006526b897-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.804 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:00:39.805 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:00:39.805 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.805 DEBUG:boto:Path: /dghqvd-48/obj7 2023-10-11T15:00:39.806 DEBUG:boto:Data: 2023-10-11T15:00:39.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.807 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.807 DEBUG:boto:Params: {} 2023-10-11T15:00:39.808 DEBUG:boto:Token: None 2023-10-11T15:00:39.808 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/obj7 2023-10-11T15:00:39.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yTAcDx1fLuQpoFtFTqIBVmPDf1A= 2023-10-11T15:00:39.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yTAcDx1fLuQpoFtFTqIBVmPDf1A='} 2023-10-11T15:00:39.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ff6be7b00cca4a3-006526b897-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.814 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:00:39.815 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:00:39.815 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.816 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:00:39.816 DEBUG:boto:Data: 2023-10-11T15:00:39.816 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.817 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.818 DEBUG:boto:Params: {} 2023-10-11T15:00:39.818 DEBUG:boto:Token: None 2023-10-11T15:00:39.818 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/ 2023-10-11T15:00:39.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y= 2023-10-11T15:00:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FV9+qBjkckB/2Ea27jDk3mrak2Y=', 'Content-Length': '0'} 2023-10-11T15:00:39.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0eed7cb3d79f3f1-006526b897-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.822 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:00:39.823 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:00:39.823 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.823 DEBUG:boto:Path: /dghqvd-48/obj8 2023-10-11T15:00:39.824 DEBUG:boto:Data: 2023-10-11T15:00:39.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:00:39.825 DEBUG:boto:Port: 8000 2023-10-11T15:00:39.825 DEBUG:boto:Params: {} 2023-10-11T15:00:39.826 DEBUG:boto:Token: None 2023-10-11T15:00:39.826 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-48/obj8 2023-10-11T15:00:39.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0T+BSkAVxMTD0wAP66FhePiETDo= 2023-10-11T15:00:39.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0T+BSkAVxMTD0wAP66FhePiETDo='} 2023-10-11T15:00:39.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df734644944b14a3-006526b897-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.875 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:39.875 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:39.876 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.876 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:39.877 DEBUG:boto:Data: 2023-10-11T15:00:39.877 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.878 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.878 DEBUG:boto:Params: {} 2023-10-11T15:00:39.879 DEBUG:boto:Token: None 2023-10-11T15:00:39.879 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/ 2023-10-11T15:00:39.880 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM= 2023-10-11T15:00:39.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM=', 'Content-Length': '0'} 2023-10-11T15:00:39.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c8975fcf71de73c-006526b897-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.883 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:00:39.884 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:00:39.884 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.884 DEBUG:boto:Path: /dghqvd-49/obj5 2023-10-11T15:00:39.885 DEBUG:boto:Data: 2023-10-11T15:00:39.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.886 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.886 DEBUG:boto:Params: {} 2023-10-11T15:00:39.887 DEBUG:boto:Token: None 2023-10-11T15:00:39.887 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/obj5 2023-10-11T15:00:39.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qC4kOgf/d7y7emZ7jifPqSqp5jM= 2023-10-11T15:00:39.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qC4kOgf/d7y7emZ7jifPqSqp5jM='} 2023-10-11T15:00:39.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ec6799f159c383c-006526b897-1679-a2'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.931 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:39.931 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:39.932 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.932 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:39.933 DEBUG:boto:Data: 2023-10-11T15:00:39.933 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.934 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.934 DEBUG:boto:Params: {} 2023-10-11T15:00:39.935 DEBUG:boto:Token: None 2023-10-11T15:00:39.935 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/ 2023-10-11T15:00:39.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM= 2023-10-11T15:00:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM=', 'Content-Length': '0'} 2023-10-11T15:00:39.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ac14d4e266fe149-006526b897-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.939 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:00:39.939 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:00:39.940 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.940 DEBUG:boto:Path: /dghqvd-49/obj6 2023-10-11T15:00:39.940 DEBUG:boto:Data: 2023-10-11T15:00:39.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.942 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.942 DEBUG:boto:Params: {} 2023-10-11T15:00:39.943 DEBUG:boto:Token: None 2023-10-11T15:00:39.943 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/obj6 2023-10-11T15:00:39.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xBfpsypE6BJxmgx/PyrbXUBvYnQ= 2023-10-11T15:00:39.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xBfpsypE6BJxmgx/PyrbXUBvYnQ='} 2023-10-11T15:00:39.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e8a7174ac140bcc-006526b897-1679-a2'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.948 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:39.949 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:39.949 DEBUG:boto:Method: HEAD 2023-10-11T15:00:39.950 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:39.950 DEBUG:boto:Data: 2023-10-11T15:00:39.950 DEBUG:boto:Headers: {} 2023-10-11T15:00:39.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.951 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.952 DEBUG:boto:Params: {} 2023-10-11T15:00:39.952 DEBUG:boto:Token: None 2023-10-11T15:00:39.953 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/ 2023-10-11T15:00:39.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM= 2023-10-11T15:00:39.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DxhwhXlF8yvz1jGXPzZx7Yr5EsM=', 'Content-Length': '0'} 2023-10-11T15:00:39.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0939e5674b478c1-006526b897-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:39.956 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:00:39.957 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:00:39.957 DEBUG:boto:Method: PUT 2023-10-11T15:00:39.958 DEBUG:boto:Path: /dghqvd-49/obj7 2023-10-11T15:00:39.958 DEBUG:boto:Data: 2023-10-11T15:00:39.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:39.959 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:39.959 DEBUG:boto:Port: 8001 2023-10-11T15:00:39.960 DEBUG:boto:Params: {} 2023-10-11T15:00:39.960 DEBUG:boto:Token: None 2023-10-11T15:00:39.960 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:39 GMT /dghqvd-49/obj7 2023-10-11T15:00:39.961 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8tTa2iKxwVxZQRhgJfAiASL+d7k= 2023-10-11T15:00:39.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8tTa2iKxwVxZQRhgJfAiASL+d7k='} 2023-10-11T15:00:40.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009021af9466981dcf-006526b897-1679-a2'), ('Date', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:40.007 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:00:40.007 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:00:40.008 DEBUG:boto:Method: HEAD 2023-10-11T15:00:40.008 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:00:40.008 DEBUG:boto:Data: 2023-10-11T15:00:40.009 DEBUG:boto:Headers: {} 2023-10-11T15:00:40.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:40.009 DEBUG:boto:Port: 8001 2023-10-11T15:00:40.010 DEBUG:boto:Params: {} 2023-10-11T15:00:40.010 DEBUG:boto:Token: None 2023-10-11T15:00:40.011 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:00:40 GMT /dghqvd-49/ 2023-10-11T15:00:40.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5P1xOZav8jw4Moa8Xl5QOCy3IYU= 2023-10-11T15:00:40.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:00:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5P1xOZav8jw4Moa8Xl5QOCy3IYU=', 'Content-Length': '0'} 2023-10-11T15:00:40.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e8052327b27fb08-006526b898-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:40.014 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:00:40.015 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:00:40.015 DEBUG:boto:Method: PUT 2023-10-11T15:00:40.016 DEBUG:boto:Path: /dghqvd-49/obj8 2023-10-11T15:00:40.016 DEBUG:boto:Data: 2023-10-11T15:00:40.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:00:40.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:00:40.017 DEBUG:boto:Port: 8001 2023-10-11T15:00:40.018 DEBUG:boto:Params: {} 2023-10-11T15:00:40.018 DEBUG:boto:Token: None 2023-10-11T15:00:40.018 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:00:40 GMT /dghqvd-49/obj8 2023-10-11T15:00:40.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T1SD07kBVJtDFlyM6vFu4HofckI= 2023-10-11T15:00:40.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:00:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T1SD07kBVJtDFlyM6vFu4HofckI='} 2023-10-11T15:00:40.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a3bcee2f028b5a5-006526b898-1679-a2'), ('Date', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:00:40.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dghqvd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:40.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dghqvd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:40.025 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dghqvd-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:00:40.077 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:40.078 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:40.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dghqvd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:40.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dghqvd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:00:40.142 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dghqvd-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:00:40.192 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:40.192 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:40.259 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-48 2023-10-11T15:00:40.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:40.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:40.260 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:00:40.318 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:40.318 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:40.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:40.350+0000 7f29753e5780 1 waiting for incremental sync to catch up: 2023-10-11T15:00:40.353 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.28.6, , 00000000001.30.6, , 00000000001.26.6, , , 00000000001.48.6, , , ] 2023-10-11T15:00:40.353 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.32.1, 00000000002.18.1, 00000000004.37.1, 00000000002.33.1, 00000000004.30.1, 00000000002.58.1, 00000000002.35.1, 00000000004.52.1, 00000000002.17.1, 00000000002.34.1, 00000000002.28.1] 2023-10-11T15:00:45.354 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:45.352+0000 7f29753e5780 1 waiting for incremental sync to catch up: 2023-10-11T15:00:45.354 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.28.6, , 00000000001.30.6, , 00000000001.26.6, , , 00000000001.48.6, , , ] 2023-10-11T15:00:45.355 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.32.1, 00000000002.18.1, 00000000004.37.1, 00000000002.33.1, 00000000004.30.1, 00000000002.58.1, 00000000002.35.1, 00000000004.52.1, 00000000002.17.1, 00000000002.34.1, 00000000002.28.1] 2023-10-11T15:00:50.355 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:50.353+0000 7f29753e5780 1 bucket sync caught up with source: 2023-10-11T15:00:50.356 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.32.1, 00000000002.18.1, 00000000004.37.1, 00000000002.33.1, 00000000004.30.1, 00000000002.58.1, 00000000002.35.1, 00000000004.52.1, 00000000002.17.1, 00000000002.34.1, 00000000002.28.1] 2023-10-11T15:00:50.356 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.32.1, 00000000002.18.1, 00000000004.37.1, 00000000002.33.1, 00000000004.30.1, 00000000002.58.1, 00000000002.35.1, 00000000004.52.1, 00000000002.17.1, 00000000002.34.1, 00000000002.28.1] 2023-10-11T15:00:50.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:50.353+0000 7f29753e5780 0 bucket checkpoint complete 2023-10-11T15:00:50.372 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-48 2023-10-11T15:00:50.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:50.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:00:50.373 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:00:50.437 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:00:50.438 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:00:50.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:50.471+0000 7f239aecf780 1 waiting to reach incremental sync.. 2023-10-11T15:00:55.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:00:55.471+0000 7f239aecf780 1 waiting to reach incremental sync.. 2023-10-11T15:01:00.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:00.472+0000 7f239aecf780 1 waiting to reach incremental sync.. 2023-10-11T15:01:05.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:05.472+0000 7f239aecf780 1 waiting to reach incremental sync.. 2023-10-11T15:01:10.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:10.474+0000 7f239aecf780 1 bucket sync caught up with source: 2023-10-11T15:01:10.477 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.235.1, 00000000002.137.1, 00000000004.174.1, 00000000002.83.1, 00000000004.172.1, 00000000002.236.1, 00000000002.258.1, 00000000004.281.1, 00000000002.120.1, 00000000002.160.1, 00000000002.238.1] 2023-10-11T15:01:10.477 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.235.1, 00000000002.137.1, 00000000004.174.1, 00000000002.83.1, 00000000004.172.1, 00000000002.236.1, 00000000002.258.1, 00000000004.281.1, 00000000002.120.1, 00000000002.160.1, 00000000002.238.1] 2023-10-11T15:01:10.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:10.474+0000 7f239aecf780 0 bucket checkpoint complete 2023-10-11T15:01:10.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-48 zones={a2, a1} 2023-10-11T15:01:10.492 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.492 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.493 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.493 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:01:10.494 DEBUG:boto:Data: 2023-10-11T15:01:10.494 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.495 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.495 DEBUG:boto:Params: {} 2023-10-11T15:01:10.496 DEBUG:boto:Token: None 2023-10-11T15:01:10.496 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Iu6yEsIfeJiESzzUBbOR0yirNQw= 2023-10-11T15:01:10.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Iu6yEsIfeJiESzzUBbOR0yirNQw=', 'Content-Length': '0'} 2023-10-11T15:01:10.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000827c8656f0bf69e3-006526b8b6-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.501 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.501 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.502 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.502 DEBUG:boto:Path: /dghqvd-48/ 2023-10-11T15:01:10.503 DEBUG:boto:Data: 2023-10-11T15:01:10.503 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.503 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.504 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.504 DEBUG:boto:Params: {} 2023-10-11T15:01:10.505 DEBUG:boto:Token: None 2023-10-11T15:01:10.505 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Iu6yEsIfeJiESzzUBbOR0yirNQw= 2023-10-11T15:01:10.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Iu6yEsIfeJiESzzUBbOR0yirNQw=', 'Content-Length': '0'} 2023-10-11T15:01:10.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092174f5228564518-006526b8b6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:01:10.510 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.510 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.510 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.511 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.511 DEBUG:boto:Method: GET 2023-10-11T15:01:10.512 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.512 DEBUG:boto:Data: 2023-10-11T15:01:10.513 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.513 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.514 DEBUG:boto:Params: {} 2023-10-11T15:01:10.514 DEBUG:boto:Token: None 2023-10-11T15:01:10.515 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/?versions 2023-10-11T15:01:10.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs= 2023-10-11T15:01:10.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs=', 'Content-Length': '0'} 2023-10-11T15:01:10.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7f7762caf1f59d-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.519 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:01:10.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:01:10.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:01:10.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:01:10.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T15:01:10.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T15:01:10.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T15:01:10.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T15:01:10.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:01:10.523 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.524 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.524 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.525 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.525 DEBUG:boto:Method: GET 2023-10-11T15:01:10.526 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.526 DEBUG:boto:Data: 2023-10-11T15:01:10.527 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.527 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.528 DEBUG:boto:Params: {} 2023-10-11T15:01:10.528 DEBUG:boto:Token: None 2023-10-11T15:01:10.529 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/?versions 2023-10-11T15:01:10.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs= 2023-10-11T15:01:10.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs=', 'Content-Length': '0'} 2023-10-11T15:01:10.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c346a5f6995c758b-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.533 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:01:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:01:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:01:10.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:01:10.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T15:01:10.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T15:01:10.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T15:01:10.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T15:01:10.537 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.537 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.537 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.538 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.538 DEBUG:boto:Method: GET 2023-10-11T15:01:10.539 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.539 DEBUG:boto:Data: 2023-10-11T15:01:10.540 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.540 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.541 DEBUG:boto:Params: {} 2023-10-11T15:01:10.541 DEBUG:boto:Token: None 2023-10-11T15:01:10.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/?versions 2023-10-11T15:01:10.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs= 2023-10-11T15:01:10.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs=', 'Content-Length': '0'} 2023-10-11T15:01:10.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb25a1d15ab1c836-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.545 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.546 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.547 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.547 DEBUG:boto:path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.548 DEBUG:boto:auth_path=/dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.548 DEBUG:boto:Method: GET 2023-10-11T15:01:10.548 DEBUG:boto:Path: /dghqvd-48/?versions&max-keys=999 2023-10-11T15:01:10.549 DEBUG:boto:Data: 2023-10-11T15:01:10.549 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.550 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.550 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.550 DEBUG:boto:Params: {} 2023-10-11T15:01:10.551 DEBUG:boto:Token: None 2023-10-11T15:01:10.551 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/?versions 2023-10-11T15:01:10.552 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs= 2023-10-11T15:01:10.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rWyZ7It191VrJGxNEMzKKEvULqs=', 'Content-Length': '0'} 2023-10-11T15:01:10.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b7e0ea3d3ef7b70-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.555 DEBUG:boto:b'dghqvd-48999falseobj1nulltrue2023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:10.557 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.557 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.558 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.558 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.558 DEBUG:boto:Method: GET 2023-10-11T15:01:10.559 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.559 DEBUG:boto:Data: 2023-10-11T15:01:10.560 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.560 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.561 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.561 DEBUG:boto:Params: {} 2023-10-11T15:01:10.561 DEBUG:boto:Token: None 2023-10-11T15:01:10.562 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.562 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww= 2023-10-11T15:01:10.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww=', 'Content-Length': '0'} 2023-10-11T15:01:10.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e76eee05a9e15d9d-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.566 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.566 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.566 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.567 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.567 DEBUG:boto:Method: GET 2023-10-11T15:01:10.568 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.568 DEBUG:boto:Data: 2023-10-11T15:01:10.569 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.569 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.570 DEBUG:boto:Params: {} 2023-10-11T15:01:10.570 DEBUG:boto:Token: None 2023-10-11T15:01:10.571 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww= 2023-10-11T15:01:10.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww=', 'Content-Length': '0'} 2023-10-11T15:01:10.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edf27b335d26bd98-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.574 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.575 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.575 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.576 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.576 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.576 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.577 DEBUG:boto:Data: 2023-10-11T15:01:10.577 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.578 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.578 DEBUG:boto:Params: {} 2023-10-11T15:01:10.579 DEBUG:boto:Token: None 2023-10-11T15:01:10.579 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Aq2eduC3rv7W1Jlzas5ACDQdhI= 2023-10-11T15:01:10.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Aq2eduC3rv7W1Jlzas5ACDQdhI=', 'Content-Length': '0'} 2023-10-11T15:01:10.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036be18da99abba1b-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.583 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.583 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.583 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.584 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.584 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.585 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.585 DEBUG:boto:Data: 2023-10-11T15:01:10.586 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.586 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.587 DEBUG:boto:Params: {} 2023-10-11T15:01:10.587 DEBUG:boto:Token: None 2023-10-11T15:01:10.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Aq2eduC3rv7W1Jlzas5ACDQdhI= 2023-10-11T15:01:10.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Aq2eduC3rv7W1Jlzas5ACDQdhI=', 'Content-Length': '0'} 2023-10-11T15:01:10.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032b6db86bb324157-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:10.591 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.592 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.592 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.593 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.593 DEBUG:boto:Method: GET 2023-10-11T15:01:10.593 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.594 DEBUG:boto:Data: 2023-10-11T15:01:10.594 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.595 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.596 DEBUG:boto:Params: {} 2023-10-11T15:01:10.596 DEBUG:boto:Token: None 2023-10-11T15:01:10.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww= 2023-10-11T15:01:10.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww=', 'Content-Length': '0'} 2023-10-11T15:01:10.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1c85f2acca0f4eb-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.600 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.600 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.601 DEBUG:boto:path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.601 DEBUG:boto:auth_path=/dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.601 DEBUG:boto:Method: GET 2023-10-11T15:01:10.602 DEBUG:boto:Path: /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.602 DEBUG:boto:Data: 2023-10-11T15:01:10.603 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.603 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.604 DEBUG:boto:Params: {} 2023-10-11T15:01:10.604 DEBUG:boto:Token: None 2023-10-11T15:01:10.605 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1?versionId=null 2023-10-11T15:01:10.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww= 2023-10-11T15:01:10.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HKrobJfrXf5yGOadfKrTXRKW5Ww=', 'Content-Length': '0'} 2023-10-11T15:01:10.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029ae447c396bede4-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.608 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.609 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.609 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.610 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.610 DEBUG:boto:Method: GET 2023-10-11T15:01:10.611 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.611 DEBUG:boto:Data: 2023-10-11T15:01:10.611 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.612 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.613 DEBUG:boto:Params: {} 2023-10-11T15:01:10.613 DEBUG:boto:Token: None 2023-10-11T15:01:10.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b5adc92875d5e2-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.658 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.659 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.659 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.660 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.660 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.661 DEBUG:boto:Method: GET 2023-10-11T15:01:10.661 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.661 DEBUG:boto:Data: 2023-10-11T15:01:10.662 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.663 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.663 DEBUG:boto:Params: {} 2023-10-11T15:01:10.664 DEBUG:boto:Token: None 2023-10-11T15:01:10.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0f930640bd923b-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.668 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.669 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.669 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.670 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.670 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:01:10.671 DEBUG:boto:Data: 2023-10-11T15:01:10.671 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.672 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.672 DEBUG:boto:Params: {} 2023-10-11T15:01:10.673 DEBUG:boto:Token: None 2023-10-11T15:01:10.673 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1 2023-10-11T15:01:10.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8/YsoItaSVxKaQuAV4a81GDbvm0= 2023-10-11T15:01:10.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8/YsoItaSVxKaQuAV4a81GDbvm0=', 'Content-Length': '0'} 2023-10-11T15:01:10.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bb9b1ef89bc6269-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.676 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.677 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.677 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.678 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:01:10.678 DEBUG:boto:Data: 2023-10-11T15:01:10.678 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.679 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.680 DEBUG:boto:Params: {} 2023-10-11T15:01:10.680 DEBUG:boto:Token: None 2023-10-11T15:01:10.680 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1 2023-10-11T15:01:10.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8/YsoItaSVxKaQuAV4a81GDbvm0= 2023-10-11T15:01:10.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8/YsoItaSVxKaQuAV4a81GDbvm0=', 'Content-Length': '0'} 2023-10-11T15:01:10.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067993319d7708460-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:10.684 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.684 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.685 DEBUG:boto:Method: GET 2023-10-11T15:01:10.685 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:01:10.686 DEBUG:boto:Data: 2023-10-11T15:01:10.686 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.687 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.687 DEBUG:boto:Params: {} 2023-10-11T15:01:10.688 DEBUG:boto:Token: None 2023-10-11T15:01:10.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1 2023-10-11T15:01:10.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rRwPQo7MnRYBop/+aXzE2MoPxOw= 2023-10-11T15:01:10.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rRwPQo7MnRYBop/+aXzE2MoPxOw=', 'Content-Length': '0'} 2023-10-11T15:01:10.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed5c49e7d682daab-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.692 DEBUG:boto:path=/dghqvd-48/obj1 2023-10-11T15:01:10.692 DEBUG:boto:auth_path=/dghqvd-48/obj1 2023-10-11T15:01:10.693 DEBUG:boto:Method: GET 2023-10-11T15:01:10.693 DEBUG:boto:Path: /dghqvd-48/obj1 2023-10-11T15:01:10.693 DEBUG:boto:Data: 2023-10-11T15:01:10.694 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.695 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.695 DEBUG:boto:Params: {} 2023-10-11T15:01:10.696 DEBUG:boto:Token: None 2023-10-11T15:01:10.696 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj1 2023-10-11T15:01:10.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rRwPQo7MnRYBop/+aXzE2MoPxOw= 2023-10-11T15:01:10.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rRwPQo7MnRYBop/+aXzE2MoPxOw=', 'Content-Length': '0'} 2023-10-11T15:01:10.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fefa206832e6af62-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.742 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.743 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.743 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.743 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.744 DEBUG:boto:Method: GET 2023-10-11T15:01:10.744 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.745 DEBUG:boto:Data: 2023-10-11T15:01:10.745 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.746 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.746 DEBUG:boto:Params: {} 2023-10-11T15:01:10.747 DEBUG:boto:Token: None 2023-10-11T15:01:10.747 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000739fbe26aa7c6978-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.751 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.752 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.752 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.753 DEBUG:boto:path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.753 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.753 DEBUG:boto:Method: GET 2023-10-11T15:01:10.754 DEBUG:boto:Path: /dghqvd-48/?prefix=obj1 2023-10-11T15:01:10.754 DEBUG:boto:Data: 2023-10-11T15:01:10.755 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.756 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.756 DEBUG:boto:Params: {} 2023-10-11T15:01:10.757 DEBUG:boto:Token: None 2023-10-11T15:01:10.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7acca5b038de65-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.761 DEBUG:boto:b'dghqvd-48obj11000falseobj12023-10-11T14:59:31.336Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:10.762 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.762 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.763 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.763 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.763 DEBUG:boto:Method: GET 2023-10-11T15:01:10.764 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.764 DEBUG:boto:Data: 2023-10-11T15:01:10.765 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.766 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.766 DEBUG:boto:Params: {} 2023-10-11T15:01:10.767 DEBUG:boto:Token: None 2023-10-11T15:01:10.767 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU= 2023-10-11T15:01:10.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU=', 'Content-Length': '0'} 2023-10-11T15:01:10.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000694497b08774debd-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.770 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.771 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.771 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.772 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.772 DEBUG:boto:Method: GET 2023-10-11T15:01:10.773 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.773 DEBUG:boto:Data: 2023-10-11T15:01:10.773 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.774 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.775 DEBUG:boto:Params: {} 2023-10-11T15:01:10.775 DEBUG:boto:Token: None 2023-10-11T15:01:10.776 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU= 2023-10-11T15:01:10.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU=', 'Content-Length': '0'} 2023-10-11T15:01:10.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040d13356c9bf6395-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.779 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.780 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.780 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.780 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.781 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.781 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.782 DEBUG:boto:Data: 2023-10-11T15:01:10.782 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.783 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.783 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.783 DEBUG:boto:Params: {} 2023-10-11T15:01:10.784 DEBUG:boto:Token: None 2023-10-11T15:01:10.784 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.785 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ygc1grTn4Yl9UhzVr6BDGiYRSsA= 2023-10-11T15:01:10.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ygc1grTn4Yl9UhzVr6BDGiYRSsA=', 'Content-Length': '0'} 2023-10-11T15:01:10.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee551decf2eeb41c-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.788 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.788 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.789 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.789 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.789 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.790 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.790 DEBUG:boto:Data: 2023-10-11T15:01:10.791 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.792 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.792 DEBUG:boto:Params: {} 2023-10-11T15:01:10.793 DEBUG:boto:Token: None 2023-10-11T15:01:10.793 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ygc1grTn4Yl9UhzVr6BDGiYRSsA= 2023-10-11T15:01:10.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ygc1grTn4Yl9UhzVr6BDGiYRSsA=', 'Content-Length': '0'} 2023-10-11T15:01:10.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c35c21f49e0aa9a-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:10.797 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.797 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.797 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.798 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.798 DEBUG:boto:Method: GET 2023-10-11T15:01:10.799 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.799 DEBUG:boto:Data: 2023-10-11T15:01:10.800 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.800 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.801 DEBUG:boto:Params: {} 2023-10-11T15:01:10.801 DEBUG:boto:Token: None 2023-10-11T15:01:10.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU= 2023-10-11T15:01:10.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU=', 'Content-Length': '0'} 2023-10-11T15:01:10.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058078db34fb33147-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.846 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.847 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.847 DEBUG:boto:path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.848 DEBUG:boto:auth_path=/dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.848 DEBUG:boto:Method: GET 2023-10-11T15:01:10.848 DEBUG:boto:Path: /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.849 DEBUG:boto:Data: 2023-10-11T15:01:10.849 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.850 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.850 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.851 DEBUG:boto:Params: {} 2023-10-11T15:01:10.851 DEBUG:boto:Token: None 2023-10-11T15:01:10.852 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2?versionId=null 2023-10-11T15:01:10.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU= 2023-10-11T15:01:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lvJ7aCicB9eHXMZgZWXb1Lqi0LU=', 'Content-Length': '0'} 2023-10-11T15:01:10.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000193928bf8a60216-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.855 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.856 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.856 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.856 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.857 DEBUG:boto:Method: GET 2023-10-11T15:01:10.857 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.858 DEBUG:boto:Data: 2023-10-11T15:01:10.858 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.859 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.859 DEBUG:boto:Params: {} 2023-10-11T15:01:10.860 DEBUG:boto:Token: None 2023-10-11T15:01:10.860 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.861 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d34f932ccd6ea6a-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.866 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.867 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.867 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.868 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.868 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.868 DEBUG:boto:Method: GET 2023-10-11T15:01:10.869 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.869 DEBUG:boto:Data: 2023-10-11T15:01:10.870 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.871 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.871 DEBUG:boto:Params: {} 2023-10-11T15:01:10.871 DEBUG:boto:Token: None 2023-10-11T15:01:10.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.872 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d590968438d4bb5a-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.922 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.923 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.923 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.924 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.924 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:01:10.924 DEBUG:boto:Data: 2023-10-11T15:01:10.925 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.926 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.926 DEBUG:boto:Params: {} 2023-10-11T15:01:10.926 DEBUG:boto:Token: None 2023-10-11T15:01:10.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2 2023-10-11T15:01:10.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kbjbstDrpDJTg67PjjXj1IMxaD4= 2023-10-11T15:01:10.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kbjbstDrpDJTg67PjjXj1IMxaD4=', 'Content-Length': '0'} 2023-10-11T15:01:10.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5d3961b54fd44bd-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.930 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.931 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.931 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.932 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:01:10.932 DEBUG:boto:Data: 2023-10-11T15:01:10.933 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.934 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.934 DEBUG:boto:Params: {} 2023-10-11T15:01:10.935 DEBUG:boto:Token: None 2023-10-11T15:01:10.935 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2 2023-10-11T15:01:10.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kbjbstDrpDJTg67PjjXj1IMxaD4= 2023-10-11T15:01:10.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kbjbstDrpDJTg67PjjXj1IMxaD4=', 'Content-Length': '0'} 2023-10-11T15:01:10.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005131c1dbee97fe1-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:10.939 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.940 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.940 DEBUG:boto:Method: GET 2023-10-11T15:01:10.941 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:01:10.941 DEBUG:boto:Data: 2023-10-11T15:01:10.941 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.942 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.943 DEBUG:boto:Params: {} 2023-10-11T15:01:10.943 DEBUG:boto:Token: None 2023-10-11T15:01:10.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2 2023-10-11T15:01:10.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V3dgmFD2bhLZ/671Z3FfoDGCvQw= 2023-10-11T15:01:10.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V3dgmFD2bhLZ/671Z3FfoDGCvQw=', 'Content-Length': '0'} 2023-10-11T15:01:10.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2f610fb85d55623-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.948 DEBUG:boto:path=/dghqvd-48/obj2 2023-10-11T15:01:10.948 DEBUG:boto:auth_path=/dghqvd-48/obj2 2023-10-11T15:01:10.949 DEBUG:boto:Method: GET 2023-10-11T15:01:10.949 DEBUG:boto:Path: /dghqvd-48/obj2 2023-10-11T15:01:10.950 DEBUG:boto:Data: 2023-10-11T15:01:10.950 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.951 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.952 DEBUG:boto:Params: {} 2023-10-11T15:01:10.952 DEBUG:boto:Token: None 2023-10-11T15:01:10.953 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj2 2023-10-11T15:01:10.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V3dgmFD2bhLZ/671Z3FfoDGCvQw= 2023-10-11T15:01:10.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V3dgmFD2bhLZ/671Z3FfoDGCvQw=', 'Content-Length': '0'} 2023-10-11T15:01:10.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c7872b71da03aa-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.956 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.956 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.957 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.957 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.958 DEBUG:boto:Method: GET 2023-10-11T15:01:10.958 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.959 DEBUG:boto:Data: 2023-10-11T15:01:10.959 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.959 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.960 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.960 DEBUG:boto:Params: {} 2023-10-11T15:01:10.961 DEBUG:boto:Token: None 2023-10-11T15:01:10.961 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00f8401550cb54f-006526b8b6-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.965 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.966 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:10.966 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:10.966 DEBUG:boto:path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.967 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.967 DEBUG:boto:Method: GET 2023-10-11T15:01:10.968 DEBUG:boto:Path: /dghqvd-48/?prefix=obj2 2023-10-11T15:01:10.968 DEBUG:boto:Data: 2023-10-11T15:01:10.969 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.969 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.970 DEBUG:boto:Params: {} 2023-10-11T15:01:10.970 DEBUG:boto:Token: None 2023-10-11T15:01:10.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/ 2023-10-11T15:01:10.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU= 2023-10-11T15:01:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+YsO/xGzcNW28sfpERXNRqe9AU=', 'Content-Length': '0'} 2023-10-11T15:01:10.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae8bc988fc8c1cd4-006526b8b6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.975 DEBUG:boto:b'dghqvd-48obj21000falseobj22023-10-11T14:59:31.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:10.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:10.976 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:10.976 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:10.977 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.977 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.977 DEBUG:boto:Method: GET 2023-10-11T15:01:10.978 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.978 DEBUG:boto:Data: 2023-10-11T15:01:10.979 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.980 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.980 DEBUG:boto:Params: {} 2023-10-11T15:01:10.981 DEBUG:boto:Token: None 2023-10-11T15:01:10.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:da6YX6AJvz+8nRITxhRCJBv9p6Y= 2023-10-11T15:01:10.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:da6YX6AJvz+8nRITxhRCJBv9p6Y=', 'Content-Length': '0'} 2023-10-11T15:01:10.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001719e5ca6e136561-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.985 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:10.985 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:10.986 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.986 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.987 DEBUG:boto:Method: GET 2023-10-11T15:01:10.987 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.987 DEBUG:boto:Data: 2023-10-11T15:01:10.988 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:10.989 DEBUG:boto:Port: 8000 2023-10-11T15:01:10.989 DEBUG:boto:Params: {} 2023-10-11T15:01:10.990 DEBUG:boto:Token: None 2023-10-11T15:01:10.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:da6YX6AJvz+8nRITxhRCJBv9p6Y= 2023-10-11T15:01:10.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:da6YX6AJvz+8nRITxhRCJBv9p6Y=', 'Content-Length': '0'} 2023-10-11T15:01:10.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e405134a025a2d6a-006526b8b6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:10.994 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:10.994 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:10.995 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.995 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.996 DEBUG:boto:Method: HEAD 2023-10-11T15:01:10.996 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.996 DEBUG:boto:Data: 2023-10-11T15:01:10.997 DEBUG:boto:Headers: {} 2023-10-11T15:01:10.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:10.998 DEBUG:boto:Port: 8001 2023-10-11T15:01:10.998 DEBUG:boto:Params: {} 2023-10-11T15:01:10.998 DEBUG:boto:Token: None 2023-10-11T15:01:10.999 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:10 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:10.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oFqt1AK6rymQRQijq6z7DsDj6Q4= 2023-10-11T15:01:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oFqt1AK6rymQRQijq6z7DsDj6Q4=', 'Content-Length': '0'} 2023-10-11T15:01:11.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dced163f2f12051-006526b8b6-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.002 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.003 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.003 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.003 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.004 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.004 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.004 DEBUG:boto:Data: 2023-10-11T15:01:11.005 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.006 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.006 DEBUG:boto:Params: {} 2023-10-11T15:01:11.006 DEBUG:boto:Token: None 2023-10-11T15:01:11.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9TE/TCTFKGniSjEqOZeDahs7E9M= 2023-10-11T15:01:11.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9TE/TCTFKGniSjEqOZeDahs7E9M=', 'Content-Length': '0'} 2023-10-11T15:01:11.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e52bce3fa949cd7-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:11.011 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.011 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.012 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.012 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.012 DEBUG:boto:Method: GET 2023-10-11T15:01:11.013 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.013 DEBUG:boto:Data: 2023-10-11T15:01:11.014 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.015 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.015 DEBUG:boto:Params: {} 2023-10-11T15:01:11.016 DEBUG:boto:Token: None 2023-10-11T15:01:11.016 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kgRg6TYEKviQ3nwr6oWE4QJg7ms= 2023-10-11T15:01:11.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kgRg6TYEKviQ3nwr6oWE4QJg7ms=', 'Content-Length': '0'} 2023-10-11T15:01:11.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f30abee2c0e8926-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.020 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.020 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.021 DEBUG:boto:path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.021 DEBUG:boto:auth_path=/dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.021 DEBUG:boto:Method: GET 2023-10-11T15:01:11.022 DEBUG:boto:Path: /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.022 DEBUG:boto:Data: 2023-10-11T15:01:11.023 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.023 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.024 DEBUG:boto:Params: {} 2023-10-11T15:01:11.024 DEBUG:boto:Token: None 2023-10-11T15:01:11.025 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3?versionId=null 2023-10-11T15:01:11.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kgRg6TYEKviQ3nwr6oWE4QJg7ms= 2023-10-11T15:01:11.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kgRg6TYEKviQ3nwr6oWE4QJg7ms=', 'Content-Length': '0'} 2023-10-11T15:01:11.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004922d4ad76e79444-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.028 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.029 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.029 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.030 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.030 DEBUG:boto:Method: GET 2023-10-11T15:01:11.031 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.031 DEBUG:boto:Data: 2023-10-11T15:01:11.031 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.032 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.033 DEBUG:boto:Params: {} 2023-10-11T15:01:11.033 DEBUG:boto:Token: None 2023-10-11T15:01:11.034 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008989d4774992a804-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.078 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.079 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.079 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.080 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.080 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.081 DEBUG:boto:Method: GET 2023-10-11T15:01:11.081 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.082 DEBUG:boto:Data: 2023-10-11T15:01:11.082 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.083 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.083 DEBUG:boto:Params: {} 2023-10-11T15:01:11.084 DEBUG:boto:Token: None 2023-10-11T15:01:11.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cc751403066e22-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.088 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.089 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.089 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.089 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.090 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:01:11.090 DEBUG:boto:Data: 2023-10-11T15:01:11.090 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.091 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.092 DEBUG:boto:Params: {} 2023-10-11T15:01:11.092 DEBUG:boto:Token: None 2023-10-11T15:01:11.092 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3 2023-10-11T15:01:11.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6/K4h78Khb86wT6b6AokB5gx7+U= 2023-10-11T15:01:11.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6/K4h78Khb86wT6b6AokB5gx7+U=', 'Content-Length': '0'} 2023-10-11T15:01:11.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089b73f9b94874e40-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.096 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.096 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.096 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.097 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:01:11.097 DEBUG:boto:Data: 2023-10-11T15:01:11.098 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.099 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.099 DEBUG:boto:Params: {} 2023-10-11T15:01:11.099 DEBUG:boto:Token: None 2023-10-11T15:01:11.100 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3 2023-10-11T15:01:11.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6/K4h78Khb86wT6b6AokB5gx7+U= 2023-10-11T15:01:11.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6/K4h78Khb86wT6b6AokB5gx7+U=', 'Content-Length': '0'} 2023-10-11T15:01:11.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019bbf0509c4d4676-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:11.104 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.104 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.105 DEBUG:boto:Method: GET 2023-10-11T15:01:11.105 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:01:11.106 DEBUG:boto:Data: 2023-10-11T15:01:11.106 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.107 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.107 DEBUG:boto:Params: {} 2023-10-11T15:01:11.108 DEBUG:boto:Token: None 2023-10-11T15:01:11.108 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3 2023-10-11T15:01:11.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:veEPpcB463tbIggDT3hs7z/mw+A= 2023-10-11T15:01:11.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:veEPpcB463tbIggDT3hs7z/mw+A=', 'Content-Length': '0'} 2023-10-11T15:01:11.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049f4e845b3a8070b-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.112 DEBUG:boto:path=/dghqvd-48/obj3 2023-10-11T15:01:11.112 DEBUG:boto:auth_path=/dghqvd-48/obj3 2023-10-11T15:01:11.112 DEBUG:boto:Method: GET 2023-10-11T15:01:11.113 DEBUG:boto:Path: /dghqvd-48/obj3 2023-10-11T15:01:11.113 DEBUG:boto:Data: 2023-10-11T15:01:11.114 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.115 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.115 DEBUG:boto:Params: {} 2023-10-11T15:01:11.116 DEBUG:boto:Token: None 2023-10-11T15:01:11.116 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj3 2023-10-11T15:01:11.116 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:veEPpcB463tbIggDT3hs7z/mw+A= 2023-10-11T15:01:11.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:veEPpcB463tbIggDT3hs7z/mw+A=', 'Content-Length': '0'} 2023-10-11T15:01:11.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcea5e5f0a0dba05-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.162 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.163 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.163 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.163 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.164 DEBUG:boto:Method: GET 2023-10-11T15:01:11.164 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.165 DEBUG:boto:Data: 2023-10-11T15:01:11.165 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.166 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.167 DEBUG:boto:Params: {} 2023-10-11T15:01:11.167 DEBUG:boto:Token: None 2023-10-11T15:01:11.168 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb3c59f141b3281-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.172 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.172 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.173 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.173 DEBUG:boto:path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.174 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.174 DEBUG:boto:Method: GET 2023-10-11T15:01:11.174 DEBUG:boto:Path: /dghqvd-48/?prefix=obj3 2023-10-11T15:01:11.175 DEBUG:boto:Data: 2023-10-11T15:01:11.175 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.176 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.177 DEBUG:boto:Params: {} 2023-10-11T15:01:11.177 DEBUG:boto:Token: None 2023-10-11T15:01:11.177 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.178 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b12ddfc630c7c429-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.182 DEBUG:boto:b'dghqvd-48obj31000falseobj32023-10-11T14:59:31.412Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:11.183 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.183 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.184 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.184 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.185 DEBUG:boto:Method: GET 2023-10-11T15:01:11.185 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.186 DEBUG:boto:Data: 2023-10-11T15:01:11.186 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.187 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.187 DEBUG:boto:Params: {} 2023-10-11T15:01:11.188 DEBUG:boto:Token: None 2023-10-11T15:01:11.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig= 2023-10-11T15:01:11.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig=', 'Content-Length': '0'} 2023-10-11T15:01:11.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008559ccfd7fdd4e1a-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.234 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.235 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.235 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.236 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.236 DEBUG:boto:Method: GET 2023-10-11T15:01:11.236 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.237 DEBUG:boto:Data: 2023-10-11T15:01:11.237 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.238 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.238 DEBUG:boto:Params: {} 2023-10-11T15:01:11.239 DEBUG:boto:Token: None 2023-10-11T15:01:11.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig= 2023-10-11T15:01:11.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig=', 'Content-Length': '0'} 2023-10-11T15:01:11.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d10ecb44804e363a-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.243 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.243 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.244 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.244 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.245 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.245 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.245 DEBUG:boto:Data: 2023-10-11T15:01:11.246 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.247 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.247 DEBUG:boto:Params: {} 2023-10-11T15:01:11.248 DEBUG:boto:Token: None 2023-10-11T15:01:11.248 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RCE5NwdEk+klNd3ds4L1Rj9uTsE= 2023-10-11T15:01:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RCE5NwdEk+klNd3ds4L1Rj9uTsE=', 'Content-Length': '0'} 2023-10-11T15:01:11.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edf1c2e2fea044d2-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.252 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.252 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.252 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.253 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.253 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.254 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.254 DEBUG:boto:Data: 2023-10-11T15:01:11.254 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.255 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.256 DEBUG:boto:Params: {} 2023-10-11T15:01:11.256 DEBUG:boto:Token: None 2023-10-11T15:01:11.256 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RCE5NwdEk+klNd3ds4L1Rj9uTsE= 2023-10-11T15:01:11.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RCE5NwdEk+klNd3ds4L1Rj9uTsE=', 'Content-Length': '0'} 2023-10-11T15:01:11.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8da74c098e80220-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:11.260 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.260 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.261 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.261 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.261 DEBUG:boto:Method: GET 2023-10-11T15:01:11.262 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.262 DEBUG:boto:Data: 2023-10-11T15:01:11.263 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.263 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.263 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.264 DEBUG:boto:Params: {} 2023-10-11T15:01:11.264 DEBUG:boto:Token: None 2023-10-11T15:01:11.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig= 2023-10-11T15:01:11.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig=', 'Content-Length': '0'} 2023-10-11T15:01:11.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052bb4ae9417a7f74-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.268 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.269 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.269 DEBUG:boto:path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.270 DEBUG:boto:auth_path=/dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.270 DEBUG:boto:Method: GET 2023-10-11T15:01:11.271 DEBUG:boto:Path: /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.271 DEBUG:boto:Data: 2023-10-11T15:01:11.271 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.272 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.273 DEBUG:boto:Params: {} 2023-10-11T15:01:11.273 DEBUG:boto:Token: None 2023-10-11T15:01:11.274 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4?versionId=null 2023-10-11T15:01:11.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig= 2023-10-11T15:01:11.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQIlqGfjjcFy0Yog82dY3VbBIig=', 'Content-Length': '0'} 2023-10-11T15:01:11.279 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c903d923133d0bcb-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.322 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.323 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.323 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.324 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.324 DEBUG:boto:Method: GET 2023-10-11T15:01:11.325 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.325 DEBUG:boto:Data: 2023-10-11T15:01:11.326 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.326 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.327 DEBUG:boto:Params: {} 2023-10-11T15:01:11.327 DEBUG:boto:Token: None 2023-10-11T15:01:11.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078a4a01c1d86a1bb-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.332 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.333 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.333 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.334 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.334 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.334 DEBUG:boto:Method: GET 2023-10-11T15:01:11.335 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.335 DEBUG:boto:Data: 2023-10-11T15:01:11.336 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.336 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.337 DEBUG:boto:Params: {} 2023-10-11T15:01:11.337 DEBUG:boto:Token: None 2023-10-11T15:01:11.338 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2aaf53f60edcb6f-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.342 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.343 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.343 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.343 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.344 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:01:11.344 DEBUG:boto:Data: 2023-10-11T15:01:11.345 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.345 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.346 DEBUG:boto:Params: {} 2023-10-11T15:01:11.346 DEBUG:boto:Token: None 2023-10-11T15:01:11.347 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4 2023-10-11T15:01:11.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KcpqMabFIoDjfKYug7yPbD8ZhH0= 2023-10-11T15:01:11.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KcpqMabFIoDjfKYug7yPbD8ZhH0=', 'Content-Length': '0'} 2023-10-11T15:01:11.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006be02dbe66376b76-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.350 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.351 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.351 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.351 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:01:11.352 DEBUG:boto:Data: 2023-10-11T15:01:11.352 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.353 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.353 DEBUG:boto:Params: {} 2023-10-11T15:01:11.354 DEBUG:boto:Token: None 2023-10-11T15:01:11.354 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4 2023-10-11T15:01:11.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KcpqMabFIoDjfKYug7yPbD8ZhH0= 2023-10-11T15:01:11.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KcpqMabFIoDjfKYug7yPbD8ZhH0=', 'Content-Length': '0'} 2023-10-11T15:01:11.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004230e929a974ecfc-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:11.358 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.359 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.359 DEBUG:boto:Method: GET 2023-10-11T15:01:11.359 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:01:11.360 DEBUG:boto:Data: 2023-10-11T15:01:11.360 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.361 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.362 DEBUG:boto:Params: {} 2023-10-11T15:01:11.362 DEBUG:boto:Token: None 2023-10-11T15:01:11.362 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4 2023-10-11T15:01:11.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:88bEq3nD4iR4lCmTqOT+lCnZkI4= 2023-10-11T15:01:11.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:88bEq3nD4iR4lCmTqOT+lCnZkI4=', 'Content-Length': '0'} 2023-10-11T15:01:11.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002794c28a908dc76e-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.406 DEBUG:boto:path=/dghqvd-48/obj4 2023-10-11T15:01:11.407 DEBUG:boto:auth_path=/dghqvd-48/obj4 2023-10-11T15:01:11.407 DEBUG:boto:Method: GET 2023-10-11T15:01:11.408 DEBUG:boto:Path: /dghqvd-48/obj4 2023-10-11T15:01:11.408 DEBUG:boto:Data: 2023-10-11T15:01:11.408 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.409 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.410 DEBUG:boto:Params: {} 2023-10-11T15:01:11.410 DEBUG:boto:Token: None 2023-10-11T15:01:11.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj4 2023-10-11T15:01:11.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:88bEq3nD4iR4lCmTqOT+lCnZkI4= 2023-10-11T15:01:11.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:88bEq3nD4iR4lCmTqOT+lCnZkI4=', 'Content-Length': '0'} 2023-10-11T15:01:11.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094c4506775e6988e-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.414 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.415 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.415 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.415 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.416 DEBUG:boto:Method: GET 2023-10-11T15:01:11.416 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.417 DEBUG:boto:Data: 2023-10-11T15:01:11.417 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.418 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.418 DEBUG:boto:Params: {} 2023-10-11T15:01:11.419 DEBUG:boto:Token: None 2023-10-11T15:01:11.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031dc54f7a92bec7b-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.423 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.424 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.425 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.425 DEBUG:boto:path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.426 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.426 DEBUG:boto:Method: GET 2023-10-11T15:01:11.427 DEBUG:boto:Path: /dghqvd-48/?prefix=obj4 2023-10-11T15:01:11.427 DEBUG:boto:Data: 2023-10-11T15:01:11.427 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.428 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.428 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.428 DEBUG:boto:Params: {} 2023-10-11T15:01:11.429 DEBUG:boto:Token: None 2023-10-11T15:01:11.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006018106b175d2abd-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.434 DEBUG:boto:b'dghqvd-48obj41000falseobj42023-10-11T14:59:31.429Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:11.435 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.435 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.436 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.436 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.437 DEBUG:boto:Method: GET 2023-10-11T15:01:11.437 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.438 DEBUG:boto:Data: 2023-10-11T15:01:11.438 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.439 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.439 DEBUG:boto:Params: {} 2023-10-11T15:01:11.440 DEBUG:boto:Token: None 2023-10-11T15:01:11.440 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU= 2023-10-11T15:01:11.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU=', 'Content-Length': '0'} 2023-10-11T15:01:11.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000839778f18caedcd4-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.444 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.445 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.445 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.446 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.446 DEBUG:boto:Method: GET 2023-10-11T15:01:11.446 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.447 DEBUG:boto:Data: 2023-10-11T15:01:11.447 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.448 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.449 DEBUG:boto:Params: {} 2023-10-11T15:01:11.449 DEBUG:boto:Token: None 2023-10-11T15:01:11.450 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU= 2023-10-11T15:01:11.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU=', 'Content-Length': '0'} 2023-10-11T15:01:11.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d94d45f85b27d2b7-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.453 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.454 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.454 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.455 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.455 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.456 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.456 DEBUG:boto:Data: 2023-10-11T15:01:11.456 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.457 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.458 DEBUG:boto:Params: {} 2023-10-11T15:01:11.458 DEBUG:boto:Token: None 2023-10-11T15:01:11.459 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fROs7mxN+63peaF9Jjrk5DbecsY= 2023-10-11T15:01:11.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fROs7mxN+63peaF9Jjrk5DbecsY=', 'Content-Length': '0'} 2023-10-11T15:01:11.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061d3a0c9127501b0-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.462 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.463 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.463 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.464 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.464 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.464 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.465 DEBUG:boto:Data: 2023-10-11T15:01:11.465 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.466 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.466 DEBUG:boto:Params: {} 2023-10-11T15:01:11.467 DEBUG:boto:Token: None 2023-10-11T15:01:11.467 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fROs7mxN+63peaF9Jjrk5DbecsY= 2023-10-11T15:01:11.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fROs7mxN+63peaF9Jjrk5DbecsY=', 'Content-Length': '0'} 2023-10-11T15:01:11.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084dc0d0282465e0b-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:11.471 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.472 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.472 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.473 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.473 DEBUG:boto:Method: GET 2023-10-11T15:01:11.473 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.474 DEBUG:boto:Data: 2023-10-11T15:01:11.474 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.475 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.476 DEBUG:boto:Params: {} 2023-10-11T15:01:11.476 DEBUG:boto:Token: None 2023-10-11T15:01:11.477 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU= 2023-10-11T15:01:11.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU=', 'Content-Length': '0'} 2023-10-11T15:01:11.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2f609ce42269803-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.481 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.481 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.482 DEBUG:boto:path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.482 DEBUG:boto:auth_path=/dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.482 DEBUG:boto:Method: GET 2023-10-11T15:01:11.483 DEBUG:boto:Path: /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.483 DEBUG:boto:Data: 2023-10-11T15:01:11.484 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.484 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.485 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.485 DEBUG:boto:Params: {} 2023-10-11T15:01:11.486 DEBUG:boto:Token: None 2023-10-11T15:01:11.486 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5?versionId=null 2023-10-11T15:01:11.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU= 2023-10-11T15:01:11.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oDr37Y93eo7daCOU9340h7MBkfU=', 'Content-Length': '0'} 2023-10-11T15:01:11.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060af6c2bb711327d-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.490 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.490 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.491 DEBUG:boto:path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.491 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.491 DEBUG:boto:Method: GET 2023-10-11T15:01:11.492 DEBUG:boto:Path: /dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.492 DEBUG:boto:Data: 2023-10-11T15:01:11.493 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.494 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.494 DEBUG:boto:Params: {} 2023-10-11T15:01:11.494 DEBUG:boto:Token: None 2023-10-11T15:01:11.495 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c73cf3f0c875f1df-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.499 DEBUG:boto:b'dghqvd-48obj51000falseobj52023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.500 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.500 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.501 DEBUG:boto:path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.501 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.502 DEBUG:boto:Method: GET 2023-10-11T15:01:11.502 DEBUG:boto:Path: /dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.502 DEBUG:boto:Data: 2023-10-11T15:01:11.503 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.503 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.504 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.504 DEBUG:boto:Params: {} 2023-10-11T15:01:11.505 DEBUG:boto:Token: None 2023-10-11T15:01:11.505 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a2c08707937630-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.509 DEBUG:boto:b'dghqvd-48obj51000falseobj52023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.510 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.510 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.511 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.511 DEBUG:boto:Path: /dghqvd-48/obj5 2023-10-11T15:01:11.512 DEBUG:boto:Data: 2023-10-11T15:01:11.512 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.513 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.513 DEBUG:boto:Params: {} 2023-10-11T15:01:11.514 DEBUG:boto:Token: None 2023-10-11T15:01:11.514 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5 2023-10-11T15:01:11.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pMxUZsK9kq8ommnacs/yeoWNkvo= 2023-10-11T15:01:11.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pMxUZsK9kq8ommnacs/yeoWNkvo=', 'Content-Length': '0'} 2023-10-11T15:01:11.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099ef116e5508fee1-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.518 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.518 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.519 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.519 DEBUG:boto:Path: /dghqvd-48/obj5 2023-10-11T15:01:11.519 DEBUG:boto:Data: 2023-10-11T15:01:11.520 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.521 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.521 DEBUG:boto:Params: {} 2023-10-11T15:01:11.521 DEBUG:boto:Token: None 2023-10-11T15:01:11.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5 2023-10-11T15:01:11.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pMxUZsK9kq8ommnacs/yeoWNkvo= 2023-10-11T15:01:11.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pMxUZsK9kq8ommnacs/yeoWNkvo=', 'Content-Length': '0'} 2023-10-11T15:01:11.525 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029f4162e87e8e345-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:11.526 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.526 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.527 DEBUG:boto:Method: GET 2023-10-11T15:01:11.527 DEBUG:boto:Path: /dghqvd-48/obj5 2023-10-11T15:01:11.528 DEBUG:boto:Data: 2023-10-11T15:01:11.528 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.529 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.529 DEBUG:boto:Params: {} 2023-10-11T15:01:11.530 DEBUG:boto:Token: None 2023-10-11T15:01:11.530 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5 2023-10-11T15:01:11.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KyVT4E9O1G3l1JXfslqt5yX9lm8= 2023-10-11T15:01:11.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KyVT4E9O1G3l1JXfslqt5yX9lm8=', 'Content-Length': '0'} 2023-10-11T15:01:11.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058b14a185c2b5627-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.578 DEBUG:boto:path=/dghqvd-48/obj5 2023-10-11T15:01:11.579 DEBUG:boto:auth_path=/dghqvd-48/obj5 2023-10-11T15:01:11.580 DEBUG:boto:Method: GET 2023-10-11T15:01:11.580 DEBUG:boto:Path: /dghqvd-48/obj5 2023-10-11T15:01:11.580 DEBUG:boto:Data: 2023-10-11T15:01:11.581 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.582 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.582 DEBUG:boto:Params: {} 2023-10-11T15:01:11.583 DEBUG:boto:Token: None 2023-10-11T15:01:11.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj5 2023-10-11T15:01:11.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KyVT4E9O1G3l1JXfslqt5yX9lm8= 2023-10-11T15:01:11.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KyVT4E9O1G3l1JXfslqt5yX9lm8=', 'Content-Length': '0'} 2023-10-11T15:01:11.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074807a206ba3aea3-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.587 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.587 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.588 DEBUG:boto:path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.588 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.589 DEBUG:boto:Method: GET 2023-10-11T15:01:11.589 DEBUG:boto:Path: /dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.589 DEBUG:boto:Data: 2023-10-11T15:01:11.590 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.591 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.591 DEBUG:boto:Params: {} 2023-10-11T15:01:11.592 DEBUG:boto:Token: None 2023-10-11T15:01:11.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dad6a4eb1883c76-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.596 DEBUG:boto:b'dghqvd-48obj51000falseobj52023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.597 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.597 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.598 DEBUG:boto:path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.598 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.598 DEBUG:boto:Method: GET 2023-10-11T15:01:11.599 DEBUG:boto:Path: /dghqvd-48/?prefix=obj5 2023-10-11T15:01:11.599 DEBUG:boto:Data: 2023-10-11T15:01:11.600 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.600 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.600 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.601 DEBUG:boto:Params: {} 2023-10-11T15:01:11.601 DEBUG:boto:Token: None 2023-10-11T15:01:11.602 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058dd640afa31eded-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.646 DEBUG:boto:b'dghqvd-48obj51000falseobj52023-10-11T15:00:39.736Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:11.648 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.648 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.649 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.649 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.650 DEBUG:boto:Method: GET 2023-10-11T15:01:11.650 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.650 DEBUG:boto:Data: 2023-10-11T15:01:11.651 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.652 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.652 DEBUG:boto:Params: {} 2023-10-11T15:01:11.653 DEBUG:boto:Token: None 2023-10-11T15:01:11.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g= 2023-10-11T15:01:11.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g=', 'Content-Length': '0'} 2023-10-11T15:01:11.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b84bfbe9a7e259df-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.657 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.657 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.658 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.658 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.659 DEBUG:boto:Method: GET 2023-10-11T15:01:11.659 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.659 DEBUG:boto:Data: 2023-10-11T15:01:11.660 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.660 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.661 DEBUG:boto:Params: {} 2023-10-11T15:01:11.661 DEBUG:boto:Token: None 2023-10-11T15:01:11.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g= 2023-10-11T15:01:11.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g=', 'Content-Length': '0'} 2023-10-11T15:01:11.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004505b9e961cb45be-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.666 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.666 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.666 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.667 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.667 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.668 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.668 DEBUG:boto:Data: 2023-10-11T15:01:11.669 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.669 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.669 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.670 DEBUG:boto:Params: {} 2023-10-11T15:01:11.670 DEBUG:boto:Token: None 2023-10-11T15:01:11.671 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.671 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CYBF017zLkoT9Z0uaADizCvm274= 2023-10-11T15:01:11.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CYBF017zLkoT9Z0uaADizCvm274=', 'Content-Length': '0'} 2023-10-11T15:01:11.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078bb6990cd337aff-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.674 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.675 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.675 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.676 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.676 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.676 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.677 DEBUG:boto:Data: 2023-10-11T15:01:11.677 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.678 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.679 DEBUG:boto:Params: {} 2023-10-11T15:01:11.679 DEBUG:boto:Token: None 2023-10-11T15:01:11.679 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CYBF017zLkoT9Z0uaADizCvm274= 2023-10-11T15:01:11.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CYBF017zLkoT9Z0uaADizCvm274=', 'Content-Length': '0'} 2023-10-11T15:01:11.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a31a786d31494aff-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:11.683 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.683 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.684 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.684 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.685 DEBUG:boto:Method: GET 2023-10-11T15:01:11.685 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.686 DEBUG:boto:Data: 2023-10-11T15:01:11.686 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.687 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.687 DEBUG:boto:Params: {} 2023-10-11T15:01:11.689 DEBUG:boto:Token: None 2023-10-11T15:01:11.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g= 2023-10-11T15:01:11.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g=', 'Content-Length': '0'} 2023-10-11T15:01:11.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029fe5558493161ce-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.693 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.694 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.694 DEBUG:boto:path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.695 DEBUG:boto:auth_path=/dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.695 DEBUG:boto:Method: GET 2023-10-11T15:01:11.695 DEBUG:boto:Path: /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.696 DEBUG:boto:Data: 2023-10-11T15:01:11.696 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.697 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.698 DEBUG:boto:Params: {} 2023-10-11T15:01:11.698 DEBUG:boto:Token: None 2023-10-11T15:01:11.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6?versionId=null 2023-10-11T15:01:11.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g= 2023-10-11T15:01:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mqrePowW3tZzLE0SGklUo5XS43g=', 'Content-Length': '0'} 2023-10-11T15:01:11.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd24d0aa236eb20d-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.702 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.703 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.703 DEBUG:boto:path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.704 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.704 DEBUG:boto:Method: GET 2023-10-11T15:01:11.704 DEBUG:boto:Path: /dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.705 DEBUG:boto:Data: 2023-10-11T15:01:11.705 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.706 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.707 DEBUG:boto:Params: {} 2023-10-11T15:01:11.707 DEBUG:boto:Token: None 2023-10-11T15:01:11.707 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007494b87a4d65c2bc-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.711 DEBUG:boto:b'dghqvd-48obj61000falseobj62023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.712 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.712 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.713 DEBUG:boto:path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.713 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.713 DEBUG:boto:Method: GET 2023-10-11T15:01:11.714 DEBUG:boto:Path: /dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.714 DEBUG:boto:Data: 2023-10-11T15:01:11.715 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.715 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.716 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.716 DEBUG:boto:Params: {} 2023-10-11T15:01:11.716 DEBUG:boto:Token: None 2023-10-11T15:01:11.717 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.717 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a7f16671169970-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.721 DEBUG:boto:b'dghqvd-48obj61000falseobj62023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.721 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.722 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.722 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.723 DEBUG:boto:Path: /dghqvd-48/obj6 2023-10-11T15:01:11.723 DEBUG:boto:Data: 2023-10-11T15:01:11.724 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.724 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.725 DEBUG:boto:Params: {} 2023-10-11T15:01:11.725 DEBUG:boto:Token: None 2023-10-11T15:01:11.726 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6 2023-10-11T15:01:11.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aIH2SZCQhAGq7YJvOvyxP0vUDeU= 2023-10-11T15:01:11.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aIH2SZCQhAGq7YJvOvyxP0vUDeU=', 'Content-Length': '0'} 2023-10-11T15:01:11.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f12583df67574698-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.729 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.730 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.730 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.731 DEBUG:boto:Path: /dghqvd-48/obj6 2023-10-11T15:01:11.731 DEBUG:boto:Data: 2023-10-11T15:01:11.731 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.732 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.733 DEBUG:boto:Params: {} 2023-10-11T15:01:11.733 DEBUG:boto:Token: None 2023-10-11T15:01:11.734 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6 2023-10-11T15:01:11.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aIH2SZCQhAGq7YJvOvyxP0vUDeU= 2023-10-11T15:01:11.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aIH2SZCQhAGq7YJvOvyxP0vUDeU=', 'Content-Length': '0'} 2023-10-11T15:01:11.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216a63d419d6fcda-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:11.737 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.738 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.738 DEBUG:boto:Method: GET 2023-10-11T15:01:11.739 DEBUG:boto:Path: /dghqvd-48/obj6 2023-10-11T15:01:11.739 DEBUG:boto:Data: 2023-10-11T15:01:11.740 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.741 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.741 DEBUG:boto:Params: {} 2023-10-11T15:01:11.742 DEBUG:boto:Token: None 2023-10-11T15:01:11.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6 2023-10-11T15:01:11.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b7h5hucccgW/Zde2dsGqj9vRFAQ= 2023-10-11T15:01:11.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b7h5hucccgW/Zde2dsGqj9vRFAQ=', 'Content-Length': '0'} 2023-10-11T15:01:11.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d4fcc544ea0ada4-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.746 DEBUG:boto:path=/dghqvd-48/obj6 2023-10-11T15:01:11.746 DEBUG:boto:auth_path=/dghqvd-48/obj6 2023-10-11T15:01:11.747 DEBUG:boto:Method: GET 2023-10-11T15:01:11.747 DEBUG:boto:Path: /dghqvd-48/obj6 2023-10-11T15:01:11.747 DEBUG:boto:Data: 2023-10-11T15:01:11.748 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.749 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.749 DEBUG:boto:Params: {} 2023-10-11T15:01:11.750 DEBUG:boto:Token: None 2023-10-11T15:01:11.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj6 2023-10-11T15:01:11.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b7h5hucccgW/Zde2dsGqj9vRFAQ= 2023-10-11T15:01:11.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b7h5hucccgW/Zde2dsGqj9vRFAQ=', 'Content-Length': '0'} 2023-10-11T15:01:11.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a865c65e41f02b17-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.754 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.755 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.755 DEBUG:boto:path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.755 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.756 DEBUG:boto:Method: GET 2023-10-11T15:01:11.756 DEBUG:boto:Path: /dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.756 DEBUG:boto:Data: 2023-10-11T15:01:11.757 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.758 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.758 DEBUG:boto:Params: {} 2023-10-11T15:01:11.758 DEBUG:boto:Token: None 2023-10-11T15:01:11.759 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.759 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e768059d9ef4f786-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.763 DEBUG:boto:b'dghqvd-48obj61000falseobj62023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.763 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.764 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.764 DEBUG:boto:path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.765 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.765 DEBUG:boto:Method: GET 2023-10-11T15:01:11.765 DEBUG:boto:Path: /dghqvd-48/?prefix=obj6 2023-10-11T15:01:11.766 DEBUG:boto:Data: 2023-10-11T15:01:11.766 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.767 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.767 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.768 DEBUG:boto:Params: {} 2023-10-11T15:01:11.768 DEBUG:boto:Token: None 2023-10-11T15:01:11.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041be1591c5fcfa96-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.772 DEBUG:boto:b'dghqvd-48obj61000falseobj62023-10-11T15:00:39.792Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:11.773 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.773 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.774 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.774 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.775 DEBUG:boto:Method: GET 2023-10-11T15:01:11.775 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.776 DEBUG:boto:Data: 2023-10-11T15:01:11.776 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.777 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.777 DEBUG:boto:Params: {} 2023-10-11T15:01:11.778 DEBUG:boto:Token: None 2023-10-11T15:01:11.778 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg= 2023-10-11T15:01:11.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg=', 'Content-Length': '0'} 2023-10-11T15:01:11.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056920e72c39e65a3-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.822 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.823 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.823 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.824 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.824 DEBUG:boto:Method: GET 2023-10-11T15:01:11.825 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.825 DEBUG:boto:Data: 2023-10-11T15:01:11.826 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.826 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.827 DEBUG:boto:Params: {} 2023-10-11T15:01:11.827 DEBUG:boto:Token: None 2023-10-11T15:01:11.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg= 2023-10-11T15:01:11.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg=', 'Content-Length': '0'} 2023-10-11T15:01:11.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a12be6f60dfe030-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.831 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.832 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.832 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.833 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.833 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.834 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.834 DEBUG:boto:Data: 2023-10-11T15:01:11.834 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.835 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.836 DEBUG:boto:Params: {} 2023-10-11T15:01:11.836 DEBUG:boto:Token: None 2023-10-11T15:01:11.837 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DuYaFUJeyAVWi7qnql/ALTsW98c= 2023-10-11T15:01:11.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DuYaFUJeyAVWi7qnql/ALTsW98c=', 'Content-Length': '0'} 2023-10-11T15:01:11.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb6bf2afea67044f-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.840 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.840 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.841 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.841 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.842 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.842 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.842 DEBUG:boto:Data: 2023-10-11T15:01:11.843 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.843 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.844 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.844 DEBUG:boto:Params: {} 2023-10-11T15:01:11.845 DEBUG:boto:Token: None 2023-10-11T15:01:11.845 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.846 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DuYaFUJeyAVWi7qnql/ALTsW98c= 2023-10-11T15:01:11.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DuYaFUJeyAVWi7qnql/ALTsW98c=', 'Content-Length': '0'} 2023-10-11T15:01:11.848 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b76ed40ebc987ab-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:11.849 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.850 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.850 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.851 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.851 DEBUG:boto:Method: GET 2023-10-11T15:01:11.851 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.852 DEBUG:boto:Data: 2023-10-11T15:01:11.852 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.853 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.853 DEBUG:boto:Params: {} 2023-10-11T15:01:11.854 DEBUG:boto:Token: None 2023-10-11T15:01:11.854 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg= 2023-10-11T15:01:11.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg=', 'Content-Length': '0'} 2023-10-11T15:01:11.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006db376f70bc2812a-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.858 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.859 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.859 DEBUG:boto:path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.859 DEBUG:boto:auth_path=/dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.860 DEBUG:boto:Method: GET 2023-10-11T15:01:11.860 DEBUG:boto:Path: /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.861 DEBUG:boto:Data: 2023-10-11T15:01:11.861 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.862 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.862 DEBUG:boto:Params: {} 2023-10-11T15:01:11.863 DEBUG:boto:Token: None 2023-10-11T15:01:11.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7?versionId=null 2023-10-11T15:01:11.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg= 2023-10-11T15:01:11.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LQ7F7dGXkTc4lRWXoyVrqKpB0Bg=', 'Content-Length': '0'} 2023-10-11T15:01:11.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d021ae47660198c-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.867 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.867 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.868 DEBUG:boto:path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.868 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.869 DEBUG:boto:Method: GET 2023-10-11T15:01:11.869 DEBUG:boto:Path: /dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.869 DEBUG:boto:Data: 2023-10-11T15:01:11.870 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.871 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.871 DEBUG:boto:Params: {} 2023-10-11T15:01:11.872 DEBUG:boto:Token: None 2023-10-11T15:01:11.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952519b698e436c7-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.876 DEBUG:boto:b'dghqvd-48obj71000falseobj72023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.877 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.877 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.878 DEBUG:boto:path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.878 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.879 DEBUG:boto:Method: GET 2023-10-11T15:01:11.879 DEBUG:boto:Path: /dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.880 DEBUG:boto:Data: 2023-10-11T15:01:11.880 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.881 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.881 DEBUG:boto:Params: {} 2023-10-11T15:01:11.882 DEBUG:boto:Token: None 2023-10-11T15:01:11.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002166d1dc451ce326-006526b8b7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.926 DEBUG:boto:b'dghqvd-48obj71000falseobj72023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:11.927 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.928 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.928 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.929 DEBUG:boto:Path: /dghqvd-48/obj7 2023-10-11T15:01:11.929 DEBUG:boto:Data: 2023-10-11T15:01:11.929 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.930 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.931 DEBUG:boto:Params: {} 2023-10-11T15:01:11.931 DEBUG:boto:Token: None 2023-10-11T15:01:11.932 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7 2023-10-11T15:01:11.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GV4zWB0tMieWoTFLHtiLq6hEg+c= 2023-10-11T15:01:11.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GV4zWB0tMieWoTFLHtiLq6hEg+c=', 'Content-Length': '0'} 2023-10-11T15:01:11.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e2c4f8a4583f2ab-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.935 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.936 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.936 DEBUG:boto:Method: HEAD 2023-10-11T15:01:11.937 DEBUG:boto:Path: /dghqvd-48/obj7 2023-10-11T15:01:11.937 DEBUG:boto:Data: 2023-10-11T15:01:11.937 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.938 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.939 DEBUG:boto:Params: {} 2023-10-11T15:01:11.939 DEBUG:boto:Token: None 2023-10-11T15:01:11.940 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7 2023-10-11T15:01:11.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GV4zWB0tMieWoTFLHtiLq6hEg+c= 2023-10-11T15:01:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GV4zWB0tMieWoTFLHtiLq6hEg+c=', 'Content-Length': '0'} 2023-10-11T15:01:11.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015fc048251286d6c-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:11.943 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.944 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.944 DEBUG:boto:Method: GET 2023-10-11T15:01:11.945 DEBUG:boto:Path: /dghqvd-48/obj7 2023-10-11T15:01:11.945 DEBUG:boto:Data: 2023-10-11T15:01:11.945 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.946 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.947 DEBUG:boto:Params: {} 2023-10-11T15:01:11.947 DEBUG:boto:Token: None 2023-10-11T15:01:11.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7 2023-10-11T15:01:11.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dC1x/XfvR36k2H9cw839kW7xLLY= 2023-10-11T15:01:11.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dC1x/XfvR36k2H9cw839kW7xLLY=', 'Content-Length': '0'} 2023-10-11T15:01:11.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8b8cba255a283ac-006526b8b7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.951 DEBUG:boto:path=/dghqvd-48/obj7 2023-10-11T15:01:11.952 DEBUG:boto:auth_path=/dghqvd-48/obj7 2023-10-11T15:01:11.952 DEBUG:boto:Method: GET 2023-10-11T15:01:11.953 DEBUG:boto:Path: /dghqvd-48/obj7 2023-10-11T15:01:11.953 DEBUG:boto:Data: 2023-10-11T15:01:11.954 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:11.955 DEBUG:boto:Port: 8000 2023-10-11T15:01:11.955 DEBUG:boto:Params: {} 2023-10-11T15:01:11.955 DEBUG:boto:Token: None 2023-10-11T15:01:11.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/obj7 2023-10-11T15:01:11.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dC1x/XfvR36k2H9cw839kW7xLLY= 2023-10-11T15:01:11.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dC1x/XfvR36k2H9cw839kW7xLLY=', 'Content-Length': '0'} 2023-10-11T15:01:11.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000683b0a0cd7370044-006526b8b7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:11.960 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:11.960 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:11.960 DEBUG:boto:path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.961 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.961 DEBUG:boto:Method: GET 2023-10-11T15:01:11.962 DEBUG:boto:Path: /dghqvd-48/?prefix=obj7 2023-10-11T15:01:11.962 DEBUG:boto:Data: 2023-10-11T15:01:11.962 DEBUG:boto:Headers: {} 2023-10-11T15:01:11.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:11.963 DEBUG:boto:Port: 8001 2023-10-11T15:01:11.964 DEBUG:boto:Params: {} 2023-10-11T15:01:11.964 DEBUG:boto:Token: None 2023-10-11T15:01:11.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:11 GMT /dghqvd-48/ 2023-10-11T15:01:11.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic= 2023-10-11T15:01:11.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WyMnm+VL+Sz/mqgQY1ci96R9ic=', 'Content-Length': '0'} 2023-10-11T15:01:11.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7c4a9eb21d01ecc-006526b8b7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.014 DEBUG:boto:b'dghqvd-48obj71000falseobj72023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.015 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:12.016 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:12.016 DEBUG:boto:path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:12.016 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj7 2023-10-11T15:01:12.017 DEBUG:boto:Method: GET 2023-10-11T15:01:12.017 DEBUG:boto:Path: /dghqvd-48/?prefix=obj7 2023-10-11T15:01:12.018 DEBUG:boto:Data: 2023-10-11T15:01:12.018 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.019 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.020 DEBUG:boto:Params: {} 2023-10-11T15:01:12.020 DEBUG:boto:Token: None 2023-10-11T15:01:12.021 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/ 2023-10-11T15:01:12.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY= 2023-10-11T15:01:12.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY=', 'Content-Length': '0'} 2023-10-11T15:01:12.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006dddbdb03a2e6ff-006526b8b8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.025 DEBUG:boto:b'dghqvd-48obj71000falseobj72023-10-11T15:00:39.809Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:12.026 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.026 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.027 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.027 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.028 DEBUG:boto:Method: GET 2023-10-11T15:01:12.028 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.029 DEBUG:boto:Data: 2023-10-11T15:01:12.029 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.030 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.030 DEBUG:boto:Params: {} 2023-10-11T15:01:12.031 DEBUG:boto:Token: None 2023-10-11T15:01:12.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU= 2023-10-11T15:01:12.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU=', 'Content-Length': '0'} 2023-10-11T15:01:12.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f37b0ec25a3a28a-006526b8b8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.035 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.035 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.036 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.036 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.037 DEBUG:boto:Method: GET 2023-10-11T15:01:12.037 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.038 DEBUG:boto:Data: 2023-10-11T15:01:12.038 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.039 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.039 DEBUG:boto:Params: {} 2023-10-11T15:01:12.040 DEBUG:boto:Token: None 2023-10-11T15:01:12.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU= 2023-10-11T15:01:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU=', 'Content-Length': '0'} 2023-10-11T15:01:12.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035faad2661117b4b-006526b8b8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.044 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.044 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.045 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.045 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.046 DEBUG:boto:Method: HEAD 2023-10-11T15:01:12.046 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.047 DEBUG:boto:Data: 2023-10-11T15:01:12.047 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.048 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.048 DEBUG:boto:Params: {} 2023-10-11T15:01:12.049 DEBUG:boto:Token: None 2023-10-11T15:01:12.049 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aiCZ6KvBNM6Kpw7A5c+YgIxfwAI= 2023-10-11T15:01:12.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aiCZ6KvBNM6Kpw7A5c+YgIxfwAI=', 'Content-Length': '0'} 2023-10-11T15:01:12.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1905ab23227175e-006526b8b8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.053 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.053 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.053 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.054 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.054 DEBUG:boto:Method: HEAD 2023-10-11T15:01:12.055 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.055 DEBUG:boto:Data: 2023-10-11T15:01:12.055 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.056 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.057 DEBUG:boto:Params: {} 2023-10-11T15:01:12.057 DEBUG:boto:Token: None 2023-10-11T15:01:12.058 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aiCZ6KvBNM6Kpw7A5c+YgIxfwAI= 2023-10-11T15:01:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aiCZ6KvBNM6Kpw7A5c+YgIxfwAI=', 'Content-Length': '0'} 2023-10-11T15:01:12.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ae32f9d2709c9ab-006526b8b8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:12.061 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.062 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.062 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.063 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.063 DEBUG:boto:Method: GET 2023-10-11T15:01:12.064 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.064 DEBUG:boto:Data: 2023-10-11T15:01:12.064 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.065 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.065 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.066 DEBUG:boto:Params: {} 2023-10-11T15:01:12.066 DEBUG:boto:Token: None 2023-10-11T15:01:12.067 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU= 2023-10-11T15:01:12.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU=', 'Content-Length': '0'} 2023-10-11T15:01:12.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb1a75dc1ff9baf1-006526b8b8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.070 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.071 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.071 DEBUG:boto:path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.072 DEBUG:boto:auth_path=/dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.072 DEBUG:boto:Method: GET 2023-10-11T15:01:12.072 DEBUG:boto:Path: /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.073 DEBUG:boto:Data: 2023-10-11T15:01:12.073 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.074 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.075 DEBUG:boto:Params: {} 2023-10-11T15:01:12.075 DEBUG:boto:Token: None 2023-10-11T15:01:12.076 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8?versionId=null 2023-10-11T15:01:12.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU= 2023-10-11T15:01:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smyS6VyXS2ldTXfrypqVAj4XYWU=', 'Content-Length': '0'} 2023-10-11T15:01:12.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbba4a5757e7b5aa-006526b8b8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.122 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:12.123 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:12.123 DEBUG:boto:path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.124 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.124 DEBUG:boto:Method: GET 2023-10-11T15:01:12.125 DEBUG:boto:Path: /dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.125 DEBUG:boto:Data: 2023-10-11T15:01:12.125 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.126 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.127 DEBUG:boto:Params: {} 2023-10-11T15:01:12.127 DEBUG:boto:Token: None 2023-10-11T15:01:12.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/ 2023-10-11T15:01:12.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY= 2023-10-11T15:01:12.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY=', 'Content-Length': '0'} 2023-10-11T15:01:12.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084b21a457eef7c3e-006526b8b8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.131 DEBUG:boto:b'dghqvd-48obj81000falseobj82023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.132 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:12.132 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:12.133 DEBUG:boto:path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.133 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.134 DEBUG:boto:Method: GET 2023-10-11T15:01:12.134 DEBUG:boto:Path: /dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.135 DEBUG:boto:Data: 2023-10-11T15:01:12.135 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.136 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.136 DEBUG:boto:Params: {} 2023-10-11T15:01:12.137 DEBUG:boto:Token: None 2023-10-11T15:01:12.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/ 2023-10-11T15:01:12.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY= 2023-10-11T15:01:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY=', 'Content-Length': '0'} 2023-10-11T15:01:12.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3b91fff0f08d55-006526b8b8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.141 DEBUG:boto:b'dghqvd-48obj81000falseobj82023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.142 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.142 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.143 DEBUG:boto:Method: HEAD 2023-10-11T15:01:12.143 DEBUG:boto:Path: /dghqvd-48/obj8 2023-10-11T15:01:12.144 DEBUG:boto:Data: 2023-10-11T15:01:12.144 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.145 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.145 DEBUG:boto:Params: {} 2023-10-11T15:01:12.146 DEBUG:boto:Token: None 2023-10-11T15:01:12.146 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8 2023-10-11T15:01:12.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CAX2eBbZfCKyRid1GDQvID/6FT8= 2023-10-11T15:01:12.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CAX2eBbZfCKyRid1GDQvID/6FT8=', 'Content-Length': '0'} 2023-10-11T15:01:12.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000853b7cd674162db5-006526b8b8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.150 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.150 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.150 DEBUG:boto:Method: HEAD 2023-10-11T15:01:12.151 DEBUG:boto:Path: /dghqvd-48/obj8 2023-10-11T15:01:12.151 DEBUG:boto:Data: 2023-10-11T15:01:12.152 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.153 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.153 DEBUG:boto:Params: {} 2023-10-11T15:01:12.153 DEBUG:boto:Token: None 2023-10-11T15:01:12.154 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8 2023-10-11T15:01:12.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CAX2eBbZfCKyRid1GDQvID/6FT8= 2023-10-11T15:01:12.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CAX2eBbZfCKyRid1GDQvID/6FT8=', 'Content-Length': '0'} 2023-10-11T15:01:12.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f211652edfd4e92-006526b8b8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:12.158 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.158 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.159 DEBUG:boto:Method: GET 2023-10-11T15:01:12.159 DEBUG:boto:Path: /dghqvd-48/obj8 2023-10-11T15:01:12.160 DEBUG:boto:Data: 2023-10-11T15:01:12.160 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.161 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.161 DEBUG:boto:Params: {} 2023-10-11T15:01:12.162 DEBUG:boto:Token: None 2023-10-11T15:01:12.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8 2023-10-11T15:01:12.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ngGKzHfbCjnTL4dq0bF/2NFbQqk= 2023-10-11T15:01:12.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ngGKzHfbCjnTL4dq0bF/2NFbQqk=', 'Content-Length': '0'} 2023-10-11T15:01:12.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-48:daa316e9-869c-4074-a756-a8adfb4a3573.5751.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a5ac5daf17e9bc3-006526b8b8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.206 DEBUG:boto:path=/dghqvd-48/obj8 2023-10-11T15:01:12.207 DEBUG:boto:auth_path=/dghqvd-48/obj8 2023-10-11T15:01:12.207 DEBUG:boto:Method: GET 2023-10-11T15:01:12.208 DEBUG:boto:Path: /dghqvd-48/obj8 2023-10-11T15:01:12.208 DEBUG:boto:Data: 2023-10-11T15:01:12.209 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.209 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.210 DEBUG:boto:Params: {} 2023-10-11T15:01:12.210 DEBUG:boto:Token: None 2023-10-11T15:01:12.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/obj8 2023-10-11T15:01:12.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ngGKzHfbCjnTL4dq0bF/2NFbQqk= 2023-10-11T15:01:12.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ngGKzHfbCjnTL4dq0bF/2NFbQqk=', 'Content-Length': '0'} 2023-10-11T15:01:12.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062b9af138e4543a5-006526b8b8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.215 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:12.215 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:12.216 DEBUG:boto:path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.216 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.217 DEBUG:boto:Method: GET 2023-10-11T15:01:12.217 DEBUG:boto:Path: /dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.217 DEBUG:boto:Data: 2023-10-11T15:01:12.218 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:12.219 DEBUG:boto:Port: 8001 2023-10-11T15:01:12.219 DEBUG:boto:Params: {} 2023-10-11T15:01:12.220 DEBUG:boto:Token: None 2023-10-11T15:01:12.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/ 2023-10-11T15:01:12.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY= 2023-10-11T15:01:12.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY=', 'Content-Length': '0'} 2023-10-11T15:01:12.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003160666b9a475384-006526b8b8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.225 DEBUG:boto:b'dghqvd-48obj81000falseobj82023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.225 DEBUG:boto:path=/dghqvd-48/ 2023-10-11T15:01:12.226 DEBUG:boto:auth_path=/dghqvd-48/ 2023-10-11T15:01:12.226 DEBUG:boto:path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.227 DEBUG:boto:auth_path=/dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.227 DEBUG:boto:Method: GET 2023-10-11T15:01:12.227 DEBUG:boto:Path: /dghqvd-48/?prefix=obj8 2023-10-11T15:01:12.228 DEBUG:boto:Data: 2023-10-11T15:01:12.228 DEBUG:boto:Headers: {} 2023-10-11T15:01:12.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:12.229 DEBUG:boto:Port: 8000 2023-10-11T15:01:12.229 DEBUG:boto:Params: {} 2023-10-11T15:01:12.230 DEBUG:boto:Token: None 2023-10-11T15:01:12.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:12 GMT /dghqvd-48/ 2023-10-11T15:01:12.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY= 2023-10-11T15:01:12.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hJHZpjAvSJ21D+h9zZCUxhdzbY=', 'Content-Length': '0'} 2023-10-11T15:01:12.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004f52331dfdec78a-006526b8b8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:12.234 DEBUG:boto:b'dghqvd-48obj81000falseobj82023-10-11T15:00:39.828Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:12.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-48 zones={a2, a1} 2023-10-11T15:01:12.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-49 2023-10-11T15:01:12.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:12.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:12.237 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:01:12.302 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:12.302 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:12.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:12.338+0000 7f756cdce780 1 bucket sync caught up with source: 2023-10-11T15:01:12.340 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.40.6, 00000000002.13.1, 00000000004.37.6, 00000000002.22.1, 00000000004.46.6, 00000000002.14.1, 00000000002.23.1, 00000000004.25.6, 00000000002.18.1, 00000000002.24.1, 00000000002.28.1] 2023-10-11T15:01:12.341 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.40.6, 00000000002.13.1, 00000000004.37.6, 00000000002.22.1, 00000000004.46.6, 00000000002.14.1, 00000000002.23.1, 00000000004.25.6, 00000000002.18.1, 00000000002.24.1, 00000000002.28.1] 2023-10-11T15:01:12.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:12.338+0000 7f756cdce780 0 bucket checkpoint complete 2023-10-11T15:01:12.353 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-49 2023-10-11T15:01:12.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:12.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:12.355 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:01:12.412 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:12.413 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:12.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:12.444+0000 7f17fa993780 1 waiting to reach incremental sync.. 2023-10-11T15:01:17.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:17.444+0000 7f17fa993780 1 waiting to reach incremental sync.. 2023-10-11T15:01:22.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:22.445+0000 7f17fa993780 1 waiting to reach incremental sync.. 2023-10-11T15:01:27.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:27.447+0000 7f17fa993780 1 bucket sync caught up with source: 2023-10-11T15:01:27.449 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000004.173.1, 00000000002.205.1, 00000000004.159.1, 00000000002.189.1, 00000000004.261.1, 00000000002.72.1, 00000000002.78.1, 00000000004.190.1, 00000000002.253.1, 00000000002.81.1, 00000000002.93.1] 2023-10-11T15:01:27.449 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000004.173.1, 00000000002.205.1, 00000000004.159.1, 00000000002.189.1, 00000000004.261.1, 00000000002.72.1, 00000000002.78.1, 00000000004.190.1, 00000000002.253.1, 00000000002.81.1, 00000000002.93.1] 2023-10-11T15:01:27.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:27.447+0000 7f17fa993780 0 bucket checkpoint complete 2023-10-11T15:01:27.463 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-49 zones={a2, a1} 2023-10-11T15:01:27.463 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.464 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.464 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.464 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:01:27.465 DEBUG:boto:Data: 2023-10-11T15:01:27.465 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.466 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.467 DEBUG:boto:Params: {} 2023-10-11T15:01:27.467 DEBUG:boto:Token: None 2023-10-11T15:01:27.468 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kEfqZrUp9mbgpkmiCw46bL/In1A= 2023-10-11T15:01:27.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kEfqZrUp9mbgpkmiCw46bL/In1A=', 'Content-Length': '0'} 2023-10-11T15:01:27.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c844b14c0a9881e1-006526b8c7-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.472 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.472 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.473 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.473 DEBUG:boto:Path: /dghqvd-49/ 2023-10-11T15:01:27.474 DEBUG:boto:Data: 2023-10-11T15:01:27.474 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.475 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.475 DEBUG:boto:Params: {} 2023-10-11T15:01:27.476 DEBUG:boto:Token: None 2023-10-11T15:01:27.476 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kEfqZrUp9mbgpkmiCw46bL/In1A= 2023-10-11T15:01:27.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kEfqZrUp9mbgpkmiCw46bL/In1A=', 'Content-Length': '0'} 2023-10-11T15:01:27.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085d1d8c31096c62c-006526b8c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:01:27.480 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.481 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.481 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.481 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.482 DEBUG:boto:Method: GET 2023-10-11T15:01:27.482 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.482 DEBUG:boto:Data: 2023-10-11T15:01:27.483 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.484 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.484 DEBUG:boto:Params: {} 2023-10-11T15:01:27.484 DEBUG:boto:Token: None 2023-10-11T15:01:27.485 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/?versions 2023-10-11T15:01:27.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8= 2023-10-11T15:01:27.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8=', 'Content-Length': '0'} 2023-10-11T15:01:27.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c0396664f055c1-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.489 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:01:27.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:01:27.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:01:27.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:01:27.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T15:01:27.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T15:01:27.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T15:01:27.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T15:01:27.493 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:01:27.493 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.493 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.494 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.494 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.494 DEBUG:boto:Method: GET 2023-10-11T15:01:27.495 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.495 DEBUG:boto:Data: 2023-10-11T15:01:27.496 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.496 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.497 DEBUG:boto:Params: {} 2023-10-11T15:01:27.497 DEBUG:boto:Token: None 2023-10-11T15:01:27.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/?versions 2023-10-11T15:01:27.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8= 2023-10-11T15:01:27.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8=', 'Content-Length': '0'} 2023-10-11T15:01:27.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d7df2145971cea-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.501 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-10-11T15:01:27.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-10-11T15:01:27.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-10-11T15:01:27.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-10-11T15:01:27.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-10-11T15:01:27.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-10-11T15:01:27.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-10-11T15:01:27.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-10-11T15:01:27.506 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.506 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.506 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.507 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.507 DEBUG:boto:Method: GET 2023-10-11T15:01:27.508 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.508 DEBUG:boto:Data: 2023-10-11T15:01:27.509 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.509 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.510 DEBUG:boto:Params: {} 2023-10-11T15:01:27.510 DEBUG:boto:Token: None 2023-10-11T15:01:27.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/?versions 2023-10-11T15:01:27.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8= 2023-10-11T15:01:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8=', 'Content-Length': '0'} 2023-10-11T15:01:27.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f974e4486c9ee878-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.514 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.515 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.516 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.516 DEBUG:boto:path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.517 DEBUG:boto:auth_path=/dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.517 DEBUG:boto:Method: GET 2023-10-11T15:01:27.517 DEBUG:boto:Path: /dghqvd-49/?versions&max-keys=999 2023-10-11T15:01:27.518 DEBUG:boto:Data: 2023-10-11T15:01:27.518 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.519 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.519 DEBUG:boto:Params: {} 2023-10-11T15:01:27.520 DEBUG:boto:Token: None 2023-10-11T15:01:27.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/?versions 2023-10-11T15:01:27.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8= 2023-10-11T15:01:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GlwbLas8wEXLxaghoSsjPt07tb8=', 'Content-Length': '0'} 2023-10-11T15:01:27.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f52a8f69a5d073e-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.524 DEBUG:boto:b'dghqvd-49999falseobj1nulltrue2023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:27.526 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.526 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.527 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.527 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.527 DEBUG:boto:Method: GET 2023-10-11T15:01:27.528 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.528 DEBUG:boto:Data: 2023-10-11T15:01:27.529 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.530 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.530 DEBUG:boto:Params: {} 2023-10-11T15:01:27.531 DEBUG:boto:Token: None 2023-10-11T15:01:27.531 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ= 2023-10-11T15:01:27.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ=', 'Content-Length': '0'} 2023-10-11T15:01:27.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ee1609041903c98-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.535 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.535 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.536 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.536 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.537 DEBUG:boto:Method: GET 2023-10-11T15:01:27.537 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.537 DEBUG:boto:Data: 2023-10-11T15:01:27.538 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.539 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.539 DEBUG:boto:Params: {} 2023-10-11T15:01:27.540 DEBUG:boto:Token: None 2023-10-11T15:01:27.540 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.541 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ= 2023-10-11T15:01:27.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ=', 'Content-Length': '0'} 2023-10-11T15:01:27.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006edce65095b77eba-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.586 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.587 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.587 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.588 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.588 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.588 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.589 DEBUG:boto:Data: 2023-10-11T15:01:27.589 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.590 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.591 DEBUG:boto:Params: {} 2023-10-11T15:01:27.591 DEBUG:boto:Token: None 2023-10-11T15:01:27.591 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:15inyQ17zwWuBNPZwLhaoKfNF34= 2023-10-11T15:01:27.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:15inyQ17zwWuBNPZwLhaoKfNF34=', 'Content-Length': '0'} 2023-10-11T15:01:27.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001658d664eb6400c4-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.595 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.595 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.596 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.596 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.596 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.597 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.597 DEBUG:boto:Data: 2023-10-11T15:01:27.598 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.598 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.598 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.599 DEBUG:boto:Params: {} 2023-10-11T15:01:27.599 DEBUG:boto:Token: None 2023-10-11T15:01:27.600 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:15inyQ17zwWuBNPZwLhaoKfNF34= 2023-10-11T15:01:27.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:15inyQ17zwWuBNPZwLhaoKfNF34=', 'Content-Length': '0'} 2023-10-11T15:01:27.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002b4f67fe272c3c7-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:27.604 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.604 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.605 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.605 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.605 DEBUG:boto:Method: GET 2023-10-11T15:01:27.606 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.606 DEBUG:boto:Data: 2023-10-11T15:01:27.607 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.608 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.608 DEBUG:boto:Params: {} 2023-10-11T15:01:27.609 DEBUG:boto:Token: None 2023-10-11T15:01:27.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ= 2023-10-11T15:01:27.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ=', 'Content-Length': '0'} 2023-10-11T15:01:27.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005df81d11209c116-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.654 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.655 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.655 DEBUG:boto:path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.656 DEBUG:boto:auth_path=/dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.656 DEBUG:boto:Method: GET 2023-10-11T15:01:27.657 DEBUG:boto:Path: /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.657 DEBUG:boto:Data: 2023-10-11T15:01:27.657 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.658 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.659 DEBUG:boto:Params: {} 2023-10-11T15:01:27.659 DEBUG:boto:Token: None 2023-10-11T15:01:27.659 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1?versionId=null 2023-10-11T15:01:27.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ= 2023-10-11T15:01:27.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F4idWA2NU1lRolrnpD5BB3OV3yQ=', 'Content-Length': '0'} 2023-10-11T15:01:27.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002febf6f99997c512-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.663 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.663 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.664 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.664 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.665 DEBUG:boto:Method: GET 2023-10-11T15:01:27.665 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.666 DEBUG:boto:Data: 2023-10-11T15:01:27.666 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.667 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.667 DEBUG:boto:Params: {} 2023-10-11T15:01:27.668 DEBUG:boto:Token: None 2023-10-11T15:01:27.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a91a1bb73dc7212-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.672 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.673 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.674 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.674 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.674 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.675 DEBUG:boto:Method: GET 2023-10-11T15:01:27.675 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.677 DEBUG:boto:Data: 2023-10-11T15:01:27.677 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.678 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.679 DEBUG:boto:Params: {} 2023-10-11T15:01:27.679 DEBUG:boto:Token: None 2023-10-11T15:01:27.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000734e58b56c0c437b-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.683 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.684 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.684 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.685 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.685 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:01:27.686 DEBUG:boto:Data: 2023-10-11T15:01:27.686 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.687 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.687 DEBUG:boto:Params: {} 2023-10-11T15:01:27.688 DEBUG:boto:Token: None 2023-10-11T15:01:27.688 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1 2023-10-11T15:01:27.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ud4/sXaXN/UNuZ18uJ6wqt9U7Gc= 2023-10-11T15:01:27.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ud4/sXaXN/UNuZ18uJ6wqt9U7Gc=', 'Content-Length': '0'} 2023-10-11T15:01:27.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8a8e5d84e740b47-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.691 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.692 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.692 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.692 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:01:27.693 DEBUG:boto:Data: 2023-10-11T15:01:27.693 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.694 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.695 DEBUG:boto:Params: {} 2023-10-11T15:01:27.695 DEBUG:boto:Token: None 2023-10-11T15:01:27.696 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1 2023-10-11T15:01:27.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ud4/sXaXN/UNuZ18uJ6wqt9U7Gc= 2023-10-11T15:01:27.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ud4/sXaXN/UNuZ18uJ6wqt9U7Gc=', 'Content-Length': '0'} 2023-10-11T15:01:27.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005194e5b6c69bfd4-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-10-11T15:01:27.699 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.699 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.700 DEBUG:boto:Method: GET 2023-10-11T15:01:27.700 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:01:27.700 DEBUG:boto:Data: 2023-10-11T15:01:27.701 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.702 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.702 DEBUG:boto:Params: {} 2023-10-11T15:01:27.703 DEBUG:boto:Token: None 2023-10-11T15:01:27.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1 2023-10-11T15:01:27.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QbWepfW+Bv3WXoTOHL3PQWJzbzs= 2023-10-11T15:01:27.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QbWepfW+Bv3WXoTOHL3PQWJzbzs=', 'Content-Length': '0'} 2023-10-11T15:01:27.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000004dff7000426856-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.746 DEBUG:boto:path=/dghqvd-49/obj1 2023-10-11T15:01:27.747 DEBUG:boto:auth_path=/dghqvd-49/obj1 2023-10-11T15:01:27.747 DEBUG:boto:Method: GET 2023-10-11T15:01:27.748 DEBUG:boto:Path: /dghqvd-49/obj1 2023-10-11T15:01:27.748 DEBUG:boto:Data: 2023-10-11T15:01:27.748 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.749 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.750 DEBUG:boto:Params: {} 2023-10-11T15:01:27.750 DEBUG:boto:Token: None 2023-10-11T15:01:27.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj1 2023-10-11T15:01:27.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QbWepfW+Bv3WXoTOHL3PQWJzbzs= 2023-10-11T15:01:27.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QbWepfW+Bv3WXoTOHL3PQWJzbzs=', 'Content-Length': '0'} 2023-10-11T15:01:27.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec8cc90e6bb1f34-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.754 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.754 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.755 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.755 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.755 DEBUG:boto:Method: GET 2023-10-11T15:01:27.756 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.756 DEBUG:boto:Data: 2023-10-11T15:01:27.757 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.758 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.758 DEBUG:boto:Params: {} 2023-10-11T15:01:27.759 DEBUG:boto:Token: None 2023-10-11T15:01:27.759 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96d2c29b72bb1a7-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.763 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.763 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.764 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.764 DEBUG:boto:path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.765 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.765 DEBUG:boto:Method: GET 2023-10-11T15:01:27.766 DEBUG:boto:Path: /dghqvd-49/?prefix=obj1 2023-10-11T15:01:27.766 DEBUG:boto:Data: 2023-10-11T15:01:27.766 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.767 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.767 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.768 DEBUG:boto:Params: {} 2023-10-11T15:01:27.768 DEBUG:boto:Token: None 2023-10-11T15:01:27.769 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66b84921fedc018-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.773 DEBUG:boto:b'dghqvd-49obj11000falseobj12023-10-11T14:59:31.449Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:27.774 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.774 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.775 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.775 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.776 DEBUG:boto:Method: GET 2023-10-11T15:01:27.776 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.776 DEBUG:boto:Data: 2023-10-11T15:01:27.777 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.778 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.778 DEBUG:boto:Params: {} 2023-10-11T15:01:27.778 DEBUG:boto:Token: None 2023-10-11T15:01:27.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU= 2023-10-11T15:01:27.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU=', 'Content-Length': '0'} 2023-10-11T15:01:27.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008173b4f6a10ec314-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.782 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.783 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.783 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.783 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.784 DEBUG:boto:Method: GET 2023-10-11T15:01:27.784 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.785 DEBUG:boto:Data: 2023-10-11T15:01:27.785 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.786 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.786 DEBUG:boto:Params: {} 2023-10-11T15:01:27.787 DEBUG:boto:Token: None 2023-10-11T15:01:27.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU= 2023-10-11T15:01:27.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU=', 'Content-Length': '0'} 2023-10-11T15:01:27.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c837637237f01da-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.791 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.791 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.792 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.792 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.792 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.793 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.793 DEBUG:boto:Data: 2023-10-11T15:01:27.793 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.794 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.795 DEBUG:boto:Params: {} 2023-10-11T15:01:27.795 DEBUG:boto:Token: None 2023-10-11T15:01:27.795 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NnusGWoghInk/j001bXmgW1ukgI= 2023-10-11T15:01:27.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NnusGWoghInk/j001bXmgW1ukgI=', 'Content-Length': '0'} 2023-10-11T15:01:27.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c828ec4fccc45b47-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.799 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.799 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.799 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.800 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.800 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.801 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.801 DEBUG:boto:Data: 2023-10-11T15:01:27.802 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.802 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.802 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.803 DEBUG:boto:Params: {} 2023-10-11T15:01:27.803 DEBUG:boto:Token: None 2023-10-11T15:01:27.804 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NnusGWoghInk/j001bXmgW1ukgI= 2023-10-11T15:01:27.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NnusGWoghInk/j001bXmgW1ukgI=', 'Content-Length': '0'} 2023-10-11T15:01:27.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b53b5c68c3753870-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:27.807 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.808 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.808 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.808 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.809 DEBUG:boto:Method: GET 2023-10-11T15:01:27.809 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.810 DEBUG:boto:Data: 2023-10-11T15:01:27.810 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.811 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.811 DEBUG:boto:Params: {} 2023-10-11T15:01:27.812 DEBUG:boto:Token: None 2023-10-11T15:01:27.812 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU= 2023-10-11T15:01:27.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU=', 'Content-Length': '0'} 2023-10-11T15:01:27.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005248a1ce73cccc30-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.815 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.816 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.816 DEBUG:boto:path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.817 DEBUG:boto:auth_path=/dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.817 DEBUG:boto:Method: GET 2023-10-11T15:01:27.817 DEBUG:boto:Path: /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.818 DEBUG:boto:Data: 2023-10-11T15:01:27.818 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.819 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.819 DEBUG:boto:Params: {} 2023-10-11T15:01:27.820 DEBUG:boto:Token: None 2023-10-11T15:01:27.820 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2?versionId=null 2023-10-11T15:01:27.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU= 2023-10-11T15:01:27.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lyK85S+s3e+kNacY28g1UL3P3uU=', 'Content-Length': '0'} 2023-10-11T15:01:27.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f0adefd29409fae-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.824 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.824 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.825 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.825 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.825 DEBUG:boto:Method: GET 2023-10-11T15:01:27.826 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.826 DEBUG:boto:Data: 2023-10-11T15:01:27.827 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.827 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.828 DEBUG:boto:Params: {} 2023-10-11T15:01:27.828 DEBUG:boto:Token: None 2023-10-11T15:01:27.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd875df5ae64000-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.874 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.875 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.875 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.876 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.876 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.877 DEBUG:boto:Method: GET 2023-10-11T15:01:27.877 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.878 DEBUG:boto:Data: 2023-10-11T15:01:27.878 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.878 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.879 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.879 DEBUG:boto:Params: {} 2023-10-11T15:01:27.880 DEBUG:boto:Token: None 2023-10-11T15:01:27.880 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071acac79382d82d1-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.884 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.884 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.885 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.885 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.886 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:01:27.886 DEBUG:boto:Data: 2023-10-11T15:01:27.887 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.887 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.888 DEBUG:boto:Params: {} 2023-10-11T15:01:27.888 DEBUG:boto:Token: None 2023-10-11T15:01:27.889 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2 2023-10-11T15:01:27.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7YqMuQjVpEuIDR/fTILoqUIzhw= 2023-10-11T15:01:27.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7YqMuQjVpEuIDR/fTILoqUIzhw=', 'Content-Length': '0'} 2023-10-11T15:01:27.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5f56d240fcdafd3-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.892 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.892 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.893 DEBUG:boto:Method: HEAD 2023-10-11T15:01:27.893 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:01:27.894 DEBUG:boto:Data: 2023-10-11T15:01:27.894 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.895 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.895 DEBUG:boto:Params: {} 2023-10-11T15:01:27.896 DEBUG:boto:Token: None 2023-10-11T15:01:27.896 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2 2023-10-11T15:01:27.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7YqMuQjVpEuIDR/fTILoqUIzhw= 2023-10-11T15:01:27.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7YqMuQjVpEuIDR/fTILoqUIzhw=', 'Content-Length': '0'} 2023-10-11T15:01:27.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059614ea1a15f031b-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-10-11T15:01:27.899 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.900 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.900 DEBUG:boto:Method: GET 2023-10-11T15:01:27.901 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:01:27.901 DEBUG:boto:Data: 2023-10-11T15:01:27.901 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.902 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.903 DEBUG:boto:Params: {} 2023-10-11T15:01:27.903 DEBUG:boto:Token: None 2023-10-11T15:01:27.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2 2023-10-11T15:01:27.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d4WyUKfSAWbK9iOFaMWOqeTPBis= 2023-10-11T15:01:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d4WyUKfSAWbK9iOFaMWOqeTPBis=', 'Content-Length': '0'} 2023-10-11T15:01:27.906 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000140773f91a80472b-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.907 DEBUG:boto:path=/dghqvd-49/obj2 2023-10-11T15:01:27.907 DEBUG:boto:auth_path=/dghqvd-49/obj2 2023-10-11T15:01:27.908 DEBUG:boto:Method: GET 2023-10-11T15:01:27.908 DEBUG:boto:Path: /dghqvd-49/obj2 2023-10-11T15:01:27.909 DEBUG:boto:Data: 2023-10-11T15:01:27.909 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.910 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.910 DEBUG:boto:Params: {} 2023-10-11T15:01:27.911 DEBUG:boto:Token: None 2023-10-11T15:01:27.911 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj2 2023-10-11T15:01:27.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d4WyUKfSAWbK9iOFaMWOqeTPBis= 2023-10-11T15:01:27.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d4WyUKfSAWbK9iOFaMWOqeTPBis=', 'Content-Length': '0'} 2023-10-11T15:01:27.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5dad9adaeaa1016-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.962 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.963 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.963 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.964 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.964 DEBUG:boto:Method: GET 2023-10-11T15:01:27.965 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.965 DEBUG:boto:Data: 2023-10-11T15:01:27.966 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.967 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.968 DEBUG:boto:Params: {} 2023-10-11T15:01:27.968 DEBUG:boto:Token: None 2023-10-11T15:01:27.969 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000658876e22e4b7f4-006526b8c7-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.973 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.973 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:27.974 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:27.974 DEBUG:boto:path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.975 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.975 DEBUG:boto:Method: GET 2023-10-11T15:01:27.976 DEBUG:boto:Path: /dghqvd-49/?prefix=obj2 2023-10-11T15:01:27.976 DEBUG:boto:Data: 2023-10-11T15:01:27.977 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.977 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.978 DEBUG:boto:Params: {} 2023-10-11T15:01:27.978 DEBUG:boto:Token: None 2023-10-11T15:01:27.979 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/ 2023-10-11T15:01:27.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE= 2023-10-11T15:01:27.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwx/8bIfRLOgOz8+XBTE/S5UDWE=', 'Content-Length': '0'} 2023-10-11T15:01:27.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d7f94300fb669e6-006526b8c7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.982 DEBUG:boto:b'dghqvd-49obj21000falseobj22023-10-11T14:59:31.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:27.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:27.984 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:27.984 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:27.984 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.985 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.985 DEBUG:boto:Method: GET 2023-10-11T15:01:27.986 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.986 DEBUG:boto:Data: 2023-10-11T15:01:27.986 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:27.987 DEBUG:boto:Port: 8001 2023-10-11T15:01:27.988 DEBUG:boto:Params: {} 2023-10-11T15:01:27.988 DEBUG:boto:Token: None 2023-10-11T15:01:27.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6T04FA+qEJBaq4Ug8LYDmZ9p+3Q= 2023-10-11T15:01:27.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6T04FA+qEJBaq4Ug8LYDmZ9p+3Q=', 'Content-Length': '0'} 2023-10-11T15:01:27.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b03584ee52a4df8-006526b8c7-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:27.992 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:27.993 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:27.993 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.993 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.994 DEBUG:boto:Method: GET 2023-10-11T15:01:27.994 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.995 DEBUG:boto:Data: 2023-10-11T15:01:27.995 DEBUG:boto:Headers: {} 2023-10-11T15:01:27.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:27.996 DEBUG:boto:Port: 8000 2023-10-11T15:01:27.996 DEBUG:boto:Params: {} 2023-10-11T15:01:27.997 DEBUG:boto:Token: None 2023-10-11T15:01:27.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:27 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:27.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6T04FA+qEJBaq4Ug8LYDmZ9p+3Q= 2023-10-11T15:01:27.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6T04FA+qEJBaq4Ug8LYDmZ9p+3Q=', 'Content-Length': '0'} 2023-10-11T15:01:28.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064c479a42b0bc502-006526b8c7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.001 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.001 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.002 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.002 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.003 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.003 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.003 DEBUG:boto:Data: 2023-10-11T15:01:28.004 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.004 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.004 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.005 DEBUG:boto:Params: {} 2023-10-11T15:01:28.005 DEBUG:boto:Token: None 2023-10-11T15:01:28.006 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.006 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yRIVfzg9W7mjlyzvlqGfjb1HfVI= 2023-10-11T15:01:28.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yRIVfzg9W7mjlyzvlqGfjb1HfVI=', 'Content-Length': '0'} 2023-10-11T15:01:28.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025eacde79d3da59b-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.009 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.010 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.010 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.011 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.011 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.011 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.012 DEBUG:boto:Data: 2023-10-11T15:01:28.012 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.013 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.013 DEBUG:boto:Params: {} 2023-10-11T15:01:28.014 DEBUG:boto:Token: None 2023-10-11T15:01:28.014 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yRIVfzg9W7mjlyzvlqGfjb1HfVI= 2023-10-11T15:01:28.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yRIVfzg9W7mjlyzvlqGfjb1HfVI=', 'Content-Length': '0'} 2023-10-11T15:01:28.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004478c84e996b5f05-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:28.018 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.018 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.019 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.019 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.019 DEBUG:boto:Method: GET 2023-10-11T15:01:28.020 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.020 DEBUG:boto:Data: 2023-10-11T15:01:28.021 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.021 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.022 DEBUG:boto:Params: {} 2023-10-11T15:01:28.022 DEBUG:boto:Token: None 2023-10-11T15:01:28.023 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQbbIKAuK+6Xgc22loX6w8sMO8U= 2023-10-11T15:01:28.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQbbIKAuK+6Xgc22loX6w8sMO8U=', 'Content-Length': '0'} 2023-10-11T15:01:28.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008937a3a4a2d4d357-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.026 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.026 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.027 DEBUG:boto:path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.027 DEBUG:boto:auth_path=/dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.028 DEBUG:boto:Method: GET 2023-10-11T15:01:28.028 DEBUG:boto:Path: /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.029 DEBUG:boto:Data: 2023-10-11T15:01:28.029 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.030 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.030 DEBUG:boto:Params: {} 2023-10-11T15:01:28.031 DEBUG:boto:Token: None 2023-10-11T15:01:28.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3?versionId=null 2023-10-11T15:01:28.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQbbIKAuK+6Xgc22loX6w8sMO8U= 2023-10-11T15:01:28.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQbbIKAuK+6Xgc22loX6w8sMO8U=', 'Content-Length': '0'} 2023-10-11T15:01:28.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df68f21b483d8419-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.035 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.035 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.036 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.036 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.037 DEBUG:boto:Method: GET 2023-10-11T15:01:28.037 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.038 DEBUG:boto:Data: 2023-10-11T15:01:28.038 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.039 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.039 DEBUG:boto:Params: {} 2023-10-11T15:01:28.040 DEBUG:boto:Token: None 2023-10-11T15:01:28.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a8e18f463f5ecf-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.044 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.044 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.045 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.045 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.045 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.046 DEBUG:boto:Method: GET 2023-10-11T15:01:28.046 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.047 DEBUG:boto:Data: 2023-10-11T15:01:28.047 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.048 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.048 DEBUG:boto:Params: {} 2023-10-11T15:01:28.049 DEBUG:boto:Token: None 2023-10-11T15:01:28.049 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c3e9644b109eff-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.053 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.054 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.054 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.055 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.055 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:01:28.055 DEBUG:boto:Data: 2023-10-11T15:01:28.056 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.057 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.057 DEBUG:boto:Params: {} 2023-10-11T15:01:28.057 DEBUG:boto:Token: None 2023-10-11T15:01:28.058 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3 2023-10-11T15:01:28.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jVuXllKtMDMmp/Gf9XN5FdrOj5U= 2023-10-11T15:01:28.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jVuXllKtMDMmp/Gf9XN5FdrOj5U=', 'Content-Length': '0'} 2023-10-11T15:01:28.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e040bb4972903642-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.062 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.062 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.063 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.063 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:01:28.063 DEBUG:boto:Data: 2023-10-11T15:01:28.064 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.065 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.065 DEBUG:boto:Params: {} 2023-10-11T15:01:28.065 DEBUG:boto:Token: None 2023-10-11T15:01:28.066 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3 2023-10-11T15:01:28.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jVuXllKtMDMmp/Gf9XN5FdrOj5U= 2023-10-11T15:01:28.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jVuXllKtMDMmp/Gf9XN5FdrOj5U=', 'Content-Length': '0'} 2023-10-11T15:01:28.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d22ea1969e4700f-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-10-11T15:01:28.070 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.070 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.070 DEBUG:boto:Method: GET 2023-10-11T15:01:28.071 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:01:28.071 DEBUG:boto:Data: 2023-10-11T15:01:28.072 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.073 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.073 DEBUG:boto:Params: {} 2023-10-11T15:01:28.074 DEBUG:boto:Token: None 2023-10-11T15:01:28.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3 2023-10-11T15:01:28.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NcLtIyB8dU0MOM/TYgUASIKSoxo= 2023-10-11T15:01:28.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NcLtIyB8dU0MOM/TYgUASIKSoxo=', 'Content-Length': '0'} 2023-10-11T15:01:28.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d9f905c170024cf-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.078 DEBUG:boto:path=/dghqvd-49/obj3 2023-10-11T15:01:28.078 DEBUG:boto:auth_path=/dghqvd-49/obj3 2023-10-11T15:01:28.078 DEBUG:boto:Method: GET 2023-10-11T15:01:28.079 DEBUG:boto:Path: /dghqvd-49/obj3 2023-10-11T15:01:28.079 DEBUG:boto:Data: 2023-10-11T15:01:28.080 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.081 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.081 DEBUG:boto:Params: {} 2023-10-11T15:01:28.081 DEBUG:boto:Token: None 2023-10-11T15:01:28.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj3 2023-10-11T15:01:28.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NcLtIyB8dU0MOM/TYgUASIKSoxo= 2023-10-11T15:01:28.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NcLtIyB8dU0MOM/TYgUASIKSoxo=', 'Content-Length': '0'} 2023-10-11T15:01:28.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6d68058fdb15d0c-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.085 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.086 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.086 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.087 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.087 DEBUG:boto:Method: GET 2023-10-11T15:01:28.088 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.088 DEBUG:boto:Data: 2023-10-11T15:01:28.089 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.089 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.090 DEBUG:boto:Params: {} 2023-10-11T15:01:28.090 DEBUG:boto:Token: None 2023-10-11T15:01:28.091 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cbcaae0b9fa951b-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.094 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.095 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.095 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.096 DEBUG:boto:path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.096 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.097 DEBUG:boto:Method: GET 2023-10-11T15:01:28.097 DEBUG:boto:Path: /dghqvd-49/?prefix=obj3 2023-10-11T15:01:28.097 DEBUG:boto:Data: 2023-10-11T15:01:28.098 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.099 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.099 DEBUG:boto:Params: {} 2023-10-11T15:01:28.100 DEBUG:boto:Token: None 2023-10-11T15:01:28.100 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d1f561b7458c73-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.103 DEBUG:boto:b'dghqvd-49obj31000falseobj32023-10-11T14:59:31.489Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:28.104 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.105 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.105 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.105 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.106 DEBUG:boto:Method: GET 2023-10-11T15:01:28.106 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.107 DEBUG:boto:Data: 2023-10-11T15:01:28.107 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.108 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.108 DEBUG:boto:Params: {} 2023-10-11T15:01:28.109 DEBUG:boto:Token: None 2023-10-11T15:01:28.109 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0= 2023-10-11T15:01:28.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0=', 'Content-Length': '0'} 2023-10-11T15:01:28.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be7d5343b1d8ae3d-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.112 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.113 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.113 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.114 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.114 DEBUG:boto:Method: GET 2023-10-11T15:01:28.114 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.115 DEBUG:boto:Data: 2023-10-11T15:01:28.115 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.116 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.117 DEBUG:boto:Params: {} 2023-10-11T15:01:28.117 DEBUG:boto:Token: None 2023-10-11T15:01:28.118 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0= 2023-10-11T15:01:28.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0=', 'Content-Length': '0'} 2023-10-11T15:01:28.120 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a905b337465dad75-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.162 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.163 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.163 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.164 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.164 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.164 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.165 DEBUG:boto:Data: 2023-10-11T15:01:28.165 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.166 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.166 DEBUG:boto:Params: {} 2023-10-11T15:01:28.167 DEBUG:boto:Token: None 2023-10-11T15:01:28.167 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:39FMj8tpKdJ4dtpL/p/l/o8t/mo= 2023-10-11T15:01:28.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:39FMj8tpKdJ4dtpL/p/l/o8t/mo=', 'Content-Length': '0'} 2023-10-11T15:01:28.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e43b66855ea56daf-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.171 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.171 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.172 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.172 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.172 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.173 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.173 DEBUG:boto:Data: 2023-10-11T15:01:28.174 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.174 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.175 DEBUG:boto:Params: {} 2023-10-11T15:01:28.175 DEBUG:boto:Token: None 2023-10-11T15:01:28.176 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:39FMj8tpKdJ4dtpL/p/l/o8t/mo= 2023-10-11T15:01:28.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:39FMj8tpKdJ4dtpL/p/l/o8t/mo=', 'Content-Length': '0'} 2023-10-11T15:01:28.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025c28b18f76aaa41-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:28.179 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.180 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.180 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.181 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.181 DEBUG:boto:Method: GET 2023-10-11T15:01:28.181 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.182 DEBUG:boto:Data: 2023-10-11T15:01:28.182 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.183 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.183 DEBUG:boto:Params: {} 2023-10-11T15:01:28.184 DEBUG:boto:Token: None 2023-10-11T15:01:28.184 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0= 2023-10-11T15:01:28.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0=', 'Content-Length': '0'} 2023-10-11T15:01:28.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaaa1c14e18d9951-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.234 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.235 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.235 DEBUG:boto:path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.236 DEBUG:boto:auth_path=/dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.236 DEBUG:boto:Method: GET 2023-10-11T15:01:28.237 DEBUG:boto:Path: /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.237 DEBUG:boto:Data: 2023-10-11T15:01:28.237 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.238 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.239 DEBUG:boto:Params: {} 2023-10-11T15:01:28.239 DEBUG:boto:Token: None 2023-10-11T15:01:28.240 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4?versionId=null 2023-10-11T15:01:28.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0= 2023-10-11T15:01:28.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:idrp7P5OBdcA1tMP7A3rpE7lnp0=', 'Content-Length': '0'} 2023-10-11T15:01:28.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000302ea6a6ccc455fe-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.243 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.244 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.244 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.245 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.245 DEBUG:boto:Method: GET 2023-10-11T15:01:28.246 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.246 DEBUG:boto:Data: 2023-10-11T15:01:28.246 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.247 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.248 DEBUG:boto:Params: {} 2023-10-11T15:01:28.248 DEBUG:boto:Token: None 2023-10-11T15:01:28.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af55260b41176136-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.252 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.253 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.254 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.254 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.254 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.255 DEBUG:boto:Method: GET 2023-10-11T15:01:28.255 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.256 DEBUG:boto:Data: 2023-10-11T15:01:28.256 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.257 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.257 DEBUG:boto:Params: {} 2023-10-11T15:01:28.258 DEBUG:boto:Token: None 2023-10-11T15:01:28.258 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004832ec9dedce9053-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.262 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.263 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.263 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.264 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.264 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:01:28.264 DEBUG:boto:Data: 2023-10-11T15:01:28.265 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.266 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.266 DEBUG:boto:Params: {} 2023-10-11T15:01:28.266 DEBUG:boto:Token: None 2023-10-11T15:01:28.267 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4 2023-10-11T15:01:28.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fAJE8YlEWkj9I86pNrC3oMu4tZ4= 2023-10-11T15:01:28.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fAJE8YlEWkj9I86pNrC3oMu4tZ4=', 'Content-Length': '0'} 2023-10-11T15:01:28.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe53eba0f5d9568c-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.270 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.271 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.271 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.272 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:01:28.272 DEBUG:boto:Data: 2023-10-11T15:01:28.272 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.273 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.274 DEBUG:boto:Params: {} 2023-10-11T15:01:28.274 DEBUG:boto:Token: None 2023-10-11T15:01:28.274 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4 2023-10-11T15:01:28.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fAJE8YlEWkj9I86pNrC3oMu4tZ4= 2023-10-11T15:01:28.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fAJE8YlEWkj9I86pNrC3oMu4tZ4=', 'Content-Length': '0'} 2023-10-11T15:01:28.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c95642b3571960b-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-10-11T15:01:28.278 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.278 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.279 DEBUG:boto:Method: GET 2023-10-11T15:01:28.279 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:01:28.280 DEBUG:boto:Data: 2023-10-11T15:01:28.280 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.281 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.281 DEBUG:boto:Params: {} 2023-10-11T15:01:28.282 DEBUG:boto:Token: None 2023-10-11T15:01:28.282 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4 2023-10-11T15:01:28.283 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d9f6M9Uh7BPQNX05sVke/z3vFls= 2023-10-11T15:01:28.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d9f6M9Uh7BPQNX05sVke/z3vFls=', 'Content-Length': '0'} 2023-10-11T15:01:28.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b4ea505a32e2365-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.285 DEBUG:boto:path=/dghqvd-49/obj4 2023-10-11T15:01:28.286 DEBUG:boto:auth_path=/dghqvd-49/obj4 2023-10-11T15:01:28.286 DEBUG:boto:Method: GET 2023-10-11T15:01:28.287 DEBUG:boto:Path: /dghqvd-49/obj4 2023-10-11T15:01:28.287 DEBUG:boto:Data: 2023-10-11T15:01:28.289 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.290 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.290 DEBUG:boto:Params: {} 2023-10-11T15:01:28.291 DEBUG:boto:Token: None 2023-10-11T15:01:28.291 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj4 2023-10-11T15:01:28.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d9f6M9Uh7BPQNX05sVke/z3vFls= 2023-10-11T15:01:28.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d9f6M9Uh7BPQNX05sVke/z3vFls=', 'Content-Length': '0'} 2023-10-11T15:01:28.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 14:59:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb59d3afff4e046b-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.295 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.295 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.296 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.296 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.297 DEBUG:boto:Method: GET 2023-10-11T15:01:28.297 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.298 DEBUG:boto:Data: 2023-10-11T15:01:28.298 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.299 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.299 DEBUG:boto:Params: {} 2023-10-11T15:01:28.300 DEBUG:boto:Token: None 2023-10-11T15:01:28.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6429f9646eff3bd-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.304 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.305 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.305 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.306 DEBUG:boto:path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.306 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.307 DEBUG:boto:Method: GET 2023-10-11T15:01:28.307 DEBUG:boto:Path: /dghqvd-49/?prefix=obj4 2023-10-11T15:01:28.308 DEBUG:boto:Data: 2023-10-11T15:01:28.308 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.308 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.309 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.309 DEBUG:boto:Params: {} 2023-10-11T15:01:28.310 DEBUG:boto:Token: None 2023-10-11T15:01:28.310 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b2c407d5fdcdba-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.354 DEBUG:boto:b'dghqvd-49obj41000falseobj42023-10-11T14:59:31.544Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:28.356 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.356 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.356 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.357 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.357 DEBUG:boto:Method: GET 2023-10-11T15:01:28.358 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.358 DEBUG:boto:Data: 2023-10-11T15:01:28.359 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.360 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.360 DEBUG:boto:Params: {} 2023-10-11T15:01:28.360 DEBUG:boto:Token: None 2023-10-11T15:01:28.361 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4= 2023-10-11T15:01:28.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4=', 'Content-Length': '0'} 2023-10-11T15:01:28.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000278c9125dce8d133-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.365 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.365 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.366 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.366 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.367 DEBUG:boto:Method: GET 2023-10-11T15:01:28.367 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.368 DEBUG:boto:Data: 2023-10-11T15:01:28.368 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.369 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.369 DEBUG:boto:Params: {} 2023-10-11T15:01:28.370 DEBUG:boto:Token: None 2023-10-11T15:01:28.370 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.371 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4= 2023-10-11T15:01:28.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4=', 'Content-Length': '0'} 2023-10-11T15:01:28.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d38ccc6e33530ced-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.374 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.375 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.375 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.376 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.376 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.376 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.377 DEBUG:boto:Data: 2023-10-11T15:01:28.377 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.378 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.379 DEBUG:boto:Params: {} 2023-10-11T15:01:28.379 DEBUG:boto:Token: None 2023-10-11T15:01:28.380 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KqT14uVKmBo1Lsyr+hMp55SwVFE= 2023-10-11T15:01:28.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KqT14uVKmBo1Lsyr+hMp55SwVFE=', 'Content-Length': '0'} 2023-10-11T15:01:28.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f6f8045e4e0b131-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.383 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.384 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.384 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.384 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.385 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.385 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.386 DEBUG:boto:Data: 2023-10-11T15:01:28.386 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.387 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.387 DEBUG:boto:Params: {} 2023-10-11T15:01:28.388 DEBUG:boto:Token: None 2023-10-11T15:01:28.388 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KqT14uVKmBo1Lsyr+hMp55SwVFE= 2023-10-11T15:01:28.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KqT14uVKmBo1Lsyr+hMp55SwVFE=', 'Content-Length': '0'} 2023-10-11T15:01:28.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f6db62f59f2b2c5-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:28.392 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.392 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.393 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.393 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.394 DEBUG:boto:Method: GET 2023-10-11T15:01:28.394 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.395 DEBUG:boto:Data: 2023-10-11T15:01:28.395 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.396 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.397 DEBUG:boto:Params: {} 2023-10-11T15:01:28.397 DEBUG:boto:Token: None 2023-10-11T15:01:28.398 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4= 2023-10-11T15:01:28.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4=', 'Content-Length': '0'} 2023-10-11T15:01:28.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4aab460f3a983ba-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.401 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.402 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.402 DEBUG:boto:path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.402 DEBUG:boto:auth_path=/dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.403 DEBUG:boto:Method: GET 2023-10-11T15:01:28.403 DEBUG:boto:Path: /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.404 DEBUG:boto:Data: 2023-10-11T15:01:28.404 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.405 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.406 DEBUG:boto:Params: {} 2023-10-11T15:01:28.406 DEBUG:boto:Token: None 2023-10-11T15:01:28.407 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5?versionId=null 2023-10-11T15:01:28.407 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4= 2023-10-11T15:01:28.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+KHX15twuSL1/MAQ8zm4VFB8CI4=', 'Content-Length': '0'} 2023-10-11T15:01:28.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef9792e75200b550-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.410 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.411 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.411 DEBUG:boto:path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.412 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.412 DEBUG:boto:Method: GET 2023-10-11T15:01:28.412 DEBUG:boto:Path: /dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.413 DEBUG:boto:Data: 2023-10-11T15:01:28.413 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.414 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.415 DEBUG:boto:Params: {} 2023-10-11T15:01:28.415 DEBUG:boto:Token: None 2023-10-11T15:01:28.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006140d5e24a322553-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.420 DEBUG:boto:b'dghqvd-49obj51000falseobj52023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.421 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.421 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.421 DEBUG:boto:path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.422 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.422 DEBUG:boto:Method: GET 2023-10-11T15:01:28.423 DEBUG:boto:Path: /dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.423 DEBUG:boto:Data: 2023-10-11T15:01:28.424 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.425 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.425 DEBUG:boto:Params: {} 2023-10-11T15:01:28.426 DEBUG:boto:Token: None 2023-10-11T15:01:28.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e29f79411dff2a-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.430 DEBUG:boto:b'dghqvd-49obj51000falseobj52023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.431 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.431 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.431 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.432 DEBUG:boto:Path: /dghqvd-49/obj5 2023-10-11T15:01:28.432 DEBUG:boto:Data: 2023-10-11T15:01:28.433 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.433 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.434 DEBUG:boto:Params: {} 2023-10-11T15:01:28.434 DEBUG:boto:Token: None 2023-10-11T15:01:28.435 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5 2023-10-11T15:01:28.435 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D7Io3UlfN61zbjeGr48F/l8sY+s= 2023-10-11T15:01:28.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D7Io3UlfN61zbjeGr48F/l8sY+s=', 'Content-Length': '0'} 2023-10-11T15:01:28.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011d101e09a60c051-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.438 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.439 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.439 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.439 DEBUG:boto:Path: /dghqvd-49/obj5 2023-10-11T15:01:28.440 DEBUG:boto:Data: 2023-10-11T15:01:28.440 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.441 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.441 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.441 DEBUG:boto:Params: {} 2023-10-11T15:01:28.442 DEBUG:boto:Token: None 2023-10-11T15:01:28.442 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5 2023-10-11T15:01:28.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D7Io3UlfN61zbjeGr48F/l8sY+s= 2023-10-11T15:01:28.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D7Io3UlfN61zbjeGr48F/l8sY+s=', 'Content-Length': '0'} 2023-10-11T15:01:28.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd6ecc1414789cfd-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-10-11T15:01:28.446 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.447 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.447 DEBUG:boto:Method: GET 2023-10-11T15:01:28.447 DEBUG:boto:Path: /dghqvd-49/obj5 2023-10-11T15:01:28.448 DEBUG:boto:Data: 2023-10-11T15:01:28.448 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.449 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.449 DEBUG:boto:Params: {} 2023-10-11T15:01:28.450 DEBUG:boto:Token: None 2023-10-11T15:01:28.450 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5 2023-10-11T15:01:28.451 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sVI8uGdMj1SUe5VEhHxPl8+Da3U= 2023-10-11T15:01:28.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sVI8uGdMj1SUe5VEhHxPl8+Da3U=', 'Content-Length': '0'} 2023-10-11T15:01:28.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9a27578ab155985-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.494 DEBUG:boto:path=/dghqvd-49/obj5 2023-10-11T15:01:28.495 DEBUG:boto:auth_path=/dghqvd-49/obj5 2023-10-11T15:01:28.495 DEBUG:boto:Method: GET 2023-10-11T15:01:28.496 DEBUG:boto:Path: /dghqvd-49/obj5 2023-10-11T15:01:28.496 DEBUG:boto:Data: 2023-10-11T15:01:28.496 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.497 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.498 DEBUG:boto:Params: {} 2023-10-11T15:01:28.498 DEBUG:boto:Token: None 2023-10-11T15:01:28.499 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj5 2023-10-11T15:01:28.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sVI8uGdMj1SUe5VEhHxPl8+Da3U= 2023-10-11T15:01:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sVI8uGdMj1SUe5VEhHxPl8+Da3U=', 'Content-Length': '0'} 2023-10-11T15:01:28.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002332e952ba1d50bf-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.502 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.503 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.503 DEBUG:boto:path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.503 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.504 DEBUG:boto:Method: GET 2023-10-11T15:01:28.504 DEBUG:boto:Path: /dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.505 DEBUG:boto:Data: 2023-10-11T15:01:28.505 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.506 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.506 DEBUG:boto:Params: {} 2023-10-11T15:01:28.507 DEBUG:boto:Token: None 2023-10-11T15:01:28.508 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457bfa6d86fdba05-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.511 DEBUG:boto:b'dghqvd-49obj51000falseobj52023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.512 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.513 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.513 DEBUG:boto:path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.514 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.514 DEBUG:boto:Method: GET 2023-10-11T15:01:28.514 DEBUG:boto:Path: /dghqvd-49/?prefix=obj5 2023-10-11T15:01:28.515 DEBUG:boto:Data: 2023-10-11T15:01:28.515 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.516 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.517 DEBUG:boto:Params: {} 2023-10-11T15:01:28.517 DEBUG:boto:Token: None 2023-10-11T15:01:28.517 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e4c24fc78f1b047-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.521 DEBUG:boto:b'dghqvd-49obj51000falseobj52023-10-11T15:00:39.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:28.522 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.523 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.523 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.523 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.524 DEBUG:boto:Method: GET 2023-10-11T15:01:28.524 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.525 DEBUG:boto:Data: 2023-10-11T15:01:28.525 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.526 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.526 DEBUG:boto:Params: {} 2023-10-11T15:01:28.527 DEBUG:boto:Token: None 2023-10-11T15:01:28.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88= 2023-10-11T15:01:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88=', 'Content-Length': '0'} 2023-10-11T15:01:28.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bdd7b65442aa803-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.530 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.531 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.531 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.532 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.532 DEBUG:boto:Method: GET 2023-10-11T15:01:28.533 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.533 DEBUG:boto:Data: 2023-10-11T15:01:28.533 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.534 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.535 DEBUG:boto:Params: {} 2023-10-11T15:01:28.535 DEBUG:boto:Token: None 2023-10-11T15:01:28.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88= 2023-10-11T15:01:28.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88=', 'Content-Length': '0'} 2023-10-11T15:01:28.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000dea18bad05575d-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.539 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.540 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.540 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.541 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.541 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.541 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.542 DEBUG:boto:Data: 2023-10-11T15:01:28.542 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.543 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.543 DEBUG:boto:Params: {} 2023-10-11T15:01:28.544 DEBUG:boto:Token: None 2023-10-11T15:01:28.544 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yl9dHjzbIqwvEBa9ZCLXwJVb8/o= 2023-10-11T15:01:28.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yl9dHjzbIqwvEBa9ZCLXwJVb8/o=', 'Content-Length': '0'} 2023-10-11T15:01:28.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a15542a543e2b51-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.547 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.548 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.548 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.549 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.549 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.549 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.550 DEBUG:boto:Data: 2023-10-11T15:01:28.550 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.551 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.551 DEBUG:boto:Params: {} 2023-10-11T15:01:28.552 DEBUG:boto:Token: None 2023-10-11T15:01:28.552 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yl9dHjzbIqwvEBa9ZCLXwJVb8/o= 2023-10-11T15:01:28.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yl9dHjzbIqwvEBa9ZCLXwJVb8/o=', 'Content-Length': '0'} 2023-10-11T15:01:28.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee2eae063c8f262e-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:28.556 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.556 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.557 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.557 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.558 DEBUG:boto:Method: GET 2023-10-11T15:01:28.558 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.559 DEBUG:boto:Data: 2023-10-11T15:01:28.559 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.560 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.560 DEBUG:boto:Params: {} 2023-10-11T15:01:28.561 DEBUG:boto:Token: None 2023-10-11T15:01:28.561 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.562 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88= 2023-10-11T15:01:28.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88=', 'Content-Length': '0'} 2023-10-11T15:01:28.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0eb1c02138555b-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.606 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.607 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.607 DEBUG:boto:path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.608 DEBUG:boto:auth_path=/dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.608 DEBUG:boto:Method: GET 2023-10-11T15:01:28.608 DEBUG:boto:Path: /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.609 DEBUG:boto:Data: 2023-10-11T15:01:28.609 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.610 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.611 DEBUG:boto:Params: {} 2023-10-11T15:01:28.611 DEBUG:boto:Token: None 2023-10-11T15:01:28.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6?versionId=null 2023-10-11T15:01:28.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88= 2023-10-11T15:01:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3+poJr5T8BHbflv1qg/vjnnFI88=', 'Content-Length': '0'} 2023-10-11T15:01:28.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025866fbb2409a6d7-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.615 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.616 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.616 DEBUG:boto:path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.616 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.617 DEBUG:boto:Method: GET 2023-10-11T15:01:28.617 DEBUG:boto:Path: /dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.618 DEBUG:boto:Data: 2023-10-11T15:01:28.618 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.619 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.619 DEBUG:boto:Params: {} 2023-10-11T15:01:28.620 DEBUG:boto:Token: None 2023-10-11T15:01:28.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e05b6c4ba2f9eb-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.624 DEBUG:boto:b'dghqvd-49obj61000falseobj62023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.625 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.625 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.626 DEBUG:boto:path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.626 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.626 DEBUG:boto:Method: GET 2023-10-11T15:01:28.627 DEBUG:boto:Path: /dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.627 DEBUG:boto:Data: 2023-10-11T15:01:28.628 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.629 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.629 DEBUG:boto:Params: {} 2023-10-11T15:01:28.629 DEBUG:boto:Token: None 2023-10-11T15:01:28.630 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eecc47038f7d265-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.634 DEBUG:boto:b'dghqvd-49obj61000falseobj62023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.634 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.635 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.635 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.636 DEBUG:boto:Path: /dghqvd-49/obj6 2023-10-11T15:01:28.636 DEBUG:boto:Data: 2023-10-11T15:01:28.636 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.637 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.638 DEBUG:boto:Params: {} 2023-10-11T15:01:28.638 DEBUG:boto:Token: None 2023-10-11T15:01:28.639 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6 2023-10-11T15:01:28.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mtj5rwoix+tjdm3GJxiiFDeaEsI= 2023-10-11T15:01:28.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mtj5rwoix+tjdm3GJxiiFDeaEsI=', 'Content-Length': '0'} 2023-10-11T15:01:28.641 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000445cee1850b467a7-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.642 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.642 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.643 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.643 DEBUG:boto:Path: /dghqvd-49/obj6 2023-10-11T15:01:28.644 DEBUG:boto:Data: 2023-10-11T15:01:28.644 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.644 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.645 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.645 DEBUG:boto:Params: {} 2023-10-11T15:01:28.646 DEBUG:boto:Token: None 2023-10-11T15:01:28.646 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6 2023-10-11T15:01:28.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mtj5rwoix+tjdm3GJxiiFDeaEsI= 2023-10-11T15:01:28.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mtj5rwoix+tjdm3GJxiiFDeaEsI=', 'Content-Length': '0'} 2023-10-11T15:01:28.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000244a15da66998788-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-10-11T15:01:28.650 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.650 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.651 DEBUG:boto:Method: GET 2023-10-11T15:01:28.651 DEBUG:boto:Path: /dghqvd-49/obj6 2023-10-11T15:01:28.651 DEBUG:boto:Data: 2023-10-11T15:01:28.652 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.653 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.653 DEBUG:boto:Params: {} 2023-10-11T15:01:28.654 DEBUG:boto:Token: None 2023-10-11T15:01:28.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6 2023-10-11T15:01:28.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uC5w4aH4rJjzyQDcM3DHXmZjlB8= 2023-10-11T15:01:28.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uC5w4aH4rJjzyQDcM3DHXmZjlB8=', 'Content-Length': '0'} 2023-10-11T15:01:28.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b8ad8e703224c44-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.657 DEBUG:boto:path=/dghqvd-49/obj6 2023-10-11T15:01:28.658 DEBUG:boto:auth_path=/dghqvd-49/obj6 2023-10-11T15:01:28.658 DEBUG:boto:Method: GET 2023-10-11T15:01:28.659 DEBUG:boto:Path: /dghqvd-49/obj6 2023-10-11T15:01:28.659 DEBUG:boto:Data: 2023-10-11T15:01:28.659 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.660 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.661 DEBUG:boto:Params: {} 2023-10-11T15:01:28.661 DEBUG:boto:Token: None 2023-10-11T15:01:28.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj6 2023-10-11T15:01:28.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uC5w4aH4rJjzyQDcM3DHXmZjlB8= 2023-10-11T15:01:28.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uC5w4aH4rJjzyQDcM3DHXmZjlB8=', 'Content-Length': '0'} 2023-10-11T15:01:28.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062573e21120485a7-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.665 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.665 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.666 DEBUG:boto:path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.666 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.667 DEBUG:boto:Method: GET 2023-10-11T15:01:28.667 DEBUG:boto:Path: /dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.668 DEBUG:boto:Data: 2023-10-11T15:01:28.668 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.669 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.669 DEBUG:boto:Params: {} 2023-10-11T15:01:28.670 DEBUG:boto:Token: None 2023-10-11T15:01:28.670 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.671 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cef520b3429a718-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.674 DEBUG:boto:b'dghqvd-49obj61000falseobj62023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.675 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.675 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.676 DEBUG:boto:path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.676 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.677 DEBUG:boto:Method: GET 2023-10-11T15:01:28.677 DEBUG:boto:Path: /dghqvd-49/?prefix=obj6 2023-10-11T15:01:28.677 DEBUG:boto:Data: 2023-10-11T15:01:28.678 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.679 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.679 DEBUG:boto:Params: {} 2023-10-11T15:01:28.679 DEBUG:boto:Token: None 2023-10-11T15:01:28.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005522311e1b05ea97-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.684 DEBUG:boto:b'dghqvd-49obj61000falseobj62023-10-11T15:00:39.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:28.685 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.685 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.686 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.686 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.686 DEBUG:boto:Method: GET 2023-10-11T15:01:28.687 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.687 DEBUG:boto:Data: 2023-10-11T15:01:28.688 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.689 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.689 DEBUG:boto:Params: {} 2023-10-11T15:01:28.690 DEBUG:boto:Token: None 2023-10-11T15:01:28.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I= 2023-10-11T15:01:28.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I=', 'Content-Length': '0'} 2023-10-11T15:01:28.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021935daf1d8a4752-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.693 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.694 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.694 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.695 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.695 DEBUG:boto:Method: GET 2023-10-11T15:01:28.696 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.696 DEBUG:boto:Data: 2023-10-11T15:01:28.696 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.697 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.698 DEBUG:boto:Params: {} 2023-10-11T15:01:28.698 DEBUG:boto:Token: None 2023-10-11T15:01:28.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I= 2023-10-11T15:01:28.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I=', 'Content-Length': '0'} 2023-10-11T15:01:28.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038bf4317469f28cc-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.702 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.703 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.703 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.704 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.704 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.705 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.705 DEBUG:boto:Data: 2023-10-11T15:01:28.705 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.706 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.707 DEBUG:boto:Params: {} 2023-10-11T15:01:28.707 DEBUG:boto:Token: None 2023-10-11T15:01:28.708 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKcBi9HXPnULEVuMtthI11dvQ8I= 2023-10-11T15:01:28.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKcBi9HXPnULEVuMtthI11dvQ8I=', 'Content-Length': '0'} 2023-10-11T15:01:28.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000753b8b285b6db3b3-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.711 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.711 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.711 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.712 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.712 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.713 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.713 DEBUG:boto:Data: 2023-10-11T15:01:28.713 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.714 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.715 DEBUG:boto:Params: {} 2023-10-11T15:01:28.715 DEBUG:boto:Token: None 2023-10-11T15:01:28.716 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKcBi9HXPnULEVuMtthI11dvQ8I= 2023-10-11T15:01:28.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKcBi9HXPnULEVuMtthI11dvQ8I=', 'Content-Length': '0'} 2023-10-11T15:01:28.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000268b1185ce2b0beb-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:28.720 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.720 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.720 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.721 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.721 DEBUG:boto:Method: GET 2023-10-11T15:01:28.722 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.722 DEBUG:boto:Data: 2023-10-11T15:01:28.723 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.724 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.724 DEBUG:boto:Params: {} 2023-10-11T15:01:28.724 DEBUG:boto:Token: None 2023-10-11T15:01:28.725 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I= 2023-10-11T15:01:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I=', 'Content-Length': '0'} 2023-10-11T15:01:28.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050a23c2053ab5c33-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.728 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.729 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.729 DEBUG:boto:path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.730 DEBUG:boto:auth_path=/dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.730 DEBUG:boto:Method: GET 2023-10-11T15:01:28.731 DEBUG:boto:Path: /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.731 DEBUG:boto:Data: 2023-10-11T15:01:28.732 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.732 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.733 DEBUG:boto:Params: {} 2023-10-11T15:01:28.733 DEBUG:boto:Token: None 2023-10-11T15:01:28.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7?versionId=null 2023-10-11T15:01:28.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I= 2023-10-11T15:01:28.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fKvFzd26oQvRjlE4Kk9KblPPl4I=', 'Content-Length': '0'} 2023-10-11T15:01:28.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c5c00ac66f22ec1-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.778 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.779 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.779 DEBUG:boto:path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.780 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.780 DEBUG:boto:Method: GET 2023-10-11T15:01:28.781 DEBUG:boto:Path: /dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.781 DEBUG:boto:Data: 2023-10-11T15:01:28.781 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.782 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.782 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.783 DEBUG:boto:Params: {} 2023-10-11T15:01:28.783 DEBUG:boto:Token: None 2023-10-11T15:01:28.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ce2d0fcea88ea4-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.787 DEBUG:boto:b'dghqvd-49obj71000falseobj72023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.788 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.788 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.789 DEBUG:boto:path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.789 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.789 DEBUG:boto:Method: GET 2023-10-11T15:01:28.790 DEBUG:boto:Path: /dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.790 DEBUG:boto:Data: 2023-10-11T15:01:28.791 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.792 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.792 DEBUG:boto:Params: {} 2023-10-11T15:01:28.792 DEBUG:boto:Token: None 2023-10-11T15:01:28.793 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000446bf1a583dcd71e-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.797 DEBUG:boto:b'dghqvd-49obj71000falseobj72023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.797 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.798 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.798 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.799 DEBUG:boto:Path: /dghqvd-49/obj7 2023-10-11T15:01:28.799 DEBUG:boto:Data: 2023-10-11T15:01:28.799 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.800 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.800 DEBUG:boto:Params: {} 2023-10-11T15:01:28.801 DEBUG:boto:Token: None 2023-10-11T15:01:28.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7 2023-10-11T15:01:28.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ynb00aCS0xuVV/Jg9wLBb/f5yX8= 2023-10-11T15:01:28.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ynb00aCS0xuVV/Jg9wLBb/f5yX8=', 'Content-Length': '0'} 2023-10-11T15:01:28.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0d20b59f889cd1c-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.805 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.805 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.806 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.806 DEBUG:boto:Path: /dghqvd-49/obj7 2023-10-11T15:01:28.806 DEBUG:boto:Data: 2023-10-11T15:01:28.807 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.807 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.808 DEBUG:boto:Params: {} 2023-10-11T15:01:28.808 DEBUG:boto:Token: None 2023-10-11T15:01:28.809 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7 2023-10-11T15:01:28.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ynb00aCS0xuVV/Jg9wLBb/f5yX8= 2023-10-11T15:01:28.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ynb00aCS0xuVV/Jg9wLBb/f5yX8=', 'Content-Length': '0'} 2023-10-11T15:01:28.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf91c725190ee518-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-10-11T15:01:28.812 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.813 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.813 DEBUG:boto:Method: GET 2023-10-11T15:01:28.814 DEBUG:boto:Path: /dghqvd-49/obj7 2023-10-11T15:01:28.814 DEBUG:boto:Data: 2023-10-11T15:01:28.814 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.815 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.816 DEBUG:boto:Params: {} 2023-10-11T15:01:28.816 DEBUG:boto:Token: None 2023-10-11T15:01:28.817 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7 2023-10-11T15:01:28.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pWiqS0BCWCAhY0UzoePBkYoW1EE= 2023-10-11T15:01:28.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWiqS0BCWCAhY0UzoePBkYoW1EE=', 'Content-Length': '0'} 2023-10-11T15:01:28.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d31160d6e7ba4c5-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.862 DEBUG:boto:path=/dghqvd-49/obj7 2023-10-11T15:01:28.863 DEBUG:boto:auth_path=/dghqvd-49/obj7 2023-10-11T15:01:28.863 DEBUG:boto:Method: GET 2023-10-11T15:01:28.864 DEBUG:boto:Path: /dghqvd-49/obj7 2023-10-11T15:01:28.864 DEBUG:boto:Data: 2023-10-11T15:01:28.865 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.865 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.866 DEBUG:boto:Params: {} 2023-10-11T15:01:28.866 DEBUG:boto:Token: None 2023-10-11T15:01:28.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj7 2023-10-11T15:01:28.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pWiqS0BCWCAhY0UzoePBkYoW1EE= 2023-10-11T15:01:28.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWiqS0BCWCAhY0UzoePBkYoW1EE=', 'Content-Length': '0'} 2023-10-11T15:01:28.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cdc1748f789755a-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.870 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.871 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.871 DEBUG:boto:path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.872 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.872 DEBUG:boto:Method: GET 2023-10-11T15:01:28.873 DEBUG:boto:Path: /dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.873 DEBUG:boto:Data: 2023-10-11T15:01:28.873 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.874 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.875 DEBUG:boto:Params: {} 2023-10-11T15:01:28.875 DEBUG:boto:Token: None 2023-10-11T15:01:28.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b9c52a5998c72d-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.880 DEBUG:boto:b'dghqvd-49obj71000falseobj72023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.880 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.881 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.881 DEBUG:boto:path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.882 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.882 DEBUG:boto:Method: GET 2023-10-11T15:01:28.882 DEBUG:boto:Path: /dghqvd-49/?prefix=obj7 2023-10-11T15:01:28.883 DEBUG:boto:Data: 2023-10-11T15:01:28.883 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.884 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.884 DEBUG:boto:Params: {} 2023-10-11T15:01:28.885 DEBUG:boto:Token: None 2023-10-11T15:01:28.885 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d042868a416acc-006526b8c8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.889 DEBUG:boto:b'dghqvd-49obj71000falseobj72023-10-11T15:00:39.962Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:28.890 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.891 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.891 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.892 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.892 DEBUG:boto:Method: GET 2023-10-11T15:01:28.892 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.893 DEBUG:boto:Data: 2023-10-11T15:01:28.893 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.894 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.895 DEBUG:boto:Params: {} 2023-10-11T15:01:28.895 DEBUG:boto:Token: None 2023-10-11T15:01:28.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g= 2023-10-11T15:01:28.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g=', 'Content-Length': '0'} 2023-10-11T15:01:28.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd494514358a44c6-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.899 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.899 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.900 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.900 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.901 DEBUG:boto:Method: GET 2023-10-11T15:01:28.901 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.902 DEBUG:boto:Data: 2023-10-11T15:01:28.902 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.903 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.903 DEBUG:boto:Params: {} 2023-10-11T15:01:28.904 DEBUG:boto:Token: None 2023-10-11T15:01:28.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g= 2023-10-11T15:01:28.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g=', 'Content-Length': '0'} 2023-10-11T15:01:28.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bee14b7ef6c4a0b1-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.908 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.908 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.909 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.909 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.909 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.910 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.910 DEBUG:boto:Data: 2023-10-11T15:01:28.911 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.911 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.912 DEBUG:boto:Params: {} 2023-10-11T15:01:28.912 DEBUG:boto:Token: None 2023-10-11T15:01:28.913 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VvcFcVTdhlNa8Ck/X+RUG7DxEWI= 2023-10-11T15:01:28.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VvcFcVTdhlNa8Ck/X+RUG7DxEWI=', 'Content-Length': '0'} 2023-10-11T15:01:28.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018a422751d142fbe-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.916 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.916 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.917 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.917 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.918 DEBUG:boto:Method: HEAD 2023-10-11T15:01:28.918 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.919 DEBUG:boto:Data: 2023-10-11T15:01:28.919 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.920 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.920 DEBUG:boto:Params: {} 2023-10-11T15:01:28.921 DEBUG:boto:Token: None 2023-10-11T15:01:28.921 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VvcFcVTdhlNa8Ck/X+RUG7DxEWI= 2023-10-11T15:01:28.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VvcFcVTdhlNa8Ck/X+RUG7DxEWI=', 'Content-Length': '0'} 2023-10-11T15:01:28.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc9b405c76d3c57b-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:28.925 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.925 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.926 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.926 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.927 DEBUG:boto:Method: GET 2023-10-11T15:01:28.927 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.928 DEBUG:boto:Data: 2023-10-11T15:01:28.928 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.929 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.930 DEBUG:boto:Params: {} 2023-10-11T15:01:28.930 DEBUG:boto:Token: None 2023-10-11T15:01:28.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g= 2023-10-11T15:01:28.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g=', 'Content-Length': '0'} 2023-10-11T15:01:28.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007cb7f2ebae7cc464-006526b8c8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.974 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:28.975 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:28.975 DEBUG:boto:path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.976 DEBUG:boto:auth_path=/dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.976 DEBUG:boto:Method: GET 2023-10-11T15:01:28.976 DEBUG:boto:Path: /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.977 DEBUG:boto:Data: 2023-10-11T15:01:28.977 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.978 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.979 DEBUG:boto:Params: {} 2023-10-11T15:01:28.979 DEBUG:boto:Token: None 2023-10-11T15:01:28.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/obj8?versionId=null 2023-10-11T15:01:28.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g= 2023-10-11T15:01:28.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:alMFXRuWpaIazR+PqDS5mzxnR7g=', 'Content-Length': '0'} 2023-10-11T15:01:28.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021eab432680e5289-006526b8c8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.983 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.984 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.984 DEBUG:boto:path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.985 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.985 DEBUG:boto:Method: GET 2023-10-11T15:01:28.986 DEBUG:boto:Path: /dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.986 DEBUG:boto:Data: 2023-10-11T15:01:28.987 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:28.988 DEBUG:boto:Port: 8001 2023-10-11T15:01:28.988 DEBUG:boto:Params: {} 2023-10-11T15:01:28.989 DEBUG:boto:Token: None 2023-10-11T15:01:28.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:28 GMT /dghqvd-49/ 2023-10-11T15:01:28.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc= 2023-10-11T15:01:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OLv8FXeRbV6CwuIayRtEkScTbfc=', 'Content-Length': '0'} 2023-10-11T15:01:28.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb7fd41e7cb5f9b-006526b8c8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:28.993 DEBUG:boto:b'dghqvd-49obj81000falseobj82023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:28.994 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:28.995 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:28.995 DEBUG:boto:path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.996 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.996 DEBUG:boto:Method: GET 2023-10-11T15:01:28.996 DEBUG:boto:Path: /dghqvd-49/?prefix=obj8 2023-10-11T15:01:28.997 DEBUG:boto:Data: 2023-10-11T15:01:28.997 DEBUG:boto:Headers: {} 2023-10-11T15:01:28.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:28.998 DEBUG:boto:Port: 8000 2023-10-11T15:01:28.999 DEBUG:boto:Params: {} 2023-10-11T15:01:28.999 DEBUG:boto:Token: None 2023-10-11T15:01:29.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/ 2023-10-11T15:01:29.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o= 2023-10-11T15:01:29.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o=', 'Content-Length': '0'} 2023-10-11T15:01:29.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c43aa71ebd1455-006526b8c9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.003 DEBUG:boto:b'dghqvd-49obj81000falseobj82023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:29.004 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:29.005 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:29.005 DEBUG:boto:Method: HEAD 2023-10-11T15:01:29.006 DEBUG:boto:Path: /dghqvd-49/obj8 2023-10-11T15:01:29.006 DEBUG:boto:Data: 2023-10-11T15:01:29.007 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:29.007 DEBUG:boto:Port: 8001 2023-10-11T15:01:29.008 DEBUG:boto:Params: {} 2023-10-11T15:01:29.008 DEBUG:boto:Token: None 2023-10-11T15:01:29.009 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/obj8 2023-10-11T15:01:29.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3WzEHM1GZr5zdbrUipGhzH/UFc= 2023-10-11T15:01:29.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3WzEHM1GZr5zdbrUipGhzH/UFc=', 'Content-Length': '0'} 2023-10-11T15:01:29.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e05ab786462656ff-006526b8c9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.012 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:29.013 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:29.013 DEBUG:boto:Method: HEAD 2023-10-11T15:01:29.014 DEBUG:boto:Path: /dghqvd-49/obj8 2023-10-11T15:01:29.014 DEBUG:boto:Data: 2023-10-11T15:01:29.015 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:29.016 DEBUG:boto:Port: 8000 2023-10-11T15:01:29.016 DEBUG:boto:Params: {} 2023-10-11T15:01:29.017 DEBUG:boto:Token: None 2023-10-11T15:01:29.017 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/obj8 2023-10-11T15:01:29.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3WzEHM1GZr5zdbrUipGhzH/UFc= 2023-10-11T15:01:29.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3WzEHM1GZr5zdbrUipGhzH/UFc=', 'Content-Length': '0'} 2023-10-11T15:01:29.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001924b4cd0d22cdf-006526b8c9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-10-11T15:01:29.021 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:29.021 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:29.022 DEBUG:boto:Method: GET 2023-10-11T15:01:29.023 DEBUG:boto:Path: /dghqvd-49/obj8 2023-10-11T15:01:29.023 DEBUG:boto:Data: 2023-10-11T15:01:29.023 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:29.024 DEBUG:boto:Port: 8001 2023-10-11T15:01:29.025 DEBUG:boto:Params: {} 2023-10-11T15:01:29.025 DEBUG:boto:Token: None 2023-10-11T15:01:29.026 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/obj8 2023-10-11T15:01:29.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x8eWwfRKRXN2UJRrrNRT1QZnMsM= 2023-10-11T15:01:29.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x8eWwfRKRXN2UJRrrNRT1QZnMsM=', 'Content-Length': '0'} 2023-10-11T15:01:29.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020d874f0c2698401-006526b8c9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.029 DEBUG:boto:path=/dghqvd-49/obj8 2023-10-11T15:01:29.030 DEBUG:boto:auth_path=/dghqvd-49/obj8 2023-10-11T15:01:29.030 DEBUG:boto:Method: GET 2023-10-11T15:01:29.030 DEBUG:boto:Path: /dghqvd-49/obj8 2023-10-11T15:01:29.031 DEBUG:boto:Data: 2023-10-11T15:01:29.031 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:29.032 DEBUG:boto:Port: 8000 2023-10-11T15:01:29.032 DEBUG:boto:Params: {} 2023-10-11T15:01:29.033 DEBUG:boto:Token: None 2023-10-11T15:01:29.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/obj8 2023-10-11T15:01:29.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x8eWwfRKRXN2UJRrrNRT1QZnMsM= 2023-10-11T15:01:29.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x8eWwfRKRXN2UJRrrNRT1QZnMsM=', 'Content-Length': '0'} 2023-10-11T15:01:29.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:00:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-49:daa316e9-869c-4074-a756-a8adfb4a3573.5751.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020f18d7121345ac0-006526b8c9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.078 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:29.079 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:29.079 DEBUG:boto:path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.080 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.080 DEBUG:boto:Method: GET 2023-10-11T15:01:29.080 DEBUG:boto:Path: /dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.081 DEBUG:boto:Data: 2023-10-11T15:01:29.081 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:29.082 DEBUG:boto:Port: 8001 2023-10-11T15:01:29.083 DEBUG:boto:Params: {} 2023-10-11T15:01:29.083 DEBUG:boto:Token: None 2023-10-11T15:01:29.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/ 2023-10-11T15:01:29.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o= 2023-10-11T15:01:29.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o=', 'Content-Length': '0'} 2023-10-11T15:01:29.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b71e32fb643242-006526b8c9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.087 DEBUG:boto:b'dghqvd-49obj81000falseobj82023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:29.088 DEBUG:boto:path=/dghqvd-49/ 2023-10-11T15:01:29.089 DEBUG:boto:auth_path=/dghqvd-49/ 2023-10-11T15:01:29.089 DEBUG:boto:path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.089 DEBUG:boto:auth_path=/dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.090 DEBUG:boto:Method: GET 2023-10-11T15:01:29.090 DEBUG:boto:Path: /dghqvd-49/?prefix=obj8 2023-10-11T15:01:29.091 DEBUG:boto:Data: 2023-10-11T15:01:29.091 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:29.092 DEBUG:boto:Port: 8000 2023-10-11T15:01:29.092 DEBUG:boto:Params: {} 2023-10-11T15:01:29.093 DEBUG:boto:Token: None 2023-10-11T15:01:29.093 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-49/ 2023-10-11T15:01:29.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o= 2023-10-11T15:01:29.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:shIqypyeBAnqITDxCi/dfGeoL/o=', 'Content-Length': '0'} 2023-10-11T15:01:29.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000516e4e697c04011e-006526b8c9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.097 DEBUG:boto:b'dghqvd-49obj81000falseobj82023-10-11T15:00:40.020Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:01:29.098 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-49 zones={a2, a1} 2023-10-11T15:01:29.098 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T15:01:29.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.100 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:29.152 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:29.198 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:01:29.198 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.199 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-11T15:01:29.199 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:07.651062Z" 2023-10-11T15:01:29.199 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.199 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.200 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-10-11T15:01:29.200 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:07.650828Z" 2023-10-11T15:01:29.200 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.200 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.200 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-11T15:01:29.201 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:07.652764Z" 2023-10-11T15:01:29.201 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.201 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.201 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000007129", 2023-10-11T15:01:29.202 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:00:40.240330Z" 2023-10-11T15:01:29.202 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.202 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:01:29.203 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000006576', 1: '00000000000000000000:00000000000000004317', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007129'} 2023-10-11T15:01:29.204 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:01:29.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:29.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:29.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:29.299 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:01:29.299 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:01:29.299 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:01:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:01:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:01:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 3631162067912812667 2023-10-11T15:01:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:01:29.301 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.301 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:01:29.301 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.301 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.302 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-10-11T15:01:29.302 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.302 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.302 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:07.651062Z" 2023-10-11T15:01:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:01:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-10-11T15:01:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:07.650828Z" 2023-10-11T15:01:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:01:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000009807", 2023-10-11T15:01:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:07.652764Z" 2023-10-11T15:01:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:01:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000007129", 2023-10-11T15:01:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:00:40.240330Z" 2023-10-11T15:01:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:01:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:01:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T15:01:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T15:01:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:01:29Z" 2023-10-11T15:01:29.312 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:01:29.313 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3631162067912812667 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:07.651062Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004317", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:07.650828Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009807", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:07.652764Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007129", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:00:40.240330Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T15:01:29Z" } 2023-10-11T15:01:29.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:07.651062Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004317', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:07.650828Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009807', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:07.652764Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:00:40.240330Z'}}] 2023-10-11T15:01:29.314 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006576', 1: '00000000000000000000:00000000000000004317', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007129'} 2023-10-11T15:01:29.314 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006576', 1: '00000000000000000000:00000000000000004317', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007129'} 2023-10-11T15:01:29.315 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:01:29.317 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T15:01:29.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.319 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:29.367 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:29.368 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:29.413 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:01:29.413 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.414 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004738", 2023-10-11T15:01:29.414 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:09.201134Z" 2023-10-11T15:01:29.414 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.414 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.414 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003048", 2023-10-11T15:01:29.415 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:09.203956Z" 2023-10-11T15:01:29.415 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.415 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.415 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-10-11T15:01:29.416 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:09.205513Z" 2023-10-11T15:01:29.416 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.416 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.416 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T15:01:29.416 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:09.207010Z" 2023-10-11T15:01:29.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.417 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:01:29.418 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004738', 1: '00000000000000000000:00000000000000003048', 2: '00000000000000000000:00000000000000008961', 3: '00000000000000000000:00000000000000004452'} 2023-10-11T15:01:29.418 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:01:29.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:29.420 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:29.471 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:29.471 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:29.513 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:01:29.513 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:01:29.513 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:01:29.514 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:01:29.514 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:01:29.514 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T15:01:29.514 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.514 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:01:29.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.515 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:01:29.515 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.515 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004738", 2023-10-11T15:01:29.516 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.516 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.516 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.516 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:09.201134Z" 2023-10-11T15:01:29.517 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.517 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.517 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.517 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:01:29.517 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.518 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.518 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000003048", 2023-10-11T15:01:29.518 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.518 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.518 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.519 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:09.203956Z" 2023-10-11T15:01:29.519 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.520 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:01:29.520 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.520 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-10-11T15:01:29.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.521 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.521 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:09.205513Z" 2023-10-11T15:01:29.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:29.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:01:29.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:29.522 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:01:29.522 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000004452", 2023-10-11T15:01:29.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:29.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:29.523 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:29.523 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:09.207010Z" 2023-10-11T15:01:29.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:29.524 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:01:29.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.524 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:01:29.524 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T15:01:29.525 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T15:01:29.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:29.525 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:01:29Z" 2023-10-11T15:01:29.525 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:01:29.527 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004738", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:09.201134Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003048", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:09.203956Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008961", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:09.205513Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004452", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:09.207010Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T15:01:29Z" } 2023-10-11T15:01:29.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004738', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:09.201134Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003048', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:09.203956Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008961', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:09.205513Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004452', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:09.207010Z'}}] 2023-10-11T15:01:29.528 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004738', 1: '00000000000000000000:00000000000000003048', 2: '00000000000000000000:00000000000000008961', 3: '00000000000000000000:00000000000000004452'} 2023-10-11T15:01:29.528 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004738', 1: '00000000000000000000:00000000000000003048', 2: '00000000000000000000:00000000000000008961', 3: '00000000000000000000:00000000000000004452'} 2023-10-11T15:01:29.528 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:01:29.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-10-11T15:01:29.530 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-50 2023-10-11T15:01:29.531 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:01:29.531 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:01:29.531 DEBUG:boto:Method: PUT 2023-10-11T15:01:29.532 DEBUG:boto:Path: /dghqvd-50/ 2023-10-11T15:01:29.532 DEBUG:boto:Data: 2023-10-11T15:01:29.533 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:29.533 DEBUG:boto:Port: 8000 2023-10-11T15:01:29.534 DEBUG:boto:Params: {} 2023-10-11T15:01:29.534 DEBUG:boto:Token: None 2023-10-11T15:01:29.535 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-50/ 2023-10-11T15:01:29.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l0xwhg77+6G7kXVV7GSXIV5ClAE= 2023-10-11T15:01:29.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l0xwhg77+6G7kXVV7GSXIV5ClAE=', 'Content-Length': '0'} 2023-10-11T15:01:29.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d3ea29e01544881-006526b8c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.549 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-51 2023-10-11T15:01:29.550 DEBUG:boto:path=/dghqvd-51/ 2023-10-11T15:01:29.550 DEBUG:boto:auth_path=/dghqvd-51/ 2023-10-11T15:01:29.551 DEBUG:boto:Method: PUT 2023-10-11T15:01:29.551 DEBUG:boto:Path: /dghqvd-51/ 2023-10-11T15:01:29.551 DEBUG:boto:Data: 2023-10-11T15:01:29.552 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:01:29.552 DEBUG:boto:Port: 8001 2023-10-11T15:01:29.553 DEBUG:boto:Params: {} 2023-10-11T15:01:29.553 DEBUG:boto:Token: None 2023-10-11T15:01:29.554 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-51/ 2023-10-11T15:01:29.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LfZHj/xEhMISfEE0ti8GE4yM3jo= 2023-10-11T15:01:29.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LfZHj/xEhMISfEE0ti8GE4yM3jo=', 'Content-Length': '0'} 2023-10-11T15:01:29.577 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003d9f01da2dec6a0-006526b8c9-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:29.578 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:29.578 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:29.578 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploads 2023-10-11T15:01:29.579 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploads 2023-10-11T15:01:29.579 DEBUG:boto:Method: POST 2023-10-11T15:01:29.579 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploads 2023-10-11T15:01:29.580 DEBUG:boto:Data: 2023-10-11T15:01:29.580 DEBUG:boto:Headers: {} 2023-10-11T15:01:29.580 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:29.581 DEBUG:boto:Port: 8000 2023-10-11T15:01:29.581 DEBUG:boto:Params: {} 2023-10-11T15:01:29.582 DEBUG:boto:Token: None 2023-10-11T15:01:29.582 DEBUG:boto:StringToSign: POST Wed, 11 Oct 2023 15:01:29 GMT /dghqvd-50/MULTIPART?uploads 2023-10-11T15:01:29.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ovIupdcPn5vWZGdi3BLmJqg0/aE= 2023-10-11T15:01:29.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ovIupdcPn5vWZGdi3BLmJqg0/aE=', 'Content-Length': '0'} 2023-10-11T15:01:31.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007505edc95c669906-006526b8c9-1675-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.527 DEBUG:boto:b'dghqvd-50MULTIPART2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34' 2023-10-11T15:01:31.559 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.560 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.560 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=1 2023-10-11T15:01:31.561 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=1 2023-10-11T15:01:31.561 DEBUG:boto:Method: PUT 2023-10-11T15:01:31.562 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=1 2023-10-11T15:01:31.562 DEBUG:boto:Data: 2023-10-11T15:01:31.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-11T15:01:31.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.563 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.564 DEBUG:boto:Params: {} 2023-10-11T15:01:31.564 DEBUG:boto:Token: None 2023-10-11T15:01:31.566 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?partNumber=1&uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bKJsM4saf6C98Yn9GYe33sMIQbo= 2023-10-11T15:01:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bKJsM4saf6C98Yn9GYe33sMIQbo='} 2023-10-11T15:01:31.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca8d1997c0e54c8b-006526b8cb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.655 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.656 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.656 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=2 2023-10-11T15:01:31.656 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=2 2023-10-11T15:01:31.657 DEBUG:boto:Method: PUT 2023-10-11T15:01:31.657 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=2 2023-10-11T15:01:31.658 DEBUG:boto:Data: 2023-10-11T15:01:31.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-11T15:01:31.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.659 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.659 DEBUG:boto:Params: {} 2023-10-11T15:01:31.660 DEBUG:boto:Token: None 2023-10-11T15:01:31.660 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?partNumber=2&uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DuJdXYQpBfslzxfeXBKK31oz93k= 2023-10-11T15:01:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DuJdXYQpBfslzxfeXBKK31oz93k='} 2023-10-11T15:01:31.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a7eae735be4dae5-006526b8cb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.730 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.731 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.732 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=3 2023-10-11T15:01:31.732 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=3 2023-10-11T15:01:31.732 DEBUG:boto:Method: PUT 2023-10-11T15:01:31.733 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=3 2023-10-11T15:01:31.733 DEBUG:boto:Data: 2023-10-11T15:01:31.734 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-11T15:01:31.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.734 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.735 DEBUG:boto:Params: {} 2023-10-11T15:01:31.735 DEBUG:boto:Token: None 2023-10-11T15:01:31.736 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?partNumber=3&uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a5iT7QA6jF27JBuUBFnMTF7vNn0= 2023-10-11T15:01:31.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a5iT7QA6jF27JBuUBFnMTF7vNn0='} 2023-10-11T15:01:31.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a6340d94ae11c9f-006526b8cb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.803 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.804 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.805 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=4 2023-10-11T15:01:31.805 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=4 2023-10-11T15:01:31.805 DEBUG:boto:Method: PUT 2023-10-11T15:01:31.806 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34&partNumber=4 2023-10-11T15:01:31.806 DEBUG:boto:Data: 2023-10-11T15:01:31.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-10-11T15:01:31.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.807 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.808 DEBUG:boto:Params: {} 2023-10-11T15:01:31.808 DEBUG:boto:Token: None 2023-10-11T15:01:31.809 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?partNumber=4&uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5e4hnnHVARZ0YepQd+hOi1hRsXo= 2023-10-11T15:01:31.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5e4hnnHVARZ0YepQd+hOi1hRsXo='} 2023-10-11T15:01:31.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063ee0ff934feb7b8-006526b8cb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.864 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.865 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.865 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.866 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.866 DEBUG:boto:Method: GET 2023-10-11T15:01:31.867 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.867 DEBUG:boto:Data: 2023-10-11T15:01:31.867 DEBUG:boto:Headers: {} 2023-10-11T15:01:31.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.868 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.869 DEBUG:boto:Params: {} 2023-10-11T15:01:31.869 DEBUG:boto:Token: None 2023-10-11T15:01:31.870 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gWhm2sqOi5wbePc+IJtBJSfLkl0= 2023-10-11T15:01:31.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gWhm2sqOi5wbePc+IJtBJSfLkl0=', 'Content-Length': '0'} 2023-10-11T15:01:31.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003524b014a2b61d3b-006526b8cb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.914 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.915 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:01:31.915 DEBUG:boto:path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.916 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.916 DEBUG:boto:Method: POST 2023-10-11T15:01:31.917 DEBUG:boto:Path: /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.917 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-10-11T15:01:31.917 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-10-11T15:01:31.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:01:31.918 DEBUG:boto:Port: 8000 2023-10-11T15:01:31.919 DEBUG:boto:Params: {} 2023-10-11T15:01:31.919 DEBUG:boto:Token: None 2023-10-11T15:01:31.920 DEBUG:boto:StringToSign: POST text/xml Wed, 11 Oct 2023 15:01:31 GMT /dghqvd-50/MULTIPART?uploadId=2~nbsVnXgmthbjeUSIL6DbqimM2X6jJ34 2023-10-11T15:01:31.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mtQzyX4eD1P07/MtKV3t5Dx+bcQ= 2023-10-11T15:01:31.921 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:01:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mtQzyX4eD1P07/MtKV3t5Dx+bcQ=', 'Content-Length': '460'} 2023-10-11T15:01:31.929 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff614da61ce07db8-006526b8cb-1675-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:01:31.929 DEBUG:boto:http://smithi158.front.sepia.ceph.com:8000/dghqvd-50/MULTIPARTdghqvd-50MULTIPART 2023-10-11T15:01:31.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:31.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:31.931 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:31.979 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:31.980 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:32.008 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:01:32.020 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:01:32.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.021 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:01:32.021 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:01:32.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.021 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:01:32.022 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:01:32.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.022 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.022 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:01:32.022 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:01:32.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.023 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036489.567363_367.1", 2023-10-11T15:01:32.023 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:01:29.567363Z" 2023-10-11T15:01:32.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.024 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:01:32.024 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036489.567363_367.1'} 2023-10-11T15:01:32.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:01:32.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:32.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:01:32.026 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:01:32.075 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:32.076 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:32.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:32.100+0000 7f39c971e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:01:32.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:32.101+0000 7f39c971e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:01:32.116 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:01:32.116 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:01:32.117 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:01:32.117 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:01:32.117 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:01:32.117 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:01:32.117 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:01:32.118 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.118 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:01:32.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.118 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:01:32.119 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:32.119 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:01:32.119 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:01:32.119 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:32.119 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:01:32.120 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:32.120 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:01:32.120 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:01:32.120 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.121 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:01:32.121 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:32.121 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:01:32.122 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:01:32.122 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:32.122 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:32.122 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:32.122 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:01:32.123 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:01:32.123 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.123 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.123 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:01:32.124 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:32.125 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:01:32.125 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:01:32.125 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:32.126 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:32.126 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:32.126 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:01:32.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:01:32.127 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.127 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.127 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:01:32.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:01:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:01:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:01:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036489.567363_367.1", 2023-10-11T15:01:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:01:32.128 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:01:32.129 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:01:32.129 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:01:29.567363Z", 2023-10-11T15:01:32.129 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:01:32.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.129 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:01:32.130 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:01:32.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.130 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:01:32.130 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:01:32.130 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:01:32.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:01:32.131 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:01:32Z" 2023-10-11T15:01:32.131 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:01:32.132 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036489.567363_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:01:29.567363Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:01:32Z" } 2023-10-11T15:01:32.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036489.567363_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:01:29.567363Z', 'realm_epoch': 2}}] 2023-10-11T15:01:32.133 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036489.567363_367.1'} 2023-10-11T15:01:32.133 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036489.567363_367.1'} 2023-10-11T15:01:32.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:01:32.134 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-50 2023-10-11T15:01:32.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:32.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:32.135 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:01:32.191 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:32.191 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:32.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:32.223+0000 7f2e07496780 1 waiting to reach incremental sync.. 2023-10-11T15:01:37.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:37.224+0000 7f2e07496780 1 waiting to reach incremental sync.. 2023-10-11T15:01:42.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:42.224+0000 7f2e07496780 1 waiting to reach incremental sync.. 2023-10-11T15:01:47.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:47.225+0000 7f2e07496780 1 waiting to reach incremental sync.. 2023-10-11T15:01:52.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:52.227+0000 7f2e07496780 1 bucket sync caught up with source: 2023-10-11T15:01:52.230 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000011.37.6, , , , , ] 2023-10-11T15:01:52.230 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000011.37.6, , , , , ] 2023-10-11T15:01:52.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:52.227+0000 7f2e07496780 0 bucket checkpoint complete 2023-10-11T15:01:52.246 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-50 2023-10-11T15:01:52.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:52.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:01:52.248 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:01:52.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:01:52.311 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:01:52.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:52.342+0000 7f951cf17780 1 waiting to reach incremental sync.. 2023-10-11T15:01:57.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:01:57.343+0000 7f951cf17780 1 waiting to reach incremental sync.. 2023-10-11T15:02:02.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:02.344+0000 7f951cf17780 1 waiting to reach incremental sync.. 2023-10-11T15:02:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:07.344+0000 7f951cf17780 1 waiting to reach incremental sync.. 2023-10-11T15:02:12.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:12.345+0000 7f951cf17780 1 bucket sync caught up with source: 2023-10-11T15:02:12.349 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.176.6, , , , , ] 2023-10-11T15:02:12.349 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.176.6, , , , , ] 2023-10-11T15:02:12.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:12.345+0000 7f951cf17780 0 bucket checkpoint complete 2023-10-11T15:02:12.364 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-50 zones={a2, a1} 2023-10-11T15:02:12.365 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.365 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.366 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.367 DEBUG:boto:Path: /dghqvd-50/ 2023-10-11T15:02:12.367 DEBUG:boto:Data: 2023-10-11T15:02:12.367 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.368 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.369 DEBUG:boto:Params: {} 2023-10-11T15:02:12.369 DEBUG:boto:Token: None 2023-10-11T15:02:12.370 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/ 2023-10-11T15:02:12.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+GfvX9WsmzJXJ6CoVGLL9aC7crg= 2023-10-11T15:02:12.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+GfvX9WsmzJXJ6CoVGLL9aC7crg=', 'Content-Length': '0'} 2023-10-11T15:02:12.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3d63a041b9f96cf-006526b8f4-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.374 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.375 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.375 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.375 DEBUG:boto:Path: /dghqvd-50/ 2023-10-11T15:02:12.376 DEBUG:boto:Data: 2023-10-11T15:02:12.376 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.377 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.377 DEBUG:boto:Params: {} 2023-10-11T15:02:12.378 DEBUG:boto:Token: None 2023-10-11T15:02:12.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/ 2023-10-11T15:02:12.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+GfvX9WsmzJXJ6CoVGLL9aC7crg= 2023-10-11T15:02:12.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+GfvX9WsmzJXJ6CoVGLL9aC7crg=', 'Content-Length': '0'} 2023-10-11T15:02:12.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023b1af0bbdf023bb-006526b8f4-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:02:12.383 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.383 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.384 DEBUG:boto:path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.384 DEBUG:boto:auth_path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.384 DEBUG:boto:Method: GET 2023-10-11T15:02:12.385 DEBUG:boto:Path: /dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.385 DEBUG:boto:Data: 2023-10-11T15:02:12.386 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.386 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.387 DEBUG:boto:Params: {} 2023-10-11T15:02:12.387 DEBUG:boto:Token: None 2023-10-11T15:02:12.388 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/?versions 2023-10-11T15:02:12.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ= 2023-10-11T15:02:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e043311cb5917fe8-006526b8f4-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.392 DEBUG:boto:b'dghqvd-50999falseMULTIPARTnulltrue2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.393 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-11T15:02:12.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:02:12.394 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.394 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.394 DEBUG:boto:path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.395 DEBUG:boto:auth_path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.395 DEBUG:boto:Method: GET 2023-10-11T15:02:12.396 DEBUG:boto:Path: /dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.396 DEBUG:boto:Data: 2023-10-11T15:02:12.396 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.397 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.398 DEBUG:boto:Params: {} 2023-10-11T15:02:12.398 DEBUG:boto:Token: None 2023-10-11T15:02:12.398 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/?versions 2023-10-11T15:02:12.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ= 2023-10-11T15:02:12.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c9ce926f0abaf74-006526b8f4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.403 DEBUG:boto:b'dghqvd-50999falseMULTIPARTnulltrue2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.403 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-10-11T15:02:12.404 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.404 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.405 DEBUG:boto:path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.405 DEBUG:boto:auth_path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.405 DEBUG:boto:Method: GET 2023-10-11T15:02:12.406 DEBUG:boto:Path: /dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.406 DEBUG:boto:Data: 2023-10-11T15:02:12.407 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.407 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.408 DEBUG:boto:Params: {} 2023-10-11T15:02:12.408 DEBUG:boto:Token: None 2023-10-11T15:02:12.409 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/?versions 2023-10-11T15:02:12.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ= 2023-10-11T15:02:12.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c756d54b0e4265a3-006526b8f4-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.413 DEBUG:boto:b'dghqvd-50999falseMULTIPARTnulltrue2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.414 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.414 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.415 DEBUG:boto:path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.415 DEBUG:boto:auth_path=/dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.415 DEBUG:boto:Method: GET 2023-10-11T15:02:12.416 DEBUG:boto:Path: /dghqvd-50/?versions&max-keys=999 2023-10-11T15:02:12.416 DEBUG:boto:Data: 2023-10-11T15:02:12.417 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.417 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.418 DEBUG:boto:Params: {} 2023-10-11T15:02:12.418 DEBUG:boto:Token: None 2023-10-11T15:02:12.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/?versions 2023-10-11T15:02:12.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ= 2023-10-11T15:02:12.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jw9H7/Xl1VcoWIPsbnnitWMrPrQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec739b0f81cf371-006526b8f4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.423 DEBUG:boto:b'dghqvd-50999falseMULTIPARTnulltrue2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-11T15:02:12.424 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.424 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.425 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.425 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.425 DEBUG:boto:Method: GET 2023-10-11T15:02:12.426 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.426 DEBUG:boto:Data: 2023-10-11T15:02:12.427 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.427 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.428 DEBUG:boto:Params: {} 2023-10-11T15:02:12.428 DEBUG:boto:Token: None 2023-10-11T15:02:12.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk= 2023-10-11T15:02:12.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk=', 'Content-Length': '0'} 2023-10-11T15:02:12.438 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-50:daa316e9-869c-4074-a756-a8adfb4a3573.5751.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008fe68791900503b4-006526b8f4-1679-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.512 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.513 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.513 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.513 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.514 DEBUG:boto:Method: GET 2023-10-11T15:02:12.514 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.515 DEBUG:boto:Data: 2023-10-11T15:02:12.515 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.516 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.517 DEBUG:boto:Params: {} 2023-10-11T15:02:12.517 DEBUG:boto:Token: None 2023-10-11T15:02:12.518 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk= 2023-10-11T15:02:12.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk=', 'Content-Length': '0'} 2023-10-11T15:02:12.528 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000085880a41b694cff1-006526b8f4-1675-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.618 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.619 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.619 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.620 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.620 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.620 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.621 DEBUG:boto:Data: 2023-10-11T15:02:12.621 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.622 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.623 DEBUG:boto:Params: {} 2023-10-11T15:02:12.623 DEBUG:boto:Token: None 2023-10-11T15:02:12.624 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Na+a87ae/NyzSV8Cz9KL0FuWsuQ= 2023-10-11T15:02:12.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Na+a87ae/NyzSV8Cz9KL0FuWsuQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.627 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-50:daa316e9-869c-4074-a756-a8adfb4a3573.5751.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000fbd6749709aacd1a-006526b8f4-1679-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.627 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.628 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.628 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.629 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.629 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.629 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.630 DEBUG:boto:Data: 2023-10-11T15:02:12.630 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.631 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.631 DEBUG:boto:Params: {} 2023-10-11T15:02:12.632 DEBUG:boto:Token: None 2023-10-11T15:02:12.632 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.633 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Na+a87ae/NyzSV8Cz9KL0FuWsuQ= 2023-10-11T15:02:12.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Na+a87ae/NyzSV8Cz9KL0FuWsuQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.635 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000709448e1ec2a730c-006526b8f4-1675-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-11T15:02:12.636 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.636 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.637 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.637 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.638 DEBUG:boto:Method: GET 2023-10-11T15:02:12.638 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.638 DEBUG:boto:Data: 2023-10-11T15:02:12.639 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.639 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.640 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.640 DEBUG:boto:Params: {} 2023-10-11T15:02:12.641 DEBUG:boto:Token: None 2023-10-11T15:02:12.641 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.642 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk= 2023-10-11T15:02:12.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk=', 'Content-Length': '0'} 2023-10-11T15:02:12.650 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-50:daa316e9-869c-4074-a756-a8adfb4a3573.5751.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004cfdc6239dc0c654-006526b8f4-1679-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.736 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.736 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.737 DEBUG:boto:path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.737 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.737 DEBUG:boto:Method: GET 2023-10-11T15:02:12.738 DEBUG:boto:Path: /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.738 DEBUG:boto:Data: 2023-10-11T15:02:12.739 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.740 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.740 DEBUG:boto:Params: {} 2023-10-11T15:02:12.741 DEBUG:boto:Token: None 2023-10-11T15:02:12.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART?versionId=null 2023-10-11T15:02:12.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk= 2023-10-11T15:02:12.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a5UI0CeatbLlIiV2jgO1tO7eAbk=', 'Content-Length': '0'} 2023-10-11T15:02:12.751 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000141e5467d5f5f459-006526b8f4-1675-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.844 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.845 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.846 DEBUG:boto:path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.846 DEBUG:boto:auth_path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.846 DEBUG:boto:Method: GET 2023-10-11T15:02:12.847 DEBUG:boto:Path: /dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.847 DEBUG:boto:Data: 2023-10-11T15:02:12.848 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.849 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.849 DEBUG:boto:Params: {} 2023-10-11T15:02:12.850 DEBUG:boto:Token: None 2023-10-11T15:02:12.850 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/ 2023-10-11T15:02:12.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KU0HS3bZUUSapvczXbvGEFM30Yg= 2023-10-11T15:02:12.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KU0HS3bZUUSapvczXbvGEFM30Yg=', 'Content-Length': '0'} 2023-10-11T15:02:12.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1db920245323fd9-006526b8f4-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.854 DEBUG:boto:b'dghqvd-50MULTIPART1000falseMULTIPART2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.855 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:12.855 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:12.856 DEBUG:boto:path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.856 DEBUG:boto:auth_path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.857 DEBUG:boto:Method: GET 2023-10-11T15:02:12.857 DEBUG:boto:Path: /dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:12.857 DEBUG:boto:Data: 2023-10-11T15:02:12.858 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.859 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.859 DEBUG:boto:Params: {} 2023-10-11T15:02:12.860 DEBUG:boto:Token: None 2023-10-11T15:02:12.860 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/ 2023-10-11T15:02:12.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KU0HS3bZUUSapvczXbvGEFM30Yg= 2023-10-11T15:02:12.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KU0HS3bZUUSapvczXbvGEFM30Yg=', 'Content-Length': '0'} 2023-10-11T15:02:12.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979229c61ba2cc3f-006526b8f4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.864 DEBUG:boto:b'dghqvd-50MULTIPART1000falseMULTIPART2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:12.864 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.865 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.865 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.866 DEBUG:boto:Path: /dghqvd-50/MULTIPART 2023-10-11T15:02:12.866 DEBUG:boto:Data: 2023-10-11T15:02:12.867 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.867 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.867 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.868 DEBUG:boto:Params: {} 2023-10-11T15:02:12.868 DEBUG:boto:Token: None 2023-10-11T15:02:12.869 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART 2023-10-11T15:02:12.869 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lc2PMmpGS13pZX45XreAymjNFOQ= 2023-10-11T15:02:12.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lc2PMmpGS13pZX45XreAymjNFOQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.871 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-50:daa316e9-869c-4074-a756-a8adfb4a3573.5751.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000021e328498947e7cd-006526b8f4-1679-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.872 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.872 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.873 DEBUG:boto:Method: HEAD 2023-10-11T15:02:12.873 DEBUG:boto:Path: /dghqvd-50/MULTIPART 2023-10-11T15:02:12.874 DEBUG:boto:Data: 2023-10-11T15:02:12.874 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.875 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.875 DEBUG:boto:Params: {} 2023-10-11T15:02:12.876 DEBUG:boto:Token: None 2023-10-11T15:02:12.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART 2023-10-11T15:02:12.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lc2PMmpGS13pZX45XreAymjNFOQ= 2023-10-11T15:02:12.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lc2PMmpGS13pZX45XreAymjNFOQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.879 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001ebc35e231fadc26-006526b8f4-1675-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-10-11T15:02:12.880 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.880 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.880 DEBUG:boto:Method: GET 2023-10-11T15:02:12.881 DEBUG:boto:Path: /dghqvd-50/MULTIPART 2023-10-11T15:02:12.881 DEBUG:boto:Data: 2023-10-11T15:02:12.882 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:12.883 DEBUG:boto:Port: 8001 2023-10-11T15:02:12.883 DEBUG:boto:Params: {} 2023-10-11T15:02:12.884 DEBUG:boto:Token: None 2023-10-11T15:02:12.884 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART 2023-10-11T15:02:12.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p+azB90YFU5CQ9naN1NRP0qUMXQ= 2023-10-11T15:02:12.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p+azB90YFU5CQ9naN1NRP0qUMXQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.893 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-50:daa316e9-869c-4074-a756-a8adfb4a3573.5751.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000558d70799b720df8-006526b8f4-1679-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:12.980 DEBUG:boto:path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.980 DEBUG:boto:auth_path=/dghqvd-50/MULTIPART 2023-10-11T15:02:12.981 DEBUG:boto:Method: GET 2023-10-11T15:02:12.981 DEBUG:boto:Path: /dghqvd-50/MULTIPART 2023-10-11T15:02:12.982 DEBUG:boto:Data: 2023-10-11T15:02:12.982 DEBUG:boto:Headers: {} 2023-10-11T15:02:12.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:12.983 DEBUG:boto:Port: 8000 2023-10-11T15:02:12.983 DEBUG:boto:Params: {} 2023-10-11T15:02:12.984 DEBUG:boto:Token: None 2023-10-11T15:02:12.984 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:12 GMT /dghqvd-50/MULTIPART 2023-10-11T15:02:12.985 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p+azB90YFU5CQ9naN1NRP0qUMXQ= 2023-10-11T15:02:12.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p+azB90YFU5CQ9naN1NRP0qUMXQ=', 'Content-Length': '0'} 2023-10-11T15:02:12.995 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:01:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004c6f9d8db5e44866-006526b8f4-1675-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.087 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:13.087 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:13.088 DEBUG:boto:path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.088 DEBUG:boto:auth_path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.088 DEBUG:boto:Method: GET 2023-10-11T15:02:13.089 DEBUG:boto:Path: /dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.089 DEBUG:boto:Data: 2023-10-11T15:02:13.090 DEBUG:boto:Headers: {} 2023-10-11T15:02:13.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:13.091 DEBUG:boto:Port: 8001 2023-10-11T15:02:13.091 DEBUG:boto:Params: {} 2023-10-11T15:02:13.092 DEBUG:boto:Token: None 2023-10-11T15:02:13.092 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:13 GMT /dghqvd-50/ 2023-10-11T15:02:13.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HGUXklXq5B5OKInPV7smGdOor8w= 2023-10-11T15:02:13.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HGUXklXq5B5OKInPV7smGdOor8w=', 'Content-Length': '0'} 2023-10-11T15:02:13.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091068e3846ae6622-006526b8f5-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.096 DEBUG:boto:b'dghqvd-50MULTIPART1000falseMULTIPART2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:13.097 DEBUG:boto:path=/dghqvd-50/ 2023-10-11T15:02:13.097 DEBUG:boto:auth_path=/dghqvd-50/ 2023-10-11T15:02:13.097 DEBUG:boto:path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.098 DEBUG:boto:auth_path=/dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.098 DEBUG:boto:Method: GET 2023-10-11T15:02:13.099 DEBUG:boto:Path: /dghqvd-50/?prefix=MULTIPART 2023-10-11T15:02:13.099 DEBUG:boto:Data: 2023-10-11T15:02:13.100 DEBUG:boto:Headers: {} 2023-10-11T15:02:13.100 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:13.100 DEBUG:boto:Port: 8000 2023-10-11T15:02:13.101 DEBUG:boto:Params: {} 2023-10-11T15:02:13.101 DEBUG:boto:Token: None 2023-10-11T15:02:13.102 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:13 GMT /dghqvd-50/ 2023-10-11T15:02:13.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HGUXklXq5B5OKInPV7smGdOor8w= 2023-10-11T15:02:13.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HGUXklXq5B5OKInPV7smGdOor8w=', 'Content-Length': '0'} 2023-10-11T15:02:13.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d63d0c6b2b1a553-006526b8f5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.150 DEBUG:boto:b'dghqvd-50MULTIPART1000falseMULTIPART2023-10-11T15:01:31.923Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:02:13.151 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-50 zones={a2, a1} 2023-10-11T15:02:13.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-10-11T15:02:13.153 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-52 2023-10-11T15:02:13.153 DEBUG:boto:path=/dghqvd-52/ 2023-10-11T15:02:13.154 DEBUG:boto:auth_path=/dghqvd-52/ 2023-10-11T15:02:13.154 DEBUG:boto:Method: PUT 2023-10-11T15:02:13.154 DEBUG:boto:Path: /dghqvd-52/ 2023-10-11T15:02:13.155 DEBUG:boto:Data: 2023-10-11T15:02:13.155 DEBUG:boto:Headers: {} 2023-10-11T15:02:13.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:13.156 DEBUG:boto:Port: 8000 2023-10-11T15:02:13.156 DEBUG:boto:Params: {} 2023-10-11T15:02:13.157 DEBUG:boto:Token: None 2023-10-11T15:02:13.157 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:02:13 GMT /dghqvd-52/ 2023-10-11T15:02:13.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WowbGUhTgv938umvOWtd1epg6GE= 2023-10-11T15:02:13.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WowbGUhTgv938umvOWtd1epg6GE=', 'Content-Length': '0'} 2023-10-11T15:02:13.172 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000389ce9c3a8fb9c06-006526b8f5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.172 DEBUG:boto:path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:13.172 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:13.173 DEBUG:boto:Method: PUT 2023-10-11T15:02:13.173 DEBUG:boto:Path: /dghqvd-52/testobj-sse-c 2023-10-11T15:02:13.174 DEBUG:boto:Data: 2023-10-11T15:02:13.174 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T15:02:13.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:13.175 DEBUG:boto:Port: 8000 2023-10-11T15:02:13.175 DEBUG:boto:Params: {} 2023-10-11T15:02:13.176 DEBUG:boto:Token: None 2023-10-11T15:02:13.176 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 11 Oct 2023 15:02:13 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dghqvd-52/testobj-sse-c 2023-10-11T15:02:13.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yaoDkKIv5bhJ8bzGI/fvfGF2Eos= 2023-10-11T15:02:13.177 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yaoDkKIv5bhJ8bzGI/fvfGF2Eos='} 2023-10-11T15:02:13.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000c574dcfd60e0b328-006526b8f5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.184 DEBUG:boto:path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:13.184 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:13.185 DEBUG:boto:Method: PUT 2023-10-11T15:02:13.185 DEBUG:boto:Path: /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:13.186 DEBUG:boto:Data: 2023-10-11T15:02:13.186 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-10-11T15:02:13.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:13.187 DEBUG:boto:Port: 8000 2023-10-11T15:02:13.187 DEBUG:boto:Params: {} 2023-10-11T15:02:13.188 DEBUG:boto:Token: None 2023-10-11T15:02:13.188 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 11 Oct 2023 15:02:13 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:13.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dSXpJIH2IUj0lbqKKrjP56nf4cw= 2023-10-11T15:02:13.189 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dSXpJIH2IUj0lbqKKrjP56nf4cw='} 2023-10-11T15:02:13.231 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZGdocXZkLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000002aedb2d98b3baeb3-006526b8f5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:13.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:13.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:13.232 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:02:13.283 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:13.283 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:13.311 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:02:13.323 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:02:13.324 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.324 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:02:13.324 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:02:13.325 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.325 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.325 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:02:13.325 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:02:13.326 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.326 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.326 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:02:13.326 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:02:13.327 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.327 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.327 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036533.169197_371.1", 2023-10-11T15:02:13.327 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:02:13.169197Z" 2023-10-11T15:02:13.328 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.328 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:02:13.329 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036533.169197_371.1'} 2023-10-11T15:02:13.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:02:13.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:13.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:13.331 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:02:13.380 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:13.381 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:13.408 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:13.406+0000 7f428313f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:02:13.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:13.407+0000 7f428313f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:02:13.421 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:02:13.421 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:02:13.421 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:02:13.422 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:02:13.422 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:02:13.422 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:02:13.422 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:13.423 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.423 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:02:13.423 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.423 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:02:13.423 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:13.424 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:13.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:02:13.424 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:13.424 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:02:13.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:13.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:02:13.425 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:13.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.426 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:02:13.426 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:13.426 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:13.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:02:13.427 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:13.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:13.427 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:13.427 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:02:13.428 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:13.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.428 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.428 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:02:13.429 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:13.429 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:13.429 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:02:13.429 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:13.429 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:13.430 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:13.430 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:02:13.430 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:13.430 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.431 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:13.431 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:02:13.431 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:13.431 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:13.432 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036533.169197_371.1", 2023-10-11T15:02:13.432 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:13.432 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:13.432 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:13.432 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:13.169197Z", 2023-10-11T15:02:13.433 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:13.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:13.433 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:02:13.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.434 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:02:13.434 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:02:13.434 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:02:13.434 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:13.435 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:02:13Z" 2023-10-11T15:02:13.435 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:02:13.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036533.169197_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:02:13.169197Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:02:13Z" } 2023-10-11T15:02:13.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036533.169197_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:02:13.169197Z', 'realm_epoch': 2}}] 2023-10-11T15:02:13.437 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036533.169197_371.1'} 2023-10-11T15:02:13.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036533.169197_371.1'} 2023-10-11T15:02:13.438 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:02:13.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:13.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:13.439 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:02:13.495 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:13.496 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:13.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:13.528+0000 7f29649ef780 1 waiting to reach incremental sync.. 2023-10-11T15:02:18.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:18.528+0000 7f29649ef780 1 waiting to reach incremental sync.. 2023-10-11T15:02:23.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:23.529+0000 7f29649ef780 1 waiting to reach incremental sync.. 2023-10-11T15:02:28.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:28.529+0000 7f29649ef780 1 waiting to reach incremental sync.. 2023-10-11T15:02:33.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:33.531+0000 7f29649ef780 1 bucket sync caught up with source: 2023-10-11T15:02:33.533 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.33.6, , , , 00000000001.36.6, , , , , ] 2023-10-11T15:02:33.533 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.33.6, , , , 00000000001.36.6, , , , , ] 2023-10-11T15:02:33.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:33.531+0000 7f29649ef780 0 bucket checkpoint complete 2023-10-11T15:02:33.548 DEBUG:boto:path=/dghqvd-52/ 2023-10-11T15:02:33.549 DEBUG:boto:auth_path=/dghqvd-52/ 2023-10-11T15:02:33.549 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.550 DEBUG:boto:Path: /dghqvd-52/ 2023-10-11T15:02:33.550 DEBUG:boto:Data: 2023-10-11T15:02:33.550 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:33.551 DEBUG:boto:Port: 8001 2023-10-11T15:02:33.552 DEBUG:boto:Params: {} 2023-10-11T15:02:33.552 DEBUG:boto:Token: None 2023-10-11T15:02:33.553 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-52/ 2023-10-11T15:02:33.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YsxLY6dursT4fBV3W5fcZndWebY= 2023-10-11T15:02:33.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YsxLY6dursT4fBV3W5fcZndWebY=', 'Content-Length': '0'} 2023-10-11T15:02:33.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f8a18534897d78-006526b909-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.557 DEBUG:boto:path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.557 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.558 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.558 DEBUG:boto:Path: /dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.559 DEBUG:boto:Data: 2023-10-11T15:02:33.559 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-11T15:02:33.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:33.560 DEBUG:boto:Port: 8001 2023-10-11T15:02:33.560 DEBUG:boto:Params: {} 2023-10-11T15:02:33.560 DEBUG:boto:Token: None 2023-10-11T15:02:33.561 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u8JscxeSbGrQqe0w+6gVuSXSSHk= 2023-10-11T15:02:33.562 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u8JscxeSbGrQqe0w+6gVuSXSSHk=', 'Content-Length': '0'} 2023-10-11T15:02:33.564 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-52:daa316e9-869c-4074-a756-a8adfb4a3573.5751.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000896922d40538e5c5-006526b909-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.564 DEBUG:boto:path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.565 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.565 DEBUG:boto:Method: GET 2023-10-11T15:02:33.566 DEBUG:boto:Path: /dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.566 DEBUG:boto:Data: 2023-10-11T15:02:33.566 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-10-11T15:02:33.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:33.567 DEBUG:boto:Port: 8001 2023-10-11T15:02:33.568 DEBUG:boto:Params: {} 2023-10-11T15:02:33.568 DEBUG:boto:Token: None 2023-10-11T15:02:33.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:33 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dghqvd-52/testobj-sse-c 2023-10-11T15:02:33.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H6Wem8R6OLPs0ZwGuU6lbTidvJM= 2023-10-11T15:02:33.569 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H6Wem8R6OLPs0ZwGuU6lbTidvJM=', 'Content-Length': '0'} 2023-10-11T15:02:33.572 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-52:daa316e9-869c-4074-a756-a8adfb4a3573.5751.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000f0e9bf9c40a241fd-006526b909-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.572 DEBUG:boto:path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.573 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.573 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.574 DEBUG:boto:Path: /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.574 DEBUG:boto:Data: 2023-10-11T15:02:33.574 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:33.575 DEBUG:boto:Port: 8001 2023-10-11T15:02:33.575 DEBUG:boto:Params: {} 2023-10-11T15:02:33.576 DEBUG:boto:Token: None 2023-10-11T15:02:33.576 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dD6AR1AhOAgY7Zkmy3iNFYYDwdg= 2023-10-11T15:02:33.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dD6AR1AhOAgY7Zkmy3iNFYYDwdg=', 'Content-Length': '0'} 2023-10-11T15:02:33.579 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-52:daa316e9-869c-4074-a756-a8adfb4a3573.5751.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000fb5fcd04af9ea66d-006526b909-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.579 DEBUG:boto:path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.580 DEBUG:boto:auth_path=/dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.580 DEBUG:boto:Method: GET 2023-10-11T15:02:33.580 DEBUG:boto:Path: /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.581 DEBUG:boto:Data: 2023-10-11T15:02:33.581 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:02:33.582 DEBUG:boto:Port: 8001 2023-10-11T15:02:33.582 DEBUG:boto:Params: {} 2023-10-11T15:02:33.583 DEBUG:boto:Token: None 2023-10-11T15:02:33.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-52/testobj-sse-kms 2023-10-11T15:02:33.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GNZwQmSUS9Fz+e0EIVxYCNn50Kg= 2023-10-11T15:02:33.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GNZwQmSUS9Fz+e0EIVxYCNn50Kg=', 'Content-Length': '0'} 2023-10-11T15:02:33.586 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-52:daa316e9-869c-4074-a756-a8adfb4a3573.5751.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000024f50edd15720c16-006526b909-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-10-11T15:02:33.588 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-53 2023-10-11T15:02:33.588 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:02:33.588 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:02:33.589 DEBUG:boto:Method: PUT 2023-10-11T15:02:33.589 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:02:33.590 DEBUG:boto:Data: 2023-10-11T15:02:33.590 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.591 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.591 DEBUG:boto:Params: {} 2023-10-11T15:02:33.592 DEBUG:boto:Token: None 2023-10-11T15:02:33.592 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/ 2023-10-11T15:02:33.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sk6vZ7lOtMDeZpcM5KDcMDPewE8= 2023-10-11T15:02:33.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sk6vZ7lOtMDeZpcM5KDcMDPewE8=', 'Content-Length': '0'} 2023-10-11T15:02:33.608 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8e246e0dbb0bcef-006526b909-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.609 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:02:33.609 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:02:33.610 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.610 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:02:33.611 DEBUG:boto:Data: 2023-10-11T15:02:33.611 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.612 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.612 DEBUG:boto:Params: {} 2023-10-11T15:02:33.612 DEBUG:boto:Token: None 2023-10-11T15:02:33.613 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/ 2023-10-11T15:02:33.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI= 2023-10-11T15:02:33.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI=', 'Content-Length': '0'} 2023-10-11T15:02:33.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a5c57bf3f9a73ac-006526b909-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.617 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:02:33.617 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:02:33.617 DEBUG:boto:Method: PUT 2023-10-11T15:02:33.618 DEBUG:boto:Path: /dghqvd-53/a 2023-10-11T15:02:33.618 DEBUG:boto:Data: 2023-10-11T15:02:33.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:02:33.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.619 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.620 DEBUG:boto:Params: {} 2023-10-11T15:02:33.620 DEBUG:boto:Token: None 2023-10-11T15:02:33.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/a 2023-10-11T15:02:33.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kr2T/N/p/7i5BTe448xcnBw/AVA= 2023-10-11T15:02:33.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kr2T/N/p/7i5BTe448xcnBw/AVA='} 2023-10-11T15:02:33.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e03f3aaa2d4fab4b-006526b909-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.629 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:02:33.629 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:02:33.629 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.630 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:02:33.630 DEBUG:boto:Data: 2023-10-11T15:02:33.631 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.631 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.631 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.632 DEBUG:boto:Params: {} 2023-10-11T15:02:33.632 DEBUG:boto:Token: None 2023-10-11T15:02:33.632 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/ 2023-10-11T15:02:33.633 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI= 2023-10-11T15:02:33.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI=', 'Content-Length': '0'} 2023-10-11T15:02:33.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c37915e7774b597-006526b909-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.636 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:02:33.636 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:02:33.637 DEBUG:boto:Method: PUT 2023-10-11T15:02:33.637 DEBUG:boto:Path: /dghqvd-53/b 2023-10-11T15:02:33.637 DEBUG:boto:Data: 2023-10-11T15:02:33.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:02:33.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.639 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.639 DEBUG:boto:Params: {} 2023-10-11T15:02:33.639 DEBUG:boto:Token: None 2023-10-11T15:02:33.640 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/b 2023-10-11T15:02:33.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bkQxrrsbBPyGruz6zSgPQrB8OJA= 2023-10-11T15:02:33.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bkQxrrsbBPyGruz6zSgPQrB8OJA='} 2023-10-11T15:02:33.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6b01579e9e77270-006526b909-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.683 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:02:33.683 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:02:33.684 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.684 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:02:33.684 DEBUG:boto:Data: 2023-10-11T15:02:33.685 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.685 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.685 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.686 DEBUG:boto:Params: {} 2023-10-11T15:02:33.686 DEBUG:boto:Token: None 2023-10-11T15:02:33.687 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/ 2023-10-11T15:02:33.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI= 2023-10-11T15:02:33.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI=', 'Content-Length': '0'} 2023-10-11T15:02:33.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cb1d18fc9a3a2b5-006526b909-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.690 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:02:33.691 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:02:33.691 DEBUG:boto:Method: PUT 2023-10-11T15:02:33.691 DEBUG:boto:Path: /dghqvd-53/c 2023-10-11T15:02:33.692 DEBUG:boto:Data: 2023-10-11T15:02:33.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:02:33.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.693 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.693 DEBUG:boto:Params: {} 2023-10-11T15:02:33.694 DEBUG:boto:Token: None 2023-10-11T15:02:33.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/c 2023-10-11T15:02:33.695 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/J1AcsOfMKEq27lUsSKKRwXMwZc= 2023-10-11T15:02:33.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/J1AcsOfMKEq27lUsSKKRwXMwZc='} 2023-10-11T15:02:33.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8dc7cb2c38f7625-006526b909-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.701 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:02:33.702 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:02:33.702 DEBUG:boto:Method: HEAD 2023-10-11T15:02:33.702 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:02:33.703 DEBUG:boto:Data: 2023-10-11T15:02:33.703 DEBUG:boto:Headers: {} 2023-10-11T15:02:33.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.704 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.704 DEBUG:boto:Params: {} 2023-10-11T15:02:33.705 DEBUG:boto:Token: None 2023-10-11T15:02:33.705 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/ 2023-10-11T15:02:33.705 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI= 2023-10-11T15:02:33.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2XlOO70FPreg3fLsecdKVG3b8XI=', 'Content-Length': '0'} 2023-10-11T15:02:33.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e629b233e0e71e-006526b909-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.709 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:02:33.709 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:02:33.709 DEBUG:boto:Method: PUT 2023-10-11T15:02:33.710 DEBUG:boto:Path: /dghqvd-53/d 2023-10-11T15:02:33.710 DEBUG:boto:Data: 2023-10-11T15:02:33.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:02:33.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:02:33.711 DEBUG:boto:Port: 8000 2023-10-11T15:02:33.712 DEBUG:boto:Params: {} 2023-10-11T15:02:33.712 DEBUG:boto:Token: None 2023-10-11T15:02:33.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:02:33 GMT /dghqvd-53/d 2023-10-11T15:02:33.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AK3yzDJSDhW3fS3XOn9S3+qbsCY= 2023-10-11T15:02:33.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:02:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AK3yzDJSDhW3fS3XOn9S3+qbsCY='} 2023-10-11T15:02:33.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff12a4fb45d43af9-006526b909-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:02:33.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:33.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:33.720 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:02:33.770 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:33.771 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:33.800 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:02:33.812 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:02:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:02:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:02:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:02:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:02:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:02:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:02:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036553.605598_375.1", 2023-10-11T15:02:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:02:33.605598Z" 2023-10-11T15:02:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.815 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:02:33.816 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036553.605598_375.1'} 2023-10-11T15:02:33.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:02:33.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:33.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:02:33.818 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:02:33.867 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:33.867 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:33.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:33.893+0000 7f14f3579780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:02:33.896 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:33.894+0000 7f14f3579780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:02:33.908 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:02:33.908 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:02:33.908 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:02:33.908 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:02:33.909 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:02:33.909 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:02:33.909 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:33.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.909 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:02:33.910 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.910 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:02:33.910 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:33.910 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:33.910 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:02:33.911 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:33.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:02:33.911 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:33.911 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:02:33.911 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:33.912 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.912 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.912 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.912 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:02:33.912 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:33.913 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:02:33.914 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:33.914 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.914 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.915 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:02:33.915 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:33.915 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:33.915 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:02:33.915 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:33.916 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:33.916 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:33.916 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:02:33.916 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:33.916 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:02:33.917 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036553.605598_375.1", 2023-10-11T15:02:33.918 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:02:33.918 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:02:33.918 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:02:33.918 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:33.605598Z", 2023-10-11T15:02:33.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:02:33.919 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.919 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:02:33.919 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:02:33.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:02:33Z" 2023-10-11T15:02:33.920 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:02:33.921 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036553.605598_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:02:33.605598Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:02:33Z" } 2023-10-11T15:02:33.922 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036553.605598_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:02:33.605598Z', 'realm_epoch': 2}}] 2023-10-11T15:02:33.923 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036553.605598_375.1'} 2023-10-11T15:02:33.923 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036553.605598_375.1'} 2023-10-11T15:02:33.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:02:33.924 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-53 2023-10-11T15:02:33.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:33.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:33.925 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:02:33.981 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:33.981 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:34.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:34.012+0000 7f2267a13780 1 waiting to reach incremental sync.. 2023-10-11T15:02:39.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:39.013+0000 7f2267a13780 1 waiting to reach incremental sync.. 2023-10-11T15:02:44.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:44.013+0000 7f2267a13780 1 waiting to reach incremental sync.. 2023-10-11T15:02:49.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:49.015+0000 7f2267a13780 1 bucket sync caught up with source: 2023-10-11T15:02:49.018 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.35.6, , , , 00000000001.42.6, , , 00000000001.27.6, , 00000000001.56.6] 2023-10-11T15:02:49.018 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.35.6, , , , 00000000001.42.6, , , 00000000001.27.6, , 00000000001.56.6] 2023-10-11T15:02:49.018 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:49.015+0000 7f2267a13780 0 bucket checkpoint complete 2023-10-11T15:02:49.034 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-53 2023-10-11T15:02:49.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:49.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:02:49.036 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:02:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:02:49.098 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:02:49.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:49.131+0000 7f8954349780 1 waiting to reach incremental sync.. 2023-10-11T15:02:54.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:54.132+0000 7f8954349780 1 waiting to reach incremental sync.. 2023-10-11T15:02:59.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:02:59.132+0000 7f8954349780 1 waiting to reach incremental sync.. 2023-10-11T15:03:04.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:04.132+0000 7f8954349780 1 waiting to reach incremental sync.. 2023-10-11T15:03:09.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:09.134+0000 7f8954349780 1 bucket sync caught up with source: 2023-10-11T15:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.181.6, , , , 00000000001.173.6, , , 00000000001.92.6, , 00000000001.288.6] 2023-10-11T15:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.181.6, , , , 00000000001.173.6, , , 00000000001.92.6, , 00000000001.288.6] 2023-10-11T15:03:09.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:09.134+0000 7f8954349780 0 bucket checkpoint complete 2023-10-11T15:03:09.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-53 zones={a2, a1} 2023-10-11T15:03:09.154 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.154 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.154 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.155 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:03:09.155 DEBUG:boto:Data: 2023-10-11T15:03:09.156 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.157 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.157 DEBUG:boto:Params: {} 2023-10-11T15:03:09.157 DEBUG:boto:Token: None 2023-10-11T15:03:09.158 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.159 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3l4RC2B0wWGKrEAj5HSSLqA+2E= 2023-10-11T15:03:09.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3l4RC2B0wWGKrEAj5HSSLqA+2E=', 'Content-Length': '0'} 2023-10-11T15:03:09.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005168dfc632d45b55-006526b92d-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.162 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.163 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.163 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.164 DEBUG:boto:Path: /dghqvd-53/ 2023-10-11T15:03:09.164 DEBUG:boto:Data: 2023-10-11T15:03:09.165 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.166 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.166 DEBUG:boto:Params: {} 2023-10-11T15:03:09.167 DEBUG:boto:Token: None 2023-10-11T15:03:09.167 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3l4RC2B0wWGKrEAj5HSSLqA+2E= 2023-10-11T15:03:09.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3l4RC2B0wWGKrEAj5HSSLqA+2E=', 'Content-Length': '0'} 2023-10-11T15:03:09.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c010f1dcb0ad14bc-006526b92d-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:03:09.171 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.172 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.172 DEBUG:boto:path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.173 DEBUG:boto:auth_path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.173 DEBUG:boto:Method: GET 2023-10-11T15:03:09.173 DEBUG:boto:Path: /dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.174 DEBUG:boto:Data: 2023-10-11T15:03:09.174 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.175 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.176 DEBUG:boto:Params: {} 2023-10-11T15:03:09.176 DEBUG:boto:Token: None 2023-10-11T15:03:09.177 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/?versions 2023-10-11T15:03:09.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc= 2023-10-11T15:03:09.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc=', 'Content-Length': '0'} 2023-10-11T15:03:09.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e68d954b1871b7a-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.180 DEBUG:boto:b'dghqvd-53999falseanulltrue2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.181 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:03:09.182 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:03:09.182 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:03:09.183 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:03:09.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:03:09.183 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.184 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.184 DEBUG:boto:path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.185 DEBUG:boto:auth_path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.185 DEBUG:boto:Method: GET 2023-10-11T15:03:09.186 DEBUG:boto:Path: /dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.186 DEBUG:boto:Data: 2023-10-11T15:03:09.186 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.187 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.188 DEBUG:boto:Params: {} 2023-10-11T15:03:09.188 DEBUG:boto:Token: None 2023-10-11T15:03:09.189 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/?versions 2023-10-11T15:03:09.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc= 2023-10-11T15:03:09.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc=', 'Content-Length': '0'} 2023-10-11T15:03:09.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae517254910f09eb-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.192 DEBUG:boto:b'dghqvd-53999falseanulltrue2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.193 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:03:09.194 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:03:09.194 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:03:09.194 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:03:09.195 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.195 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.196 DEBUG:boto:path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.196 DEBUG:boto:auth_path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.196 DEBUG:boto:Method: GET 2023-10-11T15:03:09.197 DEBUG:boto:Path: /dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.197 DEBUG:boto:Data: 2023-10-11T15:03:09.198 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.199 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.199 DEBUG:boto:Params: {} 2023-10-11T15:03:09.199 DEBUG:boto:Token: None 2023-10-11T15:03:09.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/?versions 2023-10-11T15:03:09.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc= 2023-10-11T15:03:09.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc=', 'Content-Length': '0'} 2023-10-11T15:03:09.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ad900948e0e998-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.205 DEBUG:boto:b'dghqvd-53999falseanulltrue2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.206 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.206 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.207 DEBUG:boto:path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.207 DEBUG:boto:auth_path=/dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.207 DEBUG:boto:Method: GET 2023-10-11T15:03:09.208 DEBUG:boto:Path: /dghqvd-53/?versions&max-keys=999 2023-10-11T15:03:09.208 DEBUG:boto:Data: 2023-10-11T15:03:09.209 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.209 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.210 DEBUG:boto:Params: {} 2023-10-11T15:03:09.210 DEBUG:boto:Token: None 2023-10-11T15:03:09.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/?versions 2023-10-11T15:03:09.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc= 2023-10-11T15:03:09.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+qMd9pZ0MqT+EDlL/Yq5L7wvPcc=', 'Content-Length': '0'} 2023-10-11T15:03:09.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d4e922c4da5d36-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.215 DEBUG:boto:b'dghqvd-53999falseanulltrue2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:09.216 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.217 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.217 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.217 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.218 DEBUG:boto:Method: GET 2023-10-11T15:03:09.218 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.219 DEBUG:boto:Data: 2023-10-11T15:03:09.219 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.220 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.221 DEBUG:boto:Params: {} 2023-10-11T15:03:09.221 DEBUG:boto:Token: None 2023-10-11T15:03:09.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08= 2023-10-11T15:03:09.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08=', 'Content-Length': '0'} 2023-10-11T15:03:09.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ce5d70897afcbd-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.225 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.226 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.226 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.227 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.227 DEBUG:boto:Method: GET 2023-10-11T15:03:09.228 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.228 DEBUG:boto:Data: 2023-10-11T15:03:09.228 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.229 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.230 DEBUG:boto:Params: {} 2023-10-11T15:03:09.230 DEBUG:boto:Token: None 2023-10-11T15:03:09.231 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08= 2023-10-11T15:03:09.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08=', 'Content-Length': '0'} 2023-10-11T15:03:09.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd436fc437d9648-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.278 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.279 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.279 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.279 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.280 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.280 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.281 DEBUG:boto:Data: 2023-10-11T15:03:09.281 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.282 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.282 DEBUG:boto:Params: {} 2023-10-11T15:03:09.283 DEBUG:boto:Token: None 2023-10-11T15:03:09.283 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NalsvXFb8DrCNlzRcVQoYi3JNS8= 2023-10-11T15:03:09.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NalsvXFb8DrCNlzRcVQoYi3JNS8=', 'Content-Length': '0'} 2023-10-11T15:03:09.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b0139756ec42700-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.287 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.287 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.287 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.288 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.288 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.289 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.289 DEBUG:boto:Data: 2023-10-11T15:03:09.289 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.290 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.291 DEBUG:boto:Params: {} 2023-10-11T15:03:09.291 DEBUG:boto:Token: None 2023-10-11T15:03:09.292 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NalsvXFb8DrCNlzRcVQoYi3JNS8= 2023-10-11T15:03:09.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NalsvXFb8DrCNlzRcVQoYi3JNS8=', 'Content-Length': '0'} 2023-10-11T15:03:09.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d1861a7a309cd4-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:09.295 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.296 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.296 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.296 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.297 DEBUG:boto:Method: GET 2023-10-11T15:03:09.297 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.298 DEBUG:boto:Data: 2023-10-11T15:03:09.298 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.299 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.300 DEBUG:boto:Params: {} 2023-10-11T15:03:09.300 DEBUG:boto:Token: None 2023-10-11T15:03:09.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08= 2023-10-11T15:03:09.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08=', 'Content-Length': '0'} 2023-10-11T15:03:09.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058dfd5a1bb070896-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.304 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.304 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.305 DEBUG:boto:path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.305 DEBUG:boto:auth_path=/dghqvd-53/a?versionId=null 2023-10-11T15:03:09.306 DEBUG:boto:Method: GET 2023-10-11T15:03:09.306 DEBUG:boto:Path: /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.307 DEBUG:boto:Data: 2023-10-11T15:03:09.307 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.308 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.308 DEBUG:boto:Params: {} 2023-10-11T15:03:09.309 DEBUG:boto:Token: None 2023-10-11T15:03:09.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a?versionId=null 2023-10-11T15:03:09.310 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08= 2023-10-11T15:03:09.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KmkFOox5UtRKwwLX8MckbTIua08=', 'Content-Length': '0'} 2023-10-11T15:03:09.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f65139cec49261-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.313 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.313 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.314 DEBUG:boto:path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.314 DEBUG:boto:auth_path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.315 DEBUG:boto:Method: GET 2023-10-11T15:03:09.315 DEBUG:boto:Path: /dghqvd-53/?prefix=a 2023-10-11T15:03:09.315 DEBUG:boto:Data: 2023-10-11T15:03:09.316 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.317 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.317 DEBUG:boto:Params: {} 2023-10-11T15:03:09.318 DEBUG:boto:Token: None 2023-10-11T15:03:09.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007178e5b6aa8086e3-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.322 DEBUG:boto:b'dghqvd-53a1000falsea2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.323 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.323 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.323 DEBUG:boto:path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.324 DEBUG:boto:auth_path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.324 DEBUG:boto:Method: GET 2023-10-11T15:03:09.325 DEBUG:boto:Path: /dghqvd-53/?prefix=a 2023-10-11T15:03:09.325 DEBUG:boto:Data: 2023-10-11T15:03:09.326 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.326 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.327 DEBUG:boto:Params: {} 2023-10-11T15:03:09.327 DEBUG:boto:Token: None 2023-10-11T15:03:09.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005076e8b99b778b40-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.332 DEBUG:boto:b'dghqvd-53a1000falsea2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.332 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.333 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.333 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.334 DEBUG:boto:Path: /dghqvd-53/a 2023-10-11T15:03:09.334 DEBUG:boto:Data: 2023-10-11T15:03:09.334 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.335 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.336 DEBUG:boto:Params: {} 2023-10-11T15:03:09.336 DEBUG:boto:Token: None 2023-10-11T15:03:09.336 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a 2023-10-11T15:03:09.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2FOP+vB0eoaDmscJbFimpAji30= 2023-10-11T15:03:09.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2FOP+vB0eoaDmscJbFimpAji30=', 'Content-Length': '0'} 2023-10-11T15:03:09.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0de6bff519166e8-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.340 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.340 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.341 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.341 DEBUG:boto:Path: /dghqvd-53/a 2023-10-11T15:03:09.341 DEBUG:boto:Data: 2023-10-11T15:03:09.342 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.343 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.343 DEBUG:boto:Params: {} 2023-10-11T15:03:09.343 DEBUG:boto:Token: None 2023-10-11T15:03:09.344 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a 2023-10-11T15:03:09.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2FOP+vB0eoaDmscJbFimpAji30= 2023-10-11T15:03:09.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2FOP+vB0eoaDmscJbFimpAji30=', 'Content-Length': '0'} 2023-10-11T15:03:09.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da22315e8d2b5fe-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:09.347 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.348 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.348 DEBUG:boto:Method: GET 2023-10-11T15:03:09.349 DEBUG:boto:Path: /dghqvd-53/a 2023-10-11T15:03:09.349 DEBUG:boto:Data: 2023-10-11T15:03:09.350 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.351 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.351 DEBUG:boto:Params: {} 2023-10-11T15:03:09.352 DEBUG:boto:Token: None 2023-10-11T15:03:09.352 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a 2023-10-11T15:03:09.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:na85UHjSShjb8ATuAMXgtT/bFwA= 2023-10-11T15:03:09.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:na85UHjSShjb8ATuAMXgtT/bFwA=', 'Content-Length': '0'} 2023-10-11T15:03:09.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020e36305dd67da1-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.356 DEBUG:boto:path=/dghqvd-53/a 2023-10-11T15:03:09.356 DEBUG:boto:auth_path=/dghqvd-53/a 2023-10-11T15:03:09.356 DEBUG:boto:Method: GET 2023-10-11T15:03:09.357 DEBUG:boto:Path: /dghqvd-53/a 2023-10-11T15:03:09.357 DEBUG:boto:Data: 2023-10-11T15:03:09.358 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.359 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.359 DEBUG:boto:Params: {} 2023-10-11T15:03:09.359 DEBUG:boto:Token: None 2023-10-11T15:03:09.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/a 2023-10-11T15:03:09.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:na85UHjSShjb8ATuAMXgtT/bFwA= 2023-10-11T15:03:09.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:na85UHjSShjb8ATuAMXgtT/bFwA=', 'Content-Length': '0'} 2023-10-11T15:03:09.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006493f8c84b15f138-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.406 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.407 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.407 DEBUG:boto:path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.408 DEBUG:boto:auth_path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.408 DEBUG:boto:Method: GET 2023-10-11T15:03:09.409 DEBUG:boto:Path: /dghqvd-53/?prefix=a 2023-10-11T15:03:09.409 DEBUG:boto:Data: 2023-10-11T15:03:09.409 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.410 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.411 DEBUG:boto:Params: {} 2023-10-11T15:03:09.411 DEBUG:boto:Token: None 2023-10-11T15:03:09.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371044c6bde6724d-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.416 DEBUG:boto:b'dghqvd-53a1000falsea2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.416 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.417 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.417 DEBUG:boto:path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.418 DEBUG:boto:auth_path=/dghqvd-53/?prefix=a 2023-10-11T15:03:09.418 DEBUG:boto:Method: GET 2023-10-11T15:03:09.419 DEBUG:boto:Path: /dghqvd-53/?prefix=a 2023-10-11T15:03:09.419 DEBUG:boto:Data: 2023-10-11T15:03:09.419 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.420 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.421 DEBUG:boto:Params: {} 2023-10-11T15:03:09.421 DEBUG:boto:Token: None 2023-10-11T15:03:09.422 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.422 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f95b80c7e93998-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.425 DEBUG:boto:b'dghqvd-53a1000falsea2023-10-11T15:02:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:09.426 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.427 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.427 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.428 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.428 DEBUG:boto:Method: GET 2023-10-11T15:03:09.429 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.429 DEBUG:boto:Data: 2023-10-11T15:03:09.429 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.430 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.431 DEBUG:boto:Params: {} 2023-10-11T15:03:09.431 DEBUG:boto:Token: None 2023-10-11T15:03:09.432 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4= 2023-10-11T15:03:09.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4=', 'Content-Length': '0'} 2023-10-11T15:03:09.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007932e9de96663a60-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.478 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.479 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.479 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.480 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.480 DEBUG:boto:Method: GET 2023-10-11T15:03:09.481 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.481 DEBUG:boto:Data: 2023-10-11T15:03:09.482 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.483 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.484 DEBUG:boto:Params: {} 2023-10-11T15:03:09.484 DEBUG:boto:Token: None 2023-10-11T15:03:09.485 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4= 2023-10-11T15:03:09.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4=', 'Content-Length': '0'} 2023-10-11T15:03:09.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000673f62c9c7e15ce4-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.489 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.489 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.490 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.490 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.491 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.491 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.491 DEBUG:boto:Data: 2023-10-11T15:03:09.492 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.493 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.493 DEBUG:boto:Params: {} 2023-10-11T15:03:09.494 DEBUG:boto:Token: None 2023-10-11T15:03:09.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BTT1/aCch760ios06yvG6oSPRsE= 2023-10-11T15:03:09.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BTT1/aCch760ios06yvG6oSPRsE=', 'Content-Length': '0'} 2023-10-11T15:03:09.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004944e2526dbaf519-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.497 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.498 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.498 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.499 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.499 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.500 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.500 DEBUG:boto:Data: 2023-10-11T15:03:09.500 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.501 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.502 DEBUG:boto:Params: {} 2023-10-11T15:03:09.502 DEBUG:boto:Token: None 2023-10-11T15:03:09.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.503 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BTT1/aCch760ios06yvG6oSPRsE= 2023-10-11T15:03:09.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BTT1/aCch760ios06yvG6oSPRsE=', 'Content-Length': '0'} 2023-10-11T15:03:09.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b24d4f2319e0317-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:09.507 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.507 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.508 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.508 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.508 DEBUG:boto:Method: GET 2023-10-11T15:03:09.509 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.509 DEBUG:boto:Data: 2023-10-11T15:03:09.510 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.511 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.511 DEBUG:boto:Params: {} 2023-10-11T15:03:09.512 DEBUG:boto:Token: None 2023-10-11T15:03:09.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4= 2023-10-11T15:03:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4=', 'Content-Length': '0'} 2023-10-11T15:03:09.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7029cb79cf6f2a2-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.516 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.516 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.517 DEBUG:boto:path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.517 DEBUG:boto:auth_path=/dghqvd-53/b?versionId=null 2023-10-11T15:03:09.518 DEBUG:boto:Method: GET 2023-10-11T15:03:09.518 DEBUG:boto:Path: /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.519 DEBUG:boto:Data: 2023-10-11T15:03:09.519 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.520 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.520 DEBUG:boto:Params: {} 2023-10-11T15:03:09.521 DEBUG:boto:Token: None 2023-10-11T15:03:09.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b?versionId=null 2023-10-11T15:03:09.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4= 2023-10-11T15:03:09.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uCulnfN6HP++e9sVKo1GA3gbkf4=', 'Content-Length': '0'} 2023-10-11T15:03:09.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff336ee496a1151a-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.525 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.526 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.526 DEBUG:boto:path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.526 DEBUG:boto:auth_path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.527 DEBUG:boto:Method: GET 2023-10-11T15:03:09.527 DEBUG:boto:Path: /dghqvd-53/?prefix=b 2023-10-11T15:03:09.528 DEBUG:boto:Data: 2023-10-11T15:03:09.528 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.529 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.529 DEBUG:boto:Params: {} 2023-10-11T15:03:09.530 DEBUG:boto:Token: None 2023-10-11T15:03:09.530 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54bbd3ce9be55e8-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.535 DEBUG:boto:b'dghqvd-53b1000falseb2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.535 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.536 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.536 DEBUG:boto:path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.537 DEBUG:boto:auth_path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.537 DEBUG:boto:Method: GET 2023-10-11T15:03:09.537 DEBUG:boto:Path: /dghqvd-53/?prefix=b 2023-10-11T15:03:09.538 DEBUG:boto:Data: 2023-10-11T15:03:09.538 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.539 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.540 DEBUG:boto:Params: {} 2023-10-11T15:03:09.540 DEBUG:boto:Token: None 2023-10-11T15:03:09.541 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.541 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780ac0d433ca3e4d-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.545 DEBUG:boto:b'dghqvd-53b1000falseb2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.545 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.546 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.546 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.547 DEBUG:boto:Path: /dghqvd-53/b 2023-10-11T15:03:09.547 DEBUG:boto:Data: 2023-10-11T15:03:09.548 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.548 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.549 DEBUG:boto:Params: {} 2023-10-11T15:03:09.549 DEBUG:boto:Token: None 2023-10-11T15:03:09.550 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b 2023-10-11T15:03:09.550 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uHkWctHyWA5HGvwfZBUzmOF8xtk= 2023-10-11T15:03:09.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uHkWctHyWA5HGvwfZBUzmOF8xtk=', 'Content-Length': '0'} 2023-10-11T15:03:09.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356e2b0ed1e1d970-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.553 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.553 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.554 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.554 DEBUG:boto:Path: /dghqvd-53/b 2023-10-11T15:03:09.555 DEBUG:boto:Data: 2023-10-11T15:03:09.555 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.556 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.556 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.556 DEBUG:boto:Params: {} 2023-10-11T15:03:09.557 DEBUG:boto:Token: None 2023-10-11T15:03:09.557 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b 2023-10-11T15:03:09.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uHkWctHyWA5HGvwfZBUzmOF8xtk= 2023-10-11T15:03:09.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uHkWctHyWA5HGvwfZBUzmOF8xtk=', 'Content-Length': '0'} 2023-10-11T15:03:09.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f30cdeced7c02047-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:09.561 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.562 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.562 DEBUG:boto:Method: GET 2023-10-11T15:03:09.563 DEBUG:boto:Path: /dghqvd-53/b 2023-10-11T15:03:09.563 DEBUG:boto:Data: 2023-10-11T15:03:09.563 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.564 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.565 DEBUG:boto:Params: {} 2023-10-11T15:03:09.565 DEBUG:boto:Token: None 2023-10-11T15:03:09.566 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b 2023-10-11T15:03:09.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CQ6mECE0y+FFtTG/nQ53F310x+Q= 2023-10-11T15:03:09.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CQ6mECE0y+FFtTG/nQ53F310x+Q=', 'Content-Length': '0'} 2023-10-11T15:03:09.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db152621a6a3dd0a-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.610 DEBUG:boto:path=/dghqvd-53/b 2023-10-11T15:03:09.611 DEBUG:boto:auth_path=/dghqvd-53/b 2023-10-11T15:03:09.611 DEBUG:boto:Method: GET 2023-10-11T15:03:09.612 DEBUG:boto:Path: /dghqvd-53/b 2023-10-11T15:03:09.612 DEBUG:boto:Data: 2023-10-11T15:03:09.613 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.613 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.614 DEBUG:boto:Params: {} 2023-10-11T15:03:09.614 DEBUG:boto:Token: None 2023-10-11T15:03:09.615 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/b 2023-10-11T15:03:09.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CQ6mECE0y+FFtTG/nQ53F310x+Q= 2023-10-11T15:03:09.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CQ6mECE0y+FFtTG/nQ53F310x+Q=', 'Content-Length': '0'} 2023-10-11T15:03:09.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed4bbee45abcce7-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.618 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.619 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.619 DEBUG:boto:path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.620 DEBUG:boto:auth_path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.620 DEBUG:boto:Method: GET 2023-10-11T15:03:09.621 DEBUG:boto:Path: /dghqvd-53/?prefix=b 2023-10-11T15:03:09.621 DEBUG:boto:Data: 2023-10-11T15:03:09.622 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.622 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.623 DEBUG:boto:Params: {} 2023-10-11T15:03:09.623 DEBUG:boto:Token: None 2023-10-11T15:03:09.624 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c789bf272aa8a73-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.628 DEBUG:boto:b'dghqvd-53b1000falseb2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.628 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.629 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.629 DEBUG:boto:path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.630 DEBUG:boto:auth_path=/dghqvd-53/?prefix=b 2023-10-11T15:03:09.630 DEBUG:boto:Method: GET 2023-10-11T15:03:09.631 DEBUG:boto:Path: /dghqvd-53/?prefix=b 2023-10-11T15:03:09.631 DEBUG:boto:Data: 2023-10-11T15:03:09.631 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.632 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.633 DEBUG:boto:Params: {} 2023-10-11T15:03:09.633 DEBUG:boto:Token: None 2023-10-11T15:03:09.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e93c4bd14654f4c-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.638 DEBUG:boto:b'dghqvd-53b1000falseb2023-10-11T15:02:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:09.639 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.639 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.640 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.640 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.641 DEBUG:boto:Method: GET 2023-10-11T15:03:09.641 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.641 DEBUG:boto:Data: 2023-10-11T15:03:09.642 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.643 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.643 DEBUG:boto:Params: {} 2023-10-11T15:03:09.644 DEBUG:boto:Token: None 2023-10-11T15:03:09.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0= 2023-10-11T15:03:09.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0=', 'Content-Length': '0'} 2023-10-11T15:03:09.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c4baafc468c962-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.648 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.648 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.649 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.649 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.649 DEBUG:boto:Method: GET 2023-10-11T15:03:09.650 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.650 DEBUG:boto:Data: 2023-10-11T15:03:09.651 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.652 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.652 DEBUG:boto:Params: {} 2023-10-11T15:03:09.653 DEBUG:boto:Token: None 2023-10-11T15:03:09.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0= 2023-10-11T15:03:09.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0=', 'Content-Length': '0'} 2023-10-11T15:03:09.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518de625afcb1196-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.657 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.658 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.658 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.658 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.659 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.659 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.660 DEBUG:boto:Data: 2023-10-11T15:03:09.660 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.661 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.661 DEBUG:boto:Params: {} 2023-10-11T15:03:09.662 DEBUG:boto:Token: None 2023-10-11T15:03:09.662 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8nIppmCk6P34PrDi6SNEqWSmG2U= 2023-10-11T15:03:09.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8nIppmCk6P34PrDi6SNEqWSmG2U=', 'Content-Length': '0'} 2023-10-11T15:03:09.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab16ccfa1f95829-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.666 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.666 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.667 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.667 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.668 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.668 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.668 DEBUG:boto:Data: 2023-10-11T15:03:09.669 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.669 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.670 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.670 DEBUG:boto:Params: {} 2023-10-11T15:03:09.670 DEBUG:boto:Token: None 2023-10-11T15:03:09.671 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.671 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8nIppmCk6P34PrDi6SNEqWSmG2U= 2023-10-11T15:03:09.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8nIppmCk6P34PrDi6SNEqWSmG2U=', 'Content-Length': '0'} 2023-10-11T15:03:09.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d271f1d7a3b49724-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:09.675 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.675 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.676 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.676 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.676 DEBUG:boto:Method: GET 2023-10-11T15:03:09.677 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.677 DEBUG:boto:Data: 2023-10-11T15:03:09.678 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.679 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.679 DEBUG:boto:Params: {} 2023-10-11T15:03:09.680 DEBUG:boto:Token: None 2023-10-11T15:03:09.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0= 2023-10-11T15:03:09.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0=', 'Content-Length': '0'} 2023-10-11T15:03:09.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9211481d111f332-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.726 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.727 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.727 DEBUG:boto:path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.728 DEBUG:boto:auth_path=/dghqvd-53/c?versionId=null 2023-10-11T15:03:09.728 DEBUG:boto:Method: GET 2023-10-11T15:03:09.729 DEBUG:boto:Path: /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.729 DEBUG:boto:Data: 2023-10-11T15:03:09.729 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.730 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.731 DEBUG:boto:Params: {} 2023-10-11T15:03:09.731 DEBUG:boto:Token: None 2023-10-11T15:03:09.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c?versionId=null 2023-10-11T15:03:09.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0= 2023-10-11T15:03:09.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKl034EegkF0SgSSxlXuTvkOkS0=', 'Content-Length': '0'} 2023-10-11T15:03:09.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049969f305900e5f3-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.735 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.736 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.736 DEBUG:boto:path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.736 DEBUG:boto:auth_path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.737 DEBUG:boto:Method: GET 2023-10-11T15:03:09.737 DEBUG:boto:Path: /dghqvd-53/?prefix=c 2023-10-11T15:03:09.738 DEBUG:boto:Data: 2023-10-11T15:03:09.738 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.739 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.739 DEBUG:boto:Params: {} 2023-10-11T15:03:09.740 DEBUG:boto:Token: None 2023-10-11T15:03:09.740 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a8dadf12d1267e-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.744 DEBUG:boto:b'dghqvd-53c1000falsec2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.745 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.745 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.746 DEBUG:boto:path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.746 DEBUG:boto:auth_path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.747 DEBUG:boto:Method: GET 2023-10-11T15:03:09.747 DEBUG:boto:Path: /dghqvd-53/?prefix=c 2023-10-11T15:03:09.748 DEBUG:boto:Data: 2023-10-11T15:03:09.748 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.749 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.749 DEBUG:boto:Params: {} 2023-10-11T15:03:09.750 DEBUG:boto:Token: None 2023-10-11T15:03:09.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d928ad779689ba47-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.754 DEBUG:boto:b'dghqvd-53c1000falsec2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.755 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.755 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.756 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.756 DEBUG:boto:Path: /dghqvd-53/c 2023-10-11T15:03:09.757 DEBUG:boto:Data: 2023-10-11T15:03:09.757 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.758 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.758 DEBUG:boto:Params: {} 2023-10-11T15:03:09.759 DEBUG:boto:Token: None 2023-10-11T15:03:09.759 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c 2023-10-11T15:03:09.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bjVjkUGRUukNQzI5cdI0iS4DtaE= 2023-10-11T15:03:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bjVjkUGRUukNQzI5cdI0iS4DtaE=', 'Content-Length': '0'} 2023-10-11T15:03:09.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dcbcf6ee812307c-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.763 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.763 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.763 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.764 DEBUG:boto:Path: /dghqvd-53/c 2023-10-11T15:03:09.764 DEBUG:boto:Data: 2023-10-11T15:03:09.765 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.766 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.766 DEBUG:boto:Params: {} 2023-10-11T15:03:09.766 DEBUG:boto:Token: None 2023-10-11T15:03:09.767 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c 2023-10-11T15:03:09.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bjVjkUGRUukNQzI5cdI0iS4DtaE= 2023-10-11T15:03:09.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bjVjkUGRUukNQzI5cdI0iS4DtaE=', 'Content-Length': '0'} 2023-10-11T15:03:09.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd702deda9f0e04d-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:09.771 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.771 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.772 DEBUG:boto:Method: GET 2023-10-11T15:03:09.772 DEBUG:boto:Path: /dghqvd-53/c 2023-10-11T15:03:09.772 DEBUG:boto:Data: 2023-10-11T15:03:09.773 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.774 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.774 DEBUG:boto:Params: {} 2023-10-11T15:03:09.775 DEBUG:boto:Token: None 2023-10-11T15:03:09.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c 2023-10-11T15:03:09.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bRIjRbV/2EI5uedriLSecD9ySfI= 2023-10-11T15:03:09.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bRIjRbV/2EI5uedriLSecD9ySfI=', 'Content-Length': '0'} 2023-10-11T15:03:09.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f7c716b881b27a-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.779 DEBUG:boto:path=/dghqvd-53/c 2023-10-11T15:03:09.779 DEBUG:boto:auth_path=/dghqvd-53/c 2023-10-11T15:03:09.780 DEBUG:boto:Method: GET 2023-10-11T15:03:09.780 DEBUG:boto:Path: /dghqvd-53/c 2023-10-11T15:03:09.780 DEBUG:boto:Data: 2023-10-11T15:03:09.781 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.782 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.782 DEBUG:boto:Params: {} 2023-10-11T15:03:09.783 DEBUG:boto:Token: None 2023-10-11T15:03:09.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/c 2023-10-11T15:03:09.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bRIjRbV/2EI5uedriLSecD9ySfI= 2023-10-11T15:03:09.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bRIjRbV/2EI5uedriLSecD9ySfI=', 'Content-Length': '0'} 2023-10-11T15:03:09.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895e4dd9771b25d1-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.787 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.787 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.788 DEBUG:boto:path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.788 DEBUG:boto:auth_path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.789 DEBUG:boto:Method: GET 2023-10-11T15:03:09.789 DEBUG:boto:Path: /dghqvd-53/?prefix=c 2023-10-11T15:03:09.789 DEBUG:boto:Data: 2023-10-11T15:03:09.790 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.791 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.791 DEBUG:boto:Params: {} 2023-10-11T15:03:09.792 DEBUG:boto:Token: None 2023-10-11T15:03:09.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba98b59b15159c64-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.796 DEBUG:boto:b'dghqvd-53c1000falsec2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.797 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.797 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.797 DEBUG:boto:path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.798 DEBUG:boto:auth_path=/dghqvd-53/?prefix=c 2023-10-11T15:03:09.798 DEBUG:boto:Method: GET 2023-10-11T15:03:09.799 DEBUG:boto:Path: /dghqvd-53/?prefix=c 2023-10-11T15:03:09.799 DEBUG:boto:Data: 2023-10-11T15:03:09.800 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.800 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.801 DEBUG:boto:Params: {} 2023-10-11T15:03:09.801 DEBUG:boto:Token: None 2023-10-11T15:03:09.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2aa032004c9300-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.846 DEBUG:boto:b'dghqvd-53c1000falsec2023-10-11T15:02:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:09.847 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.848 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.848 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.849 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.849 DEBUG:boto:Method: GET 2023-10-11T15:03:09.850 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.850 DEBUG:boto:Data: 2023-10-11T15:03:09.850 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.851 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.852 DEBUG:boto:Params: {} 2023-10-11T15:03:09.852 DEBUG:boto:Token: None 2023-10-11T15:03:09.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4= 2023-10-11T15:03:09.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4=', 'Content-Length': '0'} 2023-10-11T15:03:09.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b96fb923c69c5d-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.857 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.857 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.857 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.858 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.858 DEBUG:boto:Method: GET 2023-10-11T15:03:09.859 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.859 DEBUG:boto:Data: 2023-10-11T15:03:09.860 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.860 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.861 DEBUG:boto:Params: {} 2023-10-11T15:03:09.861 DEBUG:boto:Token: None 2023-10-11T15:03:09.862 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4= 2023-10-11T15:03:09.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4=', 'Content-Length': '0'} 2023-10-11T15:03:09.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2961db6d09cb4ce-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.865 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.866 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.866 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.867 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.867 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.868 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.868 DEBUG:boto:Data: 2023-10-11T15:03:09.868 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.869 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.870 DEBUG:boto:Params: {} 2023-10-11T15:03:09.870 DEBUG:boto:Token: None 2023-10-11T15:03:09.871 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FGJlJZAJGZI7z3cXWizfhMC7n5c= 2023-10-11T15:03:09.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FGJlJZAJGZI7z3cXWizfhMC7n5c=', 'Content-Length': '0'} 2023-10-11T15:03:09.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d5f7cf63a8d571-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.874 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.874 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.875 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.875 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.876 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.876 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.876 DEBUG:boto:Data: 2023-10-11T15:03:09.877 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.878 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.878 DEBUG:boto:Params: {} 2023-10-11T15:03:09.879 DEBUG:boto:Token: None 2023-10-11T15:03:09.879 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FGJlJZAJGZI7z3cXWizfhMC7n5c= 2023-10-11T15:03:09.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FGJlJZAJGZI7z3cXWizfhMC7n5c=', 'Content-Length': '0'} 2023-10-11T15:03:09.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0a50c1e04f51a0-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:09.883 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.884 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.884 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.884 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.885 DEBUG:boto:Method: GET 2023-10-11T15:03:09.885 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.886 DEBUG:boto:Data: 2023-10-11T15:03:09.886 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.887 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.887 DEBUG:boto:Params: {} 2023-10-11T15:03:09.888 DEBUG:boto:Token: None 2023-10-11T15:03:09.888 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4= 2023-10-11T15:03:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4=', 'Content-Length': '0'} 2023-10-11T15:03:09.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1b2d8b1e2ba568-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.892 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.892 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.893 DEBUG:boto:path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.893 DEBUG:boto:auth_path=/dghqvd-53/d?versionId=null 2023-10-11T15:03:09.894 DEBUG:boto:Method: GET 2023-10-11T15:03:09.894 DEBUG:boto:Path: /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.895 DEBUG:boto:Data: 2023-10-11T15:03:09.895 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.896 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.896 DEBUG:boto:Params: {} 2023-10-11T15:03:09.897 DEBUG:boto:Token: None 2023-10-11T15:03:09.897 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d?versionId=null 2023-10-11T15:03:09.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4= 2023-10-11T15:03:09.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FJ8oHRvcPQWuUuGdHKYhxOfpPM4=', 'Content-Length': '0'} 2023-10-11T15:03:09.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006505507cebbaa959-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.901 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.901 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.902 DEBUG:boto:path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.902 DEBUG:boto:auth_path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.903 DEBUG:boto:Method: GET 2023-10-11T15:03:09.903 DEBUG:boto:Path: /dghqvd-53/?prefix=d 2023-10-11T15:03:09.903 DEBUG:boto:Data: 2023-10-11T15:03:09.904 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.905 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.905 DEBUG:boto:Params: {} 2023-10-11T15:03:09.906 DEBUG:boto:Token: None 2023-10-11T15:03:09.906 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076d1862f903cda59-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.950 DEBUG:boto:b'dghqvd-53d1000falsed2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.951 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.951 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.952 DEBUG:boto:path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.952 DEBUG:boto:auth_path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.953 DEBUG:boto:Method: GET 2023-10-11T15:03:09.953 DEBUG:boto:Path: /dghqvd-53/?prefix=d 2023-10-11T15:03:09.954 DEBUG:boto:Data: 2023-10-11T15:03:09.954 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.955 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.955 DEBUG:boto:Params: {} 2023-10-11T15:03:09.956 DEBUG:boto:Token: None 2023-10-11T15:03:09.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:09.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad355cc45ff8450-006526b92d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.960 DEBUG:boto:b'dghqvd-53d1000falsed2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:09.961 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.961 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.962 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.962 DEBUG:boto:Path: /dghqvd-53/d 2023-10-11T15:03:09.963 DEBUG:boto:Data: 2023-10-11T15:03:09.963 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.964 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.964 DEBUG:boto:Params: {} 2023-10-11T15:03:09.965 DEBUG:boto:Token: None 2023-10-11T15:03:09.965 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d 2023-10-11T15:03:09.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lj3ljpTbCVCegqWILwoGiKwOpWw= 2023-10-11T15:03:09.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lj3ljpTbCVCegqWILwoGiKwOpWw=', 'Content-Length': '0'} 2023-10-11T15:03:09.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17f6ac162795839-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.968 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.969 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.969 DEBUG:boto:Method: HEAD 2023-10-11T15:03:09.970 DEBUG:boto:Path: /dghqvd-53/d 2023-10-11T15:03:09.970 DEBUG:boto:Data: 2023-10-11T15:03:09.971 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.971 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.972 DEBUG:boto:Params: {} 2023-10-11T15:03:09.972 DEBUG:boto:Token: None 2023-10-11T15:03:09.973 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d 2023-10-11T15:03:09.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lj3ljpTbCVCegqWILwoGiKwOpWw= 2023-10-11T15:03:09.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lj3ljpTbCVCegqWILwoGiKwOpWw=', 'Content-Length': '0'} 2023-10-11T15:03:09.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622a2bb3f48f2951-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:09.977 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.977 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.978 DEBUG:boto:Method: GET 2023-10-11T15:03:09.978 DEBUG:boto:Path: /dghqvd-53/d 2023-10-11T15:03:09.979 DEBUG:boto:Data: 2023-10-11T15:03:09.979 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.980 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.980 DEBUG:boto:Params: {} 2023-10-11T15:03:09.981 DEBUG:boto:Token: None 2023-10-11T15:03:09.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d 2023-10-11T15:03:09.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vFrqFW/pnPTdO4a59Gb45bFHc5U= 2023-10-11T15:03:09.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vFrqFW/pnPTdO4a59Gb45bFHc5U=', 'Content-Length': '0'} 2023-10-11T15:03:09.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a202c6f64364e4-006526b92d-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.985 DEBUG:boto:path=/dghqvd-53/d 2023-10-11T15:03:09.985 DEBUG:boto:auth_path=/dghqvd-53/d 2023-10-11T15:03:09.986 DEBUG:boto:Method: GET 2023-10-11T15:03:09.986 DEBUG:boto:Path: /dghqvd-53/d 2023-10-11T15:03:09.986 DEBUG:boto:Data: 2023-10-11T15:03:09.987 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:09.988 DEBUG:boto:Port: 8000 2023-10-11T15:03:09.988 DEBUG:boto:Params: {} 2023-10-11T15:03:09.989 DEBUG:boto:Token: None 2023-10-11T15:03:09.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/d 2023-10-11T15:03:09.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vFrqFW/pnPTdO4a59Gb45bFHc5U= 2023-10-11T15:03:09.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vFrqFW/pnPTdO4a59Gb45bFHc5U=', 'Content-Length': '0'} 2023-10-11T15:03:09.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:02:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d761803b275b29-006526b92d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:09.993 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:09.993 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:09.994 DEBUG:boto:path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.994 DEBUG:boto:auth_path=/dghqvd-53/?prefix=d 2023-10-11T15:03:09.995 DEBUG:boto:Method: GET 2023-10-11T15:03:09.995 DEBUG:boto:Path: /dghqvd-53/?prefix=d 2023-10-11T15:03:09.996 DEBUG:boto:Data: 2023-10-11T15:03:09.996 DEBUG:boto:Headers: {} 2023-10-11T15:03:09.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:09.997 DEBUG:boto:Port: 8001 2023-10-11T15:03:09.997 DEBUG:boto:Params: {} 2023-10-11T15:03:09.998 DEBUG:boto:Token: None 2023-10-11T15:03:09.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:09 GMT /dghqvd-53/ 2023-10-11T15:03:09.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w= 2023-10-11T15:03:09.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAjwul3OvPHv0MQf1HSa/wUCX7w=', 'Content-Length': '0'} 2023-10-11T15:03:10.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007838af6cc2294011-006526b92d-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.002 DEBUG:boto:b'dghqvd-53d1000falsed2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:10.003 DEBUG:boto:path=/dghqvd-53/ 2023-10-11T15:03:10.003 DEBUG:boto:auth_path=/dghqvd-53/ 2023-10-11T15:03:10.004 DEBUG:boto:path=/dghqvd-53/?prefix=d 2023-10-11T15:03:10.004 DEBUG:boto:auth_path=/dghqvd-53/?prefix=d 2023-10-11T15:03:10.004 DEBUG:boto:Method: GET 2023-10-11T15:03:10.005 DEBUG:boto:Path: /dghqvd-53/?prefix=d 2023-10-11T15:03:10.005 DEBUG:boto:Data: 2023-10-11T15:03:10.006 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.007 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.007 DEBUG:boto:Params: {} 2023-10-11T15:03:10.008 DEBUG:boto:Token: None 2023-10-11T15:03:10.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-53/ 2023-10-11T15:03:10.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GxAK1WhRcd8mo5dOEzJQB/cOnGE= 2023-10-11T15:03:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GxAK1WhRcd8mo5dOEzJQB/cOnGE=', 'Content-Length': '0'} 2023-10-11T15:03:10.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccae434ed6d44446-006526b92e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.012 DEBUG:boto:b'dghqvd-53d1000falsed2023-10-11T15:02:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:10.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-53 zones={a2, a1} 2023-10-11T15:03:10.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:10.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:10.017 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:03:10.071 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:10.072 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:10.121 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-54 2023-10-11T15:03:10.122 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:10.122 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:10.122 DEBUG:boto:Method: PUT 2023-10-11T15:03:10.123 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:10.123 DEBUG:boto:Data: 2023-10-11T15:03:10.124 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.124 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.125 DEBUG:boto:Params: {} 2023-10-11T15:03:10.125 DEBUG:boto:Token: None 2023-10-11T15:03:10.126 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/ 2023-10-11T15:03:10.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nnFYooXMGiBcf+tdB47POrLJByY= 2023-10-11T15:03:10.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nnFYooXMGiBcf+tdB47POrLJByY=', 'Content-Length': '0'} 2023-10-11T15:03:10.143 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000177a8355fa9f5297-006526b92e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.144 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:10.144 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:10.144 DEBUG:boto:Method: HEAD 2023-10-11T15:03:10.145 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:10.145 DEBUG:boto:Data: 2023-10-11T15:03:10.146 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.146 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.147 DEBUG:boto:Params: {} 2023-10-11T15:03:10.147 DEBUG:boto:Token: None 2023-10-11T15:03:10.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/ 2023-10-11T15:03:10.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg= 2023-10-11T15:03:10.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg=', 'Content-Length': '0'} 2023-10-11T15:03:10.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060b04603a6c9b2ee-006526b92e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.151 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:10.152 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:10.152 DEBUG:boto:Method: PUT 2023-10-11T15:03:10.152 DEBUG:boto:Path: /dghqvd-54/a 2023-10-11T15:03:10.153 DEBUG:boto:Data: 2023-10-11T15:03:10.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:10.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.154 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.154 DEBUG:boto:Params: {} 2023-10-11T15:03:10.155 DEBUG:boto:Token: None 2023-10-11T15:03:10.155 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/a 2023-10-11T15:03:10.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z1cMxPmVt+F02XwH20bL7pGtonA= 2023-10-11T15:03:10.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z1cMxPmVt+F02XwH20bL7pGtonA='} 2023-10-11T15:03:10.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000181942f6e4921ac4-006526b92e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.199 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:10.199 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:10.200 DEBUG:boto:Method: HEAD 2023-10-11T15:03:10.200 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:10.200 DEBUG:boto:Data: 2023-10-11T15:03:10.201 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.202 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.202 DEBUG:boto:Params: {} 2023-10-11T15:03:10.203 DEBUG:boto:Token: None 2023-10-11T15:03:10.203 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/ 2023-10-11T15:03:10.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg= 2023-10-11T15:03:10.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg=', 'Content-Length': '0'} 2023-10-11T15:03:10.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8e6947670a2589a-006526b92e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.206 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:10.207 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:10.207 DEBUG:boto:Method: PUT 2023-10-11T15:03:10.208 DEBUG:boto:Path: /dghqvd-54/b 2023-10-11T15:03:10.208 DEBUG:boto:Data: 2023-10-11T15:03:10.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:10.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.209 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.210 DEBUG:boto:Params: {} 2023-10-11T15:03:10.210 DEBUG:boto:Token: None 2023-10-11T15:03:10.211 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/b 2023-10-11T15:03:10.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YHPWaYbqn7XwlNroQsGP1OTW2Gw= 2023-10-11T15:03:10.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YHPWaYbqn7XwlNroQsGP1OTW2Gw='} 2023-10-11T15:03:10.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fd59c4362b70e7f-006526b92e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.217 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:10.217 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:10.218 DEBUG:boto:Method: HEAD 2023-10-11T15:03:10.218 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:10.219 DEBUG:boto:Data: 2023-10-11T15:03:10.219 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.220 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.220 DEBUG:boto:Params: {} 2023-10-11T15:03:10.221 DEBUG:boto:Token: None 2023-10-11T15:03:10.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/ 2023-10-11T15:03:10.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg= 2023-10-11T15:03:10.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg=', 'Content-Length': '0'} 2023-10-11T15:03:10.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cbf769cc90aff85-006526b92e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.226 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:10.226 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:10.227 DEBUG:boto:Method: PUT 2023-10-11T15:03:10.227 DEBUG:boto:Path: /dghqvd-54/c 2023-10-11T15:03:10.227 DEBUG:boto:Data: 2023-10-11T15:03:10.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:10.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.229 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.229 DEBUG:boto:Params: {} 2023-10-11T15:03:10.230 DEBUG:boto:Token: None 2023-10-11T15:03:10.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/c 2023-10-11T15:03:10.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AgHaxJs8sIJsMT5iLtFGzpjJUH0= 2023-10-11T15:03:10.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AgHaxJs8sIJsMT5iLtFGzpjJUH0='} 2023-10-11T15:03:10.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000644fbad8cbaa7b37-006526b92e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.236 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:10.237 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:10.237 DEBUG:boto:Method: HEAD 2023-10-11T15:03:10.237 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:10.238 DEBUG:boto:Data: 2023-10-11T15:03:10.238 DEBUG:boto:Headers: {} 2023-10-11T15:03:10.239 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.239 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.239 DEBUG:boto:Params: {} 2023-10-11T15:03:10.240 DEBUG:boto:Token: None 2023-10-11T15:03:10.240 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/ 2023-10-11T15:03:10.241 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg= 2023-10-11T15:03:10.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mbJEPVOibvw5EQYiQmF3XyFI6Tg=', 'Content-Length': '0'} 2023-10-11T15:03:10.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce14a615a2f86696-006526b92e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.244 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:10.244 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:10.245 DEBUG:boto:Method: PUT 2023-10-11T15:03:10.245 DEBUG:boto:Path: /dghqvd-54/d 2023-10-11T15:03:10.246 DEBUG:boto:Data: 2023-10-11T15:03:10.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:10.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:10.247 DEBUG:boto:Port: 8000 2023-10-11T15:03:10.247 DEBUG:boto:Params: {} 2023-10-11T15:03:10.248 DEBUG:boto:Token: None 2023-10-11T15:03:10.248 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:10 GMT /dghqvd-54/d 2023-10-11T15:03:10.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cg1Tw5JBcMZLojxrULmIYZ5PjRU= 2023-10-11T15:03:10.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cg1Tw5JBcMZLojxrULmIYZ5PjRU='} 2023-10-11T15:03:10.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000610d88337a9d2388-006526b92e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:10.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:10.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:10.300 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:10.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:10.378 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:03:10.390 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:03:10.390 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.390 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:03:10.391 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:03:10.391 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.391 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.391 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:03:10.391 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:03:10.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.392 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.392 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:03:10.392 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:03:10.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.393 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.393 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036590.140566_389.1", 2023-10-11T15:03:10.393 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:03:10.140566Z" 2023-10-11T15:03:10.393 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.393 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:03:10.394 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036590.140566_389.1'} 2023-10-11T15:03:10.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:03:10.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:10.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:10.396 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:10.446 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:10.446 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:10.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:10.472+0000 7fde2c5ed780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:03:10.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:10.473+0000 7fde2c5ed780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:03:10.487 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:03:10.488 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:03:10.488 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:03:10.488 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:03:10.488 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:03:10.488 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:03:10.489 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:10.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.489 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:03:10.489 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.489 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:03:10.490 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:10.490 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:10.490 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:03:10.490 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:10.490 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:03:10.491 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:10.491 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:03:10.491 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:10.491 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.491 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.492 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:03:10.492 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:10.492 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:10.492 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:03:10.493 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:10.493 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:10.493 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:10.493 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:03:10.493 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:10.494 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.494 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.494 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:03:10.494 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:10.495 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:10.495 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:03:10.495 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:10.495 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:10.495 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:10.496 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:03:10.496 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:10.496 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036590.140566_389.1", 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:10.497 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:10.498 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:10.498 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:10.140566Z", 2023-10-11T15:03:10.498 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:10.498 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:03:10.499 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:03:10.500 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:10.500 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:03:10Z" 2023-10-11T15:03:10.500 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:03:10.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036590.140566_389.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:03:10.140566Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:03:10Z" } 2023-10-11T15:03:10.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036590.140566_389.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:03:10.140566Z', 'realm_epoch': 2}}] 2023-10-11T15:03:10.502 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036590.140566_389.1'} 2023-10-11T15:03:10.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036590.140566_389.1'} 2023-10-11T15:03:10.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:03:10.503 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-54 2023-10-11T15:03:10.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:10.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:10.504 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:03:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:10.560 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:10.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:10.591+0000 7f7efe73f780 1 waiting to reach incremental sync.. 2023-10-11T15:03:15.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:15.592+0000 7f7efe73f780 1 waiting to reach incremental sync.. 2023-10-11T15:03:20.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:20.593+0000 7f7efe73f780 1 waiting to reach incremental sync.. 2023-10-11T15:03:25.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:25.593+0000 7f7efe73f780 1 waiting to reach incremental sync.. 2023-10-11T15:03:30.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:30.595+0000 7f7efe73f780 1 bucket sync caught up with source: 2023-10-11T15:03:30.598 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.21.6, , , , 00000000001.41.6, , , 00000000001.17.6, , 00000000001.15.6] 2023-10-11T15:03:30.598 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.21.6, , , , 00000000001.41.6, , , 00000000001.17.6, , 00000000001.15.6] 2023-10-11T15:03:30.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:30.595+0000 7f7efe73f780 0 bucket checkpoint complete 2023-10-11T15:03:30.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-54 2023-10-11T15:03:30.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:30.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:30.616 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:03:30.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:30.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:30.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:30.713+0000 7fe126b24780 1 waiting to reach incremental sync.. 2023-10-11T15:03:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:35.714+0000 7fe126b24780 1 waiting to reach incremental sync.. 2023-10-11T15:03:40.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:40.714+0000 7fe126b24780 1 waiting to reach incremental sync.. 2023-10-11T15:03:45.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:45.715+0000 7fe126b24780 1 waiting to reach incremental sync.. 2023-10-11T15:03:50.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:50.717+0000 7fe126b24780 1 bucket sync caught up with source: 2023-10-11T15:03:50.719 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.221.6, , , , 00000000001.180.6, , , 00000000001.57.6, , 00000000001.59.6] 2023-10-11T15:03:50.719 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.221.6, , , , 00000000001.180.6, , , 00000000001.57.6, , 00000000001.59.6] 2023-10-11T15:03:50.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:50.717+0000 7fe126b24780 0 bucket checkpoint complete 2023-10-11T15:03:50.735 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-54 zones={a2, a1} 2023-10-11T15:03:50.736 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.736 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.737 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.737 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:50.738 DEBUG:boto:Data: 2023-10-11T15:03:50.738 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.739 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.739 DEBUG:boto:Params: {} 2023-10-11T15:03:50.740 DEBUG:boto:Token: None 2023-10-11T15:03:50.741 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/ 2023-10-11T15:03:50.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KbDs9fFfQSG0b2u9z823+UyoyV4= 2023-10-11T15:03:50.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KbDs9fFfQSG0b2u9z823+UyoyV4=', 'Content-Length': '0'} 2023-10-11T15:03:50.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8a2d188d04cc93e-006526b956-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.746 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.746 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.747 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.747 DEBUG:boto:Path: /dghqvd-54/ 2023-10-11T15:03:50.748 DEBUG:boto:Data: 2023-10-11T15:03:50.748 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.749 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.749 DEBUG:boto:Params: {} 2023-10-11T15:03:50.750 DEBUG:boto:Token: None 2023-10-11T15:03:50.750 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/ 2023-10-11T15:03:50.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KbDs9fFfQSG0b2u9z823+UyoyV4= 2023-10-11T15:03:50.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KbDs9fFfQSG0b2u9z823+UyoyV4=', 'Content-Length': '0'} 2023-10-11T15:03:50.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000564cba1235686b8-006526b956-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.756 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:03:50.757 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.757 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.758 DEBUG:boto:path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.759 DEBUG:boto:auth_path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.760 DEBUG:boto:Method: GET 2023-10-11T15:03:50.760 DEBUG:boto:Path: /dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.761 DEBUG:boto:Data: 2023-10-11T15:03:50.762 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.764 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.764 DEBUG:boto:Params: {} 2023-10-11T15:03:50.765 DEBUG:boto:Token: None 2023-10-11T15:03:50.766 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/?versions 2023-10-11T15:03:50.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M= 2023-10-11T15:03:50.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M=', 'Content-Length': '0'} 2023-10-11T15:03:50.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f93e4ba6ecc4182-006526b956-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.770 DEBUG:boto:b'dghqvd-54999falseanulltrue2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.771 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:03:50.771 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:03:50.772 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:03:50.772 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:03:50.773 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:03:50.773 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.773 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.774 DEBUG:boto:path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.774 DEBUG:boto:auth_path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.775 DEBUG:boto:Method: GET 2023-10-11T15:03:50.775 DEBUG:boto:Path: /dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.776 DEBUG:boto:Data: 2023-10-11T15:03:50.776 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.777 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.777 DEBUG:boto:Params: {} 2023-10-11T15:03:50.778 DEBUG:boto:Token: None 2023-10-11T15:03:50.778 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/?versions 2023-10-11T15:03:50.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M= 2023-10-11T15:03:50.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M=', 'Content-Length': '0'} 2023-10-11T15:03:50.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cdaceb00f39a5d-006526b956-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.782 DEBUG:boto:b'dghqvd-54999falseanulltrue2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.783 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:03:50.783 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:03:50.784 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:03:50.784 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:03:50.785 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.785 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.786 DEBUG:boto:path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.786 DEBUG:boto:auth_path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.786 DEBUG:boto:Method: GET 2023-10-11T15:03:50.787 DEBUG:boto:Path: /dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.787 DEBUG:boto:Data: 2023-10-11T15:03:50.788 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.788 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.789 DEBUG:boto:Params: {} 2023-10-11T15:03:50.789 DEBUG:boto:Token: None 2023-10-11T15:03:50.790 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/?versions 2023-10-11T15:03:50.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M= 2023-10-11T15:03:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M=', 'Content-Length': '0'} 2023-10-11T15:03:50.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9afe27ccdc08cda-006526b956-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.794 DEBUG:boto:b'dghqvd-54999falseanulltrue2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.795 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.795 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.795 DEBUG:boto:path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.796 DEBUG:boto:auth_path=/dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.796 DEBUG:boto:Method: GET 2023-10-11T15:03:50.797 DEBUG:boto:Path: /dghqvd-54/?versions&max-keys=999 2023-10-11T15:03:50.797 DEBUG:boto:Data: 2023-10-11T15:03:50.797 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.798 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.799 DEBUG:boto:Params: {} 2023-10-11T15:03:50.799 DEBUG:boto:Token: None 2023-10-11T15:03:50.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/?versions 2023-10-11T15:03:50.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M= 2023-10-11T15:03:50.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HAhEUiy+ul9n6o3d6JE7LLcro8M=', 'Content-Length': '0'} 2023-10-11T15:03:50.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000190a7dd99b6f08b4-006526b956-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.804 DEBUG:boto:b'dghqvd-54999falseanulltrue2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:50.805 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.805 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.806 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.806 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.807 DEBUG:boto:Method: GET 2023-10-11T15:03:50.807 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.808 DEBUG:boto:Data: 2023-10-11T15:03:50.808 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.809 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.809 DEBUG:boto:Params: {} 2023-10-11T15:03:50.810 DEBUG:boto:Token: None 2023-10-11T15:03:50.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI= 2023-10-11T15:03:50.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI=', 'Content-Length': '0'} 2023-10-11T15:03:50.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9cba21928d357c3-006526b956-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.814 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.815 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.815 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.815 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.816 DEBUG:boto:Method: GET 2023-10-11T15:03:50.816 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.817 DEBUG:boto:Data: 2023-10-11T15:03:50.817 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.818 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.818 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.818 DEBUG:boto:Params: {} 2023-10-11T15:03:50.819 DEBUG:boto:Token: None 2023-10-11T15:03:50.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI= 2023-10-11T15:03:50.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI=', 'Content-Length': '0'} 2023-10-11T15:03:50.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006659f914a0792ea7-006526b956-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.823 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.823 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.824 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.824 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.825 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.825 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.826 DEBUG:boto:Data: 2023-10-11T15:03:50.826 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.827 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.827 DEBUG:boto:Params: {} 2023-10-11T15:03:50.828 DEBUG:boto:Token: None 2023-10-11T15:03:50.828 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7uvlQ2xrV6dVo8d3b5sgUCSFiGg= 2023-10-11T15:03:50.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7uvlQ2xrV6dVo8d3b5sgUCSFiGg=', 'Content-Length': '0'} 2023-10-11T15:03:50.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64e3f583d0c67dd-006526b956-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.832 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.832 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.832 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.833 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.833 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.834 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.834 DEBUG:boto:Data: 2023-10-11T15:03:50.835 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.835 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.836 DEBUG:boto:Params: {} 2023-10-11T15:03:50.836 DEBUG:boto:Token: None 2023-10-11T15:03:50.837 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7uvlQ2xrV6dVo8d3b5sgUCSFiGg= 2023-10-11T15:03:50.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7uvlQ2xrV6dVo8d3b5sgUCSFiGg=', 'Content-Length': '0'} 2023-10-11T15:03:50.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff86cc5bd78d39e-006526b956-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:50.840 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.841 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.841 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.842 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.842 DEBUG:boto:Method: GET 2023-10-11T15:03:50.843 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.843 DEBUG:boto:Data: 2023-10-11T15:03:50.843 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.844 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.845 DEBUG:boto:Params: {} 2023-10-11T15:03:50.845 DEBUG:boto:Token: None 2023-10-11T15:03:50.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.846 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI= 2023-10-11T15:03:50.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI=', 'Content-Length': '0'} 2023-10-11T15:03:50.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1c4fd0670829df-006526b956-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.849 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.850 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.850 DEBUG:boto:path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.850 DEBUG:boto:auth_path=/dghqvd-54/a?versionId=null 2023-10-11T15:03:50.851 DEBUG:boto:Method: GET 2023-10-11T15:03:50.851 DEBUG:boto:Path: /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.852 DEBUG:boto:Data: 2023-10-11T15:03:50.852 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.853 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.853 DEBUG:boto:Params: {} 2023-10-11T15:03:50.854 DEBUG:boto:Token: None 2023-10-11T15:03:50.854 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a?versionId=null 2023-10-11T15:03:50.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI= 2023-10-11T15:03:50.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1rm54dMu7YQDrvW6barh1AwjoTI=', 'Content-Length': '0'} 2023-10-11T15:03:50.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c675d57caa51c78b-006526b956-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.898 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.899 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.899 DEBUG:boto:path=/dghqvd-54/?prefix=a 2023-10-11T15:03:50.900 DEBUG:boto:auth_path=/dghqvd-54/?prefix=a 2023-10-11T15:03:50.900 DEBUG:boto:Method: GET 2023-10-11T15:03:50.901 DEBUG:boto:Path: /dghqvd-54/?prefix=a 2023-10-11T15:03:50.901 DEBUG:boto:Data: 2023-10-11T15:03:50.902 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.902 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.903 DEBUG:boto:Params: {} 2023-10-11T15:03:50.903 DEBUG:boto:Token: None 2023-10-11T15:03:50.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/ 2023-10-11T15:03:50.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nRHkdcbGkXgZtxWpQlhc0bR3wjA= 2023-10-11T15:03:50.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nRHkdcbGkXgZtxWpQlhc0bR3wjA=', 'Content-Length': '0'} 2023-10-11T15:03:50.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000693650dfb5177344-006526b956-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.908 DEBUG:boto:b'dghqvd-54a1000falsea2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.909 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:50.909 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:50.910 DEBUG:boto:path=/dghqvd-54/?prefix=a 2023-10-11T15:03:50.910 DEBUG:boto:auth_path=/dghqvd-54/?prefix=a 2023-10-11T15:03:50.911 DEBUG:boto:Method: GET 2023-10-11T15:03:50.911 DEBUG:boto:Path: /dghqvd-54/?prefix=a 2023-10-11T15:03:50.912 DEBUG:boto:Data: 2023-10-11T15:03:50.912 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.913 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.913 DEBUG:boto:Params: {} 2023-10-11T15:03:50.914 DEBUG:boto:Token: None 2023-10-11T15:03:50.914 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/ 2023-10-11T15:03:50.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nRHkdcbGkXgZtxWpQlhc0bR3wjA= 2023-10-11T15:03:50.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nRHkdcbGkXgZtxWpQlhc0bR3wjA=', 'Content-Length': '0'} 2023-10-11T15:03:50.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4f9e53cd0ada19-006526b956-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.919 DEBUG:boto:b'dghqvd-54a1000falsea2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:50.919 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.920 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.920 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.921 DEBUG:boto:Path: /dghqvd-54/a 2023-10-11T15:03:50.921 DEBUG:boto:Data: 2023-10-11T15:03:50.921 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.922 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.922 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.923 DEBUG:boto:Params: {} 2023-10-11T15:03:50.923 DEBUG:boto:Token: None 2023-10-11T15:03:50.924 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a 2023-10-11T15:03:50.924 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1MuZyMsWbArAduPKxj/4PAkwqA= 2023-10-11T15:03:50.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1MuZyMsWbArAduPKxj/4PAkwqA=', 'Content-Length': '0'} 2023-10-11T15:03:50.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000348edf074d6e07e0-006526b956-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.927 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.927 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.928 DEBUG:boto:Method: HEAD 2023-10-11T15:03:50.928 DEBUG:boto:Path: /dghqvd-54/a 2023-10-11T15:03:50.929 DEBUG:boto:Data: 2023-10-11T15:03:50.929 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.930 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.930 DEBUG:boto:Params: {} 2023-10-11T15:03:50.931 DEBUG:boto:Token: None 2023-10-11T15:03:50.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a 2023-10-11T15:03:50.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1MuZyMsWbArAduPKxj/4PAkwqA= 2023-10-11T15:03:50.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1MuZyMsWbArAduPKxj/4PAkwqA=', 'Content-Length': '0'} 2023-10-11T15:03:50.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004564311554153cd7-006526b956-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:03:50.937 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.938 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.938 DEBUG:boto:Method: GET 2023-10-11T15:03:50.939 DEBUG:boto:Path: /dghqvd-54/a 2023-10-11T15:03:50.939 DEBUG:boto:Data: 2023-10-11T15:03:50.940 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:50.940 DEBUG:boto:Port: 8001 2023-10-11T15:03:50.941 DEBUG:boto:Params: {} 2023-10-11T15:03:50.941 DEBUG:boto:Token: None 2023-10-11T15:03:50.942 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a 2023-10-11T15:03:50.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N/+LfLZ6QWuG16fCiei6/LB/qQ4= 2023-10-11T15:03:50.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N/+LfLZ6QWuG16fCiei6/LB/qQ4=', 'Content-Length': '0'} 2023-10-11T15:03:50.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007547ecdf350297a6-006526b956-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.986 DEBUG:boto:path=/dghqvd-54/a 2023-10-11T15:03:50.987 DEBUG:boto:auth_path=/dghqvd-54/a 2023-10-11T15:03:50.988 DEBUG:boto:Method: GET 2023-10-11T15:03:50.989 DEBUG:boto:Path: /dghqvd-54/a 2023-10-11T15:03:50.989 DEBUG:boto:Data: 2023-10-11T15:03:50.990 DEBUG:boto:Headers: {} 2023-10-11T15:03:50.991 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:50.992 DEBUG:boto:Port: 8000 2023-10-11T15:03:50.992 DEBUG:boto:Params: {} 2023-10-11T15:03:50.993 DEBUG:boto:Token: None 2023-10-11T15:03:50.994 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:50 GMT /dghqvd-54/a 2023-10-11T15:03:50.995 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N/+LfLZ6QWuG16fCiei6/LB/qQ4= 2023-10-11T15:03:50.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N/+LfLZ6QWuG16fCiei6/LB/qQ4=', 'Content-Length': '0'} 2023-10-11T15:03:50.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d5ec9857e87fe0d-006526b956-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:50.999 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.000 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.000 DEBUG:boto:path=/dghqvd-54/?prefix=a 2023-10-11T15:03:51.001 DEBUG:boto:auth_path=/dghqvd-54/?prefix=a 2023-10-11T15:03:51.001 DEBUG:boto:Method: GET 2023-10-11T15:03:51.001 DEBUG:boto:Path: /dghqvd-54/?prefix=a 2023-10-11T15:03:51.002 DEBUG:boto:Data: 2023-10-11T15:03:51.002 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.003 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.004 DEBUG:boto:Params: {} 2023-10-11T15:03:51.004 DEBUG:boto:Token: None 2023-10-11T15:03:51.005 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005817b4ea186f6a98-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.009 DEBUG:boto:b'dghqvd-54a1000falsea2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.010 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.010 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.011 DEBUG:boto:path=/dghqvd-54/?prefix=a 2023-10-11T15:03:51.011 DEBUG:boto:auth_path=/dghqvd-54/?prefix=a 2023-10-11T15:03:51.011 DEBUG:boto:Method: GET 2023-10-11T15:03:51.012 DEBUG:boto:Path: /dghqvd-54/?prefix=a 2023-10-11T15:03:51.012 DEBUG:boto:Data: 2023-10-11T15:03:51.013 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.014 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.014 DEBUG:boto:Params: {} 2023-10-11T15:03:51.014 DEBUG:boto:Token: None 2023-10-11T15:03:51.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044bb2e0a26abb9fc-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.062 DEBUG:boto:b'dghqvd-54a1000falsea2023-10-11T15:03:10.157Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:51.064 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.064 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.064 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.065 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.065 DEBUG:boto:Method: GET 2023-10-11T15:03:51.066 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.066 DEBUG:boto:Data: 2023-10-11T15:03:51.067 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.068 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.068 DEBUG:boto:Params: {} 2023-10-11T15:03:51.068 DEBUG:boto:Token: None 2023-10-11T15:03:51.069 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw= 2023-10-11T15:03:51.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw=', 'Content-Length': '0'} 2023-10-11T15:03:51.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11219edb2836e17-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.073 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.073 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.074 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.074 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.075 DEBUG:boto:Method: GET 2023-10-11T15:03:51.075 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.076 DEBUG:boto:Data: 2023-10-11T15:03:51.076 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.077 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.077 DEBUG:boto:Params: {} 2023-10-11T15:03:51.078 DEBUG:boto:Token: None 2023-10-11T15:03:51.078 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw= 2023-10-11T15:03:51.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw=', 'Content-Length': '0'} 2023-10-11T15:03:51.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae2af438536f3b5-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.082 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.082 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.083 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.083 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.084 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.084 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.084 DEBUG:boto:Data: 2023-10-11T15:03:51.085 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.086 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.086 DEBUG:boto:Params: {} 2023-10-11T15:03:51.087 DEBUG:boto:Token: None 2023-10-11T15:03:51.087 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.088 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m5pNUbgzvg4xntHCFsY2X62dmms= 2023-10-11T15:03:51.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m5pNUbgzvg4xntHCFsY2X62dmms=', 'Content-Length': '0'} 2023-10-11T15:03:51.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003c5af8f3ffb738-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.091 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.091 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.092 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.092 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.092 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.093 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.093 DEBUG:boto:Data: 2023-10-11T15:03:51.094 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.095 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.095 DEBUG:boto:Params: {} 2023-10-11T15:03:51.095 DEBUG:boto:Token: None 2023-10-11T15:03:51.096 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m5pNUbgzvg4xntHCFsY2X62dmms= 2023-10-11T15:03:51.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m5pNUbgzvg4xntHCFsY2X62dmms=', 'Content-Length': '0'} 2023-10-11T15:03:51.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009016d46ed550946d-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:51.100 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.100 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.101 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.101 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.102 DEBUG:boto:Method: GET 2023-10-11T15:03:51.102 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.103 DEBUG:boto:Data: 2023-10-11T15:03:51.103 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.104 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.104 DEBUG:boto:Params: {} 2023-10-11T15:03:51.105 DEBUG:boto:Token: None 2023-10-11T15:03:51.106 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw= 2023-10-11T15:03:51.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw=', 'Content-Length': '0'} 2023-10-11T15:03:51.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01dcaee67781a67-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.109 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.110 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.110 DEBUG:boto:path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.111 DEBUG:boto:auth_path=/dghqvd-54/b?versionId=null 2023-10-11T15:03:51.111 DEBUG:boto:Method: GET 2023-10-11T15:03:51.111 DEBUG:boto:Path: /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.112 DEBUG:boto:Data: 2023-10-11T15:03:51.112 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.113 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.113 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.114 DEBUG:boto:Params: {} 2023-10-11T15:03:51.114 DEBUG:boto:Token: None 2023-10-11T15:03:51.115 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b?versionId=null 2023-10-11T15:03:51.115 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw= 2023-10-11T15:03:51.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m3QpbeNK8umC2ZEL+IL5VSO9DMw=', 'Content-Length': '0'} 2023-10-11T15:03:51.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644097a988494292-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.118 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.119 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.119 DEBUG:boto:path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.120 DEBUG:boto:auth_path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.120 DEBUG:boto:Method: GET 2023-10-11T15:03:51.121 DEBUG:boto:Path: /dghqvd-54/?prefix=b 2023-10-11T15:03:51.121 DEBUG:boto:Data: 2023-10-11T15:03:51.121 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.122 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.123 DEBUG:boto:Params: {} 2023-10-11T15:03:51.123 DEBUG:boto:Token: None 2023-10-11T15:03:51.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc7cf05d6b20220e-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.128 DEBUG:boto:b'dghqvd-54b1000falseb2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.129 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.129 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.130 DEBUG:boto:path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.130 DEBUG:boto:auth_path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.131 DEBUG:boto:Method: GET 2023-10-11T15:03:51.131 DEBUG:boto:Path: /dghqvd-54/?prefix=b 2023-10-11T15:03:51.131 DEBUG:boto:Data: 2023-10-11T15:03:51.132 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.133 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.133 DEBUG:boto:Params: {} 2023-10-11T15:03:51.134 DEBUG:boto:Token: None 2023-10-11T15:03:51.134 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004410dc3ccd93eb66-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.138 DEBUG:boto:b'dghqvd-54b1000falseb2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.139 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.139 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.140 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.140 DEBUG:boto:Path: /dghqvd-54/b 2023-10-11T15:03:51.141 DEBUG:boto:Data: 2023-10-11T15:03:51.141 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.142 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.143 DEBUG:boto:Params: {} 2023-10-11T15:03:51.145 DEBUG:boto:Token: None 2023-10-11T15:03:51.146 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b 2023-10-11T15:03:51.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EjsaT4oZhJn6YYiwE/D7NI5IUtQ= 2023-10-11T15:03:51.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EjsaT4oZhJn6YYiwE/D7NI5IUtQ=', 'Content-Length': '0'} 2023-10-11T15:03:51.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000472d8210dcba2c0-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.149 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.150 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.150 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.150 DEBUG:boto:Path: /dghqvd-54/b 2023-10-11T15:03:51.151 DEBUG:boto:Data: 2023-10-11T15:03:51.151 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.152 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.153 DEBUG:boto:Params: {} 2023-10-11T15:03:51.153 DEBUG:boto:Token: None 2023-10-11T15:03:51.153 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b 2023-10-11T15:03:51.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EjsaT4oZhJn6YYiwE/D7NI5IUtQ= 2023-10-11T15:03:51.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EjsaT4oZhJn6YYiwE/D7NI5IUtQ=', 'Content-Length': '0'} 2023-10-11T15:03:51.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1811a5fdfd38303-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:03:51.158 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.158 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.159 DEBUG:boto:Method: GET 2023-10-11T15:03:51.159 DEBUG:boto:Path: /dghqvd-54/b 2023-10-11T15:03:51.159 DEBUG:boto:Data: 2023-10-11T15:03:51.160 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.161 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.161 DEBUG:boto:Params: {} 2023-10-11T15:03:51.162 DEBUG:boto:Token: None 2023-10-11T15:03:51.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b 2023-10-11T15:03:51.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KF069iCPBquGb/c8hR5MuECQCyw= 2023-10-11T15:03:51.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KF069iCPBquGb/c8hR5MuECQCyw=', 'Content-Length': '0'} 2023-10-11T15:03:51.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6024e0daa9f3677-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.166 DEBUG:boto:path=/dghqvd-54/b 2023-10-11T15:03:51.166 DEBUG:boto:auth_path=/dghqvd-54/b 2023-10-11T15:03:51.167 DEBUG:boto:Method: GET 2023-10-11T15:03:51.167 DEBUG:boto:Path: /dghqvd-54/b 2023-10-11T15:03:51.168 DEBUG:boto:Data: 2023-10-11T15:03:51.168 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.169 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.169 DEBUG:boto:Params: {} 2023-10-11T15:03:51.170 DEBUG:boto:Token: None 2023-10-11T15:03:51.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/b 2023-10-11T15:03:51.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KF069iCPBquGb/c8hR5MuECQCyw= 2023-10-11T15:03:51.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KF069iCPBquGb/c8hR5MuECQCyw=', 'Content-Length': '0'} 2023-10-11T15:03:51.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006474aa9dbd09994f-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.174 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.174 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.175 DEBUG:boto:path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.175 DEBUG:boto:auth_path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.176 DEBUG:boto:Method: GET 2023-10-11T15:03:51.176 DEBUG:boto:Path: /dghqvd-54/?prefix=b 2023-10-11T15:03:51.177 DEBUG:boto:Data: 2023-10-11T15:03:51.177 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.178 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.179 DEBUG:boto:Params: {} 2023-10-11T15:03:51.179 DEBUG:boto:Token: None 2023-10-11T15:03:51.180 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8b7a7b15fe91878-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.183 DEBUG:boto:b'dghqvd-54b1000falseb2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.184 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.185 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.185 DEBUG:boto:path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.186 DEBUG:boto:auth_path=/dghqvd-54/?prefix=b 2023-10-11T15:03:51.186 DEBUG:boto:Method: GET 2023-10-11T15:03:51.186 DEBUG:boto:Path: /dghqvd-54/?prefix=b 2023-10-11T15:03:51.187 DEBUG:boto:Data: 2023-10-11T15:03:51.187 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.188 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.189 DEBUG:boto:Params: {} 2023-10-11T15:03:51.189 DEBUG:boto:Token: None 2023-10-11T15:03:51.190 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.190 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045cdbf8ef680da88-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.194 DEBUG:boto:b'dghqvd-54b1000falseb2023-10-11T15:03:10.212Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:51.195 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.195 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.196 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.196 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.197 DEBUG:boto:Method: GET 2023-10-11T15:03:51.197 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.198 DEBUG:boto:Data: 2023-10-11T15:03:51.198 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.199 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.199 DEBUG:boto:Params: {} 2023-10-11T15:03:51.200 DEBUG:boto:Token: None 2023-10-11T15:03:51.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s= 2023-10-11T15:03:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s=', 'Content-Length': '0'} 2023-10-11T15:03:51.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418dba0cf92d76f2-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.246 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.247 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.247 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.248 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.248 DEBUG:boto:Method: GET 2023-10-11T15:03:51.248 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.249 DEBUG:boto:Data: 2023-10-11T15:03:51.249 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.250 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.251 DEBUG:boto:Params: {} 2023-10-11T15:03:51.251 DEBUG:boto:Token: None 2023-10-11T15:03:51.252 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s= 2023-10-11T15:03:51.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s=', 'Content-Length': '0'} 2023-10-11T15:03:51.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abb1836c7793402a-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.255 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.256 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.256 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.257 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.257 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.257 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.258 DEBUG:boto:Data: 2023-10-11T15:03:51.258 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.259 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.259 DEBUG:boto:Params: {} 2023-10-11T15:03:51.260 DEBUG:boto:Token: None 2023-10-11T15:03:51.260 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N2zeaFHvoHBZaA36VYVguY2qVlI= 2023-10-11T15:03:51.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N2zeaFHvoHBZaA36VYVguY2qVlI=', 'Content-Length': '0'} 2023-10-11T15:03:51.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69edaa2d3d008c0-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.264 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.264 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.264 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.265 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.265 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.266 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.266 DEBUG:boto:Data: 2023-10-11T15:03:51.266 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.267 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.268 DEBUG:boto:Params: {} 2023-10-11T15:03:51.268 DEBUG:boto:Token: None 2023-10-11T15:03:51.269 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N2zeaFHvoHBZaA36VYVguY2qVlI= 2023-10-11T15:03:51.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N2zeaFHvoHBZaA36VYVguY2qVlI=', 'Content-Length': '0'} 2023-10-11T15:03:51.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15e195f2c3259b4-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:51.273 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.273 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.274 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.274 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.274 DEBUG:boto:Method: GET 2023-10-11T15:03:51.275 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.275 DEBUG:boto:Data: 2023-10-11T15:03:51.276 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.277 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.277 DEBUG:boto:Params: {} 2023-10-11T15:03:51.277 DEBUG:boto:Token: None 2023-10-11T15:03:51.278 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s= 2023-10-11T15:03:51.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s=', 'Content-Length': '0'} 2023-10-11T15:03:51.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f10d632a87a1332d-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.281 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.282 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.282 DEBUG:boto:path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.283 DEBUG:boto:auth_path=/dghqvd-54/c?versionId=null 2023-10-11T15:03:51.283 DEBUG:boto:Method: GET 2023-10-11T15:03:51.284 DEBUG:boto:Path: /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.284 DEBUG:boto:Data: 2023-10-11T15:03:51.285 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.285 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.286 DEBUG:boto:Params: {} 2023-10-11T15:03:51.286 DEBUG:boto:Token: None 2023-10-11T15:03:51.287 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c?versionId=null 2023-10-11T15:03:51.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s= 2023-10-11T15:03:51.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1OznMim50WJUorzuZF+5GABqm/s=', 'Content-Length': '0'} 2023-10-11T15:03:51.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034894cfef21777e5-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.330 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.331 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.331 DEBUG:boto:path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.332 DEBUG:boto:auth_path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.332 DEBUG:boto:Method: GET 2023-10-11T15:03:51.333 DEBUG:boto:Path: /dghqvd-54/?prefix=c 2023-10-11T15:03:51.333 DEBUG:boto:Data: 2023-10-11T15:03:51.334 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.334 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.335 DEBUG:boto:Params: {} 2023-10-11T15:03:51.335 DEBUG:boto:Token: None 2023-10-11T15:03:51.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea2d637ceb14752-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.340 DEBUG:boto:b'dghqvd-54c1000falsec2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.341 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.341 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.342 DEBUG:boto:path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.342 DEBUG:boto:auth_path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.343 DEBUG:boto:Method: GET 2023-10-11T15:03:51.343 DEBUG:boto:Path: /dghqvd-54/?prefix=c 2023-10-11T15:03:51.343 DEBUG:boto:Data: 2023-10-11T15:03:51.344 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.345 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.345 DEBUG:boto:Params: {} 2023-10-11T15:03:51.346 DEBUG:boto:Token: None 2023-10-11T15:03:51.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000531a1f9563b7fd8d-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.351 DEBUG:boto:b'dghqvd-54c1000falsec2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.351 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.352 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.352 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.353 DEBUG:boto:Path: /dghqvd-54/c 2023-10-11T15:03:51.353 DEBUG:boto:Data: 2023-10-11T15:03:51.354 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.354 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.355 DEBUG:boto:Params: {} 2023-10-11T15:03:51.355 DEBUG:boto:Token: None 2023-10-11T15:03:51.356 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c 2023-10-11T15:03:51.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V5GBH6hycKNto8w3ov+Oe19RpfI= 2023-10-11T15:03:51.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V5GBH6hycKNto8w3ov+Oe19RpfI=', 'Content-Length': '0'} 2023-10-11T15:03:51.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4897ccf2ab7f92-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.359 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.360 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.360 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.361 DEBUG:boto:Path: /dghqvd-54/c 2023-10-11T15:03:51.361 DEBUG:boto:Data: 2023-10-11T15:03:51.362 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.362 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.363 DEBUG:boto:Params: {} 2023-10-11T15:03:51.363 DEBUG:boto:Token: None 2023-10-11T15:03:51.364 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c 2023-10-11T15:03:51.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V5GBH6hycKNto8w3ov+Oe19RpfI= 2023-10-11T15:03:51.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V5GBH6hycKNto8w3ov+Oe19RpfI=', 'Content-Length': '0'} 2023-10-11T15:03:51.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399a9d8ac5506990-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:03:51.368 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.369 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.369 DEBUG:boto:Method: GET 2023-10-11T15:03:51.369 DEBUG:boto:Path: /dghqvd-54/c 2023-10-11T15:03:51.370 DEBUG:boto:Data: 2023-10-11T15:03:51.370 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.371 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.372 DEBUG:boto:Params: {} 2023-10-11T15:03:51.372 DEBUG:boto:Token: None 2023-10-11T15:03:51.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c 2023-10-11T15:03:51.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1wMrQQEn+uHpX2byuDVniCP3VLg= 2023-10-11T15:03:51.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1wMrQQEn+uHpX2byuDVniCP3VLg=', 'Content-Length': '0'} 2023-10-11T15:03:51.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aafd350a7f58711d-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.418 DEBUG:boto:path=/dghqvd-54/c 2023-10-11T15:03:51.419 DEBUG:boto:auth_path=/dghqvd-54/c 2023-10-11T15:03:51.419 DEBUG:boto:Method: GET 2023-10-11T15:03:51.420 DEBUG:boto:Path: /dghqvd-54/c 2023-10-11T15:03:51.420 DEBUG:boto:Data: 2023-10-11T15:03:51.421 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.421 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.422 DEBUG:boto:Params: {} 2023-10-11T15:03:51.422 DEBUG:boto:Token: None 2023-10-11T15:03:51.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/c 2023-10-11T15:03:51.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1wMrQQEn+uHpX2byuDVniCP3VLg= 2023-10-11T15:03:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1wMrQQEn+uHpX2byuDVniCP3VLg=', 'Content-Length': '0'} 2023-10-11T15:03:51.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a6ee107a169b72-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.427 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.427 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.427 DEBUG:boto:path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.428 DEBUG:boto:auth_path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.428 DEBUG:boto:Method: GET 2023-10-11T15:03:51.429 DEBUG:boto:Path: /dghqvd-54/?prefix=c 2023-10-11T15:03:51.429 DEBUG:boto:Data: 2023-10-11T15:03:51.430 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.431 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.431 DEBUG:boto:Params: {} 2023-10-11T15:03:51.432 DEBUG:boto:Token: None 2023-10-11T15:03:51.432 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d8ec647dac7b07-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.436 DEBUG:boto:b'dghqvd-54c1000falsec2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.437 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.437 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.438 DEBUG:boto:path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.438 DEBUG:boto:auth_path=/dghqvd-54/?prefix=c 2023-10-11T15:03:51.439 DEBUG:boto:Method: GET 2023-10-11T15:03:51.439 DEBUG:boto:Path: /dghqvd-54/?prefix=c 2023-10-11T15:03:51.439 DEBUG:boto:Data: 2023-10-11T15:03:51.440 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.441 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.441 DEBUG:boto:Params: {} 2023-10-11T15:03:51.442 DEBUG:boto:Token: None 2023-10-11T15:03:51.442 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b22ab5f35c83f8-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.446 DEBUG:boto:b'dghqvd-54c1000falsec2023-10-11T15:03:10.231Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:51.447 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.448 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.448 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.448 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.449 DEBUG:boto:Method: GET 2023-10-11T15:03:51.449 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.450 DEBUG:boto:Data: 2023-10-11T15:03:51.450 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.451 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.451 DEBUG:boto:Params: {} 2023-10-11T15:03:51.452 DEBUG:boto:Token: None 2023-10-11T15:03:51.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY= 2023-10-11T15:03:51.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY=', 'Content-Length': '0'} 2023-10-11T15:03:51.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e9569228c118de-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.456 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.457 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.457 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.458 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.458 DEBUG:boto:Method: GET 2023-10-11T15:03:51.458 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.459 DEBUG:boto:Data: 2023-10-11T15:03:51.459 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.460 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.461 DEBUG:boto:Params: {} 2023-10-11T15:03:51.461 DEBUG:boto:Token: None 2023-10-11T15:03:51.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.462 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY= 2023-10-11T15:03:51.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY=', 'Content-Length': '0'} 2023-10-11T15:03:51.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052083d676dcf1128-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.465 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.466 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.466 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.466 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.467 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.467 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.468 DEBUG:boto:Data: 2023-10-11T15:03:51.468 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.469 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.469 DEBUG:boto:Params: {} 2023-10-11T15:03:51.470 DEBUG:boto:Token: None 2023-10-11T15:03:51.470 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6I9OyYIWvahSY2etF5d7PZM51Ls= 2023-10-11T15:03:51.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6I9OyYIWvahSY2etF5d7PZM51Ls=', 'Content-Length': '0'} 2023-10-11T15:03:51.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa0d4f51977374ae-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.474 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.474 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.474 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.475 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.475 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.476 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.476 DEBUG:boto:Data: 2023-10-11T15:03:51.477 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.477 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.478 DEBUG:boto:Params: {} 2023-10-11T15:03:51.478 DEBUG:boto:Token: None 2023-10-11T15:03:51.479 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6I9OyYIWvahSY2etF5d7PZM51Ls= 2023-10-11T15:03:51.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6I9OyYIWvahSY2etF5d7PZM51Ls=', 'Content-Length': '0'} 2023-10-11T15:03:51.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef17edec5bb8d52-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:51.483 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.483 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.484 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.484 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.485 DEBUG:boto:Method: GET 2023-10-11T15:03:51.485 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.485 DEBUG:boto:Data: 2023-10-11T15:03:51.486 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.487 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.487 DEBUG:boto:Params: {} 2023-10-11T15:03:51.488 DEBUG:boto:Token: None 2023-10-11T15:03:51.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY= 2023-10-11T15:03:51.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY=', 'Content-Length': '0'} 2023-10-11T15:03:51.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d9543766ebcb23-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.491 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.492 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.492 DEBUG:boto:path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.493 DEBUG:boto:auth_path=/dghqvd-54/d?versionId=null 2023-10-11T15:03:51.493 DEBUG:boto:Method: GET 2023-10-11T15:03:51.494 DEBUG:boto:Path: /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.494 DEBUG:boto:Data: 2023-10-11T15:03:51.495 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.495 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.495 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.496 DEBUG:boto:Params: {} 2023-10-11T15:03:51.496 DEBUG:boto:Token: None 2023-10-11T15:03:51.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d?versionId=null 2023-10-11T15:03:51.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY= 2023-10-11T15:03:51.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4gUk2n1AdzLqX4p37w5kz3DdoY=', 'Content-Length': '0'} 2023-10-11T15:03:51.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7545fdfab786ed-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.500 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.501 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.501 DEBUG:boto:path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.502 DEBUG:boto:auth_path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.502 DEBUG:boto:Method: GET 2023-10-11T15:03:51.503 DEBUG:boto:Path: /dghqvd-54/?prefix=d 2023-10-11T15:03:51.503 DEBUG:boto:Data: 2023-10-11T15:03:51.504 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.504 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.505 DEBUG:boto:Params: {} 2023-10-11T15:03:51.505 DEBUG:boto:Token: None 2023-10-11T15:03:51.506 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0cf9749966fbe6b-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.550 DEBUG:boto:b'dghqvd-54d1000falsed2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.551 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.551 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.552 DEBUG:boto:path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.552 DEBUG:boto:auth_path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.553 DEBUG:boto:Method: GET 2023-10-11T15:03:51.553 DEBUG:boto:Path: /dghqvd-54/?prefix=d 2023-10-11T15:03:51.553 DEBUG:boto:Data: 2023-10-11T15:03:51.554 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.555 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.555 DEBUG:boto:Params: {} 2023-10-11T15:03:51.556 DEBUG:boto:Token: None 2023-10-11T15:03:51.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000758d248ab4817200-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.560 DEBUG:boto:b'dghqvd-54d1000falsed2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.561 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.561 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.562 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.562 DEBUG:boto:Path: /dghqvd-54/d 2023-10-11T15:03:51.563 DEBUG:boto:Data: 2023-10-11T15:03:51.563 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.564 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.564 DEBUG:boto:Params: {} 2023-10-11T15:03:51.565 DEBUG:boto:Token: None 2023-10-11T15:03:51.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d 2023-10-11T15:03:51.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1iDx6aUEJFQGODZIGzoXoqiiQ8w= 2023-10-11T15:03:51.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1iDx6aUEJFQGODZIGzoXoqiiQ8w=', 'Content-Length': '0'} 2023-10-11T15:03:51.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d1cf0b819964e3-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.569 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.569 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.569 DEBUG:boto:Method: HEAD 2023-10-11T15:03:51.570 DEBUG:boto:Path: /dghqvd-54/d 2023-10-11T15:03:51.570 DEBUG:boto:Data: 2023-10-11T15:03:51.571 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.572 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.572 DEBUG:boto:Params: {} 2023-10-11T15:03:51.572 DEBUG:boto:Token: None 2023-10-11T15:03:51.573 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d 2023-10-11T15:03:51.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1iDx6aUEJFQGODZIGzoXoqiiQ8w= 2023-10-11T15:03:51.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1iDx6aUEJFQGODZIGzoXoqiiQ8w=', 'Content-Length': '0'} 2023-10-11T15:03:51.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219b8f72515f72b8-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:03:51.577 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.577 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.577 DEBUG:boto:Method: GET 2023-10-11T15:03:51.578 DEBUG:boto:Path: /dghqvd-54/d 2023-10-11T15:03:51.578 DEBUG:boto:Data: 2023-10-11T15:03:51.579 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.580 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.580 DEBUG:boto:Params: {} 2023-10-11T15:03:51.581 DEBUG:boto:Token: None 2023-10-11T15:03:51.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d 2023-10-11T15:03:51.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IWAdYI7X9U51AENMK/RUxXTOwso= 2023-10-11T15:03:51.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IWAdYI7X9U51AENMK/RUxXTOwso=', 'Content-Length': '0'} 2023-10-11T15:03:51.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-54:daa316e9-869c-4074-a756-a8adfb4a3573.5751.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9f4ec5755e3266-006526b957-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.585 DEBUG:boto:path=/dghqvd-54/d 2023-10-11T15:03:51.585 DEBUG:boto:auth_path=/dghqvd-54/d 2023-10-11T15:03:51.586 DEBUG:boto:Method: GET 2023-10-11T15:03:51.586 DEBUG:boto:Path: /dghqvd-54/d 2023-10-11T15:03:51.586 DEBUG:boto:Data: 2023-10-11T15:03:51.587 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.588 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.588 DEBUG:boto:Params: {} 2023-10-11T15:03:51.589 DEBUG:boto:Token: None 2023-10-11T15:03:51.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/d 2023-10-11T15:03:51.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IWAdYI7X9U51AENMK/RUxXTOwso= 2023-10-11T15:03:51.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IWAdYI7X9U51AENMK/RUxXTOwso=', 'Content-Length': '0'} 2023-10-11T15:03:51.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000625552ee162363-006526b957-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.634 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.635 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.635 DEBUG:boto:path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.636 DEBUG:boto:auth_path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.636 DEBUG:boto:Method: GET 2023-10-11T15:03:51.636 DEBUG:boto:Path: /dghqvd-54/?prefix=d 2023-10-11T15:03:51.637 DEBUG:boto:Data: 2023-10-11T15:03:51.637 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:03:51.638 DEBUG:boto:Port: 8001 2023-10-11T15:03:51.639 DEBUG:boto:Params: {} 2023-10-11T15:03:51.639 DEBUG:boto:Token: None 2023-10-11T15:03:51.640 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280fe73137e7938d-006526b957-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.643 DEBUG:boto:b'dghqvd-54d1000falsed2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.644 DEBUG:boto:path=/dghqvd-54/ 2023-10-11T15:03:51.645 DEBUG:boto:auth_path=/dghqvd-54/ 2023-10-11T15:03:51.645 DEBUG:boto:path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.645 DEBUG:boto:auth_path=/dghqvd-54/?prefix=d 2023-10-11T15:03:51.646 DEBUG:boto:Method: GET 2023-10-11T15:03:51.646 DEBUG:boto:Path: /dghqvd-54/?prefix=d 2023-10-11T15:03:51.647 DEBUG:boto:Data: 2023-10-11T15:03:51.647 DEBUG:boto:Headers: {} 2023-10-11T15:03:51.648 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:51.648 DEBUG:boto:Port: 8000 2023-10-11T15:03:51.648 DEBUG:boto:Params: {} 2023-10-11T15:03:51.649 DEBUG:boto:Token: None 2023-10-11T15:03:51.649 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:03:51 GMT /dghqvd-54/ 2023-10-11T15:03:51.650 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o= 2023-10-11T15:03:51.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V7sMGXogDM0yn/E0u1H7VdMwz1o=', 'Content-Length': '0'} 2023-10-11T15:03:51.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069fb15491e7a370d-006526b957-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:03:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:51.653 DEBUG:boto:b'dghqvd-54d1000falsed2023-10-11T15:03:10.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:03:51.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-54 zones={a2, a1} 2023-10-11T15:03:51.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:51.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:51.655 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:03:51.706 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:51.707 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:51.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:51.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:51.761 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:51.809 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:51.849 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T15:03:51.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:51.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:51.852 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:51.899 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:51.899 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:51.939 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:03:51.940 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.940 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "1#00000000001.35.6", 2023-10-11T15:03:51.940 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.16158698490607676235", 2023-10-11T15:03:51.940 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:03:51.941 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "a", 2023-10-11T15:03:51.941 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:03:51.941 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:03:51.941 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:03:51.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:33.623271461Z", 2023-10-11T15:03:51.942 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:03:51.942 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:03:51.942 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:03:51.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.943 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:03:51.943 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:03:51.943 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:03:51.943 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:03:51.943 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:03:51.944 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.944 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18" 2023-10-11T15:03:51.944 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:51.944 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:51.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.945 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.945 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "5#00000000001.42.6", 2023-10-11T15:03:51.945 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.18379934428367698681", 2023-10-11T15:03:51.945 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:03:51.946 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "d", 2023-10-11T15:03:51.946 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:03:51.946 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:03:51.946 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:03:51.946 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:33.714231876Z", 2023-10-11T15:03:51.947 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:03:51.947 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:03:51.947 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:03:51.947 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.947 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:03:51.948 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:03:51.948 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:03:51.948 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:03:51.948 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:03:51.948 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.949 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18" 2023-10-11T15:03:51.949 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:51.949 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:51.949 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.950 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.950 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "8#00000000001.27.6", 2023-10-11T15:03:51.950 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17932344923509454373", 2023-10-11T15:03:51.950 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:03:51.950 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "c", 2023-10-11T15:03:51.951 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:03:51.951 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:03:51.951 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:03:51.951 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:33.696355953Z", 2023-10-11T15:03:51.951 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:03:51.952 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:03:51.952 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:03:51.952 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.952 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:03:51.952 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:03:51.953 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:03:51.953 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:03:51.953 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:03:51.953 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.953 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18" 2023-10-11T15:03:51.954 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:51.954 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:51.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.954 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.955 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "10#00000000001.56.6", 2023-10-11T15:03:51.955 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17775731342590833264", 2023-10-11T15:03:51.955 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:03:51.955 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "b", 2023-10-11T15:03:51.955 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:03:51.956 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:03:51.956 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:03:51.956 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:02:33.641639586Z", 2023-10-11T15:03:51.956 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:03:51.956 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:03:51.957 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:03:51.957 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:51.957 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:03:51.957 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:03:51.957 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:03:51.958 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:03:51.958 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:03:51.958 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:51.958 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-53:daa316e9-869c-4074-a756-a8adfb4a3573.5751.18" 2023-10-11T15:03:51.958 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:51.959 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:51.959 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:51.959 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:03:51.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:51.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:03:51.961 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:03:52.007 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:52.008 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:52.049 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.047+0000 7fc41c345700 -1 req 8268887399072172830 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.049 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.047+0000 7fc335177700 -1 req 8498993464205516608 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.050 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.047+0000 7fc33f98c700 -1 req 5660790539795384439 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.050 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.048+0000 7fc40ab22700 -1 req 642600810493946139 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.052 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.051+0000 7fc41b343700 -1 req 10727176495063916062 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.053 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.052+0000 7fc40d327700 -1 req 9644241608777643909 0.002999967s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.095 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.093+0000 7fc3f72fb700 -1 req 9085023149155205924 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.096 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.095+0000 7fc405b18700 -1 req 1735168660105311841 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.099 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.097+0000 7fc3ffb0c700 -1 req 15529457671277749036 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.140 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.138+0000 7fc3d52b7700 -1 req 13822472650351790337 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.144 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.143+0000 7fc3de2c9700 -1 req 16686409023299290852 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.229 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.227+0000 7fc3caaa2700 -1 req 16954891963067817175 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.238 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.237+0000 7fc3a124f700 -1 req 8005595719524025371 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.272 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.270+0000 7fc3b6a7a700 -1 req 933526925308079262 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.282 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.280+0000 7fc3bea8a700 -1 req 8603189597776321960 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.327 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.326+0000 7fc3ad267700 -1 req 8940733375399460860 0.002999967s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.331 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.329+0000 7fc37e209700 -1 req 12215934568658069346 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-22:daa316e9-869c-4074-a756-a8adfb4a3573.4373.22,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-22:daa316e9-869c-4074-a756-a8adfb4a3573.4373.22,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.454 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.452+0000 7fc3619d0700 -1 req 17160525213969781704 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-1:daa316e9-869c-4074-a756-a8adfb4a3573.4373.1,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-1:daa316e9-869c-4074-a756-a8adfb4a3573.4373.1,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.460 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.459+0000 7fc430b6e700 -1 req 3203913908544603 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-5:daa316e9-869c-4074-a756-a8adfb4a3573.4373.5,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-5:daa316e9-869c-4074-a756-a8adfb4a3573.4373.5,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.497 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.495+0000 7fc343994700 -1 req 1998450147616309362 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-41:daa316e9-869c-4074-a756-a8adfb4a3573.5751.6,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-41:daa316e9-869c-4074-a756-a8adfb4a3573.5751.6,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.504 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.502+0000 7fc42c365700 -1 req 7116413705755430278 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-27:daa316e9-869c-4074-a756-a8adfb4a3573.4373.27,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-27:daa316e9-869c-4074-a756-a8adfb4a3573.4373.27,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.511 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.509+0000 7fc3539b4700 -1 req 18154498541059781517 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-38:daa316e9-869c-4074-a756-a8adfb4a3573.5751.3,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-38:daa316e9-869c-4074-a756-a8adfb4a3573.5751.3,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.511 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.509+0000 7fc421b50700 -1 req 6489453345074184116 0.001999978s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-3:daa316e9-869c-4074-a756-a8adfb4a3573.4373.3,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-3:daa316e9-869c-4074-a756-a8adfb4a3573.4373.3,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.540 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.538+0000 7fc42cb66700 -1 req 9162940886621173298 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-40:daa316e9-869c-4074-a756-a8adfb4a3573.5751.5,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-40:daa316e9-869c-4074-a756-a8adfb4a3573.5751.5,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.547 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.545+0000 7fc345197700 -1 req 13371328516843487025 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-29:daa316e9-869c-4074-a756-a8adfb4a3573.4373.29,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-29:daa316e9-869c-4074-a756-a8adfb4a3573.4373.29,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.554 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.552+0000 7fc42134f700 -1 req 12308345488179644665 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-37:daa316e9-869c-4074-a756-a8adfb4a3573.5751.2,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-37:daa316e9-869c-4074-a756-a8adfb4a3573.5751.2,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.589 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:03:52.588+0000 7fc3fd307700 -1 req 3708716338779592532 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-39:daa316e9-869c-4074-a756-a8adfb4a3573.5751.4,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-39:daa316e9-869c-4074-a756-a8adfb4a3573.5751.4,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:03:52.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:52.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:52.648 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:52.696 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:52.697 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:52.738 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T15:03:52.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-10-11T15:03:52.741 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-55 2023-10-11T15:03:52.742 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:03:52.742 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:03:52.743 DEBUG:boto:Method: PUT 2023-10-11T15:03:52.743 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:03:52.743 DEBUG:boto:Data: 2023-10-11T15:03:52.744 DEBUG:boto:Headers: {} 2023-10-11T15:03:52.744 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.745 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.745 DEBUG:boto:Params: {} 2023-10-11T15:03:52.745 DEBUG:boto:Token: None 2023-10-11T15:03:52.746 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/ 2023-10-11T15:03:52.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HzuaHPoN25AuhbsebIRtxjG23TA= 2023-10-11T15:03:52.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HzuaHPoN25AuhbsebIRtxjG23TA=', 'Content-Length': '0'} 2023-10-11T15:03:52.761 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000b6e88c4ddf1ed9-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.761 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:03:52.761 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:03:52.762 DEBUG:boto:Method: HEAD 2023-10-11T15:03:52.762 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:03:52.763 DEBUG:boto:Data: 2023-10-11T15:03:52.763 DEBUG:boto:Headers: {} 2023-10-11T15:03:52.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.764 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.764 DEBUG:boto:Params: {} 2023-10-11T15:03:52.765 DEBUG:boto:Token: None 2023-10-11T15:03:52.765 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/ 2023-10-11T15:03:52.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T15:03:52.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T15:03:52.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0dc4a2639a299c4-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.769 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:03:52.769 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:03:52.770 DEBUG:boto:Method: PUT 2023-10-11T15:03:52.770 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:03:52.771 DEBUG:boto:Data: 2023-10-11T15:03:52.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:52.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.772 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.772 DEBUG:boto:Params: {} 2023-10-11T15:03:52.773 DEBUG:boto:Token: None 2023-10-11T15:03:52.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/a 2023-10-11T15:03:52.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2rrqrUWT6/iwwUcyh47FRD5AHcU= 2023-10-11T15:03:52.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rrqrUWT6/iwwUcyh47FRD5AHcU='} 2023-10-11T15:03:52.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a52b94000b577e9c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.781 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:03:52.781 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:03:52.782 DEBUG:boto:Method: HEAD 2023-10-11T15:03:52.782 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:03:52.783 DEBUG:boto:Data: 2023-10-11T15:03:52.783 DEBUG:boto:Headers: {} 2023-10-11T15:03:52.783 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.784 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.784 DEBUG:boto:Params: {} 2023-10-11T15:03:52.785 DEBUG:boto:Token: None 2023-10-11T15:03:52.785 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/ 2023-10-11T15:03:52.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T15:03:52.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T15:03:52.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006275e8abcc26b14f-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.789 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:03:52.789 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:03:52.790 DEBUG:boto:Method: PUT 2023-10-11T15:03:52.790 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:03:52.791 DEBUG:boto:Data: 2023-10-11T15:03:52.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:52.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.792 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.792 DEBUG:boto:Params: {} 2023-10-11T15:03:52.793 DEBUG:boto:Token: None 2023-10-11T15:03:52.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/b 2023-10-11T15:03:52.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RVRbOvHnC0eBOC4X0dtRwhHg7yM= 2023-10-11T15:03:52.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RVRbOvHnC0eBOC4X0dtRwhHg7yM='} 2023-10-11T15:03:52.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da5aa175fed57e92-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.839 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:03:52.840 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:03:52.840 DEBUG:boto:Method: HEAD 2023-10-11T15:03:52.840 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:03:52.841 DEBUG:boto:Data: 2023-10-11T15:03:52.841 DEBUG:boto:Headers: {} 2023-10-11T15:03:52.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.842 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.842 DEBUG:boto:Params: {} 2023-10-11T15:03:52.843 DEBUG:boto:Token: None 2023-10-11T15:03:52.843 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/ 2023-10-11T15:03:52.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T15:03:52.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T15:03:52.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d784476efce0575-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.847 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:03:52.847 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:03:52.848 DEBUG:boto:Method: PUT 2023-10-11T15:03:52.848 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:03:52.849 DEBUG:boto:Data: 2023-10-11T15:03:52.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:52.850 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.850 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.851 DEBUG:boto:Params: {} 2023-10-11T15:03:52.851 DEBUG:boto:Token: None 2023-10-11T15:03:52.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/c 2023-10-11T15:03:52.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VJJcyCzXNDJ1zcSOrTpJsuNz/zQ= 2023-10-11T15:03:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VJJcyCzXNDJ1zcSOrTpJsuNz/zQ='} 2023-10-11T15:03:52.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acd9ed497b6df23c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.858 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:03:52.858 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:03:52.859 DEBUG:boto:Method: HEAD 2023-10-11T15:03:52.859 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:03:52.860 DEBUG:boto:Data: 2023-10-11T15:03:52.860 DEBUG:boto:Headers: {} 2023-10-11T15:03:52.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.861 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.861 DEBUG:boto:Params: {} 2023-10-11T15:03:52.862 DEBUG:boto:Token: None 2023-10-11T15:03:52.862 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/ 2023-10-11T15:03:52.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T15:03:52.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T15:03:52.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924d71486b5d3a11-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.866 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:03:52.867 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:03:52.867 DEBUG:boto:Method: PUT 2023-10-11T15:03:52.867 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:03:52.868 DEBUG:boto:Data: 2023-10-11T15:03:52.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:03:52.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:03:52.869 DEBUG:boto:Port: 8000 2023-10-11T15:03:52.869 DEBUG:boto:Params: {} 2023-10-11T15:03:52.870 DEBUG:boto:Token: None 2023-10-11T15:03:52.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:03:52 GMT /dghqvd-55/d 2023-10-11T15:03:52.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q0yOkRW3pAD+hvj/oh4lcrQmhrU= 2023-10-11T15:03:52.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q0yOkRW3pAD+hvj/oh4lcrQmhrU='} 2023-10-11T15:03:52.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc1fe989b5bdee0c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:03:52.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:52.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:52.916 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:52.964 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:52.965 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:52.993 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:03:53.005 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:03:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:03:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:03:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.006 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:03:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:03:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.007 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:03:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:03:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.008 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036632.758331_393.1", 2023-10-11T15:03:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:03:52.758331Z" 2023-10-11T15:03:53.009 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.009 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:03:53.010 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T15:03:53.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:03:53.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:53.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:03:53.012 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:03:53.061 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:53.062 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:53.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:53.088+0000 7f03602de780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:03:53.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:53.089+0000 7f03602de780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:03:53.103 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:03:53.103 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:03:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:03:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:03:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:03:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:03:53.104 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:03:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:03:53.105 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:03:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:53.106 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:03:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:03:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.107 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:03:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:53.108 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:03:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:03:53.109 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.110 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:03:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:03:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:53.111 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:03:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.112 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:03:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:03:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:03:53.113 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:03:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036632.758331_393.1", 2023-10-11T15:03:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:03:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:03:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:03:53.114 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:52.758331Z", 2023-10-11T15:03:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:03:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:03:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:03:53.115 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:03:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:03:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:03:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:03:53.116 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:03:53Z" 2023-10-11T15:03:53.117 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:03:53.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036632.758331_393.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:03:52.758331Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:03:53Z" } 2023-10-11T15:03:53.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036632.758331_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:03:52.758331Z', 'realm_epoch': 2}}] 2023-10-11T15:03:53.119 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T15:03:53.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T15:03:53.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:03:53.120 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T15:03:53.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:53.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:03:53.122 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:03:53.178 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:03:53.179 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:03:53.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:53.210+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T15:03:58.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:03:58.211+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T15:04:03.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:03.211+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T15:04:08.213 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:08.212+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T15:04:13.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:13.213+0000 7fe22de49780 1 bucket sync caught up with source: 2023-10-11T15:04:13.215 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.16.6, , , , 00000000001.23.6, , , 00000000001.55.6, , 00000000001.24.6] 2023-10-11T15:04:13.216 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.16.6, , , , 00000000001.23.6, , , 00000000001.55.6, , 00000000001.24.6] 2023-10-11T15:04:13.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:13.213+0000 7fe22de49780 0 bucket checkpoint complete 2023-10-11T15:04:13.231 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T15:04:13.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:04:13.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:04:13.233 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:04:13.293 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:13.293 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:13.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:13.326+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T15:04:18.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:18.326+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T15:04:23.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:23.327+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T15:04:28.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:28.327+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T15:04:33.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:33.329+0000 7f43b40db780 1 bucket sync caught up with source: 2023-10-11T15:04:33.332 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.45.6, , , , 00000000001.261.6, , , 00000000001.270.6, , 00000000001.130.6] 2023-10-11T15:04:33.332 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.45.6, , , , 00000000001.261.6, , , 00000000001.270.6, , 00000000001.130.6] 2023-10-11T15:04:33.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:33.329+0000 7f43b40db780 0 bucket checkpoint complete 2023-10-11T15:04:33.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:04:33.349 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.349 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.350 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.350 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:33.351 DEBUG:boto:Data: 2023-10-11T15:04:33.351 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.352 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.353 DEBUG:boto:Params: {} 2023-10-11T15:04:33.353 DEBUG:boto:Token: None 2023-10-11T15:04:33.354 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.354 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE= 2023-10-11T15:04:33.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE=', 'Content-Length': '0'} 2023-10-11T15:04:33.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835b372aa59b0410-006526b981-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.358 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.359 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.359 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.360 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:33.360 DEBUG:boto:Data: 2023-10-11T15:04:33.361 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.362 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.362 DEBUG:boto:Params: {} 2023-10-11T15:04:33.363 DEBUG:boto:Token: None 2023-10-11T15:04:33.363 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE= 2023-10-11T15:04:33.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE=', 'Content-Length': '0'} 2023-10-11T15:04:33.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbbcb7ea56acf70b-006526b981-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:04:33.368 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.368 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.369 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.369 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.370 DEBUG:boto:Method: GET 2023-10-11T15:04:33.370 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.371 DEBUG:boto:Data: 2023-10-11T15:04:33.371 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.372 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.373 DEBUG:boto:Params: {} 2023-10-11T15:04:33.373 DEBUG:boto:Token: None 2023-10-11T15:04:33.374 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/?versions 2023-10-11T15:04:33.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T15:04:33.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T15:04:33.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d487caf8eff413-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.378 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.379 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:04:33.379 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:04:33.380 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:04:33.380 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:04:33.381 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:04:33.381 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.382 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.383 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.383 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.384 DEBUG:boto:Method: GET 2023-10-11T15:04:33.384 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.385 DEBUG:boto:Data: 2023-10-11T15:04:33.385 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.386 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.386 DEBUG:boto:Params: {} 2023-10-11T15:04:33.387 DEBUG:boto:Token: None 2023-10-11T15:04:33.387 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/?versions 2023-10-11T15:04:33.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T15:04:33.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T15:04:33.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d77e95f411561ec-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.392 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.393 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:04:33.393 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:04:33.393 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:04:33.394 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:04:33.394 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.395 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.395 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.395 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.396 DEBUG:boto:Method: GET 2023-10-11T15:04:33.396 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.397 DEBUG:boto:Data: 2023-10-11T15:04:33.397 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.398 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.398 DEBUG:boto:Params: {} 2023-10-11T15:04:33.399 DEBUG:boto:Token: None 2023-10-11T15:04:33.399 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/?versions 2023-10-11T15:04:33.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T15:04:33.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T15:04:33.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf3cbbc09bdde8e-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.444 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.445 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.446 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.447 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.447 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.448 DEBUG:boto:Method: GET 2023-10-11T15:04:33.448 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:04:33.448 DEBUG:boto:Data: 2023-10-11T15:04:33.449 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.450 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.450 DEBUG:boto:Params: {} 2023-10-11T15:04:33.451 DEBUG:boto:Token: None 2023-10-11T15:04:33.451 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/?versions 2023-10-11T15:04:33.452 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T15:04:33.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T15:04:33.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003130f5064589dd22-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.455 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:04:33.457 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.457 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.457 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.458 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.458 DEBUG:boto:Method: GET 2023-10-11T15:04:33.459 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.459 DEBUG:boto:Data: 2023-10-11T15:04:33.460 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.461 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.461 DEBUG:boto:Params: {} 2023-10-11T15:04:33.461 DEBUG:boto:Token: None 2023-10-11T15:04:33.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T15:04:33.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T15:04:33.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72d44ccdd4e3776-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.466 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.466 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.467 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.467 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.467 DEBUG:boto:Method: GET 2023-10-11T15:04:33.468 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.469 DEBUG:boto:Data: 2023-10-11T15:04:33.470 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.471 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.472 DEBUG:boto:Params: {} 2023-10-11T15:04:33.472 DEBUG:boto:Token: None 2023-10-11T15:04:33.473 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.473 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T15:04:33.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T15:04:33.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b95c900df2c781-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.477 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.478 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.478 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.478 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.479 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.479 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.480 DEBUG:boto:Data: 2023-10-11T15:04:33.480 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.481 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.481 DEBUG:boto:Params: {} 2023-10-11T15:04:33.482 DEBUG:boto:Token: None 2023-10-11T15:04:33.482 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o= 2023-10-11T15:04:33.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o=', 'Content-Length': '0'} 2023-10-11T15:04:33.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b837400205bc50b9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.486 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.487 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.487 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.487 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.488 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.488 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.489 DEBUG:boto:Data: 2023-10-11T15:04:33.489 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.490 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.490 DEBUG:boto:Params: {} 2023-10-11T15:04:33.491 DEBUG:boto:Token: None 2023-10-11T15:04:33.491 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o= 2023-10-11T15:04:33.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o=', 'Content-Length': '0'} 2023-10-11T15:04:33.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002931de8b6d4c6460-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:04:33.496 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.496 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.497 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.497 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.497 DEBUG:boto:Method: GET 2023-10-11T15:04:33.498 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.498 DEBUG:boto:Data: 2023-10-11T15:04:33.499 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.500 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.500 DEBUG:boto:Params: {} 2023-10-11T15:04:33.501 DEBUG:boto:Token: None 2023-10-11T15:04:33.501 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T15:04:33.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T15:04:33.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee377aabbfd2fe8e-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.505 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.506 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.506 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.506 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:04:33.507 DEBUG:boto:Method: GET 2023-10-11T15:04:33.507 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.508 DEBUG:boto:Data: 2023-10-11T15:04:33.508 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.509 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.509 DEBUG:boto:Params: {} 2023-10-11T15:04:33.510 DEBUG:boto:Token: None 2023-10-11T15:04:33.510 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:04:33.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T15:04:33.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T15:04:33.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c646328abc2ad766-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.554 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.555 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.555 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.556 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.556 DEBUG:boto:Method: GET 2023-10-11T15:04:33.557 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:04:33.557 DEBUG:boto:Data: 2023-10-11T15:04:33.558 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.558 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.559 DEBUG:boto:Params: {} 2023-10-11T15:04:33.559 DEBUG:boto:Token: None 2023-10-11T15:04:33.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e02b48ae72b7c45e-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.564 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.565 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.566 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.566 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.567 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.567 DEBUG:boto:Method: GET 2023-10-11T15:04:33.568 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:04:33.568 DEBUG:boto:Data: 2023-10-11T15:04:33.569 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.569 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.570 DEBUG:boto:Params: {} 2023-10-11T15:04:33.570 DEBUG:boto:Token: None 2023-10-11T15:04:33.571 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3601ed9e9b2280-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.575 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.576 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.576 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.577 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.577 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:04:33.577 DEBUG:boto:Data: 2023-10-11T15:04:33.578 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.579 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.579 DEBUG:boto:Params: {} 2023-10-11T15:04:33.580 DEBUG:boto:Token: None 2023-10-11T15:04:33.580 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a 2023-10-11T15:04:33.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE= 2023-10-11T15:04:33.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE=', 'Content-Length': '0'} 2023-10-11T15:04:33.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000297c518b9baaa2ac-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.584 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.584 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.584 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.585 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:04:33.585 DEBUG:boto:Data: 2023-10-11T15:04:33.586 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.586 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.587 DEBUG:boto:Params: {} 2023-10-11T15:04:33.587 DEBUG:boto:Token: None 2023-10-11T15:04:33.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a 2023-10-11T15:04:33.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE= 2023-10-11T15:04:33.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE=', 'Content-Length': '0'} 2023-10-11T15:04:33.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e845db68e48c80-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:04:33.592 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.592 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.593 DEBUG:boto:Method: GET 2023-10-11T15:04:33.593 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:04:33.594 DEBUG:boto:Data: 2023-10-11T15:04:33.594 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.595 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.595 DEBUG:boto:Params: {} 2023-10-11T15:04:33.596 DEBUG:boto:Token: None 2023-10-11T15:04:33.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a 2023-10-11T15:04:33.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090= 2023-10-11T15:04:33.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090=', 'Content-Length': '0'} 2023-10-11T15:04:33.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5775789c680172-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.646 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:04:33.647 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:04:33.647 DEBUG:boto:Method: GET 2023-10-11T15:04:33.648 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:04:33.648 DEBUG:boto:Data: 2023-10-11T15:04:33.649 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.650 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.650 DEBUG:boto:Params: {} 2023-10-11T15:04:33.651 DEBUG:boto:Token: None 2023-10-11T15:04:33.651 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/a 2023-10-11T15:04:33.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090= 2023-10-11T15:04:33.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090=', 'Content-Length': '0'} 2023-10-11T15:04:33.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb07fa91458a01a8-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.655 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.655 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.656 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.656 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.656 DEBUG:boto:Method: GET 2023-10-11T15:04:33.657 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:04:33.657 DEBUG:boto:Data: 2023-10-11T15:04:33.658 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.659 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.659 DEBUG:boto:Params: {} 2023-10-11T15:04:33.660 DEBUG:boto:Token: None 2023-10-11T15:04:33.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000500430adc62074f8-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.664 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.665 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.665 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.665 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.666 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:04:33.666 DEBUG:boto:Method: GET 2023-10-11T15:04:33.667 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:04:33.667 DEBUG:boto:Data: 2023-10-11T15:04:33.668 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.669 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.669 DEBUG:boto:Params: {} 2023-10-11T15:04:33.669 DEBUG:boto:Token: None 2023-10-11T15:04:33.670 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c75861b9d9ae000-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.674 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:04:33.675 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.675 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.676 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.676 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.677 DEBUG:boto:Method: GET 2023-10-11T15:04:33.677 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.678 DEBUG:boto:Data: 2023-10-11T15:04:33.678 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.679 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.679 DEBUG:boto:Params: {} 2023-10-11T15:04:33.680 DEBUG:boto:Token: None 2023-10-11T15:04:33.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T15:04:33.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T15:04:33.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882c925d2bca3d1a-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.684 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.684 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.685 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.685 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.686 DEBUG:boto:Method: GET 2023-10-11T15:04:33.686 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.686 DEBUG:boto:Data: 2023-10-11T15:04:33.687 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.688 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.688 DEBUG:boto:Params: {} 2023-10-11T15:04:33.689 DEBUG:boto:Token: None 2023-10-11T15:04:33.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T15:04:33.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T15:04:33.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697fe8ef732d9830-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.693 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.693 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.694 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.694 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.695 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.695 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.695 DEBUG:boto:Data: 2023-10-11T15:04:33.696 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.697 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.697 DEBUG:boto:Params: {} 2023-10-11T15:04:33.698 DEBUG:boto:Token: None 2023-10-11T15:04:33.698 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU= 2023-10-11T15:04:33.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU=', 'Content-Length': '0'} 2023-10-11T15:04:33.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d85401c5854d652-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.701 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.702 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.702 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.703 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.703 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.703 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.704 DEBUG:boto:Data: 2023-10-11T15:04:33.704 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.705 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.706 DEBUG:boto:Params: {} 2023-10-11T15:04:33.706 DEBUG:boto:Token: None 2023-10-11T15:04:33.707 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU= 2023-10-11T15:04:33.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU=', 'Content-Length': '0'} 2023-10-11T15:04:33.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a12d389abb32d79-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:04:33.710 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.711 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.711 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.712 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.712 DEBUG:boto:Method: GET 2023-10-11T15:04:33.712 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.713 DEBUG:boto:Data: 2023-10-11T15:04:33.713 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.714 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.715 DEBUG:boto:Params: {} 2023-10-11T15:04:33.715 DEBUG:boto:Token: None 2023-10-11T15:04:33.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T15:04:33.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T15:04:33.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078075b43f5d5f104-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.722 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.723 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.723 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.724 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:04:33.724 DEBUG:boto:Method: GET 2023-10-11T15:04:33.725 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.725 DEBUG:boto:Data: 2023-10-11T15:04:33.726 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.726 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.726 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.727 DEBUG:boto:Params: {} 2023-10-11T15:04:33.727 DEBUG:boto:Token: None 2023-10-11T15:04:33.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:04:33.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T15:04:33.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T15:04:33.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e24a6d6d7f88f4e-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.731 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.732 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.732 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.733 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.733 DEBUG:boto:Method: GET 2023-10-11T15:04:33.734 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:04:33.734 DEBUG:boto:Data: 2023-10-11T15:04:33.734 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.735 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.735 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.736 DEBUG:boto:Params: {} 2023-10-11T15:04:33.736 DEBUG:boto:Token: None 2023-10-11T15:04:33.737 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.737 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5cff76339b8d911-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.786 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.787 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.788 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.789 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.789 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.789 DEBUG:boto:Method: GET 2023-10-11T15:04:33.790 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:04:33.790 DEBUG:boto:Data: 2023-10-11T15:04:33.791 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.792 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.792 DEBUG:boto:Params: {} 2023-10-11T15:04:33.793 DEBUG:boto:Token: None 2023-10-11T15:04:33.793 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95e11a07839a958-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.797 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.798 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.798 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.799 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.799 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:04:33.800 DEBUG:boto:Data: 2023-10-11T15:04:33.800 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.801 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.801 DEBUG:boto:Params: {} 2023-10-11T15:04:33.802 DEBUG:boto:Token: None 2023-10-11T15:04:33.802 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b 2023-10-11T15:04:33.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs= 2023-10-11T15:04:33.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs=', 'Content-Length': '0'} 2023-10-11T15:04:33.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f743fab14309295-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.806 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.806 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.807 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.807 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:04:33.808 DEBUG:boto:Data: 2023-10-11T15:04:33.808 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.809 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.809 DEBUG:boto:Params: {} 2023-10-11T15:04:33.810 DEBUG:boto:Token: None 2023-10-11T15:04:33.810 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b 2023-10-11T15:04:33.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs= 2023-10-11T15:04:33.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs=', 'Content-Length': '0'} 2023-10-11T15:04:33.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bdcf273fdf1cc89-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:04:33.814 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.815 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.815 DEBUG:boto:Method: GET 2023-10-11T15:04:33.816 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:04:33.816 DEBUG:boto:Data: 2023-10-11T15:04:33.816 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.817 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.818 DEBUG:boto:Params: {} 2023-10-11T15:04:33.818 DEBUG:boto:Token: None 2023-10-11T15:04:33.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b 2023-10-11T15:04:33.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0= 2023-10-11T15:04:33.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0=', 'Content-Length': '0'} 2023-10-11T15:04:33.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c169e58519b3c9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.822 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:04:33.823 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:04:33.823 DEBUG:boto:Method: GET 2023-10-11T15:04:33.824 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:04:33.824 DEBUG:boto:Data: 2023-10-11T15:04:33.825 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.826 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.826 DEBUG:boto:Params: {} 2023-10-11T15:04:33.827 DEBUG:boto:Token: None 2023-10-11T15:04:33.827 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/b 2023-10-11T15:04:33.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0= 2023-10-11T15:04:33.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0=', 'Content-Length': '0'} 2023-10-11T15:04:33.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f84583db586461-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.831 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.831 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.832 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.832 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.832 DEBUG:boto:Method: GET 2023-10-11T15:04:33.833 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:04:33.833 DEBUG:boto:Data: 2023-10-11T15:04:33.834 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.834 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.835 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.835 DEBUG:boto:Params: {} 2023-10-11T15:04:33.836 DEBUG:boto:Token: None 2023-10-11T15:04:33.836 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f404ee8511348b-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.840 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.841 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.841 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.842 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.842 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:04:33.843 DEBUG:boto:Method: GET 2023-10-11T15:04:33.843 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:04:33.844 DEBUG:boto:Data: 2023-10-11T15:04:33.844 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.845 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.845 DEBUG:boto:Params: {} 2023-10-11T15:04:33.846 DEBUG:boto:Token: None 2023-10-11T15:04:33.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5e0f809f22f1d8-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.850 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:04:33.851 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.852 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.852 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.852 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.853 DEBUG:boto:Method: GET 2023-10-11T15:04:33.853 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.854 DEBUG:boto:Data: 2023-10-11T15:04:33.854 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.855 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.855 DEBUG:boto:Params: {} 2023-10-11T15:04:33.856 DEBUG:boto:Token: None 2023-10-11T15:04:33.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T15:04:33.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T15:04:33.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8666d7af7a5050-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.860 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.861 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.861 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.861 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.862 DEBUG:boto:Method: GET 2023-10-11T15:04:33.862 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.863 DEBUG:boto:Data: 2023-10-11T15:04:33.863 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.864 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.864 DEBUG:boto:Params: {} 2023-10-11T15:04:33.865 DEBUG:boto:Token: None 2023-10-11T15:04:33.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T15:04:33.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T15:04:33.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c28849b3f94011c-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.869 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.870 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.870 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.870 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.871 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.871 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.872 DEBUG:boto:Data: 2023-10-11T15:04:33.872 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.873 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.873 DEBUG:boto:Params: {} 2023-10-11T15:04:33.874 DEBUG:boto:Token: None 2023-10-11T15:04:33.874 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas= 2023-10-11T15:04:33.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas=', 'Content-Length': '0'} 2023-10-11T15:04:33.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075852bd51deead0-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.877 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.878 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.878 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.879 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.879 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.880 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.880 DEBUG:boto:Data: 2023-10-11T15:04:33.880 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.881 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.882 DEBUG:boto:Params: {} 2023-10-11T15:04:33.882 DEBUG:boto:Token: None 2023-10-11T15:04:33.883 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas= 2023-10-11T15:04:33.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas=', 'Content-Length': '0'} 2023-10-11T15:04:33.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5249bf8b8a24ce-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:04:33.886 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.887 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.887 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.888 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.888 DEBUG:boto:Method: GET 2023-10-11T15:04:33.888 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.889 DEBUG:boto:Data: 2023-10-11T15:04:33.889 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.890 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.890 DEBUG:boto:Params: {} 2023-10-11T15:04:33.891 DEBUG:boto:Token: None 2023-10-11T15:04:33.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T15:04:33.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T15:04:33.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004fe2a720130e1b9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.895 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.895 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.896 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.896 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:04:33.897 DEBUG:boto:Method: GET 2023-10-11T15:04:33.897 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.897 DEBUG:boto:Data: 2023-10-11T15:04:33.898 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.899 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.899 DEBUG:boto:Params: {} 2023-10-11T15:04:33.900 DEBUG:boto:Token: None 2023-10-11T15:04:33.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:04:33.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T15:04:33.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T15:04:33.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9141478267e5d9-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.904 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.904 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.904 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:04:33.905 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:04:33.905 DEBUG:boto:Method: GET 2023-10-11T15:04:33.906 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:04:33.906 DEBUG:boto:Data: 2023-10-11T15:04:33.907 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.908 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.908 DEBUG:boto:Params: {} 2023-10-11T15:04:33.908 DEBUG:boto:Token: None 2023-10-11T15:04:33.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccbddc5056c45e8-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.954 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.955 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:33.956 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:33.956 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:04:33.957 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:04:33.957 DEBUG:boto:Method: GET 2023-10-11T15:04:33.958 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:04:33.959 DEBUG:boto:Data: 2023-10-11T15:04:33.959 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.960 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.961 DEBUG:boto:Params: {} 2023-10-11T15:04:33.962 DEBUG:boto:Token: None 2023-10-11T15:04:33.963 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/ 2023-10-11T15:04:33.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T15:04:33.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T15:04:33.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006213eac78ae7a103-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.971 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:33.972 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.973 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.974 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.974 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:04:33.975 DEBUG:boto:Data: 2023-10-11T15:04:33.976 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.977 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.977 DEBUG:boto:Params: {} 2023-10-11T15:04:33.978 DEBUG:boto:Token: None 2023-10-11T15:04:33.978 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c 2023-10-11T15:04:33.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og= 2023-10-11T15:04:33.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og=', 'Content-Length': '0'} 2023-10-11T15:04:33.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd638f435135c78-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.982 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.983 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.983 DEBUG:boto:Method: HEAD 2023-10-11T15:04:33.983 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:04:33.984 DEBUG:boto:Data: 2023-10-11T15:04:33.984 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:33.985 DEBUG:boto:Port: 8000 2023-10-11T15:04:33.986 DEBUG:boto:Params: {} 2023-10-11T15:04:33.986 DEBUG:boto:Token: None 2023-10-11T15:04:33.987 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c 2023-10-11T15:04:33.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og= 2023-10-11T15:04:33.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og=', 'Content-Length': '0'} 2023-10-11T15:04:33.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea061f48c7cae6a3-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:04:33.991 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:33.991 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:33.992 DEBUG:boto:Method: GET 2023-10-11T15:04:33.992 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:04:33.993 DEBUG:boto:Data: 2023-10-11T15:04:33.993 DEBUG:boto:Headers: {} 2023-10-11T15:04:33.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:33.994 DEBUG:boto:Port: 8001 2023-10-11T15:04:33.995 DEBUG:boto:Params: {} 2023-10-11T15:04:33.995 DEBUG:boto:Token: None 2023-10-11T15:04:33.996 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:33 GMT /dghqvd-55/c 2023-10-11T15:04:33.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dYi3sVztg4wx2IZ9PUtX6sWq8Z4= 2023-10-11T15:04:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dYi3sVztg4wx2IZ9PUtX6sWq8Z4=', 'Content-Length': '0'} 2023-10-11T15:04:33.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3809ad6b26e1cf-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:33.999 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:04:34.000 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:04:34.000 DEBUG:boto:Method: GET 2023-10-11T15:04:34.001 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:04:34.001 DEBUG:boto:Data: 2023-10-11T15:04:34.002 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.003 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.003 DEBUG:boto:Params: {} 2023-10-11T15:04:34.004 DEBUG:boto:Token: None 2023-10-11T15:04:34.004 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/c 2023-10-11T15:04:34.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E1kJzRxQPGulWefWYuXRl7IKP/w= 2023-10-11T15:04:34.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E1kJzRxQPGulWefWYuXRl7IKP/w=', 'Content-Length': '0'} 2023-10-11T15:04:34.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448293861f3d839a-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.008 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.009 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.009 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:04:34.010 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:04:34.010 DEBUG:boto:Method: GET 2023-10-11T15:04:34.011 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:04:34.011 DEBUG:boto:Data: 2023-10-11T15:04:34.011 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.012 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.013 DEBUG:boto:Params: {} 2023-10-11T15:04:34.013 DEBUG:boto:Token: None 2023-10-11T15:04:34.014 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002972b829af69f87-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.018 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.019 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.019 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.020 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:04:34.020 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:04:34.021 DEBUG:boto:Method: GET 2023-10-11T15:04:34.021 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:04:34.021 DEBUG:boto:Data: 2023-10-11T15:04:34.022 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.023 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.023 DEBUG:boto:Params: {} 2023-10-11T15:04:34.024 DEBUG:boto:Token: None 2023-10-11T15:04:34.024 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea627ff42816a92-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.028 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:04:34.030 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.030 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.031 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.031 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.032 DEBUG:boto:Method: GET 2023-10-11T15:04:34.032 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.033 DEBUG:boto:Data: 2023-10-11T15:04:34.033 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.034 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.034 DEBUG:boto:Params: {} 2023-10-11T15:04:34.035 DEBUG:boto:Token: None 2023-10-11T15:04:34.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T15:04:34.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T15:04:34.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cf862ac9a72faa-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.039 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.039 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.040 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.040 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.041 DEBUG:boto:Method: GET 2023-10-11T15:04:34.041 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.042 DEBUG:boto:Data: 2023-10-11T15:04:34.042 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.044 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.044 DEBUG:boto:Params: {} 2023-10-11T15:04:34.046 DEBUG:boto:Token: None 2023-10-11T15:04:34.046 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.047 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T15:04:34.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T15:04:34.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f363a32dd7518116-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.051 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.051 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.051 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.052 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.052 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.053 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.053 DEBUG:boto:Data: 2023-10-11T15:04:34.054 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.055 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.055 DEBUG:boto:Params: {} 2023-10-11T15:04:34.056 DEBUG:boto:Token: None 2023-10-11T15:04:34.056 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc= 2023-10-11T15:04:34.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc=', 'Content-Length': '0'} 2023-10-11T15:04:34.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024505b4a1df1e9ef-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.059 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.060 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.060 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.061 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.061 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.062 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.062 DEBUG:boto:Data: 2023-10-11T15:04:34.063 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.064 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.064 DEBUG:boto:Params: {} 2023-10-11T15:04:34.065 DEBUG:boto:Token: None 2023-10-11T15:04:34.065 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc= 2023-10-11T15:04:34.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc=', 'Content-Length': '0'} 2023-10-11T15:04:34.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be18b40a17483ab7-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:04:34.069 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.069 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.070 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.070 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.071 DEBUG:boto:Method: GET 2023-10-11T15:04:34.071 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.072 DEBUG:boto:Data: 2023-10-11T15:04:34.072 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.073 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.073 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.073 DEBUG:boto:Params: {} 2023-10-11T15:04:34.074 DEBUG:boto:Token: None 2023-10-11T15:04:34.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T15:04:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T15:04:34.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe87b5e90554a791-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.122 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.123 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.123 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.124 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:04:34.124 DEBUG:boto:Method: GET 2023-10-11T15:04:34.124 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.125 DEBUG:boto:Data: 2023-10-11T15:04:34.125 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.126 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.127 DEBUG:boto:Params: {} 2023-10-11T15:04:34.127 DEBUG:boto:Token: None 2023-10-11T15:04:34.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:04:34.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T15:04:34.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T15:04:34.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013f518bc30837c0-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.131 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.132 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.132 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.133 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.133 DEBUG:boto:Method: GET 2023-10-11T15:04:34.134 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:04:34.134 DEBUG:boto:Data: 2023-10-11T15:04:34.135 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.135 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.136 DEBUG:boto:Params: {} 2023-10-11T15:04:34.136 DEBUG:boto:Token: None 2023-10-11T15:04:34.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bdc3777de0b20f9-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.141 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.142 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.142 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.143 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.144 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.144 DEBUG:boto:Method: GET 2023-10-11T15:04:34.145 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:04:34.146 DEBUG:boto:Data: 2023-10-11T15:04:34.147 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.148 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.149 DEBUG:boto:Params: {} 2023-10-11T15:04:34.149 DEBUG:boto:Token: None 2023-10-11T15:04:34.150 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd2160d80037f9bf-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.194 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.195 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.196 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.196 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.197 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:04:34.197 DEBUG:boto:Data: 2023-10-11T15:04:34.197 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.198 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.199 DEBUG:boto:Params: {} 2023-10-11T15:04:34.199 DEBUG:boto:Token: None 2023-10-11T15:04:34.200 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d 2023-10-11T15:04:34.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4= 2023-10-11T15:04:34.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4=', 'Content-Length': '0'} 2023-10-11T15:04:34.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc0ee751fcb1a86-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.204 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.204 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.205 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.205 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:04:34.206 DEBUG:boto:Data: 2023-10-11T15:04:34.206 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.207 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.207 DEBUG:boto:Params: {} 2023-10-11T15:04:34.208 DEBUG:boto:Token: None 2023-10-11T15:04:34.208 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d 2023-10-11T15:04:34.209 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4= 2023-10-11T15:04:34.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4=', 'Content-Length': '0'} 2023-10-11T15:04:34.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31e8265838efa78-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:04:34.212 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.213 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.213 DEBUG:boto:Method: GET 2023-10-11T15:04:34.214 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:04:34.214 DEBUG:boto:Data: 2023-10-11T15:04:34.215 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.215 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.216 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.216 DEBUG:boto:Params: {} 2023-10-11T15:04:34.217 DEBUG:boto:Token: None 2023-10-11T15:04:34.217 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d 2023-10-11T15:04:34.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno= 2023-10-11T15:04:34.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno=', 'Content-Length': '0'} 2023-10-11T15:04:34.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71e961bf5aa70b4-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.221 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:04:34.221 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:04:34.222 DEBUG:boto:Method: GET 2023-10-11T15:04:34.222 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:04:34.223 DEBUG:boto:Data: 2023-10-11T15:04:34.223 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.224 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.224 DEBUG:boto:Params: {} 2023-10-11T15:04:34.225 DEBUG:boto:Token: None 2023-10-11T15:04:34.225 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/d 2023-10-11T15:04:34.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno= 2023-10-11T15:04:34.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno=', 'Content-Length': '0'} 2023-10-11T15:04:34.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77d8b7d28ae021f-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.229 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.229 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.230 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.230 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.231 DEBUG:boto:Method: GET 2023-10-11T15:04:34.231 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:04:34.232 DEBUG:boto:Data: 2023-10-11T15:04:34.232 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:04:34.233 DEBUG:boto:Port: 8001 2023-10-11T15:04:34.233 DEBUG:boto:Params: {} 2023-10-11T15:04:34.234 DEBUG:boto:Token: None 2023-10-11T15:04:34.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa8f37ce16f747af-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.239 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.240 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.240 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.241 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.241 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:04:34.241 DEBUG:boto:Method: GET 2023-10-11T15:04:34.242 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:04:34.242 DEBUG:boto:Data: 2023-10-11T15:04:34.243 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.244 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.244 DEBUG:boto:Params: {} 2023-10-11T15:04:34.245 DEBUG:boto:Token: None 2023-10-11T15:04:34.245 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T15:04:34.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T15:04:34.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa42a38d512e0a5-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.249 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:04:34.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:04:34.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.251 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:04:34.304 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:34.305 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:34.348 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:04:34.349 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.349 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding": "None", 2023-10-11T15:04:34.349 INFO:teuthology.orchestra.run.smithi158.stdout: "current_index": { 2023-10-11T15:04:34.349 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:04:34.349 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.350 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "Normal", 2023-10-11T15:04:34.350 INFO:teuthology.orchestra.run.smithi158.stdout: "normal": { 2023-10-11T15:04:34.350 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 11, 2023-10-11T15:04:34.350 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:04:34.351 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.351 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.351 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.351 INFO:teuthology.orchestra.run.smithi158.stdout: "logs": [ 2023-10-11T15:04:34.351 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.352 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:04:34.352 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.352 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:04:34.352 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:04:34.352 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:04:34.353 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.353 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 11, 2023-10-11T15:04:34.353 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:04:34.353 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.353 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.354 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.354 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.354 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.354 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.355 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:04:34.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:04:34.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:04:34.357 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:04:34.409 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:34.409 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:34.450 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:04:34.451 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.451 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "1#00000000001.16.6", 2023-10-11T15:04:34.451 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11901769168167861916", 2023-10-11T15:04:34.451 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:04:34.451 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "a", 2023-10-11T15:04:34.452 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:04:34.452 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:04:34.452 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:04:34.452 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:52.775214532Z", 2023-10-11T15:04:34.452 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:04:34.453 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:04:34.453 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:04:34.453 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.453 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:04:34.454 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:04:34.454 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:04:34.454 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:04:34.454 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:04:34.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.455 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T15:04:34.455 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.455 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.455 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.456 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.456 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "5#00000000001.23.6", 2023-10-11T15:04:34.456 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.13555810181077397004", 2023-10-11T15:04:34.456 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:04:34.456 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "d", 2023-10-11T15:04:34.457 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:04:34.457 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:04:34.457 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:04:34.457 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:52.871975063Z", 2023-10-11T15:04:34.458 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:04:34.458 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:04:34.458 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:04:34.458 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.458 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:04:34.459 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:04:34.459 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:04:34.459 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:04:34.459 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:04:34.460 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.460 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T15:04:34.460 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.460 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.461 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.461 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "8#00000000001.55.6", 2023-10-11T15:04:34.461 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12455247144329015868", 2023-10-11T15:04:34.461 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:04:34.462 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "c", 2023-10-11T15:04:34.462 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:04:34.462 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:04:34.462 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:04:34.462 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:52.853505798Z", 2023-10-11T15:04:34.463 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:04:34.463 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:04:34.463 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:04:34.463 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.464 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:04:34.464 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:04:34.464 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:04:34.464 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:04:34.464 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:04:34.465 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.465 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T15:04:34.465 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.465 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.465 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.466 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.466 INFO:teuthology.orchestra.run.smithi158.stdout: "op_id": "10#00000000001.24.6", 2023-10-11T15:04:34.466 INFO:teuthology.orchestra.run.smithi158.stdout: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15734065776330899090", 2023-10-11T15:04:34.466 INFO:teuthology.orchestra.run.smithi158.stdout: "op": "write", 2023-10-11T15:04:34.467 INFO:teuthology.orchestra.run.smithi158.stdout: "object": "b", 2023-10-11T15:04:34.467 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:04:34.467 INFO:teuthology.orchestra.run.smithi158.stdout: "state": "complete", 2023-10-11T15:04:34.467 INFO:teuthology.orchestra.run.smithi158.stdout: "index_ver": 1, 2023-10-11T15:04:34.467 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:03:52.795448596Z", 2023-10-11T15:04:34.468 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:04:34.468 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:04:34.468 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:04:34.468 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: "bilog_flags": 0, 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned": false, 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "", 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "", 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: "zones_trace": [ 2023-10-11T15:04:34.469 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.470 INFO:teuthology.orchestra.run.smithi158.stdout: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T15:04:34.470 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.470 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.470 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.471 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:04:34.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.473 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:04:34.523 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:34.524 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:34.561 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:04:34.562 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-55 2023-10-11T15:04:34.565 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 4 2023-10-11T15:04:34.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:04:34.602 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:04:34.648 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:34.648 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:34.689 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:04:34.689 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.690 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding": "None", 2023-10-11T15:04:34.690 INFO:teuthology.orchestra.run.smithi158.stdout: "current_index": { 2023-10-11T15:04:34.690 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:04:34.691 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.691 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "Normal", 2023-10-11T15:04:34.691 INFO:teuthology.orchestra.run.smithi158.stdout: "normal": { 2023-10-11T15:04:34.692 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:04:34.692 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:04:34.692 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.693 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.693 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.693 INFO:teuthology.orchestra.run.smithi158.stdout: "logs": [ 2023-10-11T15:04:34.694 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.694 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:04:34.694 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.695 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:04:34.695 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:04:34.695 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:04:34.695 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.696 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 11, 2023-10-11T15:04:34.696 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:04:34.696 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.697 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.697 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.697 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:04:34.698 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:04:34.698 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:04:34.698 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.698 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:04:34.699 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:04:34.699 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:04:34.699 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:04:34.700 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:04:34.700 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:04:34.700 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.700 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.701 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:04:34.702 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:04:34.702 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:04:34.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:04:34.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:04:34.705 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:04:34.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:34.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:34.795 INFO:teuthology.orchestra.run.smithi158.stdout:[] 2023-10-11T15:04:34.797 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.798 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.798 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.799 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:34.799 DEBUG:boto:Data: 2023-10-11T15:04:34.800 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.800 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.801 DEBUG:boto:Params: {} 2023-10-11T15:04:34.801 DEBUG:boto:Token: None 2023-10-11T15:04:34.802 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T15:04:34.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T15:04:34.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2d00e34c76c50fd-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.806 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:04:34.807 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:04:34.807 DEBUG:boto:Method: PUT 2023-10-11T15:04:34.807 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:04:34.808 DEBUG:boto:Data: 2023-10-11T15:04:34.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:04:34.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.809 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.810 DEBUG:boto:Params: {} 2023-10-11T15:04:34.810 DEBUG:boto:Token: None 2023-10-11T15:04:34.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/e 2023-10-11T15:04:34.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xbmEBWtBj+ELuuP4GyZV5IJN1f4= 2023-10-11T15:04:34.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xbmEBWtBj+ELuuP4GyZV5IJN1f4='} 2023-10-11T15:04:34.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000015a3a919010b96-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.817 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.818 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.818 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.819 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:34.819 DEBUG:boto:Data: 2023-10-11T15:04:34.819 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.820 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.821 DEBUG:boto:Params: {} 2023-10-11T15:04:34.821 DEBUG:boto:Token: None 2023-10-11T15:04:34.822 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T15:04:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T15:04:34.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000601058d522305759-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.825 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:04:34.826 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:04:34.826 DEBUG:boto:Method: PUT 2023-10-11T15:04:34.826 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:04:34.827 DEBUG:boto:Data: 2023-10-11T15:04:34.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:04:34.828 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.828 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.828 DEBUG:boto:Params: {} 2023-10-11T15:04:34.829 DEBUG:boto:Token: None 2023-10-11T15:04:34.829 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/f 2023-10-11T15:04:34.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e8D0ut42Gfm1sLQvmx8p5j8KYBc= 2023-10-11T15:04:34.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e8D0ut42Gfm1sLQvmx8p5j8KYBc='} 2023-10-11T15:04:34.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005d107a9a5583b1c-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.879 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.880 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.880 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.880 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:34.881 DEBUG:boto:Data: 2023-10-11T15:04:34.881 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.882 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.882 DEBUG:boto:Params: {} 2023-10-11T15:04:34.883 DEBUG:boto:Token: None 2023-10-11T15:04:34.883 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T15:04:34.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T15:04:34.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000017dd7c3d3e872-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.887 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:04:34.887 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:04:34.888 DEBUG:boto:Method: PUT 2023-10-11T15:04:34.889 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:04:34.889 DEBUG:boto:Data: 2023-10-11T15:04:34.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:04:34.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.892 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.893 DEBUG:boto:Params: {} 2023-10-11T15:04:34.893 DEBUG:boto:Token: None 2023-10-11T15:04:34.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/g 2023-10-11T15:04:34.895 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8sqQt1wgI8l7y78RG9YU+vtsQfM= 2023-10-11T15:04:34.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8sqQt1wgI8l7y78RG9YU+vtsQfM='} 2023-10-11T15:04:34.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa7e3df68236dfd4-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.902 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:04:34.903 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:04:34.903 DEBUG:boto:Method: HEAD 2023-10-11T15:04:34.904 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:04:34.904 DEBUG:boto:Data: 2023-10-11T15:04:34.904 DEBUG:boto:Headers: {} 2023-10-11T15:04:34.905 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.905 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.905 DEBUG:boto:Params: {} 2023-10-11T15:04:34.906 DEBUG:boto:Token: None 2023-10-11T15:04:34.906 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/ 2023-10-11T15:04:34.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T15:04:34.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T15:04:34.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027b852bf4da0543f-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.909 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:04:34.910 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:04:34.910 DEBUG:boto:Method: PUT 2023-10-11T15:04:34.911 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:04:34.911 DEBUG:boto:Data: 2023-10-11T15:04:34.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:04:34.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:04:34.912 DEBUG:boto:Port: 8000 2023-10-11T15:04:34.913 DEBUG:boto:Params: {} 2023-10-11T15:04:34.913 DEBUG:boto:Token: None 2023-10-11T15:04:34.914 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:04:34 GMT /dghqvd-55/h 2023-10-11T15:04:34.914 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ERhjfAG1uYi+23MhwwF9wHDow1M= 2023-10-11T15:04:34.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ERhjfAG1uYi+23MhwwF9wHDow1M='} 2023-10-11T15:04:34.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ff44f7181481872-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:04:34.959 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T15:04:34.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:04:34.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:04:34.960 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:04:35.019 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:04:35.019 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:04:35.052 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:35.049+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T15:04:40.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:40.050+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T15:04:45.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:45.050+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T15:04:50.054 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:50.052+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:04:50.055 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.51.6, ] 2023-10-11T15:04:50.055 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:04:55.055 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:04:55.052+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:04:55.056 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.51.6, ] 2023-10-11T15:04:55.056 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:00.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:00.053+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:00.057 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.51.6, ] 2023-10-11T15:05:00.057 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:05.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:05.054+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:05.057 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.51.6, ] 2023-10-11T15:05:05.057 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:10.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:10.055+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:10.058 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:10.058 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:15.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:15.055+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:15.059 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:15.059 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:20.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:20.057+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:20.060 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:20.060 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:25.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:25.058+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:25.060 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:25.061 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:30.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:30.059+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:30.061 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:30.061 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:35.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:35.060+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:35.062 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:35.062 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:40.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:40.060+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:40.063 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:40.063 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:45.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:45.061+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:45.063 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:45.064 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:50.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:50.062+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:50.064 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:50.064 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:05:55.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:05:55.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:05:55.065 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:05:55.065 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:00.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:00.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:00.066 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:00.066 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:05.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:05.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:05.066 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:05.066 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:10.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:10.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:10.066 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:10.067 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:15.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:15.064+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:15.067 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:15.067 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:20.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:20.065+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:20.068 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:20.068 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:25.065+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:25.069 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:25.069 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:30.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:30.066+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:30.070 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:30.070 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:35.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:35.067+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:35.070 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:35.070 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:40.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:40.068+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:40.071 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:40.071 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:45.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:45.069+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:45.072 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T15:06:45.072 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:50.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:50.071+0000 7fc78a5aa780 1 bucket sync caught up with source: 2023-10-11T15:06:50.073 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:50.073 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T15:06:50.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:50.071+0000 7fc78a5aa780 0 bucket checkpoint complete 2023-10-11T15:06:50.090 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T15:06:50.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:06:50.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:06:50.091 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:06:50.157 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:06:50.157 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:06:50.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:50.192+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:50.194 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T15:06:50.194 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:06:55.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:06:55.193+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T15:06:55.195 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T15:06:55.196 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:00.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:00.195+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T15:07:00.197 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T15:07:00.197 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:05.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:05.196+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T15:07:05.198 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T15:07:05.198 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:10.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:10.197+0000 7f41b709e780 1 bucket sync caught up with source: 2023-10-11T15:07:10.199 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:10.200 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:10.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:10.197+0000 7f41b709e780 0 bucket checkpoint complete 2023-10-11T15:07:10.216 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:07:10.216 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.217 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.217 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.218 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:07:10.218 DEBUG:boto:Data: 2023-10-11T15:07:10.219 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.219 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.220 DEBUG:boto:Params: {} 2023-10-11T15:07:10.220 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:07:10.221 DEBUG:boto:Token: None 2023-10-11T15:07:10.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw= 2023-10-11T15:07:10.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw=', 'Content-Length': '0'} 2023-10-11T15:07:10.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000620e203592925cc9-006526ba1e-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.227 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.228 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.228 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.229 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:07:10.229 DEBUG:boto:Data: 2023-10-11T15:07:10.229 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.230 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.231 DEBUG:boto:Params: {} 2023-10-11T15:07:10.231 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:07:10.232 DEBUG:boto:Token: None 2023-10-11T15:07:10.232 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw= 2023-10-11T15:07:10.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw=', 'Content-Length': '0'} 2023-10-11T15:07:10.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2745bfb80b8039e-006526ba1e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.237 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:07:10.237 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.238 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.238 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.238 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.239 DEBUG:boto:Method: GET 2023-10-11T15:07:10.239 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.240 DEBUG:boto:Data: 2023-10-11T15:07:10.240 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.241 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.241 DEBUG:boto:Params: {} 2023-10-11T15:07:10.242 DEBUG:boto:Token: None 2023-10-11T15:07:10.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/?versions 2023-10-11T15:07:10.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T15:07:10.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T15:07:10.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f72ea9fc54a10ef-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.246 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.247 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:07:10.248 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:07:10.248 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:07:10.249 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:07:10.249 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:07:10.249 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:07:10.250 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:07:10.250 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:07:10.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:07:10.251 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.251 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.252 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.252 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.253 DEBUG:boto:Method: GET 2023-10-11T15:07:10.253 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.253 DEBUG:boto:Data: 2023-10-11T15:07:10.254 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.255 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.255 DEBUG:boto:Params: {} 2023-10-11T15:07:10.256 DEBUG:boto:Token: None 2023-10-11T15:07:10.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/?versions 2023-10-11T15:07:10.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T15:07:10.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T15:07:10.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec86e5aae688a92e-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.259 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.260 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:07:10.261 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:07:10.261 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:07:10.261 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:07:10.262 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:07:10.262 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:07:10.262 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:07:10.263 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:07:10.263 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.264 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.264 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.264 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.265 DEBUG:boto:Method: GET 2023-10-11T15:07:10.265 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.266 DEBUG:boto:Data: 2023-10-11T15:07:10.266 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.267 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.267 DEBUG:boto:Params: {} 2023-10-11T15:07:10.268 DEBUG:boto:Token: None 2023-10-11T15:07:10.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/?versions 2023-10-11T15:07:10.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T15:07:10.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T15:07:10.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009141886205bdf2dd-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.272 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.273 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.273 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.274 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.274 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.275 DEBUG:boto:Method: GET 2023-10-11T15:07:10.275 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:10.275 DEBUG:boto:Data: 2023-10-11T15:07:10.276 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.277 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.277 DEBUG:boto:Params: {} 2023-10-11T15:07:10.278 DEBUG:boto:Token: None 2023-10-11T15:07:10.278 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/?versions 2023-10-11T15:07:10.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T15:07:10.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T15:07:10.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a0642635a071a3-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.282 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:10.283 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.283 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.284 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.284 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.285 DEBUG:boto:Method: GET 2023-10-11T15:07:10.285 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.285 DEBUG:boto:Data: 2023-10-11T15:07:10.286 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.287 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.287 DEBUG:boto:Params: {} 2023-10-11T15:07:10.288 DEBUG:boto:Token: None 2023-10-11T15:07:10.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T15:07:10.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T15:07:10.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038973e0f407a9c53-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.292 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.292 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.293 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.293 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.294 DEBUG:boto:Method: GET 2023-10-11T15:07:10.294 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.295 DEBUG:boto:Data: 2023-10-11T15:07:10.295 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.296 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.296 DEBUG:boto:Params: {} 2023-10-11T15:07:10.297 DEBUG:boto:Token: None 2023-10-11T15:07:10.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T15:07:10.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T15:07:10.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ff4681f2e1ae6c-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.301 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.301 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.302 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.302 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.303 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.303 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.303 DEBUG:boto:Data: 2023-10-11T15:07:10.304 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.304 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.305 DEBUG:boto:Params: {} 2023-10-11T15:07:10.305 DEBUG:boto:Token: None 2023-10-11T15:07:10.306 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous= 2023-10-11T15:07:10.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous=', 'Content-Length': '0'} 2023-10-11T15:07:10.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a83d87e221c0a6-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.309 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.309 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.310 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.310 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.311 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.311 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.311 DEBUG:boto:Data: 2023-10-11T15:07:10.312 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.313 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.313 DEBUG:boto:Params: {} 2023-10-11T15:07:10.313 DEBUG:boto:Token: None 2023-10-11T15:07:10.314 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous= 2023-10-11T15:07:10.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous=', 'Content-Length': '0'} 2023-10-11T15:07:10.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a3260567a023bc-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:10.318 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.319 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.319 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.319 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.320 DEBUG:boto:Method: GET 2023-10-11T15:07:10.320 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.321 DEBUG:boto:Data: 2023-10-11T15:07:10.321 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.322 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.322 DEBUG:boto:Params: {} 2023-10-11T15:07:10.323 DEBUG:boto:Token: None 2023-10-11T15:07:10.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T15:07:10.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T15:07:10.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f3dac7e9745154-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.366 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.367 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.367 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.368 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:10.368 DEBUG:boto:Method: GET 2023-10-11T15:07:10.368 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.369 DEBUG:boto:Data: 2023-10-11T15:07:10.369 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.370 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.371 DEBUG:boto:Params: {} 2023-10-11T15:07:10.371 DEBUG:boto:Token: None 2023-10-11T15:07:10.372 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:10.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T15:07:10.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T15:07:10.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96be835112152a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.377 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.377 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.378 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.378 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.379 DEBUG:boto:Method: GET 2023-10-11T15:07:10.379 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:10.379 DEBUG:boto:Data: 2023-10-11T15:07:10.380 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.380 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.381 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.381 DEBUG:boto:Params: {} 2023-10-11T15:07:10.382 DEBUG:boto:Token: None 2023-10-11T15:07:10.382 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e78adcd100a6c8-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.386 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.387 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.387 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.388 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.388 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.388 DEBUG:boto:Method: GET 2023-10-11T15:07:10.389 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:10.389 DEBUG:boto:Data: 2023-10-11T15:07:10.390 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.390 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.391 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.391 DEBUG:boto:Params: {} 2023-10-11T15:07:10.391 DEBUG:boto:Token: None 2023-10-11T15:07:10.392 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9d78915839094a-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.396 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.396 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.397 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.397 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.397 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:10.398 DEBUG:boto:Data: 2023-10-11T15:07:10.398 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.399 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.399 DEBUG:boto:Params: {} 2023-10-11T15:07:10.400 DEBUG:boto:Token: None 2023-10-11T15:07:10.400 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a 2023-10-11T15:07:10.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc= 2023-10-11T15:07:10.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc=', 'Content-Length': '0'} 2023-10-11T15:07:10.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aac62962c48427c-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.404 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.404 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.405 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.405 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:10.405 DEBUG:boto:Data: 2023-10-11T15:07:10.406 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.407 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.407 DEBUG:boto:Params: {} 2023-10-11T15:07:10.408 DEBUG:boto:Token: None 2023-10-11T15:07:10.408 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a 2023-10-11T15:07:10.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc= 2023-10-11T15:07:10.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc=', 'Content-Length': '0'} 2023-10-11T15:07:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39f94ced54a5a65-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:10.412 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.412 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.412 DEBUG:boto:Method: GET 2023-10-11T15:07:10.413 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:10.413 DEBUG:boto:Data: 2023-10-11T15:07:10.414 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.415 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.415 DEBUG:boto:Params: {} 2023-10-11T15:07:10.415 DEBUG:boto:Token: None 2023-10-11T15:07:10.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a 2023-10-11T15:07:10.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc= 2023-10-11T15:07:10.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc=', 'Content-Length': '0'} 2023-10-11T15:07:10.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4ac3eb4fe08005-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.419 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:10.420 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:10.420 DEBUG:boto:Method: GET 2023-10-11T15:07:10.421 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:10.421 DEBUG:boto:Data: 2023-10-11T15:07:10.422 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.423 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.423 DEBUG:boto:Params: {} 2023-10-11T15:07:10.423 DEBUG:boto:Token: None 2023-10-11T15:07:10.424 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/a 2023-10-11T15:07:10.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc= 2023-10-11T15:07:10.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc=', 'Content-Length': '0'} 2023-10-11T15:07:10.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96f8706ec71aac-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.474 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.475 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.475 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.476 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.476 DEBUG:boto:Method: GET 2023-10-11T15:07:10.477 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:10.477 DEBUG:boto:Data: 2023-10-11T15:07:10.477 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.478 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.479 DEBUG:boto:Params: {} 2023-10-11T15:07:10.479 DEBUG:boto:Token: None 2023-10-11T15:07:10.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3953552236d14af-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.483 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.484 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.484 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.485 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.485 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:10.486 DEBUG:boto:Method: GET 2023-10-11T15:07:10.486 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:10.487 DEBUG:boto:Data: 2023-10-11T15:07:10.487 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.488 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.488 DEBUG:boto:Params: {} 2023-10-11T15:07:10.489 DEBUG:boto:Token: None 2023-10-11T15:07:10.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008538207e3adc329c-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.492 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:10.494 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.494 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.494 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.495 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.495 DEBUG:boto:Method: GET 2023-10-11T15:07:10.496 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.496 DEBUG:boto:Data: 2023-10-11T15:07:10.497 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.497 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.498 DEBUG:boto:Params: {} 2023-10-11T15:07:10.498 DEBUG:boto:Token: None 2023-10-11T15:07:10.499 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T15:07:10.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T15:07:10.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d130c8a6eecadc56-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.542 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.543 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.543 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.544 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.545 DEBUG:boto:Method: GET 2023-10-11T15:07:10.545 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.545 DEBUG:boto:Data: 2023-10-11T15:07:10.546 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.547 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.547 DEBUG:boto:Params: {} 2023-10-11T15:07:10.548 DEBUG:boto:Token: None 2023-10-11T15:07:10.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T15:07:10.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T15:07:10.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021898d2614fde4ae-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.552 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.552 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.553 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.553 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.554 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.554 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.554 DEBUG:boto:Data: 2023-10-11T15:07:10.555 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.556 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.556 DEBUG:boto:Params: {} 2023-10-11T15:07:10.556 DEBUG:boto:Token: None 2023-10-11T15:07:10.557 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw= 2023-10-11T15:07:10.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw=', 'Content-Length': '0'} 2023-10-11T15:07:10.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba458bc7eea48c11-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.560 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.561 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.561 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.562 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.562 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.562 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.563 DEBUG:boto:Data: 2023-10-11T15:07:10.563 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.564 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.564 DEBUG:boto:Params: {} 2023-10-11T15:07:10.565 DEBUG:boto:Token: None 2023-10-11T15:07:10.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw= 2023-10-11T15:07:10.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw=', 'Content-Length': '0'} 2023-10-11T15:07:10.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0414a92fc113e2-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:10.569 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.570 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.570 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.570 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.571 DEBUG:boto:Method: GET 2023-10-11T15:07:10.571 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.572 DEBUG:boto:Data: 2023-10-11T15:07:10.572 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.573 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.573 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.573 DEBUG:boto:Params: {} 2023-10-11T15:07:10.574 DEBUG:boto:Token: None 2023-10-11T15:07:10.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.575 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T15:07:10.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T15:07:10.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c8d5ef678ae0e4-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.578 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.578 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.579 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.579 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:10.580 DEBUG:boto:Method: GET 2023-10-11T15:07:10.580 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.580 DEBUG:boto:Data: 2023-10-11T15:07:10.581 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.582 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.582 DEBUG:boto:Params: {} 2023-10-11T15:07:10.583 DEBUG:boto:Token: None 2023-10-11T15:07:10.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:10.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T15:07:10.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T15:07:10.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab678f818a9922a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.587 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.588 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.588 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.589 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.589 DEBUG:boto:Method: GET 2023-10-11T15:07:10.590 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:10.590 DEBUG:boto:Data: 2023-10-11T15:07:10.591 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.592 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.592 DEBUG:boto:Params: {} 2023-10-11T15:07:10.593 DEBUG:boto:Token: None 2023-10-11T15:07:10.593 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe3dba669478623-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.597 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.598 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.598 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.599 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.599 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.599 DEBUG:boto:Method: GET 2023-10-11T15:07:10.600 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:10.600 DEBUG:boto:Data: 2023-10-11T15:07:10.601 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.602 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.602 DEBUG:boto:Params: {} 2023-10-11T15:07:10.602 DEBUG:boto:Token: None 2023-10-11T15:07:10.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f764250bc201757-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.650 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.651 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.652 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.652 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.652 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:10.653 DEBUG:boto:Data: 2023-10-11T15:07:10.653 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.654 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.654 DEBUG:boto:Params: {} 2023-10-11T15:07:10.655 DEBUG:boto:Token: None 2023-10-11T15:07:10.655 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b 2023-10-11T15:07:10.656 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4= 2023-10-11T15:07:10.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4=', 'Content-Length': '0'} 2023-10-11T15:07:10.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf480253c40fd29-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.659 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.659 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.660 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.660 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:10.660 DEBUG:boto:Data: 2023-10-11T15:07:10.661 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.662 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.662 DEBUG:boto:Params: {} 2023-10-11T15:07:10.663 DEBUG:boto:Token: None 2023-10-11T15:07:10.663 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b 2023-10-11T15:07:10.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4= 2023-10-11T15:07:10.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4=', 'Content-Length': '0'} 2023-10-11T15:07:10.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d480722d2a2e3d-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:10.667 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.667 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.668 DEBUG:boto:Method: GET 2023-10-11T15:07:10.668 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:10.668 DEBUG:boto:Data: 2023-10-11T15:07:10.669 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.669 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.670 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.670 DEBUG:boto:Params: {} 2023-10-11T15:07:10.671 DEBUG:boto:Token: None 2023-10-11T15:07:10.671 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b 2023-10-11T15:07:10.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds= 2023-10-11T15:07:10.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds=', 'Content-Length': '0'} 2023-10-11T15:07:10.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee388329c794224-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.675 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:10.675 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:10.676 DEBUG:boto:Method: GET 2023-10-11T15:07:10.676 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:10.676 DEBUG:boto:Data: 2023-10-11T15:07:10.677 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.678 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.678 DEBUG:boto:Params: {} 2023-10-11T15:07:10.679 DEBUG:boto:Token: None 2023-10-11T15:07:10.679 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/b 2023-10-11T15:07:10.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds= 2023-10-11T15:07:10.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds=', 'Content-Length': '0'} 2023-10-11T15:07:10.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651844b7a42f737b-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.683 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.683 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.683 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.684 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.684 DEBUG:boto:Method: GET 2023-10-11T15:07:10.685 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:10.685 DEBUG:boto:Data: 2023-10-11T15:07:10.685 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.686 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.687 DEBUG:boto:Params: {} 2023-10-11T15:07:10.687 DEBUG:boto:Token: None 2023-10-11T15:07:10.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51297e1bb1b0231-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.691 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.692 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.693 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.693 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.693 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:10.694 DEBUG:boto:Method: GET 2023-10-11T15:07:10.694 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:10.695 DEBUG:boto:Data: 2023-10-11T15:07:10.695 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.696 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.696 DEBUG:boto:Params: {} 2023-10-11T15:07:10.697 DEBUG:boto:Token: None 2023-10-11T15:07:10.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0605f412c9d8413-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.701 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:10.702 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.703 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.703 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.703 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.704 DEBUG:boto:Method: GET 2023-10-11T15:07:10.704 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.705 DEBUG:boto:Data: 2023-10-11T15:07:10.705 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.706 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.706 DEBUG:boto:Params: {} 2023-10-11T15:07:10.707 DEBUG:boto:Token: None 2023-10-11T15:07:10.707 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T15:07:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T15:07:10.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084e06d1340b29454-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.758 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.759 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.760 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.760 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.760 DEBUG:boto:Method: GET 2023-10-11T15:07:10.761 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.761 DEBUG:boto:Data: 2023-10-11T15:07:10.762 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.762 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.763 DEBUG:boto:Params: {} 2023-10-11T15:07:10.763 DEBUG:boto:Token: None 2023-10-11T15:07:10.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T15:07:10.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T15:07:10.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065786a95f35b41af-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.768 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.768 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.769 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.769 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.770 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.770 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.770 DEBUG:boto:Data: 2023-10-11T15:07:10.771 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.772 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.772 DEBUG:boto:Params: {} 2023-10-11T15:07:10.772 DEBUG:boto:Token: None 2023-10-11T15:07:10.773 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc= 2023-10-11T15:07:10.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc=', 'Content-Length': '0'} 2023-10-11T15:07:10.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c15a5f7d23198cc-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.777 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.777 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.777 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.778 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.778 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.779 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.779 DEBUG:boto:Data: 2023-10-11T15:07:10.780 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.780 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.781 DEBUG:boto:Params: {} 2023-10-11T15:07:10.781 DEBUG:boto:Token: None 2023-10-11T15:07:10.782 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc= 2023-10-11T15:07:10.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc=', 'Content-Length': '0'} 2023-10-11T15:07:10.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924ca3a581e75ff2-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:10.785 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.786 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.786 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.787 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.787 DEBUG:boto:Method: GET 2023-10-11T15:07:10.788 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.788 DEBUG:boto:Data: 2023-10-11T15:07:10.788 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.789 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.790 DEBUG:boto:Params: {} 2023-10-11T15:07:10.790 DEBUG:boto:Token: None 2023-10-11T15:07:10.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T15:07:10.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T15:07:10.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bdd3fc54597e46a-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.794 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.795 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.795 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.796 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:10.796 DEBUG:boto:Method: GET 2023-10-11T15:07:10.796 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.797 DEBUG:boto:Data: 2023-10-11T15:07:10.797 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.798 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.799 DEBUG:boto:Params: {} 2023-10-11T15:07:10.799 DEBUG:boto:Token: None 2023-10-11T15:07:10.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:10.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T15:07:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T15:07:10.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b675860ce1942f1a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.803 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.803 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.804 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.804 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.805 DEBUG:boto:Method: GET 2023-10-11T15:07:10.805 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:10.805 DEBUG:boto:Data: 2023-10-11T15:07:10.806 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.807 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.807 DEBUG:boto:Params: {} 2023-10-11T15:07:10.808 DEBUG:boto:Token: None 2023-10-11T15:07:10.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043688fa2fdbb39db-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.812 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.813 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.813 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.814 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.814 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.815 DEBUG:boto:Method: GET 2023-10-11T15:07:10.815 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:10.815 DEBUG:boto:Data: 2023-10-11T15:07:10.816 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.817 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.817 DEBUG:boto:Params: {} 2023-10-11T15:07:10.818 DEBUG:boto:Token: None 2023-10-11T15:07:10.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddddd476a5dbc377-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.866 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.867 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.868 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.868 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.869 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:10.869 DEBUG:boto:Data: 2023-10-11T15:07:10.869 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.870 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.871 DEBUG:boto:Params: {} 2023-10-11T15:07:10.871 DEBUG:boto:Token: None 2023-10-11T15:07:10.872 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c 2023-10-11T15:07:10.872 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI= 2023-10-11T15:07:10.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI=', 'Content-Length': '0'} 2023-10-11T15:07:10.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732ea1bb283f2f71-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.875 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.876 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.876 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.877 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:10.877 DEBUG:boto:Data: 2023-10-11T15:07:10.878 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.878 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.878 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.879 DEBUG:boto:Params: {} 2023-10-11T15:07:10.879 DEBUG:boto:Token: None 2023-10-11T15:07:10.880 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c 2023-10-11T15:07:10.880 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI= 2023-10-11T15:07:10.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI=', 'Content-Length': '0'} 2023-10-11T15:07:10.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a00ab577bdffc2c-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:10.884 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.884 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.885 DEBUG:boto:Method: GET 2023-10-11T15:07:10.885 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:10.886 DEBUG:boto:Data: 2023-10-11T15:07:10.886 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.887 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.887 DEBUG:boto:Params: {} 2023-10-11T15:07:10.888 DEBUG:boto:Token: None 2023-10-11T15:07:10.888 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c 2023-10-11T15:07:10.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA= 2023-10-11T15:07:10.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA=', 'Content-Length': '0'} 2023-10-11T15:07:10.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388d3f0ea019a4a4-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.892 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:10.892 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:10.893 DEBUG:boto:Method: GET 2023-10-11T15:07:10.893 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:10.894 DEBUG:boto:Data: 2023-10-11T15:07:10.894 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.895 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.895 DEBUG:boto:Params: {} 2023-10-11T15:07:10.896 DEBUG:boto:Token: None 2023-10-11T15:07:10.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/c 2023-10-11T15:07:10.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA= 2023-10-11T15:07:10.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA=', 'Content-Length': '0'} 2023-10-11T15:07:10.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009320e0315a3c8544-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.900 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.900 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.901 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.901 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.902 DEBUG:boto:Method: GET 2023-10-11T15:07:10.902 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:10.902 DEBUG:boto:Data: 2023-10-11T15:07:10.903 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.904 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.904 DEBUG:boto:Params: {} 2023-10-11T15:07:10.905 DEBUG:boto:Token: None 2023-10-11T15:07:10.905 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.906 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c076036905df0d41-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.954 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.955 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:10.956 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:10.956 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.957 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:10.957 DEBUG:boto:Method: GET 2023-10-11T15:07:10.958 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:10.958 DEBUG:boto:Data: 2023-10-11T15:07:10.958 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.959 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.959 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.960 DEBUG:boto:Params: {} 2023-10-11T15:07:10.960 DEBUG:boto:Token: None 2023-10-11T15:07:10.961 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/ 2023-10-11T15:07:10.961 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T15:07:10.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T15:07:10.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8dabf37df1a07d-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.965 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:10.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:10.966 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:10.966 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:10.967 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.967 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.967 DEBUG:boto:Method: GET 2023-10-11T15:07:10.968 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.968 DEBUG:boto:Data: 2023-10-11T15:07:10.969 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.969 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.970 DEBUG:boto:Params: {} 2023-10-11T15:07:10.970 DEBUG:boto:Token: None 2023-10-11T15:07:10.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI= 2023-10-11T15:07:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI=', 'Content-Length': '0'} 2023-10-11T15:07:10.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7506bc2a466b897-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.975 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:10.975 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:10.975 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.976 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.976 DEBUG:boto:Method: GET 2023-10-11T15:07:10.977 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.977 DEBUG:boto:Data: 2023-10-11T15:07:10.978 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.978 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.979 DEBUG:boto:Params: {} 2023-10-11T15:07:10.979 DEBUG:boto:Token: None 2023-10-11T15:07:10.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI= 2023-10-11T15:07:10.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI=', 'Content-Length': '0'} 2023-10-11T15:07:10.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e46b3f02e98972-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.983 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:10.984 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:10.984 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.985 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.985 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.985 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.986 DEBUG:boto:Data: 2023-10-11T15:07:10.986 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:10.987 DEBUG:boto:Port: 8001 2023-10-11T15:07:10.987 DEBUG:boto:Params: {} 2023-10-11T15:07:10.988 DEBUG:boto:Token: None 2023-10-11T15:07:10.988 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc= 2023-10-11T15:07:10.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc=', 'Content-Length': '0'} 2023-10-11T15:07:10.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004370426ef0730b35-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:10.992 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:10.992 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:10.993 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.993 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:10.993 DEBUG:boto:Method: HEAD 2023-10-11T15:07:10.994 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.994 DEBUG:boto:Data: 2023-10-11T15:07:10.995 DEBUG:boto:Headers: {} 2023-10-11T15:07:10.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:10.995 DEBUG:boto:Port: 8000 2023-10-11T15:07:10.997 DEBUG:boto:Params: {} 2023-10-11T15:07:10.998 DEBUG:boto:Token: None 2023-10-11T15:07:10.998 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:10 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:10.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc= 2023-10-11T15:07:10.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc=', 'Content-Length': '0'} 2023-10-11T15:07:11.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8507a65b60329f-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:11.003 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.003 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.004 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:11.004 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:11.005 DEBUG:boto:Method: GET 2023-10-11T15:07:11.005 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:11.005 DEBUG:boto:Data: 2023-10-11T15:07:11.006 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.007 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.007 DEBUG:boto:Params: {} 2023-10-11T15:07:11.008 DEBUG:boto:Token: None 2023-10-11T15:07:11.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:11.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g= 2023-10-11T15:07:11.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g=', 'Content-Length': '0'} 2023-10-11T15:07:11.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89c3aafef346523-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.012 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.012 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.013 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:11.013 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:11.013 DEBUG:boto:Method: GET 2023-10-11T15:07:11.014 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:11.014 DEBUG:boto:Data: 2023-10-11T15:07:11.015 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.015 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.016 DEBUG:boto:Params: {} 2023-10-11T15:07:11.016 DEBUG:boto:Token: None 2023-10-11T15:07:11.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:11.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g= 2023-10-11T15:07:11.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g=', 'Content-Length': '0'} 2023-10-11T15:07:11.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564af7774e4f0e27-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.066 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.067 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.067 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.068 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.068 DEBUG:boto:Method: GET 2023-10-11T15:07:11.069 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:11.069 DEBUG:boto:Data: 2023-10-11T15:07:11.069 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.070 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.071 DEBUG:boto:Params: {} 2023-10-11T15:07:11.071 DEBUG:boto:Token: None 2023-10-11T15:07:11.072 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3bdd40e18998be-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.076 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.076 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.077 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.077 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.078 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.078 DEBUG:boto:Method: GET 2023-10-11T15:07:11.078 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:11.079 DEBUG:boto:Data: 2023-10-11T15:07:11.079 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.080 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.080 DEBUG:boto:Params: {} 2023-10-11T15:07:11.081 DEBUG:boto:Token: None 2023-10-11T15:07:11.081 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ab6d8bde9d05bd-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.085 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.086 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.086 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.086 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.087 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:11.087 DEBUG:boto:Data: 2023-10-11T15:07:11.088 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.088 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.089 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.089 DEBUG:boto:Params: {} 2023-10-11T15:07:11.089 DEBUG:boto:Token: None 2023-10-11T15:07:11.090 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d 2023-10-11T15:07:11.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo= 2023-10-11T15:07:11.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo=', 'Content-Length': '0'} 2023-10-11T15:07:11.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f22b3f5223587b-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.093 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.094 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.094 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.094 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:11.095 DEBUG:boto:Data: 2023-10-11T15:07:11.095 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.096 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.096 DEBUG:boto:Params: {} 2023-10-11T15:07:11.097 DEBUG:boto:Token: None 2023-10-11T15:07:11.097 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d 2023-10-11T15:07:11.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo= 2023-10-11T15:07:11.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo=', 'Content-Length': '0'} 2023-10-11T15:07:11.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814d492beccab93e-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:11.101 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.102 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.102 DEBUG:boto:Method: GET 2023-10-11T15:07:11.103 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:11.103 DEBUG:boto:Data: 2023-10-11T15:07:11.103 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.104 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.105 DEBUG:boto:Params: {} 2023-10-11T15:07:11.105 DEBUG:boto:Token: None 2023-10-11T15:07:11.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d 2023-10-11T15:07:11.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8= 2023-10-11T15:07:11.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8=', 'Content-Length': '0'} 2023-10-11T15:07:11.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002809d7449c00f3aa-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.150 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:11.151 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:11.151 DEBUG:boto:Method: GET 2023-10-11T15:07:11.152 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:11.152 DEBUG:boto:Data: 2023-10-11T15:07:11.153 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.153 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.153 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.154 DEBUG:boto:Params: {} 2023-10-11T15:07:11.154 DEBUG:boto:Token: None 2023-10-11T15:07:11.155 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/d 2023-10-11T15:07:11.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8= 2023-10-11T15:07:11.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8=', 'Content-Length': '0'} 2023-10-11T15:07:11.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44ecd6411e4e4c1-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.158 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.159 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.159 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.160 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.160 DEBUG:boto:Method: GET 2023-10-11T15:07:11.161 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:11.161 DEBUG:boto:Data: 2023-10-11T15:07:11.161 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.162 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.163 DEBUG:boto:Params: {} 2023-10-11T15:07:11.163 DEBUG:boto:Token: None 2023-10-11T15:07:11.164 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.164 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373a70b15e9023c1-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.168 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.168 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.169 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.169 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.170 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:11.170 DEBUG:boto:Method: GET 2023-10-11T15:07:11.171 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:11.171 DEBUG:boto:Data: 2023-10-11T15:07:11.171 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.172 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.173 DEBUG:boto:Params: {} 2023-10-11T15:07:11.173 DEBUG:boto:Token: None 2023-10-11T15:07:11.174 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0720d423dc4536f-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.177 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:11.178 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.179 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.179 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.180 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.180 DEBUG:boto:Method: GET 2023-10-11T15:07:11.181 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.181 DEBUG:boto:Data: 2023-10-11T15:07:11.181 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.182 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.183 DEBUG:boto:Params: {} 2023-10-11T15:07:11.183 DEBUG:boto:Token: None 2023-10-11T15:07:11.184 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T15:07:11.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T15:07:11.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004772139006fda781-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.188 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.188 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.189 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.189 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.189 DEBUG:boto:Method: GET 2023-10-11T15:07:11.190 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.190 DEBUG:boto:Data: 2023-10-11T15:07:11.191 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.191 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.191 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.192 DEBUG:boto:Params: {} 2023-10-11T15:07:11.192 DEBUG:boto:Token: None 2023-10-11T15:07:11.193 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.193 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T15:07:11.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T15:07:11.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa08fd7dc057966-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.198 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.198 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.199 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.199 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.200 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.200 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.200 DEBUG:boto:Data: 2023-10-11T15:07:11.201 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.202 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.202 DEBUG:boto:Params: {} 2023-10-11T15:07:11.203 DEBUG:boto:Token: None 2023-10-11T15:07:11.203 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.204 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI= 2023-10-11T15:07:11.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e70d19d87e1d924-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.207 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.207 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.207 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.208 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.208 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.209 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.209 DEBUG:boto:Data: 2023-10-11T15:07:11.209 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.210 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.211 DEBUG:boto:Params: {} 2023-10-11T15:07:11.211 DEBUG:boto:Token: None 2023-10-11T15:07:11.212 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI= 2023-10-11T15:07:11.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71f6d99042ec8d8-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:11.216 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.216 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.217 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.217 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.217 DEBUG:boto:Method: GET 2023-10-11T15:07:11.218 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.218 DEBUG:boto:Data: 2023-10-11T15:07:11.219 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.220 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.220 DEBUG:boto:Params: {} 2023-10-11T15:07:11.221 DEBUG:boto:Token: None 2023-10-11T15:07:11.221 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T15:07:11.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T15:07:11.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000184470c3097cbbe8-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.266 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.267 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.267 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.268 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:11.268 DEBUG:boto:Method: GET 2023-10-11T15:07:11.269 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.269 DEBUG:boto:Data: 2023-10-11T15:07:11.270 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.270 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.271 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.271 DEBUG:boto:Params: {} 2023-10-11T15:07:11.272 DEBUG:boto:Token: None 2023-10-11T15:07:11.272 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:11.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T15:07:11.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T15:07:11.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794f95274a561898-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.276 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.276 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.277 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.277 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.278 DEBUG:boto:Method: GET 2023-10-11T15:07:11.278 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:11.278 DEBUG:boto:Data: 2023-10-11T15:07:11.279 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.279 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.280 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.280 DEBUG:boto:Params: {} 2023-10-11T15:07:11.281 DEBUG:boto:Token: None 2023-10-11T15:07:11.281 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ba504872f3181a-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.285 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.287 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.287 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.287 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.288 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.288 DEBUG:boto:Method: GET 2023-10-11T15:07:11.289 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:11.289 DEBUG:boto:Data: 2023-10-11T15:07:11.290 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.291 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.291 DEBUG:boto:Params: {} 2023-10-11T15:07:11.292 DEBUG:boto:Token: None 2023-10-11T15:07:11.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9441a808742c994-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.297 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.298 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.299 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.299 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.300 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:11.300 DEBUG:boto:Data: 2023-10-11T15:07:11.300 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.301 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.301 DEBUG:boto:Params: {} 2023-10-11T15:07:11.302 DEBUG:boto:Token: None 2023-10-11T15:07:11.302 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e 2023-10-11T15:07:11.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8= 2023-10-11T15:07:11.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8=', 'Content-Length': '0'} 2023-10-11T15:07:11.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617934ea331eb5dd-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.306 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.306 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.307 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.307 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:11.308 DEBUG:boto:Data: 2023-10-11T15:07:11.308 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.309 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.309 DEBUG:boto:Params: {} 2023-10-11T15:07:11.310 DEBUG:boto:Token: None 2023-10-11T15:07:11.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e 2023-10-11T15:07:11.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8= 2023-10-11T15:07:11.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8=', 'Content-Length': '0'} 2023-10-11T15:07:11.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfabac2a80bcf9b3-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:11.315 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.315 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.316 DEBUG:boto:Method: GET 2023-10-11T15:07:11.316 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:11.316 DEBUG:boto:Data: 2023-10-11T15:07:11.317 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.318 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.318 DEBUG:boto:Params: {} 2023-10-11T15:07:11.319 DEBUG:boto:Token: None 2023-10-11T15:07:11.319 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e 2023-10-11T15:07:11.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE= 2023-10-11T15:07:11.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE=', 'Content-Length': '0'} 2023-10-11T15:07:11.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d484836f3ee6bc-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.323 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:11.324 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:11.324 DEBUG:boto:Method: GET 2023-10-11T15:07:11.325 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:11.325 DEBUG:boto:Data: 2023-10-11T15:07:11.325 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.326 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.327 DEBUG:boto:Params: {} 2023-10-11T15:07:11.327 DEBUG:boto:Token: None 2023-10-11T15:07:11.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/e 2023-10-11T15:07:11.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE= 2023-10-11T15:07:11.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE=', 'Content-Length': '0'} 2023-10-11T15:07:11.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0679da743cd7bd-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.331 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.332 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.332 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.333 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.333 DEBUG:boto:Method: GET 2023-10-11T15:07:11.333 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:11.334 DEBUG:boto:Data: 2023-10-11T15:07:11.334 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.335 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.336 DEBUG:boto:Params: {} 2023-10-11T15:07:11.336 DEBUG:boto:Token: None 2023-10-11T15:07:11.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f08a42464b668e-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.341 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.341 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.342 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.342 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.343 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:11.343 DEBUG:boto:Method: GET 2023-10-11T15:07:11.343 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:11.344 DEBUG:boto:Data: 2023-10-11T15:07:11.344 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.345 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.346 DEBUG:boto:Params: {} 2023-10-11T15:07:11.346 DEBUG:boto:Token: None 2023-10-11T15:07:11.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3d87de62503cbbc-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.390 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:11.392 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.392 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.393 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.393 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.393 DEBUG:boto:Method: GET 2023-10-11T15:07:11.394 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.394 DEBUG:boto:Data: 2023-10-11T15:07:11.395 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.396 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.396 DEBUG:boto:Params: {} 2023-10-11T15:07:11.397 DEBUG:boto:Token: None 2023-10-11T15:07:11.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T15:07:11.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T15:07:11.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48ef0ce168e7010-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.401 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.402 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.402 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.403 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.403 DEBUG:boto:Method: GET 2023-10-11T15:07:11.404 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.405 DEBUG:boto:Data: 2023-10-11T15:07:11.406 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.407 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.407 DEBUG:boto:Params: {} 2023-10-11T15:07:11.408 DEBUG:boto:Token: None 2023-10-11T15:07:11.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T15:07:11.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T15:07:11.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bc9c44ea7f45a0-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.412 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.412 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.413 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.413 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.414 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.414 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.414 DEBUG:boto:Data: 2023-10-11T15:07:11.415 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.416 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.416 DEBUG:boto:Params: {} 2023-10-11T15:07:11.417 DEBUG:boto:Token: None 2023-10-11T15:07:11.417 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw= 2023-10-11T15:07:11.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw=', 'Content-Length': '0'} 2023-10-11T15:07:11.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2634d1ba169081-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.421 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.421 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.421 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.422 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.422 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.423 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.423 DEBUG:boto:Data: 2023-10-11T15:07:11.424 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.424 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.425 DEBUG:boto:Params: {} 2023-10-11T15:07:11.425 DEBUG:boto:Token: None 2023-10-11T15:07:11.426 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw= 2023-10-11T15:07:11.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw=', 'Content-Length': '0'} 2023-10-11T15:07:11.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326ce9ffcf3a5b05-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:11.430 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.430 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.430 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.431 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.431 DEBUG:boto:Method: GET 2023-10-11T15:07:11.432 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.432 DEBUG:boto:Data: 2023-10-11T15:07:11.433 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.433 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.434 DEBUG:boto:Params: {} 2023-10-11T15:07:11.434 DEBUG:boto:Token: None 2023-10-11T15:07:11.435 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.435 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T15:07:11.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T15:07:11.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afca355f49c60d9c-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.439 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.439 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.439 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.440 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:11.440 DEBUG:boto:Method: GET 2023-10-11T15:07:11.441 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.441 DEBUG:boto:Data: 2023-10-11T15:07:11.441 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.442 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.443 DEBUG:boto:Params: {} 2023-10-11T15:07:11.443 DEBUG:boto:Token: None 2023-10-11T15:07:11.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:11.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T15:07:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T15:07:11.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e56fb0047973702-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.448 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.448 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.448 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.449 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.449 DEBUG:boto:Method: GET 2023-10-11T15:07:11.450 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:11.450 DEBUG:boto:Data: 2023-10-11T15:07:11.451 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.451 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.452 DEBUG:boto:Params: {} 2023-10-11T15:07:11.452 DEBUG:boto:Token: None 2023-10-11T15:07:11.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f82b9d480e2dd7d-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.457 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.458 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.458 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.459 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.459 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.459 DEBUG:boto:Method: GET 2023-10-11T15:07:11.460 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:11.460 DEBUG:boto:Data: 2023-10-11T15:07:11.461 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.461 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.462 DEBUG:boto:Params: {} 2023-10-11T15:07:11.462 DEBUG:boto:Token: None 2023-10-11T15:07:11.463 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004024ebb46f4a866b-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.506 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.507 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.508 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.508 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.508 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:11.509 DEBUG:boto:Data: 2023-10-11T15:07:11.509 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.510 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.511 DEBUG:boto:Params: {} 2023-10-11T15:07:11.511 DEBUG:boto:Token: None 2023-10-11T15:07:11.511 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f 2023-10-11T15:07:11.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0= 2023-10-11T15:07:11.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0=', 'Content-Length': '0'} 2023-10-11T15:07:11.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db5a6556aa024ed-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.515 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.515 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.516 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.516 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:11.517 DEBUG:boto:Data: 2023-10-11T15:07:11.517 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.518 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.518 DEBUG:boto:Params: {} 2023-10-11T15:07:11.519 DEBUG:boto:Token: None 2023-10-11T15:07:11.519 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f 2023-10-11T15:07:11.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0= 2023-10-11T15:07:11.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0=', 'Content-Length': '0'} 2023-10-11T15:07:11.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fb6def2def5ce5-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:11.523 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.523 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.524 DEBUG:boto:Method: GET 2023-10-11T15:07:11.524 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:11.525 DEBUG:boto:Data: 2023-10-11T15:07:11.525 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.526 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.526 DEBUG:boto:Params: {} 2023-10-11T15:07:11.527 DEBUG:boto:Token: None 2023-10-11T15:07:11.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f 2023-10-11T15:07:11.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc= 2023-10-11T15:07:11.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc=', 'Content-Length': '0'} 2023-10-11T15:07:11.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1455bb81bfe5dcc-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.574 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:11.575 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:11.576 DEBUG:boto:Method: GET 2023-10-11T15:07:11.576 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:11.576 DEBUG:boto:Data: 2023-10-11T15:07:11.577 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.578 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.578 DEBUG:boto:Params: {} 2023-10-11T15:07:11.578 DEBUG:boto:Token: None 2023-10-11T15:07:11.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/f 2023-10-11T15:07:11.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc= 2023-10-11T15:07:11.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc=', 'Content-Length': '0'} 2023-10-11T15:07:11.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d7ed31f0cc8206-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.583 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.583 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.584 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.584 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.585 DEBUG:boto:Method: GET 2023-10-11T15:07:11.585 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:11.585 DEBUG:boto:Data: 2023-10-11T15:07:11.586 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.587 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.587 DEBUG:boto:Params: {} 2023-10-11T15:07:11.587 DEBUG:boto:Token: None 2023-10-11T15:07:11.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e12551572d8f5fc-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.592 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.593 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.593 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.594 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.594 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:11.594 DEBUG:boto:Method: GET 2023-10-11T15:07:11.595 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:11.595 DEBUG:boto:Data: 2023-10-11T15:07:11.596 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.597 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.597 DEBUG:boto:Params: {} 2023-10-11T15:07:11.597 DEBUG:boto:Token: None 2023-10-11T15:07:11.598 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e959a2ec53bc1e-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.602 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:11.603 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.603 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.604 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.604 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.605 DEBUG:boto:Method: GET 2023-10-11T15:07:11.605 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.605 DEBUG:boto:Data: 2023-10-11T15:07:11.606 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.606 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.607 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.607 DEBUG:boto:Params: {} 2023-10-11T15:07:11.608 DEBUG:boto:Token: None 2023-10-11T15:07:11.608 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T15:07:11.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a86d089a7b7b92-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.612 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.612 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.613 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.613 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.613 DEBUG:boto:Method: GET 2023-10-11T15:07:11.614 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.614 DEBUG:boto:Data: 2023-10-11T15:07:11.615 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.615 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.616 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.616 DEBUG:boto:Params: {} 2023-10-11T15:07:11.617 DEBUG:boto:Token: None 2023-10-11T15:07:11.617 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T15:07:11.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad095461caea772-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.621 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.621 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.622 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.622 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.623 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.623 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.623 DEBUG:boto:Data: 2023-10-11T15:07:11.624 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.625 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.625 DEBUG:boto:Params: {} 2023-10-11T15:07:11.625 DEBUG:boto:Token: None 2023-10-11T15:07:11.626 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.627 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E= 2023-10-11T15:07:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E=', 'Content-Length': '0'} 2023-10-11T15:07:11.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a1b740f585439e1-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.630 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.631 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.631 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.631 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.632 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.632 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.633 DEBUG:boto:Data: 2023-10-11T15:07:11.633 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.633 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.634 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.634 DEBUG:boto:Params: {} 2023-10-11T15:07:11.635 DEBUG:boto:Token: None 2023-10-11T15:07:11.635 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E= 2023-10-11T15:07:11.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E=', 'Content-Length': '0'} 2023-10-11T15:07:11.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca57da46f21acd90-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:11.639 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.639 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.640 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.640 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.640 DEBUG:boto:Method: GET 2023-10-11T15:07:11.641 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.641 DEBUG:boto:Data: 2023-10-11T15:07:11.642 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.643 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.643 DEBUG:boto:Params: {} 2023-10-11T15:07:11.644 DEBUG:boto:Token: None 2023-10-11T15:07:11.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T15:07:11.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32970ea7ec66ed7-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.690 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.691 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.691 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.692 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:11.692 DEBUG:boto:Method: GET 2023-10-11T15:07:11.693 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.693 DEBUG:boto:Data: 2023-10-11T15:07:11.693 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.694 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.695 DEBUG:boto:Params: {} 2023-10-11T15:07:11.695 DEBUG:boto:Token: None 2023-10-11T15:07:11.696 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:11.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T15:07:11.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T15:07:11.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009587881313d17369-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.700 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.700 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.701 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.701 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.702 DEBUG:boto:Method: GET 2023-10-11T15:07:11.702 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:11.702 DEBUG:boto:Data: 2023-10-11T15:07:11.703 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.704 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.704 DEBUG:boto:Params: {} 2023-10-11T15:07:11.705 DEBUG:boto:Token: None 2023-10-11T15:07:11.705 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.706 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4787f59b42ff7a-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.709 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.710 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.710 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.711 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.711 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.712 DEBUG:boto:Method: GET 2023-10-11T15:07:11.712 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:11.712 DEBUG:boto:Data: 2023-10-11T15:07:11.713 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.714 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.714 DEBUG:boto:Params: {} 2023-10-11T15:07:11.715 DEBUG:boto:Token: None 2023-10-11T15:07:11.715 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34574061c7604ee-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.758 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.759 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.760 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.760 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.760 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:11.761 DEBUG:boto:Data: 2023-10-11T15:07:11.761 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.762 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.763 DEBUG:boto:Params: {} 2023-10-11T15:07:11.763 DEBUG:boto:Token: None 2023-10-11T15:07:11.764 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g 2023-10-11T15:07:11.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY= 2023-10-11T15:07:11.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY=', 'Content-Length': '0'} 2023-10-11T15:07:11.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9c14a521b22e9b-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.767 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.767 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.768 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.768 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:11.769 DEBUG:boto:Data: 2023-10-11T15:07:11.769 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.770 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.771 DEBUG:boto:Params: {} 2023-10-11T15:07:11.771 DEBUG:boto:Token: None 2023-10-11T15:07:11.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g 2023-10-11T15:07:11.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY= 2023-10-11T15:07:11.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY=', 'Content-Length': '0'} 2023-10-11T15:07:11.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4d83d516e0791a-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:11.775 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.776 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.776 DEBUG:boto:Method: GET 2023-10-11T15:07:11.777 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:11.777 DEBUG:boto:Data: 2023-10-11T15:07:11.778 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.779 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.779 DEBUG:boto:Params: {} 2023-10-11T15:07:11.780 DEBUG:boto:Token: None 2023-10-11T15:07:11.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g 2023-10-11T15:07:11.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc= 2023-10-11T15:07:11.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc=', 'Content-Length': '0'} 2023-10-11T15:07:11.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005538ecc9c3a12e67-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.783 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:11.784 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:11.784 DEBUG:boto:Method: GET 2023-10-11T15:07:11.785 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:11.785 DEBUG:boto:Data: 2023-10-11T15:07:11.786 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.787 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.787 DEBUG:boto:Params: {} 2023-10-11T15:07:11.787 DEBUG:boto:Token: None 2023-10-11T15:07:11.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/g 2023-10-11T15:07:11.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc= 2023-10-11T15:07:11.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc=', 'Content-Length': '0'} 2023-10-11T15:07:11.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e79b8191865ab60-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.791 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.792 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.792 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.793 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.793 DEBUG:boto:Method: GET 2023-10-11T15:07:11.793 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:11.794 DEBUG:boto:Data: 2023-10-11T15:07:11.794 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.795 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.796 DEBUG:boto:Params: {} 2023-10-11T15:07:11.796 DEBUG:boto:Token: None 2023-10-11T15:07:11.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024482de1ec585031-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.842 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.843 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.844 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.844 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.845 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:11.845 DEBUG:boto:Method: GET 2023-10-11T15:07:11.845 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:11.846 DEBUG:boto:Data: 2023-10-11T15:07:11.846 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.847 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.848 DEBUG:boto:Params: {} 2023-10-11T15:07:11.848 DEBUG:boto:Token: None 2023-10-11T15:07:11.849 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c370f2b74af5c51-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.852 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:11.853 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.854 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.854 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.855 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.855 DEBUG:boto:Method: GET 2023-10-11T15:07:11.855 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.856 DEBUG:boto:Data: 2023-10-11T15:07:11.856 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.857 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.858 DEBUG:boto:Params: {} 2023-10-11T15:07:11.858 DEBUG:boto:Token: None 2023-10-11T15:07:11.859 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T15:07:11.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T15:07:11.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e20a1adc8317fa-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.862 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.863 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.863 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.863 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.864 DEBUG:boto:Method: GET 2023-10-11T15:07:11.864 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.865 DEBUG:boto:Data: 2023-10-11T15:07:11.865 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.866 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.866 DEBUG:boto:Params: {} 2023-10-11T15:07:11.867 DEBUG:boto:Token: None 2023-10-11T15:07:11.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T15:07:11.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T15:07:11.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2403e99a69e0d05-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.871 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.871 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.872 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.872 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.873 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.873 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.873 DEBUG:boto:Data: 2023-10-11T15:07:11.874 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.875 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.875 DEBUG:boto:Params: {} 2023-10-11T15:07:11.876 DEBUG:boto:Token: None 2023-10-11T15:07:11.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0= 2023-10-11T15:07:11.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0=', 'Content-Length': '0'} 2023-10-11T15:07:11.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb655b630913645-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.880 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.880 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.880 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.881 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.881 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.882 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.882 DEBUG:boto:Data: 2023-10-11T15:07:11.883 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.883 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.884 DEBUG:boto:Params: {} 2023-10-11T15:07:11.884 DEBUG:boto:Token: None 2023-10-11T15:07:11.885 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0= 2023-10-11T15:07:11.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0=', 'Content-Length': '0'} 2023-10-11T15:07:11.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002039faac7c091e11-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:11.888 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.889 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.889 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.889 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.890 DEBUG:boto:Method: GET 2023-10-11T15:07:11.890 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.891 DEBUG:boto:Data: 2023-10-11T15:07:11.891 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.892 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.892 DEBUG:boto:Params: {} 2023-10-11T15:07:11.893 DEBUG:boto:Token: None 2023-10-11T15:07:11.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T15:07:11.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T15:07:11.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb4a52df2c1bb25-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.897 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.897 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.898 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.898 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:11.898 DEBUG:boto:Method: GET 2023-10-11T15:07:11.899 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.899 DEBUG:boto:Data: 2023-10-11T15:07:11.900 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.901 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.901 DEBUG:boto:Params: {} 2023-10-11T15:07:11.902 DEBUG:boto:Token: None 2023-10-11T15:07:11.902 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:11.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T15:07:11.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T15:07:11.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee14ff8b90ef0e5-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.950 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.951 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.951 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:11.952 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:11.952 DEBUG:boto:Method: GET 2023-10-11T15:07:11.953 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:11.953 DEBUG:boto:Data: 2023-10-11T15:07:11.953 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.954 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.955 DEBUG:boto:Params: {} 2023-10-11T15:07:11.955 DEBUG:boto:Token: None 2023-10-11T15:07:11.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b38e8ce9dbce30-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.960 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.960 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:11.961 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:11.961 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:11.962 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:11.962 DEBUG:boto:Method: GET 2023-10-11T15:07:11.963 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:11.963 DEBUG:boto:Data: 2023-10-11T15:07:11.963 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.964 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.965 DEBUG:boto:Params: {} 2023-10-11T15:07:11.965 DEBUG:boto:Token: None 2023-10-11T15:07:11.966 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/ 2023-10-11T15:07:11.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T15:07:11.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T15:07:11.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1dda8607746d48-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.969 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:11.970 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.970 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.971 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.971 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:11.972 DEBUG:boto:Data: 2023-10-11T15:07:11.972 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.973 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.973 DEBUG:boto:Params: {} 2023-10-11T15:07:11.974 DEBUG:boto:Token: None 2023-10-11T15:07:11.974 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h 2023-10-11T15:07:11.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU= 2023-10-11T15:07:11.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU=', 'Content-Length': '0'} 2023-10-11T15:07:11.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513e1a4843680c63-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.978 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.978 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.979 DEBUG:boto:Method: HEAD 2023-10-11T15:07:11.979 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:11.979 DEBUG:boto:Data: 2023-10-11T15:07:11.980 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:11.981 DEBUG:boto:Port: 8000 2023-10-11T15:07:11.981 DEBUG:boto:Params: {} 2023-10-11T15:07:11.982 DEBUG:boto:Token: None 2023-10-11T15:07:11.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h 2023-10-11T15:07:11.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU= 2023-10-11T15:07:11.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU=', 'Content-Length': '0'} 2023-10-11T15:07:11.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fce73e1b6fa358b-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:11.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:11.986 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:11.986 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:11.986 DEBUG:boto:Method: GET 2023-10-11T15:07:11.987 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:11.987 DEBUG:boto:Data: 2023-10-11T15:07:11.988 DEBUG:boto:Headers: {} 2023-10-11T15:07:11.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:11.989 DEBUG:boto:Port: 8001 2023-10-11T15:07:11.989 DEBUG:boto:Params: {} 2023-10-11T15:07:11.989 DEBUG:boto:Token: None 2023-10-11T15:07:11.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:11 GMT /dghqvd-55/h 2023-10-11T15:07:11.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2oiqGvi95aVg2EGY3H9fqWgwbrQ= 2023-10-11T15:07:11.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2oiqGvi95aVg2EGY3H9fqWgwbrQ=', 'Content-Length': '0'} 2023-10-11T15:07:11.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3655048d361a75e-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:12.035 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:12.036 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:12.036 DEBUG:boto:Method: GET 2023-10-11T15:07:12.036 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:12.037 DEBUG:boto:Data: 2023-10-11T15:07:12.037 DEBUG:boto:Headers: {} 2023-10-11T15:07:12.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:12.038 DEBUG:boto:Port: 8000 2023-10-11T15:07:12.039 DEBUG:boto:Params: {} 2023-10-11T15:07:12.039 DEBUG:boto:Token: None 2023-10-11T15:07:12.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:12 GMT /dghqvd-55/h 2023-10-11T15:07:12.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u0Lu9KY4btwy9kZMFDRZ0lx7Jq8= 2023-10-11T15:07:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u0Lu9KY4btwy9kZMFDRZ0lx7Jq8=', 'Content-Length': '0'} 2023-10-11T15:07:12.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c91c61e9ab139f-006526ba20-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:12.043 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:12.044 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:12.044 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:12.045 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:12.045 DEBUG:boto:Method: GET 2023-10-11T15:07:12.045 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:12.046 DEBUG:boto:Data: 2023-10-11T15:07:12.046 DEBUG:boto:Headers: {} 2023-10-11T15:07:12.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:12.047 DEBUG:boto:Port: 8001 2023-10-11T15:07:12.047 DEBUG:boto:Params: {} 2023-10-11T15:07:12.048 DEBUG:boto:Token: None 2023-10-11T15:07:12.048 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:12 GMT /dghqvd-55/ 2023-10-11T15:07:12.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg= 2023-10-11T15:07:12.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg=', 'Content-Length': '0'} 2023-10-11T15:07:12.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c542aea0a4a5f6b8-006526ba20-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:12.052 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:12.053 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:12.054 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:12.054 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:12.055 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:12.055 DEBUG:boto:Method: GET 2023-10-11T15:07:12.055 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:12.056 DEBUG:boto:Data: 2023-10-11T15:07:12.056 DEBUG:boto:Headers: {} 2023-10-11T15:07:12.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:12.057 DEBUG:boto:Port: 8000 2023-10-11T15:07:12.057 DEBUG:boto:Params: {} 2023-10-11T15:07:12.058 DEBUG:boto:Token: None 2023-10-11T15:07:12.058 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:12 GMT /dghqvd-55/ 2023-10-11T15:07:12.059 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg= 2023-10-11T15:07:12.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg=', 'Content-Length': '0'} 2023-10-11T15:07:12.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008109ecaa60fbb70a-006526ba20-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:12.062 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:12.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:07:12.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:12.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:12.064 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:12.120 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:12.120 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:12.162 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:07:12.163 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-55 2023-10-11T15:07:12.163 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 8 2023-10-11T15:07:12.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:12.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:12.190 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:12.239 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:12.280 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:07:12.280 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.281 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding": "None", 2023-10-11T15:07:12.281 INFO:teuthology.orchestra.run.smithi158.stdout: "current_index": { 2023-10-11T15:07:12.281 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:12.281 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.282 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "Normal", 2023-10-11T15:07:12.282 INFO:teuthology.orchestra.run.smithi158.stdout: "normal": { 2023-10-11T15:07:12.282 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:12.282 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:12.282 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:12.283 INFO:teuthology.orchestra.run.smithi158.stdout: "logs": [ 2023-10-11T15:07:12.283 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:12.283 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:07:12.284 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.284 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:12.284 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:12.284 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 0, 2023-10-11T15:07:12.284 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.285 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 11, 2023-10-11T15:07:12.285 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:12.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:12.286 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:12.286 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:12.286 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.287 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:12.287 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:12.287 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:12.287 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.287 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:12.288 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:12.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.288 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:12.289 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:12.289 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:12.289 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.289 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:12.290 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:12.290 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:12.290 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:12.290 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:12.290 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:12.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.292 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:07:12.292 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:12.292 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:07:12.293 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T15:07:12.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:12.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:12.295 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:07:12.350 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:12.351 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:12.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:12.381+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:07:17.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:17.382+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:07:22.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:22.383+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:07:27.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:27.383+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:07:32.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:32.384+0000 7faa967ed780 1 bucket sync caught up with empty source 2023-10-11T15:07:32.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:32.384+0000 7faa967ed780 0 bucket checkpoint complete 2023-10-11T15:07:32.403 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T15:07:32.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:32.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:32.405 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:07:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:32.469 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:32.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:32.504+0000 7f780e303780 1 bucket sync caught up with source: 2023-10-11T15:07:32.506 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:32.506 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T15:07:32.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:32.504+0000 7f780e303780 0 bucket checkpoint complete 2023-10-11T15:07:32.520 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:07:32.520 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.521 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.521 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.522 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:07:32.522 DEBUG:boto:Data: 2023-10-11T15:07:32.523 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.523 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.523 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.524 DEBUG:boto:Params: {} 2023-10-11T15:07:32.524 DEBUG:boto:Token: None 2023-10-11T15:07:32.525 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY= 2023-10-11T15:07:32.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY=', 'Content-Length': '0'} 2023-10-11T15:07:32.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bac79755a0258da3-006526ba34-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.529 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.529 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.530 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.530 DEBUG:boto:Path: /dghqvd-55/ 2023-10-11T15:07:32.530 DEBUG:boto:Data: 2023-10-11T15:07:32.531 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.532 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.532 DEBUG:boto:Params: {} 2023-10-11T15:07:32.532 DEBUG:boto:Token: None 2023-10-11T15:07:32.533 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY= 2023-10-11T15:07:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY=', 'Content-Length': '0'} 2023-10-11T15:07:32.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd405c5c63e17bac-006526ba34-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.536 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:07:32.537 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.538 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.538 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.538 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.539 DEBUG:boto:Method: GET 2023-10-11T15:07:32.539 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.540 DEBUG:boto:Data: 2023-10-11T15:07:32.540 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.541 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.541 DEBUG:boto:Params: {} 2023-10-11T15:07:32.542 DEBUG:boto:Token: None 2023-10-11T15:07:32.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/?versions 2023-10-11T15:07:32.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T15:07:32.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T15:07:32.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4da632847e1ce07-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.546 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.547 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:07:32.548 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:07:32.548 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:07:32.549 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:07:32.549 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:07:32.549 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:07:32.550 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:07:32.550 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:07:32.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:07:32.551 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.551 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.552 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.552 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.552 DEBUG:boto:Method: GET 2023-10-11T15:07:32.553 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.553 DEBUG:boto:Data: 2023-10-11T15:07:32.554 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.555 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.555 DEBUG:boto:Params: {} 2023-10-11T15:07:32.555 DEBUG:boto:Token: None 2023-10-11T15:07:32.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/?versions 2023-10-11T15:07:32.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T15:07:32.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T15:07:32.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027fdc07668d6f212-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.559 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.560 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:07:32.560 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:07:32.561 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:07:32.561 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:07:32.561 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:07:32.562 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:07:32.562 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:07:32.562 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:07:32.563 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.563 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.564 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.564 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.565 DEBUG:boto:Method: GET 2023-10-11T15:07:32.565 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.565 DEBUG:boto:Data: 2023-10-11T15:07:32.566 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.567 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.567 DEBUG:boto:Params: {} 2023-10-11T15:07:32.568 DEBUG:boto:Token: None 2023-10-11T15:07:32.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/?versions 2023-10-11T15:07:32.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T15:07:32.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T15:07:32.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778ed2e3a53564c9-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.572 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.573 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.574 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.574 DEBUG:boto:path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.575 DEBUG:boto:auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.575 DEBUG:boto:Method: GET 2023-10-11T15:07:32.575 DEBUG:boto:Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T15:07:32.576 DEBUG:boto:Data: 2023-10-11T15:07:32.576 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.577 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.577 DEBUG:boto:Params: {} 2023-10-11T15:07:32.578 DEBUG:boto:Token: None 2023-10-11T15:07:32.578 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/?versions 2023-10-11T15:07:32.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T15:07:32.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T15:07:32.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae421a07b2701c99-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.582 DEBUG:boto:b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:32.583 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.584 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.584 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.584 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.585 DEBUG:boto:Method: GET 2023-10-11T15:07:32.585 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.586 DEBUG:boto:Data: 2023-10-11T15:07:32.586 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.587 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.587 DEBUG:boto:Params: {} 2023-10-11T15:07:32.588 DEBUG:boto:Token: None 2023-10-11T15:07:32.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T15:07:32.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T15:07:32.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b687952624865766-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.591 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.592 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.592 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.593 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.593 DEBUG:boto:Method: GET 2023-10-11T15:07:32.593 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.594 DEBUG:boto:Data: 2023-10-11T15:07:32.594 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.595 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.596 DEBUG:boto:Params: {} 2023-10-11T15:07:32.596 DEBUG:boto:Token: None 2023-10-11T15:07:32.597 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T15:07:32.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T15:07:32.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da776b2e4269ade-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.642 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.643 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.643 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.644 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.644 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.645 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.645 DEBUG:boto:Data: 2023-10-11T15:07:32.645 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.646 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.647 DEBUG:boto:Params: {} 2023-10-11T15:07:32.647 DEBUG:boto:Token: None 2023-10-11T15:07:32.648 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c= 2023-10-11T15:07:32.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c=', 'Content-Length': '0'} 2023-10-11T15:07:32.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825f99769b2c5106-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.652 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.652 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.653 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.653 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.653 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.654 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.655 DEBUG:boto:Data: 2023-10-11T15:07:32.655 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.656 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.656 DEBUG:boto:Params: {} 2023-10-11T15:07:32.657 DEBUG:boto:Token: None 2023-10-11T15:07:32.658 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c= 2023-10-11T15:07:32.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c=', 'Content-Length': '0'} 2023-10-11T15:07:32.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119f1f470500b33f-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:32.662 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.663 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.663 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.664 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.664 DEBUG:boto:Method: GET 2023-10-11T15:07:32.665 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.665 DEBUG:boto:Data: 2023-10-11T15:07:32.665 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.666 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.667 DEBUG:boto:Params: {} 2023-10-11T15:07:32.667 DEBUG:boto:Token: None 2023-10-11T15:07:32.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T15:07:32.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T15:07:32.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd7adcc0a48513c-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.718 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.719 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.719 DEBUG:boto:path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.720 DEBUG:boto:auth_path=/dghqvd-55/a?versionId=null 2023-10-11T15:07:32.720 DEBUG:boto:Method: GET 2023-10-11T15:07:32.720 DEBUG:boto:Path: /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.721 DEBUG:boto:Data: 2023-10-11T15:07:32.721 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.722 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.722 DEBUG:boto:Params: {} 2023-10-11T15:07:32.723 DEBUG:boto:Token: None 2023-10-11T15:07:32.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a?versionId=null 2023-10-11T15:07:32.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T15:07:32.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T15:07:32.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae931101b83fd86-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.727 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.727 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.728 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.728 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.728 DEBUG:boto:Method: GET 2023-10-11T15:07:32.729 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:32.729 DEBUG:boto:Data: 2023-10-11T15:07:32.730 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.730 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.731 DEBUG:boto:Params: {} 2023-10-11T15:07:32.731 DEBUG:boto:Token: None 2023-10-11T15:07:32.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0153d9444e50aa-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.736 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.737 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.737 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.738 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.738 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.739 DEBUG:boto:Method: GET 2023-10-11T15:07:32.739 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:32.739 DEBUG:boto:Data: 2023-10-11T15:07:32.740 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.741 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.741 DEBUG:boto:Params: {} 2023-10-11T15:07:32.741 DEBUG:boto:Token: None 2023-10-11T15:07:32.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf33dd2fef30e4e6-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.747 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.748 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.748 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.749 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.749 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:32.750 DEBUG:boto:Data: 2023-10-11T15:07:32.750 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.751 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.751 DEBUG:boto:Params: {} 2023-10-11T15:07:32.752 DEBUG:boto:Token: None 2023-10-11T15:07:32.752 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a 2023-10-11T15:07:32.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY= 2023-10-11T15:07:32.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY=', 'Content-Length': '0'} 2023-10-11T15:07:32.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55514d4b4446fe2-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.756 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.756 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.757 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.757 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:32.757 DEBUG:boto:Data: 2023-10-11T15:07:32.758 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.759 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.759 DEBUG:boto:Params: {} 2023-10-11T15:07:32.760 DEBUG:boto:Token: None 2023-10-11T15:07:32.760 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a 2023-10-11T15:07:32.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY= 2023-10-11T15:07:32.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY=', 'Content-Length': '0'} 2023-10-11T15:07:32.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22ef61269810abf-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:07:32.764 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.765 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.765 DEBUG:boto:Method: GET 2023-10-11T15:07:32.766 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:32.766 DEBUG:boto:Data: 2023-10-11T15:07:32.767 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.767 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.768 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.768 DEBUG:boto:Params: {} 2023-10-11T15:07:32.769 DEBUG:boto:Token: None 2023-10-11T15:07:32.769 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a 2023-10-11T15:07:32.770 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE= 2023-10-11T15:07:32.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE=', 'Content-Length': '0'} 2023-10-11T15:07:32.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad00efa5fce9f6a1-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.814 DEBUG:boto:path=/dghqvd-55/a 2023-10-11T15:07:32.815 DEBUG:boto:auth_path=/dghqvd-55/a 2023-10-11T15:07:32.815 DEBUG:boto:Method: GET 2023-10-11T15:07:32.816 DEBUG:boto:Path: /dghqvd-55/a 2023-10-11T15:07:32.816 DEBUG:boto:Data: 2023-10-11T15:07:32.817 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.818 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.818 DEBUG:boto:Params: {} 2023-10-11T15:07:32.819 DEBUG:boto:Token: None 2023-10-11T15:07:32.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/a 2023-10-11T15:07:32.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE= 2023-10-11T15:07:32.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE=', 'Content-Length': '0'} 2023-10-11T15:07:32.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f865d5edeb5cf2-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.823 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.823 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.824 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.824 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.825 DEBUG:boto:Method: GET 2023-10-11T15:07:32.825 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:32.825 DEBUG:boto:Data: 2023-10-11T15:07:32.826 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.827 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.827 DEBUG:boto:Params: {} 2023-10-11T15:07:32.828 DEBUG:boto:Token: None 2023-10-11T15:07:32.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf3cafdb55bc777-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.832 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.833 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.833 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.834 DEBUG:boto:path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.834 DEBUG:boto:auth_path=/dghqvd-55/?prefix=a 2023-10-11T15:07:32.835 DEBUG:boto:Method: GET 2023-10-11T15:07:32.835 DEBUG:boto:Path: /dghqvd-55/?prefix=a 2023-10-11T15:07:32.835 DEBUG:boto:Data: 2023-10-11T15:07:32.836 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.837 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.837 DEBUG:boto:Params: {} 2023-10-11T15:07:32.838 DEBUG:boto:Token: None 2023-10-11T15:07:32.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dedffd938b857de2-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.841 DEBUG:boto:b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:32.842 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.843 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.843 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.844 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.844 DEBUG:boto:Method: GET 2023-10-11T15:07:32.844 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.845 DEBUG:boto:Data: 2023-10-11T15:07:32.845 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.846 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.847 DEBUG:boto:Params: {} 2023-10-11T15:07:32.847 DEBUG:boto:Token: None 2023-10-11T15:07:32.848 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T15:07:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T15:07:32.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c1137552c4c86b8-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.852 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.852 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.853 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.853 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.853 DEBUG:boto:Method: GET 2023-10-11T15:07:32.854 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.854 DEBUG:boto:Data: 2023-10-11T15:07:32.855 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.855 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.856 DEBUG:boto:Params: {} 2023-10-11T15:07:32.856 DEBUG:boto:Token: None 2023-10-11T15:07:32.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T15:07:32.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T15:07:32.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1d0754db27ff6d-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.860 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.861 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.861 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.862 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.862 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.863 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.863 DEBUG:boto:Data: 2023-10-11T15:07:32.863 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.864 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.865 DEBUG:boto:Params: {} 2023-10-11T15:07:32.865 DEBUG:boto:Token: None 2023-10-11T15:07:32.866 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4= 2023-10-11T15:07:32.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4=', 'Content-Length': '0'} 2023-10-11T15:07:32.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f140b49fd925dfd9-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.870 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.870 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.870 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.871 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.871 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.872 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.872 DEBUG:boto:Data: 2023-10-11T15:07:32.872 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.873 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.873 DEBUG:boto:Params: {} 2023-10-11T15:07:32.874 DEBUG:boto:Token: None 2023-10-11T15:07:32.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4= 2023-10-11T15:07:32.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4=', 'Content-Length': '0'} 2023-10-11T15:07:32.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002385c78039c2d2bb-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:32.878 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.879 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.879 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.879 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.880 DEBUG:boto:Method: GET 2023-10-11T15:07:32.880 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.881 DEBUG:boto:Data: 2023-10-11T15:07:32.881 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.882 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.882 DEBUG:boto:Params: {} 2023-10-11T15:07:32.883 DEBUG:boto:Token: None 2023-10-11T15:07:32.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T15:07:32.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T15:07:32.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1d9bf9e563dc15-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.931 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.931 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.932 DEBUG:boto:path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.932 DEBUG:boto:auth_path=/dghqvd-55/b?versionId=null 2023-10-11T15:07:32.933 DEBUG:boto:Method: GET 2023-10-11T15:07:32.933 DEBUG:boto:Path: /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.934 DEBUG:boto:Data: 2023-10-11T15:07:32.934 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.935 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.936 DEBUG:boto:Params: {} 2023-10-11T15:07:32.936 DEBUG:boto:Token: None 2023-10-11T15:07:32.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b?versionId=null 2023-10-11T15:07:32.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T15:07:32.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T15:07:32.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd4780b44dc07cd-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.941 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.941 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.942 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.942 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.943 DEBUG:boto:Method: GET 2023-10-11T15:07:32.943 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:32.943 DEBUG:boto:Data: 2023-10-11T15:07:32.944 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.945 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.945 DEBUG:boto:Params: {} 2023-10-11T15:07:32.946 DEBUG:boto:Token: None 2023-10-11T15:07:32.946 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586f3fdffdd9780e-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.950 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.953 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.953 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.953 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.954 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.954 DEBUG:boto:Method: GET 2023-10-11T15:07:32.955 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:32.955 DEBUG:boto:Data: 2023-10-11T15:07:32.956 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.956 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.957 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.957 DEBUG:boto:Params: {} 2023-10-11T15:07:32.957 DEBUG:boto:Token: None 2023-10-11T15:07:32.958 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:32.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae52c0fcbcafbe7c-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.961 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:32.962 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.962 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.963 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.963 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:32.963 DEBUG:boto:Data: 2023-10-11T15:07:32.964 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.965 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.965 DEBUG:boto:Params: {} 2023-10-11T15:07:32.966 DEBUG:boto:Token: None 2023-10-11T15:07:32.966 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b 2023-10-11T15:07:32.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ= 2023-10-11T15:07:32.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ=', 'Content-Length': '0'} 2023-10-11T15:07:32.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8e89f3a3cbc6564-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.969 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.970 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.970 DEBUG:boto:Method: HEAD 2023-10-11T15:07:32.971 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:32.971 DEBUG:boto:Data: 2023-10-11T15:07:32.972 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.972 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.973 DEBUG:boto:Params: {} 2023-10-11T15:07:32.973 DEBUG:boto:Token: None 2023-10-11T15:07:32.974 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b 2023-10-11T15:07:32.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ= 2023-10-11T15:07:32.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ=', 'Content-Length': '0'} 2023-10-11T15:07:32.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f8a73aef73dc23-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:07:32.977 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.978 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.978 DEBUG:boto:Method: GET 2023-10-11T15:07:32.979 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:32.979 DEBUG:boto:Data: 2023-10-11T15:07:32.980 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.980 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.981 DEBUG:boto:Params: {} 2023-10-11T15:07:32.981 DEBUG:boto:Token: None 2023-10-11T15:07:32.982 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b 2023-10-11T15:07:32.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk= 2023-10-11T15:07:32.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk=', 'Content-Length': '0'} 2023-10-11T15:07:32.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e575e62928df405a-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.985 DEBUG:boto:path=/dghqvd-55/b 2023-10-11T15:07:32.986 DEBUG:boto:auth_path=/dghqvd-55/b 2023-10-11T15:07:32.986 DEBUG:boto:Method: GET 2023-10-11T15:07:32.987 DEBUG:boto:Path: /dghqvd-55/b 2023-10-11T15:07:32.987 DEBUG:boto:Data: 2023-10-11T15:07:32.988 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:32.989 DEBUG:boto:Port: 8000 2023-10-11T15:07:32.989 DEBUG:boto:Params: {} 2023-10-11T15:07:32.989 DEBUG:boto:Token: None 2023-10-11T15:07:32.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/b 2023-10-11T15:07:32.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk= 2023-10-11T15:07:32.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk=', 'Content-Length': '0'} 2023-10-11T15:07:32.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009331d1da0acb998a-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:32.993 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:32.994 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:32.994 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.995 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:32.995 DEBUG:boto:Method: GET 2023-10-11T15:07:32.996 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:32.996 DEBUG:boto:Data: 2023-10-11T15:07:32.996 DEBUG:boto:Headers: {} 2023-10-11T15:07:32.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:32.997 DEBUG:boto:Port: 8001 2023-10-11T15:07:32.998 DEBUG:boto:Params: {} 2023-10-11T15:07:32.998 DEBUG:boto:Token: None 2023-10-11T15:07:32.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:32 GMT /dghqvd-55/ 2023-10-11T15:07:32.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T15:07:33.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T15:07:33.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001776325fc3f20977-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.003 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.003 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.004 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.004 DEBUG:boto:path=/dghqvd-55/?prefix=b 2023-10-11T15:07:33.005 DEBUG:boto:auth_path=/dghqvd-55/?prefix=b 2023-10-11T15:07:33.005 DEBUG:boto:Method: GET 2023-10-11T15:07:33.005 DEBUG:boto:Path: /dghqvd-55/?prefix=b 2023-10-11T15:07:33.006 DEBUG:boto:Data: 2023-10-11T15:07:33.006 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.007 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.007 DEBUG:boto:Params: {} 2023-10-11T15:07:33.008 DEBUG:boto:Token: None 2023-10-11T15:07:33.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f362b7c5e06f57-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.011 DEBUG:boto:b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:33.013 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.013 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.013 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.014 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.014 DEBUG:boto:Method: GET 2023-10-11T15:07:33.015 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.015 DEBUG:boto:Data: 2023-10-11T15:07:33.016 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.017 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.017 DEBUG:boto:Params: {} 2023-10-11T15:07:33.017 DEBUG:boto:Token: None 2023-10-11T15:07:33.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T15:07:33.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c5372c8c406e90-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.022 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.022 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.022 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.023 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.023 DEBUG:boto:Method: GET 2023-10-11T15:07:33.024 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.024 DEBUG:boto:Data: 2023-10-11T15:07:33.024 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.025 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.026 DEBUG:boto:Params: {} 2023-10-11T15:07:33.026 DEBUG:boto:Token: None 2023-10-11T15:07:33.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T15:07:33.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643dfefcb368998d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.030 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.030 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.031 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.031 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.032 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.032 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.032 DEBUG:boto:Data: 2023-10-11T15:07:33.033 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.033 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.034 DEBUG:boto:Params: {} 2023-10-11T15:07:33.034 DEBUG:boto:Token: None 2023-10-11T15:07:33.035 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw= 2023-10-11T15:07:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw=', 'Content-Length': '0'} 2023-10-11T15:07:33.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1fc2eb05ecd48cf-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.038 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.038 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.039 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.039 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.040 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.040 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.040 DEBUG:boto:Data: 2023-10-11T15:07:33.041 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.041 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.042 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.042 DEBUG:boto:Params: {} 2023-10-11T15:07:33.043 DEBUG:boto:Token: None 2023-10-11T15:07:33.043 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw= 2023-10-11T15:07:33.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw=', 'Content-Length': '0'} 2023-10-11T15:07:33.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39acd1681b5879a-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:33.047 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.047 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.048 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.048 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.049 DEBUG:boto:Method: GET 2023-10-11T15:07:33.049 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.050 DEBUG:boto:Data: 2023-10-11T15:07:33.050 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.051 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.051 DEBUG:boto:Params: {} 2023-10-11T15:07:33.052 DEBUG:boto:Token: None 2023-10-11T15:07:33.052 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T15:07:33.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d7d53b4c84e508-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.055 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.056 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.056 DEBUG:boto:path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.057 DEBUG:boto:auth_path=/dghqvd-55/c?versionId=null 2023-10-11T15:07:33.057 DEBUG:boto:Method: GET 2023-10-11T15:07:33.058 DEBUG:boto:Path: /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.058 DEBUG:boto:Data: 2023-10-11T15:07:33.059 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.060 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.060 DEBUG:boto:Params: {} 2023-10-11T15:07:33.061 DEBUG:boto:Token: None 2023-10-11T15:07:33.061 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c?versionId=null 2023-10-11T15:07:33.062 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T15:07:33.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1fd65289e444d2-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.065 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.065 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.065 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.066 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.066 DEBUG:boto:Method: GET 2023-10-11T15:07:33.067 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:33.067 DEBUG:boto:Data: 2023-10-11T15:07:33.068 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.068 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.068 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.069 DEBUG:boto:Params: {} 2023-10-11T15:07:33.069 DEBUG:boto:Token: None 2023-10-11T15:07:33.070 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.070 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4e69a7f8bcd874-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.123 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.124 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.125 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.126 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.127 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.128 DEBUG:boto:Method: GET 2023-10-11T15:07:33.128 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:33.129 DEBUG:boto:Data: 2023-10-11T15:07:33.129 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.130 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.130 DEBUG:boto:Params: {} 2023-10-11T15:07:33.131 DEBUG:boto:Token: None 2023-10-11T15:07:33.131 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b131a851217edd5b-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.135 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.136 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.136 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.137 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.137 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:33.137 DEBUG:boto:Data: 2023-10-11T15:07:33.138 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.139 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.139 DEBUG:boto:Params: {} 2023-10-11T15:07:33.140 DEBUG:boto:Token: None 2023-10-11T15:07:33.140 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c 2023-10-11T15:07:33.141 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk= 2023-10-11T15:07:33.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk=', 'Content-Length': '0'} 2023-10-11T15:07:33.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e78024897dc12f2-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.144 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.144 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.144 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.145 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:33.145 DEBUG:boto:Data: 2023-10-11T15:07:33.146 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.147 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.147 DEBUG:boto:Params: {} 2023-10-11T15:07:33.148 DEBUG:boto:Token: None 2023-10-11T15:07:33.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c 2023-10-11T15:07:33.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk= 2023-10-11T15:07:33.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk=', 'Content-Length': '0'} 2023-10-11T15:07:33.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d550c5edfea7fc9-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:07:33.152 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.152 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.153 DEBUG:boto:Method: GET 2023-10-11T15:07:33.153 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:33.154 DEBUG:boto:Data: 2023-10-11T15:07:33.154 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.155 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.155 DEBUG:boto:Params: {} 2023-10-11T15:07:33.156 DEBUG:boto:Token: None 2023-10-11T15:07:33.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c 2023-10-11T15:07:33.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ= 2023-10-11T15:07:33.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afd17c6b055ccfc-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.160 DEBUG:boto:path=/dghqvd-55/c 2023-10-11T15:07:33.161 DEBUG:boto:auth_path=/dghqvd-55/c 2023-10-11T15:07:33.161 DEBUG:boto:Method: GET 2023-10-11T15:07:33.162 DEBUG:boto:Path: /dghqvd-55/c 2023-10-11T15:07:33.162 DEBUG:boto:Data: 2023-10-11T15:07:33.162 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.163 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.164 DEBUG:boto:Params: {} 2023-10-11T15:07:33.164 DEBUG:boto:Token: None 2023-10-11T15:07:33.165 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/c 2023-10-11T15:07:33.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ= 2023-10-11T15:07:33.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ=', 'Content-Length': '0'} 2023-10-11T15:07:33.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64fe9f8b7d7273c-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.214 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.215 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.215 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.216 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.216 DEBUG:boto:Method: GET 2023-10-11T15:07:33.217 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:33.217 DEBUG:boto:Data: 2023-10-11T15:07:33.218 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.219 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.219 DEBUG:boto:Params: {} 2023-10-11T15:07:33.219 DEBUG:boto:Token: None 2023-10-11T15:07:33.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c10aea9c66f686-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.224 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.225 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.226 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.226 DEBUG:boto:path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.227 DEBUG:boto:auth_path=/dghqvd-55/?prefix=c 2023-10-11T15:07:33.227 DEBUG:boto:Method: GET 2023-10-11T15:07:33.227 DEBUG:boto:Path: /dghqvd-55/?prefix=c 2023-10-11T15:07:33.228 DEBUG:boto:Data: 2023-10-11T15:07:33.228 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.229 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.230 DEBUG:boto:Params: {} 2023-10-11T15:07:33.230 DEBUG:boto:Token: None 2023-10-11T15:07:33.231 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916f9069a1281bb7-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.234 DEBUG:boto:b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:33.235 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.236 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.236 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.236 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.237 DEBUG:boto:Method: GET 2023-10-11T15:07:33.237 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.238 DEBUG:boto:Data: 2023-10-11T15:07:33.238 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.239 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.239 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.239 DEBUG:boto:Params: {} 2023-10-11T15:07:33.240 DEBUG:boto:Token: None 2023-10-11T15:07:33.240 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.241 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T15:07:33.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T15:07:33.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a9b0450e40fc42-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.287 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.287 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.288 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.288 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.289 DEBUG:boto:Method: GET 2023-10-11T15:07:33.289 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.290 DEBUG:boto:Data: 2023-10-11T15:07:33.290 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.291 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.291 DEBUG:boto:Params: {} 2023-10-11T15:07:33.292 DEBUG:boto:Token: None 2023-10-11T15:07:33.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T15:07:33.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T15:07:33.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bcada3eb68360c-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.296 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.296 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.297 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.297 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.298 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.298 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.299 DEBUG:boto:Data: 2023-10-11T15:07:33.299 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.300 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.300 DEBUG:boto:Params: {} 2023-10-11T15:07:33.301 DEBUG:boto:Token: None 2023-10-11T15:07:33.301 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg= 2023-10-11T15:07:33.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg=', 'Content-Length': '0'} 2023-10-11T15:07:33.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff9fe4b6b64e93e-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.305 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.305 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.305 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.306 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.306 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.307 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.307 DEBUG:boto:Data: 2023-10-11T15:07:33.307 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.308 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.308 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.309 DEBUG:boto:Params: {} 2023-10-11T15:07:33.309 DEBUG:boto:Token: None 2023-10-11T15:07:33.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.310 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg= 2023-10-11T15:07:33.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg=', 'Content-Length': '0'} 2023-10-11T15:07:33.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4838022b8d4e29-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:33.313 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.314 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.314 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.315 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.315 DEBUG:boto:Method: GET 2023-10-11T15:07:33.315 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.316 DEBUG:boto:Data: 2023-10-11T15:07:33.316 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.317 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.318 DEBUG:boto:Params: {} 2023-10-11T15:07:33.318 DEBUG:boto:Token: None 2023-10-11T15:07:33.319 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T15:07:33.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T15:07:33.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0629f2e73fa97a6-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.322 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.322 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.323 DEBUG:boto:path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.323 DEBUG:boto:auth_path=/dghqvd-55/d?versionId=null 2023-10-11T15:07:33.324 DEBUG:boto:Method: GET 2023-10-11T15:07:33.324 DEBUG:boto:Path: /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.325 DEBUG:boto:Data: 2023-10-11T15:07:33.325 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.326 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.326 DEBUG:boto:Params: {} 2023-10-11T15:07:33.327 DEBUG:boto:Token: None 2023-10-11T15:07:33.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d?versionId=null 2023-10-11T15:07:33.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T15:07:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T15:07:33.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323a4b02593cad0a-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.331 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.331 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.332 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.332 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.332 DEBUG:boto:Method: GET 2023-10-11T15:07:33.333 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:33.333 DEBUG:boto:Data: 2023-10-11T15:07:33.334 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.335 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.335 DEBUG:boto:Params: {} 2023-10-11T15:07:33.335 DEBUG:boto:Token: None 2023-10-11T15:07:33.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e126e637fd10ed45-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.340 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.341 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.341 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.342 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.342 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.342 DEBUG:boto:Method: GET 2023-10-11T15:07:33.343 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:33.343 DEBUG:boto:Data: 2023-10-11T15:07:33.344 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.345 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.345 DEBUG:boto:Params: {} 2023-10-11T15:07:33.345 DEBUG:boto:Token: None 2023-10-11T15:07:33.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b55dd41f7156b1-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.390 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.391 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.392 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.392 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.393 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:33.393 DEBUG:boto:Data: 2023-10-11T15:07:33.393 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.394 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.395 DEBUG:boto:Params: {} 2023-10-11T15:07:33.395 DEBUG:boto:Token: None 2023-10-11T15:07:33.396 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d 2023-10-11T15:07:33.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE= 2023-10-11T15:07:33.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE=', 'Content-Length': '0'} 2023-10-11T15:07:33.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb68595b31ebe76-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.399 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.400 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.400 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.401 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:33.401 DEBUG:boto:Data: 2023-10-11T15:07:33.401 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.402 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.403 DEBUG:boto:Params: {} 2023-10-11T15:07:33.403 DEBUG:boto:Token: None 2023-10-11T15:07:33.404 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d 2023-10-11T15:07:33.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE= 2023-10-11T15:07:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE=', 'Content-Length': '0'} 2023-10-11T15:07:33.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d578b6d47a9c316-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:07:33.407 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.408 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.408 DEBUG:boto:Method: GET 2023-10-11T15:07:33.409 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:33.409 DEBUG:boto:Data: 2023-10-11T15:07:33.410 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.410 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.411 DEBUG:boto:Params: {} 2023-10-11T15:07:33.411 DEBUG:boto:Token: None 2023-10-11T15:07:33.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d 2023-10-11T15:07:33.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4= 2023-10-11T15:07:33.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4=', 'Content-Length': '0'} 2023-10-11T15:07:33.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11e83a83033b904-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.415 DEBUG:boto:path=/dghqvd-55/d 2023-10-11T15:07:33.416 DEBUG:boto:auth_path=/dghqvd-55/d 2023-10-11T15:07:33.416 DEBUG:boto:Method: GET 2023-10-11T15:07:33.417 DEBUG:boto:Path: /dghqvd-55/d 2023-10-11T15:07:33.417 DEBUG:boto:Data: 2023-10-11T15:07:33.417 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.418 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.419 DEBUG:boto:Params: {} 2023-10-11T15:07:33.419 DEBUG:boto:Token: None 2023-10-11T15:07:33.420 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/d 2023-10-11T15:07:33.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4= 2023-10-11T15:07:33.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4=', 'Content-Length': '0'} 2023-10-11T15:07:33.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7f2da3bb209ae9-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.423 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.423 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.424 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.424 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.425 DEBUG:boto:Method: GET 2023-10-11T15:07:33.425 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:33.426 DEBUG:boto:Data: 2023-10-11T15:07:33.426 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.427 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.427 DEBUG:boto:Params: {} 2023-10-11T15:07:33.428 DEBUG:boto:Token: None 2023-10-11T15:07:33.428 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b93b74c4f87937-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.474 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.475 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.476 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.476 DEBUG:boto:path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.477 DEBUG:boto:auth_path=/dghqvd-55/?prefix=d 2023-10-11T15:07:33.477 DEBUG:boto:Method: GET 2023-10-11T15:07:33.478 DEBUG:boto:Path: /dghqvd-55/?prefix=d 2023-10-11T15:07:33.478 DEBUG:boto:Data: 2023-10-11T15:07:33.478 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.479 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.480 DEBUG:boto:Params: {} 2023-10-11T15:07:33.480 DEBUG:boto:Token: None 2023-10-11T15:07:33.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc2a028083a888d-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.484 DEBUG:boto:b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:33.485 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.486 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.486 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.487 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.487 DEBUG:boto:Method: GET 2023-10-11T15:07:33.488 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.488 DEBUG:boto:Data: 2023-10-11T15:07:33.489 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.490 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.490 DEBUG:boto:Params: {} 2023-10-11T15:07:33.491 DEBUG:boto:Token: None 2023-10-11T15:07:33.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T15:07:33.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T15:07:33.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799195b9a3b53ab5-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.495 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.495 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.496 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.496 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.497 DEBUG:boto:Method: GET 2023-10-11T15:07:33.497 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.498 DEBUG:boto:Data: 2023-10-11T15:07:33.498 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.499 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.499 DEBUG:boto:Params: {} 2023-10-11T15:07:33.500 DEBUG:boto:Token: None 2023-10-11T15:07:33.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T15:07:33.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T15:07:33.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fba8c42e1cc374-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.505 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.505 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.506 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.506 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.506 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.507 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.507 DEBUG:boto:Data: 2023-10-11T15:07:33.508 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.509 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.509 DEBUG:boto:Params: {} 2023-10-11T15:07:33.509 DEBUG:boto:Token: None 2023-10-11T15:07:33.510 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA= 2023-10-11T15:07:33.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA=', 'Content-Length': '0'} 2023-10-11T15:07:33.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353dd576881c11a9-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.513 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.514 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.514 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.515 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.515 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.516 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.516 DEBUG:boto:Data: 2023-10-11T15:07:33.516 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.517 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.518 DEBUG:boto:Params: {} 2023-10-11T15:07:33.518 DEBUG:boto:Token: None 2023-10-11T15:07:33.519 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA= 2023-10-11T15:07:33.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA=', 'Content-Length': '0'} 2023-10-11T15:07:33.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e04e814f067444-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:33.522 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.523 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.523 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.524 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.524 DEBUG:boto:Method: GET 2023-10-11T15:07:33.525 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.525 DEBUG:boto:Data: 2023-10-11T15:07:33.525 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.526 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.527 DEBUG:boto:Params: {} 2023-10-11T15:07:33.527 DEBUG:boto:Token: None 2023-10-11T15:07:33.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T15:07:33.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T15:07:33.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75e9dbd2c14feae-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.532 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.532 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.532 DEBUG:boto:path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.533 DEBUG:boto:auth_path=/dghqvd-55/e?versionId=null 2023-10-11T15:07:33.533 DEBUG:boto:Method: GET 2023-10-11T15:07:33.534 DEBUG:boto:Path: /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.534 DEBUG:boto:Data: 2023-10-11T15:07:33.535 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.535 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.536 DEBUG:boto:Params: {} 2023-10-11T15:07:33.536 DEBUG:boto:Token: None 2023-10-11T15:07:33.537 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e?versionId=null 2023-10-11T15:07:33.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T15:07:33.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T15:07:33.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c3049cb5be21f4-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.582 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.583 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.584 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.584 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.584 DEBUG:boto:Method: GET 2023-10-11T15:07:33.585 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:33.585 DEBUG:boto:Data: 2023-10-11T15:07:33.586 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.587 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.587 DEBUG:boto:Params: {} 2023-10-11T15:07:33.588 DEBUG:boto:Token: None 2023-10-11T15:07:33.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a855b71b857cba4e-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.592 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.593 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.593 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.594 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.594 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.595 DEBUG:boto:Method: GET 2023-10-11T15:07:33.595 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:33.595 DEBUG:boto:Data: 2023-10-11T15:07:33.596 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.597 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.597 DEBUG:boto:Params: {} 2023-10-11T15:07:33.598 DEBUG:boto:Token: None 2023-10-11T15:07:33.598 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000397c5ecb4278132-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.602 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.602 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.603 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.603 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.604 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:33.604 DEBUG:boto:Data: 2023-10-11T15:07:33.604 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.605 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.605 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.606 DEBUG:boto:Params: {} 2023-10-11T15:07:33.606 DEBUG:boto:Token: None 2023-10-11T15:07:33.607 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e 2023-10-11T15:07:33.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M= 2023-10-11T15:07:33.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M=', 'Content-Length': '0'} 2023-10-11T15:07:33.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333eb761e7f8e030-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.610 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.611 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.611 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.612 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:33.612 DEBUG:boto:Data: 2023-10-11T15:07:33.612 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.613 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.614 DEBUG:boto:Params: {} 2023-10-11T15:07:33.614 DEBUG:boto:Token: None 2023-10-11T15:07:33.615 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e 2023-10-11T15:07:33.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M= 2023-10-11T15:07:33.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M=', 'Content-Length': '0'} 2023-10-11T15:07:33.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1852090f0286031-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:07:33.619 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.619 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.620 DEBUG:boto:Method: GET 2023-10-11T15:07:33.620 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:33.621 DEBUG:boto:Data: 2023-10-11T15:07:33.621 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.622 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.622 DEBUG:boto:Params: {} 2023-10-11T15:07:33.623 DEBUG:boto:Token: None 2023-10-11T15:07:33.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e 2023-10-11T15:07:33.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc= 2023-10-11T15:07:33.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc=', 'Content-Length': '0'} 2023-10-11T15:07:33.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006134feac37a54dab-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.674 DEBUG:boto:path=/dghqvd-55/e 2023-10-11T15:07:33.675 DEBUG:boto:auth_path=/dghqvd-55/e 2023-10-11T15:07:33.676 DEBUG:boto:Method: GET 2023-10-11T15:07:33.676 DEBUG:boto:Path: /dghqvd-55/e 2023-10-11T15:07:33.676 DEBUG:boto:Data: 2023-10-11T15:07:33.677 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.678 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.678 DEBUG:boto:Params: {} 2023-10-11T15:07:33.679 DEBUG:boto:Token: None 2023-10-11T15:07:33.679 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/e 2023-10-11T15:07:33.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc= 2023-10-11T15:07:33.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc=', 'Content-Length': '0'} 2023-10-11T15:07:33.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca45b92f25edf48d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.683 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.684 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.684 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.685 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.685 DEBUG:boto:Method: GET 2023-10-11T15:07:33.686 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:33.686 DEBUG:boto:Data: 2023-10-11T15:07:33.687 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.688 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.688 DEBUG:boto:Params: {} 2023-10-11T15:07:33.688 DEBUG:boto:Token: None 2023-10-11T15:07:33.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f4ac4102e1d497-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.693 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.694 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.694 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.695 DEBUG:boto:path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.695 DEBUG:boto:auth_path=/dghqvd-55/?prefix=e 2023-10-11T15:07:33.696 DEBUG:boto:Method: GET 2023-10-11T15:07:33.696 DEBUG:boto:Path: /dghqvd-55/?prefix=e 2023-10-11T15:07:33.697 DEBUG:boto:Data: 2023-10-11T15:07:33.697 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.698 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.699 DEBUG:boto:Params: {} 2023-10-11T15:07:33.699 DEBUG:boto:Token: None 2023-10-11T15:07:33.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b403d9330144a794-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.703 DEBUG:boto:b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:33.705 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.705 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.705 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.706 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.706 DEBUG:boto:Method: GET 2023-10-11T15:07:33.707 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.707 DEBUG:boto:Data: 2023-10-11T15:07:33.708 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.709 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.709 DEBUG:boto:Params: {} 2023-10-11T15:07:33.710 DEBUG:boto:Token: None 2023-10-11T15:07:33.710 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T15:07:33.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T15:07:33.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448e0b3700cdafd0-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.714 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.714 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.715 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.715 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.716 DEBUG:boto:Method: GET 2023-10-11T15:07:33.716 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.716 DEBUG:boto:Data: 2023-10-11T15:07:33.717 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.718 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.718 DEBUG:boto:Params: {} 2023-10-11T15:07:33.719 DEBUG:boto:Token: None 2023-10-11T15:07:33.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T15:07:33.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T15:07:33.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d497fd86baaec894-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.762 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.763 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.763 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.764 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.764 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.765 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.765 DEBUG:boto:Data: 2023-10-11T15:07:33.766 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.767 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.767 DEBUG:boto:Params: {} 2023-10-11T15:07:33.768 DEBUG:boto:Token: None 2023-10-11T15:07:33.768 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM= 2023-10-11T15:07:33.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM=', 'Content-Length': '0'} 2023-10-11T15:07:33.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1a0233aea80d2a-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.772 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.772 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.773 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.773 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.773 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.774 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.774 DEBUG:boto:Data: 2023-10-11T15:07:33.775 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.775 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.777 DEBUG:boto:Params: {} 2023-10-11T15:07:33.778 DEBUG:boto:Token: None 2023-10-11T15:07:33.778 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM= 2023-10-11T15:07:33.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM=', 'Content-Length': '0'} 2023-10-11T15:07:33.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081c5e54f478e067d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:33.782 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.782 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.783 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.783 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.784 DEBUG:boto:Method: GET 2023-10-11T15:07:33.784 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.784 DEBUG:boto:Data: 2023-10-11T15:07:33.785 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.786 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.786 DEBUG:boto:Params: {} 2023-10-11T15:07:33.787 DEBUG:boto:Token: None 2023-10-11T15:07:33.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T15:07:33.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T15:07:33.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000416400f393662a8e-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.790 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.791 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.791 DEBUG:boto:path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.792 DEBUG:boto:auth_path=/dghqvd-55/f?versionId=null 2023-10-11T15:07:33.792 DEBUG:boto:Method: GET 2023-10-11T15:07:33.793 DEBUG:boto:Path: /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.793 DEBUG:boto:Data: 2023-10-11T15:07:33.794 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.795 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.795 DEBUG:boto:Params: {} 2023-10-11T15:07:33.795 DEBUG:boto:Token: None 2023-10-11T15:07:33.796 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f?versionId=null 2023-10-11T15:07:33.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T15:07:33.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T15:07:33.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027be72b6f1c179f6-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.800 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.800 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.801 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.801 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.802 DEBUG:boto:Method: GET 2023-10-11T15:07:33.802 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:33.802 DEBUG:boto:Data: 2023-10-11T15:07:33.803 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.804 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.804 DEBUG:boto:Params: {} 2023-10-11T15:07:33.805 DEBUG:boto:Token: None 2023-10-11T15:07:33.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8231413ad6dda71-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.809 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.810 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.810 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.811 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.811 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.812 DEBUG:boto:Method: GET 2023-10-11T15:07:33.812 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:33.812 DEBUG:boto:Data: 2023-10-11T15:07:33.813 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.813 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.814 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.814 DEBUG:boto:Params: {} 2023-10-11T15:07:33.815 DEBUG:boto:Token: None 2023-10-11T15:07:33.815 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec04010ac5b470c2-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.819 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.819 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.820 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.820 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.821 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:33.821 DEBUG:boto:Data: 2023-10-11T15:07:33.821 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.822 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.822 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.823 DEBUG:boto:Params: {} 2023-10-11T15:07:33.823 DEBUG:boto:Token: None 2023-10-11T15:07:33.823 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f 2023-10-11T15:07:33.824 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs= 2023-10-11T15:07:33.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs=', 'Content-Length': '0'} 2023-10-11T15:07:33.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4974ee1411d4a2d-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.827 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.827 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.828 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.828 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:33.828 DEBUG:boto:Data: 2023-10-11T15:07:33.829 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.830 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.830 DEBUG:boto:Params: {} 2023-10-11T15:07:33.830 DEBUG:boto:Token: None 2023-10-11T15:07:33.831 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f 2023-10-11T15:07:33.831 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs= 2023-10-11T15:07:33.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs=', 'Content-Length': '0'} 2023-10-11T15:07:33.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996ff6092546642d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:07:33.835 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.835 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.836 DEBUG:boto:Method: GET 2023-10-11T15:07:33.836 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:33.836 DEBUG:boto:Data: 2023-10-11T15:07:33.837 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.838 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.838 DEBUG:boto:Params: {} 2023-10-11T15:07:33.839 DEBUG:boto:Token: None 2023-10-11T15:07:33.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f 2023-10-11T15:07:33.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8= 2023-10-11T15:07:33.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8=', 'Content-Length': '0'} 2023-10-11T15:07:33.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e3b5247785ce4b-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.843 DEBUG:boto:path=/dghqvd-55/f 2023-10-11T15:07:33.843 DEBUG:boto:auth_path=/dghqvd-55/f 2023-10-11T15:07:33.843 DEBUG:boto:Method: GET 2023-10-11T15:07:33.844 DEBUG:boto:Path: /dghqvd-55/f 2023-10-11T15:07:33.844 DEBUG:boto:Data: 2023-10-11T15:07:33.845 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.846 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.846 DEBUG:boto:Params: {} 2023-10-11T15:07:33.847 DEBUG:boto:Token: None 2023-10-11T15:07:33.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/f 2023-10-11T15:07:33.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8= 2023-10-11T15:07:33.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8=', 'Content-Length': '0'} 2023-10-11T15:07:33.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63794c43d34a876-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.894 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.895 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.895 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.896 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.896 DEBUG:boto:Method: GET 2023-10-11T15:07:33.896 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:33.897 DEBUG:boto:Data: 2023-10-11T15:07:33.897 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.898 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.899 DEBUG:boto:Params: {} 2023-10-11T15:07:33.899 DEBUG:boto:Token: None 2023-10-11T15:07:33.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093dcaa6528fd5367-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.903 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.904 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:33.905 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:33.905 DEBUG:boto:path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.905 DEBUG:boto:auth_path=/dghqvd-55/?prefix=f 2023-10-11T15:07:33.906 DEBUG:boto:Method: GET 2023-10-11T15:07:33.906 DEBUG:boto:Path: /dghqvd-55/?prefix=f 2023-10-11T15:07:33.907 DEBUG:boto:Data: 2023-10-11T15:07:33.907 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.908 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.909 DEBUG:boto:Params: {} 2023-10-11T15:07:33.909 DEBUG:boto:Token: None 2023-10-11T15:07:33.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/ 2023-10-11T15:07:33.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T15:07:33.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T15:07:33.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdde13e60663426d-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.913 DEBUG:boto:b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:33.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:33.915 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:33.915 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:33.915 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.916 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.916 DEBUG:boto:Method: GET 2023-10-11T15:07:33.917 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.917 DEBUG:boto:Data: 2023-10-11T15:07:33.918 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.919 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.919 DEBUG:boto:Params: {} 2023-10-11T15:07:33.920 DEBUG:boto:Token: None 2023-10-11T15:07:33.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T15:07:33.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T15:07:33.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219d68e5c42e689d-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.966 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:33.967 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:33.967 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.968 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.968 DEBUG:boto:Method: GET 2023-10-11T15:07:33.969 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.969 DEBUG:boto:Data: 2023-10-11T15:07:33.970 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.971 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.971 DEBUG:boto:Params: {} 2023-10-11T15:07:33.972 DEBUG:boto:Token: None 2023-10-11T15:07:33.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T15:07:33.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T15:07:33.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d09cd8b690bddb-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.976 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:33.976 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:33.977 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.977 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.977 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.978 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.978 DEBUG:boto:Data: 2023-10-11T15:07:33.979 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.979 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.981 DEBUG:boto:Params: {} 2023-10-11T15:07:33.982 DEBUG:boto:Token: None 2023-10-11T15:07:33.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY= 2023-10-11T15:07:33.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY=', 'Content-Length': '0'} 2023-10-11T15:07:33.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e13e56ae050c26-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.986 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:33.986 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:33.986 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.987 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.987 DEBUG:boto:Method: HEAD 2023-10-11T15:07:33.988 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.988 DEBUG:boto:Data: 2023-10-11T15:07:33.988 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:33.989 DEBUG:boto:Port: 8000 2023-10-11T15:07:33.990 DEBUG:boto:Params: {} 2023-10-11T15:07:33.990 DEBUG:boto:Token: None 2023-10-11T15:07:33.991 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY= 2023-10-11T15:07:33.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY=', 'Content-Length': '0'} 2023-10-11T15:07:33.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37252c2ff690e55-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:33.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:33.994 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:33.995 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:33.995 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.996 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:33.996 DEBUG:boto:Method: GET 2023-10-11T15:07:33.996 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:33.997 DEBUG:boto:Data: 2023-10-11T15:07:33.997 DEBUG:boto:Headers: {} 2023-10-11T15:07:33.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:33.998 DEBUG:boto:Port: 8001 2023-10-11T15:07:33.999 DEBUG:boto:Params: {} 2023-10-11T15:07:33.999 DEBUG:boto:Token: None 2023-10-11T15:07:33.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:33 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:34.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T15:07:34.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T15:07:34.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bac85b6e84bfbfe-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.003 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:34.003 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:34.004 DEBUG:boto:path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:34.004 DEBUG:boto:auth_path=/dghqvd-55/g?versionId=null 2023-10-11T15:07:34.005 DEBUG:boto:Method: GET 2023-10-11T15:07:34.005 DEBUG:boto:Path: /dghqvd-55/g?versionId=null 2023-10-11T15:07:34.006 DEBUG:boto:Data: 2023-10-11T15:07:34.006 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.007 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.007 DEBUG:boto:Params: {} 2023-10-11T15:07:34.008 DEBUG:boto:Token: None 2023-10-11T15:07:34.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/g?versionId=null 2023-10-11T15:07:34.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pv1nZn9beWkF9tgTBJdVJ/1NGWA= 2023-10-11T15:07:34.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pv1nZn9beWkF9tgTBJdVJ/1NGWA=', 'Content-Length': '0'} 2023-10-11T15:07:34.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150a4eb15b3feb0a-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.012 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.012 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.013 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.013 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.014 DEBUG:boto:Method: GET 2023-10-11T15:07:34.014 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:34.014 DEBUG:boto:Data: 2023-10-11T15:07:34.015 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.016 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.016 DEBUG:boto:Params: {} 2023-10-11T15:07:34.017 DEBUG:boto:Token: None 2023-10-11T15:07:34.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0784fd713df118-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.021 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.022 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.022 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.023 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.023 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.024 DEBUG:boto:Method: GET 2023-10-11T15:07:34.024 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:34.024 DEBUG:boto:Data: 2023-10-11T15:07:34.025 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.026 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.026 DEBUG:boto:Params: {} 2023-10-11T15:07:34.027 DEBUG:boto:Token: None 2023-10-11T15:07:34.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.028 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173cc7635199e6bf-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.070 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.071 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:34.071 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:34.072 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.072 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:34.073 DEBUG:boto:Data: 2023-10-11T15:07:34.073 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.073 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.074 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.074 DEBUG:boto:Params: {} 2023-10-11T15:07:34.075 DEBUG:boto:Token: None 2023-10-11T15:07:34.075 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/g 2023-10-11T15:07:34.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o= 2023-10-11T15:07:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o=', 'Content-Length': '0'} 2023-10-11T15:07:34.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1596d0366a5c526-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.079 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:34.079 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:34.079 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.080 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:34.080 DEBUG:boto:Data: 2023-10-11T15:07:34.081 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.081 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.082 DEBUG:boto:Params: {} 2023-10-11T15:07:34.082 DEBUG:boto:Token: None 2023-10-11T15:07:34.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/g 2023-10-11T15:07:34.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o= 2023-10-11T15:07:34.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o=', 'Content-Length': '0'} 2023-10-11T15:07:34.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06e3d97296578ee-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:07:34.086 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:34.087 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:34.087 DEBUG:boto:Method: GET 2023-10-11T15:07:34.088 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:34.088 DEBUG:boto:Data: 2023-10-11T15:07:34.088 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.089 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.090 DEBUG:boto:Params: {} 2023-10-11T15:07:34.090 DEBUG:boto:Token: None 2023-10-11T15:07:34.091 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/g 2023-10-11T15:07:34.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ= 2023-10-11T15:07:34.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ=', 'Content-Length': '0'} 2023-10-11T15:07:34.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b28eea3cc31957-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.094 DEBUG:boto:path=/dghqvd-55/g 2023-10-11T15:07:34.094 DEBUG:boto:auth_path=/dghqvd-55/g 2023-10-11T15:07:34.095 DEBUG:boto:Method: GET 2023-10-11T15:07:34.095 DEBUG:boto:Path: /dghqvd-55/g 2023-10-11T15:07:34.096 DEBUG:boto:Data: 2023-10-11T15:07:34.096 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.097 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.097 DEBUG:boto:Params: {} 2023-10-11T15:07:34.098 DEBUG:boto:Token: None 2023-10-11T15:07:34.098 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/g 2023-10-11T15:07:34.099 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ= 2023-10-11T15:07:34.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ=', 'Content-Length': '0'} 2023-10-11T15:07:34.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032de5db1fe39e2ed-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.102 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.102 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.103 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.103 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.103 DEBUG:boto:Method: GET 2023-10-11T15:07:34.104 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:34.104 DEBUG:boto:Data: 2023-10-11T15:07:34.105 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.105 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.105 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.106 DEBUG:boto:Params: {} 2023-10-11T15:07:34.106 DEBUG:boto:Token: None 2023-10-11T15:07:34.107 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.107 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.110 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008291425e12f8f01-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.110 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.111 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.111 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.112 DEBUG:boto:path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.112 DEBUG:boto:auth_path=/dghqvd-55/?prefix=g 2023-10-11T15:07:34.113 DEBUG:boto:Method: GET 2023-10-11T15:07:34.113 DEBUG:boto:Path: /dghqvd-55/?prefix=g 2023-10-11T15:07:34.114 DEBUG:boto:Data: 2023-10-11T15:07:34.114 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.115 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.115 DEBUG:boto:Params: {} 2023-10-11T15:07:34.116 DEBUG:boto:Token: None 2023-10-11T15:07:34.116 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9cd23e42ec19d3-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.120 DEBUG:boto:b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:34.121 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.121 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.122 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.122 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.122 DEBUG:boto:Method: GET 2023-10-11T15:07:34.123 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.123 DEBUG:boto:Data: 2023-10-11T15:07:34.124 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.125 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.125 DEBUG:boto:Params: {} 2023-10-11T15:07:34.125 DEBUG:boto:Token: None 2023-10-11T15:07:34.126 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T15:07:34.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T15:07:34.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1eb0cfc959ab05b-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.129 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.130 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.130 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.131 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.131 DEBUG:boto:Method: GET 2023-10-11T15:07:34.131 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.132 DEBUG:boto:Data: 2023-10-11T15:07:34.132 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.133 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.134 DEBUG:boto:Params: {} 2023-10-11T15:07:34.134 DEBUG:boto:Token: None 2023-10-11T15:07:34.135 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T15:07:34.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T15:07:34.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a169efc1e24191-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.139 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.140 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.140 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.140 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.141 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.142 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.142 DEBUG:boto:Data: 2023-10-11T15:07:34.143 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.143 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.143 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.144 DEBUG:boto:Params: {} 2023-10-11T15:07:34.144 DEBUG:boto:Token: None 2023-10-11T15:07:34.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.145 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw= 2023-10-11T15:07:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw=', 'Content-Length': '0'} 2023-10-11T15:07:34.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f1a9d2e9c2ceb9-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.148 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.149 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.149 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.149 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.150 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.150 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.151 DEBUG:boto:Data: 2023-10-11T15:07:34.151 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.152 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.152 DEBUG:boto:Params: {} 2023-10-11T15:07:34.153 DEBUG:boto:Token: None 2023-10-11T15:07:34.153 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw= 2023-10-11T15:07:34.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw=', 'Content-Length': '0'} 2023-10-11T15:07:34.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af180b49eabce22e-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:34.157 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.157 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.158 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.158 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.159 DEBUG:boto:Method: GET 2023-10-11T15:07:34.159 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.160 DEBUG:boto:Data: 2023-10-11T15:07:34.160 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.161 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.161 DEBUG:boto:Params: {} 2023-10-11T15:07:34.162 DEBUG:boto:Token: None 2023-10-11T15:07:34.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T15:07:34.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T15:07:34.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f137f25f5bfa1f-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.166 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.166 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.167 DEBUG:boto:path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.167 DEBUG:boto:auth_path=/dghqvd-55/h?versionId=null 2023-10-11T15:07:34.168 DEBUG:boto:Method: GET 2023-10-11T15:07:34.168 DEBUG:boto:Path: /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.168 DEBUG:boto:Data: 2023-10-11T15:07:34.169 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.170 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.170 DEBUG:boto:Params: {} 2023-10-11T15:07:34.171 DEBUG:boto:Token: None 2023-10-11T15:07:34.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h?versionId=null 2023-10-11T15:07:34.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T15:07:34.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T15:07:34.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233275cffee02586-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.214 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.215 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.215 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.216 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.216 DEBUG:boto:Method: GET 2023-10-11T15:07:34.216 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:34.217 DEBUG:boto:Data: 2023-10-11T15:07:34.217 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.218 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.219 DEBUG:boto:Params: {} 2023-10-11T15:07:34.219 DEBUG:boto:Token: None 2023-10-11T15:07:34.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae2372aa79f6111-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.224 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.224 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.225 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.225 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.226 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.226 DEBUG:boto:Method: GET 2023-10-11T15:07:34.227 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:34.227 DEBUG:boto:Data: 2023-10-11T15:07:34.228 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.229 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.229 DEBUG:boto:Params: {} 2023-10-11T15:07:34.229 DEBUG:boto:Token: None 2023-10-11T15:07:34.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2f62c65b41f4aa-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.234 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.235 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.235 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.235 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.236 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:34.236 DEBUG:boto:Data: 2023-10-11T15:07:34.237 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.237 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.238 DEBUG:boto:Params: {} 2023-10-11T15:07:34.239 DEBUG:boto:Token: None 2023-10-11T15:07:34.239 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h 2023-10-11T15:07:34.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A= 2023-10-11T15:07:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A=', 'Content-Length': '0'} 2023-10-11T15:07:34.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c822e375b84e81-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.242 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.243 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.243 DEBUG:boto:Method: HEAD 2023-10-11T15:07:34.243 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:34.244 DEBUG:boto:Data: 2023-10-11T15:07:34.244 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.245 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.246 DEBUG:boto:Params: {} 2023-10-11T15:07:34.246 DEBUG:boto:Token: None 2023-10-11T15:07:34.246 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h 2023-10-11T15:07:34.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A= 2023-10-11T15:07:34.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A=', 'Content-Length': '0'} 2023-10-11T15:07:34.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099524b4eee559aa5-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:07:34.250 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.251 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.251 DEBUG:boto:Method: GET 2023-10-11T15:07:34.251 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:34.252 DEBUG:boto:Data: 2023-10-11T15:07:34.252 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.253 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.253 DEBUG:boto:Params: {} 2023-10-11T15:07:34.254 DEBUG:boto:Token: None 2023-10-11T15:07:34.254 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h 2023-10-11T15:07:34.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE= 2023-10-11T15:07:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE=', 'Content-Length': '0'} 2023-10-11T15:07:34.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ce58d63d1dd29b-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.299 DEBUG:boto:path=/dghqvd-55/h 2023-10-11T15:07:34.299 DEBUG:boto:auth_path=/dghqvd-55/h 2023-10-11T15:07:34.300 DEBUG:boto:Method: GET 2023-10-11T15:07:34.300 DEBUG:boto:Path: /dghqvd-55/h 2023-10-11T15:07:34.301 DEBUG:boto:Data: 2023-10-11T15:07:34.301 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.302 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.302 DEBUG:boto:Params: {} 2023-10-11T15:07:34.303 DEBUG:boto:Token: None 2023-10-11T15:07:34.303 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/h 2023-10-11T15:07:34.304 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE= 2023-10-11T15:07:34.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE=', 'Content-Length': '0'} 2023-10-11T15:07:34.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f492461f8fd83e-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.307 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.307 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.308 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.308 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.308 DEBUG:boto:Method: GET 2023-10-11T15:07:34.309 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:34.309 DEBUG:boto:Data: 2023-10-11T15:07:34.310 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:07:34.311 DEBUG:boto:Port: 8001 2023-10-11T15:07:34.311 DEBUG:boto:Params: {} 2023-10-11T15:07:34.311 DEBUG:boto:Token: None 2023-10-11T15:07:34.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499be8118ed35ba8-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.316 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.317 DEBUG:boto:path=/dghqvd-55/ 2023-10-11T15:07:34.317 DEBUG:boto:auth_path=/dghqvd-55/ 2023-10-11T15:07:34.318 DEBUG:boto:path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.318 DEBUG:boto:auth_path=/dghqvd-55/?prefix=h 2023-10-11T15:07:34.318 DEBUG:boto:Method: GET 2023-10-11T15:07:34.319 DEBUG:boto:Path: /dghqvd-55/?prefix=h 2023-10-11T15:07:34.319 DEBUG:boto:Data: 2023-10-11T15:07:34.320 DEBUG:boto:Headers: {} 2023-10-11T15:07:34.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:34.321 DEBUG:boto:Port: 8000 2023-10-11T15:07:34.321 DEBUG:boto:Params: {} 2023-10-11T15:07:34.322 DEBUG:boto:Token: None 2023-10-11T15:07:34.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:07:34 GMT /dghqvd-55/ 2023-10-11T15:07:34.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T15:07:34.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T15:07:34.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088eacfde3c410827-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:34.325 DEBUG:boto:b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:07:34.326 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T15:07:34.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.327 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:34.375 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:34.376 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:34.418 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.417+0000 7fc432371700 -1 req 1802128087161529062 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.419 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.417+0000 7fc384a16700 -1 req 4206395784897013624 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.419 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.417+0000 7fc33717b700 -1 req 10502105560425516689 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.422 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.418+0000 7fc42e369700 -1 req 1278840046035576592 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.422 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.419+0000 7fc41c345700 -1 req 5401398298012547829 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.462 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.460+0000 7fc415b38700 -1 req 1360939669116435187 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.462 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.461+0000 7fc427b5c700 -1 req 2775820898864771056 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.467 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.465+0000 7fc413b34700 -1 req 12211526065910358041 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.504 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.502+0000 7fc422351700 -1 req 1475105592868418938 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.508 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.507+0000 7fc40f32b700 -1 req 501008692447420487 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.577 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:34.624 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:34.666 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:07:34.666 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.666 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding": "None", 2023-10-11T15:07:34.667 INFO:teuthology.orchestra.run.smithi158.stdout: "current_index": { 2023-10-11T15:07:34.667 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:34.667 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.667 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "Normal", 2023-10-11T15:07:34.668 INFO:teuthology.orchestra.run.smithi158.stdout: "normal": { 2023-10-11T15:07:34.668 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:34.668 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:34.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:34.669 INFO:teuthology.orchestra.run.smithi158.stdout: "logs": [ 2023-10-11T15:07:34.669 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:34.669 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:34.669 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.670 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:34.670 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:34.670 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:34.670 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.671 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:34.671 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:34.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.672 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:34.672 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:34.672 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:34.672 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.673 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:34.673 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:34.673 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:34.673 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:34.673 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:34.674 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:34.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.675 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.675 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:07:34.675 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:34.675 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:07:34.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.677 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:34.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:34.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:34.761 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.759+0000 7fc403313700 -1 req 4490894029272329692 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.767 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.765+0000 7fc413333700 -1 req 9408799144605805262 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.767 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.766+0000 7fc3b1a70700 -1 req 14024864955847415538 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.813 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.811+0000 7fc3a3a54700 -1 req 6312820520116418527 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.814 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.813+0000 7fc3e32d3700 -1 req 2451397968902671857 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.858 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:07:34.856+0000 7fc3ab263700 -1 req 5244464425991346776 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T15:07:34.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:07:34.920 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:07:34.966 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:34.966 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:35.005 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:07:35.006 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.006 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding": "None", 2023-10-11T15:07:35.006 INFO:teuthology.orchestra.run.smithi158.stdout: "current_index": { 2023-10-11T15:07:35.007 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:35.007 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.007 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "Normal", 2023-10-11T15:07:35.007 INFO:teuthology.orchestra.run.smithi158.stdout: "normal": { 2023-10-11T15:07:35.007 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:35.008 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:35.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.008 INFO:teuthology.orchestra.run.smithi158.stdout: "logs": [ 2023-10-11T15:07:35.009 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.009 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:35.009 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.009 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:35.010 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:35.010 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 1, 2023-10-11T15:07:35.010 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.010 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:35.010 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:35.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.011 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.012 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:35.012 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.012 INFO:teuthology.orchestra.run.smithi158.stdout: "type": "InIndex", 2023-10-11T15:07:35.012 INFO:teuthology.orchestra.run.smithi158.stdout: "in_index": { 2023-10-11T15:07:35.013 INFO:teuthology.orchestra.run.smithi158.stdout: "gen": 2, 2023-10-11T15:07:35.013 INFO:teuthology.orchestra.run.smithi158.stdout: "layout": { 2023-10-11T15:07:35.013 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 3, 2023-10-11T15:07:35.013 INFO:teuthology.orchestra.run.smithi158.stdout: "hash_type": "Mod" 2023-10-11T15:07:35.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.014 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:07:35.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.015 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:07:35.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-10-11T15:07:35.020 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:07:35.021 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:07:35.021 DEBUG:boto:Method: PUT 2023-10-11T15:07:35.022 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:07:35.022 DEBUG:boto:Data: 2023-10-11T15:07:35.023 DEBUG:boto:Headers: {} 2023-10-11T15:07:35.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:35.023 DEBUG:boto:Port: 8000 2023-10-11T15:07:35.024 DEBUG:boto:Params: {} 2023-10-11T15:07:35.024 DEBUG:boto:Token: None 2023-10-11T15:07:35.025 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:07:35 GMT /dghqvd-56/ 2023-10-11T15:07:35.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vRBPXTwxe5uMLCwBA9k65m3LQbc= 2023-10-11T15:07:35.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vRBPXTwxe5uMLCwBA9k65m3LQbc=', 'Content-Length': '0'} 2023-10-11T15:07:35.040 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000ac46d3d20ae1753-006526ba37-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:35.040 DEBUG:rgw_multi.tests:created bucket=dghqvd-56 2023-10-11T15:07:35.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:35.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:35.042 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:07:35.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:35.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:35.119 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:07:35.131 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:07:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:07:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:07:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:07:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:07:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:07:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:07:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036855.037734_407.1", 2023-10-11T15:07:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:07:35.037734Z" 2023-10-11T15:07:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.135 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:07:35.136 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036855.037734_407.1'} 2023-10-11T15:07:35.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:07:35.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:35.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:35.138 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:07:35.193 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:35.194 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:35.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:35.223+0000 7f2332b1a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:07:35.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:35.224+0000 7f2332b1a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:07:35.238 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:07:35.239 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:07:35.239 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:07:35.239 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:07:35.240 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:07:35.240 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:07:35.240 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:35.240 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.240 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:07:35.241 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.241 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:07:35.241 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:35.241 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:35.242 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:07:35.242 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:35.242 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:07:35.242 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:35.242 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:07:35.243 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:35.243 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.243 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.243 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.244 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:07:35.244 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:35.244 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:35.244 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:07:35.244 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:35.245 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:35.245 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:35.245 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:07:35.245 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:35.245 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.246 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.246 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.246 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:07:35.246 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:35.247 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:35.247 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:07:35.247 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:35.247 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:35.247 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:35.248 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:07:35.248 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:35.248 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.249 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:35.249 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:07:35.249 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:35.249 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:35.249 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036854.423950_402.1", 2023-10-11T15:07:35.250 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:35.250 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:35.250 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:35.250 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:07:34.423950Z", 2023-10-11T15:07:35.251 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:35.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:35.251 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:07:35.251 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.252 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:07:35.252 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:07:35.252 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:07:35.252 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:35.252 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:07:35Z" 2023-10-11T15:07:35.253 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:07:35.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036854.423950_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:07:34.423950Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:07:35Z" } 2023-10-11T15:07:35.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036854.423950_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:07:34.423950Z', 'realm_epoch': 2}}] 2023-10-11T15:07:35.255 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036855.037734_407.1'} 2023-10-11T15:07:35.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036854.423950_402.1'} 2023-10-11T15:07:35.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697036855.037734_407.1 target=1_1697036854.423950_402.1 2023-10-11T15:07:40.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:40.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:07:40.258 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:07:40.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:40.316 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:40.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:40.345+0000 7f8c2b497780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:07:40.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:40.346+0000 7f8c2b497780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:07:40.362 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:07:40.362 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:07:40.362 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:07:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:07:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:07:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:07:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:40.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.364 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:07:40.364 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:40.364 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:07:40.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:40.365 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:40.365 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:07:40.365 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:40.365 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:07:40.365 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:40.366 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:07:40.366 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:40.366 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:40.366 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:40.367 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:07:40.367 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:40.367 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:40.367 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:07:40.367 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:40.368 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:40.368 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:40.368 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:07:40.368 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:40.369 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:40.369 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.369 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:40.369 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:07:40.369 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:40.370 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:40.370 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:07:40.370 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:40.370 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:40.371 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:40.371 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:07:40.371 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:40.371 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:40.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:07:40.372 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:07:40.372 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:07:40.372 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:07:40.372 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036855.037734_407.1", 2023-10-11T15:07:40.373 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:07:40.373 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:07:40.373 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:07:40.373 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:07:35.037734Z", 2023-10-11T15:07:40.373 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:07:40.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:40.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:07:40.374 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:07:40.374 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.374 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:07:40.375 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:07:40.375 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:07:40.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:07:40.375 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:07:40Z" 2023-10-11T15:07:40.375 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:07:40.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036855.037734_407.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:07:35.037734Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:07:40Z" } 2023-10-11T15:07:40.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036855.037734_407.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:07:35.037734Z', 'realm_epoch': 2}}] 2023-10-11T15:07:40.377 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036855.037734_407.1'} 2023-10-11T15:07:40.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036855.037734_407.1'} 2023-10-11T15:07:40.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:07:40.379 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:07:40.379 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:07:40.379 DEBUG:boto:Method: HEAD 2023-10-11T15:07:40.380 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:07:40.380 DEBUG:boto:Data: 2023-10-11T15:07:40.380 DEBUG:boto:Headers: {} 2023-10-11T15:07:40.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.381 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.382 DEBUG:boto:Params: {} 2023-10-11T15:07:40.382 DEBUG:boto:Token: None 2023-10-11T15:07:40.382 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/ 2023-10-11T15:07:40.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs= 2023-10-11T15:07:40.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs=', 'Content-Length': '0'} 2023-10-11T15:07:40.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceab6b0ff559871a-006526ba3c-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.386 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:07:40.387 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:07:40.387 DEBUG:boto:Method: PUT 2023-10-11T15:07:40.388 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:07:40.388 DEBUG:boto:Data: 2023-10-11T15:07:40.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:07:40.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.389 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.390 DEBUG:boto:Params: {} 2023-10-11T15:07:40.390 DEBUG:boto:Token: None 2023-10-11T15:07:40.391 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/a 2023-10-11T15:07:40.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Nx3lIAXfOMaO2aHv4018w55nOWY= 2023-10-11T15:07:40.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Nx3lIAXfOMaO2aHv4018w55nOWY='} 2023-10-11T15:07:40.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000588c26b15053b41-006526ba3c-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.398 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:07:40.399 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:07:40.399 DEBUG:boto:Method: HEAD 2023-10-11T15:07:40.399 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:07:40.400 DEBUG:boto:Data: 2023-10-11T15:07:40.400 DEBUG:boto:Headers: {} 2023-10-11T15:07:40.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.401 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.401 DEBUG:boto:Params: {} 2023-10-11T15:07:40.402 DEBUG:boto:Token: None 2023-10-11T15:07:40.402 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/ 2023-10-11T15:07:40.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs= 2023-10-11T15:07:40.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs=', 'Content-Length': '0'} 2023-10-11T15:07:40.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85eec005e384a8b-006526ba3c-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.406 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:07:40.407 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:07:40.407 DEBUG:boto:Method: PUT 2023-10-11T15:07:40.407 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:07:40.408 DEBUG:boto:Data: 2023-10-11T15:07:40.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:07:40.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.409 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.409 DEBUG:boto:Params: {} 2023-10-11T15:07:40.410 DEBUG:boto:Token: None 2023-10-11T15:07:40.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/b 2023-10-11T15:07:40.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m6S8XjUqLQjImuFiKIoe/Eh0ubQ= 2023-10-11T15:07:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m6S8XjUqLQjImuFiKIoe/Eh0ubQ='} 2023-10-11T15:07:40.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000857fefe40ebf5862-006526ba3c-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.455 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:07:40.455 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:07:40.456 DEBUG:boto:Method: HEAD 2023-10-11T15:07:40.456 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:07:40.456 DEBUG:boto:Data: 2023-10-11T15:07:40.457 DEBUG:boto:Headers: {} 2023-10-11T15:07:40.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.458 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.458 DEBUG:boto:Params: {} 2023-10-11T15:07:40.458 DEBUG:boto:Token: None 2023-10-11T15:07:40.459 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/ 2023-10-11T15:07:40.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs= 2023-10-11T15:07:40.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs=', 'Content-Length': '0'} 2023-10-11T15:07:40.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea6637ba10140c55-006526ba3c-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.463 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:07:40.463 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:07:40.463 DEBUG:boto:Method: PUT 2023-10-11T15:07:40.464 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:07:40.464 DEBUG:boto:Data: 2023-10-11T15:07:40.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:07:40.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.465 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.466 DEBUG:boto:Params: {} 2023-10-11T15:07:40.466 DEBUG:boto:Token: None 2023-10-11T15:07:40.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/c 2023-10-11T15:07:40.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wn8gwhEgb03XukJgaAXh89S4XI4= 2023-10-11T15:07:40.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wn8gwhEgb03XukJgaAXh89S4XI4='} 2023-10-11T15:07:40.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b82b8c3985e681-006526ba3c-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.473 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:07:40.474 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:07:40.474 DEBUG:boto:Method: HEAD 2023-10-11T15:07:40.474 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:07:40.475 DEBUG:boto:Data: 2023-10-11T15:07:40.475 DEBUG:boto:Headers: {} 2023-10-11T15:07:40.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.476 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.476 DEBUG:boto:Params: {} 2023-10-11T15:07:40.477 DEBUG:boto:Token: None 2023-10-11T15:07:40.477 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/ 2023-10-11T15:07:40.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs= 2023-10-11T15:07:40.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eze+Xq/TQgHtgkNpCmvNMPtXlgs=', 'Content-Length': '0'} 2023-10-11T15:07:40.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028b8e0bb607195c4-006526ba3c-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.481 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:07:40.481 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:07:40.481 DEBUG:boto:Method: PUT 2023-10-11T15:07:40.482 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:07:40.482 DEBUG:boto:Data: 2023-10-11T15:07:40.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:07:40.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:07:40.483 DEBUG:boto:Port: 8000 2023-10-11T15:07:40.484 DEBUG:boto:Params: {} 2023-10-11T15:07:40.484 DEBUG:boto:Token: None 2023-10-11T15:07:40.484 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:07:40 GMT /dghqvd-56/d 2023-10-11T15:07:40.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y1LWZG4DOMwBrDgqQOx6okh7Tjw= 2023-10-11T15:07:40.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:07:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y1LWZG4DOMwBrDgqQOx6okh7Tjw='} 2023-10-11T15:07:40.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000440c609e810655b6-006526ba3c-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:07:40.490 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-56 2023-10-11T15:07:40.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:40.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:40.492 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:07:40.548 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:40.549 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:40.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:40.578+0000 7f88b7c6f780 1 waiting to reach incremental sync.. 2023-10-11T15:07:45.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:45.579+0000 7f88b7c6f780 1 waiting to reach incremental sync.. 2023-10-11T15:07:50.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:50.581+0000 7f88b7c6f780 1 bucket sync caught up with source: 2023-10-11T15:07:50.584 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.42.6, , , , 00000000001.63.6, , , 00000000001.16.6, , 00000000001.37.6] 2023-10-11T15:07:50.584 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.42.6, , , , 00000000001.63.6, , , 00000000001.16.6, , 00000000001.37.6] 2023-10-11T15:07:50.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:50.581+0000 7f88b7c6f780 0 bucket checkpoint complete 2023-10-11T15:07:50.600 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-56 2023-10-11T15:07:50.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:50.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:07:50.602 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:07:50.664 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:07:50.665 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:07:50.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:50.699+0000 7f9bc20f3780 1 waiting to reach incremental sync.. 2023-10-11T15:07:55.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:07:55.700+0000 7f9bc20f3780 1 waiting to reach incremental sync.. 2023-10-11T15:08:00.703 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:00.701+0000 7f9bc20f3780 1 waiting to reach incremental sync.. 2023-10-11T15:08:05.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:05.701+0000 7f9bc20f3780 1 waiting to reach incremental sync.. 2023-10-11T15:08:10.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:10.703+0000 7f9bc20f3780 1 bucket sync caught up with source: 2023-10-11T15:08:10.706 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, , , , 00000000001.292.6, , , 00000000001.121.6, , 00000000001.103.6] 2023-10-11T15:08:10.706 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.215.6, , , , 00000000001.292.6, , , 00000000001.121.6, , 00000000001.103.6] 2023-10-11T15:08:10.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:10.703+0000 7f9bc20f3780 0 bucket checkpoint complete 2023-10-11T15:08:10.722 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-56 zones={a2, a1} 2023-10-11T15:08:10.723 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.723 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.724 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.724 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:10.725 DEBUG:boto:Data: 2023-10-11T15:08:10.725 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.725 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.726 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.726 DEBUG:boto:Params: {} 2023-10-11T15:08:10.727 DEBUG:boto:Token: None 2023-10-11T15:08:10.727 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/ 2023-10-11T15:08:10.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:56bIUyFIjrfGGpxDeVaDF4IMiDY= 2023-10-11T15:08:10.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:56bIUyFIjrfGGpxDeVaDF4IMiDY=', 'Content-Length': '0'} 2023-10-11T15:08:10.733 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052e9c3a216afce78-006526ba5a-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.734 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.735 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.735 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.735 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:10.736 DEBUG:boto:Data: 2023-10-11T15:08:10.736 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.737 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.737 DEBUG:boto:Params: {} 2023-10-11T15:08:10.738 DEBUG:boto:Token: None 2023-10-11T15:08:10.738 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/ 2023-10-11T15:08:10.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:56bIUyFIjrfGGpxDeVaDF4IMiDY= 2023-10-11T15:08:10.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:56bIUyFIjrfGGpxDeVaDF4IMiDY=', 'Content-Length': '0'} 2023-10-11T15:08:10.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8b5af835d57d12a-006526ba5a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.744 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:08:10.744 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.745 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.745 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.745 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.746 DEBUG:boto:Method: GET 2023-10-11T15:08:10.746 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.747 DEBUG:boto:Data: 2023-10-11T15:08:10.747 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.748 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.749 DEBUG:boto:Params: {} 2023-10-11T15:08:10.749 DEBUG:boto:Token: None 2023-10-11T15:08:10.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/?versions 2023-10-11T15:08:10.750 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs= 2023-10-11T15:08:10.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs=', 'Content-Length': '0'} 2023-10-11T15:08:10.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005058a45c15bbef9a-006526ba5a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.754 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.755 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:08:10.756 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:08:10.756 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:08:10.756 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:08:10.757 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:08:10.757 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.758 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.758 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.759 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.759 DEBUG:boto:Method: GET 2023-10-11T15:08:10.759 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.760 DEBUG:boto:Data: 2023-10-11T15:08:10.760 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.761 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.762 DEBUG:boto:Params: {} 2023-10-11T15:08:10.762 DEBUG:boto:Token: None 2023-10-11T15:08:10.763 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/?versions 2023-10-11T15:08:10.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs= 2023-10-11T15:08:10.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs=', 'Content-Length': '0'} 2023-10-11T15:08:10.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004136f97b7b8f99b0-006526ba5a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.767 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.768 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:08:10.768 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:08:10.768 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:08:10.769 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:08:10.769 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.770 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.770 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.771 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.771 DEBUG:boto:Method: GET 2023-10-11T15:08:10.772 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.772 DEBUG:boto:Data: 2023-10-11T15:08:10.773 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.773 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.774 DEBUG:boto:Params: {} 2023-10-11T15:08:10.774 DEBUG:boto:Token: None 2023-10-11T15:08:10.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/?versions 2023-10-11T15:08:10.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs= 2023-10-11T15:08:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs=', 'Content-Length': '0'} 2023-10-11T15:08:10.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfb437bda2a128c0-006526ba5a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.780 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.780 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.781 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.782 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.782 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.783 DEBUG:boto:Method: GET 2023-10-11T15:08:10.783 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:08:10.784 DEBUG:boto:Data: 2023-10-11T15:08:10.784 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.785 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.785 DEBUG:boto:Params: {} 2023-10-11T15:08:10.786 DEBUG:boto:Token: None 2023-10-11T15:08:10.786 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/?versions 2023-10-11T15:08:10.787 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs= 2023-10-11T15:08:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:INx+95iKAdk44oXAv/Kl/1wwMqs=', 'Content-Length': '0'} 2023-10-11T15:08:10.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824353d91b6098e2-006526ba5a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.791 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:08:10.793 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.794 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.794 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.795 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.795 DEBUG:boto:Method: GET 2023-10-11T15:08:10.796 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.796 DEBUG:boto:Data: 2023-10-11T15:08:10.796 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.797 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.798 DEBUG:boto:Params: {} 2023-10-11T15:08:10.798 DEBUG:boto:Token: None 2023-10-11T15:08:10.799 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0= 2023-10-11T15:08:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0=', 'Content-Length': '0'} 2023-10-11T15:08:10.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f579161dece6e0-006526ba5a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.850 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.851 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.852 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.852 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.852 DEBUG:boto:Method: GET 2023-10-11T15:08:10.853 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.853 DEBUG:boto:Data: 2023-10-11T15:08:10.854 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.855 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.855 DEBUG:boto:Params: {} 2023-10-11T15:08:10.856 DEBUG:boto:Token: None 2023-10-11T15:08:10.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0= 2023-10-11T15:08:10.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0=', 'Content-Length': '0'} 2023-10-11T15:08:10.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761ec437634edff2-006526ba5a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.860 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.860 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.861 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.861 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.862 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.862 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.862 DEBUG:boto:Data: 2023-10-11T15:08:10.863 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.864 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.864 DEBUG:boto:Params: {} 2023-10-11T15:08:10.864 DEBUG:boto:Token: None 2023-10-11T15:08:10.865 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LMFd9ZYdn06mT2yJ8M6sXh9P+FI= 2023-10-11T15:08:10.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LMFd9ZYdn06mT2yJ8M6sXh9P+FI=', 'Content-Length': '0'} 2023-10-11T15:08:10.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096fd9840ea39ac35-006526ba5a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.868 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.869 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.869 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.870 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.870 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.870 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.871 DEBUG:boto:Data: 2023-10-11T15:08:10.871 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.872 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.872 DEBUG:boto:Params: {} 2023-10-11T15:08:10.873 DEBUG:boto:Token: None 2023-10-11T15:08:10.873 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LMFd9ZYdn06mT2yJ8M6sXh9P+FI= 2023-10-11T15:08:10.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LMFd9ZYdn06mT2yJ8M6sXh9P+FI=', 'Content-Length': '0'} 2023-10-11T15:08:10.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c6946f595bfca7f-006526ba5a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:08:10.877 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.878 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.878 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.879 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.879 DEBUG:boto:Method: GET 2023-10-11T15:08:10.880 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.880 DEBUG:boto:Data: 2023-10-11T15:08:10.880 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.881 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.882 DEBUG:boto:Params: {} 2023-10-11T15:08:10.882 DEBUG:boto:Token: None 2023-10-11T15:08:10.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0= 2023-10-11T15:08:10.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0=', 'Content-Length': '0'} 2023-10-11T15:08:10.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b881365176b2c8-006526ba5a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.886 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.887 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.887 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.888 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:08:10.888 DEBUG:boto:Method: GET 2023-10-11T15:08:10.888 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.889 DEBUG:boto:Data: 2023-10-11T15:08:10.889 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.890 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.891 DEBUG:boto:Params: {} 2023-10-11T15:08:10.891 DEBUG:boto:Token: None 2023-10-11T15:08:10.892 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:08:10.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0= 2023-10-11T15:08:10.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:blC00doSx2frKGzFxNb9MkboOV0=', 'Content-Length': '0'} 2023-10-11T15:08:10.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c4a8452af7515e-006526ba5a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.895 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.896 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.896 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.896 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.897 DEBUG:boto:Method: GET 2023-10-11T15:08:10.897 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:08:10.898 DEBUG:boto:Data: 2023-10-11T15:08:10.898 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.899 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.899 DEBUG:boto:Params: {} 2023-10-11T15:08:10.900 DEBUG:boto:Token: None 2023-10-11T15:08:10.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/ 2023-10-11T15:08:10.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8kkNMav0uGBTHddqcUSXTRj4zKE= 2023-10-11T15:08:10.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8kkNMav0uGBTHddqcUSXTRj4zKE=', 'Content-Length': '0'} 2023-10-11T15:08:10.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003260c0b39294455b-006526ba5a-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.904 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.905 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.906 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.906 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.907 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.907 DEBUG:boto:Method: GET 2023-10-11T15:08:10.908 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:08:10.908 DEBUG:boto:Data: 2023-10-11T15:08:10.909 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.910 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.910 DEBUG:boto:Params: {} 2023-10-11T15:08:10.911 DEBUG:boto:Token: None 2023-10-11T15:08:10.911 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/ 2023-10-11T15:08:10.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8kkNMav0uGBTHddqcUSXTRj4zKE= 2023-10-11T15:08:10.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8kkNMav0uGBTHddqcUSXTRj4zKE=', 'Content-Length': '0'} 2023-10-11T15:08:10.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b37461556d75a87a-006526ba5a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.915 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:10.916 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.917 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.917 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.918 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:08:10.918 DEBUG:boto:Data: 2023-10-11T15:08:10.919 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.919 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.920 DEBUG:boto:Params: {} 2023-10-11T15:08:10.920 DEBUG:boto:Token: None 2023-10-11T15:08:10.921 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a 2023-10-11T15:08:10.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxKe4PhYXYr9c6bTndPFNjJwtHM= 2023-10-11T15:08:10.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxKe4PhYXYr9c6bTndPFNjJwtHM=', 'Content-Length': '0'} 2023-10-11T15:08:10.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78b63c06259209a-006526ba5a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.925 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.925 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.925 DEBUG:boto:Method: HEAD 2023-10-11T15:08:10.926 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:08:10.926 DEBUG:boto:Data: 2023-10-11T15:08:10.927 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.928 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.929 DEBUG:boto:Params: {} 2023-10-11T15:08:10.929 DEBUG:boto:Token: None 2023-10-11T15:08:10.930 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a 2023-10-11T15:08:10.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxKe4PhYXYr9c6bTndPFNjJwtHM= 2023-10-11T15:08:10.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxKe4PhYXYr9c6bTndPFNjJwtHM=', 'Content-Length': '0'} 2023-10-11T15:08:10.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bcfc9c65e51450e-006526ba5a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:08:10.934 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.934 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.935 DEBUG:boto:Method: GET 2023-10-11T15:08:10.935 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:08:10.936 DEBUG:boto:Data: 2023-10-11T15:08:10.936 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.937 DEBUG:boto:Port: 8001 2023-10-11T15:08:10.938 DEBUG:boto:Params: {} 2023-10-11T15:08:10.938 DEBUG:boto:Token: None 2023-10-11T15:08:10.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a 2023-10-11T15:08:10.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hbDcV3ojNrorbUBygfe0CTpvizo= 2023-10-11T15:08:10.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hbDcV3ojNrorbUBygfe0CTpvizo=', 'Content-Length': '0'} 2023-10-11T15:08:10.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101a49419b160399-006526ba5a-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.944 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:08:10.944 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:08:10.945 DEBUG:boto:Method: GET 2023-10-11T15:08:10.945 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:08:10.946 DEBUG:boto:Data: 2023-10-11T15:08:10.946 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:10.947 DEBUG:boto:Port: 8000 2023-10-11T15:08:10.947 DEBUG:boto:Params: {} 2023-10-11T15:08:10.948 DEBUG:boto:Token: None 2023-10-11T15:08:10.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:10 GMT /dghqvd-56/a 2023-10-11T15:08:10.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hbDcV3ojNrorbUBygfe0CTpvizo= 2023-10-11T15:08:10.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hbDcV3ojNrorbUBygfe0CTpvizo=', 'Content-Length': '0'} 2023-10-11T15:08:10.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb7d2176ba4f657f-006526ba5a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:10.994 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:10.995 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:10.996 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.996 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:08:10.996 DEBUG:boto:Method: GET 2023-10-11T15:08:10.997 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:08:10.998 DEBUG:boto:Data: 2023-10-11T15:08:10.998 DEBUG:boto:Headers: {} 2023-10-11T15:08:10.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:10.999 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.000 DEBUG:boto:Params: {} 2023-10-11T15:08:11.000 DEBUG:boto:Token: None 2023-10-11T15:08:11.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007dc23e7ec0262c6-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.005 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.007 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.008 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.008 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:08:11.009 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:08:11.010 DEBUG:boto:Method: GET 2023-10-11T15:08:11.011 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:08:11.011 DEBUG:boto:Data: 2023-10-11T15:08:11.012 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.013 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.014 DEBUG:boto:Params: {} 2023-10-11T15:08:11.015 DEBUG:boto:Token: None 2023-10-11T15:08:11.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da89f79b4599961a-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.019 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:08:11.020 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.021 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.021 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.022 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.022 DEBUG:boto:Method: GET 2023-10-11T15:08:11.023 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.023 DEBUG:boto:Data: 2023-10-11T15:08:11.023 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.024 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.025 DEBUG:boto:Params: {} 2023-10-11T15:08:11.025 DEBUG:boto:Token: None 2023-10-11T15:08:11.026 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8= 2023-10-11T15:08:11.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8=', 'Content-Length': '0'} 2023-10-11T15:08:11.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e892b24e4bcc571-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.030 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.030 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.031 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.031 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.031 DEBUG:boto:Method: GET 2023-10-11T15:08:11.032 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.032 DEBUG:boto:Data: 2023-10-11T15:08:11.033 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.033 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.034 DEBUG:boto:Params: {} 2023-10-11T15:08:11.034 DEBUG:boto:Token: None 2023-10-11T15:08:11.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8= 2023-10-11T15:08:11.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8=', 'Content-Length': '0'} 2023-10-11T15:08:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c5f401a318021c-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.039 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.039 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.040 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.040 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.040 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.041 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.041 DEBUG:boto:Data: 2023-10-11T15:08:11.042 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.043 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.043 DEBUG:boto:Params: {} 2023-10-11T15:08:11.044 DEBUG:boto:Token: None 2023-10-11T15:08:11.045 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3a6DyO6xE4nr4PtKHE4MSIo4E5g= 2023-10-11T15:08:11.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3a6DyO6xE4nr4PtKHE4MSIo4E5g=', 'Content-Length': '0'} 2023-10-11T15:08:11.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005628e84d56b42de0-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.050 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.051 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.052 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.052 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.053 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.053 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.053 DEBUG:boto:Data: 2023-10-11T15:08:11.054 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.055 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.055 DEBUG:boto:Params: {} 2023-10-11T15:08:11.055 DEBUG:boto:Token: None 2023-10-11T15:08:11.057 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3a6DyO6xE4nr4PtKHE4MSIo4E5g= 2023-10-11T15:08:11.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3a6DyO6xE4nr4PtKHE4MSIo4E5g=', 'Content-Length': '0'} 2023-10-11T15:08:11.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768815c6667ca0b4-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:08:11.061 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.061 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.062 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.062 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.063 DEBUG:boto:Method: GET 2023-10-11T15:08:11.063 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.063 DEBUG:boto:Data: 2023-10-11T15:08:11.064 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.065 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.065 DEBUG:boto:Params: {} 2023-10-11T15:08:11.066 DEBUG:boto:Token: None 2023-10-11T15:08:11.066 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8= 2023-10-11T15:08:11.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8=', 'Content-Length': '0'} 2023-10-11T15:08:11.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c0ab582b633dda-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.070 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.070 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.071 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.071 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:08:11.072 DEBUG:boto:Method: GET 2023-10-11T15:08:11.072 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.073 DEBUG:boto:Data: 2023-10-11T15:08:11.073 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.074 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.074 DEBUG:boto:Params: {} 2023-10-11T15:08:11.075 DEBUG:boto:Token: None 2023-10-11T15:08:11.076 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:08:11.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8= 2023-10-11T15:08:11.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGPYNlc5svGl46cHn7Ljh6d9Pc8=', 'Content-Length': '0'} 2023-10-11T15:08:11.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114c6a3b9a737b17-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.126 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.127 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.128 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.128 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.129 DEBUG:boto:Method: GET 2023-10-11T15:08:11.129 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:08:11.129 DEBUG:boto:Data: 2023-10-11T15:08:11.130 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.131 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.131 DEBUG:boto:Params: {} 2023-10-11T15:08:11.132 DEBUG:boto:Token: None 2023-10-11T15:08:11.132 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b7a23ddb1aabf12-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.137 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.138 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.138 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.139 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.139 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.140 DEBUG:boto:Method: GET 2023-10-11T15:08:11.140 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:08:11.141 DEBUG:boto:Data: 2023-10-11T15:08:11.141 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.142 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.142 DEBUG:boto:Params: {} 2023-10-11T15:08:11.143 DEBUG:boto:Token: None 2023-10-11T15:08:11.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4fc1493efa50b0-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.149 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.150 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.151 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.151 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.152 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:08:11.152 DEBUG:boto:Data: 2023-10-11T15:08:11.152 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.153 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.153 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.154 DEBUG:boto:Params: {} 2023-10-11T15:08:11.154 DEBUG:boto:Token: None 2023-10-11T15:08:11.155 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b 2023-10-11T15:08:11.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrVMK59Y541kL1Rjy4Ai9SXJVzY= 2023-10-11T15:08:11.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrVMK59Y541kL1Rjy4Ai9SXJVzY=', 'Content-Length': '0'} 2023-10-11T15:08:11.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8656e4d14f1dd0-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.158 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.159 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.159 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.160 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:08:11.160 DEBUG:boto:Data: 2023-10-11T15:08:11.160 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.161 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.161 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.162 DEBUG:boto:Params: {} 2023-10-11T15:08:11.162 DEBUG:boto:Token: None 2023-10-11T15:08:11.162 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b 2023-10-11T15:08:11.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrVMK59Y541kL1Rjy4Ai9SXJVzY= 2023-10-11T15:08:11.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrVMK59Y541kL1Rjy4Ai9SXJVzY=', 'Content-Length': '0'} 2023-10-11T15:08:11.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3daddcb9cdbe18-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:08:11.167 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.167 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.168 DEBUG:boto:Method: GET 2023-10-11T15:08:11.168 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:08:11.168 DEBUG:boto:Data: 2023-10-11T15:08:11.169 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.170 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.170 DEBUG:boto:Params: {} 2023-10-11T15:08:11.171 DEBUG:boto:Token: None 2023-10-11T15:08:11.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b 2023-10-11T15:08:11.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I2+D9UFZuarTxUJSyzFpUIzfEe4= 2023-10-11T15:08:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I2+D9UFZuarTxUJSyzFpUIzfEe4=', 'Content-Length': '0'} 2023-10-11T15:08:11.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d7d82ae8028dae7-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.175 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:08:11.175 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:08:11.176 DEBUG:boto:Method: GET 2023-10-11T15:08:11.176 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:08:11.177 DEBUG:boto:Data: 2023-10-11T15:08:11.177 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.178 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.178 DEBUG:boto:Params: {} 2023-10-11T15:08:11.179 DEBUG:boto:Token: None 2023-10-11T15:08:11.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/b 2023-10-11T15:08:11.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I2+D9UFZuarTxUJSyzFpUIzfEe4= 2023-10-11T15:08:11.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I2+D9UFZuarTxUJSyzFpUIzfEe4=', 'Content-Length': '0'} 2023-10-11T15:08:11.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665fe6232a7e205e-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.183 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.183 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.184 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.184 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.185 DEBUG:boto:Method: GET 2023-10-11T15:08:11.185 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:08:11.186 DEBUG:boto:Data: 2023-10-11T15:08:11.186 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.187 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.187 DEBUG:boto:Params: {} 2023-10-11T15:08:11.188 DEBUG:boto:Token: None 2023-10-11T15:08:11.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed23fd92772d093a-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.192 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.193 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.194 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.194 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.195 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:08:11.195 DEBUG:boto:Method: GET 2023-10-11T15:08:11.195 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:08:11.196 DEBUG:boto:Data: 2023-10-11T15:08:11.196 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.197 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.197 DEBUG:boto:Params: {} 2023-10-11T15:08:11.198 DEBUG:boto:Token: None 2023-10-11T15:08:11.198 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004231fe8937b4f918-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.246 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:08:11.248 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.248 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.248 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.249 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.249 DEBUG:boto:Method: GET 2023-10-11T15:08:11.250 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.250 DEBUG:boto:Data: 2023-10-11T15:08:11.251 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.251 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.252 DEBUG:boto:Params: {} 2023-10-11T15:08:11.252 DEBUG:boto:Token: None 2023-10-11T15:08:11.253 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg= 2023-10-11T15:08:11.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg=', 'Content-Length': '0'} 2023-10-11T15:08:11.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4beede8f2f1c6f0-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.257 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.257 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.258 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.258 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.258 DEBUG:boto:Method: GET 2023-10-11T15:08:11.259 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.261 DEBUG:boto:Data: 2023-10-11T15:08:11.262 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.263 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.263 DEBUG:boto:Params: {} 2023-10-11T15:08:11.264 DEBUG:boto:Token: None 2023-10-11T15:08:11.264 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg= 2023-10-11T15:08:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg=', 'Content-Length': '0'} 2023-10-11T15:08:11.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000133c0af5c8f7627d-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.268 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.268 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.269 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.269 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.270 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.270 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.270 DEBUG:boto:Data: 2023-10-11T15:08:11.271 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.272 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.272 DEBUG:boto:Params: {} 2023-10-11T15:08:11.272 DEBUG:boto:Token: None 2023-10-11T15:08:11.273 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.273 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MZ7Eh+E/Ezi17qFOm3dS9eANfY0= 2023-10-11T15:08:11.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MZ7Eh+E/Ezi17qFOm3dS9eANfY0=', 'Content-Length': '0'} 2023-10-11T15:08:11.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e67bae69ea0ff15-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.276 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.277 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.277 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.278 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.278 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.279 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.279 DEBUG:boto:Data: 2023-10-11T15:08:11.280 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.280 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.281 DEBUG:boto:Params: {} 2023-10-11T15:08:11.281 DEBUG:boto:Token: None 2023-10-11T15:08:11.282 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MZ7Eh+E/Ezi17qFOm3dS9eANfY0= 2023-10-11T15:08:11.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MZ7Eh+E/Ezi17qFOm3dS9eANfY0=', 'Content-Length': '0'} 2023-10-11T15:08:11.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5bb2f69518edc5-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:08:11.285 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.286 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.286 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.287 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.287 DEBUG:boto:Method: GET 2023-10-11T15:08:11.287 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.288 DEBUG:boto:Data: 2023-10-11T15:08:11.288 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.289 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.289 DEBUG:boto:Params: {} 2023-10-11T15:08:11.290 DEBUG:boto:Token: None 2023-10-11T15:08:11.290 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg= 2023-10-11T15:08:11.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg=', 'Content-Length': '0'} 2023-10-11T15:08:11.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3dfa7875163374d-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.294 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.294 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.295 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.295 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:08:11.295 DEBUG:boto:Method: GET 2023-10-11T15:08:11.296 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.296 DEBUG:boto:Data: 2023-10-11T15:08:11.297 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.297 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.298 DEBUG:boto:Params: {} 2023-10-11T15:08:11.298 DEBUG:boto:Token: None 2023-10-11T15:08:11.299 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:08:11.299 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg= 2023-10-11T15:08:11.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:miWtru02NQ368BS1SEc+NgNgYwg=', 'Content-Length': '0'} 2023-10-11T15:08:11.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe5b4fe8405938d-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.303 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.303 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.304 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.304 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.305 DEBUG:boto:Method: GET 2023-10-11T15:08:11.305 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:08:11.305 DEBUG:boto:Data: 2023-10-11T15:08:11.306 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.307 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.307 DEBUG:boto:Params: {} 2023-10-11T15:08:11.308 DEBUG:boto:Token: None 2023-10-11T15:08:11.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e613335654f8b8d-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.312 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.312 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.313 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.313 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.314 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.314 DEBUG:boto:Method: GET 2023-10-11T15:08:11.315 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:08:11.315 DEBUG:boto:Data: 2023-10-11T15:08:11.315 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.316 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.317 DEBUG:boto:Params: {} 2023-10-11T15:08:11.317 DEBUG:boto:Token: None 2023-10-11T15:08:11.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000682d16d12fb73e83-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.322 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.323 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.323 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.323 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.324 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:08:11.324 DEBUG:boto:Data: 2023-10-11T15:08:11.325 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.325 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.326 DEBUG:boto:Params: {} 2023-10-11T15:08:11.326 DEBUG:boto:Token: None 2023-10-11T15:08:11.327 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c 2023-10-11T15:08:11.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5tba8q9GAn7QrZkLaNMa1tyx66w= 2023-10-11T15:08:11.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5tba8q9GAn7QrZkLaNMa1tyx66w=', 'Content-Length': '0'} 2023-10-11T15:08:11.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000154de78305fd4c5e-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.330 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.331 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.331 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.331 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:08:11.332 DEBUG:boto:Data: 2023-10-11T15:08:11.332 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.333 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.334 DEBUG:boto:Params: {} 2023-10-11T15:08:11.334 DEBUG:boto:Token: None 2023-10-11T15:08:11.335 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c 2023-10-11T15:08:11.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5tba8q9GAn7QrZkLaNMa1tyx66w= 2023-10-11T15:08:11.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5tba8q9GAn7QrZkLaNMa1tyx66w=', 'Content-Length': '0'} 2023-10-11T15:08:11.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899b10eba07dbf43-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:08:11.338 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.339 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.339 DEBUG:boto:Method: GET 2023-10-11T15:08:11.340 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:08:11.340 DEBUG:boto:Data: 2023-10-11T15:08:11.340 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.341 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.342 DEBUG:boto:Params: {} 2023-10-11T15:08:11.342 DEBUG:boto:Token: None 2023-10-11T15:08:11.343 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c 2023-10-11T15:08:11.343 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OeDBIw4WQd+YxHItWCfiNb8+o9I= 2023-10-11T15:08:11.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OeDBIw4WQd+YxHItWCfiNb8+o9I=', 'Content-Length': '0'} 2023-10-11T15:08:11.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003745f66d47540025-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.390 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:08:11.391 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:08:11.391 DEBUG:boto:Method: GET 2023-10-11T15:08:11.392 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:08:11.392 DEBUG:boto:Data: 2023-10-11T15:08:11.393 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.394 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.394 DEBUG:boto:Params: {} 2023-10-11T15:08:11.395 DEBUG:boto:Token: None 2023-10-11T15:08:11.395 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/c 2023-10-11T15:08:11.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OeDBIw4WQd+YxHItWCfiNb8+o9I= 2023-10-11T15:08:11.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OeDBIw4WQd+YxHItWCfiNb8+o9I=', 'Content-Length': '0'} 2023-10-11T15:08:11.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1f961a6af4b77f-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.399 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.400 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.400 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.400 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.401 DEBUG:boto:Method: GET 2023-10-11T15:08:11.401 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:08:11.402 DEBUG:boto:Data: 2023-10-11T15:08:11.402 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.403 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.404 DEBUG:boto:Params: {} 2023-10-11T15:08:11.404 DEBUG:boto:Token: None 2023-10-11T15:08:11.405 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f511813ecc59054-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.411 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.412 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.412 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.412 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.413 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:08:11.413 DEBUG:boto:Method: GET 2023-10-11T15:08:11.414 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:08:11.414 DEBUG:boto:Data: 2023-10-11T15:08:11.415 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.416 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.417 DEBUG:boto:Params: {} 2023-10-11T15:08:11.417 DEBUG:boto:Token: None 2023-10-11T15:08:11.418 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019301cbf669ea957-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.466 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:08:11.468 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.468 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.469 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.469 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.470 DEBUG:boto:Method: GET 2023-10-11T15:08:11.470 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.471 DEBUG:boto:Data: 2023-10-11T15:08:11.471 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.472 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.473 DEBUG:boto:Params: {} 2023-10-11T15:08:11.473 DEBUG:boto:Token: None 2023-10-11T15:08:11.474 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.475 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw= 2023-10-11T15:08:11.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw=', 'Content-Length': '0'} 2023-10-11T15:08:11.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637a89a4f5dce6fa-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.479 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.479 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.480 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.480 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.481 DEBUG:boto:Method: GET 2023-10-11T15:08:11.481 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.482 DEBUG:boto:Data: 2023-10-11T15:08:11.482 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.483 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.484 DEBUG:boto:Params: {} 2023-10-11T15:08:11.484 DEBUG:boto:Token: None 2023-10-11T15:08:11.485 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw= 2023-10-11T15:08:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw=', 'Content-Length': '0'} 2023-10-11T15:08:11.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d4f8cc02e19917-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.488 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.489 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.489 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.490 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.490 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.491 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.491 DEBUG:boto:Data: 2023-10-11T15:08:11.492 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.493 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.493 DEBUG:boto:Params: {} 2023-10-11T15:08:11.494 DEBUG:boto:Token: None 2023-10-11T15:08:11.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0FJjVWhXYBFLwbZaStOtsi8DJU= 2023-10-11T15:08:11.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0FJjVWhXYBFLwbZaStOtsi8DJU=', 'Content-Length': '0'} 2023-10-11T15:08:11.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5f8e7f198de1e4-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.498 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.498 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.499 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.499 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.500 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.500 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.500 DEBUG:boto:Data: 2023-10-11T15:08:11.501 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.502 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.502 DEBUG:boto:Params: {} 2023-10-11T15:08:11.503 DEBUG:boto:Token: None 2023-10-11T15:08:11.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F0FJjVWhXYBFLwbZaStOtsi8DJU= 2023-10-11T15:08:11.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F0FJjVWhXYBFLwbZaStOtsi8DJU=', 'Content-Length': '0'} 2023-10-11T15:08:11.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a3a50102bd55be3-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:08:11.507 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.508 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.508 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.508 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.509 DEBUG:boto:Method: GET 2023-10-11T15:08:11.509 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.510 DEBUG:boto:Data: 2023-10-11T15:08:11.510 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.511 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.511 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.511 DEBUG:boto:Params: {} 2023-10-11T15:08:11.512 DEBUG:boto:Token: None 2023-10-11T15:08:11.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw= 2023-10-11T15:08:11.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw=', 'Content-Length': '0'} 2023-10-11T15:08:11.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000839fdee8db4bc87-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.516 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.517 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.517 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.517 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:08:11.518 DEBUG:boto:Method: GET 2023-10-11T15:08:11.518 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.519 DEBUG:boto:Data: 2023-10-11T15:08:11.519 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.520 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.521 DEBUG:boto:Params: {} 2023-10-11T15:08:11.521 DEBUG:boto:Token: None 2023-10-11T15:08:11.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:08:11.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw= 2023-10-11T15:08:11.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jsLUEs9RBz6zrZO5RUljdPQpBNw=', 'Content-Length': '0'} 2023-10-11T15:08:11.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed39ef89b8565fb6-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.525 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.526 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.526 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.526 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.527 DEBUG:boto:Method: GET 2023-10-11T15:08:11.527 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:08:11.528 DEBUG:boto:Data: 2023-10-11T15:08:11.528 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.529 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.529 DEBUG:boto:Params: {} 2023-10-11T15:08:11.530 DEBUG:boto:Token: None 2023-10-11T15:08:11.530 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b577a60cae97125-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.535 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.536 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.536 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.537 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.537 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.537 DEBUG:boto:Method: GET 2023-10-11T15:08:11.538 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:08:11.538 DEBUG:boto:Data: 2023-10-11T15:08:11.539 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.540 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.540 DEBUG:boto:Params: {} 2023-10-11T15:08:11.541 DEBUG:boto:Token: None 2023-10-11T15:08:11.541 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002539cbe5d60c0a51-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.546 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.546 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.547 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.547 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.547 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:08:11.548 DEBUG:boto:Data: 2023-10-11T15:08:11.548 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.549 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.549 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.549 DEBUG:boto:Params: {} 2023-10-11T15:08:11.550 DEBUG:boto:Token: None 2023-10-11T15:08:11.550 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d 2023-10-11T15:08:11.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tImEHsFKHuy9AzFzdV0VmJ7CaRI= 2023-10-11T15:08:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tImEHsFKHuy9AzFzdV0VmJ7CaRI=', 'Content-Length': '0'} 2023-10-11T15:08:11.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013389ad1059608c7-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.554 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.555 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.555 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.555 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:08:11.556 DEBUG:boto:Data: 2023-10-11T15:08:11.556 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.557 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.557 DEBUG:boto:Params: {} 2023-10-11T15:08:11.558 DEBUG:boto:Token: None 2023-10-11T15:08:11.558 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d 2023-10-11T15:08:11.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tImEHsFKHuy9AzFzdV0VmJ7CaRI= 2023-10-11T15:08:11.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tImEHsFKHuy9AzFzdV0VmJ7CaRI=', 'Content-Length': '0'} 2023-10-11T15:08:11.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a29af2224c5ece-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:08:11.566 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.567 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.567 DEBUG:boto:Method: GET 2023-10-11T15:08:11.568 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:08:11.568 DEBUG:boto:Data: 2023-10-11T15:08:11.568 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.569 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.570 DEBUG:boto:Params: {} 2023-10-11T15:08:11.570 DEBUG:boto:Token: None 2023-10-11T15:08:11.571 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d 2023-10-11T15:08:11.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vu7/XuBtpEwUZDly7wYNkPw8YjU= 2023-10-11T15:08:11.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vu7/XuBtpEwUZDly7wYNkPw8YjU=', 'Content-Length': '0'} 2023-10-11T15:08:11.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae2a7b1e1585285-006526ba5b-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.614 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:08:11.615 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:08:11.616 DEBUG:boto:Method: GET 2023-10-11T15:08:11.616 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:08:11.617 DEBUG:boto:Data: 2023-10-11T15:08:11.617 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.618 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.618 DEBUG:boto:Params: {} 2023-10-11T15:08:11.619 DEBUG:boto:Token: None 2023-10-11T15:08:11.619 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/d 2023-10-11T15:08:11.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vu7/XuBtpEwUZDly7wYNkPw8YjU= 2023-10-11T15:08:11.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vu7/XuBtpEwUZDly7wYNkPw8YjU=', 'Content-Length': '0'} 2023-10-11T15:08:11.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f311e6196bef5b55-006526ba5b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.623 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.624 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.624 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.625 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.625 DEBUG:boto:Method: GET 2023-10-11T15:08:11.625 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:08:11.626 DEBUG:boto:Data: 2023-10-11T15:08:11.626 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:08:11.627 DEBUG:boto:Port: 8001 2023-10-11T15:08:11.628 DEBUG:boto:Params: {} 2023-10-11T15:08:11.628 DEBUG:boto:Token: None 2023-10-11T15:08:11.629 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.629 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000645126e7865f648f-006526ba5b-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.633 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.633 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.634 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.634 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.635 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:08:11.635 DEBUG:boto:Method: GET 2023-10-11T15:08:11.635 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:08:11.636 DEBUG:boto:Data: 2023-10-11T15:08:11.636 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.636 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.637 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.637 DEBUG:boto:Params: {} 2023-10-11T15:08:11.638 DEBUG:boto:Token: None 2023-10-11T15:08:11.638 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4= 2023-10-11T15:08:11.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AmSPD85SAk92jKxC5qSx0UkJwP4=', 'Content-Length': '0'} 2023-10-11T15:08:11.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba4e47b1e970545-006526ba5b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.642 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:08:11.643 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-56 zones={a2, a1} 2023-10-11T15:08:11.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:08:11.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:08:11.645 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:08:11.700 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:08:11.701 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:08:11.744 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:08:11.744 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-56 2023-10-11T15:08:11.747 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 4 2023-10-11T15:08:11.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:08:11.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:08:11.785 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:08:11.835 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:08:11.835 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:08:11.872 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:08:11.872 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-56 2023-10-11T15:08:11.873 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 4 2023-10-11T15:08:11.898 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.899 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.899 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.900 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:11.900 DEBUG:boto:Data: 2023-10-11T15:08:11.900 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.901 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.901 DEBUG:boto:Params: {} 2023-10-11T15:08:11.902 DEBUG:boto:Token: None 2023-10-11T15:08:11.902 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U= 2023-10-11T15:08:11.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U=', 'Content-Length': '0'} 2023-10-11T15:08:11.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7006c069d6a5cdf-006526ba5b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.906 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:08:11.907 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:08:11.907 DEBUG:boto:Method: PUT 2023-10-11T15:08:11.907 DEBUG:boto:Path: /dghqvd-56/e 2023-10-11T15:08:11.908 DEBUG:boto:Data: 2023-10-11T15:08:11.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:08:11.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.909 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.909 DEBUG:boto:Params: {} 2023-10-11T15:08:11.910 DEBUG:boto:Token: None 2023-10-11T15:08:11.910 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/e 2023-10-11T15:08:11.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:09jYcGmJx9ROOeFNluG0qdmlQjU= 2023-10-11T15:08:11.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:09jYcGmJx9ROOeFNluG0qdmlQjU='} 2023-10-11T15:08:11.917 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000978f0448fe8aff1a-006526ba5b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.917 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.918 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.919 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.919 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:11.920 DEBUG:boto:Data: 2023-10-11T15:08:11.920 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.921 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.921 DEBUG:boto:Params: {} 2023-10-11T15:08:11.922 DEBUG:boto:Token: None 2023-10-11T15:08:11.922 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U= 2023-10-11T15:08:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U=', 'Content-Length': '0'} 2023-10-11T15:08:11.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e51227b481b9822-006526ba5b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.926 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:08:11.926 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:08:11.927 DEBUG:boto:Method: PUT 2023-10-11T15:08:11.927 DEBUG:boto:Path: /dghqvd-56/f 2023-10-11T15:08:11.927 DEBUG:boto:Data: 2023-10-11T15:08:11.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:08:11.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.929 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.929 DEBUG:boto:Params: {} 2023-10-11T15:08:11.929 DEBUG:boto:Token: None 2023-10-11T15:08:11.930 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/f 2023-10-11T15:08:11.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ohFrrjmUpXwPaBy32dQZvSOdXSA= 2023-10-11T15:08:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ohFrrjmUpXwPaBy32dQZvSOdXSA='} 2023-10-11T15:08:11.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd619915cf8fe584-006526ba5b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.975 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.976 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.976 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.976 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:11.977 DEBUG:boto:Data: 2023-10-11T15:08:11.977 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.978 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.978 DEBUG:boto:Params: {} 2023-10-11T15:08:11.979 DEBUG:boto:Token: None 2023-10-11T15:08:11.979 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U= 2023-10-11T15:08:11.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U=', 'Content-Length': '0'} 2023-10-11T15:08:11.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a6a262c34388d94-006526ba5b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.983 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:08:11.983 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:08:11.984 DEBUG:boto:Method: PUT 2023-10-11T15:08:11.984 DEBUG:boto:Path: /dghqvd-56/g 2023-10-11T15:08:11.984 DEBUG:boto:Data: 2023-10-11T15:08:11.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:08:11.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.986 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.986 DEBUG:boto:Params: {} 2023-10-11T15:08:11.986 DEBUG:boto:Token: None 2023-10-11T15:08:11.987 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/g 2023-10-11T15:08:11.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1YEUZvHXZg4YwYCbR8rAAZyw0nU= 2023-10-11T15:08:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1YEUZvHXZg4YwYCbR8rAAZyw0nU='} 2023-10-11T15:08:11.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce4e7aafcb7aeafb-006526ba5b-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:11.993 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:08:11.994 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:08:11.994 DEBUG:boto:Method: HEAD 2023-10-11T15:08:11.994 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:08:11.995 DEBUG:boto:Data: 2023-10-11T15:08:11.995 DEBUG:boto:Headers: {} 2023-10-11T15:08:11.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:11.996 DEBUG:boto:Port: 8000 2023-10-11T15:08:11.996 DEBUG:boto:Params: {} 2023-10-11T15:08:11.997 DEBUG:boto:Token: None 2023-10-11T15:08:11.997 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:08:11 GMT /dghqvd-56/ 2023-10-11T15:08:11.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U= 2023-10-11T15:08:11.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:08:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q6HDnDtaY6g7MMuQbjB7Whwv/+U=', 'Content-Length': '0'} 2023-10-11T15:08:12.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b03003ad7fc9f69-006526ba5b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:12.000 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:08:12.001 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:08:12.001 DEBUG:boto:Method: PUT 2023-10-11T15:08:12.002 DEBUG:boto:Path: /dghqvd-56/h 2023-10-11T15:08:12.002 DEBUG:boto:Data: 2023-10-11T15:08:12.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:08:12.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:08:12.003 DEBUG:boto:Port: 8000 2023-10-11T15:08:12.003 DEBUG:boto:Params: {} 2023-10-11T15:08:12.004 DEBUG:boto:Token: None 2023-10-11T15:08:12.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:08:12 GMT /dghqvd-56/h 2023-10-11T15:08:12.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JG7JRSkBeayAzEFDTbPcHLIZSTM= 2023-10-11T15:08:12.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:08:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JG7JRSkBeayAzEFDTbPcHLIZSTM='} 2023-10-11T15:08:12.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd48887654cee519-006526ba5c-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:08:12.051 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-56 2023-10-11T15:08:12.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:08:12.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:08:12.053 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:08:12.114 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:08:12.115 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:08:12.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:12.148+0000 7f45e630b780 1 waiting to reach latest gen 2, on 0.. 2023-10-11T15:08:17.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:17.149+0000 7f45e630b780 1 waiting to reach latest gen 2, on 0.. 2023-10-11T15:08:22.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:22.150+0000 7f45e630b780 1 waiting to reach latest gen 2, on 0.. 2023-10-11T15:08:27.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:27.150+0000 7f45e630b780 1 waiting to reach latest gen 2, on 0.. 2023-10-11T15:08:32.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:32.151+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:08:37.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:37.152+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:08:42.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:42.152+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:08:47.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:47.153+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:08:52.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:52.154+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:08:57.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:08:57.154+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:09:02.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:02.155+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:09:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:07.156+0000 7f45e630b780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:09:12.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:12.156+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:12.159 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:12.159 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:17.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:17.157+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:17.160 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:17.161 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:22.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:22.158+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:22.161 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:22.161 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:27.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:27.158+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:27.162 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:27.162 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:32.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:32.159+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:32.162 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:32.163 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:37.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:37.160+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:37.163 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:37.163 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:42.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:42.161+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:42.164 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:42.164 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:47.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:47.161+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:47.165 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , ] 2023-10-11T15:09:47.165 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:52.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:52.163+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:52.165 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.46.6, , 00000000002.71.6] 2023-10-11T15:09:52.166 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:09:57.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:09:57.164+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:09:57.166 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.46.6, , 00000000002.71.6] 2023-10-11T15:09:57.166 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:10:02.166 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:02.165+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:02.167 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.46.6, , 00000000002.71.6] 2023-10-11T15:10:02.167 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:10:07.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:07.166+0000 7f45e630b780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:07.168 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.46.6, , 00000000002.71.6] 2023-10-11T15:10:07.168 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:10:12.168 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:12.166+0000 7f45e630b780 1 bucket sync caught up with source: 2023-10-11T15:10:12.169 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:10:12.169 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.46.6, 00000000002.64.6, 00000000002.71.6] 2023-10-11T15:10:12.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:12.166+0000 7f45e630b780 0 bucket checkpoint complete 2023-10-11T15:10:12.186 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-56 2023-10-11T15:10:12.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:10:12.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:10:12.188 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:10:12.250 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:12.251 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:12.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:12.285+0000 7f65102e0780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:12.287 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:12.287 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:17.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:17.287+0000 7f65102e0780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:17.289 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:17.289 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:22.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:22.288+0000 7f65102e0780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:22.290 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:22.290 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:27.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:27.289+0000 7f65102e0780 1 waiting for incremental sync to catch up: 2023-10-11T15:10:27.291 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:27.291 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:32.292 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:32.291+0000 7f65102e0780 1 bucket sync caught up with source: 2023-10-11T15:10:32.293 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:32.293 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.233.6, , , 00000000001.211.6, 00000000001.292.6, , , 00000000002.139.6, , 00000000002.121.6] 2023-10-11T15:10:32.293 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:32.291+0000 7f65102e0780 0 bucket checkpoint complete 2023-10-11T15:10:32.310 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-56 zones={a2, a1} 2023-10-11T15:10:32.311 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.311 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.312 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.312 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:10:32.312 DEBUG:boto:Data: 2023-10-11T15:10:32.313 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.314 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.314 DEBUG:boto:Params: {} 2023-10-11T15:10:32.315 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:10:32.315 DEBUG:boto:Token: None 2023-10-11T15:10:32.316 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OQMTNRw+AkU1mNu6YFJfBt+CHPY= 2023-10-11T15:10:32.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OQMTNRw+AkU1mNu6YFJfBt+CHPY=', 'Content-Length': '0'} 2023-10-11T15:10:32.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e119385a7924110a-006526bae8-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.322 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.322 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.323 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.323 DEBUG:boto:Path: /dghqvd-56/ 2023-10-11T15:10:32.324 DEBUG:boto:Data: 2023-10-11T15:10:32.324 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.325 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.325 DEBUG:boto:Params: {} 2023-10-11T15:10:32.326 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:10:32.326 DEBUG:boto:Token: None 2023-10-11T15:10:32.327 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OQMTNRw+AkU1mNu6YFJfBt+CHPY= 2023-10-11T15:10:32.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OQMTNRw+AkU1mNu6YFJfBt+CHPY=', 'Content-Length': '0'} 2023-10-11T15:10:32.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e25bfff282ab5ef-006526bae8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:10:32.332 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.333 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.333 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.333 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.334 DEBUG:boto:Method: GET 2023-10-11T15:10:32.334 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.335 DEBUG:boto:Data: 2023-10-11T15:10:32.335 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.336 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.336 DEBUG:boto:Params: {} 2023-10-11T15:10:32.337 DEBUG:boto:Token: None 2023-10-11T15:10:32.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/?versions 2023-10-11T15:10:32.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8= 2023-10-11T15:10:32.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8=', 'Content-Length': '0'} 2023-10-11T15:10:32.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf8ce2aaf6b9d819-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.342 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.343 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:10:32.343 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:10:32.344 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:10:32.344 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:10:32.344 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:10:32.345 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:10:32.345 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:10:32.346 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:10:32.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:10:32.346 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.347 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.347 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.348 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.348 DEBUG:boto:Method: GET 2023-10-11T15:10:32.348 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.349 DEBUG:boto:Data: 2023-10-11T15:10:32.349 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.350 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.351 DEBUG:boto:Params: {} 2023-10-11T15:10:32.351 DEBUG:boto:Token: None 2023-10-11T15:10:32.352 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/?versions 2023-10-11T15:10:32.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8= 2023-10-11T15:10:32.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8=', 'Content-Length': '0'} 2023-10-11T15:10:32.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219ce1dc7eedc503-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.355 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.356 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:10:32.356 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:10:32.357 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:10:32.357 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:10:32.357 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:10:32.358 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:10:32.358 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:10:32.358 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:10:32.359 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.359 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.360 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.360 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.361 DEBUG:boto:Method: GET 2023-10-11T15:10:32.361 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.361 DEBUG:boto:Data: 2023-10-11T15:10:32.362 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.363 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.363 DEBUG:boto:Params: {} 2023-10-11T15:10:32.363 DEBUG:boto:Token: None 2023-10-11T15:10:32.364 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/?versions 2023-10-11T15:10:32.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8= 2023-10-11T15:10:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8=', 'Content-Length': '0'} 2023-10-11T15:10:32.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ccc669febc6469-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.368 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.369 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.369 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.370 DEBUG:boto:path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.370 DEBUG:boto:auth_path=/dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.371 DEBUG:boto:Method: GET 2023-10-11T15:10:32.371 DEBUG:boto:Path: /dghqvd-56/?versions&max-keys=999 2023-10-11T15:10:32.371 DEBUG:boto:Data: 2023-10-11T15:10:32.372 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.373 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.373 DEBUG:boto:Params: {} 2023-10-11T15:10:32.374 DEBUG:boto:Token: None 2023-10-11T15:10:32.374 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/?versions 2023-10-11T15:10:32.375 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8= 2023-10-11T15:10:32.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EGLtT5ueIPq3B1nR/58RomHzWF8=', 'Content-Length': '0'} 2023-10-11T15:10:32.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b83ab4394e1b36-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.378 DEBUG:boto:b'dghqvd-56999falseanulltrue2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:10:32.380 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.380 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.381 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.381 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.381 DEBUG:boto:Method: GET 2023-10-11T15:10:32.382 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.382 DEBUG:boto:Data: 2023-10-11T15:10:32.383 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.383 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.384 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.384 DEBUG:boto:Params: {} 2023-10-11T15:10:32.385 DEBUG:boto:Token: None 2023-10-11T15:10:32.385 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE= 2023-10-11T15:10:32.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE=', 'Content-Length': '0'} 2023-10-11T15:10:32.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b4876c657cfc09-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.389 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.389 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.390 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.390 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.390 DEBUG:boto:Method: GET 2023-10-11T15:10:32.391 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.391 DEBUG:boto:Data: 2023-10-11T15:10:32.392 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.393 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.393 DEBUG:boto:Params: {} 2023-10-11T15:10:32.393 DEBUG:boto:Token: None 2023-10-11T15:10:32.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE= 2023-10-11T15:10:32.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE=', 'Content-Length': '0'} 2023-10-11T15:10:32.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464ddaa3a9642c78-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.398 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.398 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.398 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.399 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.399 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.400 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.400 DEBUG:boto:Data: 2023-10-11T15:10:32.400 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.401 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.402 DEBUG:boto:Params: {} 2023-10-11T15:10:32.402 DEBUG:boto:Token: None 2023-10-11T15:10:32.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XYOZi5oLnM7YYCbTDkaps3rnzo0= 2023-10-11T15:10:32.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XYOZi5oLnM7YYCbTDkaps3rnzo0=', 'Content-Length': '0'} 2023-10-11T15:10:32.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fc396340494b2c-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.406 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.406 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.407 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.407 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.408 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.408 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.408 DEBUG:boto:Data: 2023-10-11T15:10:32.409 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.410 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.410 DEBUG:boto:Params: {} 2023-10-11T15:10:32.410 DEBUG:boto:Token: None 2023-10-11T15:10:32.411 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XYOZi5oLnM7YYCbTDkaps3rnzo0= 2023-10-11T15:10:32.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XYOZi5oLnM7YYCbTDkaps3rnzo0=', 'Content-Length': '0'} 2023-10-11T15:10:32.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0394bb3224f2c47-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:10:32.415 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.415 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.416 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.416 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.416 DEBUG:boto:Method: GET 2023-10-11T15:10:32.417 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.417 DEBUG:boto:Data: 2023-10-11T15:10:32.418 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.419 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.419 DEBUG:boto:Params: {} 2023-10-11T15:10:32.420 DEBUG:boto:Token: None 2023-10-11T15:10:32.420 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE= 2023-10-11T15:10:32.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE=', 'Content-Length': '0'} 2023-10-11T15:10:32.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd5200c1b030de8-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.424 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.424 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.424 DEBUG:boto:path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.425 DEBUG:boto:auth_path=/dghqvd-56/a?versionId=null 2023-10-11T15:10:32.425 DEBUG:boto:Method: GET 2023-10-11T15:10:32.426 DEBUG:boto:Path: /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.426 DEBUG:boto:Data: 2023-10-11T15:10:32.427 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.427 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.428 DEBUG:boto:Params: {} 2023-10-11T15:10:32.428 DEBUG:boto:Token: None 2023-10-11T15:10:32.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a?versionId=null 2023-10-11T15:10:32.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE= 2023-10-11T15:10:32.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VwcNKN58wsL6BVNn97R2FzJ9OfE=', 'Content-Length': '0'} 2023-10-11T15:10:32.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b487c654a9bf584a-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.433 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.433 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.434 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.434 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.434 DEBUG:boto:Method: GET 2023-10-11T15:10:32.435 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:10:32.435 DEBUG:boto:Data: 2023-10-11T15:10:32.436 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.437 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.437 DEBUG:boto:Params: {} 2023-10-11T15:10:32.438 DEBUG:boto:Token: None 2023-10-11T15:10:32.438 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb172fd572554b6-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.442 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.443 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.443 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.444 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.444 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.445 DEBUG:boto:Method: GET 2023-10-11T15:10:32.445 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:10:32.446 DEBUG:boto:Data: 2023-10-11T15:10:32.446 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.447 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.448 DEBUG:boto:Params: {} 2023-10-11T15:10:32.449 DEBUG:boto:Token: None 2023-10-11T15:10:32.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd438426c2612dc-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.455 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.456 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.456 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.456 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.457 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:10:32.457 DEBUG:boto:Data: 2023-10-11T15:10:32.458 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.459 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.459 DEBUG:boto:Params: {} 2023-10-11T15:10:32.459 DEBUG:boto:Token: None 2023-10-11T15:10:32.460 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a 2023-10-11T15:10:32.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6o41nNX5doakD8UwaQfHlWcUxO0= 2023-10-11T15:10:32.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6o41nNX5doakD8UwaQfHlWcUxO0=', 'Content-Length': '0'} 2023-10-11T15:10:32.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8588856975e76c8-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.463 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.464 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.464 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.465 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:10:32.465 DEBUG:boto:Data: 2023-10-11T15:10:32.465 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.466 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.467 DEBUG:boto:Params: {} 2023-10-11T15:10:32.467 DEBUG:boto:Token: None 2023-10-11T15:10:32.468 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a 2023-10-11T15:10:32.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6o41nNX5doakD8UwaQfHlWcUxO0= 2023-10-11T15:10:32.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6o41nNX5doakD8UwaQfHlWcUxO0=', 'Content-Length': '0'} 2023-10-11T15:10:32.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dee8f9c1a72a708-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:10:32.471 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.472 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.472 DEBUG:boto:Method: GET 2023-10-11T15:10:32.473 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:10:32.473 DEBUG:boto:Data: 2023-10-11T15:10:32.474 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.474 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.475 DEBUG:boto:Params: {} 2023-10-11T15:10:32.475 DEBUG:boto:Token: None 2023-10-11T15:10:32.476 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a 2023-10-11T15:10:32.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Th1/U7MTSPthYoc6B2DIkhjFA8= 2023-10-11T15:10:32.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Th1/U7MTSPthYoc6B2DIkhjFA8=', 'Content-Length': '0'} 2023-10-11T15:10:32.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096dfd83c285ddd4a-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.526 DEBUG:boto:path=/dghqvd-56/a 2023-10-11T15:10:32.527 DEBUG:boto:auth_path=/dghqvd-56/a 2023-10-11T15:10:32.528 DEBUG:boto:Method: GET 2023-10-11T15:10:32.528 DEBUG:boto:Path: /dghqvd-56/a 2023-10-11T15:10:32.528 DEBUG:boto:Data: 2023-10-11T15:10:32.529 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.530 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.530 DEBUG:boto:Params: {} 2023-10-11T15:10:32.531 DEBUG:boto:Token: None 2023-10-11T15:10:32.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/a 2023-10-11T15:10:32.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Th1/U7MTSPthYoc6B2DIkhjFA8= 2023-10-11T15:10:32.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Th1/U7MTSPthYoc6B2DIkhjFA8=', 'Content-Length': '0'} 2023-10-11T15:10:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f63c6a10191262-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.537 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.538 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.539 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.540 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.540 DEBUG:boto:Method: GET 2023-10-11T15:10:32.541 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:10:32.542 DEBUG:boto:Data: 2023-10-11T15:10:32.543 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.544 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.545 DEBUG:boto:Params: {} 2023-10-11T15:10:32.546 DEBUG:boto:Token: None 2023-10-11T15:10:32.547 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269d08e204f0e23b-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.553 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.554 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.555 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.555 DEBUG:boto:path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.556 DEBUG:boto:auth_path=/dghqvd-56/?prefix=a 2023-10-11T15:10:32.557 DEBUG:boto:Method: GET 2023-10-11T15:10:32.558 DEBUG:boto:Path: /dghqvd-56/?prefix=a 2023-10-11T15:10:32.559 DEBUG:boto:Data: 2023-10-11T15:10:32.559 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.560 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.561 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.561 DEBUG:boto:Params: {} 2023-10-11T15:10:32.562 DEBUG:boto:Token: None 2023-10-11T15:10:32.563 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000325a15ec2742d129-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.567 DEBUG:boto:b'dghqvd-56a1000falsea2023-10-11T15:07:40.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:10:32.568 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.568 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.569 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.569 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.570 DEBUG:boto:Method: GET 2023-10-11T15:10:32.570 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.571 DEBUG:boto:Data: 2023-10-11T15:10:32.571 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.572 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.572 DEBUG:boto:Params: {} 2023-10-11T15:10:32.573 DEBUG:boto:Token: None 2023-10-11T15:10:32.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA= 2023-10-11T15:10:32.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA=', 'Content-Length': '0'} 2023-10-11T15:10:32.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106d9fce58772fb1-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.579 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.580 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.580 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.581 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.581 DEBUG:boto:Method: GET 2023-10-11T15:10:32.581 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.582 DEBUG:boto:Data: 2023-10-11T15:10:32.582 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.583 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.584 DEBUG:boto:Params: {} 2023-10-11T15:10:32.584 DEBUG:boto:Token: None 2023-10-11T15:10:32.585 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA= 2023-10-11T15:10:32.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA=', 'Content-Length': '0'} 2023-10-11T15:10:32.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000148129b985b052cd-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.588 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.589 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.589 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.590 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.590 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.590 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.591 DEBUG:boto:Data: 2023-10-11T15:10:32.591 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.592 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.592 DEBUG:boto:Params: {} 2023-10-11T15:10:32.593 DEBUG:boto:Token: None 2023-10-11T15:10:32.594 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kARvAPH0rti+OsLC6UlTRelgiUw= 2023-10-11T15:10:32.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kARvAPH0rti+OsLC6UlTRelgiUw=', 'Content-Length': '0'} 2023-10-11T15:10:32.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f5710705c99b3f-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.598 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.599 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.600 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.600 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.601 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.601 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.602 DEBUG:boto:Data: 2023-10-11T15:10:32.603 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.604 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.604 DEBUG:boto:Params: {} 2023-10-11T15:10:32.605 DEBUG:boto:Token: None 2023-10-11T15:10:32.605 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kARvAPH0rti+OsLC6UlTRelgiUw= 2023-10-11T15:10:32.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kARvAPH0rti+OsLC6UlTRelgiUw=', 'Content-Length': '0'} 2023-10-11T15:10:32.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b1613c780811b3-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:10:32.610 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.610 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.610 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.611 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.611 DEBUG:boto:Method: GET 2023-10-11T15:10:32.612 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.612 DEBUG:boto:Data: 2023-10-11T15:10:32.613 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.613 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.614 DEBUG:boto:Params: {} 2023-10-11T15:10:32.614 DEBUG:boto:Token: None 2023-10-11T15:10:32.615 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA= 2023-10-11T15:10:32.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA=', 'Content-Length': '0'} 2023-10-11T15:10:32.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8064be8eaacf64d-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.658 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.659 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.660 DEBUG:boto:path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.661 DEBUG:boto:auth_path=/dghqvd-56/b?versionId=null 2023-10-11T15:10:32.662 DEBUG:boto:Method: GET 2023-10-11T15:10:32.662 DEBUG:boto:Path: /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.663 DEBUG:boto:Data: 2023-10-11T15:10:32.664 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.665 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.666 DEBUG:boto:Params: {} 2023-10-11T15:10:32.667 DEBUG:boto:Token: None 2023-10-11T15:10:32.667 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b?versionId=null 2023-10-11T15:10:32.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA= 2023-10-11T15:10:32.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4ejpiMJD9x26X7xkTzsqT8rMdDA=', 'Content-Length': '0'} 2023-10-11T15:10:32.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab9ceab0e79048e-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.671 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.671 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.672 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.672 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.673 DEBUG:boto:Method: GET 2023-10-11T15:10:32.673 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:10:32.673 DEBUG:boto:Data: 2023-10-11T15:10:32.674 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.675 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.675 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.676 DEBUG:boto:Params: {} 2023-10-11T15:10:32.676 DEBUG:boto:Token: None 2023-10-11T15:10:32.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8fc15f6d81f3dde-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.681 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.682 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.682 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.683 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.683 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.684 DEBUG:boto:Method: GET 2023-10-11T15:10:32.684 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:10:32.684 DEBUG:boto:Data: 2023-10-11T15:10:32.685 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.685 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.686 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.686 DEBUG:boto:Params: {} 2023-10-11T15:10:32.687 DEBUG:boto:Token: None 2023-10-11T15:10:32.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3cc8a59840bbb7-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.730 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.731 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.732 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.732 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.733 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:10:32.733 DEBUG:boto:Data: 2023-10-11T15:10:32.734 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.734 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.735 DEBUG:boto:Params: {} 2023-10-11T15:10:32.735 DEBUG:boto:Token: None 2023-10-11T15:10:32.736 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b 2023-10-11T15:10:32.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/TOPB6m9e02qXOpCknm96sqnpw0= 2023-10-11T15:10:32.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/TOPB6m9e02qXOpCknm96sqnpw0=', 'Content-Length': '0'} 2023-10-11T15:10:32.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006086fb58f9a49c55-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.739 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.740 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.740 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.741 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:10:32.741 DEBUG:boto:Data: 2023-10-11T15:10:32.741 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.742 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.743 DEBUG:boto:Params: {} 2023-10-11T15:10:32.743 DEBUG:boto:Token: None 2023-10-11T15:10:32.744 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b 2023-10-11T15:10:32.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/TOPB6m9e02qXOpCknm96sqnpw0= 2023-10-11T15:10:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/TOPB6m9e02qXOpCknm96sqnpw0=', 'Content-Length': '0'} 2023-10-11T15:10:32.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfebd53fc867ebc-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:10:32.748 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.748 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.749 DEBUG:boto:Method: GET 2023-10-11T15:10:32.749 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:10:32.750 DEBUG:boto:Data: 2023-10-11T15:10:32.750 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.751 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.751 DEBUG:boto:Params: {} 2023-10-11T15:10:32.752 DEBUG:boto:Token: None 2023-10-11T15:10:32.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b 2023-10-11T15:10:32.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XrbBZoCb7Tcq/SpY265xzrtRiLs= 2023-10-11T15:10:32.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XrbBZoCb7Tcq/SpY265xzrtRiLs=', 'Content-Length': '0'} 2023-10-11T15:10:32.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d18a100038be9c-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.756 DEBUG:boto:path=/dghqvd-56/b 2023-10-11T15:10:32.757 DEBUG:boto:auth_path=/dghqvd-56/b 2023-10-11T15:10:32.757 DEBUG:boto:Method: GET 2023-10-11T15:10:32.758 DEBUG:boto:Path: /dghqvd-56/b 2023-10-11T15:10:32.758 DEBUG:boto:Data: 2023-10-11T15:10:32.759 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.760 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.760 DEBUG:boto:Params: {} 2023-10-11T15:10:32.760 DEBUG:boto:Token: None 2023-10-11T15:10:32.761 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/b 2023-10-11T15:10:32.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XrbBZoCb7Tcq/SpY265xzrtRiLs= 2023-10-11T15:10:32.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XrbBZoCb7Tcq/SpY265xzrtRiLs=', 'Content-Length': '0'} 2023-10-11T15:10:32.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f85bb896ba0ea69-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.765 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.765 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.766 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.766 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.766 DEBUG:boto:Method: GET 2023-10-11T15:10:32.767 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:10:32.767 DEBUG:boto:Data: 2023-10-11T15:10:32.768 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.769 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.769 DEBUG:boto:Params: {} 2023-10-11T15:10:32.770 DEBUG:boto:Token: None 2023-10-11T15:10:32.770 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000417cff49e4c5b183-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.774 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.775 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.775 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.776 DEBUG:boto:path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.776 DEBUG:boto:auth_path=/dghqvd-56/?prefix=b 2023-10-11T15:10:32.777 DEBUG:boto:Method: GET 2023-10-11T15:10:32.777 DEBUG:boto:Path: /dghqvd-56/?prefix=b 2023-10-11T15:10:32.778 DEBUG:boto:Data: 2023-10-11T15:10:32.778 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.779 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.779 DEBUG:boto:Params: {} 2023-10-11T15:10:32.780 DEBUG:boto:Token: None 2023-10-11T15:10:32.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694292b0fe50ae60-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.784 DEBUG:boto:b'dghqvd-56b1000falseb2023-10-11T15:07:40.412Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:10:32.785 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.785 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.786 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.786 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.787 DEBUG:boto:Method: GET 2023-10-11T15:10:32.787 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.788 DEBUG:boto:Data: 2023-10-11T15:10:32.788 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.789 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.789 DEBUG:boto:Params: {} 2023-10-11T15:10:32.790 DEBUG:boto:Token: None 2023-10-11T15:10:32.790 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M= 2023-10-11T15:10:32.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M=', 'Content-Length': '0'} 2023-10-11T15:10:32.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032da65cd753a4d5c-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.794 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.795 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.795 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.795 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.796 DEBUG:boto:Method: GET 2023-10-11T15:10:32.796 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.797 DEBUG:boto:Data: 2023-10-11T15:10:32.797 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.798 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.799 DEBUG:boto:Params: {} 2023-10-11T15:10:32.799 DEBUG:boto:Token: None 2023-10-11T15:10:32.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M= 2023-10-11T15:10:32.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M=', 'Content-Length': '0'} 2023-10-11T15:10:32.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092935f60c86a96ee-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.846 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.847 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.847 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.848 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.848 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.848 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.849 DEBUG:boto:Data: 2023-10-11T15:10:32.849 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.850 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.850 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.851 DEBUG:boto:Params: {} 2023-10-11T15:10:32.851 DEBUG:boto:Token: None 2023-10-11T15:10:32.852 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tv31tP5jz0N8PlDLp0is7EKnbpw= 2023-10-11T15:10:32.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tv31tP5jz0N8PlDLp0is7EKnbpw=', 'Content-Length': '0'} 2023-10-11T15:10:32.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bdd96e5d246a4e6-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.856 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.856 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.857 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.857 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.858 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.858 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.859 DEBUG:boto:Data: 2023-10-11T15:10:32.859 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.860 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.860 DEBUG:boto:Params: {} 2023-10-11T15:10:32.861 DEBUG:boto:Token: None 2023-10-11T15:10:32.861 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tv31tP5jz0N8PlDLp0is7EKnbpw= 2023-10-11T15:10:32.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tv31tP5jz0N8PlDLp0is7EKnbpw=', 'Content-Length': '0'} 2023-10-11T15:10:32.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd0483e272df462-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:10:32.866 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.866 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.866 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.867 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.867 DEBUG:boto:Method: GET 2023-10-11T15:10:32.868 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.868 DEBUG:boto:Data: 2023-10-11T15:10:32.869 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.869 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.870 DEBUG:boto:Params: {} 2023-10-11T15:10:32.870 DEBUG:boto:Token: None 2023-10-11T15:10:32.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M= 2023-10-11T15:10:32.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M=', 'Content-Length': '0'} 2023-10-11T15:10:32.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b277dfbaee26214e-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.915 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.915 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.916 DEBUG:boto:path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.916 DEBUG:boto:auth_path=/dghqvd-56/c?versionId=null 2023-10-11T15:10:32.916 DEBUG:boto:Method: GET 2023-10-11T15:10:32.917 DEBUG:boto:Path: /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.917 DEBUG:boto:Data: 2023-10-11T15:10:32.918 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.919 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.919 DEBUG:boto:Params: {} 2023-10-11T15:10:32.920 DEBUG:boto:Token: None 2023-10-11T15:10:32.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c?versionId=null 2023-10-11T15:10:32.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M= 2023-10-11T15:10:32.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bJq/Inwicv7+RbQG34x6D2WB93M=', 'Content-Length': '0'} 2023-10-11T15:10:32.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af6714de73a150a-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.924 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.925 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.925 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:10:32.925 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:10:32.926 DEBUG:boto:Method: GET 2023-10-11T15:10:32.926 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:10:32.927 DEBUG:boto:Data: 2023-10-11T15:10:32.927 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.928 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.929 DEBUG:boto:Params: {} 2023-10-11T15:10:32.929 DEBUG:boto:Token: None 2023-10-11T15:10:32.930 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000593739798c612bba-006526bae8-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.934 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.934 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:32.935 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:32.935 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:10:32.936 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:10:32.936 DEBUG:boto:Method: GET 2023-10-11T15:10:32.937 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:10:32.937 DEBUG:boto:Data: 2023-10-11T15:10:32.937 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.938 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.939 DEBUG:boto:Params: {} 2023-10-11T15:10:32.939 DEBUG:boto:Token: None 2023-10-11T15:10:32.940 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/ 2023-10-11T15:10:32.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss= 2023-10-11T15:10:32.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:woZh3tr3JmG8uUKm1pWDwYoWJss=', 'Content-Length': '0'} 2023-10-11T15:10:32.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cef5aba58a015ed-006526bae8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.943 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:32.944 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.944 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.945 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.945 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:10:32.946 DEBUG:boto:Data: 2023-10-11T15:10:32.946 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.947 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.947 DEBUG:boto:Params: {} 2023-10-11T15:10:32.948 DEBUG:boto:Token: None 2023-10-11T15:10:32.948 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c 2023-10-11T15:10:32.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JG1pr2deAQ8KfLZ2BX36uGyeKq8= 2023-10-11T15:10:32.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JG1pr2deAQ8KfLZ2BX36uGyeKq8=', 'Content-Length': '0'} 2023-10-11T15:10:32.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c648ad8f81d5f5db-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.952 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.952 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.952 DEBUG:boto:Method: HEAD 2023-10-11T15:10:32.953 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:10:32.953 DEBUG:boto:Data: 2023-10-11T15:10:32.954 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.955 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.955 DEBUG:boto:Params: {} 2023-10-11T15:10:32.955 DEBUG:boto:Token: None 2023-10-11T15:10:32.956 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c 2023-10-11T15:10:32.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JG1pr2deAQ8KfLZ2BX36uGyeKq8= 2023-10-11T15:10:32.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JG1pr2deAQ8KfLZ2BX36uGyeKq8=', 'Content-Length': '0'} 2023-10-11T15:10:32.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d29e2fc2d42dffe5-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:10:32.960 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.960 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.961 DEBUG:boto:Method: GET 2023-10-11T15:10:32.961 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:10:32.962 DEBUG:boto:Data: 2023-10-11T15:10:32.962 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:32.963 DEBUG:boto:Port: 8001 2023-10-11T15:10:32.963 DEBUG:boto:Params: {} 2023-10-11T15:10:32.964 DEBUG:boto:Token: None 2023-10-11T15:10:32.964 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c 2023-10-11T15:10:32.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKTYgl82iw20NOGB/cEWenaN6/A= 2023-10-11T15:10:32.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKTYgl82iw20NOGB/cEWenaN6/A=', 'Content-Length': '0'} 2023-10-11T15:10:32.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f77bcedbe51218-006526bae8-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:32.968 DEBUG:boto:path=/dghqvd-56/c 2023-10-11T15:10:32.968 DEBUG:boto:auth_path=/dghqvd-56/c 2023-10-11T15:10:32.969 DEBUG:boto:Method: GET 2023-10-11T15:10:32.969 DEBUG:boto:Path: /dghqvd-56/c 2023-10-11T15:10:32.970 DEBUG:boto:Data: 2023-10-11T15:10:32.970 DEBUG:boto:Headers: {} 2023-10-11T15:10:32.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:32.971 DEBUG:boto:Port: 8000 2023-10-11T15:10:32.971 DEBUG:boto:Params: {} 2023-10-11T15:10:32.972 DEBUG:boto:Token: None 2023-10-11T15:10:32.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:32 GMT /dghqvd-56/c 2023-10-11T15:10:32.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FKTYgl82iw20NOGB/cEWenaN6/A= 2023-10-11T15:10:32.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FKTYgl82iw20NOGB/cEWenaN6/A=', 'Content-Length': '0'} 2023-10-11T15:10:32.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba97da51a4c1b7e-006526bae8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.022 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.023 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.024 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:10:33.024 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:10:33.025 DEBUG:boto:Method: GET 2023-10-11T15:10:33.025 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:10:33.025 DEBUG:boto:Data: 2023-10-11T15:10:33.026 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.027 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.027 DEBUG:boto:Params: {} 2023-10-11T15:10:33.028 DEBUG:boto:Token: None 2023-10-11T15:10:33.028 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004cd7dfb8ade63c6-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.033 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.034 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.035 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.035 DEBUG:boto:path=/dghqvd-56/?prefix=c 2023-10-11T15:10:33.036 DEBUG:boto:auth_path=/dghqvd-56/?prefix=c 2023-10-11T15:10:33.036 DEBUG:boto:Method: GET 2023-10-11T15:10:33.036 DEBUG:boto:Path: /dghqvd-56/?prefix=c 2023-10-11T15:10:33.037 DEBUG:boto:Data: 2023-10-11T15:10:33.037 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.038 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.039 DEBUG:boto:Params: {} 2023-10-11T15:10:33.039 DEBUG:boto:Token: None 2023-10-11T15:10:33.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e0ce22ba7780052-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.043 DEBUG:boto:b'dghqvd-56c1000falsec2023-10-11T15:07:40.468Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:10:33.045 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.045 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.045 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.046 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.046 DEBUG:boto:Method: GET 2023-10-11T15:10:33.047 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.047 DEBUG:boto:Data: 2023-10-11T15:10:33.048 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.048 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.049 DEBUG:boto:Params: {} 2023-10-11T15:10:33.049 DEBUG:boto:Token: None 2023-10-11T15:10:33.050 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38= 2023-10-11T15:10:33.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38=', 'Content-Length': '0'} 2023-10-11T15:10:33.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1555e93d2168a67-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.054 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.054 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.055 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.055 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.056 DEBUG:boto:Method: GET 2023-10-11T15:10:33.056 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.056 DEBUG:boto:Data: 2023-10-11T15:10:33.057 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.058 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.058 DEBUG:boto:Params: {} 2023-10-11T15:10:33.059 DEBUG:boto:Token: None 2023-10-11T15:10:33.059 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38= 2023-10-11T15:10:33.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38=', 'Content-Length': '0'} 2023-10-11T15:10:33.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065bf31d9bfc865fa-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.063 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.063 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.064 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.064 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.064 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.065 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.065 DEBUG:boto:Data: 2023-10-11T15:10:33.066 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.067 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.067 DEBUG:boto:Params: {} 2023-10-11T15:10:33.067 DEBUG:boto:Token: None 2023-10-11T15:10:33.068 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5k5lDKbp5pPEpIwWp9pkV1pCVlA= 2023-10-11T15:10:33.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5k5lDKbp5pPEpIwWp9pkV1pCVlA=', 'Content-Length': '0'} 2023-10-11T15:10:33.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5ec5a9b08b9361-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.071 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.072 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.072 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.073 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.073 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.074 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.074 DEBUG:boto:Data: 2023-10-11T15:10:33.074 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.075 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.076 DEBUG:boto:Params: {} 2023-10-11T15:10:33.076 DEBUG:boto:Token: None 2023-10-11T15:10:33.077 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.077 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5k5lDKbp5pPEpIwWp9pkV1pCVlA= 2023-10-11T15:10:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5k5lDKbp5pPEpIwWp9pkV1pCVlA=', 'Content-Length': '0'} 2023-10-11T15:10:33.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc69e9a2a92ec092-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:10:33.081 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.081 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.081 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.082 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.082 DEBUG:boto:Method: GET 2023-10-11T15:10:33.083 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.083 DEBUG:boto:Data: 2023-10-11T15:10:33.084 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.085 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.085 DEBUG:boto:Params: {} 2023-10-11T15:10:33.086 DEBUG:boto:Token: None 2023-10-11T15:10:33.086 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38= 2023-10-11T15:10:33.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38=', 'Content-Length': '0'} 2023-10-11T15:10:33.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d839d81a1a5133cb-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.130 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.131 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.132 DEBUG:boto:path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.132 DEBUG:boto:auth_path=/dghqvd-56/d?versionId=null 2023-10-11T15:10:33.132 DEBUG:boto:Method: GET 2023-10-11T15:10:33.133 DEBUG:boto:Path: /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.133 DEBUG:boto:Data: 2023-10-11T15:10:33.134 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.135 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.135 DEBUG:boto:Params: {} 2023-10-11T15:10:33.136 DEBUG:boto:Token: None 2023-10-11T15:10:33.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d?versionId=null 2023-10-11T15:10:33.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38= 2023-10-11T15:10:33.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJnSPhEAAE+HO+sh3dMhmo6oh38=', 'Content-Length': '0'} 2023-10-11T15:10:33.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071d86d7d28985a26-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.140 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.141 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.141 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.142 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.142 DEBUG:boto:Method: GET 2023-10-11T15:10:33.142 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:10:33.143 DEBUG:boto:Data: 2023-10-11T15:10:33.143 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.144 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.145 DEBUG:boto:Params: {} 2023-10-11T15:10:33.145 DEBUG:boto:Token: None 2023-10-11T15:10:33.146 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a826b3aee9253741-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.150 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.150 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.151 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.151 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.152 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.152 DEBUG:boto:Method: GET 2023-10-11T15:10:33.153 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:10:33.153 DEBUG:boto:Data: 2023-10-11T15:10:33.154 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.155 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.155 DEBUG:boto:Params: {} 2023-10-11T15:10:33.155 DEBUG:boto:Token: None 2023-10-11T15:10:33.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef7d8dec79eed3a7-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.159 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.160 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.160 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.161 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.161 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:10:33.162 DEBUG:boto:Data: 2023-10-11T15:10:33.162 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.163 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.163 DEBUG:boto:Params: {} 2023-10-11T15:10:33.164 DEBUG:boto:Token: None 2023-10-11T15:10:33.164 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d 2023-10-11T15:10:33.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iDGx4IyxcS/suNbr2qBJefj+Bso= 2023-10-11T15:10:33.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iDGx4IyxcS/suNbr2qBJefj+Bso=', 'Content-Length': '0'} 2023-10-11T15:10:33.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020697c40d91390af-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.168 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.168 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.169 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.169 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:10:33.169 DEBUG:boto:Data: 2023-10-11T15:10:33.170 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.171 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.171 DEBUG:boto:Params: {} 2023-10-11T15:10:33.172 DEBUG:boto:Token: None 2023-10-11T15:10:33.172 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d 2023-10-11T15:10:33.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iDGx4IyxcS/suNbr2qBJefj+Bso= 2023-10-11T15:10:33.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iDGx4IyxcS/suNbr2qBJefj+Bso=', 'Content-Length': '0'} 2023-10-11T15:10:33.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c3411b88c2a0f1-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:10:33.176 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.177 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.177 DEBUG:boto:Method: GET 2023-10-11T15:10:33.177 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:10:33.178 DEBUG:boto:Data: 2023-10-11T15:10:33.178 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.179 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.179 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.180 DEBUG:boto:Params: {} 2023-10-11T15:10:33.180 DEBUG:boto:Token: None 2023-10-11T15:10:33.180 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d 2023-10-11T15:10:33.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgNMg7HtBa5khb2CQerk28GFKPg= 2023-10-11T15:10:33.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgNMg7HtBa5khb2CQerk28GFKPg=', 'Content-Length': '0'} 2023-10-11T15:10:33.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a329fb8525da77ff-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.184 DEBUG:boto:path=/dghqvd-56/d 2023-10-11T15:10:33.184 DEBUG:boto:auth_path=/dghqvd-56/d 2023-10-11T15:10:33.185 DEBUG:boto:Method: GET 2023-10-11T15:10:33.185 DEBUG:boto:Path: /dghqvd-56/d 2023-10-11T15:10:33.186 DEBUG:boto:Data: 2023-10-11T15:10:33.186 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.187 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.187 DEBUG:boto:Params: {} 2023-10-11T15:10:33.188 DEBUG:boto:Token: None 2023-10-11T15:10:33.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/d 2023-10-11T15:10:33.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgNMg7HtBa5khb2CQerk28GFKPg= 2023-10-11T15:10:33.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgNMg7HtBa5khb2CQerk28GFKPg=', 'Content-Length': '0'} 2023-10-11T15:10:33.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:07:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbd9706311603523-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.238 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.239 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.240 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.240 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.240 DEBUG:boto:Method: GET 2023-10-11T15:10:33.241 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:10:33.241 DEBUG:boto:Data: 2023-10-11T15:10:33.242 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.243 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.243 DEBUG:boto:Params: {} 2023-10-11T15:10:33.244 DEBUG:boto:Token: None 2023-10-11T15:10:33.244 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deef8c0c85ec97ec-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.248 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.249 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.249 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.250 DEBUG:boto:path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.250 DEBUG:boto:auth_path=/dghqvd-56/?prefix=d 2023-10-11T15:10:33.251 DEBUG:boto:Method: GET 2023-10-11T15:10:33.251 DEBUG:boto:Path: /dghqvd-56/?prefix=d 2023-10-11T15:10:33.252 DEBUG:boto:Data: 2023-10-11T15:10:33.252 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.253 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.253 DEBUG:boto:Params: {} 2023-10-11T15:10:33.254 DEBUG:boto:Token: None 2023-10-11T15:10:33.254 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d394650e989f0b34-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.258 DEBUG:boto:b'dghqvd-56d1000falsed2023-10-11T15:07:40.486Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:10:33.259 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.259 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.260 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.260 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.261 DEBUG:boto:Method: GET 2023-10-11T15:10:33.261 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.262 DEBUG:boto:Data: 2023-10-11T15:10:33.262 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.263 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.263 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.264 DEBUG:boto:Params: {} 2023-10-11T15:10:33.264 DEBUG:boto:Token: None 2023-10-11T15:10:33.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU= 2023-10-11T15:10:33.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU=', 'Content-Length': '0'} 2023-10-11T15:10:33.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001acac15f7f90213c-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.310 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.311 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.311 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.312 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.312 DEBUG:boto:Method: GET 2023-10-11T15:10:33.313 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.313 DEBUG:boto:Data: 2023-10-11T15:10:33.314 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.315 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.315 DEBUG:boto:Params: {} 2023-10-11T15:10:33.316 DEBUG:boto:Token: None 2023-10-11T15:10:33.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU= 2023-10-11T15:10:33.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU=', 'Content-Length': '0'} 2023-10-11T15:10:33.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4399f5cd54253f-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.320 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.321 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.321 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.322 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.322 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.322 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.323 DEBUG:boto:Data: 2023-10-11T15:10:33.323 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.324 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.324 DEBUG:boto:Params: {} 2023-10-11T15:10:33.325 DEBUG:boto:Token: None 2023-10-11T15:10:33.325 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:piK0ri3HMJy9UG5ekakJ2RWpZYQ= 2023-10-11T15:10:33.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:piK0ri3HMJy9UG5ekakJ2RWpZYQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad47ed55e5436f9-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.329 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.329 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.330 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.330 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.331 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.331 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.331 DEBUG:boto:Data: 2023-10-11T15:10:33.332 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.333 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.333 DEBUG:boto:Params: {} 2023-10-11T15:10:33.334 DEBUG:boto:Token: None 2023-10-11T15:10:33.334 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:piK0ri3HMJy9UG5ekakJ2RWpZYQ= 2023-10-11T15:10:33.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:piK0ri3HMJy9UG5ekakJ2RWpZYQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc27466afc9d0c8-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:10:33.338 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.339 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.339 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.340 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.340 DEBUG:boto:Method: GET 2023-10-11T15:10:33.340 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.341 DEBUG:boto:Data: 2023-10-11T15:10:33.341 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.342 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.342 DEBUG:boto:Params: {} 2023-10-11T15:10:33.343 DEBUG:boto:Token: None 2023-10-11T15:10:33.343 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU= 2023-10-11T15:10:33.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU=', 'Content-Length': '0'} 2023-10-11T15:10:33.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007121758452d08216-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.347 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.347 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.348 DEBUG:boto:path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.348 DEBUG:boto:auth_path=/dghqvd-56/e?versionId=null 2023-10-11T15:10:33.349 DEBUG:boto:Method: GET 2023-10-11T15:10:33.349 DEBUG:boto:Path: /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.350 DEBUG:boto:Data: 2023-10-11T15:10:33.350 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.351 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.352 DEBUG:boto:Params: {} 2023-10-11T15:10:33.352 DEBUG:boto:Token: None 2023-10-11T15:10:33.353 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e?versionId=null 2023-10-11T15:10:33.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU= 2023-10-11T15:10:33.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RORiFf3i5yExoYmDeEwcvqXjthU=', 'Content-Length': '0'} 2023-10-11T15:10:33.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d013bd903fdc450-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.357 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.357 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.358 DEBUG:boto:path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.358 DEBUG:boto:auth_path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.358 DEBUG:boto:Method: GET 2023-10-11T15:10:33.359 DEBUG:boto:Path: /dghqvd-56/?prefix=e 2023-10-11T15:10:33.359 DEBUG:boto:Data: 2023-10-11T15:10:33.360 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.360 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.360 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.361 DEBUG:boto:Params: {} 2023-10-11T15:10:33.361 DEBUG:boto:Token: None 2023-10-11T15:10:33.362 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.362 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1146e95e48ad85-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.366 DEBUG:boto:b'dghqvd-56e1000falsee2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.367 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.367 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.368 DEBUG:boto:path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.368 DEBUG:boto:auth_path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.369 DEBUG:boto:Method: GET 2023-10-11T15:10:33.369 DEBUG:boto:Path: /dghqvd-56/?prefix=e 2023-10-11T15:10:33.369 DEBUG:boto:Data: 2023-10-11T15:10:33.370 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.371 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.371 DEBUG:boto:Params: {} 2023-10-11T15:10:33.372 DEBUG:boto:Token: None 2023-10-11T15:10:33.372 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad7a86fa587ff30b-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.376 DEBUG:boto:b'dghqvd-56e1000falsee2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.376 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.377 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.377 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.377 DEBUG:boto:Path: /dghqvd-56/e 2023-10-11T15:10:33.378 DEBUG:boto:Data: 2023-10-11T15:10:33.378 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.379 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.380 DEBUG:boto:Params: {} 2023-10-11T15:10:33.380 DEBUG:boto:Token: None 2023-10-11T15:10:33.380 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e 2023-10-11T15:10:33.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mrRZXlxH7BcL5TKMMjQMp0qLNZo= 2023-10-11T15:10:33.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mrRZXlxH7BcL5TKMMjQMp0qLNZo=', 'Content-Length': '0'} 2023-10-11T15:10:33.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fbe0291287975a-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.384 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.385 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.385 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.385 DEBUG:boto:Path: /dghqvd-56/e 2023-10-11T15:10:33.386 DEBUG:boto:Data: 2023-10-11T15:10:33.386 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.387 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.388 DEBUG:boto:Params: {} 2023-10-11T15:10:33.388 DEBUG:boto:Token: None 2023-10-11T15:10:33.389 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e 2023-10-11T15:10:33.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mrRZXlxH7BcL5TKMMjQMp0qLNZo= 2023-10-11T15:10:33.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mrRZXlxH7BcL5TKMMjQMp0qLNZo=', 'Content-Length': '0'} 2023-10-11T15:10:33.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c023528f9d78a18d-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:10:33.393 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.393 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.394 DEBUG:boto:Method: GET 2023-10-11T15:10:33.394 DEBUG:boto:Path: /dghqvd-56/e 2023-10-11T15:10:33.395 DEBUG:boto:Data: 2023-10-11T15:10:33.395 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.396 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.396 DEBUG:boto:Params: {} 2023-10-11T15:10:33.397 DEBUG:boto:Token: None 2023-10-11T15:10:33.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e 2023-10-11T15:10:33.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0m+LuBaQgYGxrHhP3N8mt9SuM3g= 2023-10-11T15:10:33.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0m+LuBaQgYGxrHhP3N8mt9SuM3g=', 'Content-Length': '0'} 2023-10-11T15:10:33.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040df0cdf64111e98-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.401 DEBUG:boto:path=/dghqvd-56/e 2023-10-11T15:10:33.401 DEBUG:boto:auth_path=/dghqvd-56/e 2023-10-11T15:10:33.402 DEBUG:boto:Method: GET 2023-10-11T15:10:33.402 DEBUG:boto:Path: /dghqvd-56/e 2023-10-11T15:10:33.403 DEBUG:boto:Data: 2023-10-11T15:10:33.403 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.404 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.405 DEBUG:boto:Params: {} 2023-10-11T15:10:33.405 DEBUG:boto:Token: None 2023-10-11T15:10:33.406 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/e 2023-10-11T15:10:33.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0m+LuBaQgYGxrHhP3N8mt9SuM3g= 2023-10-11T15:10:33.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0m+LuBaQgYGxrHhP3N8mt9SuM3g=', 'Content-Length': '0'} 2023-10-11T15:10:33.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004846d658a42a8fd0-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.450 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.451 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.452 DEBUG:boto:path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.452 DEBUG:boto:auth_path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.452 DEBUG:boto:Method: GET 2023-10-11T15:10:33.453 DEBUG:boto:Path: /dghqvd-56/?prefix=e 2023-10-11T15:10:33.453 DEBUG:boto:Data: 2023-10-11T15:10:33.454 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.455 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.455 DEBUG:boto:Params: {} 2023-10-11T15:10:33.456 DEBUG:boto:Token: None 2023-10-11T15:10:33.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2369906145399f-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.460 DEBUG:boto:b'dghqvd-56e1000falsee2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.461 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.462 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.462 DEBUG:boto:path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.463 DEBUG:boto:auth_path=/dghqvd-56/?prefix=e 2023-10-11T15:10:33.463 DEBUG:boto:Method: GET 2023-10-11T15:10:33.463 DEBUG:boto:Path: /dghqvd-56/?prefix=e 2023-10-11T15:10:33.464 DEBUG:boto:Data: 2023-10-11T15:10:33.464 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.465 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.466 DEBUG:boto:Params: {} 2023-10-11T15:10:33.466 DEBUG:boto:Token: None 2023-10-11T15:10:33.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009883c0be01b958e7-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.472 DEBUG:boto:b'dghqvd-56e1000falsee2023-10-11T15:08:11.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:10:33.473 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.473 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.474 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.474 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.475 DEBUG:boto:Method: GET 2023-10-11T15:10:33.475 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.475 DEBUG:boto:Data: 2023-10-11T15:10:33.476 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.476 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.477 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.477 DEBUG:boto:Params: {} 2023-10-11T15:10:33.478 DEBUG:boto:Token: None 2023-10-11T15:10:33.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4= 2023-10-11T15:10:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4=', 'Content-Length': '0'} 2023-10-11T15:10:33.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000202c4ed8f9283860-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.522 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.523 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.523 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.524 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.524 DEBUG:boto:Method: GET 2023-10-11T15:10:33.525 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.525 DEBUG:boto:Data: 2023-10-11T15:10:33.526 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.526 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.527 DEBUG:boto:Params: {} 2023-10-11T15:10:33.528 DEBUG:boto:Token: None 2023-10-11T15:10:33.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4= 2023-10-11T15:10:33.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4=', 'Content-Length': '0'} 2023-10-11T15:10:33.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2361f23b5be6c9-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.533 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.534 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.535 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.535 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.535 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.536 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.536 DEBUG:boto:Data: 2023-10-11T15:10:33.537 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.537 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.538 DEBUG:boto:Params: {} 2023-10-11T15:10:33.538 DEBUG:boto:Token: None 2023-10-11T15:10:33.539 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M8vBg645M79XGD6ovoky/VWddqc= 2023-10-11T15:10:33.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M8vBg645M79XGD6ovoky/VWddqc=', 'Content-Length': '0'} 2023-10-11T15:10:33.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002280a767cedb011a-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.543 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.543 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.544 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.544 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.544 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.545 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.545 DEBUG:boto:Data: 2023-10-11T15:10:33.546 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.547 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.547 DEBUG:boto:Params: {} 2023-10-11T15:10:33.547 DEBUG:boto:Token: None 2023-10-11T15:10:33.548 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M8vBg645M79XGD6ovoky/VWddqc= 2023-10-11T15:10:33.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M8vBg645M79XGD6ovoky/VWddqc=', 'Content-Length': '0'} 2023-10-11T15:10:33.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000162026d8d9405d-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:10:33.552 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.553 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.553 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.554 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.554 DEBUG:boto:Method: GET 2023-10-11T15:10:33.554 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.555 DEBUG:boto:Data: 2023-10-11T15:10:33.555 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.556 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.556 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.557 DEBUG:boto:Params: {} 2023-10-11T15:10:33.557 DEBUG:boto:Token: None 2023-10-11T15:10:33.558 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4= 2023-10-11T15:10:33.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4=', 'Content-Length': '0'} 2023-10-11T15:10:33.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069229788978c5b0a-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.561 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.562 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.562 DEBUG:boto:path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.563 DEBUG:boto:auth_path=/dghqvd-56/f?versionId=null 2023-10-11T15:10:33.563 DEBUG:boto:Method: GET 2023-10-11T15:10:33.564 DEBUG:boto:Path: /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.564 DEBUG:boto:Data: 2023-10-11T15:10:33.565 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.565 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.565 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.566 DEBUG:boto:Params: {} 2023-10-11T15:10:33.566 DEBUG:boto:Token: None 2023-10-11T15:10:33.567 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f?versionId=null 2023-10-11T15:10:33.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4= 2023-10-11T15:10:33.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EpKcq2R6Na9LYmqrATzJzgmlFb4=', 'Content-Length': '0'} 2023-10-11T15:10:33.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aca7144767b7a7d-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.571 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.571 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.572 DEBUG:boto:path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.572 DEBUG:boto:auth_path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.572 DEBUG:boto:Method: GET 2023-10-11T15:10:33.573 DEBUG:boto:Path: /dghqvd-56/?prefix=f 2023-10-11T15:10:33.573 DEBUG:boto:Data: 2023-10-11T15:10:33.574 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.575 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.575 DEBUG:boto:Params: {} 2023-10-11T15:10:33.576 DEBUG:boto:Token: None 2023-10-11T15:10:33.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020cbbf013da35e39-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.580 DEBUG:boto:b'dghqvd-56f1000falsef2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.581 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.581 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.582 DEBUG:boto:path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.582 DEBUG:boto:auth_path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.583 DEBUG:boto:Method: GET 2023-10-11T15:10:33.583 DEBUG:boto:Path: /dghqvd-56/?prefix=f 2023-10-11T15:10:33.584 DEBUG:boto:Data: 2023-10-11T15:10:33.584 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.585 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.585 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.586 DEBUG:boto:Params: {} 2023-10-11T15:10:33.586 DEBUG:boto:Token: None 2023-10-11T15:10:33.587 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.587 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df08983b5e65e8f6-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.630 DEBUG:boto:b'dghqvd-56f1000falsef2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.631 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.632 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.632 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.633 DEBUG:boto:Path: /dghqvd-56/f 2023-10-11T15:10:33.633 DEBUG:boto:Data: 2023-10-11T15:10:33.633 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.634 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.635 DEBUG:boto:Params: {} 2023-10-11T15:10:33.635 DEBUG:boto:Token: None 2023-10-11T15:10:33.636 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f 2023-10-11T15:10:33.637 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5Bk0mmJfXtJK69/6dDJ7OZ57Syg= 2023-10-11T15:10:33.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5Bk0mmJfXtJK69/6dDJ7OZ57Syg=', 'Content-Length': '0'} 2023-10-11T15:10:33.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5259c72fbf0c8d4-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.640 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.640 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.641 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.641 DEBUG:boto:Path: /dghqvd-56/f 2023-10-11T15:10:33.641 DEBUG:boto:Data: 2023-10-11T15:10:33.642 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.643 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.643 DEBUG:boto:Params: {} 2023-10-11T15:10:33.643 DEBUG:boto:Token: None 2023-10-11T15:10:33.644 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f 2023-10-11T15:10:33.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5Bk0mmJfXtJK69/6dDJ7OZ57Syg= 2023-10-11T15:10:33.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5Bk0mmJfXtJK69/6dDJ7OZ57Syg=', 'Content-Length': '0'} 2023-10-11T15:10:33.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db171fc4a9f46f2a-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:10:33.648 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.648 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.649 DEBUG:boto:Method: GET 2023-10-11T15:10:33.649 DEBUG:boto:Path: /dghqvd-56/f 2023-10-11T15:10:33.650 DEBUG:boto:Data: 2023-10-11T15:10:33.650 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.651 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.651 DEBUG:boto:Params: {} 2023-10-11T15:10:33.652 DEBUG:boto:Token: None 2023-10-11T15:10:33.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f 2023-10-11T15:10:33.653 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fEEVIbJeIn/8BsBfkeK5ZhxW8tk= 2023-10-11T15:10:33.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fEEVIbJeIn/8BsBfkeK5ZhxW8tk=', 'Content-Length': '0'} 2023-10-11T15:10:33.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad91720f196b648f-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.656 DEBUG:boto:path=/dghqvd-56/f 2023-10-11T15:10:33.656 DEBUG:boto:auth_path=/dghqvd-56/f 2023-10-11T15:10:33.657 DEBUG:boto:Method: GET 2023-10-11T15:10:33.657 DEBUG:boto:Path: /dghqvd-56/f 2023-10-11T15:10:33.658 DEBUG:boto:Data: 2023-10-11T15:10:33.658 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.659 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.659 DEBUG:boto:Params: {} 2023-10-11T15:10:33.660 DEBUG:boto:Token: None 2023-10-11T15:10:33.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/f 2023-10-11T15:10:33.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fEEVIbJeIn/8BsBfkeK5ZhxW8tk= 2023-10-11T15:10:33.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fEEVIbJeIn/8BsBfkeK5ZhxW8tk=', 'Content-Length': '0'} 2023-10-11T15:10:33.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002af61c775bb61068-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.664 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.664 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.664 DEBUG:boto:path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.665 DEBUG:boto:auth_path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.665 DEBUG:boto:Method: GET 2023-10-11T15:10:33.666 DEBUG:boto:Path: /dghqvd-56/?prefix=f 2023-10-11T15:10:33.666 DEBUG:boto:Data: 2023-10-11T15:10:33.667 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.667 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.668 DEBUG:boto:Params: {} 2023-10-11T15:10:33.668 DEBUG:boto:Token: None 2023-10-11T15:10:33.669 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002822ecf6795b4f38-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.714 DEBUG:boto:b'dghqvd-56f1000falsef2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.715 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.716 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.716 DEBUG:boto:path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.717 DEBUG:boto:auth_path=/dghqvd-56/?prefix=f 2023-10-11T15:10:33.717 DEBUG:boto:Method: GET 2023-10-11T15:10:33.718 DEBUG:boto:Path: /dghqvd-56/?prefix=f 2023-10-11T15:10:33.718 DEBUG:boto:Data: 2023-10-11T15:10:33.718 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.719 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.720 DEBUG:boto:Params: {} 2023-10-11T15:10:33.720 DEBUG:boto:Token: None 2023-10-11T15:10:33.721 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d935fcbce9f334ec-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.725 DEBUG:boto:b'dghqvd-56f1000falsef2023-10-11T15:08:11.931Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:10:33.726 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.726 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.727 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.727 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.728 DEBUG:boto:Method: GET 2023-10-11T15:10:33.728 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.729 DEBUG:boto:Data: 2023-10-11T15:10:33.729 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.731 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.732 DEBUG:boto:Params: {} 2023-10-11T15:10:33.732 DEBUG:boto:Token: None 2023-10-11T15:10:33.733 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA= 2023-10-11T15:10:33.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA=', 'Content-Length': '0'} 2023-10-11T15:10:33.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2ca5caa7bedb32-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.737 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.737 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.738 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.738 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.738 DEBUG:boto:Method: GET 2023-10-11T15:10:33.739 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.739 DEBUG:boto:Data: 2023-10-11T15:10:33.740 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.741 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.741 DEBUG:boto:Params: {} 2023-10-11T15:10:33.742 DEBUG:boto:Token: None 2023-10-11T15:10:33.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA= 2023-10-11T15:10:33.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA=', 'Content-Length': '0'} 2023-10-11T15:10:33.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204dd2282909dbfe-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.746 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.746 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.747 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.747 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.747 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.748 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.748 DEBUG:boto:Data: 2023-10-11T15:10:33.749 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.750 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.750 DEBUG:boto:Params: {} 2023-10-11T15:10:33.750 DEBUG:boto:Token: None 2023-10-11T15:10:33.751 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cKcpq8ttthAaaL55MlXyRluH7v4= 2023-10-11T15:10:33.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cKcpq8ttthAaaL55MlXyRluH7v4=', 'Content-Length': '0'} 2023-10-11T15:10:33.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f32cbcc82fafd73-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.754 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.755 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.755 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.756 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.756 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.756 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.757 DEBUG:boto:Data: 2023-10-11T15:10:33.757 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.758 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.759 DEBUG:boto:Params: {} 2023-10-11T15:10:33.759 DEBUG:boto:Token: None 2023-10-11T15:10:33.759 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cKcpq8ttthAaaL55MlXyRluH7v4= 2023-10-11T15:10:33.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cKcpq8ttthAaaL55MlXyRluH7v4=', 'Content-Length': '0'} 2023-10-11T15:10:33.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec453129b4693f4-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:10:33.763 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.764 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.764 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.764 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.765 DEBUG:boto:Method: GET 2023-10-11T15:10:33.765 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.766 DEBUG:boto:Data: 2023-10-11T15:10:33.766 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.767 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.767 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.767 DEBUG:boto:Params: {} 2023-10-11T15:10:33.768 DEBUG:boto:Token: None 2023-10-11T15:10:33.769 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA= 2023-10-11T15:10:33.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA=', 'Content-Length': '0'} 2023-10-11T15:10:33.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6487a44678d23c-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.772 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.772 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.773 DEBUG:boto:path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.773 DEBUG:boto:auth_path=/dghqvd-56/g?versionId=null 2023-10-11T15:10:33.774 DEBUG:boto:Method: GET 2023-10-11T15:10:33.774 DEBUG:boto:Path: /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.775 DEBUG:boto:Data: 2023-10-11T15:10:33.775 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.776 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.776 DEBUG:boto:Params: {} 2023-10-11T15:10:33.777 DEBUG:boto:Token: None 2023-10-11T15:10:33.777 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g?versionId=null 2023-10-11T15:10:33.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA= 2023-10-11T15:10:33.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KJ8Ltx3X8c1LX/B/bDMdakc6TSA=', 'Content-Length': '0'} 2023-10-11T15:10:33.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee082ce27d58a744-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.822 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.823 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.824 DEBUG:boto:path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.824 DEBUG:boto:auth_path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.825 DEBUG:boto:Method: GET 2023-10-11T15:10:33.825 DEBUG:boto:Path: /dghqvd-56/?prefix=g 2023-10-11T15:10:33.826 DEBUG:boto:Data: 2023-10-11T15:10:33.826 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.827 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.828 DEBUG:boto:Params: {} 2023-10-11T15:10:33.828 DEBUG:boto:Token: None 2023-10-11T15:10:33.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000608cb00866c0a596-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.833 DEBUG:boto:b'dghqvd-56g1000falseg2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.834 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.834 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.835 DEBUG:boto:path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.835 DEBUG:boto:auth_path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.836 DEBUG:boto:Method: GET 2023-10-11T15:10:33.836 DEBUG:boto:Path: /dghqvd-56/?prefix=g 2023-10-11T15:10:33.837 DEBUG:boto:Data: 2023-10-11T15:10:33.837 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.838 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.839 DEBUG:boto:Params: {} 2023-10-11T15:10:33.839 DEBUG:boto:Token: None 2023-10-11T15:10:33.840 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b514dd5a0556ff1-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.843 DEBUG:boto:b'dghqvd-56g1000falseg2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.844 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.844 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.845 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.845 DEBUG:boto:Path: /dghqvd-56/g 2023-10-11T15:10:33.846 DEBUG:boto:Data: 2023-10-11T15:10:33.846 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.847 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.847 DEBUG:boto:Params: {} 2023-10-11T15:10:33.848 DEBUG:boto:Token: None 2023-10-11T15:10:33.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g 2023-10-11T15:10:33.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oI/o8aDaZ4/4rFQ21accKeEXIQI= 2023-10-11T15:10:33.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oI/o8aDaZ4/4rFQ21accKeEXIQI=', 'Content-Length': '0'} 2023-10-11T15:10:33.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c06f8df6b5e782a1-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.852 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.852 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.853 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.853 DEBUG:boto:Path: /dghqvd-56/g 2023-10-11T15:10:33.853 DEBUG:boto:Data: 2023-10-11T15:10:33.854 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.855 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.855 DEBUG:boto:Params: {} 2023-10-11T15:10:33.855 DEBUG:boto:Token: None 2023-10-11T15:10:33.856 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g 2023-10-11T15:10:33.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oI/o8aDaZ4/4rFQ21accKeEXIQI= 2023-10-11T15:10:33.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oI/o8aDaZ4/4rFQ21accKeEXIQI=', 'Content-Length': '0'} 2023-10-11T15:10:33.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d756c33f25015c3a-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:10:33.860 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.861 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.861 DEBUG:boto:Method: GET 2023-10-11T15:10:33.861 DEBUG:boto:Path: /dghqvd-56/g 2023-10-11T15:10:33.862 DEBUG:boto:Data: 2023-10-11T15:10:33.863 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.864 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.864 DEBUG:boto:Params: {} 2023-10-11T15:10:33.864 DEBUG:boto:Token: None 2023-10-11T15:10:33.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g 2023-10-11T15:10:33.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8nHL8bYGChtm9N1+po4TMJsAeF8= 2023-10-11T15:10:33.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8nHL8bYGChtm9N1+po4TMJsAeF8=', 'Content-Length': '0'} 2023-10-11T15:10:33.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ec57b1b133e637-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.869 DEBUG:boto:path=/dghqvd-56/g 2023-10-11T15:10:33.869 DEBUG:boto:auth_path=/dghqvd-56/g 2023-10-11T15:10:33.870 DEBUG:boto:Method: GET 2023-10-11T15:10:33.871 DEBUG:boto:Path: /dghqvd-56/g 2023-10-11T15:10:33.871 DEBUG:boto:Data: 2023-10-11T15:10:33.872 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.873 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.873 DEBUG:boto:Params: {} 2023-10-11T15:10:33.874 DEBUG:boto:Token: None 2023-10-11T15:10:33.874 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/g 2023-10-11T15:10:33.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8nHL8bYGChtm9N1+po4TMJsAeF8= 2023-10-11T15:10:33.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8nHL8bYGChtm9N1+po4TMJsAeF8=', 'Content-Length': '0'} 2023-10-11T15:10:33.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008206f9a9a7c83869-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.918 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.919 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.920 DEBUG:boto:path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.920 DEBUG:boto:auth_path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.921 DEBUG:boto:Method: GET 2023-10-11T15:10:33.921 DEBUG:boto:Path: /dghqvd-56/?prefix=g 2023-10-11T15:10:33.922 DEBUG:boto:Data: 2023-10-11T15:10:33.922 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.923 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.923 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.924 DEBUG:boto:Params: {} 2023-10-11T15:10:33.924 DEBUG:boto:Token: None 2023-10-11T15:10:33.925 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dadfd12d4cc030a6-006526bae9-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.929 DEBUG:boto:b'dghqvd-56g1000falseg2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.930 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:33.931 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:33.931 DEBUG:boto:path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.932 DEBUG:boto:auth_path=/dghqvd-56/?prefix=g 2023-10-11T15:10:33.933 DEBUG:boto:Method: GET 2023-10-11T15:10:33.933 DEBUG:boto:Path: /dghqvd-56/?prefix=g 2023-10-11T15:10:33.934 DEBUG:boto:Data: 2023-10-11T15:10:33.934 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.935 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.936 DEBUG:boto:Params: {} 2023-10-11T15:10:33.936 DEBUG:boto:Token: None 2023-10-11T15:10:33.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/ 2023-10-11T15:10:33.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ= 2023-10-11T15:10:33.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7x7/vLsEAnb9LCE6G3WbZUeVuPQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3871c7948a3d115-006526bae9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.940 DEBUG:boto:b'dghqvd-56g1000falseg2023-10-11T15:08:11.989Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:33.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:10:33.942 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.942 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.943 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.943 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.944 DEBUG:boto:Method: GET 2023-10-11T15:10:33.945 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.945 DEBUG:boto:Data: 2023-10-11T15:10:33.946 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.946 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.947 DEBUG:boto:Params: {} 2023-10-11T15:10:33.947 DEBUG:boto:Token: None 2023-10-11T15:10:33.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU= 2023-10-11T15:10:33.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU=', 'Content-Length': '0'} 2023-10-11T15:10:33.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382c5ebff150606a-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.953 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.954 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.954 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.955 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.955 DEBUG:boto:Method: GET 2023-10-11T15:10:33.955 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.956 DEBUG:boto:Data: 2023-10-11T15:10:33.956 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.957 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.958 DEBUG:boto:Params: {} 2023-10-11T15:10:33.958 DEBUG:boto:Token: None 2023-10-11T15:10:33.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU= 2023-10-11T15:10:33.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU=', 'Content-Length': '0'} 2023-10-11T15:10:33.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91c5d68860244c5-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.963 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.964 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.964 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.965 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.966 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.967 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.967 DEBUG:boto:Data: 2023-10-11T15:10:33.968 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.968 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.969 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.969 DEBUG:boto:Params: {} 2023-10-11T15:10:33.970 DEBUG:boto:Token: None 2023-10-11T15:10:33.970 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2rs1+N6u5k4T88oZNmFaEzz6pBQ= 2023-10-11T15:10:33.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rs1+N6u5k4T88oZNmFaEzz6pBQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000876ff90d72d1b99-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.974 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.975 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.976 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.976 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.977 DEBUG:boto:Method: HEAD 2023-10-11T15:10:33.977 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.978 DEBUG:boto:Data: 2023-10-11T15:10:33.978 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:33.979 DEBUG:boto:Port: 8000 2023-10-11T15:10:33.980 DEBUG:boto:Params: {} 2023-10-11T15:10:33.980 DEBUG:boto:Token: None 2023-10-11T15:10:33.981 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2rs1+N6u5k4T88oZNmFaEzz6pBQ= 2023-10-11T15:10:33.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rs1+N6u5k4T88oZNmFaEzz6pBQ=', 'Content-Length': '0'} 2023-10-11T15:10:33.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9bd2ec11f9fa635-006526bae9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:10:33.985 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.985 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.986 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.986 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.987 DEBUG:boto:Method: GET 2023-10-11T15:10:33.988 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.988 DEBUG:boto:Data: 2023-10-11T15:10:33.989 DEBUG:boto:Headers: {} 2023-10-11T15:10:33.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:33.990 DEBUG:boto:Port: 8001 2023-10-11T15:10:33.990 DEBUG:boto:Params: {} 2023-10-11T15:10:33.991 DEBUG:boto:Token: None 2023-10-11T15:10:33.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:33 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU= 2023-10-11T15:10:33.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DG3WfLIwt83YvacZ5yyJu6Y2NnU=', 'Content-Length': '0'} 2023-10-11T15:10:33.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec7580907afd4a9-006526bae9-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:33.996 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:33.996 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:33.997 DEBUG:boto:path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.997 DEBUG:boto:auth_path=/dghqvd-56/h?versionId=null 2023-10-11T15:10:33.998 DEBUG:boto:Method: GET 2023-10-11T15:10:33.998 DEBUG:boto:Path: /dghqvd-56/h?versionId=null 2023-10-11T15:10:33.999 DEBUG:boto:Data: 2023-10-11T15:10:33.999 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.000 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.001 DEBUG:boto:Params: {} 2023-10-11T15:10:34.001 DEBUG:boto:Token: None 2023-10-11T15:10:34.002 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/h?versionId=null 2023-10-11T15:10:34.003 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YpeRNjSgEV4qmWtrswWYqv7qDLg= 2023-10-11T15:10:34.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YpeRNjSgEV4qmWtrswWYqv7qDLg=', 'Content-Length': '0'} 2023-10-11T15:10:34.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7f61cf35e1e193-006526baea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.006 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:34.007 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:34.007 DEBUG:boto:path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.008 DEBUG:boto:auth_path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.008 DEBUG:boto:Method: GET 2023-10-11T15:10:34.008 DEBUG:boto:Path: /dghqvd-56/?prefix=h 2023-10-11T15:10:34.009 DEBUG:boto:Data: 2023-10-11T15:10:34.009 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:34.013 DEBUG:boto:Port: 8001 2023-10-11T15:10:34.014 DEBUG:boto:Params: {} 2023-10-11T15:10:34.014 DEBUG:boto:Token: None 2023-10-11T15:10:34.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/ 2023-10-11T15:10:34.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA= 2023-10-11T15:10:34.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA=', 'Content-Length': '0'} 2023-10-11T15:10:34.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4680acb43ace3f-006526baea-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.062 DEBUG:boto:b'dghqvd-56h1000falseh2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:34.063 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:34.064 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:34.064 DEBUG:boto:path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.065 DEBUG:boto:auth_path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.065 DEBUG:boto:Method: GET 2023-10-11T15:10:34.065 DEBUG:boto:Path: /dghqvd-56/?prefix=h 2023-10-11T15:10:34.066 DEBUG:boto:Data: 2023-10-11T15:10:34.066 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.067 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.068 DEBUG:boto:Params: {} 2023-10-11T15:10:34.068 DEBUG:boto:Token: None 2023-10-11T15:10:34.069 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/ 2023-10-11T15:10:34.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA= 2023-10-11T15:10:34.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA=', 'Content-Length': '0'} 2023-10-11T15:10:34.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a42ccb12649ca6-006526baea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.072 DEBUG:boto:b'dghqvd-56h1000falseh2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:34.073 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:34.074 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:34.074 DEBUG:boto:Method: HEAD 2023-10-11T15:10:34.074 DEBUG:boto:Path: /dghqvd-56/h 2023-10-11T15:10:34.075 DEBUG:boto:Data: 2023-10-11T15:10:34.075 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:34.076 DEBUG:boto:Port: 8001 2023-10-11T15:10:34.076 DEBUG:boto:Params: {} 2023-10-11T15:10:34.077 DEBUG:boto:Token: None 2023-10-11T15:10:34.077 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/h 2023-10-11T15:10:34.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UYzwf9Lcysk1j4KLm9pXNugtJZw= 2023-10-11T15:10:34.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UYzwf9Lcysk1j4KLm9pXNugtJZw=', 'Content-Length': '0'} 2023-10-11T15:10:34.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d16257bbd9e1fb-006526baea-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.081 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:34.081 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:34.082 DEBUG:boto:Method: HEAD 2023-10-11T15:10:34.082 DEBUG:boto:Path: /dghqvd-56/h 2023-10-11T15:10:34.083 DEBUG:boto:Data: 2023-10-11T15:10:34.083 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.084 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.084 DEBUG:boto:Params: {} 2023-10-11T15:10:34.085 DEBUG:boto:Token: None 2023-10-11T15:10:34.085 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/h 2023-10-11T15:10:34.086 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UYzwf9Lcysk1j4KLm9pXNugtJZw= 2023-10-11T15:10:34.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UYzwf9Lcysk1j4KLm9pXNugtJZw=', 'Content-Length': '0'} 2023-10-11T15:10:34.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f4aa4c23ded481-006526baea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:10:34.090 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:34.090 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:34.090 DEBUG:boto:Method: GET 2023-10-11T15:10:34.091 DEBUG:boto:Path: /dghqvd-56/h 2023-10-11T15:10:34.091 DEBUG:boto:Data: 2023-10-11T15:10:34.092 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:34.093 DEBUG:boto:Port: 8001 2023-10-11T15:10:34.093 DEBUG:boto:Params: {} 2023-10-11T15:10:34.093 DEBUG:boto:Token: None 2023-10-11T15:10:34.094 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/h 2023-10-11T15:10:34.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ezRVKbVsak9KioFrXtRAxn9iY+A= 2023-10-11T15:10:34.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ezRVKbVsak9KioFrXtRAxn9iY+A=', 'Content-Length': '0'} 2023-10-11T15:10:34.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-56:daa316e9-869c-4074-a756-a8adfb4a3573.5751.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219dda41fe5182db-006526baea-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.098 DEBUG:boto:path=/dghqvd-56/h 2023-10-11T15:10:34.098 DEBUG:boto:auth_path=/dghqvd-56/h 2023-10-11T15:10:34.099 DEBUG:boto:Method: GET 2023-10-11T15:10:34.099 DEBUG:boto:Path: /dghqvd-56/h 2023-10-11T15:10:34.100 DEBUG:boto:Data: 2023-10-11T15:10:34.100 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.101 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.102 DEBUG:boto:Params: {} 2023-10-11T15:10:34.102 DEBUG:boto:Token: None 2023-10-11T15:10:34.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/h 2023-10-11T15:10:34.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ezRVKbVsak9KioFrXtRAxn9iY+A= 2023-10-11T15:10:34.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ezRVKbVsak9KioFrXtRAxn9iY+A=', 'Content-Length': '0'} 2023-10-11T15:10:34.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:08:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b18709856aef306-006526baea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.106 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:34.107 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:34.107 DEBUG:boto:path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.108 DEBUG:boto:auth_path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.108 DEBUG:boto:Method: GET 2023-10-11T15:10:34.108 DEBUG:boto:Path: /dghqvd-56/?prefix=h 2023-10-11T15:10:34.109 DEBUG:boto:Data: 2023-10-11T15:10:34.109 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:34.110 DEBUG:boto:Port: 8001 2023-10-11T15:10:34.111 DEBUG:boto:Params: {} 2023-10-11T15:10:34.111 DEBUG:boto:Token: None 2023-10-11T15:10:34.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/ 2023-10-11T15:10:34.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA= 2023-10-11T15:10:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA=', 'Content-Length': '0'} 2023-10-11T15:10:34.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000296dfb40415d0e4a-006526baea-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.115 DEBUG:boto:b'dghqvd-56h1000falseh2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:34.116 DEBUG:boto:path=/dghqvd-56/ 2023-10-11T15:10:34.117 DEBUG:boto:auth_path=/dghqvd-56/ 2023-10-11T15:10:34.117 DEBUG:boto:path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.118 DEBUG:boto:auth_path=/dghqvd-56/?prefix=h 2023-10-11T15:10:34.118 DEBUG:boto:Method: GET 2023-10-11T15:10:34.119 DEBUG:boto:Path: /dghqvd-56/?prefix=h 2023-10-11T15:10:34.119 DEBUG:boto:Data: 2023-10-11T15:10:34.120 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.120 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.121 DEBUG:boto:Params: {} 2023-10-11T15:10:34.121 DEBUG:boto:Token: None 2023-10-11T15:10:34.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-56/ 2023-10-11T15:10:34.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA= 2023-10-11T15:10:34.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QWlbZgbIsRe+q3nIn9bRgR8efGA=', 'Content-Length': '0'} 2023-10-11T15:10:34.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f88fac415ec20d5-006526baea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.125 DEBUG:boto:b'dghqvd-56h1000falseh2023-10-11T15:08:12.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:10:34.126 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-56 zones={a2, a1} 2023-10-11T15:10:34.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-10-11T15:10:34.128 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:34.128 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:34.129 DEBUG:boto:Method: PUT 2023-10-11T15:10:34.129 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:34.129 DEBUG:boto:Data: 2023-10-11T15:10:34.130 DEBUG:boto:Headers: {} 2023-10-11T15:10:34.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:34.131 DEBUG:boto:Port: 8000 2023-10-11T15:10:34.131 DEBUG:boto:Params: {} 2023-10-11T15:10:34.132 DEBUG:boto:Token: None 2023-10-11T15:10:34.132 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:10:34 GMT /dghqvd-57/ 2023-10-11T15:10:34.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SMLOb9BJ633FURDAjwBUbDzdqx0= 2023-10-11T15:10:34.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SMLOb9BJ633FURDAjwBUbDzdqx0=', 'Content-Length': '0'} 2023-10-11T15:10:34.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001101b5c9ca853d7c-006526baea-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:34.152 DEBUG:rgw_multi.tests:created bucket=dghqvd-57 2023-10-11T15:10:34.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:34.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:34.154 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:10:34.210 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:34.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:34.246 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:10:34.259 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:10:34.259 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.260 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:10:34.260 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:10:34.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.260 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:10:34.261 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:10:34.261 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.261 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:10:34.262 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:10:34.262 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.262 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.262 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037034.149247_423.1", 2023-10-11T15:10:34.262 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:10:34.149247Z" 2023-10-11T15:10:34.263 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.263 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:10:34.264 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037034.149247_423.1'} 2023-10-11T15:10:34.264 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:10:34.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:34.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:34.266 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:10:34.318 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:34.319 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:34.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:34.347+0000 7f7877a39780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:10:34.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:34.349+0000 7f7877a39780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:10:34.362 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:10:34.362 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:10:34.363 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:10:34.363 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:10:34.363 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:10:34.363 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:10:34.364 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:34.364 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.364 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:10:34.364 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.365 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:10:34.365 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:34.365 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:34.365 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:10:34.365 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:34.366 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:10:34.366 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:34.366 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:10:34.366 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:34.366 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.367 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.367 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.367 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:10:34.367 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:34.368 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:34.368 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:10:34.368 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:34.368 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:34.368 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:34.369 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:10:34.369 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:34.369 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.369 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.369 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.370 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:10:34.370 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:34.370 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:34.370 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:10:34.371 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:34.371 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:34.371 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:34.371 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:10:34.371 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:34.372 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:34.372 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:10:34.372 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:34.373 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:34.373 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697036891.876035_418.1", 2023-10-11T15:10:34.373 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:34.373 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:34.373 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:08:11.876035Z", 2023-10-11T15:10:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:10:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:10:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:10:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:10:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:10:34Z" 2023-10-11T15:10:34.376 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:10:34.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697036891.876035_418.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:08:11.876035Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:10:34Z" } 2023-10-11T15:10:34.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036891.876035_418.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:08:11.876035Z', 'realm_epoch': 2}}] 2023-10-11T15:10:34.378 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037034.149247_423.1'} 2023-10-11T15:10:34.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036891.876035_418.1'} 2023-10-11T15:10:34.379 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037034.149247_423.1 target=1_1697036891.876035_418.1 2023-10-11T15:10:39.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:39.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:10:39.382 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:10:39.440 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:39.441 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:39.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:39.470+0000 7f8184c21780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:10:39.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:39.472+0000 7f8184c21780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:10:39.487 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:10:39.487 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:10:39.487 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:10:39.488 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:10:39.488 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:10:39.488 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:10:39.488 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:39.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.489 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:10:39.489 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:39.489 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:10:39.490 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:39.490 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:39.490 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:10:39.490 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:39.491 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:10:39.491 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:39.491 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:10:39.491 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:39.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:39.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:39.492 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:10:39.493 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:39.493 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:39.493 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:10:39.493 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:39.494 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:39.494 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:39.494 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:10:39.494 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:39.494 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:39.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.495 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:39.495 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:10:39.495 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:39.496 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:39.496 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:10:39.496 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:39.496 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:39.496 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:39.497 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:10:39.497 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:39.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:39.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.498 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:10:39.498 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:10:39.498 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:10:39.498 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:10:39.499 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037034.149247_423.1", 2023-10-11T15:10:39.499 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:10:39.499 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:10:39.499 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:10:39.500 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:10:34.149247Z", 2023-10-11T15:10:39.500 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:10:39.500 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:39.500 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:10:39.501 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:10:39.501 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.501 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:10:39.501 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:10:39.502 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:10:39.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:10:39.502 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:10:39Z" 2023-10-11T15:10:39.502 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:10:39.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037034.149247_423.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:10:34.149247Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:10:39Z" } 2023-10-11T15:10:39.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037034.149247_423.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:10:34.149247Z', 'realm_epoch': 2}}] 2023-10-11T15:10:39.505 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037034.149247_423.1'} 2023-10-11T15:10:39.506 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037034.149247_423.1'} 2023-10-11T15:10:39.506 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:10:39.507 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:10:39.507 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:10:39.508 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:10:39.506+0000 7fc45fe82700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 122395) UID: 0 2023-10-11T15:10:39.508 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:10:39.506+0000 7fc4640eb7c0 -1 shutting down 2023-10-11T15:10:39.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:10:45.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2023-10-11T15:10:45.609 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:10:45.610 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.610 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.611 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.611 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.612 DEBUG:boto:Data: 2023-10-11T15:10:45.612 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.613 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.614 DEBUG:boto:Params: {} 2023-10-11T15:10:45.614 DEBUG:boto:Token: None 2023-10-11T15:10:45.615 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd8a6d269fe151e2-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.619 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:10:45.620 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:10:45.620 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.621 DEBUG:boto:Path: /dghqvd-57/a 2023-10-11T15:10:45.621 DEBUG:boto:Data: 2023-10-11T15:10:45.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.622 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.623 DEBUG:boto:Params: {} 2023-10-11T15:10:45.623 DEBUG:boto:Token: None 2023-10-11T15:10:45.623 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/a 2023-10-11T15:10:45.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gqsA+CF5OIT6iAj5hjkjCE0sLqQ= 2023-10-11T15:10:45.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gqsA+CF5OIT6iAj5hjkjCE0sLqQ='} 2023-10-11T15:10:45.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc13edf8908d09ae-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.672 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.673 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.674 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.675 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.675 DEBUG:boto:Data: 2023-10-11T15:10:45.676 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.678 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.679 DEBUG:boto:Params: {} 2023-10-11T15:10:45.679 DEBUG:boto:Token: None 2023-10-11T15:10:45.680 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fd10472c7b2d696-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.688 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:10:45.688 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:10:45.689 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.690 DEBUG:boto:Path: /dghqvd-57/b 2023-10-11T15:10:45.690 DEBUG:boto:Data: 2023-10-11T15:10:45.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.692 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.693 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.693 DEBUG:boto:Params: {} 2023-10-11T15:10:45.694 DEBUG:boto:Token: None 2023-10-11T15:10:45.695 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/b 2023-10-11T15:10:45.695 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hLHnKSOA6dQUntWrzaKtyRH4X20= 2023-10-11T15:10:45.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hLHnKSOA6dQUntWrzaKtyRH4X20='} 2023-10-11T15:10:45.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e23d83d260404ae6-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.703 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.704 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.704 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.705 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.705 DEBUG:boto:Data: 2023-10-11T15:10:45.705 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.706 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.707 DEBUG:boto:Params: {} 2023-10-11T15:10:45.707 DEBUG:boto:Token: None 2023-10-11T15:10:45.708 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc99e42a8441268d-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.712 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:10:45.712 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:10:45.713 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.713 DEBUG:boto:Path: /dghqvd-57/c 2023-10-11T15:10:45.714 DEBUG:boto:Data: 2023-10-11T15:10:45.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.715 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.715 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.716 DEBUG:boto:Params: {} 2023-10-11T15:10:45.716 DEBUG:boto:Token: None 2023-10-11T15:10:45.717 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/c 2023-10-11T15:10:45.718 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7hKmenTzMxDfd1lKkYF1+WImKPA= 2023-10-11T15:10:45.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7hKmenTzMxDfd1lKkYF1+WImKPA='} 2023-10-11T15:10:45.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083656c99e1e64330-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.763 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.764 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.764 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.765 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.765 DEBUG:boto:Data: 2023-10-11T15:10:45.766 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.767 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.768 DEBUG:boto:Params: {} 2023-10-11T15:10:45.769 DEBUG:boto:Token: None 2023-10-11T15:10:45.770 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.770 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005362d91dafae3727-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.774 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:10:45.775 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:10:45.776 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.777 DEBUG:boto:Path: /dghqvd-57/d 2023-10-11T15:10:45.778 DEBUG:boto:Data: 2023-10-11T15:10:45.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.781 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.781 DEBUG:boto:Params: {} 2023-10-11T15:10:45.782 DEBUG:boto:Token: None 2023-10-11T15:10:45.783 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/d 2023-10-11T15:10:45.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GXrBxGF0dRsH9ls8QaVaw1EjjPU= 2023-10-11T15:10:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GXrBxGF0dRsH9ls8QaVaw1EjjPU='} 2023-10-11T15:10:45.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004241da59d9c634ed-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:10:45.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:10:45.794 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:10:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:45.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:45.893 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:10:45.893 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-57 2023-10-11T15:10:45.897 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 4 2023-10-11T15:10:45.934 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.935 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.936 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.936 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.936 DEBUG:boto:Data: 2023-10-11T15:10:45.937 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.938 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.939 DEBUG:boto:Params: {} 2023-10-11T15:10:45.939 DEBUG:boto:Token: None 2023-10-11T15:10:45.940 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1618b85b09b6ed1-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.944 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:10:45.944 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:10:45.944 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.945 DEBUG:boto:Path: /dghqvd-57/e 2023-10-11T15:10:45.945 DEBUG:boto:Data: 2023-10-11T15:10:45.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.947 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.947 DEBUG:boto:Params: {} 2023-10-11T15:10:45.948 DEBUG:boto:Token: None 2023-10-11T15:10:45.948 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/e 2023-10-11T15:10:45.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WEY7JQVZs6ftrKCpXqEMOG5PO8w= 2023-10-11T15:10:45.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WEY7JQVZs6ftrKCpXqEMOG5PO8w='} 2023-10-11T15:10:45.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b6ed6ec6606ecb7-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.955 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:45.955 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:45.956 DEBUG:boto:Method: HEAD 2023-10-11T15:10:45.956 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:45.957 DEBUG:boto:Data: 2023-10-11T15:10:45.957 DEBUG:boto:Headers: {} 2023-10-11T15:10:45.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.958 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.959 DEBUG:boto:Params: {} 2023-10-11T15:10:45.959 DEBUG:boto:Token: None 2023-10-11T15:10:45.960 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/ 2023-10-11T15:10:45.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0= 2023-10-11T15:10:45.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOtEhvJSN2m0aK8dAJdnxh1HDB0=', 'Content-Length': '0'} 2023-10-11T15:10:45.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea1aa007cb99daee-006526baf5-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:45.963 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:10:45.964 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:10:45.964 DEBUG:boto:Method: PUT 2023-10-11T15:10:45.964 DEBUG:boto:Path: /dghqvd-57/f 2023-10-11T15:10:45.965 DEBUG:boto:Data: 2023-10-11T15:10:45.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:45.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:45.966 DEBUG:boto:Port: 8000 2023-10-11T15:10:45.967 DEBUG:boto:Params: {} 2023-10-11T15:10:45.967 DEBUG:boto:Token: None 2023-10-11T15:10:45.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:45 GMT /dghqvd-57/f 2023-10-11T15:10:45.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:di0qc5Se/uMCBUtFCtkwqKwUFpo= 2023-10-11T15:10:45.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:45 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:di0qc5Se/uMCBUtFCtkwqKwUFpo='} 2023-10-11T15:10:46.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca64d2dd190f6af9-006526baf5-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:46.015 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:46.016 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:46.016 DEBUG:boto:Method: HEAD 2023-10-11T15:10:46.016 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:46.017 DEBUG:boto:Data: 2023-10-11T15:10:46.018 DEBUG:boto:Headers: {} 2023-10-11T15:10:46.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:46.019 DEBUG:boto:Port: 8000 2023-10-11T15:10:46.020 DEBUG:boto:Params: {} 2023-10-11T15:10:46.021 DEBUG:boto:Token: None 2023-10-11T15:10:46.021 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:46 GMT /dghqvd-57/ 2023-10-11T15:10:46.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ncsC7Q012jA9tKrP7CoyYYbE0LU= 2023-10-11T15:10:46.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ncsC7Q012jA9tKrP7CoyYYbE0LU=', 'Content-Length': '0'} 2023-10-11T15:10:46.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1e1816ed06f0154-006526baf6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:46.025 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:10:46.025 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:10:46.025 DEBUG:boto:Method: PUT 2023-10-11T15:10:46.026 DEBUG:boto:Path: /dghqvd-57/g 2023-10-11T15:10:46.026 DEBUG:boto:Data: 2023-10-11T15:10:46.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:46.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:46.028 DEBUG:boto:Port: 8000 2023-10-11T15:10:46.028 DEBUG:boto:Params: {} 2023-10-11T15:10:46.029 DEBUG:boto:Token: None 2023-10-11T15:10:46.030 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:46 GMT /dghqvd-57/g 2023-10-11T15:10:46.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M3cp2qpjqfRVLU2Djkqc4f7rEqg= 2023-10-11T15:10:46.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M3cp2qpjqfRVLU2Djkqc4f7rEqg='} 2023-10-11T15:10:46.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a441c6e2a3cc1aa5-006526baf6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:46.037 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:10:46.037 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:10:46.038 DEBUG:boto:Method: HEAD 2023-10-11T15:10:46.038 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:10:46.039 DEBUG:boto:Data: 2023-10-11T15:10:46.039 DEBUG:boto:Headers: {} 2023-10-11T15:10:46.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:46.040 DEBUG:boto:Port: 8000 2023-10-11T15:10:46.040 DEBUG:boto:Params: {} 2023-10-11T15:10:46.041 DEBUG:boto:Token: None 2023-10-11T15:10:46.041 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:10:46 GMT /dghqvd-57/ 2023-10-11T15:10:46.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ncsC7Q012jA9tKrP7CoyYYbE0LU= 2023-10-11T15:10:46.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:10:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ncsC7Q012jA9tKrP7CoyYYbE0LU=', 'Content-Length': '0'} 2023-10-11T15:10:46.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d42371a7f56b3ca6-006526baf6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:46.045 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:10:46.045 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:10:46.046 DEBUG:boto:Method: PUT 2023-10-11T15:10:46.046 DEBUG:boto:Path: /dghqvd-57/h 2023-10-11T15:10:46.047 DEBUG:boto:Data: 2023-10-11T15:10:46.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:10:46.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:10:46.049 DEBUG:boto:Port: 8000 2023-10-11T15:10:46.050 DEBUG:boto:Params: {} 2023-10-11T15:10:46.051 DEBUG:boto:Token: None 2023-10-11T15:10:46.052 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:10:46 GMT /dghqvd-57/h 2023-10-11T15:10:46.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ttc6hnOeHF8SRDBfVQfeDG4kJIA= 2023-10-11T15:10:46.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:10:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ttc6hnOeHF8SRDBfVQfeDG4kJIA='} 2023-10-11T15:10:46.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000830b7dd4c2b86ee2-006526baf6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:10:46.058 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:10:46.059 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:10:46.061 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:10:46.062 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:46.083 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:10:46.084 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:10:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:10:46.086 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:10:47.088 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:10:47.114 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:10:47.114 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:10:47.122 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 23777 0 --:--:-- --:--:-- --:--:-- 26750 2023-10-11T15:10:47.123 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:10:47.124 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-57 2023-10-11T15:10:47.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:10:47.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:10:47.126 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:10:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:10:47.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:10:47.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:47.260+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:10:51.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:10:52.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:52.261+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:10:57.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:10:57.261+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:02.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:02.262+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:07.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:07.262+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:12.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:12.263+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:17.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:17.263+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:22.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:22.264+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:27.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:27.265+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:32.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:32.266+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:37.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:37.266+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:42.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:42.267+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:47.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:47.267+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:52.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:52.268+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:11:57.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:11:57.268+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:02.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:02.269+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:07.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:07.269+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:12.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:12.270+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:17.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:17.269+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:22.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:22.270+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:27.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:27.270+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:32.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:32.271+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:37.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:37.272+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:42.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:42.272+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:47.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:47.273+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:52.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:52.274+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:12:57.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:12:57.275+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:13:02.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:02.276+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:13:07.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:07.277+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:13:12.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:12.277+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:13:17.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:17.278+0000 7ff47d44d780 1 waiting to reach incremental sync.. 2023-10-11T15:13:22.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:22.279+0000 7ff47d44d780 1 bucket sync caught up with source: 2023-10-11T15:13:22.281 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000003.58.6, 00000000002.31.6, 00000000002.52.6] 2023-10-11T15:13:22.281 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000003.58.6, 00000000002.31.6, 00000000002.52.6] 2023-10-11T15:13:22.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:22.279+0000 7ff47d44d780 0 bucket checkpoint complete 2023-10-11T15:13:22.298 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-57 2023-10-11T15:13:22.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:22.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:22.300 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:13:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:22.364 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:22.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:22.399+0000 7fe0035a5780 1 waiting for incremental sync to catch up: 2023-10-11T15:13:22.401 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.269.6, , , , 00000000001.244.6, , , 00000000002.182.6, , 00000000001.186.6] 2023-10-11T15:13:22.401 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.269.6, , , 00000000001.102.6, 00000000001.244.6, , , 00000000002.182.6, , 00000000002.194.6] 2023-10-11T15:13:27.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:27.400+0000 7fe0035a5780 1 waiting for incremental sync to catch up: 2023-10-11T15:13:27.403 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.269.6, , , , 00000000001.244.6, , , 00000000002.182.6, , 00000000001.186.6] 2023-10-11T15:13:27.403 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.269.6, , , 00000000001.102.6, 00000000001.244.6, , , 00000000002.182.6, , 00000000002.194.6] 2023-10-11T15:13:32.403 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:32.402+0000 7fe0035a5780 1 bucket sync caught up with source: 2023-10-11T15:13:32.404 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.269.6, , , 00000000001.102.6, 00000000001.244.6, , , 00000000002.182.6, , 00000000002.194.6] 2023-10-11T15:13:32.404 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.269.6, , , 00000000001.102.6, 00000000001.244.6, , , 00000000002.182.6, , 00000000002.194.6] 2023-10-11T15:13:32.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:32.402+0000 7fe0035a5780 0 bucket checkpoint complete 2023-10-11T15:13:32.420 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-57 zones={a2, a1} 2023-10-11T15:13:32.420 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.421 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.421 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.422 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:13:32.422 DEBUG:boto:Data: 2023-10-11T15:13:32.423 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.424 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.424 DEBUG:boto:Params: {} 2023-10-11T15:13:32.425 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:13:32.425 DEBUG:boto:Token: None 2023-10-11T15:13:32.426 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:46o/fe4QKeiyYCLEyAn29K7nC/k= 2023-10-11T15:13:32.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:46o/fe4QKeiyYCLEyAn29K7nC/k=', 'Content-Length': '0'} 2023-10-11T15:13:32.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6c5729adebc4110-006526bb9c-1ab5-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.434 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.434 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.435 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.435 DEBUG:boto:Path: /dghqvd-57/ 2023-10-11T15:13:32.436 DEBUG:boto:Data: 2023-10-11T15:13:32.436 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.437 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.437 DEBUG:boto:Params: {} 2023-10-11T15:13:32.438 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:13:32.438 DEBUG:boto:Token: None 2023-10-11T15:13:32.439 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:46o/fe4QKeiyYCLEyAn29K7nC/k= 2023-10-11T15:13:32.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:46o/fe4QKeiyYCLEyAn29K7nC/k=', 'Content-Length': '0'} 2023-10-11T15:13:32.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000066c3b7f3dbb77f2-006526bb9c-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.444 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:13:32.444 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.445 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.445 DEBUG:boto:path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.445 DEBUG:boto:auth_path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.446 DEBUG:boto:Method: GET 2023-10-11T15:13:32.446 DEBUG:boto:Path: /dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.447 DEBUG:boto:Data: 2023-10-11T15:13:32.447 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.448 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.449 DEBUG:boto:Params: {} 2023-10-11T15:13:32.449 DEBUG:boto:Token: None 2023-10-11T15:13:32.450 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/?versions 2023-10-11T15:13:32.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ= 2023-10-11T15:13:32.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b83b0af177ca82b-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.454 DEBUG:boto:b'dghqvd-57999falseanulltrue2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.455 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:13:32.455 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:13:32.456 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:13:32.456 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:13:32.456 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:13:32.457 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:13:32.457 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:13:32.458 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:13:32.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:13:32.458 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.459 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.459 DEBUG:boto:path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.460 DEBUG:boto:auth_path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.460 DEBUG:boto:Method: GET 2023-10-11T15:13:32.461 DEBUG:boto:Path: /dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.461 DEBUG:boto:Data: 2023-10-11T15:13:32.461 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.462 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.463 DEBUG:boto:Params: {} 2023-10-11T15:13:32.463 DEBUG:boto:Token: None 2023-10-11T15:13:32.464 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/?versions 2023-10-11T15:13:32.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ= 2023-10-11T15:13:32.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d792cabcee30bf3e-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.467 DEBUG:boto:b'dghqvd-57999falseanulltrue2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.468 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-10-11T15:13:32.468 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-10-11T15:13:32.469 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-10-11T15:13:32.469 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-10-11T15:13:32.470 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-10-11T15:13:32.470 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-10-11T15:13:32.470 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-10-11T15:13:32.471 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-10-11T15:13:32.471 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.472 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.472 DEBUG:boto:path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.473 DEBUG:boto:auth_path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.473 DEBUG:boto:Method: GET 2023-10-11T15:13:32.473 DEBUG:boto:Path: /dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.474 DEBUG:boto:Data: 2023-10-11T15:13:32.474 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.475 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.476 DEBUG:boto:Params: {} 2023-10-11T15:13:32.476 DEBUG:boto:Token: None 2023-10-11T15:13:32.477 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/?versions 2023-10-11T15:13:32.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ= 2023-10-11T15:13:32.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088dd9cc2f749714c-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.480 DEBUG:boto:b'dghqvd-57999falseanulltrue2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.481 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.482 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.482 DEBUG:boto:path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.483 DEBUG:boto:auth_path=/dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.483 DEBUG:boto:Method: GET 2023-10-11T15:13:32.484 DEBUG:boto:Path: /dghqvd-57/?versions&max-keys=999 2023-10-11T15:13:32.484 DEBUG:boto:Data: 2023-10-11T15:13:32.484 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.485 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.486 DEBUG:boto:Params: {} 2023-10-11T15:13:32.486 DEBUG:boto:Token: None 2023-10-11T15:13:32.487 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/?versions 2023-10-11T15:13:32.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ= 2023-10-11T15:13:32.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l05Sb25WkOwiX16oVrOoi+j62QQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fff3a309a8eaa1b-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.490 DEBUG:boto:b'dghqvd-57999falseanulltrue2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:13:32.492 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.492 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.493 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.493 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.494 DEBUG:boto:Method: GET 2023-10-11T15:13:32.494 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.494 DEBUG:boto:Data: 2023-10-11T15:13:32.495 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.495 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.496 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.496 DEBUG:boto:Params: {} 2023-10-11T15:13:32.497 DEBUG:boto:Token: None 2023-10-11T15:13:32.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo= 2023-10-11T15:13:32.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo=', 'Content-Length': '0'} 2023-10-11T15:13:32.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001084c828b8d27ee-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.501 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.502 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.502 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.502 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.503 DEBUG:boto:Method: GET 2023-10-11T15:13:32.503 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.504 DEBUG:boto:Data: 2023-10-11T15:13:32.504 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.505 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.506 DEBUG:boto:Params: {} 2023-10-11T15:13:32.506 DEBUG:boto:Token: None 2023-10-11T15:13:32.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo= 2023-10-11T15:13:32.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo=', 'Content-Length': '0'} 2023-10-11T15:13:32.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8af328169809ebe-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.510 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.511 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.511 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.512 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.512 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.512 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.513 DEBUG:boto:Data: 2023-10-11T15:13:32.513 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.514 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.515 DEBUG:boto:Params: {} 2023-10-11T15:13:32.515 DEBUG:boto:Token: None 2023-10-11T15:13:32.515 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhQA75iB5yod6PPVBbYyF0FsafY= 2023-10-11T15:13:32.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhQA75iB5yod6PPVBbYyF0FsafY=', 'Content-Length': '0'} 2023-10-11T15:13:32.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa34962abc0f3c36-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.519 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.519 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.520 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.520 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.520 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.521 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.521 DEBUG:boto:Data: 2023-10-11T15:13:32.522 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.522 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.523 DEBUG:boto:Params: {} 2023-10-11T15:13:32.523 DEBUG:boto:Token: None 2023-10-11T15:13:32.524 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhQA75iB5yod6PPVBbYyF0FsafY= 2023-10-11T15:13:32.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhQA75iB5yod6PPVBbYyF0FsafY=', 'Content-Length': '0'} 2023-10-11T15:13:32.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000132f70c9d26e4db8-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:13:32.528 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.528 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.528 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.529 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.529 DEBUG:boto:Method: GET 2023-10-11T15:13:32.530 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.530 DEBUG:boto:Data: 2023-10-11T15:13:32.531 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.532 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.532 DEBUG:boto:Params: {} 2023-10-11T15:13:32.532 DEBUG:boto:Token: None 2023-10-11T15:13:32.533 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo= 2023-10-11T15:13:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo=', 'Content-Length': '0'} 2023-10-11T15:13:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b243bcc2313ae93-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.536 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.537 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.537 DEBUG:boto:path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.538 DEBUG:boto:auth_path=/dghqvd-57/a?versionId=null 2023-10-11T15:13:32.538 DEBUG:boto:Method: GET 2023-10-11T15:13:32.539 DEBUG:boto:Path: /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.539 DEBUG:boto:Data: 2023-10-11T15:13:32.539 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.540 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.541 DEBUG:boto:Params: {} 2023-10-11T15:13:32.541 DEBUG:boto:Token: None 2023-10-11T15:13:32.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a?versionId=null 2023-10-11T15:13:32.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo= 2023-10-11T15:13:32.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmmyJV6/Hr17knn5Ft20jTS74Qo=', 'Content-Length': '0'} 2023-10-11T15:13:32.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0327ec545a60fb-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.590 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.591 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.591 DEBUG:boto:path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.592 DEBUG:boto:auth_path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.592 DEBUG:boto:Method: GET 2023-10-11T15:13:32.593 DEBUG:boto:Path: /dghqvd-57/?prefix=a 2023-10-11T15:13:32.593 DEBUG:boto:Data: 2023-10-11T15:13:32.593 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.594 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.595 DEBUG:boto:Params: {} 2023-10-11T15:13:32.595 DEBUG:boto:Token: None 2023-10-11T15:13:32.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bd30034641a410c-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.600 DEBUG:boto:b'dghqvd-57a1000falsea2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.601 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.601 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.602 DEBUG:boto:path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.602 DEBUG:boto:auth_path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.605 DEBUG:boto:Method: GET 2023-10-11T15:13:32.605 DEBUG:boto:Path: /dghqvd-57/?prefix=a 2023-10-11T15:13:32.606 DEBUG:boto:Data: 2023-10-11T15:13:32.606 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.607 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.608 DEBUG:boto:Params: {} 2023-10-11T15:13:32.608 DEBUG:boto:Token: None 2023-10-11T15:13:32.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b620068e889f98b1-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.613 DEBUG:boto:b'dghqvd-57a1000falsea2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.614 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.614 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.614 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.615 DEBUG:boto:Path: /dghqvd-57/a 2023-10-11T15:13:32.615 DEBUG:boto:Data: 2023-10-11T15:13:32.616 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.616 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.617 DEBUG:boto:Params: {} 2023-10-11T15:13:32.617 DEBUG:boto:Token: None 2023-10-11T15:13:32.618 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a 2023-10-11T15:13:32.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MjccjGsYK4PPvZz+dlAY7tBJvjw= 2023-10-11T15:13:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MjccjGsYK4PPvZz+dlAY7tBJvjw=', 'Content-Length': '0'} 2023-10-11T15:13:32.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0064b9e1a877e02-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.621 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.622 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.622 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.623 DEBUG:boto:Path: /dghqvd-57/a 2023-10-11T15:13:32.623 DEBUG:boto:Data: 2023-10-11T15:13:32.623 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.624 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.624 DEBUG:boto:Params: {} 2023-10-11T15:13:32.625 DEBUG:boto:Token: None 2023-10-11T15:13:32.625 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a 2023-10-11T15:13:32.626 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MjccjGsYK4PPvZz+dlAY7tBJvjw= 2023-10-11T15:13:32.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MjccjGsYK4PPvZz+dlAY7tBJvjw=', 'Content-Length': '0'} 2023-10-11T15:13:32.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abf6327ab11853e-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-10-11T15:13:32.630 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.630 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.631 DEBUG:boto:Method: GET 2023-10-11T15:13:32.631 DEBUG:boto:Path: /dghqvd-57/a 2023-10-11T15:13:32.631 DEBUG:boto:Data: 2023-10-11T15:13:32.632 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.633 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.633 DEBUG:boto:Params: {} 2023-10-11T15:13:32.634 DEBUG:boto:Token: None 2023-10-11T15:13:32.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a 2023-10-11T15:13:32.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8voxhXehh8r5W8oQi52RwIiu3Y4= 2023-10-11T15:13:32.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8voxhXehh8r5W8oQi52RwIiu3Y4=', 'Content-Length': '0'} 2023-10-11T15:13:32.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbba8360b5276aca-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.678 DEBUG:boto:path=/dghqvd-57/a 2023-10-11T15:13:32.679 DEBUG:boto:auth_path=/dghqvd-57/a 2023-10-11T15:13:32.680 DEBUG:boto:Method: GET 2023-10-11T15:13:32.680 DEBUG:boto:Path: /dghqvd-57/a 2023-10-11T15:13:32.680 DEBUG:boto:Data: 2023-10-11T15:13:32.681 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.682 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.682 DEBUG:boto:Params: {} 2023-10-11T15:13:32.683 DEBUG:boto:Token: None 2023-10-11T15:13:32.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/a 2023-10-11T15:13:32.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8voxhXehh8r5W8oQi52RwIiu3Y4= 2023-10-11T15:13:32.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8voxhXehh8r5W8oQi52RwIiu3Y4=', 'Content-Length': '0'} 2023-10-11T15:13:32.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca106010ca1a1ac-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.687 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.687 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.688 DEBUG:boto:path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.688 DEBUG:boto:auth_path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.689 DEBUG:boto:Method: GET 2023-10-11T15:13:32.689 DEBUG:boto:Path: /dghqvd-57/?prefix=a 2023-10-11T15:13:32.690 DEBUG:boto:Data: 2023-10-11T15:13:32.690 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.691 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.691 DEBUG:boto:Params: {} 2023-10-11T15:13:32.692 DEBUG:boto:Token: None 2023-10-11T15:13:32.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5148666d8b95c4c-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.696 DEBUG:boto:b'dghqvd-57a1000falsea2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.697 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.698 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.698 DEBUG:boto:path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.699 DEBUG:boto:auth_path=/dghqvd-57/?prefix=a 2023-10-11T15:13:32.699 DEBUG:boto:Method: GET 2023-10-11T15:13:32.700 DEBUG:boto:Path: /dghqvd-57/?prefix=a 2023-10-11T15:13:32.700 DEBUG:boto:Data: 2023-10-11T15:13:32.700 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.701 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.702 DEBUG:boto:Params: {} 2023-10-11T15:13:32.702 DEBUG:boto:Token: None 2023-10-11T15:13:32.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002333d718340e1c57-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.707 DEBUG:boto:b'dghqvd-57a1000falsea2023-10-11T15:10:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:13:32.708 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.708 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.709 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.709 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.710 DEBUG:boto:Method: GET 2023-10-11T15:13:32.710 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.710 DEBUG:boto:Data: 2023-10-11T15:13:32.711 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.712 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.712 DEBUG:boto:Params: {} 2023-10-11T15:13:32.713 DEBUG:boto:Token: None 2023-10-11T15:13:32.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg= 2023-10-11T15:13:32.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg=', 'Content-Length': '0'} 2023-10-11T15:13:32.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ff7824245586ad-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.717 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.718 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.718 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.719 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.719 DEBUG:boto:Method: GET 2023-10-11T15:13:32.719 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.720 DEBUG:boto:Data: 2023-10-11T15:13:32.720 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.721 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.722 DEBUG:boto:Params: {} 2023-10-11T15:13:32.722 DEBUG:boto:Token: None 2023-10-11T15:13:32.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg= 2023-10-11T15:13:32.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg=', 'Content-Length': '0'} 2023-10-11T15:13:32.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c95d336ee6460a1f-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.766 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.767 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.768 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.768 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.768 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.769 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.769 DEBUG:boto:Data: 2023-10-11T15:13:32.770 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.770 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.771 DEBUG:boto:Params: {} 2023-10-11T15:13:32.771 DEBUG:boto:Token: None 2023-10-11T15:13:32.772 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vyhOaIF2dQu2Tdblyh8IarAnwX8= 2023-10-11T15:13:32.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vyhOaIF2dQu2Tdblyh8IarAnwX8=', 'Content-Length': '0'} 2023-10-11T15:13:32.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008782262bc529a19c-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.775 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.776 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.776 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.776 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.777 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.777 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.778 DEBUG:boto:Data: 2023-10-11T15:13:32.778 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.779 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.779 DEBUG:boto:Params: {} 2023-10-11T15:13:32.780 DEBUG:boto:Token: None 2023-10-11T15:13:32.780 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vyhOaIF2dQu2Tdblyh8IarAnwX8= 2023-10-11T15:13:32.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vyhOaIF2dQu2Tdblyh8IarAnwX8=', 'Content-Length': '0'} 2023-10-11T15:13:32.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab7822ac0a9157d-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:13:32.784 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.785 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.785 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.785 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.786 DEBUG:boto:Method: GET 2023-10-11T15:13:32.786 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.787 DEBUG:boto:Data: 2023-10-11T15:13:32.787 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.788 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.788 DEBUG:boto:Params: {} 2023-10-11T15:13:32.789 DEBUG:boto:Token: None 2023-10-11T15:13:32.789 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg= 2023-10-11T15:13:32.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg=', 'Content-Length': '0'} 2023-10-11T15:13:32.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002335ff7ccf5936e6-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.838 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.839 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.840 DEBUG:boto:path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.840 DEBUG:boto:auth_path=/dghqvd-57/b?versionId=null 2023-10-11T15:13:32.840 DEBUG:boto:Method: GET 2023-10-11T15:13:32.841 DEBUG:boto:Path: /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.841 DEBUG:boto:Data: 2023-10-11T15:13:32.842 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.843 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.843 DEBUG:boto:Params: {} 2023-10-11T15:13:32.844 DEBUG:boto:Token: None 2023-10-11T15:13:32.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b?versionId=null 2023-10-11T15:13:32.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg= 2023-10-11T15:13:32.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z2GbnNdqxa5D9nJapGYjFBPEZcg=', 'Content-Length': '0'} 2023-10-11T15:13:32.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee2d41e8020616d-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.848 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.849 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.850 DEBUG:boto:path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.850 DEBUG:boto:auth_path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.851 DEBUG:boto:Method: GET 2023-10-11T15:13:32.851 DEBUG:boto:Path: /dghqvd-57/?prefix=b 2023-10-11T15:13:32.852 DEBUG:boto:Data: 2023-10-11T15:13:32.852 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.853 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.854 DEBUG:boto:Params: {} 2023-10-11T15:13:32.854 DEBUG:boto:Token: None 2023-10-11T15:13:32.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff90e3850452ed5f-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.859 DEBUG:boto:b'dghqvd-57b1000falseb2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.860 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.860 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.861 DEBUG:boto:path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.861 DEBUG:boto:auth_path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.862 DEBUG:boto:Method: GET 2023-10-11T15:13:32.862 DEBUG:boto:Path: /dghqvd-57/?prefix=b 2023-10-11T15:13:32.862 DEBUG:boto:Data: 2023-10-11T15:13:32.863 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.864 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.864 DEBUG:boto:Params: {} 2023-10-11T15:13:32.865 DEBUG:boto:Token: None 2023-10-11T15:13:32.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006379a3bdfff53d33-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.869 DEBUG:boto:b'dghqvd-57b1000falseb2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.870 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.870 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.871 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.871 DEBUG:boto:Path: /dghqvd-57/b 2023-10-11T15:13:32.871 DEBUG:boto:Data: 2023-10-11T15:13:32.872 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.873 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.873 DEBUG:boto:Params: {} 2023-10-11T15:13:32.874 DEBUG:boto:Token: None 2023-10-11T15:13:32.874 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b 2023-10-11T15:13:32.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKFoGIQseXfgw90LzDrULm5tZGQ= 2023-10-11T15:13:32.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKFoGIQseXfgw90LzDrULm5tZGQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f201fc5cfc3fb6-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.878 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.878 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.879 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.879 DEBUG:boto:Path: /dghqvd-57/b 2023-10-11T15:13:32.879 DEBUG:boto:Data: 2023-10-11T15:13:32.880 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.881 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.881 DEBUG:boto:Params: {} 2023-10-11T15:13:32.881 DEBUG:boto:Token: None 2023-10-11T15:13:32.882 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b 2023-10-11T15:13:32.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKFoGIQseXfgw90LzDrULm5tZGQ= 2023-10-11T15:13:32.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKFoGIQseXfgw90LzDrULm5tZGQ=', 'Content-Length': '0'} 2023-10-11T15:13:32.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8412b8e9c234ce0-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-10-11T15:13:32.886 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.886 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.887 DEBUG:boto:Method: GET 2023-10-11T15:13:32.888 DEBUG:boto:Path: /dghqvd-57/b 2023-10-11T15:13:32.888 DEBUG:boto:Data: 2023-10-11T15:13:32.889 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.889 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.890 DEBUG:boto:Params: {} 2023-10-11T15:13:32.890 DEBUG:boto:Token: None 2023-10-11T15:13:32.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b 2023-10-11T15:13:32.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmU9Hf71+Al0yfMdEopCNI5wwKo= 2023-10-11T15:13:32.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmU9Hf71+Al0yfMdEopCNI5wwKo=', 'Content-Length': '0'} 2023-10-11T15:13:32.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac9d570a691f9cc-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.895 DEBUG:boto:path=/dghqvd-57/b 2023-10-11T15:13:32.895 DEBUG:boto:auth_path=/dghqvd-57/b 2023-10-11T15:13:32.896 DEBUG:boto:Method: GET 2023-10-11T15:13:32.896 DEBUG:boto:Path: /dghqvd-57/b 2023-10-11T15:13:32.896 DEBUG:boto:Data: 2023-10-11T15:13:32.897 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.898 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.898 DEBUG:boto:Params: {} 2023-10-11T15:13:32.899 DEBUG:boto:Token: None 2023-10-11T15:13:32.899 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/b 2023-10-11T15:13:32.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmU9Hf71+Al0yfMdEopCNI5wwKo= 2023-10-11T15:13:32.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmU9Hf71+Al0yfMdEopCNI5wwKo=', 'Content-Length': '0'} 2023-10-11T15:13:32.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042990c6411960bd0-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.903 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.903 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.904 DEBUG:boto:path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.904 DEBUG:boto:auth_path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.905 DEBUG:boto:Method: GET 2023-10-11T15:13:32.905 DEBUG:boto:Path: /dghqvd-57/?prefix=b 2023-10-11T15:13:32.906 DEBUG:boto:Data: 2023-10-11T15:13:32.906 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.907 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.908 DEBUG:boto:Params: {} 2023-10-11T15:13:32.908 DEBUG:boto:Token: None 2023-10-11T15:13:32.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c98ebaaa4cd0fd3-006526bb9c-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.958 DEBUG:boto:b'dghqvd-57b1000falseb2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.959 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:32.960 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:32.961 DEBUG:boto:path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.961 DEBUG:boto:auth_path=/dghqvd-57/?prefix=b 2023-10-11T15:13:32.962 DEBUG:boto:Method: GET 2023-10-11T15:13:32.963 DEBUG:boto:Path: /dghqvd-57/?prefix=b 2023-10-11T15:13:32.964 DEBUG:boto:Data: 2023-10-11T15:13:32.964 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.966 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.967 DEBUG:boto:Params: {} 2023-10-11T15:13:32.968 DEBUG:boto:Token: None 2023-10-11T15:13:32.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/ 2023-10-11T15:13:32.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA= 2023-10-11T15:13:32.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ov39Fg47QhC+cXzQyvH6q+71OMA=', 'Content-Length': '0'} 2023-10-11T15:13:32.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041abcf036699a9d2-006526bb9c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.972 DEBUG:boto:b'dghqvd-57b1000falseb2023-10-11T15:10:45.697Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:32.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:13:32.974 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:32.974 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:32.975 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.975 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.975 DEBUG:boto:Method: GET 2023-10-11T15:13:32.976 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:32.976 DEBUG:boto:Data: 2023-10-11T15:13:32.977 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:32.978 DEBUG:boto:Port: 8001 2023-10-11T15:13:32.978 DEBUG:boto:Params: {} 2023-10-11T15:13:32.979 DEBUG:boto:Token: None 2023-10-11T15:13:32.979 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:32.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OyUct/f9vjqx9OOe6xbtnTgxeEg= 2023-10-11T15:13:32.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OyUct/f9vjqx9OOe6xbtnTgxeEg=', 'Content-Length': '0'} 2023-10-11T15:13:32.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae15cbb2798bf8da-006526bb9c-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.983 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:32.984 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:32.984 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.984 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.985 DEBUG:boto:Method: GET 2023-10-11T15:13:32.985 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:32.986 DEBUG:boto:Data: 2023-10-11T15:13:32.986 DEBUG:boto:Headers: {} 2023-10-11T15:13:32.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:32.987 DEBUG:boto:Port: 8000 2023-10-11T15:13:32.988 DEBUG:boto:Params: {} 2023-10-11T15:13:32.989 DEBUG:boto:Token: None 2023-10-11T15:13:32.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:32 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:32.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OyUct/f9vjqx9OOe6xbtnTgxeEg= 2023-10-11T15:13:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OyUct/f9vjqx9OOe6xbtnTgxeEg=', 'Content-Length': '0'} 2023-10-11T15:13:32.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059aded7becce7c6d-006526bb9c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:32.995 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:32.996 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:32.997 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.997 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:32.998 DEBUG:boto:Method: HEAD 2023-10-11T15:13:32.999 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:32.999 DEBUG:boto:Data: 2023-10-11T15:13:33.000 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.001 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.002 DEBUG:boto:Params: {} 2023-10-11T15:13:33.003 DEBUG:boto:Token: None 2023-10-11T15:13:33.004 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGBMIvCOMU0soK8n9RK46dvGY68= 2023-10-11T15:13:33.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGBMIvCOMU0soK8n9RK46dvGY68=', 'Content-Length': '0'} 2023-10-11T15:13:33.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07e4a2324741957-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.009 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.010 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.010 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.013 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.014 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.014 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.015 DEBUG:boto:Data: 2023-10-11T15:13:33.015 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.016 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.017 DEBUG:boto:Params: {} 2023-10-11T15:13:33.017 DEBUG:boto:Token: None 2023-10-11T15:13:33.018 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGBMIvCOMU0soK8n9RK46dvGY68= 2023-10-11T15:13:33.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGBMIvCOMU0soK8n9RK46dvGY68=', 'Content-Length': '0'} 2023-10-11T15:13:33.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d23acb7adb481ea-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:13:33.022 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.023 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.023 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.024 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.025 DEBUG:boto:Method: GET 2023-10-11T15:13:33.026 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.026 DEBUG:boto:Data: 2023-10-11T15:13:33.027 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.029 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.029 DEBUG:boto:Params: {} 2023-10-11T15:13:33.030 DEBUG:boto:Token: None 2023-10-11T15:13:33.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ysbb4Li1nu4fqv2kEHFxNsVO+Vg= 2023-10-11T15:13:33.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ysbb4Li1nu4fqv2kEHFxNsVO+Vg=', 'Content-Length': '0'} 2023-10-11T15:13:33.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc9f3eec837a898-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.035 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.035 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.036 DEBUG:boto:path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.036 DEBUG:boto:auth_path=/dghqvd-57/c?versionId=null 2023-10-11T15:13:33.037 DEBUG:boto:Method: GET 2023-10-11T15:13:33.037 DEBUG:boto:Path: /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.037 DEBUG:boto:Data: 2023-10-11T15:13:33.038 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.039 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.039 DEBUG:boto:Params: {} 2023-10-11T15:13:33.040 DEBUG:boto:Token: None 2023-10-11T15:13:33.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c?versionId=null 2023-10-11T15:13:33.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ysbb4Li1nu4fqv2kEHFxNsVO+Vg= 2023-10-11T15:13:33.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ysbb4Li1nu4fqv2kEHFxNsVO+Vg=', 'Content-Length': '0'} 2023-10-11T15:13:33.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225380f9a7239b20-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.044 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.044 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.044 DEBUG:boto:path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.045 DEBUG:boto:auth_path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.045 DEBUG:boto:Method: GET 2023-10-11T15:13:33.046 DEBUG:boto:Path: /dghqvd-57/?prefix=c 2023-10-11T15:13:33.046 DEBUG:boto:Data: 2023-10-11T15:13:33.047 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.047 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.048 DEBUG:boto:Params: {} 2023-10-11T15:13:33.048 DEBUG:boto:Token: None 2023-10-11T15:13:33.049 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008732a8f8811e578d-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.094 DEBUG:boto:b'dghqvd-57c1000falsec2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.095 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.096 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.096 DEBUG:boto:path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.097 DEBUG:boto:auth_path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.097 DEBUG:boto:Method: GET 2023-10-11T15:13:33.098 DEBUG:boto:Path: /dghqvd-57/?prefix=c 2023-10-11T15:13:33.098 DEBUG:boto:Data: 2023-10-11T15:13:33.098 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.099 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.100 DEBUG:boto:Params: {} 2023-10-11T15:13:33.100 DEBUG:boto:Token: None 2023-10-11T15:13:33.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff81b6853bf1a35-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.104 DEBUG:boto:b'dghqvd-57c1000falsec2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.105 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.106 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.106 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.106 DEBUG:boto:Path: /dghqvd-57/c 2023-10-11T15:13:33.107 DEBUG:boto:Data: 2023-10-11T15:13:33.107 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.108 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.109 DEBUG:boto:Params: {} 2023-10-11T15:13:33.109 DEBUG:boto:Token: None 2023-10-11T15:13:33.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c 2023-10-11T15:13:33.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q6+x+1l+zwnM4UWmU+P3AQbh9a4= 2023-10-11T15:13:33.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q6+x+1l+zwnM4UWmU+P3AQbh9a4=', 'Content-Length': '0'} 2023-10-11T15:13:33.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b69a94cfe3a45c-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.113 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.114 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.114 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.115 DEBUG:boto:Path: /dghqvd-57/c 2023-10-11T15:13:33.115 DEBUG:boto:Data: 2023-10-11T15:13:33.116 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.116 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.117 DEBUG:boto:Params: {} 2023-10-11T15:13:33.117 DEBUG:boto:Token: None 2023-10-11T15:13:33.118 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c 2023-10-11T15:13:33.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q6+x+1l+zwnM4UWmU+P3AQbh9a4= 2023-10-11T15:13:33.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q6+x+1l+zwnM4UWmU+P3AQbh9a4=', 'Content-Length': '0'} 2023-10-11T15:13:33.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf6b441b38ffa8a-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-10-11T15:13:33.122 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.122 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.123 DEBUG:boto:Method: GET 2023-10-11T15:13:33.123 DEBUG:boto:Path: /dghqvd-57/c 2023-10-11T15:13:33.124 DEBUG:boto:Data: 2023-10-11T15:13:33.124 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.125 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.125 DEBUG:boto:Params: {} 2023-10-11T15:13:33.126 DEBUG:boto:Token: None 2023-10-11T15:13:33.126 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c 2023-10-11T15:13:33.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vMSL9Iuoz7RxkOzMBuDfH8OFClg= 2023-10-11T15:13:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vMSL9Iuoz7RxkOzMBuDfH8OFClg=', 'Content-Length': '0'} 2023-10-11T15:13:33.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec126f92ac6ffc21-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.130 DEBUG:boto:path=/dghqvd-57/c 2023-10-11T15:13:33.131 DEBUG:boto:auth_path=/dghqvd-57/c 2023-10-11T15:13:33.131 DEBUG:boto:Method: GET 2023-10-11T15:13:33.132 DEBUG:boto:Path: /dghqvd-57/c 2023-10-11T15:13:33.132 DEBUG:boto:Data: 2023-10-11T15:13:33.132 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.133 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.134 DEBUG:boto:Params: {} 2023-10-11T15:13:33.134 DEBUG:boto:Token: None 2023-10-11T15:13:33.135 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/c 2023-10-11T15:13:33.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vMSL9Iuoz7RxkOzMBuDfH8OFClg= 2023-10-11T15:13:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vMSL9Iuoz7RxkOzMBuDfH8OFClg=', 'Content-Length': '0'} 2023-10-11T15:13:33.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cae9397a01319d-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.138 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.139 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.139 DEBUG:boto:path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.140 DEBUG:boto:auth_path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.140 DEBUG:boto:Method: GET 2023-10-11T15:13:33.141 DEBUG:boto:Path: /dghqvd-57/?prefix=c 2023-10-11T15:13:33.141 DEBUG:boto:Data: 2023-10-11T15:13:33.141 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.142 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.143 DEBUG:boto:Params: {} 2023-10-11T15:13:33.143 DEBUG:boto:Token: None 2023-10-11T15:13:33.144 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b0c0c61bbd380c9-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.148 DEBUG:boto:b'dghqvd-57c1000falsec2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.148 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.149 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.149 DEBUG:boto:path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.150 DEBUG:boto:auth_path=/dghqvd-57/?prefix=c 2023-10-11T15:13:33.150 DEBUG:boto:Method: GET 2023-10-11T15:13:33.151 DEBUG:boto:Path: /dghqvd-57/?prefix=c 2023-10-11T15:13:33.151 DEBUG:boto:Data: 2023-10-11T15:13:33.152 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.153 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.153 DEBUG:boto:Params: {} 2023-10-11T15:13:33.154 DEBUG:boto:Token: None 2023-10-11T15:13:33.154 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000569153c6e3e26bea-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.158 DEBUG:boto:b'dghqvd-57c1000falsec2023-10-11T15:10:45.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:13:33.159 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.160 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.160 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.161 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.161 DEBUG:boto:Method: GET 2023-10-11T15:13:33.162 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.162 DEBUG:boto:Data: 2023-10-11T15:13:33.163 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.164 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.164 DEBUG:boto:Params: {} 2023-10-11T15:13:33.165 DEBUG:boto:Token: None 2023-10-11T15:13:33.165 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.166 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY= 2023-10-11T15:13:33.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY=', 'Content-Length': '0'} 2023-10-11T15:13:33.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009147d74cc1f303d2-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.169 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.169 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.170 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.170 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.170 DEBUG:boto:Method: GET 2023-10-11T15:13:33.171 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.171 DEBUG:boto:Data: 2023-10-11T15:13:33.172 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.172 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.173 DEBUG:boto:Params: {} 2023-10-11T15:13:33.173 DEBUG:boto:Token: None 2023-10-11T15:13:33.174 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY= 2023-10-11T15:13:33.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY=', 'Content-Length': '0'} 2023-10-11T15:13:33.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b541a5a3fcbf9484-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.178 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.178 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.178 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.179 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.179 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.180 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.180 DEBUG:boto:Data: 2023-10-11T15:13:33.180 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.181 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.182 DEBUG:boto:Params: {} 2023-10-11T15:13:33.182 DEBUG:boto:Token: None 2023-10-11T15:13:33.183 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.183 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OqGJij+NNvqycfqzHcmH7fC7Ls0= 2023-10-11T15:13:33.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OqGJij+NNvqycfqzHcmH7fC7Ls0=', 'Content-Length': '0'} 2023-10-11T15:13:33.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b874aac108530e-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.186 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.186 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.187 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.187 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.188 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.188 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.188 DEBUG:boto:Data: 2023-10-11T15:13:33.189 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.190 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.190 DEBUG:boto:Params: {} 2023-10-11T15:13:33.191 DEBUG:boto:Token: None 2023-10-11T15:13:33.191 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OqGJij+NNvqycfqzHcmH7fC7Ls0= 2023-10-11T15:13:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OqGJij+NNvqycfqzHcmH7fC7Ls0=', 'Content-Length': '0'} 2023-10-11T15:13:33.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974c647c53832839-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:13:33.195 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.195 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.196 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.196 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.196 DEBUG:boto:Method: GET 2023-10-11T15:13:33.197 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.197 DEBUG:boto:Data: 2023-10-11T15:13:33.198 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.199 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.199 DEBUG:boto:Params: {} 2023-10-11T15:13:33.200 DEBUG:boto:Token: None 2023-10-11T15:13:33.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY= 2023-10-11T15:13:33.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY=', 'Content-Length': '0'} 2023-10-11T15:13:33.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000227d64f222e7aaa0-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.204 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.204 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.205 DEBUG:boto:path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.205 DEBUG:boto:auth_path=/dghqvd-57/d?versionId=null 2023-10-11T15:13:33.205 DEBUG:boto:Method: GET 2023-10-11T15:13:33.206 DEBUG:boto:Path: /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.206 DEBUG:boto:Data: 2023-10-11T15:13:33.207 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.208 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.208 DEBUG:boto:Params: {} 2023-10-11T15:13:33.209 DEBUG:boto:Token: None 2023-10-11T15:13:33.209 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d?versionId=null 2023-10-11T15:13:33.210 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY= 2023-10-11T15:13:33.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fDaMAt/NpqK71x0Harv13P4ubtY=', 'Content-Length': '0'} 2023-10-11T15:13:33.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a1c5127c9cbeeb-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.213 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.213 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.214 DEBUG:boto:path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.218 DEBUG:boto:auth_path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.219 DEBUG:boto:Method: GET 2023-10-11T15:13:33.219 DEBUG:boto:Path: /dghqvd-57/?prefix=d 2023-10-11T15:13:33.220 DEBUG:boto:Data: 2023-10-11T15:13:33.220 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.221 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.221 DEBUG:boto:Params: {} 2023-10-11T15:13:33.222 DEBUG:boto:Token: None 2023-10-11T15:13:33.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688a343670acaeba-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.270 DEBUG:boto:b'dghqvd-57d1000falsed2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.271 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.272 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.272 DEBUG:boto:path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.273 DEBUG:boto:auth_path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.273 DEBUG:boto:Method: GET 2023-10-11T15:13:33.274 DEBUG:boto:Path: /dghqvd-57/?prefix=d 2023-10-11T15:13:33.274 DEBUG:boto:Data: 2023-10-11T15:13:33.275 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.277 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.278 DEBUG:boto:Params: {} 2023-10-11T15:13:33.279 DEBUG:boto:Token: None 2023-10-11T15:13:33.279 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000684caf91e404296e-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.284 DEBUG:boto:b'dghqvd-57d1000falsed2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.285 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.286 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.286 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.287 DEBUG:boto:Path: /dghqvd-57/d 2023-10-11T15:13:33.287 DEBUG:boto:Data: 2023-10-11T15:13:33.287 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.288 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.288 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.289 DEBUG:boto:Params: {} 2023-10-11T15:13:33.289 DEBUG:boto:Token: None 2023-10-11T15:13:33.290 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d 2023-10-11T15:13:33.290 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VDgh/q7w8hDlzILkGCXnc0fqZBE= 2023-10-11T15:13:33.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VDgh/q7w8hDlzILkGCXnc0fqZBE=', 'Content-Length': '0'} 2023-10-11T15:13:33.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030653a254a04b447-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.293 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.294 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.294 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.295 DEBUG:boto:Path: /dghqvd-57/d 2023-10-11T15:13:33.295 DEBUG:boto:Data: 2023-10-11T15:13:33.296 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.297 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.297 DEBUG:boto:Params: {} 2023-10-11T15:13:33.297 DEBUG:boto:Token: None 2023-10-11T15:13:33.298 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d 2023-10-11T15:13:33.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VDgh/q7w8hDlzILkGCXnc0fqZBE= 2023-10-11T15:13:33.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VDgh/q7w8hDlzILkGCXnc0fqZBE=', 'Content-Length': '0'} 2023-10-11T15:13:33.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006913c021fb08a3da-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-10-11T15:13:33.302 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.302 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.303 DEBUG:boto:Method: GET 2023-10-11T15:13:33.303 DEBUG:boto:Path: /dghqvd-57/d 2023-10-11T15:13:33.304 DEBUG:boto:Data: 2023-10-11T15:13:33.304 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.305 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.305 DEBUG:boto:Params: {} 2023-10-11T15:13:33.306 DEBUG:boto:Token: None 2023-10-11T15:13:33.307 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d 2023-10-11T15:13:33.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2uQ3mjJrGT4nR/oC0u3WwK8gT1Q= 2023-10-11T15:13:33.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2uQ3mjJrGT4nR/oC0u3WwK8gT1Q=', 'Content-Length': '0'} 2023-10-11T15:13:33.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eaa656d4f291dba-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.311 DEBUG:boto:path=/dghqvd-57/d 2023-10-11T15:13:33.311 DEBUG:boto:auth_path=/dghqvd-57/d 2023-10-11T15:13:33.312 DEBUG:boto:Method: GET 2023-10-11T15:13:33.312 DEBUG:boto:Path: /dghqvd-57/d 2023-10-11T15:13:33.313 DEBUG:boto:Data: 2023-10-11T15:13:33.314 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.315 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.316 DEBUG:boto:Params: {} 2023-10-11T15:13:33.317 DEBUG:boto:Token: None 2023-10-11T15:13:33.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/d 2023-10-11T15:13:33.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2uQ3mjJrGT4nR/oC0u3WwK8gT1Q= 2023-10-11T15:13:33.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2uQ3mjJrGT4nR/oC0u3WwK8gT1Q=', 'Content-Length': '0'} 2023-10-11T15:13:33.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005918753d4511be43-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.322 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.322 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.323 DEBUG:boto:path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.323 DEBUG:boto:auth_path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.324 DEBUG:boto:Method: GET 2023-10-11T15:13:33.324 DEBUG:boto:Path: /dghqvd-57/?prefix=d 2023-10-11T15:13:33.325 DEBUG:boto:Data: 2023-10-11T15:13:33.325 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.326 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.326 DEBUG:boto:Params: {} 2023-10-11T15:13:33.327 DEBUG:boto:Token: None 2023-10-11T15:13:33.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008797c992de089278-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.332 DEBUG:boto:b'dghqvd-57d1000falsed2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.334 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.334 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.335 DEBUG:boto:path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.336 DEBUG:boto:auth_path=/dghqvd-57/?prefix=d 2023-10-11T15:13:33.336 DEBUG:boto:Method: GET 2023-10-11T15:13:33.337 DEBUG:boto:Path: /dghqvd-57/?prefix=d 2023-10-11T15:13:33.338 DEBUG:boto:Data: 2023-10-11T15:13:33.338 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.339 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.339 DEBUG:boto:Params: {} 2023-10-11T15:13:33.340 DEBUG:boto:Token: None 2023-10-11T15:13:33.340 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fde5232fa5544ac-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.344 DEBUG:boto:b'dghqvd-57d1000falsed2023-10-11T15:10:45.786Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:13:33.345 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.346 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.346 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.347 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.347 DEBUG:boto:Method: GET 2023-10-11T15:13:33.348 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.348 DEBUG:boto:Data: 2023-10-11T15:13:33.348 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.349 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.350 DEBUG:boto:Params: {} 2023-10-11T15:13:33.350 DEBUG:boto:Token: None 2023-10-11T15:13:33.351 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM= 2023-10-11T15:13:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM=', 'Content-Length': '0'} 2023-10-11T15:13:33.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006610d24f9c7d2a6a-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.354 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.355 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.355 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.356 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.356 DEBUG:boto:Method: GET 2023-10-11T15:13:33.356 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.357 DEBUG:boto:Data: 2023-10-11T15:13:33.357 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.358 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.359 DEBUG:boto:Params: {} 2023-10-11T15:13:33.359 DEBUG:boto:Token: None 2023-10-11T15:13:33.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM= 2023-10-11T15:13:33.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM=', 'Content-Length': '0'} 2023-10-11T15:13:33.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fc81153f0cbacb-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.363 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.364 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.364 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.365 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.365 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.366 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.366 DEBUG:boto:Data: 2023-10-11T15:13:33.366 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.367 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.368 DEBUG:boto:Params: {} 2023-10-11T15:13:33.368 DEBUG:boto:Token: None 2023-10-11T15:13:33.369 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1Q2LKPOIZrWdJOJEegpVP+DptUU= 2023-10-11T15:13:33.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1Q2LKPOIZrWdJOJEegpVP+DptUU=', 'Content-Length': '0'} 2023-10-11T15:13:33.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193f4033aa238694-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.372 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.373 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.373 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.373 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.374 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.374 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.375 DEBUG:boto:Data: 2023-10-11T15:13:33.375 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.375 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.376 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.376 DEBUG:boto:Params: {} 2023-10-11T15:13:33.377 DEBUG:boto:Token: None 2023-10-11T15:13:33.377 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1Q2LKPOIZrWdJOJEegpVP+DptUU= 2023-10-11T15:13:33.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1Q2LKPOIZrWdJOJEegpVP+DptUU=', 'Content-Length': '0'} 2023-10-11T15:13:33.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81727a00ffc24dc-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:13:33.381 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.381 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.382 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.382 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.383 DEBUG:boto:Method: GET 2023-10-11T15:13:33.383 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.383 DEBUG:boto:Data: 2023-10-11T15:13:33.384 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.385 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.385 DEBUG:boto:Params: {} 2023-10-11T15:13:33.386 DEBUG:boto:Token: None 2023-10-11T15:13:33.386 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM= 2023-10-11T15:13:33.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM=', 'Content-Length': '0'} 2023-10-11T15:13:33.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3157517bf21e5d2-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.390 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.390 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.391 DEBUG:boto:path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.391 DEBUG:boto:auth_path=/dghqvd-57/e?versionId=null 2023-10-11T15:13:33.391 DEBUG:boto:Method: GET 2023-10-11T15:13:33.392 DEBUG:boto:Path: /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.392 DEBUG:boto:Data: 2023-10-11T15:13:33.393 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.393 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.394 DEBUG:boto:Params: {} 2023-10-11T15:13:33.394 DEBUG:boto:Token: None 2023-10-11T15:13:33.395 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e?versionId=null 2023-10-11T15:13:33.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM= 2023-10-11T15:13:33.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w+fXMOcpUwm2Pq/PYJyycZznQCM=', 'Content-Length': '0'} 2023-10-11T15:13:33.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2bab4d134a458d-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.438 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.439 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.439 DEBUG:boto:path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.440 DEBUG:boto:auth_path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.440 DEBUG:boto:Method: GET 2023-10-11T15:13:33.441 DEBUG:boto:Path: /dghqvd-57/?prefix=e 2023-10-11T15:13:33.441 DEBUG:boto:Data: 2023-10-11T15:13:33.442 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.442 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.443 DEBUG:boto:Params: {} 2023-10-11T15:13:33.443 DEBUG:boto:Token: None 2023-10-11T15:13:33.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc1b9685be0c3f0-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.448 DEBUG:boto:b'dghqvd-57e1000falsee2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.449 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.449 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.449 DEBUG:boto:path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.450 DEBUG:boto:auth_path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.450 DEBUG:boto:Method: GET 2023-10-11T15:13:33.451 DEBUG:boto:Path: /dghqvd-57/?prefix=e 2023-10-11T15:13:33.451 DEBUG:boto:Data: 2023-10-11T15:13:33.452 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.453 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.453 DEBUG:boto:Params: {} 2023-10-11T15:13:33.453 DEBUG:boto:Token: None 2023-10-11T15:13:33.454 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d14eb033c776c3-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.458 DEBUG:boto:b'dghqvd-57e1000falsee2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.458 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.459 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.459 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.460 DEBUG:boto:Path: /dghqvd-57/e 2023-10-11T15:13:33.460 DEBUG:boto:Data: 2023-10-11T15:13:33.460 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.461 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.462 DEBUG:boto:Params: {} 2023-10-11T15:13:33.462 DEBUG:boto:Token: None 2023-10-11T15:13:33.462 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e 2023-10-11T15:13:33.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pR5Z0epuxUQ8klUd8UTyP3IESOo= 2023-10-11T15:13:33.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pR5Z0epuxUQ8klUd8UTyP3IESOo=', 'Content-Length': '0'} 2023-10-11T15:13:33.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004377740520e7643e-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.466 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.466 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.467 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.467 DEBUG:boto:Path: /dghqvd-57/e 2023-10-11T15:13:33.468 DEBUG:boto:Data: 2023-10-11T15:13:33.468 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.469 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.469 DEBUG:boto:Params: {} 2023-10-11T15:13:33.470 DEBUG:boto:Token: None 2023-10-11T15:13:33.470 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e 2023-10-11T15:13:33.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pR5Z0epuxUQ8klUd8UTyP3IESOo= 2023-10-11T15:13:33.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pR5Z0epuxUQ8klUd8UTyP3IESOo=', 'Content-Length': '0'} 2023-10-11T15:13:33.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9fa8c59b8421018-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-10-11T15:13:33.474 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.475 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.475 DEBUG:boto:Method: GET 2023-10-11T15:13:33.475 DEBUG:boto:Path: /dghqvd-57/e 2023-10-11T15:13:33.476 DEBUG:boto:Data: 2023-10-11T15:13:33.476 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.477 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.477 DEBUG:boto:Params: {} 2023-10-11T15:13:33.478 DEBUG:boto:Token: None 2023-10-11T15:13:33.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e 2023-10-11T15:13:33.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S/iBY/k21c5WddT1FVZIefQLtY8= 2023-10-11T15:13:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S/iBY/k21c5WddT1FVZIefQLtY8=', 'Content-Length': '0'} 2023-10-11T15:13:33.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc055ef3b93cb77-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.522 DEBUG:boto:path=/dghqvd-57/e 2023-10-11T15:13:33.523 DEBUG:boto:auth_path=/dghqvd-57/e 2023-10-11T15:13:33.523 DEBUG:boto:Method: GET 2023-10-11T15:13:33.524 DEBUG:boto:Path: /dghqvd-57/e 2023-10-11T15:13:33.524 DEBUG:boto:Data: 2023-10-11T15:13:33.525 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.526 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.526 DEBUG:boto:Params: {} 2023-10-11T15:13:33.527 DEBUG:boto:Token: None 2023-10-11T15:13:33.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/e 2023-10-11T15:13:33.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S/iBY/k21c5WddT1FVZIefQLtY8= 2023-10-11T15:13:33.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S/iBY/k21c5WddT1FVZIefQLtY8=', 'Content-Length': '0'} 2023-10-11T15:13:33.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449a12482be56dc6-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.531 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.531 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.532 DEBUG:boto:path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.532 DEBUG:boto:auth_path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.533 DEBUG:boto:Method: GET 2023-10-11T15:13:33.533 DEBUG:boto:Path: /dghqvd-57/?prefix=e 2023-10-11T15:13:33.534 DEBUG:boto:Data: 2023-10-11T15:13:33.534 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.535 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.535 DEBUG:boto:Params: {} 2023-10-11T15:13:33.536 DEBUG:boto:Token: None 2023-10-11T15:13:33.536 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb0bde514435172c-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.540 DEBUG:boto:b'dghqvd-57e1000falsee2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.541 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.541 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.542 DEBUG:boto:path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.542 DEBUG:boto:auth_path=/dghqvd-57/?prefix=e 2023-10-11T15:13:33.543 DEBUG:boto:Method: GET 2023-10-11T15:13:33.543 DEBUG:boto:Path: /dghqvd-57/?prefix=e 2023-10-11T15:13:33.544 DEBUG:boto:Data: 2023-10-11T15:13:33.544 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.545 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.545 DEBUG:boto:Params: {} 2023-10-11T15:13:33.546 DEBUG:boto:Token: None 2023-10-11T15:13:33.546 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b5af427733d34d0-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.550 DEBUG:boto:b'dghqvd-57e1000falsee2023-10-11T15:10:45.950Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:13:33.551 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.551 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.552 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.552 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.553 DEBUG:boto:Method: GET 2023-10-11T15:13:33.553 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.553 DEBUG:boto:Data: 2023-10-11T15:13:33.554 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.555 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.555 DEBUG:boto:Params: {} 2023-10-11T15:13:33.556 DEBUG:boto:Token: None 2023-10-11T15:13:33.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8= 2023-10-11T15:13:33.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8=', 'Content-Length': '0'} 2023-10-11T15:13:33.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b866e644f596a948-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.560 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.560 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.561 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.561 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.562 DEBUG:boto:Method: GET 2023-10-11T15:13:33.562 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.563 DEBUG:boto:Data: 2023-10-11T15:13:33.563 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.564 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.564 DEBUG:boto:Params: {} 2023-10-11T15:13:33.565 DEBUG:boto:Token: None 2023-10-11T15:13:33.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8= 2023-10-11T15:13:33.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8=', 'Content-Length': '0'} 2023-10-11T15:13:33.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a20e5c70317ef08-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.569 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.569 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.570 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.570 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.571 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.571 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.571 DEBUG:boto:Data: 2023-10-11T15:13:33.572 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.573 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.573 DEBUG:boto:Params: {} 2023-10-11T15:13:33.573 DEBUG:boto:Token: None 2023-10-11T15:13:33.574 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:68obp+c2Bb1iF0J4gF4fLRE+i9M= 2023-10-11T15:13:33.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:68obp+c2Bb1iF0J4gF4fLRE+i9M=', 'Content-Length': '0'} 2023-10-11T15:13:33.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ffb7a70ce841b5-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.577 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.578 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.578 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.579 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.579 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.579 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.580 DEBUG:boto:Data: 2023-10-11T15:13:33.580 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.581 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.581 DEBUG:boto:Params: {} 2023-10-11T15:13:33.582 DEBUG:boto:Token: None 2023-10-11T15:13:33.582 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:68obp+c2Bb1iF0J4gF4fLRE+i9M= 2023-10-11T15:13:33.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:68obp+c2Bb1iF0J4gF4fLRE+i9M=', 'Content-Length': '0'} 2023-10-11T15:13:33.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5ecbab89a2b42e-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:13:33.586 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.587 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.587 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.588 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.588 DEBUG:boto:Method: GET 2023-10-11T15:13:33.588 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.589 DEBUG:boto:Data: 2023-10-11T15:13:33.589 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.590 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.591 DEBUG:boto:Params: {} 2023-10-11T15:13:33.591 DEBUG:boto:Token: None 2023-10-11T15:13:33.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8= 2023-10-11T15:13:33.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8=', 'Content-Length': '0'} 2023-10-11T15:13:33.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0d785da07d5402-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.642 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.643 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.644 DEBUG:boto:path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.644 DEBUG:boto:auth_path=/dghqvd-57/f?versionId=null 2023-10-11T15:13:33.644 DEBUG:boto:Method: GET 2023-10-11T15:13:33.645 DEBUG:boto:Path: /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.645 DEBUG:boto:Data: 2023-10-11T15:13:33.646 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.647 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.647 DEBUG:boto:Params: {} 2023-10-11T15:13:33.647 DEBUG:boto:Token: None 2023-10-11T15:13:33.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f?versionId=null 2023-10-11T15:13:33.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8= 2023-10-11T15:13:33.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OeDTrbSIpa20+F+i++TZe75Ge8=', 'Content-Length': '0'} 2023-10-11T15:13:33.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af4c2cfca717fe5-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.652 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.652 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.653 DEBUG:boto:path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.653 DEBUG:boto:auth_path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.653 DEBUG:boto:Method: GET 2023-10-11T15:13:33.654 DEBUG:boto:Path: /dghqvd-57/?prefix=f 2023-10-11T15:13:33.654 DEBUG:boto:Data: 2023-10-11T15:13:33.655 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.656 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.656 DEBUG:boto:Params: {} 2023-10-11T15:13:33.657 DEBUG:boto:Token: None 2023-10-11T15:13:33.657 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da13d7e7b13b913e-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.661 DEBUG:boto:b'dghqvd-57f1000falsef2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.662 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.662 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.663 DEBUG:boto:path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.663 DEBUG:boto:auth_path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.663 DEBUG:boto:Method: GET 2023-10-11T15:13:33.664 DEBUG:boto:Path: /dghqvd-57/?prefix=f 2023-10-11T15:13:33.664 DEBUG:boto:Data: 2023-10-11T15:13:33.665 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.666 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.666 DEBUG:boto:Params: {} 2023-10-11T15:13:33.666 DEBUG:boto:Token: None 2023-10-11T15:13:33.667 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.667 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe191031c6b315c-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.710 DEBUG:boto:b'dghqvd-57f1000falsef2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.711 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.712 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.712 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.713 DEBUG:boto:Path: /dghqvd-57/f 2023-10-11T15:13:33.713 DEBUG:boto:Data: 2023-10-11T15:13:33.713 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.714 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.715 DEBUG:boto:Params: {} 2023-10-11T15:13:33.715 DEBUG:boto:Token: None 2023-10-11T15:13:33.716 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f 2023-10-11T15:13:33.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kl4+X4kQxa1X3jIGh5YllPccywg= 2023-10-11T15:13:33.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kl4+X4kQxa1X3jIGh5YllPccywg=', 'Content-Length': '0'} 2023-10-11T15:13:33.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6fc9a18a7e21f1d-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.719 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.720 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.720 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.720 DEBUG:boto:Path: /dghqvd-57/f 2023-10-11T15:13:33.721 DEBUG:boto:Data: 2023-10-11T15:13:33.721 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.722 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.723 DEBUG:boto:Params: {} 2023-10-11T15:13:33.723 DEBUG:boto:Token: None 2023-10-11T15:13:33.724 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f 2023-10-11T15:13:33.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kl4+X4kQxa1X3jIGh5YllPccywg= 2023-10-11T15:13:33.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kl4+X4kQxa1X3jIGh5YllPccywg=', 'Content-Length': '0'} 2023-10-11T15:13:33.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30220a8f557445e-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-10-11T15:13:33.729 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.729 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.730 DEBUG:boto:Method: GET 2023-10-11T15:13:33.730 DEBUG:boto:Path: /dghqvd-57/f 2023-10-11T15:13:33.731 DEBUG:boto:Data: 2023-10-11T15:13:33.731 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.732 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.733 DEBUG:boto:Params: {} 2023-10-11T15:13:33.733 DEBUG:boto:Token: None 2023-10-11T15:13:33.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f 2023-10-11T15:13:33.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4sRrqM9HY8V0BPrEOllj6xtt658= 2023-10-11T15:13:33.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4sRrqM9HY8V0BPrEOllj6xtt658=', 'Content-Length': '0'} 2023-10-11T15:13:33.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6fac6dfd85d944-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.738 DEBUG:boto:path=/dghqvd-57/f 2023-10-11T15:13:33.738 DEBUG:boto:auth_path=/dghqvd-57/f 2023-10-11T15:13:33.739 DEBUG:boto:Method: GET 2023-10-11T15:13:33.739 DEBUG:boto:Path: /dghqvd-57/f 2023-10-11T15:13:33.740 DEBUG:boto:Data: 2023-10-11T15:13:33.740 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.741 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.742 DEBUG:boto:Params: {} 2023-10-11T15:13:33.742 DEBUG:boto:Token: None 2023-10-11T15:13:33.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/f 2023-10-11T15:13:33.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4sRrqM9HY8V0BPrEOllj6xtt658= 2023-10-11T15:13:33.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4sRrqM9HY8V0BPrEOllj6xtt658=', 'Content-Length': '0'} 2023-10-11T15:13:33.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80a801f929cf82a-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.746 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.747 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.747 DEBUG:boto:path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.748 DEBUG:boto:auth_path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.748 DEBUG:boto:Method: GET 2023-10-11T15:13:33.749 DEBUG:boto:Path: /dghqvd-57/?prefix=f 2023-10-11T15:13:33.749 DEBUG:boto:Data: 2023-10-11T15:13:33.750 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.751 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.751 DEBUG:boto:Params: {} 2023-10-11T15:13:33.753 DEBUG:boto:Token: None 2023-10-11T15:13:33.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a6b6226766c1e7-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.758 DEBUG:boto:b'dghqvd-57f1000falsef2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.759 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.759 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.760 DEBUG:boto:path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.760 DEBUG:boto:auth_path=/dghqvd-57/?prefix=f 2023-10-11T15:13:33.760 DEBUG:boto:Method: GET 2023-10-11T15:13:33.761 DEBUG:boto:Path: /dghqvd-57/?prefix=f 2023-10-11T15:13:33.761 DEBUG:boto:Data: 2023-10-11T15:13:33.762 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.763 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.763 DEBUG:boto:Params: {} 2023-10-11T15:13:33.764 DEBUG:boto:Token: None 2023-10-11T15:13:33.765 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baf4c049c9284059-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.768 DEBUG:boto:b'dghqvd-57f1000falsef2023-10-11T15:10:45.970Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:13:33.770 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.770 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.771 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.771 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.771 DEBUG:boto:Method: GET 2023-10-11T15:13:33.772 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.772 DEBUG:boto:Data: 2023-10-11T15:13:33.773 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.774 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.774 DEBUG:boto:Params: {} 2023-10-11T15:13:33.775 DEBUG:boto:Token: None 2023-10-11T15:13:33.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY= 2023-10-11T15:13:33.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY=', 'Content-Length': '0'} 2023-10-11T15:13:33.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1a52fdce16620f-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.779 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.779 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.780 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.780 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.781 DEBUG:boto:Method: GET 2023-10-11T15:13:33.781 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.782 DEBUG:boto:Data: 2023-10-11T15:13:33.782 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.783 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.783 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.784 DEBUG:boto:Params: {} 2023-10-11T15:13:33.784 DEBUG:boto:Token: None 2023-10-11T15:13:33.785 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.785 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY= 2023-10-11T15:13:33.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY=', 'Content-Length': '0'} 2023-10-11T15:13:33.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f43465ef6b157e-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.788 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.789 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.789 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.790 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.790 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.791 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.791 DEBUG:boto:Data: 2023-10-11T15:13:33.791 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.792 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.792 DEBUG:boto:Params: {} 2023-10-11T15:13:33.793 DEBUG:boto:Token: None 2023-10-11T15:13:33.793 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u8fZrApruEwmTe434797uvtnPmE= 2023-10-11T15:13:33.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u8fZrApruEwmTe434797uvtnPmE=', 'Content-Length': '0'} 2023-10-11T15:13:33.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387233ac4174f28f-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.796 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.797 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.797 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.798 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.798 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.799 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.799 DEBUG:boto:Data: 2023-10-11T15:13:33.800 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.800 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.801 DEBUG:boto:Params: {} 2023-10-11T15:13:33.801 DEBUG:boto:Token: None 2023-10-11T15:13:33.802 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u8fZrApruEwmTe434797uvtnPmE= 2023-10-11T15:13:33.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u8fZrApruEwmTe434797uvtnPmE=', 'Content-Length': '0'} 2023-10-11T15:13:33.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008665a32bdbe2c278-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:13:33.806 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.806 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.806 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.807 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.807 DEBUG:boto:Method: GET 2023-10-11T15:13:33.808 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.808 DEBUG:boto:Data: 2023-10-11T15:13:33.808 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.809 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.810 DEBUG:boto:Params: {} 2023-10-11T15:13:33.810 DEBUG:boto:Token: None 2023-10-11T15:13:33.811 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY= 2023-10-11T15:13:33.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY=', 'Content-Length': '0'} 2023-10-11T15:13:33.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008679b66f278bde90-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.854 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.855 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.855 DEBUG:boto:path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.856 DEBUG:boto:auth_path=/dghqvd-57/g?versionId=null 2023-10-11T15:13:33.856 DEBUG:boto:Method: GET 2023-10-11T15:13:33.857 DEBUG:boto:Path: /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.857 DEBUG:boto:Data: 2023-10-11T15:13:33.857 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.858 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.859 DEBUG:boto:Params: {} 2023-10-11T15:13:33.859 DEBUG:boto:Token: None 2023-10-11T15:13:33.860 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g?versionId=null 2023-10-11T15:13:33.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY= 2023-10-11T15:13:33.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yOccogaK8eTnB6B5eotV05NoqdY=', 'Content-Length': '0'} 2023-10-11T15:13:33.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009055b744959166e4-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.864 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.864 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.865 DEBUG:boto:path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.865 DEBUG:boto:auth_path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.866 DEBUG:boto:Method: GET 2023-10-11T15:13:33.866 DEBUG:boto:Path: /dghqvd-57/?prefix=g 2023-10-11T15:13:33.867 DEBUG:boto:Data: 2023-10-11T15:13:33.867 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.868 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.869 DEBUG:boto:Params: {} 2023-10-11T15:13:33.869 DEBUG:boto:Token: None 2023-10-11T15:13:33.870 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5c82e617b9d1aed-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.874 DEBUG:boto:b'dghqvd-57g1000falseg2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.875 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.875 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.876 DEBUG:boto:path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.876 DEBUG:boto:auth_path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.877 DEBUG:boto:Method: GET 2023-10-11T15:13:33.877 DEBUG:boto:Path: /dghqvd-57/?prefix=g 2023-10-11T15:13:33.877 DEBUG:boto:Data: 2023-10-11T15:13:33.878 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.878 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.879 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.879 DEBUG:boto:Params: {} 2023-10-11T15:13:33.880 DEBUG:boto:Token: None 2023-10-11T15:13:33.881 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d732c84d3130fc21-006526bb9d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.926 DEBUG:boto:b'dghqvd-57g1000falseg2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:33.927 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.927 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.928 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.928 DEBUG:boto:Path: /dghqvd-57/g 2023-10-11T15:13:33.929 DEBUG:boto:Data: 2023-10-11T15:13:33.929 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.930 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.930 DEBUG:boto:Params: {} 2023-10-11T15:13:33.931 DEBUG:boto:Token: None 2023-10-11T15:13:33.932 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g 2023-10-11T15:13:33.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pZ67lXM1scl87nt9a540KveLNw= 2023-10-11T15:13:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pZ67lXM1scl87nt9a540KveLNw=', 'Content-Length': '0'} 2023-10-11T15:13:33.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a405a6784de7b96-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.935 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.936 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.936 DEBUG:boto:Method: HEAD 2023-10-11T15:13:33.937 DEBUG:boto:Path: /dghqvd-57/g 2023-10-11T15:13:33.937 DEBUG:boto:Data: 2023-10-11T15:13:33.937 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.938 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.939 DEBUG:boto:Params: {} 2023-10-11T15:13:33.939 DEBUG:boto:Token: None 2023-10-11T15:13:33.940 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g 2023-10-11T15:13:33.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+pZ67lXM1scl87nt9a540KveLNw= 2023-10-11T15:13:33.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+pZ67lXM1scl87nt9a540KveLNw=', 'Content-Length': '0'} 2023-10-11T15:13:33.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb266c3e2282c742-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-10-11T15:13:33.944 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.944 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.945 DEBUG:boto:Method: GET 2023-10-11T15:13:33.945 DEBUG:boto:Path: /dghqvd-57/g 2023-10-11T15:13:33.945 DEBUG:boto:Data: 2023-10-11T15:13:33.946 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.947 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.947 DEBUG:boto:Params: {} 2023-10-11T15:13:33.948 DEBUG:boto:Token: None 2023-10-11T15:13:33.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g 2023-10-11T15:13:33.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IE2CrD07zRBRwjRK85itD30oIz0= 2023-10-11T15:13:33.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IE2CrD07zRBRwjRK85itD30oIz0=', 'Content-Length': '0'} 2023-10-11T15:13:33.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c18a7733cd1e6f3-006526bb9d-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.952 DEBUG:boto:path=/dghqvd-57/g 2023-10-11T15:13:33.952 DEBUG:boto:auth_path=/dghqvd-57/g 2023-10-11T15:13:33.953 DEBUG:boto:Method: GET 2023-10-11T15:13:33.953 DEBUG:boto:Path: /dghqvd-57/g 2023-10-11T15:13:33.954 DEBUG:boto:Data: 2023-10-11T15:13:33.954 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:33.955 DEBUG:boto:Port: 8000 2023-10-11T15:13:33.955 DEBUG:boto:Params: {} 2023-10-11T15:13:33.956 DEBUG:boto:Token: None 2023-10-11T15:13:33.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/g 2023-10-11T15:13:33.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IE2CrD07zRBRwjRK85itD30oIz0= 2023-10-11T15:13:33.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IE2CrD07zRBRwjRK85itD30oIz0=', 'Content-Length': '0'} 2023-10-11T15:13:33.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9089123dec331fc-006526bb9d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:33.960 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:33.960 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:33.960 DEBUG:boto:path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.961 DEBUG:boto:auth_path=/dghqvd-57/?prefix=g 2023-10-11T15:13:33.961 DEBUG:boto:Method: GET 2023-10-11T15:13:33.962 DEBUG:boto:Path: /dghqvd-57/?prefix=g 2023-10-11T15:13:33.962 DEBUG:boto:Data: 2023-10-11T15:13:33.962 DEBUG:boto:Headers: {} 2023-10-11T15:13:33.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:33.963 DEBUG:boto:Port: 8001 2023-10-11T15:13:33.964 DEBUG:boto:Params: {} 2023-10-11T15:13:33.964 DEBUG:boto:Token: None 2023-10-11T15:13:33.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:33 GMT /dghqvd-57/ 2023-10-11T15:13:33.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI= 2023-10-11T15:13:33.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+JAolkWHRE6RnSNPD1Zd8h5ngGI=', 'Content-Length': '0'} 2023-10-11T15:13:33.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004995b4bb60c382c6-006526bb9d-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.010 DEBUG:boto:b'dghqvd-57g1000falseg2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.011 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:34.011 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:34.012 DEBUG:boto:path=/dghqvd-57/?prefix=g 2023-10-11T15:13:34.012 DEBUG:boto:auth_path=/dghqvd-57/?prefix=g 2023-10-11T15:13:34.013 DEBUG:boto:Method: GET 2023-10-11T15:13:34.013 DEBUG:boto:Path: /dghqvd-57/?prefix=g 2023-10-11T15:13:34.013 DEBUG:boto:Data: 2023-10-11T15:13:34.014 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.015 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.015 DEBUG:boto:Params: {} 2023-10-11T15:13:34.015 DEBUG:boto:Token: None 2023-10-11T15:13:34.016 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/ 2023-10-11T15:13:34.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4= 2023-10-11T15:13:34.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4=', 'Content-Length': '0'} 2023-10-11T15:13:34.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047067abcd8af3f6f-006526bb9e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.020 DEBUG:boto:b'dghqvd-57g1000falseg2023-10-11T15:10:46.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:13:34.021 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.021 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.022 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.022 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.022 DEBUG:boto:Method: GET 2023-10-11T15:13:34.023 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.023 DEBUG:boto:Data: 2023-10-11T15:13:34.024 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.025 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.025 DEBUG:boto:Params: {} 2023-10-11T15:13:34.026 DEBUG:boto:Token: None 2023-10-11T15:13:34.026 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU= 2023-10-11T15:13:34.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU=', 'Content-Length': '0'} 2023-10-11T15:13:34.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3622db897ca6c9d-006526bb9e-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.030 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.030 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.031 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.031 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.032 DEBUG:boto:Method: GET 2023-10-11T15:13:34.032 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.032 DEBUG:boto:Data: 2023-10-11T15:13:34.033 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.034 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.034 DEBUG:boto:Params: {} 2023-10-11T15:13:34.035 DEBUG:boto:Token: None 2023-10-11T15:13:34.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU= 2023-10-11T15:13:34.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU=', 'Content-Length': '0'} 2023-10-11T15:13:34.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea567151f903e36-006526bb9e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.078 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.079 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.079 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.080 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.080 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.080 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.081 DEBUG:boto:Data: 2023-10-11T15:13:34.081 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.082 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.082 DEBUG:boto:Params: {} 2023-10-11T15:13:34.083 DEBUG:boto:Token: None 2023-10-11T15:13:34.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:llYZnV42OMgTmaBA7sd4EVDNIsY= 2023-10-11T15:13:34.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:llYZnV42OMgTmaBA7sd4EVDNIsY=', 'Content-Length': '0'} 2023-10-11T15:13:34.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad00fb0882c0a89a-006526bb9e-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.087 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.087 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.088 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.088 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.089 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.089 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.090 DEBUG:boto:Data: 2023-10-11T15:13:34.090 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.091 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.091 DEBUG:boto:Params: {} 2023-10-11T15:13:34.092 DEBUG:boto:Token: None 2023-10-11T15:13:34.092 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:llYZnV42OMgTmaBA7sd4EVDNIsY= 2023-10-11T15:13:34.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:llYZnV42OMgTmaBA7sd4EVDNIsY=', 'Content-Length': '0'} 2023-10-11T15:13:34.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd31af743d28767e-006526bb9e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:13:34.096 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.097 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.097 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.097 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.098 DEBUG:boto:Method: GET 2023-10-11T15:13:34.098 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.099 DEBUG:boto:Data: 2023-10-11T15:13:34.099 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.100 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.100 DEBUG:boto:Params: {} 2023-10-11T15:13:34.101 DEBUG:boto:Token: None 2023-10-11T15:13:34.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU= 2023-10-11T15:13:34.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU=', 'Content-Length': '0'} 2023-10-11T15:13:34.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089756b1251bd81bd-006526bb9e-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.105 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.105 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.106 DEBUG:boto:path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.106 DEBUG:boto:auth_path=/dghqvd-57/h?versionId=null 2023-10-11T15:13:34.107 DEBUG:boto:Method: GET 2023-10-11T15:13:34.107 DEBUG:boto:Path: /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.108 DEBUG:boto:Data: 2023-10-11T15:13:34.108 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.109 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.109 DEBUG:boto:Params: {} 2023-10-11T15:13:34.109 DEBUG:boto:Token: None 2023-10-11T15:13:34.110 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h?versionId=null 2023-10-11T15:13:34.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU= 2023-10-11T15:13:34.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oy2FagxCMwabrZz58OFZHCtacEU=', 'Content-Length': '0'} 2023-10-11T15:13:34.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0d14cee36e1cff2-006526bb9e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.114 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:34.115 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:34.115 DEBUG:boto:path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.115 DEBUG:boto:auth_path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.116 DEBUG:boto:Method: GET 2023-10-11T15:13:34.116 DEBUG:boto:Path: /dghqvd-57/?prefix=h 2023-10-11T15:13:34.117 DEBUG:boto:Data: 2023-10-11T15:13:34.117 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.118 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.118 DEBUG:boto:Params: {} 2023-10-11T15:13:34.119 DEBUG:boto:Token: None 2023-10-11T15:13:34.119 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/ 2023-10-11T15:13:34.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4= 2023-10-11T15:13:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4=', 'Content-Length': '0'} 2023-10-11T15:13:34.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad999bacc913049e-006526bb9e-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.123 DEBUG:boto:b'dghqvd-57h1000falseh2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.124 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:34.124 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:34.125 DEBUG:boto:path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.125 DEBUG:boto:auth_path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.126 DEBUG:boto:Method: GET 2023-10-11T15:13:34.126 DEBUG:boto:Path: /dghqvd-57/?prefix=h 2023-10-11T15:13:34.127 DEBUG:boto:Data: 2023-10-11T15:13:34.127 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.128 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.128 DEBUG:boto:Params: {} 2023-10-11T15:13:34.129 DEBUG:boto:Token: None 2023-10-11T15:13:34.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/ 2023-10-11T15:13:34.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4= 2023-10-11T15:13:34.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4=', 'Content-Length': '0'} 2023-10-11T15:13:34.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035463c205712bcd6-006526bb9e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.133 DEBUG:boto:b'dghqvd-57h1000falseh2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.134 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.134 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.134 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.135 DEBUG:boto:Path: /dghqvd-57/h 2023-10-11T15:13:34.135 DEBUG:boto:Data: 2023-10-11T15:13:34.136 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.136 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.137 DEBUG:boto:Params: {} 2023-10-11T15:13:34.137 DEBUG:boto:Token: None 2023-10-11T15:13:34.138 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h 2023-10-11T15:13:34.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BxeBzD9XRpoG4vS727KwheWjoyM= 2023-10-11T15:13:34.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxeBzD9XRpoG4vS727KwheWjoyM=', 'Content-Length': '0'} 2023-10-11T15:13:34.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e811e99c3fcbbfd-006526bb9e-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.141 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.141 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.142 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.142 DEBUG:boto:Path: /dghqvd-57/h 2023-10-11T15:13:34.143 DEBUG:boto:Data: 2023-10-11T15:13:34.143 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.143 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.144 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.144 DEBUG:boto:Params: {} 2023-10-11T15:13:34.145 DEBUG:boto:Token: None 2023-10-11T15:13:34.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h 2023-10-11T15:13:34.145 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BxeBzD9XRpoG4vS727KwheWjoyM= 2023-10-11T15:13:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxeBzD9XRpoG4vS727KwheWjoyM=', 'Content-Length': '0'} 2023-10-11T15:13:34.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a0d0581f199b52-006526bb9e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-10-11T15:13:34.149 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.150 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.150 DEBUG:boto:Method: GET 2023-10-11T15:13:34.150 DEBUG:boto:Path: /dghqvd-57/h 2023-10-11T15:13:34.151 DEBUG:boto:Data: 2023-10-11T15:13:34.151 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.152 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.153 DEBUG:boto:Params: {} 2023-10-11T15:13:34.153 DEBUG:boto:Token: None 2023-10-11T15:13:34.154 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h 2023-10-11T15:13:34.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pZVyIOQZ6i0ejBeYTO5kSzzLhuM= 2023-10-11T15:13:34.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pZVyIOQZ6i0ejBeYTO5kSzzLhuM=', 'Content-Length': '0'} 2023-10-11T15:13:34.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-57:daa316e9-869c-4074-a756-a8adfb4a3573.5751.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030baa41897ecaddc-006526bb9e-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.198 DEBUG:boto:path=/dghqvd-57/h 2023-10-11T15:13:34.199 DEBUG:boto:auth_path=/dghqvd-57/h 2023-10-11T15:13:34.199 DEBUG:boto:Method: GET 2023-10-11T15:13:34.200 DEBUG:boto:Path: /dghqvd-57/h 2023-10-11T15:13:34.200 DEBUG:boto:Data: 2023-10-11T15:13:34.201 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.201 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.202 DEBUG:boto:Params: {} 2023-10-11T15:13:34.202 DEBUG:boto:Token: None 2023-10-11T15:13:34.203 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/h 2023-10-11T15:13:34.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pZVyIOQZ6i0ejBeYTO5kSzzLhuM= 2023-10-11T15:13:34.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pZVyIOQZ6i0ejBeYTO5kSzzLhuM=', 'Content-Length': '0'} 2023-10-11T15:13:34.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:10:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000917bce4a685c21ce-006526bb9e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.207 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:34.207 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:34.208 DEBUG:boto:path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.208 DEBUG:boto:auth_path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.209 DEBUG:boto:Method: GET 2023-10-11T15:13:34.209 DEBUG:boto:Path: /dghqvd-57/?prefix=h 2023-10-11T15:13:34.209 DEBUG:boto:Data: 2023-10-11T15:13:34.210 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.211 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.211 DEBUG:boto:Params: {} 2023-10-11T15:13:34.212 DEBUG:boto:Token: None 2023-10-11T15:13:34.212 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/ 2023-10-11T15:13:34.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4= 2023-10-11T15:13:34.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4=', 'Content-Length': '0'} 2023-10-11T15:13:34.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e368274d083b7ea7-006526bb9e-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.216 DEBUG:boto:b'dghqvd-57h1000falseh2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.217 DEBUG:boto:path=/dghqvd-57/ 2023-10-11T15:13:34.217 DEBUG:boto:auth_path=/dghqvd-57/ 2023-10-11T15:13:34.218 DEBUG:boto:path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.218 DEBUG:boto:auth_path=/dghqvd-57/?prefix=h 2023-10-11T15:13:34.218 DEBUG:boto:Method: GET 2023-10-11T15:13:34.219 DEBUG:boto:Path: /dghqvd-57/?prefix=h 2023-10-11T15:13:34.219 DEBUG:boto:Data: 2023-10-11T15:13:34.220 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.221 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.221 DEBUG:boto:Params: {} 2023-10-11T15:13:34.221 DEBUG:boto:Token: None 2023-10-11T15:13:34.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-57/ 2023-10-11T15:13:34.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4= 2023-10-11T15:13:34.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:82h96rL4ycSKT1p0AnafDB6/9F4=', 'Content-Length': '0'} 2023-10-11T15:13:34.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000921f6a04d05795df-006526bb9e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.266 DEBUG:boto:b'dghqvd-57h1000falseh2023-10-11T15:10:46.054Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:34.267 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-57 zones={a2, a1} 2023-10-11T15:13:34.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-10-11T15:13:34.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.270 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:34.328 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:34.328 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:34.361 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:13:34.374 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:13:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.374 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:13:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:13:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:13:34.375 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:13:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:13:34.376 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:13:34.377 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.377 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.377 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037045.900686_427.1", 2023-10-11T15:13:34.377 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:10:45.900686Z" 2023-10-11T15:13:34.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.378 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:13:34.379 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037045.900686_427.1'} 2023-10-11T15:13:34.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:13:34.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.381 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:34.433 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:34.433 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:34.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:34.461+0000 7fb5033ba780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:13:34.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:34.462+0000 7fb5033ba780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:13:34.476 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:13:34.477 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:13:34.477 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:13:34.477 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:13:34.477 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:13:34.477 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:13:34.478 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.478 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.478 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:13:34.478 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.479 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:13:34.479 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.479 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.479 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:13:34.479 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.480 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:13:34.480 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.480 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:13:34.480 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.480 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.481 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.481 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:13:34.481 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.482 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.482 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:13:34.482 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.482 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.482 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.483 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:13:34.483 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.483 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.483 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.483 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.484 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:13:34.484 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.484 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.484 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:13:34.484 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.485 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.485 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.485 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:13:34.485 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.485 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.486 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.486 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:13:34.486 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.487 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.487 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037045.900686_427.1", 2023-10-11T15:13:34.487 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.487 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.487 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.488 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:10:45.900686Z", 2023-10-11T15:13:34.488 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.488 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.488 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.488 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:13:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:13:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:13:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:13:34.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.490 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:13:34Z" 2023-10-11T15:13:34.490 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:13:34.491 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037045.900686_427.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:10:45.900686Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:13:34Z" } 2023-10-11T15:13:34.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037045.900686_427.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:10:45.900686Z', 'realm_epoch': 2}}] 2023-10-11T15:13:34.492 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037045.900686_427.1'} 2023-10-11T15:13:34.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037045.900686_427.1'} 2023-10-11T15:13:34.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:13:34.493 DEBUG:boto:path=/ 2023-10-11T15:13:34.493 DEBUG:boto:auth_path=/ 2023-10-11T15:13:34.494 DEBUG:boto:Method: GET 2023-10-11T15:13:34.494 DEBUG:boto:Path: / 2023-10-11T15:13:34.495 DEBUG:boto:Data: 2023-10-11T15:13:34.495 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.496 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.496 DEBUG:boto:Params: {} 2023-10-11T15:13:34.497 DEBUG:boto:Token: None 2023-10-11T15:13:34.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT / 2023-10-11T15:13:34.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pwXzNjZTAhpAq6+kBX1YKjdYc9E= 2023-10-11T15:13:34.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pwXzNjZTAhpAq6+kBX1YKjdYc9E=', 'Content-Length': '0'} 2023-10-11T15:13:34.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb877e85aaec4458-006526bb9e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.503 DEBUG:boto:path=/ 2023-10-11T15:13:34.503 DEBUG:boto:auth_path=/ 2023-10-11T15:13:34.503 DEBUG:boto:Method: GET 2023-10-11T15:13:34.504 DEBUG:boto:Path: / 2023-10-11T15:13:34.504 DEBUG:boto:Data: 2023-10-11T15:13:34.505 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:34.505 DEBUG:boto:Port: 8001 2023-10-11T15:13:34.506 DEBUG:boto:Params: {} 2023-10-11T15:13:34.506 DEBUG:boto:Token: None 2023-10-11T15:13:34.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:34 GMT / 2023-10-11T15:13:34.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pwXzNjZTAhpAq6+kBX1YKjdYc9E= 2023-10-11T15:13:34.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pwXzNjZTAhpAq6+kBX1YKjdYc9E=', 'Content-Length': '0'} 2023-10-11T15:13:34.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088bdc520a2bf01e-006526bb9e-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-10-11T15:13:34.512 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.512 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.513 DEBUG:boto:Method: PUT 2023-10-11T15:13:34.513 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.513 DEBUG:boto:Data: 2023-10-11T15:13:34.514 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.515 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.515 DEBUG:boto:Params: {} 2023-10-11T15:13:34.515 DEBUG:boto:Token: None 2023-10-11T15:13:34.516 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/ 2023-10-11T15:13:34.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o69bJ/ICc8PmKwClgsutbkj4324= 2023-10-11T15:13:34.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o69bJ/ICc8PmKwClgsutbkj4324=', 'Content-Length': '0'} 2023-10-11T15:13:34.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2f6dec3ef5d235b-006526bb9e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.531 DEBUG:rgw_multi.tests:created bucket=dghqvd-58 2023-10-11T15:13:34.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.532 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:34.585 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:34.585 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:34.614 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:13:34.626 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:13:34.627 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.627 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:13:34.627 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:13:34.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.628 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:13:34.628 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:13:34.628 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.629 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:13:34.629 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:13:34.629 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.629 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.630 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:13:34.630 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:13:34.528226Z" 2023-10-11T15:13:34.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.630 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:13:34.631 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:13:34.632 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:13:34.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.633 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:34.682 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:34.683 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:34.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:34.709+0000 7ff03de15780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:13:34.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:34.710+0000 7ff03de15780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:13:34.723 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:13:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:13:34.723 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:13:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:13:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:13:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:13:34.724 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:13:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:13:34.725 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:13:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.726 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:13:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:13:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.728 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.728 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:13:34.728 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.728 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.728 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:13:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:13:34.729 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.730 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.730 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.730 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:13:34.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.731 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.731 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:13:34.731 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.731 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.732 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.732 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:13:34.732 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.732 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.733 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:13:34.733 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:13:34.733 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:13:34.733 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:13:34.734 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:13:34.734 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:13:34.734 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:13:34.734 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:13:34.735 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:13:34.735 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:13:34.735 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.735 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:13:34.735 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:13:34.736 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.736 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:13:34.736 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:13:34.736 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:13:34.737 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:13:34.737 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:13:34Z" 2023-10-11T15:13:34.737 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:13:34.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:13:34Z" } 2023-10-11T15:13:34.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:13:34.739 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:13:34.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:13:34.740 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:13:34.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:34.741 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:34.793 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:34.793 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:34.832 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:13:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:13:34.832 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:13:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000005", 2023-10-11T15:13:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:13:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:13:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:13:34.833 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:13:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:13:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:13:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000013", 2023-10-11T15:13:34.834 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:13:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006" 2023-10-11T15:13:34.835 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:13:34.835 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:13:34.836 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.836 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.837 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.837 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.838 DEBUG:boto:Data: 2023-10-11T15:13:34.838 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.839 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.839 DEBUG:boto:Params: {} 2023-10-11T15:13:34.840 DEBUG:boto:Token: None 2023-10-11T15:13:34.840 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/ 2023-10-11T15:13:34.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0= 2023-10-11T15:13:34.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0=', 'Content-Length': '0'} 2023-10-11T15:13:34.843 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc67eeb74536080d-006526bb9e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.844 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:34.844 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:34.845 DEBUG:boto:Method: PUT 2023-10-11T15:13:34.845 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:34.846 DEBUG:boto:Data: 2023-10-11T15:13:34.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:34.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.847 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.847 DEBUG:boto:Params: {} 2023-10-11T15:13:34.848 DEBUG:boto:Token: None 2023-10-11T15:13:34.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:34.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TYSQRV0vt/ot8YQvM6X9g5Savs4= 2023-10-11T15:13:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TYSQRV0vt/ot8YQvM6X9g5Savs4='} 2023-10-11T15:13:34.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009226d0489bcfafe9-006526bb9e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.856 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.856 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.857 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.857 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.857 DEBUG:boto:Data: 2023-10-11T15:13:34.858 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.858 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.859 DEBUG:boto:Params: {} 2023-10-11T15:13:34.859 DEBUG:boto:Token: None 2023-10-11T15:13:34.860 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/ 2023-10-11T15:13:34.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0= 2023-10-11T15:13:34.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0=', 'Content-Length': '0'} 2023-10-11T15:13:34.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058fc58e75a718b4a-006526bb9e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.863 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:34.864 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:34.864 DEBUG:boto:Method: PUT 2023-10-11T15:13:34.865 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:34.865 DEBUG:boto:Data: 2023-10-11T15:13:34.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:34.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.866 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.867 DEBUG:boto:Params: {} 2023-10-11T15:13:34.867 DEBUG:boto:Token: None 2023-10-11T15:13:34.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:34.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d/DS3TDvFlXbalCl8BmaKE9kJVU= 2023-10-11T15:13:34.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d/DS3TDvFlXbalCl8BmaKE9kJVU='} 2023-10-11T15:13:34.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1e01d6a19d988e0-006526bb9e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.915 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.916 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.916 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.916 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.917 DEBUG:boto:Data: 2023-10-11T15:13:34.918 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.918 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.919 DEBUG:boto:Params: {} 2023-10-11T15:13:34.919 DEBUG:boto:Token: None 2023-10-11T15:13:34.920 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/ 2023-10-11T15:13:34.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0= 2023-10-11T15:13:34.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0=', 'Content-Length': '0'} 2023-10-11T15:13:34.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d34fa92dd89cda89-006526bb9e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.924 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:34.924 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:34.924 DEBUG:boto:Method: PUT 2023-10-11T15:13:34.925 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:34.925 DEBUG:boto:Data: 2023-10-11T15:13:34.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:34.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.926 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.927 DEBUG:boto:Params: {} 2023-10-11T15:13:34.927 DEBUG:boto:Token: None 2023-10-11T15:13:34.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:34.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lbeYZYrBkP16FuTcb0/T7u2d7a8= 2023-10-11T15:13:34.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lbeYZYrBkP16FuTcb0/T7u2d7a8='} 2023-10-11T15:13:34.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fcae0c821ad0cb9-006526bb9e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.934 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.935 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.935 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.935 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.936 DEBUG:boto:Data: 2023-10-11T15:13:34.936 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.937 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.937 DEBUG:boto:Params: {} 2023-10-11T15:13:34.938 DEBUG:boto:Token: None 2023-10-11T15:13:34.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/ 2023-10-11T15:13:34.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0= 2023-10-11T15:13:34.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jy/UkhCdipfMlc/cDaQHgx13CS0=', 'Content-Length': '0'} 2023-10-11T15:13:34.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000821b205afd683c4d-006526bb9e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.942 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:34.942 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:34.943 DEBUG:boto:Method: PUT 2023-10-11T15:13:34.943 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:34.944 DEBUG:boto:Data: 2023-10-11T15:13:34.944 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:34.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.945 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.945 DEBUG:boto:Params: {} 2023-10-11T15:13:34.946 DEBUG:boto:Token: None 2023-10-11T15:13:34.946 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:34 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:34.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x5ou4NNKWvBHge6rWN5OWC7Gvvw= 2023-10-11T15:13:34.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x5ou4NNKWvBHge6rWN5OWC7Gvvw='} 2023-10-11T15:13:34.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092ab778c23bf9474-006526bb9e-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:34.995 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:34.996 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:34.996 DEBUG:boto:Method: HEAD 2023-10-11T15:13:34.997 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:34.997 DEBUG:boto:Data: 2023-10-11T15:13:34.997 DEBUG:boto:Headers: {} 2023-10-11T15:13:34.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:34.998 DEBUG:boto:Port: 8000 2023-10-11T15:13:34.999 DEBUG:boto:Params: {} 2023-10-11T15:13:34.999 DEBUG:boto:Token: None 2023-10-11T15:13:35.000 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a938530fbccbabbe-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.004 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:35.004 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:35.004 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.005 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:35.005 DEBUG:boto:Data: 2023-10-11T15:13:35.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.007 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.007 DEBUG:boto:Params: {} 2023-10-11T15:13:35.008 DEBUG:boto:Token: None 2023-10-11T15:13:35.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:35.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:24escT9Y/03k1XORS/Wvat0ufGc= 2023-10-11T15:13:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:24escT9Y/03k1XORS/Wvat0ufGc='} 2023-10-11T15:13:35.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee36d5b5eb39af17-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.014 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:35.015 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:35.015 DEBUG:boto:Method: HEAD 2023-10-11T15:13:35.015 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:35.016 DEBUG:boto:Data: 2023-10-11T15:13:35.016 DEBUG:boto:Headers: {} 2023-10-11T15:13:35.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.017 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.018 DEBUG:boto:Params: {} 2023-10-11T15:13:35.018 DEBUG:boto:Token: None 2023-10-11T15:13:35.019 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ca4999ca231346b-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.022 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:35.023 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:35.023 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.024 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:35.024 DEBUG:boto:Data: 2023-10-11T15:13:35.025 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.025 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.026 DEBUG:boto:Params: {} 2023-10-11T15:13:35.026 DEBUG:boto:Token: None 2023-10-11T15:13:35.027 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:35.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gRMstQEaqOf2HP8GQKuv2BoOnB8= 2023-10-11T15:13:35.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gRMstQEaqOf2HP8GQKuv2BoOnB8='} 2023-10-11T15:13:35.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c08dded4c20c7cec-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.077 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:35.078 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:35.078 DEBUG:boto:Method: HEAD 2023-10-11T15:13:35.079 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:35.079 DEBUG:boto:Data: 2023-10-11T15:13:35.079 DEBUG:boto:Headers: {} 2023-10-11T15:13:35.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.080 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.081 DEBUG:boto:Params: {} 2023-10-11T15:13:35.081 DEBUG:boto:Token: None 2023-10-11T15:13:35.082 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f19f3753d159c442-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.086 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:35.087 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:35.087 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.088 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:35.088 DEBUG:boto:Data: 2023-10-11T15:13:35.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.089 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.090 DEBUG:boto:Params: {} 2023-10-11T15:13:35.090 DEBUG:boto:Token: None 2023-10-11T15:13:35.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:35.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pG/4RswkYu6+6mPcasm0LX+jNn8= 2023-10-11T15:13:35.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pG/4RswkYu6+6mPcasm0LX+jNn8='} 2023-10-11T15:13:35.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f38ebb45f11ac032-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.097 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:35.097 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:35.097 DEBUG:boto:Method: HEAD 2023-10-11T15:13:35.098 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:35.098 DEBUG:boto:Data: 2023-10-11T15:13:35.099 DEBUG:boto:Headers: {} 2023-10-11T15:13:35.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.100 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.100 DEBUG:boto:Params: {} 2023-10-11T15:13:35.100 DEBUG:boto:Token: None 2023-10-11T15:13:35.101 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060fe6b06ba50ef1c-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.105 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:35.105 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:35.106 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.106 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:35.106 DEBUG:boto:Data: 2023-10-11T15:13:35.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.108 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.108 DEBUG:boto:Params: {} 2023-10-11T15:13:35.108 DEBUG:boto:Token: None 2023-10-11T15:13:35.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:35.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Eci11QB6e4cUeLXrOXYcWI8H81Y= 2023-10-11T15:13:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Eci11QB6e4cUeLXrOXYcWI8H81Y='} 2023-10-11T15:13:35.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d86f87ea22cdb7c8-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.115 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:35.116 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:35.116 DEBUG:boto:Method: HEAD 2023-10-11T15:13:35.116 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:35.117 DEBUG:boto:Data: 2023-10-11T15:13:35.117 DEBUG:boto:Headers: {} 2023-10-11T15:13:35.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.118 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.119 DEBUG:boto:Params: {} 2023-10-11T15:13:35.119 DEBUG:boto:Token: None 2023-10-11T15:13:35.120 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000983c441ad847e880-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.124 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:35.124 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:35.124 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.125 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:35.125 DEBUG:boto:Data: 2023-10-11T15:13:35.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.126 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.127 DEBUG:boto:Params: {} 2023-10-11T15:13:35.127 DEBUG:boto:Token: None 2023-10-11T15:13:35.128 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:35.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t/qI6lOWv1pUq7wouAN2oVovWfk= 2023-10-11T15:13:35.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t/qI6lOWv1pUq7wouAN2oVovWfk='} 2023-10-11T15:13:35.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002276c26bbbd1dc38-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.175 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:35.176 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:35.176 DEBUG:boto:Method: HEAD 2023-10-11T15:13:35.176 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:35.177 DEBUG:boto:Data: 2023-10-11T15:13:35.177 DEBUG:boto:Headers: {} 2023-10-11T15:13:35.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.178 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.178 DEBUG:boto:Params: {} 2023-10-11T15:13:35.179 DEBUG:boto:Token: None 2023-10-11T15:13:35.179 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/ 2023-10-11T15:13:35.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY= 2023-10-11T15:13:35.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HQCRzP4hjVBIpKSI6ogXvwDQnMY=', 'Content-Length': '0'} 2023-10-11T15:13:35.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e53f2117923b0c9-006526bb9f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.183 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:35.183 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:35.184 DEBUG:boto:Method: PUT 2023-10-11T15:13:35.184 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:35.185 DEBUG:boto:Data: 2023-10-11T15:13:35.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:35.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:35.186 DEBUG:boto:Port: 8000 2023-10-11T15:13:35.186 DEBUG:boto:Params: {} 2023-10-11T15:13:35.187 DEBUG:boto:Token: None 2023-10-11T15:13:35.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:35 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:35.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MkJzQH54H1nfoqMY2g/OPZst3T8= 2023-10-11T15:13:35.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MkJzQH54H1nfoqMY2g/OPZst3T8='} 2023-10-11T15:13:35.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0398f90e51e7b75-006526bb9f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:35.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-58 2023-10-11T15:13:35.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:35.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:35.195 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:13:35.251 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:35.251 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:35.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:35.283+0000 7f013a4af780 1 waiting to reach incremental sync.. 2023-10-11T15:13:40.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:40.285+0000 7f013a4af780 1 bucket sync caught up with source: 2023-10-11T15:13:40.287 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.34.6, 00000000001.26.6, 00000000001.61.6, 00000000001.74.6, 00000000001.52.6, 00000000001.31.6, 00000000001.29.6, 00000000001.46.6, 00000000001.31.6, 00000000001.44.6, ] 2023-10-11T15:13:40.288 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.34.6, 00000000001.26.6, 00000000001.61.6, 00000000001.74.6, 00000000001.52.6, 00000000001.31.6, 00000000001.29.6, 00000000001.46.6, 00000000001.31.6, 00000000001.44.6, ] 2023-10-11T15:13:40.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:40.285+0000 7f013a4af780 0 bucket checkpoint complete 2023-10-11T15:13:40.304 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-58 2023-10-11T15:13:40.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:40.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:13:40.306 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:13:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:40.370 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:40.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:40.408+0000 7f4bbe0a5780 1 waiting to reach incremental sync.. 2023-10-11T15:13:45.410 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:45.408+0000 7f4bbe0a5780 1 waiting to reach incremental sync.. 2023-10-11T15:13:50.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:50.409+0000 7f4bbe0a5780 1 bucket sync caught up with source: 2023-10-11T15:13:50.412 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.272.6, 00000000001.62.6, 00000000001.185.6, 00000000001.311.6, 00000000001.295.6, 00000000001.142.6, 00000000001.141.6, 00000000001.251.6, 00000000001.233.6, 00000000001.278.6, ] 2023-10-11T15:13:50.412 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.272.6, 00000000001.62.6, 00000000001.185.6, 00000000001.311.6, 00000000001.295.6, 00000000001.142.6, 00000000001.141.6, 00000000001.251.6, 00000000001.233.6, 00000000001.278.6, ] 2023-10-11T15:13:50.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:13:50.409+0000 7f4bbe0a5780 0 bucket checkpoint complete 2023-10-11T15:13:50.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-58 zones={a2, a1} 2023-10-11T15:13:50.430 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.431 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.431 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.432 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:50.432 DEBUG:boto:Data: 2023-10-11T15:13:50.433 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.433 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.434 DEBUG:boto:Params: {} 2023-10-11T15:13:50.434 DEBUG:boto:Token: None 2023-10-11T15:13:50.435 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.435 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vaAhnZGyp3vgH2VoLE7ki0WCtK8= 2023-10-11T15:13:50.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vaAhnZGyp3vgH2VoLE7ki0WCtK8=', 'Content-Length': '0'} 2023-10-11T15:13:50.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b04713d3d3b3f949-006526bbae-1ab5-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.439 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.440 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.440 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.440 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:50.441 DEBUG:boto:Data: 2023-10-11T15:13:50.441 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.442 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.442 DEBUG:boto:Params: {} 2023-10-11T15:13:50.445 DEBUG:boto:Token: None 2023-10-11T15:13:50.445 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.446 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vaAhnZGyp3vgH2VoLE7ki0WCtK8= 2023-10-11T15:13:50.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vaAhnZGyp3vgH2VoLE7ki0WCtK8=', 'Content-Length': '0'} 2023-10-11T15:13:50.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c75a03fd3012fb1a-006526bbae-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.449 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:13:50.450 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.450 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.451 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.451 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.451 DEBUG:boto:Method: GET 2023-10-11T15:13:50.452 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.452 DEBUG:boto:Data: 2023-10-11T15:13:50.453 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.453 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.454 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.454 DEBUG:boto:Params: {} 2023-10-11T15:13:50.454 DEBUG:boto:Token: None 2023-10-11T15:13:50.455 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/?versions 2023-10-11T15:13:50.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY= 2023-10-11T15:13:50.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY=', 'Content-Length': '0'} 2023-10-11T15:13:50.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c25e13c8568b93-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.459 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-11T15:13:50.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-11T15:13:50.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-11T15:13:50.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-11T15:13:50.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-11T15:13:50.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-11T15:13:50.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-11T15:13:50.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-11T15:13:50.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-11T15:13:50.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-11T15:13:50.464 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:13:50.465 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.465 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.466 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.466 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.466 DEBUG:boto:Method: GET 2023-10-11T15:13:50.467 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.467 DEBUG:boto:Data: 2023-10-11T15:13:50.468 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.468 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.469 DEBUG:boto:Params: {} 2023-10-11T15:13:50.469 DEBUG:boto:Token: None 2023-10-11T15:13:50.470 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/?versions 2023-10-11T15:13:50.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY= 2023-10-11T15:13:50.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY=', 'Content-Length': '0'} 2023-10-11T15:13:50.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeb69a0af19fef2c-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.474 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-11T15:13:50.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-11T15:13:50.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-11T15:13:50.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-11T15:13:50.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-11T15:13:50.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-11T15:13:50.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-11T15:13:50.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-11T15:13:50.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-11T15:13:50.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-11T15:13:50.479 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.479 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.480 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.480 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.481 DEBUG:boto:Method: GET 2023-10-11T15:13:50.481 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.482 DEBUG:boto:Data: 2023-10-11T15:13:50.482 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.483 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.483 DEBUG:boto:Params: {} 2023-10-11T15:13:50.484 DEBUG:boto:Token: None 2023-10-11T15:13:50.484 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/?versions 2023-10-11T15:13:50.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY= 2023-10-11T15:13:50.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY=', 'Content-Length': '0'} 2023-10-11T15:13:50.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c1bcdc9fa599186-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.488 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.489 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.490 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.490 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.491 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.491 DEBUG:boto:Method: GET 2023-10-11T15:13:50.491 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:13:50.492 DEBUG:boto:Data: 2023-10-11T15:13:50.492 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.493 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.493 DEBUG:boto:Params: {} 2023-10-11T15:13:50.494 DEBUG:boto:Token: None 2023-10-11T15:13:50.494 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/?versions 2023-10-11T15:13:50.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY= 2023-10-11T15:13:50.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpZM4biHEjtO0PocuW2odiDIOoY=', 'Content-Length': '0'} 2023-10-11T15:13:50.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9c757ceb5a4149-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.499 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:13:50.500 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.501 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.501 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.501 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.502 DEBUG:boto:Method: GET 2023-10-11T15:13:50.502 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.503 DEBUG:boto:Data: 2023-10-11T15:13:50.503 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.504 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.504 DEBUG:boto:Params: {} 2023-10-11T15:13:50.505 DEBUG:boto:Token: None 2023-10-11T15:13:50.505 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo= 2023-10-11T15:13:50.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo=', 'Content-Length': '0'} 2023-10-11T15:13:50.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096211cfbd769ba61-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.550 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.551 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.552 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.552 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.553 DEBUG:boto:Method: GET 2023-10-11T15:13:50.553 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.554 DEBUG:boto:Data: 2023-10-11T15:13:50.554 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.555 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.555 DEBUG:boto:Params: {} 2023-10-11T15:13:50.556 DEBUG:boto:Token: None 2023-10-11T15:13:50.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo= 2023-10-11T15:13:50.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo=', 'Content-Length': '0'} 2023-10-11T15:13:50.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106f97885accf977-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.560 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.560 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.561 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.561 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.562 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.562 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.563 DEBUG:boto:Data: 2023-10-11T15:13:50.563 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.564 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.564 DEBUG:boto:Params: {} 2023-10-11T15:13:50.565 DEBUG:boto:Token: None 2023-10-11T15:13:50.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgbI9nul85FFFwCLlORBSTzmyjE= 2023-10-11T15:13:50.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgbI9nul85FFFwCLlORBSTzmyjE=', 'Content-Length': '0'} 2023-10-11T15:13:50.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070456625f756b463-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.569 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.569 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.570 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.570 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.571 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.571 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.572 DEBUG:boto:Data: 2023-10-11T15:13:50.572 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.573 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.573 DEBUG:boto:Params: {} 2023-10-11T15:13:50.574 DEBUG:boto:Token: None 2023-10-11T15:13:50.574 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.575 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgbI9nul85FFFwCLlORBSTzmyjE= 2023-10-11T15:13:50.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgbI9nul85FFFwCLlORBSTzmyjE=', 'Content-Length': '0'} 2023-10-11T15:13:50.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c47bc91e24513d-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:13:50.578 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.578 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.579 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.579 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.580 DEBUG:boto:Method: GET 2023-10-11T15:13:50.580 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.581 DEBUG:boto:Data: 2023-10-11T15:13:50.581 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.582 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.582 DEBUG:boto:Params: {} 2023-10-11T15:13:50.583 DEBUG:boto:Token: None 2023-10-11T15:13:50.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo= 2023-10-11T15:13:50.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo=', 'Content-Length': '0'} 2023-10-11T15:13:50.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b111c119c4f4e255-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.587 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.588 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.588 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.588 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.589 DEBUG:boto:Method: GET 2023-10-11T15:13:50.589 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.590 DEBUG:boto:Data: 2023-10-11T15:13:50.590 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.591 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.591 DEBUG:boto:Params: {} 2023-10-11T15:13:50.592 DEBUG:boto:Token: None 2023-10-11T15:13:50.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:13:50.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo= 2023-10-11T15:13:50.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YxEvTyhyf7q8EJggoPbj9JeKNRo=', 'Content-Length': '0'} 2023-10-11T15:13:50.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f07f6c0c3923284a-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.596 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.597 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.597 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.598 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.598 DEBUG:boto:Method: GET 2023-10-11T15:13:50.598 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.599 DEBUG:boto:Data: 2023-10-11T15:13:50.599 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.600 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.600 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.601 DEBUG:boto:Params: {} 2023-10-11T15:13:50.601 DEBUG:boto:Token: None 2023-10-11T15:13:50.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000098403394dcf64e2-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.606 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.606 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.607 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.607 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.608 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.608 DEBUG:boto:Method: GET 2023-10-11T15:13:50.609 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.609 DEBUG:boto:Data: 2023-10-11T15:13:50.610 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.611 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.611 DEBUG:boto:Params: {} 2023-10-11T15:13:50.612 DEBUG:boto:Token: None 2023-10-11T15:13:50.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028039d77eaa09907-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.616 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.617 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.617 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.618 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.618 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.619 DEBUG:boto:Data: 2023-10-11T15:13:50.619 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.620 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.620 DEBUG:boto:Params: {} 2023-10-11T15:13:50.621 DEBUG:boto:Token: None 2023-10-11T15:13:50.621 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.622 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yqHqHbuqZAQV03Ppz643tgP+Avk= 2023-10-11T15:13:50.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yqHqHbuqZAQV03Ppz643tgP+Avk=', 'Content-Length': '0'} 2023-10-11T15:13:50.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e438e6959674e3f2-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.625 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.625 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.626 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.626 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.627 DEBUG:boto:Data: 2023-10-11T15:13:50.627 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.628 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.628 DEBUG:boto:Params: {} 2023-10-11T15:13:50.629 DEBUG:boto:Token: None 2023-10-11T15:13:50.629 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yqHqHbuqZAQV03Ppz643tgP+Avk= 2023-10-11T15:13:50.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yqHqHbuqZAQV03Ppz643tgP+Avk=', 'Content-Length': '0'} 2023-10-11T15:13:50.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f946fc33619cb77-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:13:50.633 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.633 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.634 DEBUG:boto:Method: GET 2023-10-11T15:13:50.634 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.635 DEBUG:boto:Data: 2023-10-11T15:13:50.635 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.635 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.636 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.636 DEBUG:boto:Params: {} 2023-10-11T15:13:50.637 DEBUG:boto:Token: None 2023-10-11T15:13:50.637 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.638 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xU1ZhgiibjTpQ/9ogrraHHsSQqE= 2023-10-11T15:13:50.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xU1ZhgiibjTpQ/9ogrraHHsSQqE=', 'Content-Length': '0'} 2023-10-11T15:13:50.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848787492969c995-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.641 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.641 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.642 DEBUG:boto:Method: GET 2023-10-11T15:13:50.642 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.642 DEBUG:boto:Data: 2023-10-11T15:13:50.643 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.643 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.644 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.644 DEBUG:boto:Params: {} 2023-10-11T15:13:50.645 DEBUG:boto:Token: None 2023-10-11T15:13:50.645 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:13:50.646 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xU1ZhgiibjTpQ/9ogrraHHsSQqE= 2023-10-11T15:13:50.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xU1ZhgiibjTpQ/9ogrraHHsSQqE=', 'Content-Length': '0'} 2023-10-11T15:13:50.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c343aed4140a5e0f-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.690 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.691 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.691 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.692 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.692 DEBUG:boto:Method: GET 2023-10-11T15:13:50.693 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.693 DEBUG:boto:Data: 2023-10-11T15:13:50.694 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.694 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.695 DEBUG:boto:Params: {} 2023-10-11T15:13:50.695 DEBUG:boto:Token: None 2023-10-11T15:13:50.696 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016862ac32aaabfc4-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.700 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.701 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.702 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.702 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.702 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.703 DEBUG:boto:Method: GET 2023-10-11T15:13:50.703 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:13:50.704 DEBUG:boto:Data: 2023-10-11T15:13:50.704 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.705 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.705 DEBUG:boto:Params: {} 2023-10-11T15:13:50.706 DEBUG:boto:Token: None 2023-10-11T15:13:50.706 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1322484e13266fc-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.710 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:13:35.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:13:50.711 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.712 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.712 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.713 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.713 DEBUG:boto:Method: GET 2023-10-11T15:13:50.713 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.714 DEBUG:boto:Data: 2023-10-11T15:13:50.714 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.715 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.715 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.715 DEBUG:boto:Params: {} 2023-10-11T15:13:50.716 DEBUG:boto:Token: None 2023-10-11T15:13:50.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.717 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM= 2023-10-11T15:13:50.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM=', 'Content-Length': '0'} 2023-10-11T15:13:50.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0230f38bb2d937-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.762 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.763 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.763 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.764 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.764 DEBUG:boto:Method: GET 2023-10-11T15:13:50.765 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.765 DEBUG:boto:Data: 2023-10-11T15:13:50.766 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.767 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.767 DEBUG:boto:Params: {} 2023-10-11T15:13:50.768 DEBUG:boto:Token: None 2023-10-11T15:13:50.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM= 2023-10-11T15:13:50.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM=', 'Content-Length': '0'} 2023-10-11T15:13:50.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76578b7101de6ea-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.772 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.772 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.772 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.773 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.773 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.774 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.774 DEBUG:boto:Data: 2023-10-11T15:13:50.775 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.775 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.776 DEBUG:boto:Params: {} 2023-10-11T15:13:50.776 DEBUG:boto:Token: None 2023-10-11T15:13:50.777 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Bm1JXBi63uDY+YN3yy1F/IqdGA= 2023-10-11T15:13:50.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Bm1JXBi63uDY+YN3yy1F/IqdGA=', 'Content-Length': '0'} 2023-10-11T15:13:50.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c757c7c58d294c-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.780 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.781 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.781 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.781 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.782 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.782 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.783 DEBUG:boto:Data: 2023-10-11T15:13:50.783 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.784 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.784 DEBUG:boto:Params: {} 2023-10-11T15:13:50.785 DEBUG:boto:Token: None 2023-10-11T15:13:50.785 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Bm1JXBi63uDY+YN3yy1F/IqdGA= 2023-10-11T15:13:50.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Bm1JXBi63uDY+YN3yy1F/IqdGA=', 'Content-Length': '0'} 2023-10-11T15:13:50.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056663fac4c250c95-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:13:50.789 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.790 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.790 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.791 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.791 DEBUG:boto:Method: GET 2023-10-11T15:13:50.791 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.792 DEBUG:boto:Data: 2023-10-11T15:13:50.792 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.793 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.794 DEBUG:boto:Params: {} 2023-10-11T15:13:50.794 DEBUG:boto:Token: None 2023-10-11T15:13:50.795 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM= 2023-10-11T15:13:50.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM=', 'Content-Length': '0'} 2023-10-11T15:13:50.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7a94ebc1a6eec0-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.798 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.798 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.799 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.799 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.800 DEBUG:boto:Method: GET 2023-10-11T15:13:50.800 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.800 DEBUG:boto:Data: 2023-10-11T15:13:50.801 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.802 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.802 DEBUG:boto:Params: {} 2023-10-11T15:13:50.803 DEBUG:boto:Token: None 2023-10-11T15:13:50.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:13:50.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM= 2023-10-11T15:13:50.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P8OZPAAMJkE+Vs3gNNa8MWalcbM=', 'Content-Length': '0'} 2023-10-11T15:13:50.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9216f937fecf3a-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.807 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.807 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.807 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.808 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.808 DEBUG:boto:Method: GET 2023-10-11T15:13:50.809 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.809 DEBUG:boto:Data: 2023-10-11T15:13:50.809 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.810 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.811 DEBUG:boto:Params: {} 2023-10-11T15:13:50.811 DEBUG:boto:Token: None 2023-10-11T15:13:50.812 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198dd54cf7266b1e-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.815 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.816 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.817 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.817 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.818 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.818 DEBUG:boto:Method: GET 2023-10-11T15:13:50.819 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.819 DEBUG:boto:Data: 2023-10-11T15:13:50.819 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.820 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.821 DEBUG:boto:Params: {} 2023-10-11T15:13:50.821 DEBUG:boto:Token: None 2023-10-11T15:13:50.822 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d160bb5105d3ffab-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.826 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.826 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.827 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.827 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.828 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.828 DEBUG:boto:Data: 2023-10-11T15:13:50.828 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.829 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.830 DEBUG:boto:Params: {} 2023-10-11T15:13:50.830 DEBUG:boto:Token: None 2023-10-11T15:13:50.831 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.831 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gB3fNJMMgDHBhfv2ArhhaQckPWI= 2023-10-11T15:13:50.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gB3fNJMMgDHBhfv2ArhhaQckPWI=', 'Content-Length': '0'} 2023-10-11T15:13:50.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4206d2f29416289-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.834 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.834 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.835 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.835 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.836 DEBUG:boto:Data: 2023-10-11T15:13:50.836 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.837 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.837 DEBUG:boto:Params: {} 2023-10-11T15:13:50.838 DEBUG:boto:Token: None 2023-10-11T15:13:50.838 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gB3fNJMMgDHBhfv2ArhhaQckPWI= 2023-10-11T15:13:50.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gB3fNJMMgDHBhfv2ArhhaQckPWI=', 'Content-Length': '0'} 2023-10-11T15:13:50.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bd48c24cf5ff8e-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:13:50.842 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.842 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.843 DEBUG:boto:Method: GET 2023-10-11T15:13:50.843 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.844 DEBUG:boto:Data: 2023-10-11T15:13:50.844 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.845 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.845 DEBUG:boto:Params: {} 2023-10-11T15:13:50.846 DEBUG:boto:Token: None 2023-10-11T15:13:50.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dCO2eeONaJwuuVFO/S5wTO5uZCg= 2023-10-11T15:13:50.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dCO2eeONaJwuuVFO/S5wTO5uZCg=', 'Content-Length': '0'} 2023-10-11T15:13:50.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003612bf5a39575a6c-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.850 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.850 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.851 DEBUG:boto:Method: GET 2023-10-11T15:13:50.851 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.851 DEBUG:boto:Data: 2023-10-11T15:13:50.852 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.852 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.853 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.853 DEBUG:boto:Params: {} 2023-10-11T15:13:50.853 DEBUG:boto:Token: None 2023-10-11T15:13:50.854 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:50.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dCO2eeONaJwuuVFO/S5wTO5uZCg= 2023-10-11T15:13:50.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dCO2eeONaJwuuVFO/S5wTO5uZCg=', 'Content-Length': '0'} 2023-10-11T15:13:50.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3fb51242154e26-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.857 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.858 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.858 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.859 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.859 DEBUG:boto:Method: GET 2023-10-11T15:13:50.859 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.860 DEBUG:boto:Data: 2023-10-11T15:13:50.860 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.861 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.862 DEBUG:boto:Params: {} 2023-10-11T15:13:50.862 DEBUG:boto:Token: None 2023-10-11T15:13:50.862 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d40bcb3a546b1867-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.867 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.867 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.868 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.868 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.869 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.869 DEBUG:boto:Method: GET 2023-10-11T15:13:50.870 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:13:50.870 DEBUG:boto:Data: 2023-10-11T15:13:50.871 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.871 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.872 DEBUG:boto:Params: {} 2023-10-11T15:13:50.872 DEBUG:boto:Token: None 2023-10-11T15:13:50.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a6b5bdbd30be768-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.877 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:13:35.010Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:13:50.878 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.878 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.879 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.879 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.880 DEBUG:boto:Method: GET 2023-10-11T15:13:50.880 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.881 DEBUG:boto:Data: 2023-10-11T15:13:50.881 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.882 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.882 DEBUG:boto:Params: {} 2023-10-11T15:13:50.883 DEBUG:boto:Token: None 2023-10-11T15:13:50.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk= 2023-10-11T15:13:50.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk=', 'Content-Length': '0'} 2023-10-11T15:13:50.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7587708a4b8b0f-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.926 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.927 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.928 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.928 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.929 DEBUG:boto:Method: GET 2023-10-11T15:13:50.929 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.929 DEBUG:boto:Data: 2023-10-11T15:13:50.930 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.931 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.931 DEBUG:boto:Params: {} 2023-10-11T15:13:50.932 DEBUG:boto:Token: None 2023-10-11T15:13:50.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk= 2023-10-11T15:13:50.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk=', 'Content-Length': '0'} 2023-10-11T15:13:50.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d914deee6e44140-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.936 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.937 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.937 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.938 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.938 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.939 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.939 DEBUG:boto:Data: 2023-10-11T15:13:50.940 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.940 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.941 DEBUG:boto:Params: {} 2023-10-11T15:13:50.941 DEBUG:boto:Token: None 2023-10-11T15:13:50.942 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3WkSHYCNZCAvL9hQJavdmly9ec= 2023-10-11T15:13:50.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3WkSHYCNZCAvL9hQJavdmly9ec=', 'Content-Length': '0'} 2023-10-11T15:13:50.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8923e0170041f91-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.945 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.946 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.946 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.947 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.947 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.948 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.948 DEBUG:boto:Data: 2023-10-11T15:13:50.948 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.949 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.950 DEBUG:boto:Params: {} 2023-10-11T15:13:50.950 DEBUG:boto:Token: None 2023-10-11T15:13:50.951 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3WkSHYCNZCAvL9hQJavdmly9ec= 2023-10-11T15:13:50.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3WkSHYCNZCAvL9hQJavdmly9ec=', 'Content-Length': '0'} 2023-10-11T15:13:50.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009989f0bff8144088-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:13:50.955 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.955 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.955 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.956 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.956 DEBUG:boto:Method: GET 2023-10-11T15:13:50.957 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.957 DEBUG:boto:Data: 2023-10-11T15:13:50.958 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.958 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.959 DEBUG:boto:Params: {} 2023-10-11T15:13:50.959 DEBUG:boto:Token: None 2023-10-11T15:13:50.960 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk= 2023-10-11T15:13:50.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk=', 'Content-Length': '0'} 2023-10-11T15:13:50.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c03d8a8b0fc913e-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.963 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.964 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.964 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.965 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.965 DEBUG:boto:Method: GET 2023-10-11T15:13:50.966 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.966 DEBUG:boto:Data: 2023-10-11T15:13:50.966 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.967 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.968 DEBUG:boto:Params: {} 2023-10-11T15:13:50.968 DEBUG:boto:Token: None 2023-10-11T15:13:50.969 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:13:50.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk= 2023-10-11T15:13:50.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yeN5uKL5lbA0kU0nqONbOwFDCMk=', 'Content-Length': '0'} 2023-10-11T15:13:50.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e651a933dd3401c4-006526bbae-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.973 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.973 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.974 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.974 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.974 DEBUG:boto:Method: GET 2023-10-11T15:13:50.975 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.975 DEBUG:boto:Data: 2023-10-11T15:13:50.976 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.977 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.977 DEBUG:boto:Params: {} 2023-10-11T15:13:50.977 DEBUG:boto:Token: None 2023-10-11T15:13:50.978 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe89ed3da8839cf-006526bbae-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.982 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.983 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:50.983 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:50.984 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.984 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.984 DEBUG:boto:Method: GET 2023-10-11T15:13:50.985 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:50.985 DEBUG:boto:Data: 2023-10-11T15:13:50.986 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:50.987 DEBUG:boto:Port: 8000 2023-10-11T15:13:50.987 DEBUG:boto:Params: {} 2023-10-11T15:13:50.988 DEBUG:boto:Token: None 2023-10-11T15:13:50.988 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/ 2023-10-11T15:13:50.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM= 2023-10-11T15:13:50.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vj4WGIAp0DXTXuqubqNwAoZtojM=', 'Content-Length': '0'} 2023-10-11T15:13:50.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a136e66bb28a62-006526bbae-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:50.992 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:50.993 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.993 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.994 DEBUG:boto:Method: HEAD 2023-10-11T15:13:50.994 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.995 DEBUG:boto:Data: 2023-10-11T15:13:50.995 DEBUG:boto:Headers: {} 2023-10-11T15:13:50.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:50.996 DEBUG:boto:Port: 8001 2023-10-11T15:13:50.996 DEBUG:boto:Params: {} 2023-10-11T15:13:50.997 DEBUG:boto:Token: None 2023-10-11T15:13:50.997 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:50 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:50.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V2PWwyNhTFXGylzeE53qOpz5uxg= 2023-10-11T15:13:50.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V2PWwyNhTFXGylzeE53qOpz5uxg=', 'Content-Length': '0'} 2023-10-11T15:13:51.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c70541f938f419-006526bbae-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.001 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.001 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.002 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.002 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.002 DEBUG:boto:Data: 2023-10-11T15:13:51.003 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.004 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.004 DEBUG:boto:Params: {} 2023-10-11T15:13:51.005 DEBUG:boto:Token: None 2023-10-11T15:13:51.005 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x0KeJzf04XoqgeszsHIGYWpLjR8= 2023-10-11T15:13:51.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x0KeJzf04XoqgeszsHIGYWpLjR8=', 'Content-Length': '0'} 2023-10-11T15:13:51.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b87134b13eb9436-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:13:51.009 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.009 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.010 DEBUG:boto:Method: GET 2023-10-11T15:13:51.010 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.011 DEBUG:boto:Data: 2023-10-11T15:13:51.011 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.012 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.012 DEBUG:boto:Params: {} 2023-10-11T15:13:51.013 DEBUG:boto:Token: None 2023-10-11T15:13:51.013 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I/SCh+OUtmZI2cPTsgLtMeYIMLE= 2023-10-11T15:13:51.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I/SCh+OUtmZI2cPTsgLtMeYIMLE=', 'Content-Length': '0'} 2023-10-11T15:13:51.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a25a6b5f35ee19-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.017 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.018 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.018 DEBUG:boto:Method: GET 2023-10-11T15:13:51.019 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.019 DEBUG:boto:Data: 2023-10-11T15:13:51.020 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.021 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.021 DEBUG:boto:Params: {} 2023-10-11T15:13:51.022 DEBUG:boto:Token: None 2023-10-11T15:13:51.022 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:13:51.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I/SCh+OUtmZI2cPTsgLtMeYIMLE= 2023-10-11T15:13:51.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I/SCh+OUtmZI2cPTsgLtMeYIMLE=', 'Content-Length': '0'} 2023-10-11T15:13:51.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1918b186e42d3f9-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.067 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.069 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.069 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.069 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.070 DEBUG:boto:Method: GET 2023-10-11T15:13:51.070 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.071 DEBUG:boto:Data: 2023-10-11T15:13:51.071 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.073 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.073 DEBUG:boto:Params: {} 2023-10-11T15:13:51.074 DEBUG:boto:Token: None 2023-10-11T15:13:51.075 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74150cae53c04d7-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.080 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.081 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.081 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.082 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.082 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.082 DEBUG:boto:Method: GET 2023-10-11T15:13:51.083 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:13:51.083 DEBUG:boto:Data: 2023-10-11T15:13:51.084 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.084 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.085 DEBUG:boto:Params: {} 2023-10-11T15:13:51.085 DEBUG:boto:Token: None 2023-10-11T15:13:51.086 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.086 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c232ce6c24cdae-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.090 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:13:34.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:13:51.091 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.092 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.092 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.092 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.093 DEBUG:boto:Method: GET 2023-10-11T15:13:51.093 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.094 DEBUG:boto:Data: 2023-10-11T15:13:51.094 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.095 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.095 DEBUG:boto:Params: {} 2023-10-11T15:13:51.096 DEBUG:boto:Token: None 2023-10-11T15:13:51.096 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.097 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4= 2023-10-11T15:13:51.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4=', 'Content-Length': '0'} 2023-10-11T15:13:51.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074843a7fd82a28c2-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.146 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.147 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.147 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.148 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.148 DEBUG:boto:Method: GET 2023-10-11T15:13:51.149 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.149 DEBUG:boto:Data: 2023-10-11T15:13:51.150 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.151 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.151 DEBUG:boto:Params: {} 2023-10-11T15:13:51.151 DEBUG:boto:Token: None 2023-10-11T15:13:51.152 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4= 2023-10-11T15:13:51.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4=', 'Content-Length': '0'} 2023-10-11T15:13:51.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e540acd55d26b1db-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.156 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.156 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.156 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.157 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.157 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.158 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.158 DEBUG:boto:Data: 2023-10-11T15:13:51.159 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.159 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.160 DEBUG:boto:Params: {} 2023-10-11T15:13:51.160 DEBUG:boto:Token: None 2023-10-11T15:13:51.161 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RlKBrw2ELCoPFnhOLLfcFivsVwk= 2023-10-11T15:13:51.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RlKBrw2ELCoPFnhOLLfcFivsVwk=', 'Content-Length': '0'} 2023-10-11T15:13:51.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5b21de48acf631-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.164 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.164 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.165 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.165 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.165 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.166 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.166 DEBUG:boto:Data: 2023-10-11T15:13:51.167 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.167 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.167 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.168 DEBUG:boto:Params: {} 2023-10-11T15:13:51.168 DEBUG:boto:Token: None 2023-10-11T15:13:51.169 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RlKBrw2ELCoPFnhOLLfcFivsVwk= 2023-10-11T15:13:51.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RlKBrw2ELCoPFnhOLLfcFivsVwk=', 'Content-Length': '0'} 2023-10-11T15:13:51.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d45b2b4009a207b-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:13:51.173 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.173 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.173 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.174 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.174 DEBUG:boto:Method: GET 2023-10-11T15:13:51.175 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.175 DEBUG:boto:Data: 2023-10-11T15:13:51.176 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.176 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.177 DEBUG:boto:Params: {} 2023-10-11T15:13:51.177 DEBUG:boto:Token: None 2023-10-11T15:13:51.178 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.178 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4= 2023-10-11T15:13:51.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4=', 'Content-Length': '0'} 2023-10-11T15:13:51.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a851662a4ed2c69-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.181 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.182 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.182 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.183 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.183 DEBUG:boto:Method: GET 2023-10-11T15:13:51.184 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.184 DEBUG:boto:Data: 2023-10-11T15:13:51.184 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.185 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.186 DEBUG:boto:Params: {} 2023-10-11T15:13:51.186 DEBUG:boto:Token: None 2023-10-11T15:13:51.187 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:13:51.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4= 2023-10-11T15:13:51.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AsO8niKMxkNojGc44HyAY3dQif4=', 'Content-Length': '0'} 2023-10-11T15:13:51.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f26a5e4c8118b64-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.191 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.191 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.191 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.192 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.192 DEBUG:boto:Method: GET 2023-10-11T15:13:51.193 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.193 DEBUG:boto:Data: 2023-10-11T15:13:51.194 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.195 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.195 DEBUG:boto:Params: {} 2023-10-11T15:13:51.195 DEBUG:boto:Token: None 2023-10-11T15:13:51.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093c9b38fef32fac2-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.200 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.201 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.201 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.201 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.202 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.202 DEBUG:boto:Method: GET 2023-10-11T15:13:51.203 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.203 DEBUG:boto:Data: 2023-10-11T15:13:51.203 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.204 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.205 DEBUG:boto:Params: {} 2023-10-11T15:13:51.205 DEBUG:boto:Token: None 2023-10-11T15:13:51.206 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d231ca1a3cef56e-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.210 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.210 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.211 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.211 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.212 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.212 DEBUG:boto:Data: 2023-10-11T15:13:51.212 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.213 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.213 DEBUG:boto:Params: {} 2023-10-11T15:13:51.214 DEBUG:boto:Token: None 2023-10-11T15:13:51.214 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MWQyl5KoNR85Dp/bigzyYNDuhlw= 2023-10-11T15:13:51.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MWQyl5KoNR85Dp/bigzyYNDuhlw=', 'Content-Length': '0'} 2023-10-11T15:13:51.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29aee8f68e02ef7-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.218 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.218 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.218 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.219 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.219 DEBUG:boto:Data: 2023-10-11T15:13:51.220 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.220 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.221 DEBUG:boto:Params: {} 2023-10-11T15:13:51.221 DEBUG:boto:Token: None 2023-10-11T15:13:51.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MWQyl5KoNR85Dp/bigzyYNDuhlw= 2023-10-11T15:13:51.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MWQyl5KoNR85Dp/bigzyYNDuhlw=', 'Content-Length': '0'} 2023-10-11T15:13:51.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b47d466ec743ef69-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:13:51.225 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.225 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.226 DEBUG:boto:Method: GET 2023-10-11T15:13:51.226 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.227 DEBUG:boto:Data: 2023-10-11T15:13:51.227 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.228 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.228 DEBUG:boto:Params: {} 2023-10-11T15:13:51.229 DEBUG:boto:Token: None 2023-10-11T15:13:51.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sDzLBoSekIWorBZ2Srb6MWerxEs= 2023-10-11T15:13:51.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sDzLBoSekIWorBZ2Srb6MWerxEs=', 'Content-Length': '0'} 2023-10-11T15:13:51.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b9fb798772379b-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.233 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.233 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.234 DEBUG:boto:Method: GET 2023-10-11T15:13:51.234 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.235 DEBUG:boto:Data: 2023-10-11T15:13:51.235 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.236 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.236 DEBUG:boto:Params: {} 2023-10-11T15:13:51.237 DEBUG:boto:Token: None 2023-10-11T15:13:51.237 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:13:51.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sDzLBoSekIWorBZ2Srb6MWerxEs= 2023-10-11T15:13:51.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sDzLBoSekIWorBZ2Srb6MWerxEs=', 'Content-Length': '0'} 2023-10-11T15:13:51.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000287cfcdd2986dbb5-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.241 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.242 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.242 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.242 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.243 DEBUG:boto:Method: GET 2023-10-11T15:13:51.243 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.244 DEBUG:boto:Data: 2023-10-11T15:13:51.244 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.245 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.245 DEBUG:boto:Params: {} 2023-10-11T15:13:51.246 DEBUG:boto:Token: None 2023-10-11T15:13:51.246 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005d6ed7eaba92d0-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.250 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.251 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.252 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.252 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.253 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.253 DEBUG:boto:Method: GET 2023-10-11T15:13:51.253 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:13:51.254 DEBUG:boto:Data: 2023-10-11T15:13:51.254 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.255 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.256 DEBUG:boto:Params: {} 2023-10-11T15:13:51.256 DEBUG:boto:Token: None 2023-10-11T15:13:51.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0ff968f53f9234-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.261 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:13:35.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:13:51.262 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.262 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.263 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.263 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.264 DEBUG:boto:Method: GET 2023-10-11T15:13:51.264 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.264 DEBUG:boto:Data: 2023-10-11T15:13:51.265 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.266 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.266 DEBUG:boto:Params: {} 2023-10-11T15:13:51.267 DEBUG:boto:Token: None 2023-10-11T15:13:51.267 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c= 2023-10-11T15:13:51.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c=', 'Content-Length': '0'} 2023-10-11T15:13:51.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac98be080d67c48a-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.271 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.271 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.272 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.272 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.272 DEBUG:boto:Method: GET 2023-10-11T15:13:51.273 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.273 DEBUG:boto:Data: 2023-10-11T15:13:51.274 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.275 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.275 DEBUG:boto:Params: {} 2023-10-11T15:13:51.276 DEBUG:boto:Token: None 2023-10-11T15:13:51.276 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c= 2023-10-11T15:13:51.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c=', 'Content-Length': '0'} 2023-10-11T15:13:51.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0dfb714fb63018-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.280 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.280 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.280 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.281 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.281 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.282 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.282 DEBUG:boto:Data: 2023-10-11T15:13:51.282 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.283 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.283 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.284 DEBUG:boto:Params: {} 2023-10-11T15:13:51.284 DEBUG:boto:Token: None 2023-10-11T15:13:51.285 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o/zkoCPeEIN2G6zStWRXxF5L2z4= 2023-10-11T15:13:51.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o/zkoCPeEIN2G6zStWRXxF5L2z4=', 'Content-Length': '0'} 2023-10-11T15:13:51.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e6edbe6c6c9e69-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.288 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.289 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.290 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.290 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.291 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.291 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.292 DEBUG:boto:Data: 2023-10-11T15:13:51.292 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.293 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.293 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.293 DEBUG:boto:Params: {} 2023-10-11T15:13:51.294 DEBUG:boto:Token: None 2023-10-11T15:13:51.295 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.295 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o/zkoCPeEIN2G6zStWRXxF5L2z4= 2023-10-11T15:13:51.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o/zkoCPeEIN2G6zStWRXxF5L2z4=', 'Content-Length': '0'} 2023-10-11T15:13:51.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c431f8184b256c-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:13:51.299 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.299 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.300 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.300 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.301 DEBUG:boto:Method: GET 2023-10-11T15:13:51.301 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.301 DEBUG:boto:Data: 2023-10-11T15:13:51.302 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.303 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.303 DEBUG:boto:Params: {} 2023-10-11T15:13:51.304 DEBUG:boto:Token: None 2023-10-11T15:13:51.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c= 2023-10-11T15:13:51.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c=', 'Content-Length': '0'} 2023-10-11T15:13:51.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb7597963c935703-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.308 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.308 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.309 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.309 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.309 DEBUG:boto:Method: GET 2023-10-11T15:13:51.310 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.310 DEBUG:boto:Data: 2023-10-11T15:13:51.311 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.312 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.312 DEBUG:boto:Params: {} 2023-10-11T15:13:51.313 DEBUG:boto:Token: None 2023-10-11T15:13:51.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:13:51.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c= 2023-10-11T15:13:51.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ew4OQvVgWfUvvMYS4NguZgGPt9c=', 'Content-Length': '0'} 2023-10-11T15:13:51.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047ae45c6e61650a5-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.362 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.363 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.363 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.364 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.364 DEBUG:boto:Method: GET 2023-10-11T15:13:51.365 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.365 DEBUG:boto:Data: 2023-10-11T15:13:51.366 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.366 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.367 DEBUG:boto:Params: {} 2023-10-11T15:13:51.367 DEBUG:boto:Token: None 2023-10-11T15:13:51.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad28c06fe8572c7-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.372 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.373 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.373 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.374 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.374 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.374 DEBUG:boto:Method: GET 2023-10-11T15:13:51.375 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.375 DEBUG:boto:Data: 2023-10-11T15:13:51.376 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.377 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.377 DEBUG:boto:Params: {} 2023-10-11T15:13:51.378 DEBUG:boto:Token: None 2023-10-11T15:13:51.378 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005585313660b07b92-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.382 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.383 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.383 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.384 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.384 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.385 DEBUG:boto:Data: 2023-10-11T15:13:51.385 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.386 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.386 DEBUG:boto:Params: {} 2023-10-11T15:13:51.387 DEBUG:boto:Token: None 2023-10-11T15:13:51.387 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uFq6dTY5btdmgAxN2m9eaNRn3AM= 2023-10-11T15:13:51.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uFq6dTY5btdmgAxN2m9eaNRn3AM=', 'Content-Length': '0'} 2023-10-11T15:13:51.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004050bd296823f16a-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.391 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.391 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.392 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.392 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.392 DEBUG:boto:Data: 2023-10-11T15:13:51.393 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.394 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.394 DEBUG:boto:Params: {} 2023-10-11T15:13:51.395 DEBUG:boto:Token: None 2023-10-11T15:13:51.395 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uFq6dTY5btdmgAxN2m9eaNRn3AM= 2023-10-11T15:13:51.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uFq6dTY5btdmgAxN2m9eaNRn3AM=', 'Content-Length': '0'} 2023-10-11T15:13:51.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25f939458281885-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:13:51.399 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.399 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.400 DEBUG:boto:Method: GET 2023-10-11T15:13:51.400 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.401 DEBUG:boto:Data: 2023-10-11T15:13:51.401 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.402 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.402 DEBUG:boto:Params: {} 2023-10-11T15:13:51.403 DEBUG:boto:Token: None 2023-10-11T15:13:51.403 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TXr8o5owzswSPQjrAwmBotmPIMY= 2023-10-11T15:13:51.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TXr8o5owzswSPQjrAwmBotmPIMY=', 'Content-Length': '0'} 2023-10-11T15:13:51.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028a95be46afb2a8e-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.454 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.455 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.455 DEBUG:boto:Method: GET 2023-10-11T15:13:51.456 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.456 DEBUG:boto:Data: 2023-10-11T15:13:51.457 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.458 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.458 DEBUG:boto:Params: {} 2023-10-11T15:13:51.459 DEBUG:boto:Token: None 2023-10-11T15:13:51.459 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:51.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TXr8o5owzswSPQjrAwmBotmPIMY= 2023-10-11T15:13:51.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TXr8o5owzswSPQjrAwmBotmPIMY=', 'Content-Length': '0'} 2023-10-11T15:13:51.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000495cf78cc78dc5f5-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.465 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.466 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.467 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.467 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.468 DEBUG:boto:Method: GET 2023-10-11T15:13:51.468 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.468 DEBUG:boto:Data: 2023-10-11T15:13:51.469 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.470 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.470 DEBUG:boto:Params: {} 2023-10-11T15:13:51.471 DEBUG:boto:Token: None 2023-10-11T15:13:51.471 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c38be10d1ab5b2a-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.475 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.476 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.476 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.477 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.477 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.477 DEBUG:boto:Method: GET 2023-10-11T15:13:51.478 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:13:51.478 DEBUG:boto:Data: 2023-10-11T15:13:51.479 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.480 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.480 DEBUG:boto:Params: {} 2023-10-11T15:13:51.481 DEBUG:boto:Token: None 2023-10-11T15:13:51.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.482 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c79c9fd1d911c44b-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.486 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:13:35.092Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:13:51.487 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.488 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.488 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.488 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.489 DEBUG:boto:Method: GET 2023-10-11T15:13:51.489 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.490 DEBUG:boto:Data: 2023-10-11T15:13:51.490 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.491 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.492 DEBUG:boto:Params: {} 2023-10-11T15:13:51.492 DEBUG:boto:Token: None 2023-10-11T15:13:51.492 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k= 2023-10-11T15:13:51.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k=', 'Content-Length': '0'} 2023-10-11T15:13:51.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0127f90dca2d4b5-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.496 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.497 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.497 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.497 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.498 DEBUG:boto:Method: GET 2023-10-11T15:13:51.498 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.499 DEBUG:boto:Data: 2023-10-11T15:13:51.499 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.500 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.501 DEBUG:boto:Params: {} 2023-10-11T15:13:51.501 DEBUG:boto:Token: None 2023-10-11T15:13:51.502 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k= 2023-10-11T15:13:51.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k=', 'Content-Length': '0'} 2023-10-11T15:13:51.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009314a6a1153eeff2-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.505 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.506 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.506 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.507 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.507 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.508 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.508 DEBUG:boto:Data: 2023-10-11T15:13:51.508 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.509 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.510 DEBUG:boto:Params: {} 2023-10-11T15:13:51.510 DEBUG:boto:Token: None 2023-10-11T15:13:51.511 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYYcikqTE7uRshYzGxrsBfsCdeE= 2023-10-11T15:13:51.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYYcikqTE7uRshYzGxrsBfsCdeE=', 'Content-Length': '0'} 2023-10-11T15:13:51.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698fb7390e4ed5ff-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.514 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.514 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.515 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.515 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.515 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.516 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.516 DEBUG:boto:Data: 2023-10-11T15:13:51.517 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.518 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.518 DEBUG:boto:Params: {} 2023-10-11T15:13:51.518 DEBUG:boto:Token: None 2023-10-11T15:13:51.519 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYYcikqTE7uRshYzGxrsBfsCdeE= 2023-10-11T15:13:51.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYYcikqTE7uRshYzGxrsBfsCdeE=', 'Content-Length': '0'} 2023-10-11T15:13:51.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d22bbf592515f8-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:13:51.523 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.523 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.524 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.524 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.525 DEBUG:boto:Method: GET 2023-10-11T15:13:51.525 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.525 DEBUG:boto:Data: 2023-10-11T15:13:51.526 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.527 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.527 DEBUG:boto:Params: {} 2023-10-11T15:13:51.527 DEBUG:boto:Token: None 2023-10-11T15:13:51.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k= 2023-10-11T15:13:51.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k=', 'Content-Length': '0'} 2023-10-11T15:13:51.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d4afb91d6a3041-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.574 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.575 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.576 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.576 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.576 DEBUG:boto:Method: GET 2023-10-11T15:13:51.577 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.577 DEBUG:boto:Data: 2023-10-11T15:13:51.578 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.579 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.579 DEBUG:boto:Params: {} 2023-10-11T15:13:51.580 DEBUG:boto:Token: None 2023-10-11T15:13:51.580 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:13:51.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k= 2023-10-11T15:13:51.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTrtII8bSQTg+d7HB5N/QcwpW9k=', 'Content-Length': '0'} 2023-10-11T15:13:51.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001969feff9620d012-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.584 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.584 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.585 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.585 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.586 DEBUG:boto:Method: GET 2023-10-11T15:13:51.586 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.587 DEBUG:boto:Data: 2023-10-11T15:13:51.587 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.588 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.588 DEBUG:boto:Params: {} 2023-10-11T15:13:51.589 DEBUG:boto:Token: None 2023-10-11T15:13:51.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000863138ccfa4da772-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.594 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.595 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.595 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.596 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.596 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.596 DEBUG:boto:Method: GET 2023-10-11T15:13:51.597 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.597 DEBUG:boto:Data: 2023-10-11T15:13:51.598 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.598 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.599 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.599 DEBUG:boto:Params: {} 2023-10-11T15:13:51.600 DEBUG:boto:Token: None 2023-10-11T15:13:51.600 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003307c60f3a99e69f-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.646 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.647 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.648 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.648 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.649 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.649 DEBUG:boto:Data: 2023-10-11T15:13:51.649 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.650 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.651 DEBUG:boto:Params: {} 2023-10-11T15:13:51.651 DEBUG:boto:Token: None 2023-10-11T15:13:51.652 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eo9rc61aYHLDXeTwcPJvaG+QdoY= 2023-10-11T15:13:51.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eo9rc61aYHLDXeTwcPJvaG+QdoY=', 'Content-Length': '0'} 2023-10-11T15:13:51.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0fe762d400ae8ed-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.655 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.656 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.656 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.656 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.657 DEBUG:boto:Data: 2023-10-11T15:13:51.657 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.658 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.658 DEBUG:boto:Params: {} 2023-10-11T15:13:51.659 DEBUG:boto:Token: None 2023-10-11T15:13:51.659 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eo9rc61aYHLDXeTwcPJvaG+QdoY= 2023-10-11T15:13:51.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eo9rc61aYHLDXeTwcPJvaG+QdoY=', 'Content-Length': '0'} 2023-10-11T15:13:51.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000997723d4d3c8c387-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:13:51.663 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.664 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.664 DEBUG:boto:Method: GET 2023-10-11T15:13:51.665 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.665 DEBUG:boto:Data: 2023-10-11T15:13:51.665 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.666 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.667 DEBUG:boto:Params: {} 2023-10-11T15:13:51.667 DEBUG:boto:Token: None 2023-10-11T15:13:51.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ce8+AvtG87OzalQCD4EDl613Dgs= 2023-10-11T15:13:51.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ce8+AvtG87OzalQCD4EDl613Dgs=', 'Content-Length': '0'} 2023-10-11T15:13:51.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98a96656d092b87-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.673 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.673 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.674 DEBUG:boto:Method: GET 2023-10-11T15:13:51.674 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.675 DEBUG:boto:Data: 2023-10-11T15:13:51.675 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.676 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.677 DEBUG:boto:Params: {} 2023-10-11T15:13:51.677 DEBUG:boto:Token: None 2023-10-11T15:13:51.677 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:51.678 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ce8+AvtG87OzalQCD4EDl613Dgs= 2023-10-11T15:13:51.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ce8+AvtG87OzalQCD4EDl613Dgs=', 'Content-Length': '0'} 2023-10-11T15:13:51.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc9276a5752389e-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.681 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.682 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.682 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.682 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.683 DEBUG:boto:Method: GET 2023-10-11T15:13:51.683 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.684 DEBUG:boto:Data: 2023-10-11T15:13:51.684 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.684 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.685 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.685 DEBUG:boto:Params: {} 2023-10-11T15:13:51.686 DEBUG:boto:Token: None 2023-10-11T15:13:51.686 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d90d8a432badf124-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.734 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.735 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.736 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.736 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.737 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.737 DEBUG:boto:Method: GET 2023-10-11T15:13:51.737 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:13:51.738 DEBUG:boto:Data: 2023-10-11T15:13:51.738 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.739 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.740 DEBUG:boto:Params: {} 2023-10-11T15:13:51.740 DEBUG:boto:Token: None 2023-10-11T15:13:51.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b768b7eb9bccbb-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.745 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:13:34.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:13:51.746 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.746 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.747 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.747 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.748 DEBUG:boto:Method: GET 2023-10-11T15:13:51.748 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.749 DEBUG:boto:Data: 2023-10-11T15:13:51.749 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.750 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.750 DEBUG:boto:Params: {} 2023-10-11T15:13:51.751 DEBUG:boto:Token: None 2023-10-11T15:13:51.751 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE= 2023-10-11T15:13:51.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE=', 'Content-Length': '0'} 2023-10-11T15:13:51.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a97fe79d736e7b-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.755 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.756 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.756 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.756 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.757 DEBUG:boto:Method: GET 2023-10-11T15:13:51.757 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.758 DEBUG:boto:Data: 2023-10-11T15:13:51.758 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.759 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.759 DEBUG:boto:Params: {} 2023-10-11T15:13:51.760 DEBUG:boto:Token: None 2023-10-11T15:13:51.760 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE= 2023-10-11T15:13:51.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE=', 'Content-Length': '0'} 2023-10-11T15:13:51.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba48f4ff2e81d99-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.764 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.765 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.765 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.765 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.766 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.766 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.767 DEBUG:boto:Data: 2023-10-11T15:13:51.767 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.768 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.768 DEBUG:boto:Params: {} 2023-10-11T15:13:51.769 DEBUG:boto:Token: None 2023-10-11T15:13:51.769 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.770 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:espuiABt+aBxebZnlBUbmRgsGj8= 2023-10-11T15:13:51.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:espuiABt+aBxebZnlBUbmRgsGj8=', 'Content-Length': '0'} 2023-10-11T15:13:51.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045a2d9c67684934-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.773 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.773 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.774 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.774 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.775 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.775 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.775 DEBUG:boto:Data: 2023-10-11T15:13:51.776 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.777 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.777 DEBUG:boto:Params: {} 2023-10-11T15:13:51.777 DEBUG:boto:Token: None 2023-10-11T15:13:51.778 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:espuiABt+aBxebZnlBUbmRgsGj8= 2023-10-11T15:13:51.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:espuiABt+aBxebZnlBUbmRgsGj8=', 'Content-Length': '0'} 2023-10-11T15:13:51.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c07cd99f8912f5-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:13:51.782 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.782 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.783 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.783 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.784 DEBUG:boto:Method: GET 2023-10-11T15:13:51.784 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.785 DEBUG:boto:Data: 2023-10-11T15:13:51.785 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.786 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.786 DEBUG:boto:Params: {} 2023-10-11T15:13:51.787 DEBUG:boto:Token: None 2023-10-11T15:13:51.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE= 2023-10-11T15:13:51.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE=', 'Content-Length': '0'} 2023-10-11T15:13:51.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008508f80c0b882b78-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.791 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.792 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.792 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.792 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.793 DEBUG:boto:Method: GET 2023-10-11T15:13:51.793 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.794 DEBUG:boto:Data: 2023-10-11T15:13:51.794 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.795 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.795 DEBUG:boto:Params: {} 2023-10-11T15:13:51.796 DEBUG:boto:Token: None 2023-10-11T15:13:51.796 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:13:51.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE= 2023-10-11T15:13:51.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/3clG53M7OLeAfwDyZCBKZy8dE=', 'Content-Length': '0'} 2023-10-11T15:13:51.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000633a9d66ad7ea127-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.800 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.801 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.801 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.801 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.802 DEBUG:boto:Method: GET 2023-10-11T15:13:51.802 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.803 DEBUG:boto:Data: 2023-10-11T15:13:51.803 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.804 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.804 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.805 DEBUG:boto:Params: {} 2023-10-11T15:13:51.805 DEBUG:boto:Token: None 2023-10-11T15:13:51.806 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f64788fbcbd4df2-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.854 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.855 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.856 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.856 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.857 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.857 DEBUG:boto:Method: GET 2023-10-11T15:13:51.858 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.858 DEBUG:boto:Data: 2023-10-11T15:13:51.859 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.859 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.860 DEBUG:boto:Params: {} 2023-10-11T15:13:51.860 DEBUG:boto:Token: None 2023-10-11T15:13:51.861 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.861 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000eac5b9e70b56b5-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.865 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.865 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.866 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.866 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.867 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.867 DEBUG:boto:Data: 2023-10-11T15:13:51.867 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.868 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.869 DEBUG:boto:Params: {} 2023-10-11T15:13:51.869 DEBUG:boto:Token: None 2023-10-11T15:13:51.870 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BluP+WVGuzssN4adrfFcOv0YMHY= 2023-10-11T15:13:51.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BluP+WVGuzssN4adrfFcOv0YMHY=', 'Content-Length': '0'} 2023-10-11T15:13:51.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066884ed0e3bd2b2e-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.873 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.873 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.874 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.874 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.875 DEBUG:boto:Data: 2023-10-11T15:13:51.875 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.878 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.879 DEBUG:boto:Params: {} 2023-10-11T15:13:51.879 DEBUG:boto:Token: None 2023-10-11T15:13:51.880 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BluP+WVGuzssN4adrfFcOv0YMHY= 2023-10-11T15:13:51.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BluP+WVGuzssN4adrfFcOv0YMHY=', 'Content-Length': '0'} 2023-10-11T15:13:51.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef753eccb70a945e-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:13:51.885 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.885 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.886 DEBUG:boto:Method: GET 2023-10-11T15:13:51.886 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.886 DEBUG:boto:Data: 2023-10-11T15:13:51.887 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.888 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.888 DEBUG:boto:Params: {} 2023-10-11T15:13:51.889 DEBUG:boto:Token: None 2023-10-11T15:13:51.890 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.890 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/h4HMcqslqgrrJoiMrB4OHM/sDw= 2023-10-11T15:13:51.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/h4HMcqslqgrrJoiMrB4OHM/sDw=', 'Content-Length': '0'} 2023-10-11T15:13:51.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7fbf5e39dd7e6f-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.893 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.894 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.894 DEBUG:boto:Method: GET 2023-10-11T15:13:51.895 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.895 DEBUG:boto:Data: 2023-10-11T15:13:51.896 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.896 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.897 DEBUG:boto:Params: {} 2023-10-11T15:13:51.897 DEBUG:boto:Token: None 2023-10-11T15:13:51.898 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:51.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/h4HMcqslqgrrJoiMrB4OHM/sDw= 2023-10-11T15:13:51.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/h4HMcqslqgrrJoiMrB4OHM/sDw=', 'Content-Length': '0'} 2023-10-11T15:13:51.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69f09aa4bee255c-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.942 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.943 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.944 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.944 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.944 DEBUG:boto:Method: GET 2023-10-11T15:13:51.945 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.945 DEBUG:boto:Data: 2023-10-11T15:13:51.946 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.947 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.947 DEBUG:boto:Params: {} 2023-10-11T15:13:51.948 DEBUG:boto:Token: None 2023-10-11T15:13:51.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef270b381c2dcf7-006526bbaf-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.952 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.953 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:51.954 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:51.954 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.954 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.955 DEBUG:boto:Method: GET 2023-10-11T15:13:51.955 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:13:51.956 DEBUG:boto:Data: 2023-10-11T15:13:51.956 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.957 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.958 DEBUG:boto:Params: {} 2023-10-11T15:13:51.958 DEBUG:boto:Token: None 2023-10-11T15:13:51.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/ 2023-10-11T15:13:51.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ= 2023-10-11T15:13:51.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vEyI2hzO6pSQbV4AW5DyCce7/OQ=', 'Content-Length': '0'} 2023-10-11T15:13:51.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e116637e85663715-006526bbaf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.963 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:13:34.929Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:51.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:13:51.964 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.965 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.965 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.966 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.966 DEBUG:boto:Method: GET 2023-10-11T15:13:51.966 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.967 DEBUG:boto:Data: 2023-10-11T15:13:51.967 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.968 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.968 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.969 DEBUG:boto:Params: {} 2023-10-11T15:13:51.969 DEBUG:boto:Token: None 2023-10-11T15:13:51.970 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.970 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFqBCpTgFWmzVPCcpoaFfeyptEo= 2023-10-11T15:13:51.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFqBCpTgFWmzVPCcpoaFfeyptEo=', 'Content-Length': '0'} 2023-10-11T15:13:51.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006593b1de70b7f640-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.974 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.974 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.975 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.975 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.975 DEBUG:boto:Method: GET 2023-10-11T15:13:51.976 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.976 DEBUG:boto:Data: 2023-10-11T15:13:51.977 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.977 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.978 DEBUG:boto:Params: {} 2023-10-11T15:13:51.978 DEBUG:boto:Token: None 2023-10-11T15:13:51.979 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rFqBCpTgFWmzVPCcpoaFfeyptEo= 2023-10-11T15:13:51.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rFqBCpTgFWmzVPCcpoaFfeyptEo=', 'Content-Length': '0'} 2023-10-11T15:13:51.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9ee395d4644cd6-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.983 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.983 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.984 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.984 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.985 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.985 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.985 DEBUG:boto:Data: 2023-10-11T15:13:51.986 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:51.987 DEBUG:boto:Port: 8001 2023-10-11T15:13:51.987 DEBUG:boto:Params: {} 2023-10-11T15:13:51.987 DEBUG:boto:Token: None 2023-10-11T15:13:51.988 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ku7I8emu9JgcHqy5XWqPbpraVqU= 2023-10-11T15:13:51.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ku7I8emu9JgcHqy5XWqPbpraVqU=', 'Content-Length': '0'} 2023-10-11T15:13:51.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b705c384dadaac1-006526bbaf-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:51.992 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.992 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:51.993 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.993 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.993 DEBUG:boto:Method: HEAD 2023-10-11T15:13:51.994 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.994 DEBUG:boto:Data: 2023-10-11T15:13:51.995 DEBUG:boto:Headers: {} 2023-10-11T15:13:51.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:51.995 DEBUG:boto:Port: 8000 2023-10-11T15:13:51.996 DEBUG:boto:Params: {} 2023-10-11T15:13:51.996 DEBUG:boto:Token: None 2023-10-11T15:13:51.997 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:51 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:51.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ku7I8emu9JgcHqy5XWqPbpraVqU= 2023-10-11T15:13:51.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ku7I8emu9JgcHqy5XWqPbpraVqU=', 'Content-Length': '0'} 2023-10-11T15:13:52.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5ad939ef6f4dc0-006526bbaf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:13:52.001 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.001 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.002 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.002 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.003 DEBUG:boto:Method: GET 2023-10-11T15:13:52.003 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.004 DEBUG:boto:Data: 2023-10-11T15:13:52.004 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.004 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.005 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.005 DEBUG:boto:Params: {} 2023-10-11T15:13:52.006 DEBUG:boto:Token: None 2023-10-11T15:13:52.006 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vu+IEYEK6tzwqLdlVTOC6M50tE= 2023-10-11T15:13:52.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vu+IEYEK6tzwqLdlVTOC6M50tE=', 'Content-Length': '0'} 2023-10-11T15:13:52.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f96e26dd77e2d09-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.050 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.051 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.052 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.052 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.052 DEBUG:boto:Method: GET 2023-10-11T15:13:52.053 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.053 DEBUG:boto:Data: 2023-10-11T15:13:52.054 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.055 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.055 DEBUG:boto:Params: {} 2023-10-11T15:13:52.056 DEBUG:boto:Token: None 2023-10-11T15:13:52.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:13:52.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vu+IEYEK6tzwqLdlVTOC6M50tE= 2023-10-11T15:13:52.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vu+IEYEK6tzwqLdlVTOC6M50tE=', 'Content-Length': '0'} 2023-10-11T15:13:52.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f95fcd6a1bddfab-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.061 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.061 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.062 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.062 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.063 DEBUG:boto:Method: GET 2023-10-11T15:13:52.063 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.063 DEBUG:boto:Data: 2023-10-11T15:13:52.064 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.065 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.065 DEBUG:boto:Params: {} 2023-10-11T15:13:52.066 DEBUG:boto:Token: None 2023-10-11T15:13:52.066 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412ac565396107a2-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.071 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.072 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.072 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.073 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.073 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.074 DEBUG:boto:Method: GET 2023-10-11T15:13:52.074 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.074 DEBUG:boto:Data: 2023-10-11T15:13:52.075 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.076 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.076 DEBUG:boto:Params: {} 2023-10-11T15:13:52.077 DEBUG:boto:Token: None 2023-10-11T15:13:52.078 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008355817bb7fbf274-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.082 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.083 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.083 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.086 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.086 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.086 DEBUG:boto:Data: 2023-10-11T15:13:52.087 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.088 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.088 DEBUG:boto:Params: {} 2023-10-11T15:13:52.089 DEBUG:boto:Token: None 2023-10-11T15:13:52.089 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QXSNBsZS1LbOMOrMAYVXQM8Nibk= 2023-10-11T15:13:52.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QXSNBsZS1LbOMOrMAYVXQM8Nibk=', 'Content-Length': '0'} 2023-10-11T15:13:52.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4dea2d035b7822-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.093 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.093 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.094 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.094 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.095 DEBUG:boto:Data: 2023-10-11T15:13:52.095 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.096 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.096 DEBUG:boto:Params: {} 2023-10-11T15:13:52.097 DEBUG:boto:Token: None 2023-10-11T15:13:52.097 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QXSNBsZS1LbOMOrMAYVXQM8Nibk= 2023-10-11T15:13:52.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QXSNBsZS1LbOMOrMAYVXQM8Nibk=', 'Content-Length': '0'} 2023-10-11T15:13:52.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720ef6e120aa3d46-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:13:52.101 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.102 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.102 DEBUG:boto:Method: GET 2023-10-11T15:13:52.103 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.103 DEBUG:boto:Data: 2023-10-11T15:13:52.103 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.104 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.105 DEBUG:boto:Params: {} 2023-10-11T15:13:52.105 DEBUG:boto:Token: None 2023-10-11T15:13:52.106 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iDvGpeAOBeTbVLUE4AKBOOv6jFw= 2023-10-11T15:13:52.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iDvGpeAOBeTbVLUE4AKBOOv6jFw=', 'Content-Length': '0'} 2023-10-11T15:13:52.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664945bd8fdfd270-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.110 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.110 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.110 DEBUG:boto:Method: GET 2023-10-11T15:13:52.111 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.111 DEBUG:boto:Data: 2023-10-11T15:13:52.112 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.113 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.113 DEBUG:boto:Params: {} 2023-10-11T15:13:52.114 DEBUG:boto:Token: None 2023-10-11T15:13:52.114 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:13:52.115 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iDvGpeAOBeTbVLUE4AKBOOv6jFw= 2023-10-11T15:13:52.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iDvGpeAOBeTbVLUE4AKBOOv6jFw=', 'Content-Length': '0'} 2023-10-11T15:13:52.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d728a20ffcd2d2b-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.118 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.118 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.119 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.119 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.119 DEBUG:boto:Method: GET 2023-10-11T15:13:52.120 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.120 DEBUG:boto:Data: 2023-10-11T15:13:52.121 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.122 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.122 DEBUG:boto:Params: {} 2023-10-11T15:13:52.122 DEBUG:boto:Token: None 2023-10-11T15:13:52.123 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530f74286ae9a900-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.127 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.127 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.128 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.128 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.129 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.129 DEBUG:boto:Method: GET 2023-10-11T15:13:52.130 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:13:52.130 DEBUG:boto:Data: 2023-10-11T15:13:52.131 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.131 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.132 DEBUG:boto:Params: {} 2023-10-11T15:13:52.132 DEBUG:boto:Token: None 2023-10-11T15:13:52.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397ba45662d682c3-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.137 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:13:35.029Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:13:52.138 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.138 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.139 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.139 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.140 DEBUG:boto:Method: GET 2023-10-11T15:13:52.140 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.140 DEBUG:boto:Data: 2023-10-11T15:13:52.141 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.142 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.142 DEBUG:boto:Params: {} 2023-10-11T15:13:52.143 DEBUG:boto:Token: None 2023-10-11T15:13:52.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro= 2023-10-11T15:13:52.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro=', 'Content-Length': '0'} 2023-10-11T15:13:52.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000434210c63faf76d-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.190 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.191 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.191 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.192 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.192 DEBUG:boto:Method: GET 2023-10-11T15:13:52.193 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.193 DEBUG:boto:Data: 2023-10-11T15:13:52.194 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.195 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.195 DEBUG:boto:Params: {} 2023-10-11T15:13:52.196 DEBUG:boto:Token: None 2023-10-11T15:13:52.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro= 2023-10-11T15:13:52.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro=', 'Content-Length': '0'} 2023-10-11T15:13:52.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a39fd1da1710d0-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.200 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.200 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.201 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.201 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.202 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.202 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.202 DEBUG:boto:Data: 2023-10-11T15:13:52.203 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.204 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.204 DEBUG:boto:Params: {} 2023-10-11T15:13:52.205 DEBUG:boto:Token: None 2023-10-11T15:13:52.205 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xvEFM875tg9JsXkq5RV+blhVO2s= 2023-10-11T15:13:52.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xvEFM875tg9JsXkq5RV+blhVO2s=', 'Content-Length': '0'} 2023-10-11T15:13:52.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a42b3fbf963c07e-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.208 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.209 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.209 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.210 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.210 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.210 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.211 DEBUG:boto:Data: 2023-10-11T15:13:52.211 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.212 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.212 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.212 DEBUG:boto:Params: {} 2023-10-11T15:13:52.213 DEBUG:boto:Token: None 2023-10-11T15:13:52.213 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.214 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xvEFM875tg9JsXkq5RV+blhVO2s= 2023-10-11T15:13:52.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xvEFM875tg9JsXkq5RV+blhVO2s=', 'Content-Length': '0'} 2023-10-11T15:13:52.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff29582bea46502-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:13:52.217 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.218 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.218 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.219 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.219 DEBUG:boto:Method: GET 2023-10-11T15:13:52.220 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.220 DEBUG:boto:Data: 2023-10-11T15:13:52.220 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.221 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.222 DEBUG:boto:Params: {} 2023-10-11T15:13:52.222 DEBUG:boto:Token: None 2023-10-11T15:13:52.223 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro= 2023-10-11T15:13:52.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro=', 'Content-Length': '0'} 2023-10-11T15:13:52.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa4ebebea674e82-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.226 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.227 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.227 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.228 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.228 DEBUG:boto:Method: GET 2023-10-11T15:13:52.228 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.229 DEBUG:boto:Data: 2023-10-11T15:13:52.229 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.230 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.231 DEBUG:boto:Params: {} 2023-10-11T15:13:52.231 DEBUG:boto:Token: None 2023-10-11T15:13:52.231 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:13:52.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro= 2023-10-11T15:13:52.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IKyNRXPUAZks/HQpLTTP4rNYCro=', 'Content-Length': '0'} 2023-10-11T15:13:52.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004decb5eda78675ad-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.278 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.279 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.279 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.280 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.280 DEBUG:boto:Method: GET 2023-10-11T15:13:52.281 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.281 DEBUG:boto:Data: 2023-10-11T15:13:52.282 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.282 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.283 DEBUG:boto:Params: {} 2023-10-11T15:13:52.283 DEBUG:boto:Token: None 2023-10-11T15:13:52.284 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d08a47e30c3282-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.288 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.288 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.289 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.289 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.290 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.290 DEBUG:boto:Method: GET 2023-10-11T15:13:52.291 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.291 DEBUG:boto:Data: 2023-10-11T15:13:52.291 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.292 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.293 DEBUG:boto:Params: {} 2023-10-11T15:13:52.293 DEBUG:boto:Token: None 2023-10-11T15:13:52.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000275560f18f4a45f3-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.298 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.299 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.299 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.299 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.300 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.300 DEBUG:boto:Data: 2023-10-11T15:13:52.301 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.301 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.302 DEBUG:boto:Params: {} 2023-10-11T15:13:52.302 DEBUG:boto:Token: None 2023-10-11T15:13:52.303 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rOmS0qmfJlNNB14EDvqVdykKvuo= 2023-10-11T15:13:52.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rOmS0qmfJlNNB14EDvqVdykKvuo=', 'Content-Length': '0'} 2023-10-11T15:13:52.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252b92770a8be4d2-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.306 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.306 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.307 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.307 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.307 DEBUG:boto:Data: 2023-10-11T15:13:52.308 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.308 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.309 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.309 DEBUG:boto:Params: {} 2023-10-11T15:13:52.310 DEBUG:boto:Token: None 2023-10-11T15:13:52.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rOmS0qmfJlNNB14EDvqVdykKvuo= 2023-10-11T15:13:52.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rOmS0qmfJlNNB14EDvqVdykKvuo=', 'Content-Length': '0'} 2023-10-11T15:13:52.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009787b1c63d345dda-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:13:52.314 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.314 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.315 DEBUG:boto:Method: GET 2023-10-11T15:13:52.315 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.316 DEBUG:boto:Data: 2023-10-11T15:13:52.316 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.317 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.317 DEBUG:boto:Params: {} 2023-10-11T15:13:52.318 DEBUG:boto:Token: None 2023-10-11T15:13:52.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CpB/Ri63lMINZfHnkLeVJ8FeFsM= 2023-10-11T15:13:52.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CpB/Ri63lMINZfHnkLeVJ8FeFsM=', 'Content-Length': '0'} 2023-10-11T15:13:52.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042555a26d9cf0d43-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.322 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.322 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.323 DEBUG:boto:Method: GET 2023-10-11T15:13:52.323 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.324 DEBUG:boto:Data: 2023-10-11T15:13:52.324 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.325 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.325 DEBUG:boto:Params: {} 2023-10-11T15:13:52.326 DEBUG:boto:Token: None 2023-10-11T15:13:52.326 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:13:52.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CpB/Ri63lMINZfHnkLeVJ8FeFsM= 2023-10-11T15:13:52.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CpB/Ri63lMINZfHnkLeVJ8FeFsM=', 'Content-Length': '0'} 2023-10-11T15:13:52.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6973a272a7cd4fe-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.330 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.330 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.331 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.331 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.332 DEBUG:boto:Method: GET 2023-10-11T15:13:52.332 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.333 DEBUG:boto:Data: 2023-10-11T15:13:52.333 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.334 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.334 DEBUG:boto:Params: {} 2023-10-11T15:13:52.335 DEBUG:boto:Token: None 2023-10-11T15:13:52.335 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f2a7da180a73d2-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.340 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.340 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.341 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.341 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.342 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.342 DEBUG:boto:Method: GET 2023-10-11T15:13:52.342 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:13:52.343 DEBUG:boto:Data: 2023-10-11T15:13:52.343 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.344 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.345 DEBUG:boto:Params: {} 2023-10-11T15:13:52.345 DEBUG:boto:Token: None 2023-10-11T15:13:52.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1e79ac3714f088-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.350 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:13:35.189Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:13:52.351 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.352 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.352 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.352 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.353 DEBUG:boto:Method: GET 2023-10-11T15:13:52.353 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.354 DEBUG:boto:Data: 2023-10-11T15:13:52.354 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.355 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.355 DEBUG:boto:Params: {} 2023-10-11T15:13:52.356 DEBUG:boto:Token: None 2023-10-11T15:13:52.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM= 2023-10-11T15:13:52.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM=', 'Content-Length': '0'} 2023-10-11T15:13:52.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f487a56318db4493-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.402 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.403 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.404 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.404 DEBUG:boto:Method: GET 2023-10-11T15:13:52.405 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.405 DEBUG:boto:Data: 2023-10-11T15:13:52.406 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.406 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.407 DEBUG:boto:Params: {} 2023-10-11T15:13:52.407 DEBUG:boto:Token: None 2023-10-11T15:13:52.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM= 2023-10-11T15:13:52.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM=', 'Content-Length': '0'} 2023-10-11T15:13:52.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa38086660819ceb-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.412 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.412 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.413 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.413 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.414 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.414 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.414 DEBUG:boto:Data: 2023-10-11T15:13:52.415 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.416 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.416 DEBUG:boto:Params: {} 2023-10-11T15:13:52.416 DEBUG:boto:Token: None 2023-10-11T15:13:52.417 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7nDjEb1/LNQfuWXoaT2ddwSnZlE= 2023-10-11T15:13:52.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7nDjEb1/LNQfuWXoaT2ddwSnZlE=', 'Content-Length': '0'} 2023-10-11T15:13:52.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062eb83ed9c9858dc-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.420 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.421 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.421 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.422 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.422 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.423 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.423 DEBUG:boto:Data: 2023-10-11T15:13:52.423 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.424 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.425 DEBUG:boto:Params: {} 2023-10-11T15:13:52.425 DEBUG:boto:Token: None 2023-10-11T15:13:52.426 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7nDjEb1/LNQfuWXoaT2ddwSnZlE= 2023-10-11T15:13:52.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7nDjEb1/LNQfuWXoaT2ddwSnZlE=', 'Content-Length': '0'} 2023-10-11T15:13:52.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d2b4c23567bf565-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:13:52.429 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.430 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.430 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.431 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.431 DEBUG:boto:Method: GET 2023-10-11T15:13:52.431 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.432 DEBUG:boto:Data: 2023-10-11T15:13:52.432 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.433 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.434 DEBUG:boto:Params: {} 2023-10-11T15:13:52.434 DEBUG:boto:Token: None 2023-10-11T15:13:52.435 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.435 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM= 2023-10-11T15:13:52.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM=', 'Content-Length': '0'} 2023-10-11T15:13:52.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b924e28f654c66-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.439 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.439 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.440 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.440 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.440 DEBUG:boto:Method: GET 2023-10-11T15:13:52.441 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.441 DEBUG:boto:Data: 2023-10-11T15:13:52.442 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.443 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.443 DEBUG:boto:Params: {} 2023-10-11T15:13:52.444 DEBUG:boto:Token: None 2023-10-11T15:13:52.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:13:52.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM= 2023-10-11T15:13:52.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG3WRokUe5EELXOcGedC8TvYJQM=', 'Content-Length': '0'} 2023-10-11T15:13:52.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8953d3f99a1507-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.448 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.448 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.449 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.449 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.450 DEBUG:boto:Method: GET 2023-10-11T15:13:52.450 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.451 DEBUG:boto:Data: 2023-10-11T15:13:52.451 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.452 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.452 DEBUG:boto:Params: {} 2023-10-11T15:13:52.453 DEBUG:boto:Token: None 2023-10-11T15:13:52.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7437d15228743f3-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.457 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.458 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.458 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.459 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.459 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.460 DEBUG:boto:Method: GET 2023-10-11T15:13:52.460 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.461 DEBUG:boto:Data: 2023-10-11T15:13:52.461 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.462 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.462 DEBUG:boto:Params: {} 2023-10-11T15:13:52.463 DEBUG:boto:Token: None 2023-10-11T15:13:52.463 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb01f87eb0c1463-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.515 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.516 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.516 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.517 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.517 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.517 DEBUG:boto:Data: 2023-10-11T15:13:52.518 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.519 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.519 DEBUG:boto:Params: {} 2023-10-11T15:13:52.519 DEBUG:boto:Token: None 2023-10-11T15:13:52.520 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wwNXFCnOblcrwiSdq55FkBzvAEg= 2023-10-11T15:13:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wwNXFCnOblcrwiSdq55FkBzvAEg=', 'Content-Length': '0'} 2023-10-11T15:13:52.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561e1553499d4391-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.524 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.524 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.525 DEBUG:boto:Method: HEAD 2023-10-11T15:13:52.525 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.525 DEBUG:boto:Data: 2023-10-11T15:13:52.526 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.527 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.527 DEBUG:boto:Params: {} 2023-10-11T15:13:52.528 DEBUG:boto:Token: None 2023-10-11T15:13:52.528 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wwNXFCnOblcrwiSdq55FkBzvAEg= 2023-10-11T15:13:52.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wwNXFCnOblcrwiSdq55FkBzvAEg=', 'Content-Length': '0'} 2023-10-11T15:13:52.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3e0b15f59e322c-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:13:52.532 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.533 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.533 DEBUG:boto:Method: GET 2023-10-11T15:13:52.533 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.534 DEBUG:boto:Data: 2023-10-11T15:13:52.534 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.535 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.536 DEBUG:boto:Params: {} 2023-10-11T15:13:52.536 DEBUG:boto:Token: None 2023-10-11T15:13:52.537 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MP7UNEN+sLhvpoHeGfnP966NmZQ= 2023-10-11T15:13:52.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MP7UNEN+sLhvpoHeGfnP966NmZQ=', 'Content-Length': '0'} 2023-10-11T15:13:52.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c3afe281931f97-006526bbb0-1ab5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.540 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.541 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.541 DEBUG:boto:Method: GET 2023-10-11T15:13:52.541 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.542 DEBUG:boto:Data: 2023-10-11T15:13:52.542 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.543 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.544 DEBUG:boto:Params: {} 2023-10-11T15:13:52.544 DEBUG:boto:Token: None 2023-10-11T15:13:52.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:52.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MP7UNEN+sLhvpoHeGfnP966NmZQ= 2023-10-11T15:13:52.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MP7UNEN+sLhvpoHeGfnP966NmZQ=', 'Content-Length': '0'} 2023-10-11T15:13:52.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e379299ac153d19f-006526bbb0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.548 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.549 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.549 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.550 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.550 DEBUG:boto:Method: GET 2023-10-11T15:13:52.551 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.551 DEBUG:boto:Data: 2023-10-11T15:13:52.551 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:13:52.552 DEBUG:boto:Port: 8001 2023-10-11T15:13:52.553 DEBUG:boto:Params: {} 2023-10-11T15:13:52.553 DEBUG:boto:Token: None 2023-10-11T15:13:52.554 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007183665e8d5faf7e-006526bbb0-1ab5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.558 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.558 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:52.559 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:52.559 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.560 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.560 DEBUG:boto:Method: GET 2023-10-11T15:13:52.560 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:13:52.561 DEBUG:boto:Data: 2023-10-11T15:13:52.561 DEBUG:boto:Headers: {} 2023-10-11T15:13:52.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:52.562 DEBUG:boto:Port: 8000 2023-10-11T15:13:52.562 DEBUG:boto:Params: {} 2023-10-11T15:13:52.563 DEBUG:boto:Token: None 2023-10-11T15:13:52.563 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:13:52 GMT /dghqvd-58/ 2023-10-11T15:13:52.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc= 2023-10-11T15:13:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iYxATUIRJUePQcZWOWha+2fIhhc=', 'Content-Length': '0'} 2023-10-11T15:13:52.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec2489d2f53b25b-006526bbb0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:13:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:52.567 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:13:34.948Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:13:52.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-58 zones={a2, a1} 2023-10-11T15:13:52.568 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:13:52.569 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:13:52.570 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:13:52.567+0000 7f952c969700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 157276) UID: 0 2023-10-11T15:13:52.570 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:13:52.567+0000 7f9530bd27c0 -1 shutting down 2023-10-11T15:13:53.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:13:58.668 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:13:58.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:13:58.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:13:58.669 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:13:58.728 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:58.729 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:58.772 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:13:58.773 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:13:58.777 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 10 2023-10-11T15:13:58.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:58.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:58.813 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:58.866 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:58.866 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:58.906 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:13:58.907 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:13:58.907 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000007", 2023-10-11T15:13:58.907 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000016", 2023-10-11T15:13:58.907 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000024", 2023-10-11T15:13:58.908 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000020", 2023-10-11T15:13:58.908 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000028", 2023-10-11T15:13:58.908 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:13:58.908 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000032", 2023-10-11T15:13:58.909 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:13:58.909 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:13:58.909 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000019", 2023-10-11T15:13:58.909 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000013", 2023-10-11T15:13:58.909 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021", 2023-10-11T15:13:58.910 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000030", 2023-10-11T15:13:58.910 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000026", 2023-10-11T15:13:58.910 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000033", 2023-10-11T15:13:58.910 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:13:58.911 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000076" 2023-10-11T15:13:58.911 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:13:58.911 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:13:58.912 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:58.914 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:58.915 DEBUG:boto:Method: HEAD 2023-10-11T15:13:58.915 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:58.916 DEBUG:boto:Data: 2023-10-11T15:13:58.916 DEBUG:boto:Headers: {} 2023-10-11T15:13:58.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:58.917 DEBUG:boto:Port: 8000 2023-10-11T15:13:58.917 DEBUG:boto:Params: {} 2023-10-11T15:13:58.918 DEBUG:boto:Token: None 2023-10-11T15:13:58.918 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:58 GMT /dghqvd-58/ 2023-10-11T15:13:58.919 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ= 2023-10-11T15:13:58.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ=', 'Content-Length': '0'} 2023-10-11T15:13:58.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2078e68b4125db0-006526bbb6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:58.923 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:13:58.923 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:13:58.924 DEBUG:boto:Method: PUT 2023-10-11T15:13:58.924 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:13:58.924 DEBUG:boto:Data: 2023-10-11T15:13:58.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:58.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:58.926 DEBUG:boto:Port: 8000 2023-10-11T15:13:58.926 DEBUG:boto:Params: {} 2023-10-11T15:13:58.926 DEBUG:boto:Token: None 2023-10-11T15:13:58.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:58 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:13:58.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BkJ2ocJFrpIRjKZABf0JBJBHUGI= 2023-10-11T15:13:58.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BkJ2ocJFrpIRjKZABf0JBJBHUGI='} 2023-10-11T15:13:58.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adc6cc9d762e7d77-006526bbb6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:58.933 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:58.934 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:58.934 DEBUG:boto:Method: HEAD 2023-10-11T15:13:58.935 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:58.935 DEBUG:boto:Data: 2023-10-11T15:13:58.935 DEBUG:boto:Headers: {} 2023-10-11T15:13:58.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:58.936 DEBUG:boto:Port: 8000 2023-10-11T15:13:58.937 DEBUG:boto:Params: {} 2023-10-11T15:13:58.937 DEBUG:boto:Token: None 2023-10-11T15:13:58.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:58 GMT /dghqvd-58/ 2023-10-11T15:13:58.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ= 2023-10-11T15:13:58.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ=', 'Content-Length': '0'} 2023-10-11T15:13:58.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038affbf270cca90a-006526bbb6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:58.941 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:13:58.942 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:13:58.942 DEBUG:boto:Method: PUT 2023-10-11T15:13:58.943 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:13:58.943 DEBUG:boto:Data: 2023-10-11T15:13:58.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:58.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:58.944 DEBUG:boto:Port: 8000 2023-10-11T15:13:58.945 DEBUG:boto:Params: {} 2023-10-11T15:13:58.945 DEBUG:boto:Token: None 2023-10-11T15:13:58.946 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:58 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:13:58.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Fnrw3QZVl0ZYXyDWjWmKuEuOM+g= 2023-10-11T15:13:58.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Fnrw3QZVl0ZYXyDWjWmKuEuOM+g='} 2023-10-11T15:13:58.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007da4c1f7a0e694f-006526bbb6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:58.991 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:58.992 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:58.992 DEBUG:boto:Method: HEAD 2023-10-11T15:13:58.993 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:58.994 DEBUG:boto:Data: 2023-10-11T15:13:58.994 DEBUG:boto:Headers: {} 2023-10-11T15:13:58.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:58.996 DEBUG:boto:Port: 8000 2023-10-11T15:13:58.997 DEBUG:boto:Params: {} 2023-10-11T15:13:58.998 DEBUG:boto:Token: None 2023-10-11T15:13:58.999 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:58 GMT /dghqvd-58/ 2023-10-11T15:13:59.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ= 2023-10-11T15:13:59.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQHkbkrM47ePx16ORGpOks3VVcQ=', 'Content-Length': '0'} 2023-10-11T15:13:59.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a5585aa00205674-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.004 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.004 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.005 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.005 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.006 DEBUG:boto:Data: 2023-10-11T15:13:59.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.007 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.007 DEBUG:boto:Params: {} 2023-10-11T15:13:59.008 DEBUG:boto:Token: None 2023-10-11T15:13:59.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xp/grqE1B59g3PzmCiSrp1ST9/k= 2023-10-11T15:13:59.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xp/grqE1B59g3PzmCiSrp1ST9/k='} 2023-10-11T15:13:59.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9e413d95218832d-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.015 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.015 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.016 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.016 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.017 DEBUG:boto:Data: 2023-10-11T15:13:59.017 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.018 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.019 DEBUG:boto:Params: {} 2023-10-11T15:13:59.019 DEBUG:boto:Token: None 2023-10-11T15:13:59.020 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083be716c8919a18a-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.024 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.025 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.025 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.026 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.027 DEBUG:boto:Data: 2023-10-11T15:13:59.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.028 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.029 DEBUG:boto:Params: {} 2023-10-11T15:13:59.029 DEBUG:boto:Token: None 2023-10-11T15:13:59.030 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Iz1klLS+xQ4PJ/tl8itkJwC9538= 2023-10-11T15:13:59.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Iz1klLS+xQ4PJ/tl8itkJwC9538='} 2023-10-11T15:13:59.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3fca3a56ec81636-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.075 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.076 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.076 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.077 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.077 DEBUG:boto:Data: 2023-10-11T15:13:59.077 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.078 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.079 DEBUG:boto:Params: {} 2023-10-11T15:13:59.079 DEBUG:boto:Token: None 2023-10-11T15:13:59.080 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021c1df4ff750e166-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.084 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:13:59.084 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:13:59.085 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.085 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076 2023-10-11T15:13:59.086 DEBUG:boto:Data: 2023-10-11T15:13:59.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.087 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.088 DEBUG:boto:Params: {} 2023-10-11T15:13:59.088 DEBUG:boto:Token: None 2023-10-11T15:13:59.089 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000076 2023-10-11T15:13:59.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R6TgTXsalHZp1AKEoMJebBNis+8= 2023-10-11T15:13:59.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R6TgTXsalHZp1AKEoMJebBNis+8='} 2023-10-11T15:13:59.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4b4aad0cfa56c06-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.095 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.096 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.097 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.098 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.098 DEBUG:boto:Data: 2023-10-11T15:13:59.099 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.100 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.101 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.101 DEBUG:boto:Params: {} 2023-10-11T15:13:59.102 DEBUG:boto:Token: None 2023-10-11T15:13:59.103 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000856c0d2edbfcef56-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.107 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:59.107 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:13:59.108 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.108 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:59.109 DEBUG:boto:Data: 2023-10-11T15:13:59.109 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.110 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.110 DEBUG:boto:Params: {} 2023-10-11T15:13:59.111 DEBUG:boto:Token: None 2023-10-11T15:13:59.111 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:13:59.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xcT10MwCCcBAzIxPGW7XYqxvl+w= 2023-10-11T15:13:59.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xcT10MwCCcBAzIxPGW7XYqxvl+w='} 2023-10-11T15:13:59.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a148bf2758dae8e5-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.155 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.156 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.156 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.156 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.157 DEBUG:boto:Data: 2023-10-11T15:13:59.157 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.158 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.159 DEBUG:boto:Params: {} 2023-10-11T15:13:59.159 DEBUG:boto:Token: None 2023-10-11T15:13:59.160 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b837de8957097a65-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.164 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:13:59.164 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:13:59.165 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.165 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028 2023-10-11T15:13:59.165 DEBUG:boto:Data: 2023-10-11T15:13:59.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.167 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.167 DEBUG:boto:Params: {} 2023-10-11T15:13:59.168 DEBUG:boto:Token: None 2023-10-11T15:13:59.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000028 2023-10-11T15:13:59.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qrFPKRaOh0CHExEt53nf9mcflTk= 2023-10-11T15:13:59.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qrFPKRaOh0CHExEt53nf9mcflTk='} 2023-10-11T15:13:59.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bdc4157c31c5042-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.175 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.175 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.176 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.176 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.176 DEBUG:boto:Data: 2023-10-11T15:13:59.177 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.178 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.178 DEBUG:boto:Params: {} 2023-10-11T15:13:59.179 DEBUG:boto:Token: None 2023-10-11T15:13:59.179 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000842179d653080cee-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.184 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.184 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.185 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.185 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.185 DEBUG:boto:Data: 2023-10-11T15:13:59.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.187 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.187 DEBUG:boto:Params: {} 2023-10-11T15:13:59.188 DEBUG:boto:Token: None 2023-10-11T15:13:59.188 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J4aJ2ttb+ebMYWSv74/plXekd7c= 2023-10-11T15:13:59.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J4aJ2ttb+ebMYWSv74/plXekd7c='} 2023-10-11T15:13:59.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:13:59.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000dd0f908edd7e76c-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.235 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.236 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.236 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.236 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.237 DEBUG:boto:Data: 2023-10-11T15:13:59.237 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.238 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.239 DEBUG:boto:Params: {} 2023-10-11T15:13:59.239 DEBUG:boto:Token: None 2023-10-11T15:13:59.239 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eed712ed18e158c-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.243 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:13:59.244 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:13:59.244 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.245 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:13:59.245 DEBUG:boto:Data: 2023-10-11T15:13:59.245 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.246 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.247 DEBUG:boto:Params: {} 2023-10-11T15:13:59.247 DEBUG:boto:Token: None 2023-10-11T15:13:59.248 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:13:59.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5Auy56iwdF8UxsxEpLz8MxTwLXE= 2023-10-11T15:13:59.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5Auy56iwdF8UxsxEpLz8MxTwLXE='} 2023-10-11T15:13:59.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b6cf846f08cd9c6-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.254 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.255 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.255 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.256 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.256 DEBUG:boto:Data: 2023-10-11T15:13:59.256 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.257 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.258 DEBUG:boto:Params: {} 2023-10-11T15:13:59.259 DEBUG:boto:Token: None 2023-10-11T15:13:59.259 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc2ef567ed2afcdf-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.263 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:13:59.264 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:13:59.264 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.265 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:13:59.265 DEBUG:boto:Data: 2023-10-11T15:13:59.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.267 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.267 DEBUG:boto:Params: {} 2023-10-11T15:13:59.267 DEBUG:boto:Token: None 2023-10-11T15:13:59.268 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:13:59.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l6qWPgU54mjYGI61s5ClFax2eEk= 2023-10-11T15:13:59.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l6qWPgU54mjYGI61s5ClFax2eEk='} 2023-10-11T15:13:59.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc3b1f2faa1068bc-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.315 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.315 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.316 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.316 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.316 DEBUG:boto:Data: 2023-10-11T15:13:59.317 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.318 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.318 DEBUG:boto:Params: {} 2023-10-11T15:13:59.319 DEBUG:boto:Token: None 2023-10-11T15:13:59.319 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066927f5c9d86d7b1-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.323 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:13:59.324 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:13:59.324 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.325 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:13:59.325 DEBUG:boto:Data: 2023-10-11T15:13:59.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.326 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.327 DEBUG:boto:Params: {} 2023-10-11T15:13:59.327 DEBUG:boto:Token: None 2023-10-11T15:13:59.327 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:13:59.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yx0Q7BNcMGx/BfjrsBUKjWm8c5k= 2023-10-11T15:13:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yx0Q7BNcMGx/BfjrsBUKjWm8c5k='} 2023-10-11T15:13:59.333 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000685e6aee7c04a7bd-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.333 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.334 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.334 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.335 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.335 DEBUG:boto:Data: 2023-10-11T15:13:59.335 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.336 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.337 DEBUG:boto:Params: {} 2023-10-11T15:13:59.337 DEBUG:boto:Token: None 2023-10-11T15:13:59.338 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004324f46051bd0f37-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.342 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:59.342 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:13:59.343 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.343 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:59.344 DEBUG:boto:Data: 2023-10-11T15:13:59.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.345 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.345 DEBUG:boto:Params: {} 2023-10-11T15:13:59.346 DEBUG:boto:Token: None 2023-10-11T15:13:59.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:13:59.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TH/5KMB2lRUmj9RcHUv6xjSfpzQ= 2023-10-11T15:13:59.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TH/5KMB2lRUmj9RcHUv6xjSfpzQ='} 2023-10-11T15:13:59.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005b0c13397f57adf-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.391 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.391 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.392 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.392 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.392 DEBUG:boto:Data: 2023-10-11T15:13:59.393 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.393 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.394 DEBUG:boto:Params: {} 2023-10-11T15:13:59.394 DEBUG:boto:Token: None 2023-10-11T15:13:59.395 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb8a655187ddd235-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.399 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:59.399 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:13:59.400 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.400 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:59.400 DEBUG:boto:Data: 2023-10-11T15:13:59.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.402 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.402 DEBUG:boto:Params: {} 2023-10-11T15:13:59.402 DEBUG:boto:Token: None 2023-10-11T15:13:59.403 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:13:59.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dwuwC87QIY2Ued9oLgrGTRLVJTM= 2023-10-11T15:13:59.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dwuwC87QIY2Ued9oLgrGTRLVJTM='} 2023-10-11T15:13:59.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f91e57b309a9c468-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.411 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.411 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.411 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.412 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.412 DEBUG:boto:Data: 2023-10-11T15:13:59.413 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.413 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.413 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.414 DEBUG:boto:Params: {} 2023-10-11T15:13:59.414 DEBUG:boto:Token: None 2023-10-11T15:13:59.415 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000837f102507ffec1e-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.419 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:13:59.419 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:13:59.420 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.420 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:13:59.420 DEBUG:boto:Data: 2023-10-11T15:13:59.421 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.422 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.422 DEBUG:boto:Params: {} 2023-10-11T15:13:59.423 DEBUG:boto:Token: None 2023-10-11T15:13:59.423 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:13:59.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6cz4+y8D0bOABWGgPZgjHT/kdHA= 2023-10-11T15:13:59.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6cz4+y8D0bOABWGgPZgjHT/kdHA='} 2023-10-11T15:13:59.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046b547c53421dd11-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.467 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.468 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.468 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.468 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.469 DEBUG:boto:Data: 2023-10-11T15:13:59.469 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.470 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.470 DEBUG:boto:Params: {} 2023-10-11T15:13:59.471 DEBUG:boto:Token: None 2023-10-11T15:13:59.471 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe83a8ccf1738975-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.475 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:13:59.476 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:13:59.476 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.476 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:13:59.477 DEBUG:boto:Data: 2023-10-11T15:13:59.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.478 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.479 DEBUG:boto:Params: {} 2023-10-11T15:13:59.479 DEBUG:boto:Token: None 2023-10-11T15:13:59.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:13:59.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMS5OfL1SSUs9fcUzPyBnMsJrLw= 2023-10-11T15:13:59.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMS5OfL1SSUs9fcUzPyBnMsJrLw='} 2023-10-11T15:13:59.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6d280ef70d68c09-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.486 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.486 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.486 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.487 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.487 DEBUG:boto:Data: 2023-10-11T15:13:59.488 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.488 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.489 DEBUG:boto:Params: {} 2023-10-11T15:13:59.489 DEBUG:boto:Token: None 2023-10-11T15:13:59.490 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da748838c767fc0b-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.494 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:59.494 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:13:59.495 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.495 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:59.495 DEBUG:boto:Data: 2023-10-11T15:13:59.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.497 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.497 DEBUG:boto:Params: {} 2023-10-11T15:13:59.497 DEBUG:boto:Token: None 2023-10-11T15:13:59.498 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:13:59.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XehboSH1HCSF7FY57uFYp5zhk+Y= 2023-10-11T15:13:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XehboSH1HCSF7FY57uFYp5zhk+Y='} 2023-10-11T15:13:59.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff8ac982e340211f-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:13:59.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:13:59.544 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:13:59.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:59.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:59.639 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:13:59.640 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:13:59.645 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 22 2023-10-11T15:13:59.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:59.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:13:59.689 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:13:59.740 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:13:59.740 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:13:59.779 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:13:59.779 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:13:59.780 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000005", 2023-10-11T15:13:59.780 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:13:59.780 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000016", 2023-10-11T15:13:59.780 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021", 2023-10-11T15:13:59.780 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:13:59.781 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000025", 2023-10-11T15:13:59.781 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000030", 2023-10-11T15:13:59.781 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000015", 2023-10-11T15:13:59.781 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000019", 2023-10-11T15:13:59.782 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:13:59.782 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000009", 2023-10-11T15:13:59.782 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000014", 2023-10-11T15:13:59.782 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:13:59.782 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:13:59.783 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:13:59.783 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006", 2023-10-11T15:13:59.783 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:13:59.783 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:13:59.783 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000007" 2023-10-11T15:13:59.784 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:13:59.784 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:13:59.785 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.785 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.786 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.786 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.787 DEBUG:boto:Data: 2023-10-11T15:13:59.787 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.788 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.788 DEBUG:boto:Params: {} 2023-10-11T15:13:59.789 DEBUG:boto:Token: None 2023-10-11T15:13:59.789 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a56cce9e7f345e5-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.793 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:13:59.794 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:13:59.794 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.795 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:13:59.795 DEBUG:boto:Data: 2023-10-11T15:13:59.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.797 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.797 DEBUG:boto:Params: {} 2023-10-11T15:13:59.797 DEBUG:boto:Token: None 2023-10-11T15:13:59.798 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:13:59.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dgt6zYqLZP9C2APsIDk7ebFtxB4= 2023-10-11T15:13:59.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dgt6zYqLZP9C2APsIDk7ebFtxB4='} 2023-10-11T15:13:59.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa6410b9f57ebfbb-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.806 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.806 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.807 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.807 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.807 DEBUG:boto:Data: 2023-10-11T15:13:59.808 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.809 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.809 DEBUG:boto:Params: {} 2023-10-11T15:13:59.809 DEBUG:boto:Token: None 2023-10-11T15:13:59.810 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018c88aa6c8ebe458-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.814 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.814 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.815 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.815 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.816 DEBUG:boto:Data: 2023-10-11T15:13:59.816 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.817 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.817 DEBUG:boto:Params: {} 2023-10-11T15:13:59.817 DEBUG:boto:Token: None 2023-10-11T15:13:59.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:13:59.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J4aJ2ttb+ebMYWSv74/plXekd7c= 2023-10-11T15:13:59.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J4aJ2ttb+ebMYWSv74/plXekd7c='} 2023-10-11T15:13:59.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e77c8563a249dd1-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.826 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.827 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.827 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.827 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.828 DEBUG:boto:Data: 2023-10-11T15:13:59.828 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.829 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.829 DEBUG:boto:Params: {} 2023-10-11T15:13:59.830 DEBUG:boto:Token: None 2023-10-11T15:13:59.830 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.831 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acf5f68e5c46ef4b-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.834 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:13:59.835 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:13:59.835 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.835 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:13:59.836 DEBUG:boto:Data: 2023-10-11T15:13:59.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.837 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.838 DEBUG:boto:Params: {} 2023-10-11T15:13:59.838 DEBUG:boto:Token: None 2023-10-11T15:13:59.839 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:13:59.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0/Z7HBQ6L3Ey6Viv/YygwqC6I00= 2023-10-11T15:13:59.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0/Z7HBQ6L3Ey6Viv/YygwqC6I00='} 2023-10-11T15:13:59.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e865383c790cf780-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.845 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.845 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.846 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.846 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.847 DEBUG:boto:Data: 2023-10-11T15:13:59.847 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.848 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.848 DEBUG:boto:Params: {} 2023-10-11T15:13:59.849 DEBUG:boto:Token: None 2023-10-11T15:13:59.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b110a594338d5b97-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.853 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.853 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.854 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.854 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.855 DEBUG:boto:Data: 2023-10-11T15:13:59.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.856 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.856 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.856 DEBUG:boto:Params: {} 2023-10-11T15:13:59.857 DEBUG:boto:Token: None 2023-10-11T15:13:59.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:13:59.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Iz1klLS+xQ4PJ/tl8itkJwC9538= 2023-10-11T15:13:59.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Iz1klLS+xQ4PJ/tl8itkJwC9538='} 2023-10-11T15:13:59.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af75ad2b6fa4123d-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.907 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.907 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.908 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.908 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.909 DEBUG:boto:Data: 2023-10-11T15:13:59.909 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.910 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.910 DEBUG:boto:Params: {} 2023-10-11T15:13:59.911 DEBUG:boto:Token: None 2023-10-11T15:13:59.911 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089f07c4c6f216ca5-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.915 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:13:59.915 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:13:59.916 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.916 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:13:59.917 DEBUG:boto:Data: 2023-10-11T15:13:59.917 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.918 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.918 DEBUG:boto:Params: {} 2023-10-11T15:13:59.919 DEBUG:boto:Token: None 2023-10-11T15:13:59.919 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:13:59.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TirewnyK0Eeh7Jjpn3N5YO+WopY= 2023-10-11T15:13:59.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TirewnyK0Eeh7Jjpn3N5YO+WopY='} 2023-10-11T15:13:59.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018a6e942899ccfa8-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.925 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.926 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.926 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.927 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.927 DEBUG:boto:Data: 2023-10-11T15:13:59.927 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.928 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.928 DEBUG:boto:Params: {} 2023-10-11T15:13:59.929 DEBUG:boto:Token: None 2023-10-11T15:13:59.929 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e942a620884b6ee-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.934 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.934 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.934 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.935 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.935 DEBUG:boto:Data: 2023-10-11T15:13:59.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.937 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.937 DEBUG:boto:Params: {} 2023-10-11T15:13:59.937 DEBUG:boto:Token: None 2023-10-11T15:13:59.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:13:59.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xp/grqE1B59g3PzmCiSrp1ST9/k= 2023-10-11T15:13:59.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xp/grqE1B59g3PzmCiSrp1ST9/k='} 2023-10-11T15:13:59.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e061a87cd58aa5a-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.983 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:13:59.984 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:13:59.984 DEBUG:boto:Method: HEAD 2023-10-11T15:13:59.985 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:13:59.985 DEBUG:boto:Data: 2023-10-11T15:13:59.985 DEBUG:boto:Headers: {} 2023-10-11T15:13:59.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.986 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.986 DEBUG:boto:Params: {} 2023-10-11T15:13:59.987 DEBUG:boto:Token: None 2023-10-11T15:13:59.987 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/ 2023-10-11T15:13:59.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk= 2023-10-11T15:13:59.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e/ui6vFrgL86k+e3C/MzMVUAXsk=', 'Content-Length': '0'} 2023-10-11T15:13:59.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077b6fc943e26c377-006526bbb7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:13:59.991 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:59.992 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:13:59.992 DEBUG:boto:Method: PUT 2023-10-11T15:13:59.992 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:59.993 DEBUG:boto:Data: 2023-10-11T15:13:59.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:13:59.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:13:59.994 DEBUG:boto:Port: 8000 2023-10-11T15:13:59.994 DEBUG:boto:Params: {} 2023-10-11T15:13:59.995 DEBUG:boto:Token: None 2023-10-11T15:13:59.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:13:59 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:13:59.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6/2d7E3moHD0upA2uVWhO/dgLmY= 2023-10-11T15:13:59.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:13:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6/2d7E3moHD0upA2uVWhO/dgLmY='} 2023-10-11T15:14:00.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c13ed5250353664-006526bbb7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.003 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.003 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.004 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.004 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.004 DEBUG:boto:Data: 2023-10-11T15:14:00.005 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.006 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.006 DEBUG:boto:Params: {} 2023-10-11T15:14:00.007 DEBUG:boto:Token: None 2023-10-11T15:14:00.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7bffc9e7bc77eb2-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.011 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:00.011 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:00.012 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.012 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:00.013 DEBUG:boto:Data: 2023-10-11T15:14:00.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.014 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.014 DEBUG:boto:Params: {} 2023-10-11T15:14:00.015 DEBUG:boto:Token: None 2023-10-11T15:14:00.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:00.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jNUbG+tTe/sTX0bEoY8863R2KU= 2023-10-11T15:14:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jNUbG+tTe/sTX0bEoY8863R2KU='} 2023-10-11T15:14:00.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a81055f170b625e3-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.059 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.059 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.060 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.060 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.060 DEBUG:boto:Data: 2023-10-11T15:14:00.061 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.061 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.062 DEBUG:boto:Params: {} 2023-10-11T15:14:00.062 DEBUG:boto:Token: None 2023-10-11T15:14:00.063 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8af4e2ff7b1e417-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.067 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.067 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.068 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.068 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.069 DEBUG:boto:Data: 2023-10-11T15:14:00.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.070 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.070 DEBUG:boto:Params: {} 2023-10-11T15:14:00.071 DEBUG:boto:Token: None 2023-10-11T15:14:00.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I4d8mllFVt3+FwUvmzYOtzcOmXk= 2023-10-11T15:14:00.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I4d8mllFVt3+FwUvmzYOtzcOmXk='} 2023-10-11T15:14:00.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff0d0e4568178857-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.079 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.079 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.080 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.080 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.080 DEBUG:boto:Data: 2023-10-11T15:14:00.081 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.082 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.082 DEBUG:boto:Params: {} 2023-10-11T15:14:00.082 DEBUG:boto:Token: None 2023-10-11T15:14:00.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026ed123e10ddab25-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.087 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:00.087 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:00.088 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.088 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:00.088 DEBUG:boto:Data: 2023-10-11T15:14:00.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.090 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.090 DEBUG:boto:Params: {} 2023-10-11T15:14:00.090 DEBUG:boto:Token: None 2023-10-11T15:14:00.091 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:00.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tdrwcfWaJOU73emnkyyDKVl7AZU= 2023-10-11T15:14:00.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tdrwcfWaJOU73emnkyyDKVl7AZU='} 2023-10-11T15:14:00.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e45b3f30b3d1698-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.135 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.136 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.136 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.137 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.137 DEBUG:boto:Data: 2023-10-11T15:14:00.138 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.138 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.139 DEBUG:boto:Params: {} 2023-10-11T15:14:00.139 DEBUG:boto:Token: None 2023-10-11T15:14:00.140 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc28b977bab0d2e1-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.144 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:00.144 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:00.145 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.146 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:00.146 DEBUG:boto:Data: 2023-10-11T15:14:00.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.149 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.149 DEBUG:boto:Params: {} 2023-10-11T15:14:00.150 DEBUG:boto:Token: None 2023-10-11T15:14:00.151 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:00.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ePY/t5JWpVm6sorq032yItZQCiA= 2023-10-11T15:14:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ePY/t5JWpVm6sorq032yItZQCiA='} 2023-10-11T15:14:00.159 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003eb94d98f4467cdc-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.159 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.160 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.160 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.160 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.161 DEBUG:boto:Data: 2023-10-11T15:14:00.161 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.162 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.163 DEBUG:boto:Params: {} 2023-10-11T15:14:00.163 DEBUG:boto:Token: None 2023-10-11T15:14:00.163 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.164 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009274c3a715e1650a-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.168 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:00.168 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:00.169 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.169 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:00.170 DEBUG:boto:Data: 2023-10-11T15:14:00.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.171 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.171 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.171 DEBUG:boto:Params: {} 2023-10-11T15:14:00.172 DEBUG:boto:Token: None 2023-10-11T15:14:00.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:00.173 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+ai00r+8eywIIoNHUxRBEBQi5IA= 2023-10-11T15:14:00.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+ai00r+8eywIIoNHUxRBEBQi5IA='} 2023-10-11T15:14:00.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc219bafb465311a-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.220 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.220 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.220 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.221 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.221 DEBUG:boto:Data: 2023-10-11T15:14:00.222 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.223 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.223 DEBUG:boto:Params: {} 2023-10-11T15:14:00.223 DEBUG:boto:Token: None 2023-10-11T15:14:00.224 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cb0612fecba03ba-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.228 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:00.228 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:00.229 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.229 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:00.230 DEBUG:boto:Data: 2023-10-11T15:14:00.230 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.231 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.232 DEBUG:boto:Params: {} 2023-10-11T15:14:00.232 DEBUG:boto:Token: None 2023-10-11T15:14:00.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:00.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wrXk7MU1Vs07MxgWabSX3Yxz9fs= 2023-10-11T15:14:00.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wrXk7MU1Vs07MxgWabSX3Yxz9fs='} 2023-10-11T15:14:00.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf910f761adaed71-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.242 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.242 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.243 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.243 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.244 DEBUG:boto:Data: 2023-10-11T15:14:00.244 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.245 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.245 DEBUG:boto:Params: {} 2023-10-11T15:14:00.246 DEBUG:boto:Token: None 2023-10-11T15:14:00.246 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ffce9500a81c05c-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.251 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:00.252 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:00.252 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.253 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:00.253 DEBUG:boto:Data: 2023-10-11T15:14:00.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.254 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.255 DEBUG:boto:Params: {} 2023-10-11T15:14:00.255 DEBUG:boto:Token: None 2023-10-11T15:14:00.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:00.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pNuwFLr3LjgUu7NJwL+XZW/IW10= 2023-10-11T15:14:00.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pNuwFLr3LjgUu7NJwL+XZW/IW10='} 2023-10-11T15:14:00.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036d96f109eb047d3-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.303 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.304 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.304 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.304 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.305 DEBUG:boto:Data: 2023-10-11T15:14:00.305 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.306 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.306 DEBUG:boto:Params: {} 2023-10-11T15:14:00.307 DEBUG:boto:Token: None 2023-10-11T15:14:00.307 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c3774000863e697-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.312 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:00.312 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:00.313 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.313 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:00.313 DEBUG:boto:Data: 2023-10-11T15:14:00.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.315 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.315 DEBUG:boto:Params: {} 2023-10-11T15:14:00.315 DEBUG:boto:Token: None 2023-10-11T15:14:00.316 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:00.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:36vzUhKI2/AyDvFYgu2f4njLczo= 2023-10-11T15:14:00.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:36vzUhKI2/AyDvFYgu2f4njLczo='} 2023-10-11T15:14:00.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1415d5c4fae674f-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.323 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.324 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.324 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.325 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.325 DEBUG:boto:Data: 2023-10-11T15:14:00.325 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.326 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.327 DEBUG:boto:Params: {} 2023-10-11T15:14:00.327 DEBUG:boto:Token: None 2023-10-11T15:14:00.327 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.331 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000548b53cf3c2ac5cc-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.332 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.332 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.332 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.333 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.333 DEBUG:boto:Data: 2023-10-11T15:14:00.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.334 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.335 DEBUG:boto:Params: {} 2023-10-11T15:14:00.335 DEBUG:boto:Token: None 2023-10-11T15:14:00.336 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NpPF1FK9mBuBnIFBo/FGvQAvyuA= 2023-10-11T15:14:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NpPF1FK9mBuBnIFBo/FGvQAvyuA='} 2023-10-11T15:14:00.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b0983e80f402608-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.379 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.380 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.380 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.381 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.381 DEBUG:boto:Data: 2023-10-11T15:14:00.382 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.383 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.383 DEBUG:boto:Params: {} 2023-10-11T15:14:00.383 DEBUG:boto:Token: None 2023-10-11T15:14:00.384 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c44dba7c8509056-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.388 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:00.389 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:00.389 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.390 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:00.390 DEBUG:boto:Data: 2023-10-11T15:14:00.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.391 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.391 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.392 DEBUG:boto:Params: {} 2023-10-11T15:14:00.392 DEBUG:boto:Token: None 2023-10-11T15:14:00.393 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:00.393 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q2qFheDAc+PkxihwawLSfTVymQ= 2023-10-11T15:14:00.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q2qFheDAc+PkxihwawLSfTVymQ='} 2023-10-11T15:14:00.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001103177d7fea55e0-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.401 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.401 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.401 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.402 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.402 DEBUG:boto:Data: 2023-10-11T15:14:00.403 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.404 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.404 DEBUG:boto:Params: {} 2023-10-11T15:14:00.405 DEBUG:boto:Token: None 2023-10-11T15:14:00.405 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085240ddfef87f6dc-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.410 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:00.411 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:00.411 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.411 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:00.412 DEBUG:boto:Data: 2023-10-11T15:14:00.412 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.413 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.413 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.413 DEBUG:boto:Params: {} 2023-10-11T15:14:00.414 DEBUG:boto:Token: None 2023-10-11T15:14:00.415 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:00.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eGdlcD6/5CU2FyB2X7RIUAbebY8= 2023-10-11T15:14:00.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGdlcD6/5CU2FyB2X7RIUAbebY8='} 2023-10-11T15:14:00.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003afb88244360c7d9-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:00.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:00.464 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:14:00.517 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:00.518 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:00.562 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:14:00.562 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:14:00.569 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 27 2023-10-11T15:14:00.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:00.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:00.616 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:00.668 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:00.668 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:00.707 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:00.708 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:14:00.709 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000058", 2023-10-11T15:14:00.709 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000026", 2023-10-11T15:14:00.709 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:14:00.710 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000046", 2023-10-11T15:14:00.710 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000014", 2023-10-11T15:14:00.710 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000050", 2023-10-11T15:14:00.711 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000032", 2023-10-11T15:14:00.711 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:14:00.711 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000039", 2023-10-11T15:14:00.712 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000007", 2023-10-11T15:14:00.712 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006", 2023-10-11T15:14:00.712 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000040", 2023-10-11T15:14:00.712 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000009", 2023-10-11T15:14:00.713 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000045", 2023-10-11T15:14:00.713 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000013", 2023-10-11T15:14:00.713 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:14:00.713 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000033", 2023-10-11T15:14:00.714 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:14:00.714 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:14:00.714 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021", 2023-10-11T15:14:00.714 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:14:00.715 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000055", 2023-10-11T15:14:00.715 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000008" 2023-10-11T15:14:00.715 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:00.715 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:00.717 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.717 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.718 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.718 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.719 DEBUG:boto:Data: 2023-10-11T15:14:00.719 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.720 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.720 DEBUG:boto:Params: {} 2023-10-11T15:14:00.721 DEBUG:boto:Token: None 2023-10-11T15:14:00.721 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.722 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e86f7dad3d8d9b8f-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.726 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:00.727 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:00.727 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.728 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:00.728 DEBUG:boto:Data: 2023-10-11T15:14:00.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.729 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.730 DEBUG:boto:Params: {} 2023-10-11T15:14:00.730 DEBUG:boto:Token: None 2023-10-11T15:14:00.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:00.731 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VYqjAq2sH4ujhz5kkw2rzh/hHew= 2023-10-11T15:14:00.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VYqjAq2sH4ujhz5kkw2rzh/hHew='} 2023-10-11T15:14:00.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcc4cdbf862feacb-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.738 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.739 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.739 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.739 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.740 DEBUG:boto:Data: 2023-10-11T15:14:00.740 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.741 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.741 DEBUG:boto:Params: {} 2023-10-11T15:14:00.742 DEBUG:boto:Token: None 2023-10-11T15:14:00.742 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dce421d517457b48-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.748 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:00.748 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:00.749 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.750 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:00.750 DEBUG:boto:Data: 2023-10-11T15:14:00.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.752 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.753 DEBUG:boto:Params: {} 2023-10-11T15:14:00.753 DEBUG:boto:Token: None 2023-10-11T15:14:00.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:00.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0ZL8mfH1Im1Iq4EtR6XiR1iKQ68= 2023-10-11T15:14:00.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0ZL8mfH1Im1Iq4EtR6XiR1iKQ68='} 2023-10-11T15:14:00.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd1a89debe3ccd37-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.762 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.762 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.763 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.763 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.763 DEBUG:boto:Data: 2023-10-11T15:14:00.764 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.765 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.765 DEBUG:boto:Params: {} 2023-10-11T15:14:00.765 DEBUG:boto:Token: None 2023-10-11T15:14:00.766 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ee8dc27b7f88d54-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.771 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:00.771 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:00.772 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.772 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:00.773 DEBUG:boto:Data: 2023-10-11T15:14:00.773 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.774 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.774 DEBUG:boto:Params: {} 2023-10-11T15:14:00.775 DEBUG:boto:Token: None 2023-10-11T15:14:00.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:00.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uoetloDJ9LdByZ30jBrVwbYnIe4= 2023-10-11T15:14:00.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uoetloDJ9LdByZ30jBrVwbYnIe4='} 2023-10-11T15:14:00.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006726d5a4308e5f74-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.819 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.820 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.820 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.821 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.821 DEBUG:boto:Data: 2023-10-11T15:14:00.822 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.822 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.823 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.823 DEBUG:boto:Params: {} 2023-10-11T15:14:00.824 DEBUG:boto:Token: None 2023-10-11T15:14:00.825 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f590b18a06d6f38-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.830 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.831 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.831 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.832 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.832 DEBUG:boto:Data: 2023-10-11T15:14:00.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.833 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.834 DEBUG:boto:Params: {} 2023-10-11T15:14:00.834 DEBUG:boto:Token: None 2023-10-11T15:14:00.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:00.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I4d8mllFVt3+FwUvmzYOtzcOmXk= 2023-10-11T15:14:00.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I4d8mllFVt3+FwUvmzYOtzcOmXk='} 2023-10-11T15:14:00.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000010d0e51fc1a5c6e-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.843 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.844 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.845 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.845 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.845 DEBUG:boto:Data: 2023-10-11T15:14:00.846 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.847 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.847 DEBUG:boto:Params: {} 2023-10-11T15:14:00.848 DEBUG:boto:Token: None 2023-10-11T15:14:00.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006691d015c3a2d82a-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.853 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:00.853 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:00.853 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.854 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:00.854 DEBUG:boto:Data: 2023-10-11T15:14:00.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.856 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.856 DEBUG:boto:Params: {} 2023-10-11T15:14:00.856 DEBUG:boto:Token: None 2023-10-11T15:14:00.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:00.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g/V20lipoupNMjyvb9ZEpGLuw5Q= 2023-10-11T15:14:00.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g/V20lipoupNMjyvb9ZEpGLuw5Q='} 2023-10-11T15:14:00.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f21cff1052eaf6c0-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.903 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.904 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.904 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.905 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.905 DEBUG:boto:Data: 2023-10-11T15:14:00.905 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.906 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.907 DEBUG:boto:Params: {} 2023-10-11T15:14:00.907 DEBUG:boto:Token: None 2023-10-11T15:14:00.908 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3a81a44940bfe7b-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.912 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.913 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.913 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.914 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.914 DEBUG:boto:Data: 2023-10-11T15:14:00.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.915 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.916 DEBUG:boto:Params: {} 2023-10-11T15:14:00.916 DEBUG:boto:Token: None 2023-10-11T15:14:00.917 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:00.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NpPF1FK9mBuBnIFBo/FGvQAvyuA= 2023-10-11T15:14:00.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NpPF1FK9mBuBnIFBo/FGvQAvyuA='} 2023-10-11T15:14:00.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003777a0323f8ee24a-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.925 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.926 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.926 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.927 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.927 DEBUG:boto:Data: 2023-10-11T15:14:00.927 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.929 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.930 DEBUG:boto:Params: {} 2023-10-11T15:14:00.930 DEBUG:boto:Token: None 2023-10-11T15:14:00.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:00.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:00.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:00.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036d15788f84f2420-006526bbb8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.937 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:00.938 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:00.939 DEBUG:boto:Method: PUT 2023-10-11T15:14:00.939 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:00.940 DEBUG:boto:Data: 2023-10-11T15:14:00.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:00.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.942 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.943 DEBUG:boto:Params: {} 2023-10-11T15:14:00.944 DEBUG:boto:Token: None 2023-10-11T15:14:00.944 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:00.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hJj1ndL7PaSXROsunu+s9c/7/rE= 2023-10-11T15:14:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hJj1ndL7PaSXROsunu+s9c/7/rE='} 2023-10-11T15:14:00.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f94156fc7c7da99-006526bbb8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:00.992 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:00.993 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:00.993 DEBUG:boto:Method: HEAD 2023-10-11T15:14:00.994 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:00.995 DEBUG:boto:Data: 2023-10-11T15:14:00.996 DEBUG:boto:Headers: {} 2023-10-11T15:14:00.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:00.997 DEBUG:boto:Port: 8000 2023-10-11T15:14:00.998 DEBUG:boto:Params: {} 2023-10-11T15:14:00.999 DEBUG:boto:Token: None 2023-10-11T15:14:01.000 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:00 GMT /dghqvd-58/ 2023-10-11T15:14:01.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg= 2023-10-11T15:14:01.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tA77dY7u8zw3AwfNwdb2d3NApYg=', 'Content-Length': '0'} 2023-10-11T15:14:01.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009975b95b19e38e41-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.006 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:01.006 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:01.007 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.008 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:01.009 DEBUG:boto:Data: 2023-10-11T15:14:01.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.011 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.012 DEBUG:boto:Params: {} 2023-10-11T15:14:01.012 DEBUG:boto:Token: None 2023-10-11T15:14:01.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:01.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZHjzrAe7iPHralepFtLlvO+t85Y= 2023-10-11T15:14:01.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZHjzrAe7iPHralepFtLlvO+t85Y='} 2023-10-11T15:14:01.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6cf9f43f6868df3-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.021 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.021 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.022 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.023 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.024 DEBUG:boto:Data: 2023-10-11T15:14:01.024 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.026 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.026 DEBUG:boto:Params: {} 2023-10-11T15:14:01.027 DEBUG:boto:Token: None 2023-10-11T15:14:01.028 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afa646d0dc502cc6-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.034 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:01.035 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:01.035 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.036 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:01.037 DEBUG:boto:Data: 2023-10-11T15:14:01.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.039 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.040 DEBUG:boto:Params: {} 2023-10-11T15:14:01.041 DEBUG:boto:Token: None 2023-10-11T15:14:01.041 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:01.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IWiFFWLeuB4sz/mMEABFIAJtzZw= 2023-10-11T15:14:01.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IWiFFWLeuB4sz/mMEABFIAJtzZw='} 2023-10-11T15:14:01.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6d8f190fe56635e-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.048 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.049 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.049 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.050 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.050 DEBUG:boto:Data: 2023-10-11T15:14:01.050 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.051 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.052 DEBUG:boto:Params: {} 2023-10-11T15:14:01.052 DEBUG:boto:Token: None 2023-10-11T15:14:01.052 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3cee247b79bbaf9-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.057 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:01.057 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:01.058 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.058 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:01.059 DEBUG:boto:Data: 2023-10-11T15:14:01.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.060 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.060 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.060 DEBUG:boto:Params: {} 2023-10-11T15:14:01.061 DEBUG:boto:Token: None 2023-10-11T15:14:01.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:01.062 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r4tC4u8n3cTSGfRcR7Tha0Mef90= 2023-10-11T15:14:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r4tC4u8n3cTSGfRcR7Tha0Mef90='} 2023-10-11T15:14:01.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e02a958996d84e51-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.111 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.112 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.112 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.112 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.113 DEBUG:boto:Data: 2023-10-11T15:14:01.113 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.114 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.115 DEBUG:boto:Params: {} 2023-10-11T15:14:01.115 DEBUG:boto:Token: None 2023-10-11T15:14:01.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082cc03f235004ba8-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.122 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:01.122 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:01.123 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.123 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:01.123 DEBUG:boto:Data: 2023-10-11T15:14:01.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.125 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.125 DEBUG:boto:Params: {} 2023-10-11T15:14:01.125 DEBUG:boto:Token: None 2023-10-11T15:14:01.126 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:01.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DVIY8J5SrDeveK88g1nfLGzwnzA= 2023-10-11T15:14:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVIY8J5SrDeveK88g1nfLGzwnzA='} 2023-10-11T15:14:01.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083e6fcd9d42ab635-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.133 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.134 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.134 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.135 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.135 DEBUG:boto:Data: 2023-10-11T15:14:01.136 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.136 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.137 DEBUG:boto:Params: {} 2023-10-11T15:14:01.137 DEBUG:boto:Token: None 2023-10-11T15:14:01.138 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d932df85d12d9e9-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.142 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.142 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.143 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.143 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.143 DEBUG:boto:Data: 2023-10-11T15:14:01.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.145 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.145 DEBUG:boto:Params: {} 2023-10-11T15:14:01.146 DEBUG:boto:Token: None 2023-10-11T15:14:01.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TFe0pc2EvhXpGkeMt31DZ8fJ4Rw= 2023-10-11T15:14:01.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TFe0pc2EvhXpGkeMt31DZ8fJ4Rw='} 2023-10-11T15:14:01.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0724cc3de54ecc6-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.153 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.154 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.154 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.155 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.155 DEBUG:boto:Data: 2023-10-11T15:14:01.155 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.156 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.157 DEBUG:boto:Params: {} 2023-10-11T15:14:01.157 DEBUG:boto:Token: None 2023-10-11T15:14:01.157 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004419aa52d194bfcb-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.163 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:01.163 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:01.164 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.165 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:01.165 DEBUG:boto:Data: 2023-10-11T15:14:01.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.167 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.167 DEBUG:boto:Params: {} 2023-10-11T15:14:01.168 DEBUG:boto:Token: None 2023-10-11T15:14:01.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:01.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TbENt4cmzGqOBtjAOk5waBizVA0= 2023-10-11T15:14:01.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TbENt4cmzGqOBtjAOk5waBizVA0='} 2023-10-11T15:14:01.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f62e334ccda8039-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.220 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.220 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.221 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.222 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.222 DEBUG:boto:Data: 2023-10-11T15:14:01.223 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.223 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.223 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.224 DEBUG:boto:Params: {} 2023-10-11T15:14:01.224 DEBUG:boto:Token: None 2023-10-11T15:14:01.225 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.225 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055da54388a36a07d-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.229 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:01.229 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:01.230 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.230 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:01.231 DEBUG:boto:Data: 2023-10-11T15:14:01.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.232 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.232 DEBUG:boto:Params: {} 2023-10-11T15:14:01.233 DEBUG:boto:Token: None 2023-10-11T15:14:01.233 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:01.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yDO8yoVAvM8T/GIOq0+i5QSxv58= 2023-10-11T15:14:01.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yDO8yoVAvM8T/GIOq0+i5QSxv58='} 2023-10-11T15:14:01.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ad83422452ed1c9-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.240 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.240 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.241 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.242 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.243 DEBUG:boto:Data: 2023-10-11T15:14:01.243 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.245 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.246 DEBUG:boto:Params: {} 2023-10-11T15:14:01.246 DEBUG:boto:Token: None 2023-10-11T15:14:01.247 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000403b88db883f2a6e-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.253 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:01.253 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:01.253 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.254 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:01.254 DEBUG:boto:Data: 2023-10-11T15:14:01.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.255 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.256 DEBUG:boto:Params: {} 2023-10-11T15:14:01.256 DEBUG:boto:Token: None 2023-10-11T15:14:01.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:01.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HDrcWEQDuQMM5KbWwT6h3R2Ucm4= 2023-10-11T15:14:01.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HDrcWEQDuQMM5KbWwT6h3R2Ucm4='} 2023-10-11T15:14:01.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045c1120ef60e337b-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.303 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.304 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.304 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.305 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.305 DEBUG:boto:Data: 2023-10-11T15:14:01.306 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.307 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.307 DEBUG:boto:Params: {} 2023-10-11T15:14:01.307 DEBUG:boto:Token: None 2023-10-11T15:14:01.308 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fabff9feb8788db-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.313 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:01.313 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:01.314 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.315 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:01.315 DEBUG:boto:Data: 2023-10-11T15:14:01.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.317 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.317 DEBUG:boto:Params: {} 2023-10-11T15:14:01.318 DEBUG:boto:Token: None 2023-10-11T15:14:01.318 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:01.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n6UCdlk5gyiOvF6gV0TQMgQ7rhw= 2023-10-11T15:14:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n6UCdlk5gyiOvF6gV0TQMgQ7rhw='} 2023-10-11T15:14:01.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f43a5ac8f556e5c0-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.326 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.326 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.327 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.327 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.328 DEBUG:boto:Data: 2023-10-11T15:14:01.328 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.329 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.329 DEBUG:boto:Params: {} 2023-10-11T15:14:01.330 DEBUG:boto:Token: None 2023-10-11T15:14:01.330 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063239b4260bf5502-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.334 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:01.335 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:01.335 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.336 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:01.336 DEBUG:boto:Data: 2023-10-11T15:14:01.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.337 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.337 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.338 DEBUG:boto:Params: {} 2023-10-11T15:14:01.338 DEBUG:boto:Token: None 2023-10-11T15:14:01.339 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:01.339 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ce7LZtUa7fy0iuPPK1KpPUdZfCI= 2023-10-11T15:14:01.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ce7LZtUa7fy0iuPPK1KpPUdZfCI='} 2023-10-11T15:14:01.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047ffbc24ad34607f-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.383 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.384 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.384 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.384 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.385 DEBUG:boto:Data: 2023-10-11T15:14:01.385 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.386 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.387 DEBUG:boto:Params: {} 2023-10-11T15:14:01.387 DEBUG:boto:Token: None 2023-10-11T15:14:01.388 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccc677a7c09caf73-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.392 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:01.392 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:01.393 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.393 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:01.394 DEBUG:boto:Data: 2023-10-11T15:14:01.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.396 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.397 DEBUG:boto:Params: {} 2023-10-11T15:14:01.397 DEBUG:boto:Token: None 2023-10-11T15:14:01.398 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:01.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jOPc64YfDbJLAEXXpMw6LRLLM/I= 2023-10-11T15:14:01.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jOPc64YfDbJLAEXXpMw6LRLLM/I='} 2023-10-11T15:14:01.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039161e1c51ce0797-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.406 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.406 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.407 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.407 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.407 DEBUG:boto:Data: 2023-10-11T15:14:01.408 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.408 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.409 DEBUG:boto:Params: {} 2023-10-11T15:14:01.409 DEBUG:boto:Token: None 2023-10-11T15:14:01.410 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.410 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006db55ca504d6ef78-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.414 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:01.415 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:01.415 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.416 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:01.416 DEBUG:boto:Data: 2023-10-11T15:14:01.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.417 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.418 DEBUG:boto:Params: {} 2023-10-11T15:14:01.418 DEBUG:boto:Token: None 2023-10-11T15:14:01.419 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:01.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qkq9ZvLIwZFxlVB13Wz3amncbpE= 2023-10-11T15:14:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qkq9ZvLIwZFxlVB13Wz3amncbpE='} 2023-10-11T15:14:01.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000419aaedd1e32ecc5-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.463 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.464 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.464 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.464 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.465 DEBUG:boto:Data: 2023-10-11T15:14:01.465 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.466 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.467 DEBUG:boto:Params: {} 2023-10-11T15:14:01.467 DEBUG:boto:Token: None 2023-10-11T15:14:01.468 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.469 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000740a8a496fd15504-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.473 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:01.473 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:01.474 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.474 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:01.475 DEBUG:boto:Data: 2023-10-11T15:14:01.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.476 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.476 DEBUG:boto:Params: {} 2023-10-11T15:14:01.477 DEBUG:boto:Token: None 2023-10-11T15:14:01.477 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:01.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NCQdiD6/+au8awh762nFIdd0dlk= 2023-10-11T15:14:01.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NCQdiD6/+au8awh762nFIdd0dlk='} 2023-10-11T15:14:01.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fe37547dcc58a3c-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.486 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.487 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.487 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.488 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.488 DEBUG:boto:Data: 2023-10-11T15:14:01.488 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.489 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.490 DEBUG:boto:Params: {} 2023-10-11T15:14:01.490 DEBUG:boto:Token: None 2023-10-11T15:14:01.491 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d05688da812490f-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.496 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:01.496 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:01.497 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.497 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:01.498 DEBUG:boto:Data: 2023-10-11T15:14:01.498 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.499 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.500 DEBUG:boto:Params: {} 2023-10-11T15:14:01.500 DEBUG:boto:Token: None 2023-10-11T15:14:01.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:01.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UIMtVKWFaU9UCTpllZ+hZyldIZM= 2023-10-11T15:14:01.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UIMtVKWFaU9UCTpllZ+hZyldIZM='} 2023-10-11T15:14:01.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e482837ad72b616-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:01.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:01.550 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:14:01.602 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:01.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:01.647 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:14:01.648 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:14:01.656 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 34 2023-10-11T15:14:01.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:01.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:01.709 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:01.760 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:01.760 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:01.800 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:01.800 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:14:01.800 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006", 2023-10-11T15:14:01.800 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000059", 2023-10-11T15:14:01.801 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000027", 2023-10-11T15:14:01.801 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:14:01.801 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:14:01.801 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:14:01.801 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000019", 2023-10-11T15:14:01.802 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000049", 2023-10-11T15:14:01.802 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000017", 2023-10-11T15:14:01.802 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000008", 2023-10-11T15:14:01.802 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000015", 2023-10-11T15:14:01.802 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000031", 2023-10-11T15:14:01.803 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000022", 2023-10-11T15:14:01.803 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000013", 2023-10-11T15:14:01.803 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000020", 2023-10-11T15:14:01.803 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000037", 2023-10-11T15:14:01.804 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000005", 2023-10-11T15:14:01.804 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:14:01.804 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:14:01.804 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000042", 2023-10-11T15:14:01.804 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:14:01.805 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:14:01.805 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000009", 2023-10-11T15:14:01.805 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000025", 2023-10-11T15:14:01.805 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000016", 2023-10-11T15:14:01.805 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:14:01.806 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000014", 2023-10-11T15:14:01.806 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000030", 2023-10-11T15:14:01.806 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021" 2023-10-11T15:14:01.806 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:01.806 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:01.807 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.808 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.808 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.809 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.809 DEBUG:boto:Data: 2023-10-11T15:14:01.810 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.810 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.811 DEBUG:boto:Params: {} 2023-10-11T15:14:01.811 DEBUG:boto:Token: None 2023-10-11T15:14:01.812 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be1c1172510d72ca-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.816 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.817 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.817 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.818 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.818 DEBUG:boto:Data: 2023-10-11T15:14:01.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.819 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.820 DEBUG:boto:Params: {} 2023-10-11T15:14:01.820 DEBUG:boto:Token: None 2023-10-11T15:14:01.821 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:01.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TFe0pc2EvhXpGkeMt31DZ8fJ4Rw= 2023-10-11T15:14:01.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TFe0pc2EvhXpGkeMt31DZ8fJ4Rw='} 2023-10-11T15:14:01.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000210116e5190978f-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.829 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.829 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.829 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.830 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.830 DEBUG:boto:Data: 2023-10-11T15:14:01.830 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.831 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.831 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.832 DEBUG:boto:Params: {} 2023-10-11T15:14:01.832 DEBUG:boto:Token: None 2023-10-11T15:14:01.833 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.833 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002285c8e7f07b301f-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.837 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:01.838 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:01.838 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.838 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:01.839 DEBUG:boto:Data: 2023-10-11T15:14:01.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.840 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.840 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.840 DEBUG:boto:Params: {} 2023-10-11T15:14:01.841 DEBUG:boto:Token: None 2023-10-11T15:14:01.841 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:01.842 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+ymBNAIfT8uxdTMe0xoX21hTxtI= 2023-10-11T15:14:01.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+ymBNAIfT8uxdTMe0xoX21hTxtI='} 2023-10-11T15:14:01.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000168b2ed83d34acdb-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.847 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.848 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.848 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.849 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.849 DEBUG:boto:Data: 2023-10-11T15:14:01.849 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.850 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.850 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.851 DEBUG:boto:Params: {} 2023-10-11T15:14:01.851 DEBUG:boto:Token: None 2023-10-11T15:14:01.851 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d261cb81db6bc1dd-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.856 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:01.856 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:01.857 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.857 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:01.857 DEBUG:boto:Data: 2023-10-11T15:14:01.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.859 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.859 DEBUG:boto:Params: {} 2023-10-11T15:14:01.860 DEBUG:boto:Token: None 2023-10-11T15:14:01.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:01.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4fzb0mlWQLLV9nluqFLGBUpj0t8= 2023-10-11T15:14:01.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4fzb0mlWQLLV9nluqFLGBUpj0t8='} 2023-10-11T15:14:01.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b7a431481ad4c72-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.907 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.908 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.908 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.908 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.909 DEBUG:boto:Data: 2023-10-11T15:14:01.909 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.910 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.910 DEBUG:boto:Params: {} 2023-10-11T15:14:01.911 DEBUG:boto:Token: None 2023-10-11T15:14:01.911 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba3c301aa2068506-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.918 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:01.918 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:01.919 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.919 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:01.919 DEBUG:boto:Data: 2023-10-11T15:14:01.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.921 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.921 DEBUG:boto:Params: {} 2023-10-11T15:14:01.921 DEBUG:boto:Token: None 2023-10-11T15:14:01.922 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:01.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nDU7UKgz4bAgwDi35y8Pjra6wyM= 2023-10-11T15:14:01.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nDU7UKgz4bAgwDi35y8Pjra6wyM='} 2023-10-11T15:14:01.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aaa7858e0dc70148-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.930 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.930 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.931 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.931 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.931 DEBUG:boto:Data: 2023-10-11T15:14:01.932 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.932 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.933 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.933 DEBUG:boto:Params: {} 2023-10-11T15:14:01.933 DEBUG:boto:Token: None 2023-10-11T15:14:01.934 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.934 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008244a45047eec049-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.938 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:01.939 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:01.939 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.939 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:01.940 DEBUG:boto:Data: 2023-10-11T15:14:01.940 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.941 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.941 DEBUG:boto:Params: {} 2023-10-11T15:14:01.942 DEBUG:boto:Token: None 2023-10-11T15:14:01.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:01.943 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ddkf6MklCk8+RslsxyV6h8H2gkg= 2023-10-11T15:14:01.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ddkf6MklCk8+RslsxyV6h8H2gkg='} 2023-10-11T15:14:01.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8f99f4514ac1a0d-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.949 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:01.949 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:01.949 DEBUG:boto:Method: HEAD 2023-10-11T15:14:01.950 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:01.950 DEBUG:boto:Data: 2023-10-11T15:14:01.951 DEBUG:boto:Headers: {} 2023-10-11T15:14:01.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.951 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.952 DEBUG:boto:Params: {} 2023-10-11T15:14:01.952 DEBUG:boto:Token: None 2023-10-11T15:14:01.953 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/ 2023-10-11T15:14:01.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs= 2023-10-11T15:14:01.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROIlBN+o1t8TeMI5LeGOmTDkCJs=', 'Content-Length': '0'} 2023-10-11T15:14:01.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed233b15136e9872-006526bbb9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:01.957 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:01.958 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:01.958 DEBUG:boto:Method: PUT 2023-10-11T15:14:01.959 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:01.959 DEBUG:boto:Data: 2023-10-11T15:14:01.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:01.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:01.960 DEBUG:boto:Port: 8000 2023-10-11T15:14:01.961 DEBUG:boto:Params: {} 2023-10-11T15:14:01.961 DEBUG:boto:Token: None 2023-10-11T15:14:01.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:01 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:01.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t2H6Cjc04L/Z92CF1KVUxViFPuI= 2023-10-11T15:14:01.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t2H6Cjc04L/Z92CF1KVUxViFPuI='} 2023-10-11T15:14:02.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000904bb19371c22eb0-006526bbb9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.011 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.012 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.012 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.012 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.013 DEBUG:boto:Data: 2023-10-11T15:14:02.013 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.014 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.014 DEBUG:boto:Params: {} 2023-10-11T15:14:02.015 DEBUG:boto:Token: None 2023-10-11T15:14:02.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000810f1ecb0b23179a-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.020 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:02.021 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:02.021 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.021 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:02.022 DEBUG:boto:Data: 2023-10-11T15:14:02.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.023 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.023 DEBUG:boto:Params: {} 2023-10-11T15:14:02.024 DEBUG:boto:Token: None 2023-10-11T15:14:02.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:02.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NhhXzR79CTI9Dw+uEPMQe4K9txs= 2023-10-11T15:14:02.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NhhXzR79CTI9Dw+uEPMQe4K9txs='} 2023-10-11T15:14:02.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d7bc48ebb5d12c3-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.032 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.032 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.033 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.033 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.034 DEBUG:boto:Data: 2023-10-11T15:14:02.034 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.035 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.035 DEBUG:boto:Params: {} 2023-10-11T15:14:02.036 DEBUG:boto:Token: None 2023-10-11T15:14:02.036 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.037 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988d660d741ad06c-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.041 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:02.041 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:02.041 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.042 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:02.042 DEBUG:boto:Data: 2023-10-11T15:14:02.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.044 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.044 DEBUG:boto:Params: {} 2023-10-11T15:14:02.044 DEBUG:boto:Token: None 2023-10-11T15:14:02.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:02.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wTTyPY3qIB3ysj4NKutLXBnvwOE= 2023-10-11T15:14:02.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wTTyPY3qIB3ysj4NKutLXBnvwOE='} 2023-10-11T15:14:02.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000012d896da47d0502-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.091 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.092 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.092 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.092 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.093 DEBUG:boto:Data: 2023-10-11T15:14:02.093 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.094 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.095 DEBUG:boto:Params: {} 2023-10-11T15:14:02.095 DEBUG:boto:Token: None 2023-10-11T15:14:02.095 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c70e686be9a7951-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.100 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:02.101 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:02.101 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.101 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:02.102 DEBUG:boto:Data: 2023-10-11T15:14:02.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.103 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.103 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.103 DEBUG:boto:Params: {} 2023-10-11T15:14:02.104 DEBUG:boto:Token: None 2023-10-11T15:14:02.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:02.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5s/OvmDJK4b+diYqtOb49L3X8Z0= 2023-10-11T15:14:02.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5s/OvmDJK4b+diYqtOb49L3X8Z0='} 2023-10-11T15:14:02.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003eea26270135c01a-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.112 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.113 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.113 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.114 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.114 DEBUG:boto:Data: 2023-10-11T15:14:02.114 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.115 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.116 DEBUG:boto:Params: {} 2023-10-11T15:14:02.116 DEBUG:boto:Token: None 2023-10-11T15:14:02.117 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f9132a74ce4ac89-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.121 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:02.121 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:02.122 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.122 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:02.123 DEBUG:boto:Data: 2023-10-11T15:14:02.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.123 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.124 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.124 DEBUG:boto:Params: {} 2023-10-11T15:14:02.125 DEBUG:boto:Token: None 2023-10-11T15:14:02.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:02.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jAaT/d9Dx775EmwbrxhCOZUg9Xs= 2023-10-11T15:14:02.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jAaT/d9Dx775EmwbrxhCOZUg9Xs='} 2023-10-11T15:14:02.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071185f5b29d8dd8d-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.175 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.176 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.176 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.177 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.177 DEBUG:boto:Data: 2023-10-11T15:14:02.178 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.178 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.179 DEBUG:boto:Params: {} 2023-10-11T15:14:02.179 DEBUG:boto:Token: None 2023-10-11T15:14:02.180 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000997e142ce2c744fe-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.185 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:02.186 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:02.186 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.187 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:02.187 DEBUG:boto:Data: 2023-10-11T15:14:02.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.188 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.189 DEBUG:boto:Params: {} 2023-10-11T15:14:02.189 DEBUG:boto:Token: None 2023-10-11T15:14:02.190 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:02.190 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:88PtX+R+1fIA1CkVZao34dlAptI= 2023-10-11T15:14:02.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:88PtX+R+1fIA1CkVZao34dlAptI='} 2023-10-11T15:14:02.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eee550dde6aec15a-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.197 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.198 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.198 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.199 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.199 DEBUG:boto:Data: 2023-10-11T15:14:02.200 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.200 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.201 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.201 DEBUG:boto:Params: {} 2023-10-11T15:14:02.201 DEBUG:boto:Token: None 2023-10-11T15:14:02.202 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.202 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e7577842a1f23c2-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.207 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:02.207 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:02.208 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.208 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:02.209 DEBUG:boto:Data: 2023-10-11T15:14:02.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.210 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.210 DEBUG:boto:Params: {} 2023-10-11T15:14:02.211 DEBUG:boto:Token: None 2023-10-11T15:14:02.211 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:02.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NpeWvTZMrtYDwGqfMf8YP1p7+Gc= 2023-10-11T15:14:02.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NpeWvTZMrtYDwGqfMf8YP1p7+Gc='} 2023-10-11T15:14:02.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015f2f7052ca39041-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.259 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.260 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.260 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.260 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.261 DEBUG:boto:Data: 2023-10-11T15:14:02.261 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.262 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.263 DEBUG:boto:Params: {} 2023-10-11T15:14:02.264 DEBUG:boto:Token: None 2023-10-11T15:14:02.264 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094464485516282b9-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.269 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:02.270 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:02.270 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.271 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:02.271 DEBUG:boto:Data: 2023-10-11T15:14:02.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.272 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.273 DEBUG:boto:Params: {} 2023-10-11T15:14:02.273 DEBUG:boto:Token: None 2023-10-11T15:14:02.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:02.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijz2GesM6R1dSsc5Da6DW8dAOuE= 2023-10-11T15:14:02.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijz2GesM6R1dSsc5Da6DW8dAOuE='} 2023-10-11T15:14:02.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000502f77e5f814b354-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.280 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.280 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.280 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.281 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.281 DEBUG:boto:Data: 2023-10-11T15:14:02.282 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.282 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.283 DEBUG:boto:Params: {} 2023-10-11T15:14:02.283 DEBUG:boto:Token: None 2023-10-11T15:14:02.284 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051d0fef718e372ea-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.289 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:02.290 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:02.290 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.291 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:02.291 DEBUG:boto:Data: 2023-10-11T15:14:02.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.292 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.293 DEBUG:boto:Params: {} 2023-10-11T15:14:02.293 DEBUG:boto:Token: None 2023-10-11T15:14:02.294 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:02.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tQvb+xQ1NZZfArxonBu3S6urrNM= 2023-10-11T15:14:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tQvb+xQ1NZZfArxonBu3S6urrNM='} 2023-10-11T15:14:02.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0b9bf52029af5c1-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.339 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.340 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.340 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.340 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.341 DEBUG:boto:Data: 2023-10-11T15:14:02.341 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.342 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.342 DEBUG:boto:Params: {} 2023-10-11T15:14:02.343 DEBUG:boto:Token: None 2023-10-11T15:14:02.343 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c3d2a60565ef631-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.348 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:02.349 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:02.349 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.349 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:02.350 DEBUG:boto:Data: 2023-10-11T15:14:02.350 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.351 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.352 DEBUG:boto:Params: {} 2023-10-11T15:14:02.352 DEBUG:boto:Token: None 2023-10-11T15:14:02.352 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:02.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KfaT3jSzMfVcf5puXPHx36MEHys= 2023-10-11T15:14:02.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KfaT3jSzMfVcf5puXPHx36MEHys='} 2023-10-11T15:14:02.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001477741a8af175cd-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.361 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.361 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.361 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.362 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.362 DEBUG:boto:Data: 2023-10-11T15:14:02.363 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.363 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.364 DEBUG:boto:Params: {} 2023-10-11T15:14:02.364 DEBUG:boto:Token: None 2023-10-11T15:14:02.365 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000861c75fa549b5d5e-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.369 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:02.370 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:02.370 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.371 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:02.371 DEBUG:boto:Data: 2023-10-11T15:14:02.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.372 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.373 DEBUG:boto:Params: {} 2023-10-11T15:14:02.373 DEBUG:boto:Token: None 2023-10-11T15:14:02.374 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:02.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u2hub7GKAluNClhRuKuR0nPPoOY= 2023-10-11T15:14:02.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u2hub7GKAluNClhRuKuR0nPPoOY='} 2023-10-11T15:14:02.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec61f6ee75b1310b-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.419 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.420 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.420 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.420 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.421 DEBUG:boto:Data: 2023-10-11T15:14:02.421 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.422 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.422 DEBUG:boto:Params: {} 2023-10-11T15:14:02.423 DEBUG:boto:Token: None 2023-10-11T15:14:02.423 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8449d7de06fca9f-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.428 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:02.428 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:02.429 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.429 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:02.430 DEBUG:boto:Data: 2023-10-11T15:14:02.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.431 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.432 DEBUG:boto:Params: {} 2023-10-11T15:14:02.432 DEBUG:boto:Token: None 2023-10-11T15:14:02.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:02.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dMvVgUrA7hH7227VfFXeSAul/tQ= 2023-10-11T15:14:02.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dMvVgUrA7hH7227VfFXeSAul/tQ='} 2023-10-11T15:14:02.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec269561819bd70c-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.440 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.441 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.441 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.441 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.442 DEBUG:boto:Data: 2023-10-11T15:14:02.443 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.443 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.444 DEBUG:boto:Params: {} 2023-10-11T15:14:02.444 DEBUG:boto:Token: None 2023-10-11T15:14:02.445 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1256b917adfa184-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.449 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:02.450 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:02.450 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.451 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:02.451 DEBUG:boto:Data: 2023-10-11T15:14:02.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.452 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.453 DEBUG:boto:Params: {} 2023-10-11T15:14:02.453 DEBUG:boto:Token: None 2023-10-11T15:14:02.454 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:02.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fak3NEddQop0qVqrTN1wayGufjM= 2023-10-11T15:14:02.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fak3NEddQop0qVqrTN1wayGufjM='} 2023-10-11T15:14:02.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035c9d26dd305d844-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.499 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.500 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.500 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.501 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.501 DEBUG:boto:Data: 2023-10-11T15:14:02.501 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.502 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.502 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.503 DEBUG:boto:Params: {} 2023-10-11T15:14:02.503 DEBUG:boto:Token: None 2023-10-11T15:14:02.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043e5854a938ef346-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.508 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:02.509 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:02.509 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.509 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:02.510 DEBUG:boto:Data: 2023-10-11T15:14:02.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.511 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.511 DEBUG:boto:Params: {} 2023-10-11T15:14:02.512 DEBUG:boto:Token: None 2023-10-11T15:14:02.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:02.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sqtsm4WwJmdXfmMpv5HI1L/g0nU= 2023-10-11T15:14:02.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sqtsm4WwJmdXfmMpv5HI1L/g0nU='} 2023-10-11T15:14:02.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd25a3842f918521-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.520 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.520 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.521 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.521 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.521 DEBUG:boto:Data: 2023-10-11T15:14:02.522 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.523 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.523 DEBUG:boto:Params: {} 2023-10-11T15:14:02.524 DEBUG:boto:Token: None 2023-10-11T15:14:02.524 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eeae6db3c0925758-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.532 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:02.532 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:02.532 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.533 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:02.533 DEBUG:boto:Data: 2023-10-11T15:14:02.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.535 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.535 DEBUG:boto:Params: {} 2023-10-11T15:14:02.536 DEBUG:boto:Token: None 2023-10-11T15:14:02.536 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:02.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:trsXXlE5lWPI1gU0KhsFdSYDCYs= 2023-10-11T15:14:02.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:trsXXlE5lWPI1gU0KhsFdSYDCYs='} 2023-10-11T15:14:02.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020708cf3f8a9fd74-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.583 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.584 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.584 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.584 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.585 DEBUG:boto:Data: 2023-10-11T15:14:02.585 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.586 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.586 DEBUG:boto:Params: {} 2023-10-11T15:14:02.587 DEBUG:boto:Token: None 2023-10-11T15:14:02.587 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002cd5624626f96ed-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.592 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:02.593 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:02.593 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.594 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:02.594 DEBUG:boto:Data: 2023-10-11T15:14:02.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.595 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.596 DEBUG:boto:Params: {} 2023-10-11T15:14:02.596 DEBUG:boto:Token: None 2023-10-11T15:14:02.597 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:02.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H7tMu1LHYfmqlvJgiUb/1BShde0= 2023-10-11T15:14:02.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H7tMu1LHYfmqlvJgiUb/1BShde0='} 2023-10-11T15:14:02.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b68d63911ee7bd5-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.605 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.605 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.605 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.606 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.606 DEBUG:boto:Data: 2023-10-11T15:14:02.607 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.608 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.608 DEBUG:boto:Params: {} 2023-10-11T15:14:02.608 DEBUG:boto:Token: None 2023-10-11T15:14:02.609 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009aa61acfb3be9a4c-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.613 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:02.614 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:02.614 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.615 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:02.615 DEBUG:boto:Data: 2023-10-11T15:14:02.616 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.616 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.617 DEBUG:boto:Params: {} 2023-10-11T15:14:02.617 DEBUG:boto:Token: None 2023-10-11T15:14:02.618 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:02.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qqf44k7jVL69IJR1yDISPwWjZAw= 2023-10-11T15:14:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qqf44k7jVL69IJR1yDISPwWjZAw='} 2023-10-11T15:14:02.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d157fcf089e1963d-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.663 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.664 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.664 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.665 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.665 DEBUG:boto:Data: 2023-10-11T15:14:02.666 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.667 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.667 DEBUG:boto:Params: {} 2023-10-11T15:14:02.668 DEBUG:boto:Token: None 2023-10-11T15:14:02.668 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063192b62067b5e38-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.674 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:02.675 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:02.675 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.675 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:02.676 DEBUG:boto:Data: 2023-10-11T15:14:02.676 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.677 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.677 DEBUG:boto:Params: {} 2023-10-11T15:14:02.678 DEBUG:boto:Token: None 2023-10-11T15:14:02.678 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:02.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HeiefifgPmm3jNsRoTCF5awK3SA= 2023-10-11T15:14:02.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HeiefifgPmm3jNsRoTCF5awK3SA='} 2023-10-11T15:14:02.685 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f89bb7c38c0ffa27-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.685 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.686 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.686 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.687 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.687 DEBUG:boto:Data: 2023-10-11T15:14:02.688 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.688 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.689 DEBUG:boto:Params: {} 2023-10-11T15:14:02.689 DEBUG:boto:Token: None 2023-10-11T15:14:02.690 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2f2a782e5e04034-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.694 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:02.695 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:02.695 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.696 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:02.696 DEBUG:boto:Data: 2023-10-11T15:14:02.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.698 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.698 DEBUG:boto:Params: {} 2023-10-11T15:14:02.698 DEBUG:boto:Token: None 2023-10-11T15:14:02.699 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:02.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YjD4XdlkV4NBdgFNYwf5WSPf/bU= 2023-10-11T15:14:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YjD4XdlkV4NBdgFNYwf5WSPf/bU='} 2023-10-11T15:14:02.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bae13b1b465cf554-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.705 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.706 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.706 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.706 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.707 DEBUG:boto:Data: 2023-10-11T15:14:02.707 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.708 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.708 DEBUG:boto:Params: {} 2023-10-11T15:14:02.709 DEBUG:boto:Token: None 2023-10-11T15:14:02.709 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe9d92543b71613b-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.714 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:02.715 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:02.715 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.715 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:02.716 DEBUG:boto:Data: 2023-10-11T15:14:02.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.717 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.717 DEBUG:boto:Params: {} 2023-10-11T15:14:02.718 DEBUG:boto:Token: None 2023-10-11T15:14:02.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:02.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KZF8IMfLm+OuixIDKE+PbjJw230= 2023-10-11T15:14:02.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KZF8IMfLm+OuixIDKE+PbjJw230='} 2023-10-11T15:14:02.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c35492603d5cc898-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.763 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.764 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.764 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.764 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.765 DEBUG:boto:Data: 2023-10-11T15:14:02.765 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.766 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.767 DEBUG:boto:Params: {} 2023-10-11T15:14:02.767 DEBUG:boto:Token: None 2023-10-11T15:14:02.767 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bebb49b8f8f1beb-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.773 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:02.773 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:02.773 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.774 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:02.774 DEBUG:boto:Data: 2023-10-11T15:14:02.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.776 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.776 DEBUG:boto:Params: {} 2023-10-11T15:14:02.776 DEBUG:boto:Token: None 2023-10-11T15:14:02.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:02.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ln5hO/QKF/+FkxzNa2MFfPs4B0c= 2023-10-11T15:14:02.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ln5hO/QKF/+FkxzNa2MFfPs4B0c='} 2023-10-11T15:14:02.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db4eb207a9dcffc1-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.783 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:02.784 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:02.784 DEBUG:boto:Method: HEAD 2023-10-11T15:14:02.785 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:02.785 DEBUG:boto:Data: 2023-10-11T15:14:02.785 DEBUG:boto:Headers: {} 2023-10-11T15:14:02.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.786 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.787 DEBUG:boto:Params: {} 2023-10-11T15:14:02.787 DEBUG:boto:Token: None 2023-10-11T15:14:02.788 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/ 2023-10-11T15:14:02.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk= 2023-10-11T15:14:02.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MByBYHApIfVgyrvc1jxNqznMODk=', 'Content-Length': '0'} 2023-10-11T15:14:02.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000010febd4f4482ba9-006526bbba-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.792 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:02.793 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:02.793 DEBUG:boto:Method: PUT 2023-10-11T15:14:02.793 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:02.794 DEBUG:boto:Data: 2023-10-11T15:14:02.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:02.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:02.795 DEBUG:boto:Port: 8000 2023-10-11T15:14:02.796 DEBUG:boto:Params: {} 2023-10-11T15:14:02.796 DEBUG:boto:Token: None 2023-10-11T15:14:02.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:02 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:02.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tpEPddRvZ0rpV/f3OBwPGxG4hdg= 2023-10-11T15:14:02.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:02 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tpEPddRvZ0rpV/f3OBwPGxG4hdg='} 2023-10-11T15:14:02.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ec44234529e6f12-006526bbba-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:02.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:02.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:02.848 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:14:02.902 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:02.902 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:02.950 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:14:02.950 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:14:02.961 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 42 2023-10-11T15:14:03.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:03.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:03.019 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:03.070 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:03.070 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:03.110 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:14:03.110 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000019", 2023-10-11T15:14:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006", 2023-10-11T15:14:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000467", 2023-10-11T15:14:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000005", 2023-10-11T15:14:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000056", 2023-10-11T15:14:03.111 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:14:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000023", 2023-10-11T15:14:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000317", 2023-10-11T15:14:03.112 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000022", 2023-10-11T15:14:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:14:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021", 2023-10-11T15:14:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000009", 2023-10-11T15:14:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000271", 2023-10-11T15:14:03.113 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000008", 2023-10-11T15:14:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000018", 2023-10-11T15:14:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000007", 2023-10-11T15:14:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000017", 2023-10-11T15:14:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000162", 2023-10-11T15:14:03.114 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000016", 2023-10-11T15:14:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:14:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000423", 2023-10-11T15:14:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:14:03.115 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:14:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:14:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:14:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000283", 2023-10-11T15:14:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:14:03.116 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000061", 2023-10-11T15:14:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000050", 2023-10-11T15:14:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000029", 2023-10-11T15:14:03.117 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000133" 2023-10-11T15:14:03.118 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:03.118 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:03.120 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.120 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.121 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.122 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.123 DEBUG:boto:Data: 2023-10-11T15:14:03.123 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.124 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.125 DEBUG:boto:Params: {} 2023-10-11T15:14:03.126 DEBUG:boto:Token: None 2023-10-11T15:14:03.127 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095fba4f7a02abb6d-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.132 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:03.132 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:03.132 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.133 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:03.133 DEBUG:boto:Data: 2023-10-11T15:14:03.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.135 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.135 DEBUG:boto:Params: {} 2023-10-11T15:14:03.136 DEBUG:boto:Token: None 2023-10-11T15:14:03.136 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:03.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rv6aat8oSOF3osesjHtKM4drgPc= 2023-10-11T15:14:03.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rv6aat8oSOF3osesjHtKM4drgPc='} 2023-10-11T15:14:03.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b41ac79cc3e5d050-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.144 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.144 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.145 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.145 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.146 DEBUG:boto:Data: 2023-10-11T15:14:03.146 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.147 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.147 DEBUG:boto:Params: {} 2023-10-11T15:14:03.148 DEBUG:boto:Token: None 2023-10-11T15:14:03.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b839b36ec79b96c-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.153 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:03.154 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:03.155 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.155 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:03.156 DEBUG:boto:Data: 2023-10-11T15:14:03.157 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.158 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.159 DEBUG:boto:Params: {} 2023-10-11T15:14:03.160 DEBUG:boto:Token: None 2023-10-11T15:14:03.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:03.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EzC96lNUfCq8Hl5ukr3PHMQfvUg= 2023-10-11T15:14:03.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EzC96lNUfCq8Hl5ukr3PHMQfvUg='} 2023-10-11T15:14:03.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e2e06c2c6c62b29-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.170 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.170 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.171 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.171 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.171 DEBUG:boto:Data: 2023-10-11T15:14:03.172 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.173 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.173 DEBUG:boto:Params: {} 2023-10-11T15:14:03.174 DEBUG:boto:Token: None 2023-10-11T15:14:03.174 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000899a53613f4915b9-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.179 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:03.180 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:03.180 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.181 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:03.181 DEBUG:boto:Data: 2023-10-11T15:14:03.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.183 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.184 DEBUG:boto:Params: {} 2023-10-11T15:14:03.184 DEBUG:boto:Token: None 2023-10-11T15:14:03.185 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:03.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lloi/pD04/NsqbUbx62PtJpa4vY= 2023-10-11T15:14:03.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lloi/pD04/NsqbUbx62PtJpa4vY='} 2023-10-11T15:14:03.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000705882b2c4791c39-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.235 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.236 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.236 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.236 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.237 DEBUG:boto:Data: 2023-10-11T15:14:03.238 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.239 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.239 DEBUG:boto:Params: {} 2023-10-11T15:14:03.240 DEBUG:boto:Token: None 2023-10-11T15:14:03.240 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.241 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e1de857a28aa264-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.245 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:03.246 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:03.246 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.247 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:03.247 DEBUG:boto:Data: 2023-10-11T15:14:03.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.248 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.248 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.249 DEBUG:boto:Params: {} 2023-10-11T15:14:03.249 DEBUG:boto:Token: None 2023-10-11T15:14:03.250 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:03.250 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6MeYz+oJACXyaQb1KhaOUA8scro= 2023-10-11T15:14:03.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6MeYz+oJACXyaQb1KhaOUA8scro='} 2023-10-11T15:14:03.257 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004fcb6f742ad8f29-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.257 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.258 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.258 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.259 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.259 DEBUG:boto:Data: 2023-10-11T15:14:03.259 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.260 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.260 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.261 DEBUG:boto:Params: {} 2023-10-11T15:14:03.261 DEBUG:boto:Token: None 2023-10-11T15:14:03.262 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.262 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1b31c5cc54d3178-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.266 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:03.266 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:03.267 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.267 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:03.268 DEBUG:boto:Data: 2023-10-11T15:14:03.268 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.269 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.269 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.269 DEBUG:boto:Params: {} 2023-10-11T15:14:03.270 DEBUG:boto:Token: None 2023-10-11T15:14:03.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:03.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+hZgN1i/jvYoKS4xIMEQe4hcWCs= 2023-10-11T15:14:03.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+hZgN1i/jvYoKS4xIMEQe4hcWCs='} 2023-10-11T15:14:03.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff33138de7c928f3-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.277 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.277 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.278 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.278 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.279 DEBUG:boto:Data: 2023-10-11T15:14:03.279 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.280 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.280 DEBUG:boto:Params: {} 2023-10-11T15:14:03.281 DEBUG:boto:Token: None 2023-10-11T15:14:03.281 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c822426562092b5-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.286 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:03.286 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:03.287 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.287 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:03.287 DEBUG:boto:Data: 2023-10-11T15:14:03.288 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.288 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.289 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.289 DEBUG:boto:Params: {} 2023-10-11T15:14:03.290 DEBUG:boto:Token: None 2023-10-11T15:14:03.290 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:03.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FsY7LxXWcfpIEcNW42xVzpnO5K8= 2023-10-11T15:14:03.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FsY7LxXWcfpIEcNW42xVzpnO5K8='} 2023-10-11T15:14:03.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b2f048eaaed3e57-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.336 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.336 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.337 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.337 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.338 DEBUG:boto:Data: 2023-10-11T15:14:03.338 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.339 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.339 DEBUG:boto:Params: {} 2023-10-11T15:14:03.340 DEBUG:boto:Token: None 2023-10-11T15:14:03.340 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e702b40da8eeb37e-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.345 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:03.345 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:03.346 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.346 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:03.347 DEBUG:boto:Data: 2023-10-11T15:14:03.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.350 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.350 DEBUG:boto:Params: {} 2023-10-11T15:14:03.351 DEBUG:boto:Token: None 2023-10-11T15:14:03.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:03.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aG2LbsJ/Uykh5W1pHVoa8Ss0L5M= 2023-10-11T15:14:03.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aG2LbsJ/Uykh5W1pHVoa8Ss0L5M='} 2023-10-11T15:14:03.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9ea2bc09217f9a0-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.358 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.359 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.359 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.360 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.360 DEBUG:boto:Data: 2023-10-11T15:14:03.360 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.361 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.361 DEBUG:boto:Params: {} 2023-10-11T15:14:03.362 DEBUG:boto:Token: None 2023-10-11T15:14:03.362 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcf229a549532d76-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.367 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:03.368 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:03.368 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.369 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:03.369 DEBUG:boto:Data: 2023-10-11T15:14:03.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.370 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.371 DEBUG:boto:Params: {} 2023-10-11T15:14:03.371 DEBUG:boto:Token: None 2023-10-11T15:14:03.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:03.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E/EkB9YorAJtdue3OJ/MCZJNsRU= 2023-10-11T15:14:03.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E/EkB9YorAJtdue3OJ/MCZJNsRU='} 2023-10-11T15:14:03.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d06f25a4206733b1-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.415 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.416 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.416 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.416 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.417 DEBUG:boto:Data: 2023-10-11T15:14:03.417 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.418 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.418 DEBUG:boto:Params: {} 2023-10-11T15:14:03.419 DEBUG:boto:Token: None 2023-10-11T15:14:03.419 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053d79fa91308fe72-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.424 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:03.425 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:03.425 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.426 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:03.426 DEBUG:boto:Data: 2023-10-11T15:14:03.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.427 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.428 DEBUG:boto:Params: {} 2023-10-11T15:14:03.428 DEBUG:boto:Token: None 2023-10-11T15:14:03.429 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:03.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F9+RkCusXMns+/bwCENFoYmko7M= 2023-10-11T15:14:03.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F9+RkCusXMns+/bwCENFoYmko7M='} 2023-10-11T15:14:03.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e1bcdb8ce507931-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.435 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.435 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.436 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.436 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.436 DEBUG:boto:Data: 2023-10-11T15:14:03.437 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.438 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.438 DEBUG:boto:Params: {} 2023-10-11T15:14:03.439 DEBUG:boto:Token: None 2023-10-11T15:14:03.439 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.443 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007395fe292a6c31a7-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.444 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:03.444 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:03.444 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.445 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:03.445 DEBUG:boto:Data: 2023-10-11T15:14:03.446 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.446 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.447 DEBUG:boto:Params: {} 2023-10-11T15:14:03.447 DEBUG:boto:Token: None 2023-10-11T15:14:03.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:03.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DUP1AyqZnMylSApTztH1Kh6elNY= 2023-10-11T15:14:03.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DUP1AyqZnMylSApTztH1Kh6elNY='} 2023-10-11T15:14:03.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f5e17dcf76ac7c5-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.491 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.492 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.492 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.492 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.493 DEBUG:boto:Data: 2023-10-11T15:14:03.493 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.494 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.495 DEBUG:boto:Params: {} 2023-10-11T15:14:03.495 DEBUG:boto:Token: None 2023-10-11T15:14:03.495 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.496 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4bf60f66df39577-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.500 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:03.501 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:03.501 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.502 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:03.502 DEBUG:boto:Data: 2023-10-11T15:14:03.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.505 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.505 DEBUG:boto:Params: {} 2023-10-11T15:14:03.506 DEBUG:boto:Token: None 2023-10-11T15:14:03.506 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:03.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ok8jIR29uE9s7dyLomDoobySigk= 2023-10-11T15:14:03.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ok8jIR29uE9s7dyLomDoobySigk='} 2023-10-11T15:14:03.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000528dee29fde7b16a-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.513 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.513 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.514 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.514 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.515 DEBUG:boto:Data: 2023-10-11T15:14:03.515 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.516 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.516 DEBUG:boto:Params: {} 2023-10-11T15:14:03.517 DEBUG:boto:Token: None 2023-10-11T15:14:03.517 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.521 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000872b9fc865621fa6-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.522 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:03.522 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:03.523 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.523 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:03.524 DEBUG:boto:Data: 2023-10-11T15:14:03.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.525 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.525 DEBUG:boto:Params: {} 2023-10-11T15:14:03.526 DEBUG:boto:Token: None 2023-10-11T15:14:03.526 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:03.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pWKnIhlR3s1dnjyW+vqZqZOSZ94= 2023-10-11T15:14:03.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWKnIhlR3s1dnjyW+vqZqZOSZ94='} 2023-10-11T15:14:03.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000496c7098b6f7e77d-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.575 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.576 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.576 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.576 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.577 DEBUG:boto:Data: 2023-10-11T15:14:03.577 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.578 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.578 DEBUG:boto:Params: {} 2023-10-11T15:14:03.579 DEBUG:boto:Token: None 2023-10-11T15:14:03.579 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086cf440287b9460b-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.584 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:03.585 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:03.585 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.585 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:03.586 DEBUG:boto:Data: 2023-10-11T15:14:03.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.587 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.588 DEBUG:boto:Params: {} 2023-10-11T15:14:03.588 DEBUG:boto:Token: None 2023-10-11T15:14:03.589 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:03.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Sxz08qtEnu4zrpHLtPvFmvMzXW0= 2023-10-11T15:14:03.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sxz08qtEnu4zrpHLtPvFmvMzXW0='} 2023-10-11T15:14:03.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f4cc46034ae3517-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.596 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.597 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.597 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.597 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.598 DEBUG:boto:Data: 2023-10-11T15:14:03.598 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.599 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.600 DEBUG:boto:Params: {} 2023-10-11T15:14:03.600 DEBUG:boto:Token: None 2023-10-11T15:14:03.600 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec325307070c596f-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.605 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:03.606 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:03.606 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.607 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:03.607 DEBUG:boto:Data: 2023-10-11T15:14:03.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.608 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.608 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.609 DEBUG:boto:Params: {} 2023-10-11T15:14:03.609 DEBUG:boto:Token: None 2023-10-11T15:14:03.610 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:03.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p6FJXosq8+ra2b9bBTYLqIIvfXY= 2023-10-11T15:14:03.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p6FJXosq8+ra2b9bBTYLqIIvfXY='} 2023-10-11T15:14:03.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c06863c70942162-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.655 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.656 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.656 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.656 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.657 DEBUG:boto:Data: 2023-10-11T15:14:03.657 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.658 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.659 DEBUG:boto:Params: {} 2023-10-11T15:14:03.659 DEBUG:boto:Token: None 2023-10-11T15:14:03.660 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.664 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a87816e0266df6be-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.665 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:03.665 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:03.665 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.666 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:03.666 DEBUG:boto:Data: 2023-10-11T15:14:03.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.668 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.668 DEBUG:boto:Params: {} 2023-10-11T15:14:03.668 DEBUG:boto:Token: None 2023-10-11T15:14:03.669 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:03.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q2HgWe6h5MRHKOxi1iyMSaGm5Hg= 2023-10-11T15:14:03.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q2HgWe6h5MRHKOxi1iyMSaGm5Hg='} 2023-10-11T15:14:03.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d087338bf678f0da-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.677 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.677 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.678 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.678 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.678 DEBUG:boto:Data: 2023-10-11T15:14:03.679 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.680 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.680 DEBUG:boto:Params: {} 2023-10-11T15:14:03.680 DEBUG:boto:Token: None 2023-10-11T15:14:03.681 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.685 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c273dd7d7ff1f9e-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.686 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:03.686 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:03.686 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.687 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:03.687 DEBUG:boto:Data: 2023-10-11T15:14:03.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.688 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.689 DEBUG:boto:Params: {} 2023-10-11T15:14:03.689 DEBUG:boto:Token: None 2023-10-11T15:14:03.690 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:03.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fxNVlPWsRW+hVuTN1x2/aZz9s9g= 2023-10-11T15:14:03.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fxNVlPWsRW+hVuTN1x2/aZz9s9g='} 2023-10-11T15:14:03.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b85c28c07399700-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.739 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.740 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.740 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.741 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.741 DEBUG:boto:Data: 2023-10-11T15:14:03.741 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.742 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.743 DEBUG:boto:Params: {} 2023-10-11T15:14:03.743 DEBUG:boto:Token: None 2023-10-11T15:14:03.743 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000827b7e88e974be56-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.748 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:03.749 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:03.749 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.749 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:03.750 DEBUG:boto:Data: 2023-10-11T15:14:03.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.751 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.751 DEBUG:boto:Params: {} 2023-10-11T15:14:03.752 DEBUG:boto:Token: None 2023-10-11T15:14:03.752 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:03.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bJiGz39gIsMOPLw/W6PYE3ixflo= 2023-10-11T15:14:03.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bJiGz39gIsMOPLw/W6PYE3ixflo='} 2023-10-11T15:14:03.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009433f1f36bd5ee72-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.762 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.762 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.762 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.763 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.763 DEBUG:boto:Data: 2023-10-11T15:14:03.764 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.764 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.765 DEBUG:boto:Params: {} 2023-10-11T15:14:03.765 DEBUG:boto:Token: None 2023-10-11T15:14:03.766 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000453bde28a5fe3ed1-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.771 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:03.771 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:03.772 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.772 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:03.772 DEBUG:boto:Data: 2023-10-11T15:14:03.773 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.774 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.774 DEBUG:boto:Params: {} 2023-10-11T15:14:03.774 DEBUG:boto:Token: None 2023-10-11T15:14:03.775 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:03.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbjaDbHnJRqbmkfX/Qx+ZdigpAE= 2023-10-11T15:14:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbjaDbHnJRqbmkfX/Qx+ZdigpAE='} 2023-10-11T15:14:03.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3289bd5f17f075a-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.819 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.820 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.820 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.821 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.821 DEBUG:boto:Data: 2023-10-11T15:14:03.821 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.822 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.823 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.823 DEBUG:boto:Params: {} 2023-10-11T15:14:03.824 DEBUG:boto:Token: None 2023-10-11T15:14:03.825 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef7e4e16dc7c1f71-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.831 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:03.831 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:03.832 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.832 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:03.833 DEBUG:boto:Data: 2023-10-11T15:14:03.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.834 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.834 DEBUG:boto:Params: {} 2023-10-11T15:14:03.835 DEBUG:boto:Token: None 2023-10-11T15:14:03.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:03.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:avF8ZK9Ai6F5ZiCAAgmsV0SGG/c= 2023-10-11T15:14:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:avF8ZK9Ai6F5ZiCAAgmsV0SGG/c='} 2023-10-11T15:14:03.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7d3fe433e6725d3-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.843 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.844 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.845 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.845 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.846 DEBUG:boto:Data: 2023-10-11T15:14:03.846 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.847 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.847 DEBUG:boto:Params: {} 2023-10-11T15:14:03.848 DEBUG:boto:Token: None 2023-10-11T15:14:03.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075a1c8ca4a856af0-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.855 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:03.856 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:03.856 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.857 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:03.858 DEBUG:boto:Data: 2023-10-11T15:14:03.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.860 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.860 DEBUG:boto:Params: {} 2023-10-11T15:14:03.861 DEBUG:boto:Token: None 2023-10-11T15:14:03.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:03.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3mI7LNix9wNuIcopcf3SlisDAi0= 2023-10-11T15:14:03.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3mI7LNix9wNuIcopcf3SlisDAi0='} 2023-10-11T15:14:03.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ae4c97e3940126e-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.907 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.908 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.908 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.909 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.909 DEBUG:boto:Data: 2023-10-11T15:14:03.910 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.911 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.911 DEBUG:boto:Params: {} 2023-10-11T15:14:03.912 DEBUG:boto:Token: None 2023-10-11T15:14:03.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fa7b9dc3a9853d4-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.918 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:03.918 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:03.919 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.919 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:03.919 DEBUG:boto:Data: 2023-10-11T15:14:03.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.921 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.921 DEBUG:boto:Params: {} 2023-10-11T15:14:03.921 DEBUG:boto:Token: None 2023-10-11T15:14:03.922 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:03.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tDcXlTaUR9megZ8EP8gWuCY/RhM= 2023-10-11T15:14:03.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tDcXlTaUR9megZ8EP8gWuCY/RhM='} 2023-10-11T15:14:03.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae38982e523e88d6-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.928 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.929 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.929 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.930 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.930 DEBUG:boto:Data: 2023-10-11T15:14:03.931 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.931 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.931 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.932 DEBUG:boto:Params: {} 2023-10-11T15:14:03.932 DEBUG:boto:Token: None 2023-10-11T15:14:03.933 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:03.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5dfff483265b2cc-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.937 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:03.938 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:03.938 DEBUG:boto:Method: PUT 2023-10-11T15:14:03.939 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:03.939 DEBUG:boto:Data: 2023-10-11T15:14:03.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:03.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.940 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.941 DEBUG:boto:Params: {} 2023-10-11T15:14:03.941 DEBUG:boto:Token: None 2023-10-11T15:14:03.942 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:03.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rqDy04OZbiRUR+xtU9M0oSdsNvM= 2023-10-11T15:14:03.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqDy04OZbiRUR+xtU9M0oSdsNvM='} 2023-10-11T15:14:03.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aff796d48fde88f-006526bbbb-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:03.991 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:03.992 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:03.992 DEBUG:boto:Method: HEAD 2023-10-11T15:14:03.993 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:03.993 DEBUG:boto:Data: 2023-10-11T15:14:03.993 DEBUG:boto:Headers: {} 2023-10-11T15:14:03.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:03.994 DEBUG:boto:Port: 8000 2023-10-11T15:14:03.995 DEBUG:boto:Params: {} 2023-10-11T15:14:03.995 DEBUG:boto:Token: None 2023-10-11T15:14:03.996 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:03 GMT /dghqvd-58/ 2023-10-11T15:14:03.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M= 2023-10-11T15:14:03.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:03 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HgzvOyClo6bH1KP2Pk/cNNS/E8M=', 'Content-Length': '0'} 2023-10-11T15:14:04.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000946a1ca4b3571a79-006526bbbb-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.000 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:04.001 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:04.001 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.002 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:04.002 DEBUG:boto:Data: 2023-10-11T15:14:04.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.003 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.004 DEBUG:boto:Params: {} 2023-10-11T15:14:04.004 DEBUG:boto:Token: None 2023-10-11T15:14:04.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:04.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i7HZfqdzBaBbxsjLOiNyQPlNCOM= 2023-10-11T15:14:04.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i7HZfqdzBaBbxsjLOiNyQPlNCOM='} 2023-10-11T15:14:04.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d492e7589ee4036-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.011 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.011 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.012 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.012 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.013 DEBUG:boto:Data: 2023-10-11T15:14:04.013 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.014 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.014 DEBUG:boto:Params: {} 2023-10-11T15:14:04.015 DEBUG:boto:Token: None 2023-10-11T15:14:04.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c35111ed719ece56-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.020 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:04.020 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:04.020 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.021 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:04.021 DEBUG:boto:Data: 2023-10-11T15:14:04.022 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.022 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.023 DEBUG:boto:Params: {} 2023-10-11T15:14:04.024 DEBUG:boto:Token: None 2023-10-11T15:14:04.024 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:04.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lpxoKsC5XEOc1E0O/8s/4KHHl7A= 2023-10-11T15:14:04.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lpxoKsC5XEOc1E0O/8s/4KHHl7A='} 2023-10-11T15:14:04.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004eace3a4367810b3-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.075 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.076 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.076 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.077 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.077 DEBUG:boto:Data: 2023-10-11T15:14:04.077 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.078 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.078 DEBUG:boto:Params: {} 2023-10-11T15:14:04.079 DEBUG:boto:Token: None 2023-10-11T15:14:04.079 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000234cfd1a99c8af8-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.085 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:04.085 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:04.085 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.086 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:04.086 DEBUG:boto:Data: 2023-10-11T15:14:04.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.087 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.088 DEBUG:boto:Params: {} 2023-10-11T15:14:04.088 DEBUG:boto:Token: None 2023-10-11T15:14:04.089 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:04.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LL2rxa/wr5tqD0Irh6RUniyS+IY= 2023-10-11T15:14:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LL2rxa/wr5tqD0Irh6RUniyS+IY='} 2023-10-11T15:14:04.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018555f0a38503325-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.097 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.097 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.097 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.098 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.098 DEBUG:boto:Data: 2023-10-11T15:14:04.099 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.099 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.100 DEBUG:boto:Params: {} 2023-10-11T15:14:04.100 DEBUG:boto:Token: None 2023-10-11T15:14:04.101 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e8a2f0f6a08e6bb-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.105 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:04.106 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:04.106 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.106 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:04.107 DEBUG:boto:Data: 2023-10-11T15:14:04.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.108 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.108 DEBUG:boto:Params: {} 2023-10-11T15:14:04.109 DEBUG:boto:Token: None 2023-10-11T15:14:04.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:04.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RkjMHILejC7JzwvpREQLlRcVVKQ= 2023-10-11T15:14:04.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RkjMHILejC7JzwvpREQLlRcVVKQ='} 2023-10-11T15:14:04.158 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb6db97573a21d9e-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.159 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.160 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.160 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.160 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.161 DEBUG:boto:Data: 2023-10-11T15:14:04.161 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.162 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.162 DEBUG:boto:Params: {} 2023-10-11T15:14:04.163 DEBUG:boto:Token: None 2023-10-11T15:14:04.163 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.164 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8891b92c0c1a017-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.169 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:04.169 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:04.170 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.170 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:04.171 DEBUG:boto:Data: 2023-10-11T15:14:04.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.171 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.172 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.172 DEBUG:boto:Params: {} 2023-10-11T15:14:04.173 DEBUG:boto:Token: None 2023-10-11T15:14:04.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:04.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OeDhDq+BTu/VLjcSaC/ZVh3tnVQ= 2023-10-11T15:14:04.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OeDhDq+BTu/VLjcSaC/ZVh3tnVQ='} 2023-10-11T15:14:04.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2ef0c6b8c775e4d-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.181 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.181 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.181 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.182 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.182 DEBUG:boto:Data: 2023-10-11T15:14:04.183 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.184 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.184 DEBUG:boto:Params: {} 2023-10-11T15:14:04.184 DEBUG:boto:Token: None 2023-10-11T15:14:04.185 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7f1b677bc41a517-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.190 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:04.190 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:04.191 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.191 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:04.191 DEBUG:boto:Data: 2023-10-11T15:14:04.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.192 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.193 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.193 DEBUG:boto:Params: {} 2023-10-11T15:14:04.193 DEBUG:boto:Token: None 2023-10-11T15:14:04.194 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:04.194 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lhucrkO9bsf20Y1aQfi9/8Cvoeg= 2023-10-11T15:14:04.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lhucrkO9bsf20Y1aQfi9/8Cvoeg='} 2023-10-11T15:14:04.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034d890111853116a-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:04.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:04.244 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:14:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:04.297 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:04.345 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:14:04.345 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-58 2023-10-11T15:14:04.356 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 53 2023-10-11T15:14:04.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:04.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:04.417 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:04.469 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:04.509 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:04.509 INFO:teuthology.orchestra.run.smithi158.stdout: "objs": [ 2023-10-11T15:14:04.510 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000060", 2023-10-11T15:14:04.510 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000000", 2023-10-11T15:14:04.510 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000011", 2023-10-11T15:14:04.510 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000001", 2023-10-11T15:14:04.511 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000033", 2023-10-11T15:14:04.511 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000065", 2023-10-11T15:14:04.511 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000005", 2023-10-11T15:14:04.511 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000016", 2023-10-11T15:14:04.511 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000048", 2023-10-11T15:14:04.512 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000015", 2023-10-11T15:14:04.512 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000026", 2023-10-11T15:14:04.512 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000030", 2023-10-11T15:14:04.512 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000062", 2023-10-11T15:14:04.512 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000009", 2023-10-11T15:14:04.513 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000040", 2023-10-11T15:14:04.513 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000024", 2023-10-11T15:14:04.513 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000012", 2023-10-11T15:14:04.513 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000002", 2023-10-11T15:14:04.514 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000013", 2023-10-11T15:14:04.514 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000045", 2023-10-11T15:14:04.514 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000006", 2023-10-11T15:14:04.514 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000017", 2023-10-11T15:14:04.515 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000007", 2023-10-11T15:14:04.515 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000039", 2023-10-11T15:14:04.515 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000027", 2023-10-11T15:14:04.515 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000010", 2023-10-11T15:14:04.515 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000021", 2023-10-11T15:14:04.516 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000053", 2023-10-11T15:14:04.516 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000020", 2023-10-11T15:14:04.516 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000004", 2023-10-11T15:14:04.516 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000036", 2023-10-11T15:14:04.516 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000003", 2023-10-11T15:14:04.517 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000014", 2023-10-11T15:14:04.517 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000019", 2023-10-11T15:14:04.517 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000057", 2023-10-11T15:14:04.517 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000018", 2023-10-11T15:14:04.518 INFO:teuthology.orchestra.run.smithi158.stdout: "obj00000000000000000008" 2023-10-11T15:14:04.518 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:04.518 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:04.519 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.520 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.520 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.521 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.521 DEBUG:boto:Data: 2023-10-11T15:14:04.521 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.522 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.523 DEBUG:boto:Params: {} 2023-10-11T15:14:04.523 DEBUG:boto:Token: None 2023-10-11T15:14:04.524 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000173b9ec1397941ea-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.529 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:04.529 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:04.530 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.530 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:04.530 DEBUG:boto:Data: 2023-10-11T15:14:04.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.532 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.532 DEBUG:boto:Params: {} 2023-10-11T15:14:04.532 DEBUG:boto:Token: None 2023-10-11T15:14:04.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:04.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VMhug/2lwvUTmApwKiWZrdPkzk8= 2023-10-11T15:14:04.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VMhug/2lwvUTmApwKiWZrdPkzk8='} 2023-10-11T15:14:04.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f41d65669b668fa9-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.540 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.540 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.540 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.541 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.541 DEBUG:boto:Data: 2023-10-11T15:14:04.541 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.542 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.543 DEBUG:boto:Params: {} 2023-10-11T15:14:04.543 DEBUG:boto:Token: None 2023-10-11T15:14:04.544 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093e400acec431c3f-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.548 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:04.549 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:04.549 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.549 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:04.550 DEBUG:boto:Data: 2023-10-11T15:14:04.550 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.551 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.551 DEBUG:boto:Params: {} 2023-10-11T15:14:04.552 DEBUG:boto:Token: None 2023-10-11T15:14:04.552 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:04.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JmNc8tIQ60HBncUEf3Ba6QY5qu0= 2023-10-11T15:14:04.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JmNc8tIQ60HBncUEf3Ba6QY5qu0='} 2023-10-11T15:14:04.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1758a5c71ca1774-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.603 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.604 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.604 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.605 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.605 DEBUG:boto:Data: 2023-10-11T15:14:04.605 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.606 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.606 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.607 DEBUG:boto:Params: {} 2023-10-11T15:14:04.607 DEBUG:boto:Token: None 2023-10-11T15:14:04.608 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.608 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000693a4223bab0fcf0-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.612 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:04.613 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:04.613 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.614 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:04.614 DEBUG:boto:Data: 2023-10-11T15:14:04.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.615 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.616 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.616 DEBUG:boto:Params: {} 2023-10-11T15:14:04.616 DEBUG:boto:Token: None 2023-10-11T15:14:04.617 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:04.617 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T34+pO6XebLEA5HGTaimYG8u+Gk= 2023-10-11T15:14:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T34+pO6XebLEA5HGTaimYG8u+Gk='} 2023-10-11T15:14:04.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000654b6b9ddfe0642b-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.625 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.625 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.626 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.626 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.627 DEBUG:boto:Data: 2023-10-11T15:14:04.627 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.628 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.628 DEBUG:boto:Params: {} 2023-10-11T15:14:04.629 DEBUG:boto:Token: None 2023-10-11T15:14:04.629 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005495f638b059b4d6-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.634 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:04.635 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:04.635 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.636 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:04.636 DEBUG:boto:Data: 2023-10-11T15:14:04.637 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.637 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.638 DEBUG:boto:Params: {} 2023-10-11T15:14:04.638 DEBUG:boto:Token: None 2023-10-11T15:14:04.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:04.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6/4ju+hdKhKGB5YsLoEa7gSmlNA= 2023-10-11T15:14:04.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6/4ju+hdKhKGB5YsLoEa7gSmlNA='} 2023-10-11T15:14:04.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0f4ae0d1bce5d94-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.687 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.688 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.688 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.688 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.689 DEBUG:boto:Data: 2023-10-11T15:14:04.689 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.690 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.690 DEBUG:boto:Params: {} 2023-10-11T15:14:04.691 DEBUG:boto:Token: None 2023-10-11T15:14:04.691 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa219cd960fd5f12-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.696 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:04.697 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:04.697 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.697 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:04.698 DEBUG:boto:Data: 2023-10-11T15:14:04.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.699 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.699 DEBUG:boto:Params: {} 2023-10-11T15:14:04.700 DEBUG:boto:Token: None 2023-10-11T15:14:04.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:04.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iz+9mu5fYqh+61gio/hkipcEgr0= 2023-10-11T15:14:04.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iz+9mu5fYqh+61gio/hkipcEgr0='} 2023-10-11T15:14:04.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000686a0b08f24e7ef2-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.707 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.707 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.708 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.708 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.708 DEBUG:boto:Data: 2023-10-11T15:14:04.709 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.710 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.710 DEBUG:boto:Params: {} 2023-10-11T15:14:04.710 DEBUG:boto:Token: None 2023-10-11T15:14:04.711 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008517037a3f70c352-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.716 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:04.716 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:04.716 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.717 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:04.717 DEBUG:boto:Data: 2023-10-11T15:14:04.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.718 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.719 DEBUG:boto:Params: {} 2023-10-11T15:14:04.719 DEBUG:boto:Token: None 2023-10-11T15:14:04.719 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:04.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8S7S+kVAMKwbZKO21Dv9kkpt7nE= 2023-10-11T15:14:04.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8S7S+kVAMKwbZKO21Dv9kkpt7nE='} 2023-10-11T15:14:04.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed90917a263f4114-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.767 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.768 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.768 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.768 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.769 DEBUG:boto:Data: 2023-10-11T15:14:04.769 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.770 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.770 DEBUG:boto:Params: {} 2023-10-11T15:14:04.771 DEBUG:boto:Token: None 2023-10-11T15:14:04.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000538dba772d6427b2-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.776 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:04.777 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:04.777 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.777 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:04.778 DEBUG:boto:Data: 2023-10-11T15:14:04.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.779 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.779 DEBUG:boto:Params: {} 2023-10-11T15:14:04.780 DEBUG:boto:Token: None 2023-10-11T15:14:04.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:04.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FpV1V/ICHW7J8eCcyLhpD5voYWM= 2023-10-11T15:14:04.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FpV1V/ICHW7J8eCcyLhpD5voYWM='} 2023-10-11T15:14:04.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a02c8c9318d28cfa-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.788 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.788 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.789 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.789 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.790 DEBUG:boto:Data: 2023-10-11T15:14:04.790 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.793 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.793 DEBUG:boto:Params: {} 2023-10-11T15:14:04.794 DEBUG:boto:Token: None 2023-10-11T15:14:04.794 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d18d425bdbd34e4-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.799 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:04.800 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:04.800 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.801 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:04.801 DEBUG:boto:Data: 2023-10-11T15:14:04.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.802 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.802 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.803 DEBUG:boto:Params: {} 2023-10-11T15:14:04.803 DEBUG:boto:Token: None 2023-10-11T15:14:04.804 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:04.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V2uIltnKtqmZgQo0Pj0yycND9OQ= 2023-10-11T15:14:04.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V2uIltnKtqmZgQo0Pj0yycND9OQ='} 2023-10-11T15:14:04.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f3d0570a16c754c-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.851 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.852 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.852 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.852 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.853 DEBUG:boto:Data: 2023-10-11T15:14:04.853 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.854 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.854 DEBUG:boto:Params: {} 2023-10-11T15:14:04.855 DEBUG:boto:Token: None 2023-10-11T15:14:04.855 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad93fe08e7c8c8d6-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.860 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:04.861 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:04.861 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.862 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:04.862 DEBUG:boto:Data: 2023-10-11T15:14:04.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.863 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.864 DEBUG:boto:Params: {} 2023-10-11T15:14:04.864 DEBUG:boto:Token: None 2023-10-11T15:14:04.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:04.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LfkCuRQUC09meE74kvkSUud92kw= 2023-10-11T15:14:04.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LfkCuRQUC09meE74kvkSUud92kw='} 2023-10-11T15:14:04.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009276f111b3c00555-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.873 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.873 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.873 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.874 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.874 DEBUG:boto:Data: 2023-10-11T15:14:04.875 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.875 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.876 DEBUG:boto:Params: {} 2023-10-11T15:14:04.876 DEBUG:boto:Token: None 2023-10-11T15:14:04.877 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cae706005edd5e42-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.882 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:04.882 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:04.883 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.883 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:04.884 DEBUG:boto:Data: 2023-10-11T15:14:04.884 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.885 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.885 DEBUG:boto:Params: {} 2023-10-11T15:14:04.886 DEBUG:boto:Token: None 2023-10-11T15:14:04.886 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:04.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2sFyCK6xBDi7exBLFvMcbuH1Xo= 2023-10-11T15:14:04.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2sFyCK6xBDi7exBLFvMcbuH1Xo='} 2023-10-11T15:14:04.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b418bf3cb905c14-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.935 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.936 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.936 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.936 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.937 DEBUG:boto:Data: 2023-10-11T15:14:04.937 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.938 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.938 DEBUG:boto:Params: {} 2023-10-11T15:14:04.939 DEBUG:boto:Token: None 2023-10-11T15:14:04.939 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f7e4ab63abe64cd-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.945 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:04.945 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:04.946 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.946 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:04.947 DEBUG:boto:Data: 2023-10-11T15:14:04.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.948 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.948 DEBUG:boto:Params: {} 2023-10-11T15:14:04.949 DEBUG:boto:Token: None 2023-10-11T15:14:04.949 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:04.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5INd4iEJs5iHk7Y24bp1fUFKYG4= 2023-10-11T15:14:04.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5INd4iEJs5iHk7Y24bp1fUFKYG4='} 2023-10-11T15:14:04.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3b87b11da508ebd-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.957 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:04.958 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:04.958 DEBUG:boto:Method: HEAD 2023-10-11T15:14:04.958 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:04.959 DEBUG:boto:Data: 2023-10-11T15:14:04.959 DEBUG:boto:Headers: {} 2023-10-11T15:14:04.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.960 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.960 DEBUG:boto:Params: {} 2023-10-11T15:14:04.961 DEBUG:boto:Token: None 2023-10-11T15:14:04.961 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/ 2023-10-11T15:14:04.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk= 2023-10-11T15:14:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CFhNqD0DW5Cjey7OD0A+Fup6Wuk=', 'Content-Length': '0'} 2023-10-11T15:14:04.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023c837a74c5b3345-006526bbbc-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:04.966 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:04.967 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:04.967 DEBUG:boto:Method: PUT 2023-10-11T15:14:04.967 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:04.968 DEBUG:boto:Data: 2023-10-11T15:14:04.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:04.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:04.969 DEBUG:boto:Port: 8000 2023-10-11T15:14:04.969 DEBUG:boto:Params: {} 2023-10-11T15:14:04.970 DEBUG:boto:Token: None 2023-10-11T15:14:04.970 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:04 GMT /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:04.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kKy5O3ALi0T5eKTRahWLBVl2CXw= 2023-10-11T15:14:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:04 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kKy5O3ALi0T5eKTRahWLBVl2CXw='} 2023-10-11T15:14:05.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024412467d07624e6-006526bbbc-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.019 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.020 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.020 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.021 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.021 DEBUG:boto:Data: 2023-10-11T15:14:05.021 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.022 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.023 DEBUG:boto:Params: {} 2023-10-11T15:14:05.023 DEBUG:boto:Token: None 2023-10-11T15:14:05.024 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074843036d2e3d4a8-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.029 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:05.029 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:05.030 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.030 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:05.031 DEBUG:boto:Data: 2023-10-11T15:14:05.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.032 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.032 DEBUG:boto:Params: {} 2023-10-11T15:14:05.033 DEBUG:boto:Token: None 2023-10-11T15:14:05.033 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:05.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4rscLreX8UeWlD4N6xXQ2kE/wVk= 2023-10-11T15:14:05.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4rscLreX8UeWlD4N6xXQ2kE/wVk='} 2023-10-11T15:14:05.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3949c30f4e60526-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.041 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.041 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.042 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.042 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.043 DEBUG:boto:Data: 2023-10-11T15:14:05.043 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.044 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.044 DEBUG:boto:Params: {} 2023-10-11T15:14:05.045 DEBUG:boto:Token: None 2023-10-11T15:14:05.045 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bae1746163bc5590-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.050 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:05.051 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:05.051 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.052 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:05.052 DEBUG:boto:Data: 2023-10-11T15:14:05.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.053 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.054 DEBUG:boto:Params: {} 2023-10-11T15:14:05.054 DEBUG:boto:Token: None 2023-10-11T15:14:05.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:05.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QU3FF8hMq1qrcPMAwmmCKZnDsvc= 2023-10-11T15:14:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QU3FF8hMq1qrcPMAwmmCKZnDsvc='} 2023-10-11T15:14:05.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c39861d4841f5e0b-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.099 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.100 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.100 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.100 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.101 DEBUG:boto:Data: 2023-10-11T15:14:05.101 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.102 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.102 DEBUG:boto:Params: {} 2023-10-11T15:14:05.103 DEBUG:boto:Token: None 2023-10-11T15:14:05.103 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d1eb974c3c9f158-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.108 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:05.108 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:05.109 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.109 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:05.109 DEBUG:boto:Data: 2023-10-11T15:14:05.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.111 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.111 DEBUG:boto:Params: {} 2023-10-11T15:14:05.111 DEBUG:boto:Token: None 2023-10-11T15:14:05.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:05.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kb+gYoQ8AjbdmpW/1Efx+ZRUEac= 2023-10-11T15:14:05.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kb+gYoQ8AjbdmpW/1Efx+ZRUEac='} 2023-10-11T15:14:05.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049c5d22c0dbc96fc-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.118 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.119 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.119 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.119 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.120 DEBUG:boto:Data: 2023-10-11T15:14:05.120 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.121 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.122 DEBUG:boto:Params: {} 2023-10-11T15:14:05.122 DEBUG:boto:Token: None 2023-10-11T15:14:05.123 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e1058f58e3ff999-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.128 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:05.128 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:05.129 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.129 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:05.129 DEBUG:boto:Data: 2023-10-11T15:14:05.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.131 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.131 DEBUG:boto:Params: {} 2023-10-11T15:14:05.132 DEBUG:boto:Token: None 2023-10-11T15:14:05.132 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:05.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpcG8r0/zdZUR2KxGkmMTxGnMBA= 2023-10-11T15:14:05.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpcG8r0/zdZUR2KxGkmMTxGnMBA='} 2023-10-11T15:14:05.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000549b99f394c37844-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.179 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.180 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.180 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.181 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.181 DEBUG:boto:Data: 2023-10-11T15:14:05.181 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.182 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.183 DEBUG:boto:Params: {} 2023-10-11T15:14:05.183 DEBUG:boto:Token: None 2023-10-11T15:14:05.184 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059e1786cdbcdf94c-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.189 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:05.189 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:05.190 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.190 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:05.191 DEBUG:boto:Data: 2023-10-11T15:14:05.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.192 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.192 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.192 DEBUG:boto:Params: {} 2023-10-11T15:14:05.193 DEBUG:boto:Token: None 2023-10-11T15:14:05.193 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:05.194 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+AErKpZD79OS5F3IhtI+mW0wgIQ= 2023-10-11T15:14:05.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+AErKpZD79OS5F3IhtI+mW0wgIQ='} 2023-10-11T15:14:05.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068adbf29ab0d321c-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.202 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.202 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.203 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.203 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.204 DEBUG:boto:Data: 2023-10-11T15:14:05.204 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.205 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.205 DEBUG:boto:Params: {} 2023-10-11T15:14:05.206 DEBUG:boto:Token: None 2023-10-11T15:14:05.206 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009086f4cb5649451d-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.211 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:05.211 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:05.212 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.212 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:05.213 DEBUG:boto:Data: 2023-10-11T15:14:05.213 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.214 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.214 DEBUG:boto:Params: {} 2023-10-11T15:14:05.215 DEBUG:boto:Token: None 2023-10-11T15:14:05.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:05.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oYLP6EUGmmR+kx5FMvqEvnk/3b8= 2023-10-11T15:14:05.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oYLP6EUGmmR+kx5FMvqEvnk/3b8='} 2023-10-11T15:14:05.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090b40c98f6804f32-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.259 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.260 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.260 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.261 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.261 DEBUG:boto:Data: 2023-10-11T15:14:05.262 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.262 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.263 DEBUG:boto:Params: {} 2023-10-11T15:14:05.263 DEBUG:boto:Token: None 2023-10-11T15:14:05.264 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7a0e519c9c20598-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.270 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:05.270 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:05.271 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.271 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:05.272 DEBUG:boto:Data: 2023-10-11T15:14:05.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.273 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.273 DEBUG:boto:Params: {} 2023-10-11T15:14:05.274 DEBUG:boto:Token: None 2023-10-11T15:14:05.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:05.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tlIJdmhqvuJzKbqrrAEUMgdLwuo= 2023-10-11T15:14:05.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tlIJdmhqvuJzKbqrrAEUMgdLwuo='} 2023-10-11T15:14:05.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061140e9c5347b8e9-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.283 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.283 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.283 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.284 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.284 DEBUG:boto:Data: 2023-10-11T15:14:05.285 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.286 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.287 DEBUG:boto:Params: {} 2023-10-11T15:14:05.287 DEBUG:boto:Token: None 2023-10-11T15:14:05.287 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000436b6dea764e1109-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.292 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:05.293 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:05.293 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.294 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:05.294 DEBUG:boto:Data: 2023-10-11T15:14:05.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.295 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.296 DEBUG:boto:Params: {} 2023-10-11T15:14:05.296 DEBUG:boto:Token: None 2023-10-11T15:14:05.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:05.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kARgxrQK20O/A6rISWayIgh4/VE= 2023-10-11T15:14:05.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kARgxrQK20O/A6rISWayIgh4/VE='} 2023-10-11T15:14:05.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afa146e54be8b02e-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.343 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.344 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.344 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.345 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.345 DEBUG:boto:Data: 2023-10-11T15:14:05.345 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.346 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.347 DEBUG:boto:Params: {} 2023-10-11T15:14:05.347 DEBUG:boto:Token: None 2023-10-11T15:14:05.348 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076c4080d24cabcd2-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.352 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:05.353 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:05.353 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.354 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:05.354 DEBUG:boto:Data: 2023-10-11T15:14:05.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.355 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.356 DEBUG:boto:Params: {} 2023-10-11T15:14:05.356 DEBUG:boto:Token: None 2023-10-11T15:14:05.357 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:05.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X2n+VBqV6dl9KUzqB+v4oQVWmyU= 2023-10-11T15:14:05.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X2n+VBqV6dl9KUzqB+v4oQVWmyU='} 2023-10-11T15:14:05.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fdee74716c8793e9-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.364 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.365 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.365 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.366 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.366 DEBUG:boto:Data: 2023-10-11T15:14:05.366 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.367 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.368 DEBUG:boto:Params: {} 2023-10-11T15:14:05.368 DEBUG:boto:Token: None 2023-10-11T15:14:05.369 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e431798d478e063-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.374 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:05.374 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:05.375 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.375 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:05.376 DEBUG:boto:Data: 2023-10-11T15:14:05.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.377 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.378 DEBUG:boto:Params: {} 2023-10-11T15:14:05.378 DEBUG:boto:Token: None 2023-10-11T15:14:05.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:05.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:etMFyZgcLZJEQFdcJamOWAU8W6s= 2023-10-11T15:14:05.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:etMFyZgcLZJEQFdcJamOWAU8W6s='} 2023-10-11T15:14:05.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007cc42fc7f52af008-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.423 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.424 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.424 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.425 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.425 DEBUG:boto:Data: 2023-10-11T15:14:05.425 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.426 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.427 DEBUG:boto:Params: {} 2023-10-11T15:14:05.427 DEBUG:boto:Token: None 2023-10-11T15:14:05.428 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ed80438949bdbc5-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.433 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:05.433 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:05.434 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.434 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:05.434 DEBUG:boto:Data: 2023-10-11T15:14:05.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.436 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.436 DEBUG:boto:Params: {} 2023-10-11T15:14:05.437 DEBUG:boto:Token: None 2023-10-11T15:14:05.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:05.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gkwpDD2Bb4yP4VHgiOdtiCelUpI= 2023-10-11T15:14:05.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gkwpDD2Bb4yP4VHgiOdtiCelUpI='} 2023-10-11T15:14:05.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9c48343e1fd1d1b-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.445 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.446 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.446 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.447 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.447 DEBUG:boto:Data: 2023-10-11T15:14:05.448 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.448 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.449 DEBUG:boto:Params: {} 2023-10-11T15:14:05.449 DEBUG:boto:Token: None 2023-10-11T15:14:05.450 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0934512c6ee2756-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:05.455 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:05.456 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.456 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:05.457 DEBUG:boto:Data: 2023-10-11T15:14:05.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.458 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.458 DEBUG:boto:Params: {} 2023-10-11T15:14:05.459 DEBUG:boto:Token: None 2023-10-11T15:14:05.459 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:05.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m+c2mRrrbGt+wjAc02gTOxxYrUI= 2023-10-11T15:14:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m+c2mRrrbGt+wjAc02gTOxxYrUI='} 2023-10-11T15:14:05.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056f6c4edea1c1698-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.503 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.504 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.504 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.505 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.505 DEBUG:boto:Data: 2023-10-11T15:14:05.506 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.507 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.507 DEBUG:boto:Params: {} 2023-10-11T15:14:05.508 DEBUG:boto:Token: None 2023-10-11T15:14:05.508 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075fbd4c378ad4c46-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.514 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:05.515 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:05.515 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.515 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:05.516 DEBUG:boto:Data: 2023-10-11T15:14:05.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.517 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.517 DEBUG:boto:Params: {} 2023-10-11T15:14:05.518 DEBUG:boto:Token: None 2023-10-11T15:14:05.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:05.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W67VRuQikhRbgu/E/VsNKf6Ac/4= 2023-10-11T15:14:05.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W67VRuQikhRbgu/E/VsNKf6Ac/4='} 2023-10-11T15:14:05.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000746fc8f85c290102-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.526 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.527 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.527 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.527 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.528 DEBUG:boto:Data: 2023-10-11T15:14:05.528 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.529 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.529 DEBUG:boto:Params: {} 2023-10-11T15:14:05.530 DEBUG:boto:Token: None 2023-10-11T15:14:05.530 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc769e45ad4a60bd-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.535 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:05.535 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:05.536 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.536 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:05.537 DEBUG:boto:Data: 2023-10-11T15:14:05.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.538 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.538 DEBUG:boto:Params: {} 2023-10-11T15:14:05.539 DEBUG:boto:Token: None 2023-10-11T15:14:05.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:05.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eo3eRMhlxOp9ZKjzFqyDZzPibCw= 2023-10-11T15:14:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eo3eRMhlxOp9ZKjzFqyDZzPibCw='} 2023-10-11T15:14:05.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c90d17c950f819d0-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.584 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.584 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.584 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.585 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.585 DEBUG:boto:Data: 2023-10-11T15:14:05.586 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.586 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.587 DEBUG:boto:Params: {} 2023-10-11T15:14:05.587 DEBUG:boto:Token: None 2023-10-11T15:14:05.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000567337d6f2e2886b-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.593 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:05.594 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:05.594 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.595 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:05.595 DEBUG:boto:Data: 2023-10-11T15:14:05.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.596 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.597 DEBUG:boto:Params: {} 2023-10-11T15:14:05.597 DEBUG:boto:Token: None 2023-10-11T15:14:05.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:05.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ztqgcUjWR9c0iJbqkPdIV7v/UD8= 2023-10-11T15:14:05.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ztqgcUjWR9c0iJbqkPdIV7v/UD8='} 2023-10-11T15:14:05.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000964186a6f6dac6f0-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.604 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.605 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.605 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.606 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.606 DEBUG:boto:Data: 2023-10-11T15:14:05.606 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.607 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.608 DEBUG:boto:Params: {} 2023-10-11T15:14:05.608 DEBUG:boto:Token: None 2023-10-11T15:14:05.609 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ba772ac8e16bf8f-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.613 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:05.614 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:05.614 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.615 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:05.618 DEBUG:boto:Data: 2023-10-11T15:14:05.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.621 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.621 DEBUG:boto:Params: {} 2023-10-11T15:14:05.622 DEBUG:boto:Token: None 2023-10-11T15:14:05.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:05.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+Fj2n6++zSgFPuyNxebKcjP3F4A= 2023-10-11T15:14:05.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+Fj2n6++zSgFPuyNxebKcjP3F4A='} 2023-10-11T15:14:05.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e3d8fb3fe566fc3-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.671 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.672 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.672 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.673 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.673 DEBUG:boto:Data: 2023-10-11T15:14:05.673 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.674 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.675 DEBUG:boto:Params: {} 2023-10-11T15:14:05.675 DEBUG:boto:Token: None 2023-10-11T15:14:05.676 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa045b9c002721ae-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.681 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:05.682 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:05.682 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.683 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:05.683 DEBUG:boto:Data: 2023-10-11T15:14:05.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.684 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.684 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.685 DEBUG:boto:Params: {} 2023-10-11T15:14:05.685 DEBUG:boto:Token: None 2023-10-11T15:14:05.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:05.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6rRbR/Tm30XipFNOMli5g+a5eWA= 2023-10-11T15:14:05.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6rRbR/Tm30XipFNOMli5g+a5eWA='} 2023-10-11T15:14:05.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bb349b4e08b9d71-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.693 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.694 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.694 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.695 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.695 DEBUG:boto:Data: 2023-10-11T15:14:05.696 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.697 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.697 DEBUG:boto:Params: {} 2023-10-11T15:14:05.698 DEBUG:boto:Token: None 2023-10-11T15:14:05.698 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000486d09293b35f1bc-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.703 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:05.704 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:05.704 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.705 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:05.705 DEBUG:boto:Data: 2023-10-11T15:14:05.706 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.707 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.707 DEBUG:boto:Params: {} 2023-10-11T15:14:05.708 DEBUG:boto:Token: None 2023-10-11T15:14:05.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:05.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wnWaCAd9+VN2cPGilvJq0nbKoqQ= 2023-10-11T15:14:05.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wnWaCAd9+VN2cPGilvJq0nbKoqQ='} 2023-10-11T15:14:05.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033dd1e0c244b8bf6-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.755 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.755 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.756 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.756 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.757 DEBUG:boto:Data: 2023-10-11T15:14:05.757 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.758 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.759 DEBUG:boto:Params: {} 2023-10-11T15:14:05.759 DEBUG:boto:Token: None 2023-10-11T15:14:05.760 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079f2ab26edbc339a-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.765 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:05.765 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:05.766 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.766 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:05.767 DEBUG:boto:Data: 2023-10-11T15:14:05.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.767 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.768 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.768 DEBUG:boto:Params: {} 2023-10-11T15:14:05.769 DEBUG:boto:Token: None 2023-10-11T15:14:05.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:05.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PflvrZgGyAdw3er/LB19NSu/28A= 2023-10-11T15:14:05.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PflvrZgGyAdw3er/LB19NSu/28A='} 2023-10-11T15:14:05.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bd1f7ff7c9ad322-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.776 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.777 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.777 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.778 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.778 DEBUG:boto:Data: 2023-10-11T15:14:05.778 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.779 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.780 DEBUG:boto:Params: {} 2023-10-11T15:14:05.780 DEBUG:boto:Token: None 2023-10-11T15:14:05.780 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f57fa4594808c09-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.785 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:05.785 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:05.786 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.786 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:05.786 DEBUG:boto:Data: 2023-10-11T15:14:05.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.788 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.788 DEBUG:boto:Params: {} 2023-10-11T15:14:05.789 DEBUG:boto:Token: None 2023-10-11T15:14:05.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:05.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k1uKs9qlD62lUqCEHLAPNB9c9Ts= 2023-10-11T15:14:05.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k1uKs9qlD62lUqCEHLAPNB9c9Ts='} 2023-10-11T15:14:05.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006016d07df34a2794-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.836 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.836 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.837 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.837 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.837 DEBUG:boto:Data: 2023-10-11T15:14:05.838 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.839 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.839 DEBUG:boto:Params: {} 2023-10-11T15:14:05.840 DEBUG:boto:Token: None 2023-10-11T15:14:05.840 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.841 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a12cc9f9824fc122-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.845 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:05.845 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:05.846 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.846 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:05.847 DEBUG:boto:Data: 2023-10-11T15:14:05.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.848 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.848 DEBUG:boto:Params: {} 2023-10-11T15:14:05.849 DEBUG:boto:Token: None 2023-10-11T15:14:05.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:05.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D9BgdBoN6so69tb/YtnKz2LXKJU= 2023-10-11T15:14:05.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D9BgdBoN6so69tb/YtnKz2LXKJU='} 2023-10-11T15:14:05.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ba1f3fdbcac93f3-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.857 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:05.857 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:05.858 DEBUG:boto:Method: HEAD 2023-10-11T15:14:05.858 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:05.859 DEBUG:boto:Data: 2023-10-11T15:14:05.859 DEBUG:boto:Headers: {} 2023-10-11T15:14:05.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.860 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.860 DEBUG:boto:Params: {} 2023-10-11T15:14:05.861 DEBUG:boto:Token: None 2023-10-11T15:14:05.861 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/ 2023-10-11T15:14:05.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps= 2023-10-11T15:14:05.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jPcVUFWMwF3P7HrvC9yNVUUBwps=', 'Content-Length': '0'} 2023-10-11T15:14:05.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ba43ed389a1c83b-006526bbbd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.866 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:05.867 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:05.867 DEBUG:boto:Method: PUT 2023-10-11T15:14:05.868 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:05.868 DEBUG:boto:Data: 2023-10-11T15:14:05.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:14:05.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:05.869 DEBUG:boto:Port: 8000 2023-10-11T15:14:05.870 DEBUG:boto:Params: {} 2023-10-11T15:14:05.870 DEBUG:boto:Token: None 2023-10-11T15:14:05.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:14:05 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:05.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4GFFi8TwiKt/KUmlvG2LaDtA6Jk= 2023-10-11T15:14:05.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:14:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4GFFi8TwiKt/KUmlvG2LaDtA6Jk='} 2023-10-11T15:14:05.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ecb2b5fadecbd07b-006526bbbd-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:05.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:05.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:14:05.920 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:14:05.972 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:05.972 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:06.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.036 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 2 2023-10-11T15:14:06.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000001.34.6 2023-10-11T15:14:06.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 2 2023-10-11T15:14:06.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.032+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000001.26.6 2023-10-11T15:14:06.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 2 2023-10-11T15:14:06.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-11T15:14:06.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.038 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 2 2023-10-11T15:14:06.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000001.31.6 2023-10-11T15:14:06.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 2 2023-10-11T15:14:06.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000001.31.6 2023-10-11T15:14:06.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.033+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 2 2023-10-11T15:14:06.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.034+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000001.74.6 2023-10-11T15:14:06.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 2 2023-10-11T15:14:06.042 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000001.29.6 2023-10-11T15:14:06.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 2 2023-10-11T15:14:06.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 2 2023-10-11T15:14:06.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 2 2023-10-11T15:14:06.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000001.52.6 2023-10-11T15:14:06.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000001.46.6 2023-10-11T15:14:06.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.035+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-11T15:14:06.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.036+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 2 2023-10-11T15:14:06.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.036+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.090+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.090+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.090+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.090+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.091+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.092+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:06.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.093+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:06.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.094+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.095+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.095+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.095+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.095+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000010 2023-10-11T15:14:06.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:13:59.405189+0000 2023-10-11T15:14:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-10-11T15:14:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000010[0] 2023-10-11T15:14:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000002.80.6 2023-10-11T15:14:06.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000019 2023-10-11T15:14:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:13:59.329391+0000 2023-10-11T15:14:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-11T15:14:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-10-11T15:14:06.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-11T15:14:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.096+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000019 2023-10-11T15:14:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-11T15:14:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000013 2023-10-11T15:14:06.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:13:59.113712+0000 2023-10-11T15:14:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-11T15:14:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-11T15:14:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2/obj00000000000000000024 2023-10-11T15:14:06.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: entry->timestamp=2023-10-11T15:13:58.947932+0000 2023-10-11T15:14:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-11T15:14:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-11T15:14:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000030 2023-10-11T15:14:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:13:59.032075+0000 2023-10-11T15:14:06.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-10-11T15:14:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-11T15:14:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13/obj00000000000000000026 2023-10-11T15:14:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: entry->timestamp=2023-10-11T15:13:59.270391+0000 2023-10-11T15:14:06.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-10-11T15:14:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000013[0] 2023-10-11T15:14:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-11T15:14:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000024[0] 2023-10-11T15:14:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-11T15:14:06.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000030[0] 2023-10-11T15:14:06.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000001.60.6 2023-10-11T15:14:06.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000026[0] 2023-10-11T15:14:06.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position=00000000001.57.6 2023-10-11T15:14:06.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000001.14.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000021 2023-10-11T15:14:06.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:13:59.010567+0000 2023-10-11T15:14:06.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.14.6 op=0 op_state=1 2023-10-11T15:14:06.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:06.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.097+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000001.14.6 2023-10-11T15:14:06.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:06.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000001 2023-10-11T15:14:06.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:13:59.500219+0000 2023-10-11T15:14:06.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.21.6 op=0 op_state=1 2023-10-11T15:14:06.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-11T15:14:06.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000004 2023-10-11T15:14:06.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:13:59.348383+0000 2023-10-11T15:14:06.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-11T15:14:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2023-10-11T15:14:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000002.21.6 2023-10-11T15:14:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000004[0] 2023-10-11T15:14:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-11T15:14:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.098+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000001 2023-10-11T15:14:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-11T15:14:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000033 2023-10-11T15:14:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:13:59.250098+0000 2023-10-11T15:14:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.62.6 op=0 op_state=1 2023-10-11T15:14:06.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000033[0] 2023-10-11T15:14:06.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.099+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000001.62.6 2023-10-11T15:14:06.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000026]: success 2023-10-11T15:14:06.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13:1 marker=00000000001.57.6 timestamp=2023-10-11T15:13:59.270391+0000 2023-10-11T15:14:06.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000024]: success 2023-10-11T15:14:06.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2:1 marker=00000000001.58.6 timestamp=2023-10-11T15:13:58.947932+0000 2023-10-11T15:14:06.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000004]: success 2023-10-11T15:14:06.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:1 marker=00000000002.50.6 timestamp=2023-10-11T15:13:59.348383+0000 2023-10-11T15:14:06.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000013]: success 2023-10-11T15:14:06.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.111+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10:1 marker=00000000002.35.6 timestamp=2023-10-11T15:13:59.113712+0000 2023-10-11T15:14:06.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.114+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000030]: success 2023-10-11T15:14:06.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.114+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:1 marker=00000000001.60.6 timestamp=2023-10-11T15:13:59.032075+0000 2023-10-11T15:14:06.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.114+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000010]: success 2023-10-11T15:14:06.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.114+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:1 marker=00000000002.80.6 timestamp=2023-10-11T15:13:59.405189+0000 2023-10-11T15:14:06.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000001.27.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-11T15:14:06.136 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000016 2023-10-11T15:14:06.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:13:59.190560+0000 2023-10-11T15:14:06.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.27.6 op=0 op_state=1 2023-10-11T15:14:06.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000016[0] 2023-10-11T15:14:06.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000001.27.6 2023-10-11T15:14:06.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:06.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000007 2023-10-11T15:14:06.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:13:58.929259+0000 2023-10-11T15:14:06.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-11T15:14:06.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-11T15:14:06.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000020 2023-10-11T15:14:06.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:13:59.481398+0000 2023-10-11T15:14:06.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.82.6 op=0 op_state=1 2023-10-11T15:14:06.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-11T15:14:06.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000032 2023-10-11T15:14:06.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:13:59.425426+0000 2023-10-11T15:14:06.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-10-11T15:14:06.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2023-10-11T15:14:06.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-11T15:14:06.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000020[0] 2023-10-11T15:14:06.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000001.82.6 2023-10-11T15:14:06.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: : injecting data sync error on key=obj00000000000000000032 2023-10-11T15:14:06.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000001.52.6 2023-10-11T15:14:06.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000007 2023-10-11T15:14:06.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.133+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000032 2023-10-11T15:14:06.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-10-11T15:14:06.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4/obj00000000000000000028 2023-10-11T15:14:06.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: entry->timestamp=2023-10-11T15:13:59.170304+0000 2023-10-11T15:14:06.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-10-11T15:14:06.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000028[0] 2023-10-11T15:14:06.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.136+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-11T15:14:06.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-10-11T15:14:06.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16/obj00000000000000000076 2023-10-11T15:14:06.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: entry->timestamp=2023-10-11T15:13:59.090821+0000 2023-10-11T15:14:06.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-10-11T15:14:06.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000076[0] 2023-10-11T15:14:06.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-11T15:14:06.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: a sync operation returned error 2023-10-11T15:14:06.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: backing out with sync_status=-5 2023-10-11T15:14:06.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.139+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.140+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.140+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.142+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: a sync operation returned error 2023-10-11T15:14:06.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.142+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: backing out with sync_status=-5 2023-10-11T15:14:06.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.142+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.142+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.144+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.144+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.144+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.153+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000021]: success 2023-10-11T15:14:06.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.153+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:1 marker=00000000001.14.6 timestamp=2023-10-11T15:13:59.010567+0000 2023-10-11T15:14:06.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.154+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000033]: success 2023-10-11T15:14:06.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.154+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:1 marker=00000000001.62.6 timestamp=2023-10-11T15:13:59.250098+0000 2023-10-11T15:14:06.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: a sync operation returned error 2023-10-11T15:14:06.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: a sync operation returned error 2023-10-11T15:14:06.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: backing out with sync_status=-5 2023-10-11T15:14:06.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: backing out with sync_status=-5 2023-10-11T15:14:06.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.176+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.177+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.177+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.177+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.177+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.177+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000028]: success 2023-10-11T15:14:06.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4:1 marker=00000000001.56.6 timestamp=2023-10-11T15:13:59.170304+0000 2023-10-11T15:14:06.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000016]: success 2023-10-11T15:14:06.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:1 marker=00000000001.27.6 timestamp=2023-10-11T15:13:59.190560+0000 2023-10-11T15:14:06.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000020]: success 2023-10-11T15:14:06.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.181+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:1 marker=00000000001.82.6 timestamp=2023-10-11T15:13:59.481398+0000 2023-10-11T15:14:06.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000019 2023-10-11T15:14:06.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:13:59.329391+0000 2023-10-11T15:14:06.186 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-11T15:14:06.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-10-11T15:14:06.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-11T15:14:06.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.187 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.183+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000019 2023-10-11T15:14:06.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.185+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000076]: success 2023-10-11T15:14:06.188 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.185+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16:1 marker=00000000001.41.6 timestamp=2023-10-11T15:13:59.090821+0000 2023-10-11T15:14:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000001 2023-10-11T15:14:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:13:59.500219+0000 2023-10-11T15:14:06.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.21.6 op=0 op_state=1 2023-10-11T15:14:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000001[0] 2023-10-11T15:14:06.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.186+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000002.21.6 2023-10-11T15:14:06.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-11T15:14:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000032 2023-10-11T15:14:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:13:59.425426+0000 2023-10-11T15:14:06.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-10-11T15:14:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000032[0] 2023-10-11T15:14:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.219+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000001.52.6 2023-10-11T15:14:06.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:06.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000007 2023-10-11T15:14:06.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:13:58.929259+0000 2023-10-11T15:14:06.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-10-11T15:14:06.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000007[0] 2023-10-11T15:14:06.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.220+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000001.39.6 2023-10-11T15:14:06.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.223+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: success 2023-10-11T15:14:06.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.223+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:1 marker=00000000001.52.6 timestamp=2023-10-11T15:13:59.425426+0000 2023-10-11T15:14:06.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.224+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000007]: success 2023-10-11T15:14:06.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.224+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:1 marker=00000000001.39.6 timestamp=2023-10-11T15:13:58.929259+0000 2023-10-11T15:14:06.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: a sync operation returned error 2023-10-11T15:14:06.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: backing out with sync_status=-5 2023-10-11T15:14:06.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.226+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.232+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000001]: success 2023-10-11T15:14:06.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.232+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:1 marker=00000000002.21.6 timestamp=2023-10-11T15:13:59.500219+0000 2023-10-11T15:14:06.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000019 2023-10-11T15:14:06.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:13:59.329391+0000 2023-10-11T15:14:06.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-11T15:14:06.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000019[0] 2023-10-11T15:14:06.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.268+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-11T15:14:06.314 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.312+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000019]: success 2023-10-11T15:14:06.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.312+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:1 marker=00000000001.43.6 timestamp=2023-10-11T15:13:59.329391+0000 2023-10-11T15:14:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.316+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.319 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.320 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.317+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.318+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:06.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.318+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.325 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.326 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:06.327 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-11T15:14:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000005 2023-10-11T15:14:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:13:59.997657+0000 2023-10-11T15:14:06.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-10-11T15:14:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-11T15:14:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4/obj00000000000000000004 2023-10-11T15:14:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: entry->timestamp=2023-10-11T15:14:00.416647+0000 2023-10-11T15:14:06.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-10-11T15:14:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000005[0] 2023-10-11T15:14:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000002.36.6 2023-10-11T15:14:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000004[0] 2023-10-11T15:14:06.330 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000002.32.6 2023-10-11T15:14:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.319+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.331 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.320+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.332 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:06.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:06.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.333 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.334 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.335 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:06.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.321+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:06.336 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:06.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-11T15:14:06.337 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000015 2023-10-11T15:14:06.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:13:59.921224+0000 2023-10-11T15:14:06.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-10-11T15:14:06.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-11T15:14:06.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000030 2023-10-11T15:14:06.338 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:13:59.859572+0000 2023-10-11T15:14:06.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-10-11T15:14:06.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-11T15:14:06.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000009 2023-10-11T15:14:06.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:00.337921+0000 2023-10-11T15:14:06.339 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-11T15:14:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000004]: success 2023-10-11T15:14:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4:2 marker=00000000002.32.6 timestamp=2023-10-11T15:14:00.416647+0000 2023-10-11T15:14:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000015[0] 2023-10-11T15:14:06.340 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000002.30.6 2023-10-11T15:14:06.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000030[0] 2023-10-11T15:14:06.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.33.6 2023-10-11T15:14:06.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000009[0] 2023-10-11T15:14:06.341 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-11T15:14:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket shard: 0 2023-10-11T15:14:06.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.323+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.324+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.324+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000003 2023-10-11T15:14:06.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:14:00.017014+0000 2023-10-11T15:14:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-11T15:14:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000003[0] 2023-10-11T15:14:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-11T15:14:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:06.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000007 2023-10-11T15:14:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:13:59.800213+0000 2023-10-11T15:14:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-11T15:14:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-11T15:14:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000000 2023-10-11T15:14:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:00.236376+0000 2023-10-11T15:14:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-11T15:14:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17/obj00000000000000000001 2023-10-11T15:14:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: entry->timestamp=2023-10-11T15:14:00.257941+0000 2023-10-11T15:14:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-11T15:14:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000007[0] 2023-10-11T15:14:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-11T15:14:06.348 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000000[0] 2023-10-11T15:14:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-11T15:14:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000006 2023-10-11T15:14:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:14:00.395034+0000 2023-10-11T15:14:06.349 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-11T15:14:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000030]: success 2023-10-11T15:14:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:2 marker=00000000002.33.6 timestamp=2023-10-11T15:13:59.859572+0000 2023-10-11T15:14:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000001[0] 2023-10-11T15:14:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-11T15:14:06.350 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000006[0] 2023-10-11T15:14:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.327+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-11T15:14:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.331+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000007]: success 2023-10-11T15:14:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.331+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18:2 marker=00000000002.47.6 timestamp=2023-10-11T15:13:59.800213+0000 2023-10-11T15:14:06.351 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.332+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: success 2023-10-11T15:14:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.332+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17:2 marker=00000000002.37.6 timestamp=2023-10-11T15:14:00.257941+0000 2023-10-11T15:14:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.333+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000005]: success 2023-10-11T15:14:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.333+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:2 marker=00000000002.36.6 timestamp=2023-10-11T15:13:59.997657+0000 2023-10-11T15:14:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.335+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000009]: success 2023-10-11T15:14:06.352 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.335+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10:2 marker=00000000001.61.6 timestamp=2023-10-11T15:14:00.337921+0000 2023-10-11T15:14:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.339+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000003]: success 2023-10-11T15:14:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.339+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:2 marker=00000000002.35.6 timestamp=2023-10-11T15:14:00.017014+0000 2023-10-11T15:14:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.339+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000006]: success 2023-10-11T15:14:06.353 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.339+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:2 marker=00000000002.54.6 timestamp=2023-10-11T15:14:00.395034+0000 2023-10-11T15:14:06.361 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-11T15:14:06.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2/obj00000000000000000016 2023-10-11T15:14:06.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: entry->timestamp=2023-10-11T15:13:59.819998+0000 2023-10-11T15:14:06.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-11T15:14:06.362 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000016[0] 2023-10-11T15:14:06.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.359+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-11T15:14:06.363 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.360+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000016]: success 2023-10-11T15:14:06.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.360+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2:2 marker=00000000002.50.6 timestamp=2023-10-11T15:13:59.819998+0000 2023-10-11T15:14:06.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000021 2023-10-11T15:14:06.364 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:13:59.939847+0000 2023-10-11T15:14:06.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-11T15:14:06.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:06.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.361+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-11T15:14:06.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-11T15:14:06.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000025 2023-10-11T15:14:06.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: entry->timestamp=2023-10-11T15:13:59.840670+0000 2023-10-11T15:14:06.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-10-11T15:14:06.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000025[0] 2023-10-11T15:14:06.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-11T15:14:06.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000021]: success 2023-10-11T15:14:06.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.362+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:2 marker=00000000002.31.6 timestamp=2023-10-11T15:13:59.939847+0000 2023-10-11T15:14:06.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000019 2023-10-11T15:14:06.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:00.318179+0000 2023-10-11T15:14:06.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-11T15:14:06.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000019[0] 2023-10-11T15:14:06.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-11T15:14:06.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000001.19.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-11T15:14:06.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000014 2023-10-11T15:14:06.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:14:00.174344+0000 2023-10-11T15:14:06.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.19.6 op=0 op_state=1 2023-10-11T15:14:06.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-11T15:14:06.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000012 2023-10-11T15:14:06.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:14:00.092957+0000 2023-10-11T15:14:06.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-10-11T15:14:06.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000014[0] 2023-10-11T15:14:06.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000001.19.6 2023-10-11T15:14:06.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000012[0] 2023-10-11T15:14:06.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.366+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000002.66.6 2023-10-11T15:14:06.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-11T15:14:06.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000010 2023-10-11T15:14:06.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:00.153109+0000 2023-10-11T15:14:06.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.24.6 op=0 op_state=1 2023-10-11T15:14:06.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000010[0] 2023-10-11T15:14:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000002.24.6 2023-10-11T15:14:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16/obj00000000000000000011 2023-10-11T15:14:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: entry->timestamp=2023-10-11T15:14:00.073552+0000 2023-10-11T15:14:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-10-11T15:14:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000011[0] 2023-10-11T15:14:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.367+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position=00000000002.33.6 2023-10-11T15:14:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.369+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000019]: success 2023-10-11T15:14:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.369+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:2 marker=00000000002.45.6 timestamp=2023-10-11T15:14:00.318179+0000 2023-10-11T15:14:06.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.370+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000010]: success 2023-10-11T15:14:06.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.370+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:2 marker=00000000002.24.6 timestamp=2023-10-11T15:14:00.153109+0000 2023-10-11T15:14:06.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.372+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000015]: success 2023-10-11T15:14:06.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.372+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:2 marker=00000000002.30.6 timestamp=2023-10-11T15:13:59.921224+0000 2023-10-11T15:14:06.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.375+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000000]: success 2023-10-11T15:14:06.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.375+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:2 marker=00000000002.61.6 timestamp=2023-10-11T15:14:00.236376+0000 2023-10-11T15:14:06.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.406+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000025]: success 2023-10-11T15:14:06.409 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.406+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5:2 marker=00000000001.41.6 timestamp=2023-10-11T15:13:59.840670+0000 2023-10-11T15:14:06.415 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.413+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000012]: success 2023-10-11T15:14:06.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.413+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:2 marker=00000000002.66.6 timestamp=2023-10-11T15:14:00.092957+0000 2023-10-11T15:14:06.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.413+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000014]: success 2023-10-11T15:14:06.416 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.413+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:2 marker=00000000001.19.6 timestamp=2023-10-11T15:14:00.174344+0000 2023-10-11T15:14:06.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.414+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000011]: success 2023-10-11T15:14:06.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.414+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16:2 marker=00000000002.33.6 timestamp=2023-10-11T15:14:00.073552+0000 2023-10-11T15:14:06.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.422+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.425 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.422+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.422+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.422+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.422+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.426 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.427 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:06.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.428 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.429 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.423+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.424+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.424+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.424+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.424+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.430 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:06.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.431 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.432 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.433 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.425+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.434 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.435 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:06.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.437 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:06.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:06.438 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.426+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.440 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:06.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket shard: 0 2023-10-11T15:14:06.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.427+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:06.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.442 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:06.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:06.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket shard: 0 2023-10-11T15:14:06.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket shard: 0 2023-10-11T15:14:06.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.443 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.428+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket shard: 0 2023-10-11T15:14:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.444 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000006 2023-10-11T15:14:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:01.420848+0000 2023-10-11T15:14:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-10-11T15:14:06.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-11T15:14:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000040 2023-10-11T15:14:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:14:01.401435+0000 2023-10-11T15:14:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-11T15:14:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-11T15:14:06.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000009 2023-10-11T15:14:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:14:00.919710+0000 2023-10-11T15:14:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-11T15:14:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-11T15:14:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000012 2023-10-11T15:14:06.447 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:14:01.320561+0000 2023-10-11T15:14:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-10-11T15:14:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000006[0] 2023-10-11T15:14:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.69.6 2023-10-11T15:14:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000040[0] 2023-10-11T15:14:06.448 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-11T15:14:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000009[0] 2023-10-11T15:14:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-11T15:14:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000012[0] 2023-10-11T15:14:06.449 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000002.37.6 2023-10-11T15:14:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket shard: 0 2023-10-11T15:14:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.429+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000007 2023-10-11T15:14:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:14:00.756076+0000 2023-10-11T15:14:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-11T15:14:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000007[0] 2023-10-11T15:14:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.430+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-11T15:14:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-11T15:14:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13/obj00000000000000000045 2023-10-11T15:14:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: entry->timestamp=2023-10-11T15:14:01.063436+0000 2023-10-11T15:14:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-10-11T15:14:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000045[0] 2023-10-11T15:14:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-11T15:14:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20/obj00000000000000000003 2023-10-11T15:14:06.454 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: entry->timestamp=2023-10-11T15:14:01.148028+0000 2023-10-11T15:14:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-10-11T15:14:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-11T15:14:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22/obj00000000000000000008 2023-10-11T15:14:06.455 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: entry->timestamp=2023-10-11T15:14:00.778019+0000 2023-10-11T15:14:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-10-11T15:14:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19/obj00000000000000000021 2023-10-11T15:14:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: entry->timestamp=2023-10-11T15:14:01.503324+0000 2023-10-11T15:14:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-10-11T15:14:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000003[0] 2023-10-11T15:14:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position=00000000002.47.6 2023-10-11T15:14:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000008[0] 2023-10-11T15:14:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position=00000000002.35.6 2023-10-11T15:14:06.457 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.432+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position=00000000002.72.6 2023-10-11T15:14:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.433+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000009]: success 2023-10-11T15:14:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.433+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:3 marker=00000000002.71.6 timestamp=2023-10-11T15:14:00.919710+0000 2023-10-11T15:14:06.458 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.433+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000006]: success 2023-10-11T15:14:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.433+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10:3 marker=00000000002.69.6 timestamp=2023-10-11T15:14:01.420848+0000 2023-10-11T15:14:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17/obj00000000000000000001 2023-10-11T15:14:06.459 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: entry->timestamp=2023-10-11T15:14:01.479707+0000 2023-10-11T15:14:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-10-11T15:14:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000001[0] 2023-10-11T15:14:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position=00000000002.29.6 2023-10-11T15:14:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000007]: success 2023-10-11T15:14:06.460 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.434+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:3 marker=00000000002.51.6 timestamp=2023-10-11T15:14:00.756076+0000 2023-10-11T15:14:06.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.435+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000012]: success 2023-10-11T15:14:06.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.435+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:3 marker=00000000002.37.6 timestamp=2023-10-11T15:14:01.320561+0000 2023-10-11T15:14:06.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.436+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000003]: success 2023-10-11T15:14:06.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.436+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20:3 marker=00000000002.47.6 timestamp=2023-10-11T15:14:01.148028+0000 2023-10-11T15:14:06.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.438+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000001]: success 2023-10-11T15:14:06.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.438+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17:3 marker=00000000002.29.6 timestamp=2023-10-11T15:14:01.479707+0000 2023-10-11T15:14:06.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.439+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000021]: success 2023-10-11T15:14:06.462 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.439+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19:3 marker=00000000002.72.6 timestamp=2023-10-11T15:14:01.503324+0000 2023-10-11T15:14:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000001.75.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-10-11T15:14:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000058 2023-10-11T15:14:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:14:01.340678+0000 2023-10-11T15:14:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.75.6 op=0 op_state=1 2023-10-11T15:14:06.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000058[0] 2023-10-11T15:14:06.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000001.75.6 2023-10-11T15:14:06.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-10-11T15:14:06.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000026 2023-10-11T15:14:06.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:00.732689+0000 2023-10-11T15:14:06.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-10-11T15:14:06.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000026[0] 2023-10-11T15:14:06.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.465+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000002.88.6 2023-10-11T15:14:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000001.107.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-10-11T15:14:06.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000046 2023-10-11T15:14:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:14:01.044242+0000 2023-10-11T15:14:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.107.6 op=0 op_state=1 2023-10-11T15:14:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000046[0] 2023-10-11T15:14:06.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000001.107.6 2023-10-11T15:14:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-10-11T15:14:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000032 2023-10-11T15:14:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:14:01.259150+0000 2023-10-11T15:14:06.473 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-11T15:14:06.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:06.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2/obj00000000000000000011 2023-10-11T15:14:06.474 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: entry->timestamp=2023-10-11T15:14:00.837746+0000 2023-10-11T15:14:06.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-11T15:14:06.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000032[0] 2023-10-11T15:14:06.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-11T15:14:06.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000011[0] 2023-10-11T15:14:06.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.466+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-11T15:14:06.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-11T15:14:06.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000039 2023-10-11T15:14:06.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:01.235534+0000 2023-10-11T15:14:06.476 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.73.6 op=0 op_state=1 2023-10-11T15:14:06.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-10-11T15:14:06.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000002 2023-10-11T15:14:06.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:14:01.128132+0000 2023-10-11T15:14:06.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-10-11T15:14:06.477 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000039[0] 2023-10-11T15:14:06.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000001.73.6 2023-10-11T15:14:06.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000002[0] 2023-10-11T15:14:06.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.467+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000002.39.6 2023-10-11T15:14:06.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.468+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000026]: success 2023-10-11T15:14:06.479 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.468+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:3 marker=00000000002.88.6 timestamp=2023-10-11T15:14:00.732689+0000 2023-10-11T15:14:06.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.469+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: success 2023-10-11T15:14:06.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.469+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2:3 marker=00000000002.43.6 timestamp=2023-10-11T15:14:00.837746+0000 2023-10-11T15:14:06.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.471+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000032]: success 2023-10-11T15:14:06.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.471+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:3 marker=00000000002.71.6 timestamp=2023-10-11T15:14:01.259150+0000 2023-10-11T15:14:06.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-11T15:14:06.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000050 2023-10-11T15:14:06.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: entry->timestamp=2023-10-11T15:14:01.015710+0000 2023-10-11T15:14:06.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-10-11T15:14:06.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000050[0] 2023-10-11T15:14:06.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.472+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-11T15:14:06.482 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-11T15:14:06.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000013 2023-10-11T15:14:06.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:00.859284+0000 2023-10-11T15:14:06.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-11T15:14:06.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000013[0] 2023-10-11T15:14:06.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-11T15:14:06.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-11T15:14:06.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16/obj00000000000000000033 2023-10-11T15:14:06.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: entry->timestamp=2023-10-11T15:14:00.946731+0000 2023-10-11T15:14:06.484 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-10-11T15:14:06.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000033[0] 2023-10-11T15:14:06.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.473+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position=00000000002.26.6 2023-10-11T15:14:06.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-11T15:14:06.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000000 2023-10-11T15:14:06.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:14:01.170777+0000 2023-10-11T15:14:06.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-11T15:14:06.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000000[0] 2023-10-11T15:14:06.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.474+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-11T15:14:06.486 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000039]: success 2023-10-11T15:14:06.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:3 marker=00000000001.73.6 timestamp=2023-10-11T15:14:01.235534+0000 2023-10-11T15:14:06.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000013]: success 2023-10-11T15:14:06.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:3 marker=00000000002.42.6 timestamp=2023-10-11T15:14:00.859284+0000 2023-10-11T15:14:06.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000033]: success 2023-10-11T15:14:06.487 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.475+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16:3 marker=00000000002.26.6 timestamp=2023-10-11T15:14:00.946731+0000 2023-10-11T15:14:06.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.476+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000000]: success 2023-10-11T15:14:06.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.476+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18:3 marker=00000000002.70.6 timestamp=2023-10-11T15:14:01.170777+0000 2023-10-11T15:14:06.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.479+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000040]: success 2023-10-11T15:14:06.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.480+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:3 marker=00000000001.49.6 timestamp=2023-10-11T15:14:01.401435+0000 2023-10-11T15:14:06.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.485+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000008]: success 2023-10-11T15:14:06.489 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.485+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22:3 marker=00000000002.35.6 timestamp=2023-10-11T15:14:00.778019+0000 2023-10-11T15:14:06.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.485+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000045]: success 2023-10-11T15:14:06.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.485+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13:3 marker=00000000001.45.6 timestamp=2023-10-11T15:14:01.063436+0000 2023-10-11T15:14:06.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.510+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000058]: success 2023-10-11T15:14:06.513 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.510+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:3 marker=00000000001.75.6 timestamp=2023-10-11T15:14:01.340678+0000 2023-10-11T15:14:06.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.514+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000046]: success 2023-10-11T15:14:06.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.514+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:3 marker=00000000001.107.6 timestamp=2023-10-11T15:14:01.044242+0000 2023-10-11T15:14:06.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.514+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000002]: success 2023-10-11T15:14:06.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.514+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:3 marker=00000000002.39.6 timestamp=2023-10-11T15:14:01.128132+0000 2023-10-11T15:14:06.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.520+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000050]: success 2023-10-11T15:14:06.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.520+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5:3 marker=00000000001.67.6 timestamp=2023-10-11T15:14:01.015710+0000 2023-10-11T15:14:06.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.533+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.533+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.533+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.534+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:06.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.535+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:06.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:06.546 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.547 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.548 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.536+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.537+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.549 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.537+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:06.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:06.550 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.551 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.538+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:06.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.552 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.553 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket shard: 0 2023-10-11T15:14:06.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.539+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket shard: 0 2023-10-11T15:14:06.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:06.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket shard: 0 2023-10-11T15:14:06.554 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:06.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket shard: 0 2023-10-11T15:14:06.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket shard: 0 2023-10-11T15:14:06.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket shard: 0 2023-10-11T15:14:06.555 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket shard: 0 2023-10-11T15:14:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket shard: 0 2023-10-11T15:14:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket shard: 0 2023-10-11T15:14:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket shard: 0 2023-10-11T15:14:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket shard: 0 2023-10-11T15:14:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-11T15:14:06.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000017 2023-10-11T15:14:06.559 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:01.843422+0000 2023-10-11T15:14:06.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.540+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-10-11T15:14:06.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.541+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000017[0] 2023-10-11T15:14:06.560 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.541+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000001.42.6 2023-10-11T15:14:06.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-11T15:14:06.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000020 2023-10-11T15:14:06.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:01.924163+0000 2023-10-11T15:14:06.563 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000001.74.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-10-11T15:14:06.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000037 2023-10-11T15:14:06.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:14:02.275683+0000 2023-10-11T15:14:06.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.74.6 op=0 op_state=1 2023-10-11T15:14:06.564 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-11T15:14:06.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16/obj00000000000000000005 2023-10-11T15:14:06.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: entry->timestamp=2023-10-11T15:14:02.455969+0000 2023-10-11T15:14:06.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-11T15:14:06.565 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: : injecting data sync error on key=obj00000000000000000020 2023-10-11T15:14:06.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000037[0] 2023-10-11T15:14:06.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000001.74.6 2023-10-11T15:14:06.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000005[0] 2023-10-11T15:14:06.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-11T15:14:06.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.542+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000020 2023-10-11T15:14:06.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:06.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000003 2023-10-11T15:14:06.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:14:01.823024+0000 2023-10-11T15:14:06.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-10-11T15:14:06.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-10-11T15:14:06.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17/obj00000000000000000012 2023-10-11T15:14:06.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: entry->timestamp=2023-10-11T15:14:02.106669+0000 2023-10-11T15:14:06.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-11T15:14:06.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000003 2023-10-11T15:14:06.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000012[0] 2023-10-11T15:14:06.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.543+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-11T15:14:06.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-11T15:14:06.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26/obj00000000000000000014 2023-10-11T15:14:06.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: entry->timestamp=2023-10-11T15:14:02.213609+0000 2023-10-11T15:14:06.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-11T15:14:06.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28/obj00000000000000000021 2023-10-11T15:14:06.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: entry->timestamp=2023-10-11T15:14:02.798825+0000 2023-10-11T15:14:06.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-11T15:14:06.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-11T15:14:06.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24/obj00000000000000000016 2023-10-11T15:14:06.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: entry->timestamp=2023-10-11T15:14:02.538233+0000 2023-10-11T15:14:06.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-11T15:14:06.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: parsed entry: id=00000000002.20.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-11T15:14:06.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27/obj00000000000000000030 2023-10-11T15:14:06.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: entry->timestamp=2023-10-11T15:14:02.047194+0000 2023-10-11T15:14:06.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.20.6 op=0 op_state=1 2023-10-11T15:14:06.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: : injecting data sync error on key=obj00000000000000000014 2023-10-11T15:14:06.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-11T15:14:06.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:06.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-11T15:14:06.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000016[0] 2023-10-11T15:14:06.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-11T15:14:06.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000030[0] 2023-10-11T15:14:06.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position=00000000002.20.6 2023-10-11T15:14:06.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.545+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26/obj00000000000000000014 2023-10-11T15:14:06.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-11T15:14:06.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13/obj00000000000000000013 2023-10-11T15:14:06.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: entry->timestamp=2023-10-11T15:14:02.376280+0000 2023-10-11T15:14:06.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-11T15:14:06.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000013[0] 2023-10-11T15:14:06.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-11T15:14:06.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-11T15:14:06.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22/obj00000000000000000009 2023-10-11T15:14:06.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: entry->timestamp=2023-10-11T15:14:02.192261+0000 2023-10-11T15:14:06.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-11T15:14:06.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-11T15:14:06.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25/obj00000000000000000000 2023-10-11T15:14:06.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: entry->timestamp=2023-10-11T15:14:02.434571+0000 2023-10-11T15:14:06.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-11T15:14:06.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-11T15:14:06.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23/obj00000000000000000025 2023-10-11T15:14:06.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: entry->timestamp=2023-10-11T15:14:02.720463+0000 2023-10-11T15:14:06.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-10-11T15:14:06.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000005]: success 2023-10-11T15:14:06.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16:4 marker=00000000002.42.6 timestamp=2023-10-11T15:14:02.455969+0000 2023-10-11T15:14:06.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000009[0] 2023-10-11T15:14:06.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-11T15:14:06.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000000[0] 2023-10-11T15:14:06.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.548+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-11T15:14:06.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: : injecting data sync error on key=obj00000000000000000025 2023-10-11T15:14:06.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position=00000000002.76.6 2023-10-11T15:14:06.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23/obj00000000000000000025 2023-10-11T15:14:06.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000021]: success 2023-10-11T15:14:06.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28:4 marker=00000000002.62.6 timestamp=2023-10-11T15:14:02.798825+0000 2023-10-11T15:14:06.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000012]: success 2023-10-11T15:14:06.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.549+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17:4 marker=00000000002.40.6 timestamp=2023-10-11T15:14:02.106669+0000 2023-10-11T15:14:06.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000030]: success 2023-10-11T15:14:06.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27:4 marker=00000000002.20.6 timestamp=2023-10-11T15:14:02.047194+0000 2023-10-11T15:14:06.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000013]: success 2023-10-11T15:14:06.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13:4 marker=00000000002.31.6 timestamp=2023-10-11T15:14:02.376280+0000 2023-10-11T15:14:06.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000009]: success 2023-10-11T15:14:06.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22:4 marker=00000000002.70.6 timestamp=2023-10-11T15:14:02.192261+0000 2023-10-11T15:14:06.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000000]: success 2023-10-11T15:14:06.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.552+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25:4 marker=00000000002.79.6 timestamp=2023-10-11T15:14:02.434571+0000 2023-10-11T15:14:06.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.554+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000016]: success 2023-10-11T15:14:06.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.554+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24:4 marker=00000000002.60.6 timestamp=2023-10-11T15:14:02.538233+0000 2023-10-11T15:14:06.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.555+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000017]: success 2023-10-11T15:14:06.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.555+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:4 marker=00000000001.42.6 timestamp=2023-10-11T15:14:01.843422+0000 2023-10-11T15:14:06.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: a sync operation returned error 2023-10-11T15:14:06.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: backing out with sync_status=-5 2023-10-11T15:14:06.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket shard: 0 2023-10-11T15:14:06.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.559+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-10-11T15:14:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000059 2023-10-11T15:14:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:01.862268+0000 2023-10-11T15:14:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-10-11T15:14:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000059[0] 2023-10-11T15:14:06.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.576+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000001.43.6 2023-10-11T15:14:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000006 2023-10-11T15:14:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:14:02.026685+0000 2023-10-11T15:14:06.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-10-11T15:14:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000006[0] 2023-10-11T15:14:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000002.40.6 2023-10-11T15:14:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-11T15:14:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000004 2023-10-11T15:14:06.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:14:02.354564+0000 2023-10-11T15:14:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-10-11T15:14:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000019 2023-10-11T15:14:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:14:01.963808+0000 2023-10-11T15:14:06.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-11T15:14:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000004[0] 2023-10-11T15:14:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000002.83.6 2023-10-11T15:14:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-10-11T15:14:06.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-11T15:14:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.577+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000019 2023-10-11T15:14:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-11T15:14:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2/obj00000000000000000027 2023-10-11T15:14:06.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: entry->timestamp=2023-10-11T15:14:02.701131+0000 2023-10-11T15:14:06.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-10-11T15:14:06.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000027[0] 2023-10-11T15:14:06.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.578+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000001.45.6 2023-10-11T15:14:06.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.579+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000006]: success 2023-10-11T15:14:06.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.579+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:4 marker=00000000002.40.6 timestamp=2023-10-11T15:14:02.026685+0000 2023-10-11T15:14:06.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:06.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4/obj00000000000000000011 2023-10-11T15:14:06.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: entry->timestamp=2023-10-11T15:14:02.514082+0000 2023-10-11T15:14:06.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-11T15:14:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000011[0] 2023-10-11T15:14:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-11T15:14:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-10-11T15:14:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000031 2023-10-11T15:14:06.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:14:02.779233+0000 2023-10-11T15:14:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-10-11T15:14:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-10-11T15:14:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000049 2023-10-11T15:14:06.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:14:01.944664+0000 2023-10-11T15:14:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-10-11T15:14:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-11T15:14:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000008 2023-10-11T15:14:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:14:02.599109+0000 2023-10-11T15:14:06.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-10-11T15:14:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-11T15:14:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000015 2023-10-11T15:14:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:02.127634+0000 2023-10-11T15:14:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-10-11T15:14:06.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000031[0] 2023-10-11T15:14:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000001.40.6 2023-10-11T15:14:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000049[0] 2023-10-11T15:14:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000001.58.6 2023-10-11T15:14:06.604 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000004]: success 2023-10-11T15:14:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:4 marker=00000000002.83.6 timestamp=2023-10-11T15:14:02.354564+0000 2023-10-11T15:14:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000008[0] 2023-10-11T15:14:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000002.72.6 2023-10-11T15:14:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000015[0] 2023-10-11T15:14:06.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.580+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.49.6 2023-10-11T15:14:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-11T15:14:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000022 2023-10-11T15:14:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:14:02.296231+0000 2023-10-11T15:14:06.606 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-10-11T15:14:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000022[0] 2023-10-11T15:14:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.581+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000002.46.6 2023-10-11T15:14:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.583+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000011]: success 2023-10-11T15:14:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.583+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4:4 marker=00000000002.52.6 timestamp=2023-10-11T15:14:02.514082+0000 2023-10-11T15:14:06.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-10-11T15:14:06.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19/obj00000000000000000042 2023-10-11T15:14:06.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: entry->timestamp=2023-10-11T15:14:02.681262+0000 2023-10-11T15:14:06.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.85.6 op=0 op_state=1 2023-10-11T15:14:06.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000042[0] 2023-10-11T15:14:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.584+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position=00000000001.85.6 2023-10-11T15:14:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-11T15:14:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20/obj00000000000000000010 2023-10-11T15:14:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: entry->timestamp=2023-10-11T15:14:02.619877+0000 2023-10-11T15:14:06.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-11T15:14:06.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000010[0] 2023-10-11T15:14:06.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-11T15:14:06.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: a sync operation returned error 2023-10-11T15:14:06.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: backing out with sync_status=-5 2023-10-11T15:14:06.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000015]: success 2023-10-11T15:14:06.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10:4 marker=00000000002.49.6 timestamp=2023-10-11T15:14:02.127634+0000 2023-10-11T15:14:06.611 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.585+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000008]: success 2023-10-11T15:14:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.586+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:4 marker=00000000002.72.6 timestamp=2023-10-11T15:14:02.599109+0000 2023-10-11T15:14:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.586+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: a sync operation returned error 2023-10-11T15:14:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.586+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: backing out with sync_status=-5 2023-10-11T15:14:06.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.586+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.586+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: a sync operation returned error 2023-10-11T15:14:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: backing out with sync_status=-5 2023-10-11T15:14:06.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000010]: success 2023-10-11T15:14:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.588+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20:4 marker=00000000002.58.6 timestamp=2023-10-11T15:14:02.619877+0000 2023-10-11T15:14:06.614 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.589+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket shard: 0 2023-10-11T15:14:06.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.590+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.591+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000042]: success 2023-10-11T15:14:06.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.591+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19:4 marker=00000000001.85.6 timestamp=2023-10-11T15:14:02.681262+0000 2023-10-11T15:14:06.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.592+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:06.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.592+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.593+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000022]: success 2023-10-11T15:14:06.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.593+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:4 marker=00000000002.46.6 timestamp=2023-10-11T15:14:02.296231+0000 2023-10-11T15:14:06.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.593+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000037]: success 2023-10-11T15:14:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.593+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:4 marker=00000000001.74.6 timestamp=2023-10-11T15:14:02.275683+0000 2023-10-11T15:14:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-10-11T15:14:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23/obj00000000000000000025 2023-10-11T15:14:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: entry->timestamp=2023-10-11T15:14:02.720463+0000 2023-10-11T15:14:06.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-10-11T15:14:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000025[0] 2023-10-11T15:14:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.601+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position=00000000002.76.6 2023-10-11T15:14:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.603+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000025]: success 2023-10-11T15:14:06.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.603+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23:4 marker=00000000002.76.6 timestamp=2023-10-11T15:14:02.720463+0000 2023-10-11T15:14:06.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: a sync operation returned error 2023-10-11T15:14:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: backing out with sync_status=-5 2023-10-11T15:14:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000059]: success 2023-10-11T15:14:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:4 marker=00000000001.43.6 timestamp=2023-10-11T15:14:01.862268+0000 2023-10-11T15:14:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.621+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.625 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.622+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.622+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.625+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000027]: success 2023-10-11T15:14:06.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.625+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2:4 marker=00000000001.45.6 timestamp=2023-10-11T15:14:02.701131+0000 2023-10-11T15:14:06.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.628+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000049]: success 2023-10-11T15:14:06.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.628+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:4 marker=00000000001.58.6 timestamp=2023-10-11T15:14:01.944664+0000 2023-10-11T15:14:06.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.630+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000031]: success 2023-10-11T15:14:06.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.630+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:4 marker=00000000001.40.6 timestamp=2023-10-11T15:14:02.779233+0000 2023-10-11T15:14:06.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-11T15:14:06.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000020 2023-10-11T15:14:06.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:01.924163+0000 2023-10-11T15:14:06.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000020[0] 2023-10-11T15:14:06.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.632+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-11T15:14:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26/obj00000000000000000014 2023-10-11T15:14:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: entry->timestamp=2023-10-11T15:14:02.213609+0000 2023-10-11T15:14:06.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-11T15:14:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000014[0] 2023-10-11T15:14:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.633+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-11T15:14:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:06.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000003 2023-10-11T15:14:06.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:14:01.823024+0000 2023-10-11T15:14:06.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000003[0] 2023-10-11T15:14:06.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000020]: success 2023-10-11T15:14:06.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.634+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:4 marker=00000000002.56.6 timestamp=2023-10-11T15:14:01.924163+0000 2023-10-11T15:14:06.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.636+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000014]: success 2023-10-11T15:14:06.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.636+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26:4 marker=00000000002.52.6 timestamp=2023-10-11T15:14:02.213609+0000 2023-10-11T15:14:06.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.636+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000003]: success 2023-10-11T15:14:06.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.636+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18:4 marker=00000000002.56.6 timestamp=2023-10-11T15:14:01.823024+0000 2023-10-11T15:14:06.667 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000019 2023-10-11T15:14:06.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:14:01.963808+0000 2023-10-11T15:14:06.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-10-11T15:14:06.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000019[0] 2023-10-11T15:14:06.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.665+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.44.6 2023-10-11T15:14:06.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.667+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000019]: success 2023-10-11T15:14:06.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.667+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:4 marker=00000000002.44.6 timestamp=2023-10-11T15:14:01.963808+0000 2023-10-11T15:14:06.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.710+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:06.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.711+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:06.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.712+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:06.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:06.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:06.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:06.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.713+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.714+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:06.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.714+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.714+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:06.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.714+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:06.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-10-11T15:14:06.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000023 2023-10-11T15:14:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:14:03.777412+0000 2023-10-11T15:14:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-11T15:14:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000023[0] 2023-10-11T15:14:06.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-11T15:14:06.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000001.74.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-10-11T15:14:06.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000317 2023-10-11T15:14:06.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:14:04.026360+0000 2023-10-11T15:14:06.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.74.6 op=0 op_state=1 2023-10-11T15:14:06.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000317[0] 2023-10-11T15:14:06.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000001.74.6 2023-10-11T15:14:06.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-10-11T15:14:06.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4/obj00000000000000000056 2023-10-11T15:14:06.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: entry->timestamp=2023-10-11T15:14:03.924438+0000 2023-10-11T15:14:06.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-10-11T15:14:06.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000056[0] 2023-10-11T15:14:06.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.715+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000001.53.6 2023-10-11T15:14:06.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-11T15:14:06.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000022 2023-10-11T15:14:06.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:03.865067+0000 2023-10-11T15:14:06.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-11T15:14:06.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2023-10-11T15:14:06.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-11T15:14:06.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000022 2023-10-11T15:14:06.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.716+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket shard: 0 2023-10-11T15:14:06.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:06.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:06.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:06.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket shard: 0 2023-10-11T15:14:06.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:06.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.717+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket shard: 0 2023-10-11T15:14:06.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket shard: 0 2023-10-11T15:14:06.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:06.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.742 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket shard: 0 2023-10-11T15:14:06.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket shard: 0 2023-10-11T15:14:06.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket shard: 0 2023-10-11T15:14:06.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.718+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket shard: 0 2023-10-11T15:14:06.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket shard: 0 2023-10-11T15:14:06.745 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket shard: 0 2023-10-11T15:14:06.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket shard: 0 2023-10-11T15:14:06.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket shard: 0 2023-10-11T15:14:06.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.719+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.720+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: a sync operation returned error 2023-10-11T15:14:06.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.720+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: backing out with sync_status=-5 2023-10-11T15:14:06.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.720+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.720+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.721+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: sync status for source bucket shard: 0 2023-10-11T15:14:06.748 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.721+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-10-11T15:14:06.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20/obj00000000000000000423 2023-10-11T15:14:06.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: entry->timestamp=2023-10-11T15:14:04.006914+0000 2023-10-11T15:14:06.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-10-11T15:14:06.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000423[0] 2023-10-11T15:14:06.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position=00000000001.44.6 2023-10-11T15:14:06.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:06.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000007 2023-10-11T15:14:06.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:14:03.353199+0000 2023-10-11T15:14:06.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-11T15:14:06.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-11T15:14:06.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000000 2023-10-11T15:14:06.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:14:03.754478+0000 2023-10-11T15:14:06.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-10-11T15:14:06.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:06.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19/obj00000000000000000003 2023-10-11T15:14:06.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: entry->timestamp=2023-10-11T15:14:03.373738+0000 2023-10-11T15:14:06.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-10-11T15:14:06.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:06.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24/obj00000000000000000011 2023-10-11T15:14:06.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: entry->timestamp=2023-10-11T15:14:04.175551+0000 2023-10-11T15:14:06.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-11T15:14:06.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-11T15:14:06.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28/obj00000000000000000050 2023-10-11T15:14:06.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: entry->timestamp=2023-10-11T15:14:03.837579+0000 2023-10-11T15:14:06.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-11T15:14:06.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:06.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23/obj00000000000000000001 2023-10-11T15:14:06.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: entry->timestamp=2023-10-11T15:14:03.611798+0000 2023-10-11T15:14:06.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-10-11T15:14:06.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-10-11T15:14:06.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29/obj00000000000000000029 2023-10-11T15:14:06.756 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: entry->timestamp=2023-10-11T15:14:03.187274+0000 2023-10-11T15:14:06.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-11T15:14:06.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-11T15:14:06.757 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26/obj00000000000000000010 2023-10-11T15:14:06.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: entry->timestamp=2023-10-11T15:14:03.692076+0000 2023-10-11T15:14:06.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-10-11T15:14:06.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-10-11T15:14:06.758 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27/obj00000000000000000061 2023-10-11T15:14:06.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: entry->timestamp=2023-10-11T15:14:03.430765+0000 2023-10-11T15:14:06.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.72.6 op=0 op_state=1 2023-10-11T15:14:06.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-10-11T15:14:06.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25/obj00000000000000000283 2023-10-11T15:14:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: entry->timestamp=2023-10-11T15:14:04.111758+0000 2023-10-11T15:14:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-11T15:14:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000007[0] 2023-10-11T15:14:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-11T15:14:06.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000000[0] 2023-10-11T15:14:06.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000002.29.6 2023-10-11T15:14:06.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000003[0] 2023-10-11T15:14:06.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position=00000000002.58.6 2023-10-11T15:14:06.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2023-10-11T15:14:06.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-11T15:14:06.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: : injecting data sync error on key=obj00000000000000000050 2023-10-11T15:14:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-11T15:14:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000001[0] 2023-10-11T15:14:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position=00000000002.93.6 2023-10-11T15:14:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2023-10-11T15:14:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-11T15:14:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000010[0] 2023-10-11T15:14:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position=00000000002.48.6 2023-10-11T15:14:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-10-11T15:14:06.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30/obj00000000000000000133 2023-10-11T15:14:06.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: entry->timestamp=2023-10-11T15:14:03.508570+0000 2023-10-11T15:14:06.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-10-11T15:14:06.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-11T15:14:06.765 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000018 2023-10-11T15:14:06.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:03.528096+0000 2023-10-11T15:14:06.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-10-11T15:14:06.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000061[0] 2023-10-11T15:14:06.766 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position=00000000001.72.6 2023-10-11T15:14:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: : injecting data sync error on key=obj00000000000000000283 2023-10-11T15:14:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-11T15:14:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.767 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24/obj00000000000000000011 2023-10-11T15:14:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28/obj00000000000000000050 2023-10-11T15:14:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29/obj00000000000000000029 2023-10-11T15:14:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000133[0] 2023-10-11T15:14:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: listing bilog for incremental sync; position=00000000001.57.6 2023-10-11T15:14:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000018[0] 2023-10-11T15:14:06.769 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.51.6 2023-10-11T15:14:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.725+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25/obj00000000000000000283 2023-10-11T15:14:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.727+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000317]: success 2023-10-11T15:14:06.770 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.727+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:5 marker=00000000001.74.6 timestamp=2023-10-11T15:14:04.026360+0000 2023-10-11T15:14:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000056]: success 2023-10-11T15:14:06.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4:5 marker=00000000001.53.6 timestamp=2023-10-11T15:14:03.924438+0000 2023-10-11T15:14:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000023]: success 2023-10-11T15:14:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:5 marker=00000000002.50.6 timestamp=2023-10-11T15:14:03.777412+0000 2023-10-11T15:14:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000001]: success 2023-10-11T15:14:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23:5 marker=00000000002.93.6 timestamp=2023-10-11T15:14:03.611798+0000 2023-10-11T15:14:06.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000007]: success 2023-10-11T15:14:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:5 marker=00000000002.70.6 timestamp=2023-10-11T15:14:03.353199+0000 2023-10-11T15:14:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000003]: success 2023-10-11T15:14:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.732+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19:5 marker=00000000002.58.6 timestamp=2023-10-11T15:14:03.373738+0000 2023-10-11T15:14:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.733+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000000]: success 2023-10-11T15:14:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.733+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:5 marker=00000000002.29.6 timestamp=2023-10-11T15:14:03.754478+0000 2023-10-11T15:14:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.735+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000010]: success 2023-10-11T15:14:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.735+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26:5 marker=00000000002.48.6 timestamp=2023-10-11T15:14:03.692076+0000 2023-10-11T15:14:06.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.738+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000061]: success 2023-10-11T15:14:06.780 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.738+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27:5 marker=00000000001.72.6 timestamp=2023-10-11T15:14:03.430765+0000 2023-10-11T15:14:06.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.738+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000018]: success 2023-10-11T15:14:06.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.738+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:5 marker=00000000002.51.6 timestamp=2023-10-11T15:14:03.528096+0000 2023-10-11T15:14:06.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:06.781 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000006 2023-10-11T15:14:06.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:03.164110+0000 2023-10-11T15:14:06.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: : injecting data sync error on key=obj00000000000000000006 2023-10-11T15:14:06.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.753+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000006 2023-10-11T15:14:06.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000002.112.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:06.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000019 2023-10-11T15:14:06.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:14:03.671032+0000 2023-10-11T15:14:06.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.112.6 op=0 op_state=1 2023-10-11T15:14:06.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000019[0] 2023-10-11T15:14:06.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000002.112.6 2023-10-11T15:14:06.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-11T15:14:06.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000005 2023-10-11T15:14:06.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:14:03.252207+0000 2023-10-11T15:14:06.785 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-11T15:14:06.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-11T15:14:06.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000004 2023-10-11T15:14:06.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: entry->timestamp=2023-10-11T15:14:04.196080+0000 2023-10-11T15:14:06.786 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-10-11T15:14:06.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-10-11T15:14:06.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-11T15:14:06.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000004[0] 2023-10-11T15:14:06.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000002.73.6 2023-10-11T15:14:06.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.754+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000005 2023-10-11T15:14:06.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.756+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000019]: success 2023-10-11T15:14:06.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.756+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:5 marker=00000000002.112.6 timestamp=2023-10-11T15:14:03.671032+0000 2023-10-11T15:14:06.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.756+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000004]: success 2023-10-11T15:14:06.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.756+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5:5 marker=00000000002.73.6 timestamp=2023-10-11T15:14:04.196080+0000 2023-10-11T15:14:06.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.789 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:03.450022+0000 2023-10-11T15:14:06.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-11T15:14:06.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2023-10-11T15:14:06.790 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-11T15:14:06.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.757+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-11T15:14:06.791 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000009 2023-10-11T15:14:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:14:03.138283+0000 2023-10-11T15:14:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-11T15:14:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000009[0] 2023-10-11T15:14:06.792 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.758+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-11T15:14:06.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000001.27.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-10-11T15:14:06.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000271 2023-10-11T15:14:06.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:14:03.272549+0000 2023-10-11T15:14:06.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.27.6 op=0 op_state=1 2023-10-11T15:14:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000271[0] 2023-10-11T15:14:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000001.27.6 2023-10-11T15:14:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-11T15:14:06.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13/obj00000000000000000008 2023-10-11T15:14:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: entry->timestamp=2023-10-11T15:14:03.591020+0000 2023-10-11T15:14:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-10-11T15:14:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000008[0] 2023-10-11T15:14:06.795 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.759+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position=00000000002.38.6 2023-10-11T15:14:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.760+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000009]: success 2023-10-11T15:14:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.760+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:5 marker=00000000002.68.6 timestamp=2023-10-11T15:14:03.138283+0000 2023-10-11T15:14:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.761+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000008]: success 2023-10-11T15:14:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.761+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13:5 marker=00000000002.38.6 timestamp=2023-10-11T15:14:03.591020+0000 2023-10-11T15:14:06.796 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-10-11T15:14:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17/obj00000000000000000162 2023-10-11T15:14:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: entry->timestamp=2023-10-11T15:14:03.943763+0000 2023-10-11T15:14:06.797 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-10-11T15:14:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000162[0] 2023-10-11T15:14:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.764+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position=00000000001.59.6 2023-10-11T15:14:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-11T15:14:06.798 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16/obj00000000000000000017 2023-10-11T15:14:06.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: entry->timestamp=2023-10-11T15:14:04.091107+0000 2023-10-11T15:14:06.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-11T15:14:06.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000017[0] 2023-10-11T15:14:06.799 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-11T15:14:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-11T15:14:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000016 2023-10-11T15:14:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:14:03.292266+0000 2023-10-11T15:14:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-11T15:14:06.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000016[0] 2023-10-11T15:14:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.765+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-11T15:14:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.766+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:entry[obj00000000000000000017]: success 2023-10-11T15:14:06.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.767+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16:5 marker=00000000002.57.6 timestamp=2023-10-11T15:14:04.091107+0000 2023-10-11T15:14:06.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.767+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000016]: success 2023-10-11T15:14:06.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.767+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18:5 marker=00000000002.61.6 timestamp=2023-10-11T15:14:03.292266+0000 2023-10-11T15:14:06.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: a sync operation returned error 2023-10-11T15:14:06.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: a sync operation returned error 2023-10-11T15:14:06.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: a sync operation returned error 2023-10-11T15:14:06.803 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: backing out with sync_status=-5 2023-10-11T15:14:06.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: backing out with sync_status=-5 2023-10-11T15:14:06.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: backing out with sync_status=-5 2023-10-11T15:14:06.804 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.805 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000423]: success 2023-10-11T15:14:06.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.772+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20:5 marker=00000000001.44.6 timestamp=2023-10-11T15:14:04.006914+0000 2023-10-11T15:14:06.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.773+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: a sync operation returned error 2023-10-11T15:14:06.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.773+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: backing out with sync_status=-5 2023-10-11T15:14:06.806 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.773+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.773+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.807 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket shard: 0 2023-10-11T15:14:06.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket shard: 0 2023-10-11T15:14:06.808 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.775+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-10-11T15:14:06.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000022 2023-10-11T15:14:06.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:03.865067+0000 2023-10-11T15:14:06.809 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-11T15:14:06.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000022[0] 2023-10-11T15:14:06.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-11T15:14:06.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket shard: 0 2023-10-11T15:14:06.810 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.776+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.777+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket shard: 0 2023-10-11T15:14:06.811 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.777+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.813 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.778+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000022]: success 2023-10-11T15:14:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.778+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:5 marker=00000000002.57.6 timestamp=2023-10-11T15:14:03.865067+0000 2023-10-11T15:14:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.780+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000133]: success 2023-10-11T15:14:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.780+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30:5 marker=00000000001.57.6 timestamp=2023-10-11T15:14:03.508570+0000 2023-10-11T15:14:06.814 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.795+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: a sync operation returned error 2023-10-11T15:14:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.795+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: backing out with sync_status=-5 2023-10-11T15:14:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.795+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.795+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.815 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.796+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.796+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.796+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.797+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: a sync operation returned error 2023-10-11T15:14:06.816 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.797+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: backing out with sync_status=-5 2023-10-11T15:14:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.797+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.797+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.798+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.817 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.798+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.798+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: a sync operation returned error 2023-10-11T15:14:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: backing out with sync_status=-5 2023-10-11T15:14:06.818 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.799+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.819 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.804+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000271]: success 2023-10-11T15:14:06.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.804+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:5 marker=00000000001.27.6 timestamp=2023-10-11T15:14:03.272549+0000 2023-10-11T15:14:06.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.809+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:entry[obj00000000000000000162]: success 2023-10-11T15:14:06.820 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.810+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17:5 marker=00000000001.59.6 timestamp=2023-10-11T15:14:03.943763+0000 2023-10-11T15:14:06.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:06.821 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24/obj00000000000000000011 2023-10-11T15:14:06.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: entry->timestamp=2023-10-11T15:14:04.175551+0000 2023-10-11T15:14:06.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-11T15:14:06.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000011[0] 2023-10-11T15:14:06.822 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-11T15:14:06.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-10-11T15:14:06.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28/obj00000000000000000050 2023-10-11T15:14:06.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: entry->timestamp=2023-10-11T15:14:03.837579+0000 2023-10-11T15:14:06.823 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-10-11T15:14:06.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000050[0] 2023-10-11T15:14:06.824 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.818+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position=00000000002.31.6 2023-10-11T15:14:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-10-11T15:14:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29/obj00000000000000000029 2023-10-11T15:14:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: entry->timestamp=2023-10-11T15:14:03.187274+0000 2023-10-11T15:14:06.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-10-11T15:14:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000029[0] 2023-10-11T15:14:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position=00000000001.49.6 2023-10-11T15:14:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-10-11T15:14:06.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25/obj00000000000000000283 2023-10-11T15:14:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: entry->timestamp=2023-10-11T15:14:04.111758+0000 2023-10-11T15:14:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-10-11T15:14:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000283[0] 2023-10-11T15:14:06.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.819+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position=00000000001.61.6 2023-10-11T15:14:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.820+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000050]: success 2023-10-11T15:14:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.820+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28:5 marker=00000000002.31.6 timestamp=2023-10-11T15:14:03.837579+0000 2023-10-11T15:14:06.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.821+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000011]: success 2023-10-11T15:14:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.821+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24:5 marker=00000000002.45.6 timestamp=2023-10-11T15:14:04.175551+0000 2023-10-11T15:14:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.824+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000283]: success 2023-10-11T15:14:06.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.824+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25:5 marker=00000000001.61.6 timestamp=2023-10-11T15:14:04.111758+0000 2023-10-11T15:14:06.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000006 2023-10-11T15:14:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:03.164110+0000 2023-10-11T15:14:06.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000006[0] 2023-10-11T15:14:06.842 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.838+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.839+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000006]: success 2023-10-11T15:14:06.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.839+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:5 marker=00000000002.56.6 timestamp=2023-10-11T15:14:03.164110+0000 2023-10-11T15:14:06.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-11T15:14:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000005 2023-10-11T15:14:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:14:03.252207+0000 2023-10-11T15:14:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-10-11T15:14:06.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000005[0] 2023-10-11T15:14:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000002.42.6 2023-10-11T15:14:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:03.450022+0000 2023-10-11T15:14:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-11T15:14:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2023-10-11T15:14:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-11T15:14:06.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.841+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.892 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.842+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000005]: success 2023-10-11T15:14:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.842+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:5 marker=00000000002.42.6 timestamp=2023-10-11T15:14:03.252207+0000 2023-10-11T15:14:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.864+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000029]: success 2023-10-11T15:14:06.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.864+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29:5 marker=00000000001.49.6 timestamp=2023-10-11T15:14:03.187274+0000 2023-10-11T15:14:06.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.883+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: a sync operation returned error 2023-10-11T15:14:06.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.883+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: backing out with sync_status=-5 2023-10-11T15:14:06.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.883+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.894 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.883+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.883+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.884+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.895 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.884+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:06.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:06.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:03.450022+0000 2023-10-11T15:14:06.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-11T15:14:06.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2023-10-11T15:14:06.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-11T15:14:06.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:06.930 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.926+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:06.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.968+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: a sync operation returned error 2023-10-11T15:14:06.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.968+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: backing out with sync_status=-5 2023-10-11T15:14:06.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.968+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:06.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.968+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:06.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.968+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:06.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.969+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:06.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:06.969+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:07.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10/obj00000000000000000021 2023-10-11T15:14:07.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: entry->timestamp=2023-10-11T15:14:03.450022+0000 2023-10-11T15:14:07.014 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-10-11T15:14:07.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:07.015 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.011+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position=00000000002.79.6 2023-10-11T15:14:07.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.013+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:entry[obj00000000000000000021]: success 2023-10-11T15:14:07.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.013+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10:5 marker=00000000002.79.6 timestamp=2023-10-11T15:14:03.450022+0000 2023-10-11T15:14:07.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.057+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.057+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.057+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.057+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: sync status for source bucket shard: 0 2023-10-11T15:14:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: sync status for source bucket shard: 0 2023-10-11T15:14:07.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: sync status for source bucket shard: 0 2023-10-11T15:14:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.058+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: sync status for source bucket shard: 0 2023-10-11T15:14:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:07.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-10-11T15:14:07.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4/obj00000000000000000033 2023-10-11T15:14:07.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: entry->timestamp=2023-10-11T15:14:05.035423+0000 2023-10-11T15:14:07.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-11T15:14:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000033[0] 2023-10-11T15:14:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-11T15:14:07.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: parsed entry: id=00000000001.62.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-10-11T15:14:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0/obj00000000000000000060 2023-10-11T15:14:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: entry->timestamp=2023-10-11T15:14:05.113916+0000 2023-10-11T15:14:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.62.6 op=0 op_state=1 2023-10-11T15:14:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000060[0] 2023-10-11T15:14:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.059+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: listing bilog for incremental sync; position=00000000001.62.6 2023-10-11T15:14:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.073 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: sync status for source bucket shard: 0 2023-10-11T15:14:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: sync status for source bucket shard: 0 2023-10-11T15:14:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: sync status for source bucket shard: 0 2023-10-11T15:14:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:10]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.060+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: sync status for source bucket shard: 0 2023-10-11T15:14:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: sync status for source bucket shard: 0 2023-10-11T15:14:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: sync status for source bucket shard: 0 2023-10-11T15:14:07.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: sync status for source bucket shard: 0 2023-10-11T15:14:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: sync status for source bucket shard: 0 2023-10-11T15:14:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: sync status for source bucket shard: 0 2023-10-11T15:14:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:16]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-10-11T15:14:07.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3/obj00000000000000000001 2023-10-11T15:14:07.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: entry->timestamp=2023-10-11T15:14:05.872986+0000 2023-10-11T15:14:07.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.72.6 op=0 op_state=1 2023-10-11T15:14:07.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000001[0] 2023-10-11T15:14:07.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.061+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: listing bilog for incremental sync; position=00000000002.72.6 2023-10-11T15:14:07.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: sync status for source bucket shard: 0 2023-10-11T15:14:07.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:17]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-10-11T15:14:07.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14/obj00000000000000000040 2023-10-11T15:14:07.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: entry->timestamp=2023-10-11T15:14:04.641229+0000 2023-10-11T15:14:07.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-10-11T15:14:07.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000040[0] 2023-10-11T15:14:07.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: listing bilog for incremental sync; position=00000000002.62.6 2023-10-11T15:14:07.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-10-11T15:14:07.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6/obj00000000000000000005 2023-10-11T15:14:07.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: entry->timestamp=2023-10-11T15:14:04.782685+0000 2023-10-11T15:14:07.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.91.6 op=0 op_state=1 2023-10-11T15:14:07.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-10-11T15:14:07.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13/obj00000000000000000009 2023-10-11T15:14:07.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: entry->timestamp=2023-10-11T15:14:05.196140+0000 2023-10-11T15:14:07.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-10-11T15:14:07.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000005[0] 2023-10-11T15:14:07.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: listing bilog for incremental sync; position=00000000002.91.6 2023-10-11T15:14:07.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000009[0] 2023-10-11T15:14:07.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.063+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: listing bilog for incremental sync; position=00000000002.64.6 2023-10-11T15:14:07.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.064+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: sync status for source bucket shard: 0 2023-10-11T15:14:07.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.064+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: sync status for source bucket shard: 0 2023-10-11T15:14:07.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: sync status for source bucket shard: 0 2023-10-11T15:14:07.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: sync status for source bucket shard: 0 2023-10-11T15:14:07.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: sync status for source bucket shard: 0 2023-10-11T15:14:07.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:entry[obj00000000000000000033]: success 2023-10-11T15:14:07.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:4:6 marker=00000000002.59.6 timestamp=2023-10-11T15:14:05.035423+0000 2023-10-11T15:14:07.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.065+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.066+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: sync status for source bucket shard: 0 2023-10-11T15:14:07.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.066+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.066+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: sync status for source bucket shard: 0 2023-10-11T15:14:07.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.066+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: sync status for source bucket shard: 0 2023-10-11T15:14:07.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: sync status for source bucket shard: 0 2023-10-11T15:14:07.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: sync status for source bucket shard: 0 2023-10-11T15:14:07.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: sync status for source bucket shard: 0 2023-10-11T15:14:07.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: sync status for source bucket shard: 0 2023-10-11T15:14:07.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: parsed entry: id=00000000001.27.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-10-11T15:14:07.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8/obj00000000000000000048 2023-10-11T15:14:07.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: entry->timestamp=2023-10-11T15:14:04.535335+0000 2023-10-11T15:14:07.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.27.6 op=0 op_state=1 2023-10-11T15:14:07.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000048[0] 2023-10-11T15:14:07.098 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: listing bilog for incremental sync; position=00000000001.27.6 2023-10-11T15:14:07.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:entry[obj00000000000000000001]: success 2023-10-11T15:14:07.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.067+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:3:6 marker=00000000002.72.6 timestamp=2023-10-11T15:14:05.872986+0000 2023-10-11T15:14:07.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.068+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: sync status for source bucket shard: 0 2023-10-11T15:14:07.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.068+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.099 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: sync status for source bucket shard: 0 2023-10-11T15:14:07.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-10-11T15:14:07.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9/obj00000000000000000015 2023-10-11T15:14:07.100 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: entry->timestamp=2023-10-11T15:14:05.439809+0000 2023-10-11T15:14:07.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-10-11T15:14:07.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-10-11T15:14:07.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12/obj00000000000000000062 2023-10-11T15:14:07.101 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: entry->timestamp=2023-10-11T15:14:04.702427+0000 2023-10-11T15:14:07.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-10-11T15:14:07.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-10-11T15:14:07.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25/obj00000000000000000010 2023-10-11T15:14:07.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: entry->timestamp=2023-10-11T15:14:04.619273+0000 2023-10-11T15:14:07.102 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-10-11T15:14:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-10-11T15:14:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27/obj00000000000000000053 2023-10-11T15:14:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: entry->timestamp=2023-10-11T15:14:05.217608+0000 2023-10-11T15:14:07.103 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-10-11T15:14:07.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000015[0] 2023-10-11T15:14:07.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: listing bilog for incremental sync; position=00000000002.45.6 2023-10-11T15:14:07.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000062[0] 2023-10-11T15:14:07.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: listing bilog for incremental sync; position=00000000001.41.6 2023-10-11T15:14:07.104 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000010[0] 2023-10-11T15:14:07.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: listing bilog for incremental sync; position=00000000002.70.6 2023-10-11T15:14:07.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000053[0] 2023-10-11T15:14:07.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: listing bilog for incremental sync; position=00000000002.56.6 2023-10-11T15:14:07.105 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: sync status for source bucket shard: 0 2023-10-11T15:14:07.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: sync status for source bucket shard: 0 2023-10-11T15:14:07.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: sync status for source bucket shard: 0 2023-10-11T15:14:07.106 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:entry[obj00000000000000000040]: success 2023-10-11T15:14:07.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:14:6 marker=00000000002.62.6 timestamp=2023-10-11T15:14:04.641229+0000 2023-10-11T15:14:07.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.107 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.070+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.071+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:entry[obj00000000000000000009]: success 2023-10-11T15:14:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.071+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:13:6 marker=00000000002.64.6 timestamp=2023-10-11T15:14:05.196140+0000 2023-10-11T15:14:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: sync status for source bucket shard: 0 2023-10-11T15:14:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: sync status for source bucket shard: 0 2023-10-11T15:14:07.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: sync status for source bucket shard: 0 2023-10-11T15:14:07.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: sync status for source bucket shard: 0 2023-10-11T15:14:07.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: sync status for source bucket shard: 0 2023-10-11T15:14:07.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:entry[obj00000000000000000005]: success 2023-10-11T15:14:07.110 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:6:6 marker=00000000002.91.6 timestamp=2023-10-11T15:14:04.782685+0000 2023-10-11T15:14:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-10-11T15:14:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23/obj00000000000000000039 2023-10-11T15:14:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: entry->timestamp=2023-10-11T15:14:05.688083+0000 2023-10-11T15:14:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.84.6 op=0 op_state=1 2023-10-11T15:14:07.111 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-10-11T15:14:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26/obj00000000000000000021 2023-10-11T15:14:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: entry->timestamp=2023-10-11T15:14:05.277119+0000 2023-10-11T15:14:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-11T15:14:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-10-11T15:14:07.112 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34/obj00000000000000000057 2023-10-11T15:14:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: entry->timestamp=2023-10-11T15:14:05.599951+0000 2023-10-11T15:14:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.073+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-10-11T15:14:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000039[0] 2023-10-11T15:14:07.113 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: listing bilog for incremental sync; position=00000000002.84.6 2023-10-11T15:14:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000021[0] 2023-10-11T15:14:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-11T15:14:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000057[0] 2023-10-11T15:14:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.074+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: listing bilog for incremental sync; position=00000000001.69.6 2023-10-11T15:14:07.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:entry[obj00000000000000000015]: success 2023-10-11T15:14:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:9:6 marker=00000000002.45.6 timestamp=2023-10-11T15:14:05.439809+0000 2023-10-11T15:14:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: parsed entry: id=00000000002.100.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-10-11T15:14:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29/obj00000000000000000004 2023-10-11T15:14:07.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: entry->timestamp=2023-10-11T15:14:05.520612+0000 2023-10-11T15:14:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.100.6 op=0 op_state=1 2023-10-11T15:14:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000004[0] 2023-10-11T15:14:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.075+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: listing bilog for incremental sync; position=00000000002.100.6 2023-10-11T15:14:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-11T15:14:07.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35/obj00000000000000000018 2023-10-11T15:14:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: entry->timestamp=2023-10-11T15:14:05.298694+0000 2023-10-11T15:14:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-11T15:14:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-10-11T15:14:07.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30/obj00000000000000000036 2023-10-11T15:14:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: entry->timestamp=2023-10-11T15:14:04.972589+0000 2023-10-11T15:14:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-10-11T15:14:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: parsed entry: id=00000000002.81.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-10-11T15:14:07.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28/obj00000000000000000020 2023-10-11T15:14:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: entry->timestamp=2023-10-11T15:14:05.851771+0000 2023-10-11T15:14:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.81.6 op=0 op_state=1 2023-10-11T15:14:07.119 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-10-11T15:14:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33/obj00000000000000000019 2023-10-11T15:14:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: entry->timestamp=2023-10-11T15:14:04.805830+0000 2023-10-11T15:14:07.120 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.84.6 op=0 op_state=1 2023-10-11T15:14:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-10-11T15:14:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31/obj00000000000000000003 2023-10-11T15:14:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: entry->timestamp=2023-10-11T15:14:05.624738+0000 2023-10-11T15:14:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-10-11T15:14:07.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-10-11T15:14:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32/obj00000000000000000014 2023-10-11T15:14:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: entry->timestamp=2023-10-11T15:14:05.358929+0000 2023-10-11T15:14:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-10-11T15:14:07.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:entry[obj00000000000000000010]: success 2023-10-11T15:14:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:25:6 marker=00000000002.70.6 timestamp=2023-10-11T15:14:04.619273+0000 2023-10-11T15:14:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-10-11T15:14:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-11T15:14:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:entry[obj00000000000000000039]: success 2023-10-11T15:14:07.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:23:6 marker=00000000002.84.6 timestamp=2023-10-11T15:14:05.688083+0000 2023-10-11T15:14:07.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:entry[obj00000000000000000060]: success 2023-10-11T15:14:07.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:0:6 marker=00000000001.62.6 timestamp=2023-10-11T15:14:05.113916+0000 2023-10-11T15:14:07.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000036[0] 2023-10-11T15:14:07.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: listing bilog for incremental sync; position=00000000001.67.6 2023-10-11T15:14:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000020[0] 2023-10-11T15:14:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: listing bilog for incremental sync; position=00000000002.81.6 2023-10-11T15:14:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:entry[obj00000000000000000021]: success 2023-10-11T15:14:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:26:6 marker=00000000002.59.6 timestamp=2023-10-11T15:14:05.277119+0000 2023-10-11T15:14:07.125 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000019[0] 2023-10-11T15:14:07.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: listing bilog for incremental sync; position=00000000002.84.6 2023-10-11T15:14:07.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000003[0] 2023-10-11T15:14:07.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: listing bilog for incremental sync; position=00000000002.50.6 2023-10-11T15:14:07.126 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000014[0] 2023-10-11T15:14:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: listing bilog for incremental sync; position=00000000002.61.6 2023-10-11T15:14:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.079+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35/obj00000000000000000018 2023-10-11T15:14:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-10-11T15:14:07.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1/obj00000000000000000000 2023-10-11T15:14:07.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: entry->timestamp=2023-10-11T15:14:04.867099+0000 2023-10-11T15:14:07.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-10-11T15:14:07.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000000[0] 2023-10-11T15:14:07.128 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.080+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: listing bilog for incremental sync; position=00000000002.60.6 2023-10-11T15:14:07.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.084+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:entry[obj00000000000000000004]: success 2023-10-11T15:14:07.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.084+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:29:6 marker=00000000002.100.6 timestamp=2023-10-11T15:14:05.520612+0000 2023-10-11T15:14:07.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:entry[obj00000000000000000003]: success 2023-10-11T15:14:07.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:31:6 marker=00000000002.50.6 timestamp=2023-10-11T15:14:05.624738+0000 2023-10-11T15:14:07.129 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:entry[obj00000000000000000014]: success 2023-10-11T15:14:07.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:32:6 marker=00000000002.61.6 timestamp=2023-10-11T15:14:05.358929+0000 2023-10-11T15:14:07.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:entry[obj00000000000000000019]: success 2023-10-11T15:14:07.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:33:6 marker=00000000002.84.6 timestamp=2023-10-11T15:14:04.805830+0000 2023-10-11T15:14:07.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:entry[obj00000000000000000020]: success 2023-10-11T15:14:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:28:6 marker=00000000002.81.6 timestamp=2023-10-11T15:14:05.851771+0000 2023-10-11T15:14:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:entry[obj00000000000000000000]: success 2023-10-11T15:14:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.086+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:1:6 marker=00000000002.60.6 timestamp=2023-10-11T15:14:04.867099+0000 2023-10-11T15:14:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.087+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:entry[obj00000000000000000057]: success 2023-10-11T15:14:07.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.087+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:34:6 marker=00000000001.69.6 timestamp=2023-10-11T15:14:05.599951+0000 2023-10-11T15:14:07.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.087+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:entry[obj00000000000000000053]: success 2023-10-11T15:14:07.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.087+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:27:6 marker=00000000002.56.6 timestamp=2023-10-11T15:14:05.217608+0000 2023-10-11T15:14:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: parsed entry: id=00000000002.102.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-10-11T15:14:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2/obj00000000000000000011 2023-10-11T15:14:07.137 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: entry->timestamp=2023-10-11T15:14:05.710918+0000 2023-10-11T15:14:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.102.6 op=0 op_state=1 2023-10-11T15:14:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000011[0] 2023-10-11T15:14:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.100+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: listing bilog for incremental sync; position=00000000002.102.6 2023-10-11T15:14:07.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.102+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:entry[obj00000000000000000011]: success 2023-10-11T15:14:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.102+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:2:6 marker=00000000002.102.6 timestamp=2023-10-11T15:14:05.710918+0000 2023-10-11T15:14:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-10-11T15:14:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000065 2023-10-11T15:14:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: entry->timestamp=2023-10-11T15:14:05.056868+0000 2023-10-11T15:14:07.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-11T15:14:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: : injecting data sync error on key=obj00000000000000000065 2023-10-11T15:14:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-11T15:14:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:14:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.103+0000 7fe5b57d6780 0 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: ERROR: failed to sync object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000065 2023-10-11T15:14:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-10-11T15:14:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11/obj00000000000000000030 2023-10-11T15:14:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: entry->timestamp=2023-10-11T15:14:05.380646+0000 2023-10-11T15:14:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-10-11T15:14:07.141 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000030[0] 2023-10-11T15:14:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.106+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: listing bilog for incremental sync; position=00000000002.53.6 2023-10-11T15:14:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.107+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:entry[obj00000000000000000030]: success 2023-10-11T15:14:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.107+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:11:6 marker=00000000002.53.6 timestamp=2023-10-11T15:14:05.380646+0000 2023-10-11T15:14:07.142 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-10-11T15:14:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15/obj00000000000000000024 2023-10-11T15:14:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: entry->timestamp=2023-10-11T15:14:05.541424+0000 2023-10-11T15:14:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-10-11T15:14:07.143 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000024[0] 2023-10-11T15:14:07.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.109+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: listing bilog for incremental sync; position=00000000002.29.6 2023-10-11T15:14:07.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.110+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:entry[obj00000000000000000024]: success 2023-10-11T15:14:07.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.110+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:15:6 marker=00000000002.29.6 timestamp=2023-10-11T15:14:05.541424+0000 2023-10-11T15:14:07.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-10-11T15:14:07.144 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18/obj00000000000000000013 2023-10-11T15:14:07.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: entry->timestamp=2023-10-11T15:14:04.721417+0000 2023-10-11T15:14:07.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-10-11T15:14:07.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-10-11T15:14:07.145 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21/obj00000000000000000017 2023-10-11T15:14:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: entry->timestamp=2023-10-11T15:14:05.791403+0000 2023-10-11T15:14:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-10-11T15:14:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-10-11T15:14:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24/obj00000000000000000027 2023-10-11T15:14:07.146 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: entry->timestamp=2023-10-11T15:14:04.951461+0000 2023-10-11T15:14:07.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-10-11T15:14:07.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-10-11T15:14:07.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19/obj00000000000000000045 2023-10-11T15:14:07.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: entry->timestamp=2023-10-11T15:14:05.771122+0000 2023-10-11T15:14:07.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-10-11T15:14:07.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-10-11T15:14:07.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22/obj00000000000000000007 2023-10-11T15:14:07.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: entry->timestamp=2023-10-11T15:14:04.888646+0000 2023-10-11T15:14:07.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-10-11T15:14:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000013[0] 2023-10-11T15:14:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: listing bilog for incremental sync; position=00000000002.68.6 2023-10-11T15:14:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000017[0] 2023-10-11T15:14:07.149 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: listing bilog for incremental sync; position=00000000002.54.6 2023-10-11T15:14:07.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000027[0] 2023-10-11T15:14:07.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: listing bilog for incremental sync; position=00000000002.65.6 2023-10-11T15:14:07.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000045[0] 2023-10-11T15:14:07.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: listing bilog for incremental sync; position=00000000002.71.6 2023-10-11T15:14:07.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000007[0] 2023-10-11T15:14:07.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.112+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: listing bilog for incremental sync; position=00000000002.57.6 2023-10-11T15:14:07.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-10-11T15:14:07.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7/obj00000000000000000016 2023-10-11T15:14:07.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: entry->timestamp=2023-10-11T15:14:04.554577+0000 2023-10-11T15:14:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-10-11T15:14:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000016[0] 2023-10-11T15:14:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.113+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: listing bilog for incremental sync; position=00000000002.52.6 2023-10-11T15:14:07.152 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:entry[obj00000000000000000007]: success 2023-10-11T15:14:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:22:6 marker=00000000002.57.6 timestamp=2023-10-11T15:14:04.888646+0000 2023-10-11T15:14:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:entry[obj00000000000000000013]: success 2023-10-11T15:14:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:18:6 marker=00000000002.68.6 timestamp=2023-10-11T15:14:04.721417+0000 2023-10-11T15:14:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:entry[obj00000000000000000017]: success 2023-10-11T15:14:07.153 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:21:6 marker=00000000002.54.6 timestamp=2023-10-11T15:14:05.791403+0000 2023-10-11T15:14:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:entry[obj00000000000000000027]: success 2023-10-11T15:14:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.115+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:24:6 marker=00000000002.65.6 timestamp=2023-10-11T15:14:04.951461+0000 2023-10-11T15:14:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.116+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:entry[obj00000000000000000045]: success 2023-10-11T15:14:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.116+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:19:6 marker=00000000002.71.6 timestamp=2023-10-11T15:14:05.771122+0000 2023-10-11T15:14:07.154 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.116+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:entry[obj00000000000000000016]: success 2023-10-11T15:14:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.116+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:7:6 marker=00000000002.52.6 timestamp=2023-10-11T15:14:04.554577+0000 2023-10-11T15:14:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-10-11T15:14:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20/obj00000000000000000006 2023-10-11T15:14:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: entry->timestamp=2023-10-11T15:14:05.461664+0000 2023-10-11T15:14:07.155 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-10-11T15:14:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000006[0] 2023-10-11T15:14:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.117+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: listing bilog for incremental sync; position=00000000002.78.6 2023-10-11T15:14:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.119+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:entry[obj00000000000000000048]: success 2023-10-11T15:14:07.156 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.119+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:8:6 marker=00000000001.27.6 timestamp=2023-10-11T15:14:04.535335+0000 2023-10-11T15:14:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.119+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:entry[obj00000000000000000006]: success 2023-10-11T15:14:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.119+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:20:6 marker=00000000002.78.6 timestamp=2023-10-11T15:14:05.461664+0000 2023-10-11T15:14:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-10-11T15:14:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36/obj00000000000000000008 2023-10-11T15:14:07.157 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: entry->timestamp=2023-10-11T15:14:05.134751+0000 2023-10-11T15:14:07.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-10-11T15:14:07.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000008[0] 2023-10-11T15:14:07.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.120+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: listing bilog for incremental sync; position=00000000002.59.6 2023-10-11T15:14:07.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.121+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:entry[obj00000000000000000008]: success 2023-10-11T15:14:07.158 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.121+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:36:6 marker=00000000002.59.6 timestamp=2023-10-11T15:14:05.134751+0000 2023-10-11T15:14:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.122+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:entry[obj00000000000000000062]: success 2023-10-11T15:14:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.122+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:12:6 marker=00000000001.41.6 timestamp=2023-10-11T15:14:04.702427+0000 2023-10-11T15:14:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.124+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: a sync operation returned error 2023-10-11T15:14:07.159 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.124+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: backing out with sync_status=-5 2023-10-11T15:14:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.124+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.124+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.124+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.125+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: sync status for source bucket shard: 0 2023-10-11T15:14:07.160 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.125+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.126+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:entry[obj00000000000000000036]: success 2023-10-11T15:14:07.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.126+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:30:6 marker=00000000001.67.6 timestamp=2023-10-11T15:14:04.972589+0000 2023-10-11T15:14:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: a sync operation returned error 2023-10-11T15:14:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: backing out with sync_status=-5 2023-10-11T15:14:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: incremental sync on bucket failed, retcode=-5 2023-10-11T15:14:07.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: ERROR: incremental sync failed. error: -5 2023-10-11T15:14:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:14:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: sync status for source bucket shard: 0 2023-10-11T15:14:07.163 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.145+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position= 2023-10-11T15:14:07.169 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-10-11T15:14:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35/obj00000000000000000018 2023-10-11T15:14:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: entry->timestamp=2023-10-11T15:14:05.298694+0000 2023-10-11T15:14:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-10-11T15:14:07.170 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000018[0] 2023-10-11T15:14:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.167+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: listing bilog for incremental sync; position=00000000002.43.6 2023-10-11T15:14:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.168+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:entry[obj00000000000000000018]: success 2023-10-11T15:14:07.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.168+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:35:6 marker=00000000002.43.6 timestamp=2023-10-11T15:14:05.298694+0000 2023-10-11T15:14:07.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-10-11T15:14:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: syncing object: dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5/obj00000000000000000065 2023-10-11T15:14:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: entry->timestamp=2023-10-11T15:14:05.056868+0000 2023-10-11T15:14:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-10-11T15:14:07.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 5 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-58[daa316e9-869c-4074-a756-a8adfb4a3573.5751.23])/obj00000000000000000065[0] 2023-10-11T15:14:07.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.187+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: listing bilog for incremental sync; position=00000000001.37.6 2023-10-11T15:14:07.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.231+0000 7fe5b57d6780 10 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:entry[obj00000000000000000065]: success 2023-10-11T15:14:07.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:07.231+0000 7fe5b57d6780 20 RGW-SYNC:bucket[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23<-dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]:inc_sync[dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5]: updating marker marker_oid=bucket.sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23:5:6 marker=00000000001.37.6 timestamp=2023-10-11T15:14:05.056868+0000 2023-10-11T15:14:07.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:07.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:07.257 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:07.347 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:07.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:07.380 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:14:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000000", 2023-10-11T15:14:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 26 2023-10-11T15:14:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.867099Z", 2023-10-11T15:14:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10553887835121780053", 2023-10-11T15:14:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000001", 2023-10-11T15:14:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.387 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.387 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.387 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 18 2023-10-11T15:14:07.387 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.387 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.388 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.388 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.388 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.388 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.388 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.872986Z", 2023-10-11T15:14:07.389 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.389 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.389 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.389 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.389 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.390 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.390 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.390 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.390 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.390 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17055894827889840251", 2023-10-11T15:14:07.391 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.391 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.391 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.391 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.391 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.392 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000002", 2023-10-11T15:14:07.392 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.392 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.392 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.392 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:14:07.393 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.393 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.393 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.393 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.393 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.394 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.394 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.128132Z", 2023-10-11T15:14:07.394 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.394 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.394 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.395 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.395 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.395 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.395 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.395 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.396 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.396 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.9504562076109682229", 2023-10-11T15:14:07.396 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.396 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.397 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.397 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.397 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.397 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000003", 2023-10-11T15:14:07.397 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.398 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.398 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.398 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 25 2023-10-11T15:14:07.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.398 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.399 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.399 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.399 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.399 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.400 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.624738Z", 2023-10-11T15:14:07.400 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.400 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.400 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.400 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.401 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.401 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.401 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.401 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.401 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.402 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.9096584825587265475", 2023-10-11T15:14:07.402 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.402 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.402 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.403 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.403 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.403 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000004", 2023-10-11T15:14:07.403 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.403 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.404 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.404 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 16 2023-10-11T15:14:07.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.404 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.404 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.405 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.405 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.405 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.405 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.520612Z", 2023-10-11T15:14:07.405 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.406 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.406 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.406 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.406 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.407 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.407 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.407 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.407 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8390145599838159106", 2023-10-11T15:14:07.408 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.408 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.408 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.408 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.409 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000005", 2023-10-11T15:14:07.409 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.409 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.409 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 17 2023-10-11T15:14:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.410 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.411 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.411 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.411 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.782685Z", 2023-10-11T15:14:07.411 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.412 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.413 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.413 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.413 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11541754508448271610", 2023-10-11T15:14:07.413 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.413 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.414 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.414 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.414 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.414 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000006", 2023-10-11T15:14:07.414 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 16 2023-10-11T15:14:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.461664Z", 2023-10-11T15:14:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6266412457646626456", 2023-10-11T15:14:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000007", 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:14:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.888646Z", 2023-10-11T15:14:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.811083287111949332", 2023-10-11T15:14:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000008", 2023-10-11T15:14:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:14:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.428 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.428 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.134751Z", 2023-10-11T15:14:07.428 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.428 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.428 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.429 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.429 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.429 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.429 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.429 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.430 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.430 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6096635792028039236", 2023-10-11T15:14:07.430 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.430 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.430 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.431 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.431 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000009", 2023-10-11T15:14:07.431 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.431 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.432 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.432 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:14:07.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.432 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.432 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.433 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.433 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.433 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.433 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.196140Z", 2023-10-11T15:14:07.433 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.434 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.434 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.434 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.434 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.434 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.435 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.435 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.435 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.435 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.7542895136599781916", 2023-10-11T15:14:07.436 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.436 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.436 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.436 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.437 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000010", 2023-10-11T15:14:07.437 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.437 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.437 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.437 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 24 2023-10-11T15:14:07.438 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.438 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.438 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.438 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.439 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.439 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.439 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.619273Z", 2023-10-11T15:14:07.439 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.439 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.440 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.440 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.440 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.440 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.440 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.441 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.441 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.441 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.7299045946894083115", 2023-10-11T15:14:07.441 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.441 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.442 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.442 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.442 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.442 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000011", 2023-10-11T15:14:07.442 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.443 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.443 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.443 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:14:07.443 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.443 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.444 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.444 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.444 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.444 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.444 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.710918Z", 2023-10-11T15:14:07.445 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.445 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.445 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.445 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.446 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.446 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.446 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.446 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.446 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.447 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.3737176303284751350", 2023-10-11T15:14:07.447 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.447 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.447 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.447 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.448 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.448 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000012", 2023-10-11T15:14:07.448 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.448 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.448 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.449 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:14:07.449 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.449 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.449 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.449 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.450 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.450 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.450 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.106669Z", 2023-10-11T15:14:07.450 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.450 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.451 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.451 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.451 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.451 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.453 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.453 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.453 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.453 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.4533477923867705370", 2023-10-11T15:14:07.454 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.454 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.454 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.454 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.455 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000013", 2023-10-11T15:14:07.455 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.455 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.455 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.456 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 45 2023-10-11T15:14:07.456 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.456 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.456 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.456 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.457 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.457 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.457 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.721417Z", 2023-10-11T15:14:07.457 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.457 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.458 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.458 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.458 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.458 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.458 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.459 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.459 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.459 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17118342137448972564", 2023-10-11T15:14:07.459 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.459 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.460 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.460 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.460 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000014", 2023-10-11T15:14:07.460 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.461 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.461 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.461 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.462 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.462 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.462 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.462 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.462 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.463 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.358929Z", 2023-10-11T15:14:07.463 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.463 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.463 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.463 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.464 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.464 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.464 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.464 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.465 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.18297690366553854953", 2023-10-11T15:14:07.465 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.465 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.465 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.465 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.466 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.466 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000015", 2023-10-11T15:14:07.466 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.466 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.466 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.467 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 50 2023-10-11T15:14:07.467 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.467 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.467 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.467 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.468 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.468 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.468 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.439809Z", 2023-10-11T15:14:07.468 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.468 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.469 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.469 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.469 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.469 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.470 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.470 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.470 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.470 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14538889824636312859", 2023-10-11T15:14:07.470 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.471 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.471 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.471 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.471 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.471 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000016", 2023-10-11T15:14:07.472 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.472 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.472 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.472 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:14:07.472 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.473 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.473 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.473 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.473 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.473 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.474 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.554577Z", 2023-10-11T15:14:07.474 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.474 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.474 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.474 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.475 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.475 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.475 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.475 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.476 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.476 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12787278846639085428", 2023-10-11T15:14:07.476 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.476 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.476 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.477 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.477 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.477 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000017", 2023-10-11T15:14:07.477 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.477 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.478 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.478 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:14:07.478 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.478 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.478 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.479 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.479 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.479 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.479 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.791403Z", 2023-10-11T15:14:07.479 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.480 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.480 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.480 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.480 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.480 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.481 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.481 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.481 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.481 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6923950716499928980", 2023-10-11T15:14:07.481 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.482 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.482 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.482 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.483 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000018", 2023-10-11T15:14:07.483 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.483 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.483 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.483 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 23 2023-10-11T15:14:07.484 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.484 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.484 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.484 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.484 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.485 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.485 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.298694Z", 2023-10-11T15:14:07.485 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.485 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.485 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.486 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.486 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.486 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.486 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.486 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.487 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.487 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12655474378522800174", 2023-10-11T15:14:07.487 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.487 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.487 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.488 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.488 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.488 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000019", 2023-10-11T15:14:07.488 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.488 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.489 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.489 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:14:07.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.489 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.489 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.490 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.490 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.490 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.490 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.805830Z", 2023-10-11T15:14:07.490 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.491 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.492 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.492 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.492 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1098039865450984780", 2023-10-11T15:14:07.492 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.493 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.493 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.493 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.493 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000020", 2023-10-11T15:14:07.494 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.494 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.494 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.494 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:14:07.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.495 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.495 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.495 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.495 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.495 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.496 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.851771Z", 2023-10-11T15:14:07.496 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.496 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.496 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.496 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.497 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.497 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.497 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.497 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.498 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5449905295212712947", 2023-10-11T15:14:07.498 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.498 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.498 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.499 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.499 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000021", 2023-10-11T15:14:07.499 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.499 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.499 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.500 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:14:07.500 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.500 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.500 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.500 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.501 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.501 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.501 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.277119Z", 2023-10-11T15:14:07.501 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.501 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.502 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.502 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.502 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.502 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.502 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.503 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.503 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6995232185788119273", 2023-10-11T15:14:07.503 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.504 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.504 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.504 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000022", 2023-10-11T15:14:07.505 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.505 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.505 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.505 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:14:07.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.506 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.506 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.506 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.506 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.506 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.507 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.865067Z", 2023-10-11T15:14:07.507 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.507 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.507 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.507 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.508 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.508 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.508 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.508 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.508 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.509 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10008345815887647342", 2023-10-11T15:14:07.509 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.509 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.509 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.510 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.510 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000023", 2023-10-11T15:14:07.510 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.510 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.510 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.511 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 12 2023-10-11T15:14:07.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.511 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.511 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.777412Z", 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.512 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.513 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.513 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.513 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.513 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.514 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.514 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.514 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.514 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.16368504188860041050", 2023-10-11T15:14:07.514 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.515 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.515 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.515 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000024", 2023-10-11T15:14:07.516 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.516 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.516 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.516 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:14:07.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.517 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.517 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.517 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.517 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.517 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.518 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.541424Z", 2023-10-11T15:14:07.518 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.518 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.518 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.519 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.519 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.519 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.519 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.519 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.520 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14487261729735055824", 2023-10-11T15:14:07.520 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.520 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.520 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.521 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000025", 2023-10-11T15:14:07.521 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.521 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.522 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.522 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.522 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.522 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.523 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.523 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.523 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.523 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.720463Z", 2023-10-11T15:14:07.524 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.524 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.524 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.524 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.524 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.525 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.525 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.525 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.525 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14075035677483649176", 2023-10-11T15:14:07.526 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.526 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.526 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.526 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.527 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000026", 2023-10-11T15:14:07.527 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.527 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.527 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.527 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:14:07.528 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.528 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.528 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.528 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.529 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.529 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.529 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:00.732689Z", 2023-10-11T15:14:07.529 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.529 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15908066006253169355", 2023-10-11T15:14:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000027", 2023-10-11T15:14:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 47 2023-10-11T15:14:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.534 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.534 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.534 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.534 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.534 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.535 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.951461Z", 2023-10-11T15:14:07.535 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.535 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.535 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.535 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.536 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.536 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.536 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.536 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.537 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12950236045111234237", 2023-10-11T15:14:07.537 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.537 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.537 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.538 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.538 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000028", 2023-10-11T15:14:07.538 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.539 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.539 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.539 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.539 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.539 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.540 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.540 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.540 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.540 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.540 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:13:59.170304Z", 2023-10-11T15:14:07.541 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.541 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.541 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.541 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.541 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.542 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.542 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.542 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.542 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.542 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6619237397519159362", 2023-10-11T15:14:07.543 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.543 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.543 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.543 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.543 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.544 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000029", 2023-10-11T15:14:07.544 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.544 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.544 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.544 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.545 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.546 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.546 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.187274Z", 2023-10-11T15:14:07.546 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.546 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.546 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.547 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.547 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.547 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.547 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.548 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.548 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8095364034510527545", 2023-10-11T15:14:07.548 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000030", 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.549 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.550 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:14:07.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.550 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.551 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.551 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.551 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.551 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.551 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.380646Z", 2023-10-11T15:14:07.552 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.552 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.552 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.552 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.552 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8990363291996581896", 2023-10-11T15:14:07.553 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.554 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.554 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.554 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.554 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.554 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000031", 2023-10-11T15:14:07.555 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.555 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.555 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.555 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.556 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.779233Z", 2023-10-11T15:14:07.557 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.557 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.557 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.557 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.557 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.558 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.558 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.558 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.558 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.558 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15802763888473997249", 2023-10-11T15:14:07.559 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.559 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.559 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.560 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.560 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000032", 2023-10-11T15:14:07.560 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.560 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.560 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.561 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:07.561 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.561 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.561 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.561 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.562 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.562 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.562 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.259150Z", 2023-10-11T15:14:07.562 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.562 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.563 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.563 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.563 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.563 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.563 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.564 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.564 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.564 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5026318514589152123", 2023-10-11T15:14:07.564 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.565 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.565 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.565 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.565 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.565 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000033", 2023-10-11T15:14:07.566 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.566 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.566 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.566 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 19 2023-10-11T15:14:07.566 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.567 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.567 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.567 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.567 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.567 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.568 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.035423Z", 2023-10-11T15:14:07.568 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.568 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.568 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.569 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.569 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.569 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.569 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.569 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.570 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.570 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17551825381850744102", 2023-10-11T15:14:07.570 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.570 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.570 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.571 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.571 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.571 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000036", 2023-10-11T15:14:07.571 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.571 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.572 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.572 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 48 2023-10-11T15:14:07.572 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.572 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.572 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.573 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.573 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.573 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.573 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.972589Z", 2023-10-11T15:14:07.573 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.574 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.574 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.574 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.574 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.574 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.575 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.575 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.575 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.575 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.2612409287149233382", 2023-10-11T15:14:07.575 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.576 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.576 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.576 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.576 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.576 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000037", 2023-10-11T15:14:07.577 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.577 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.577 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.577 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.577 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.578 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.578 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.578 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.578 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.579 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.579 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.275683Z", 2023-10-11T15:14:07.579 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.579 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.579 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.580 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.580 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.580 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.580 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.580 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.581 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.581 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5777968676532958036", 2023-10-11T15:14:07.581 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.581 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.581 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.582 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.582 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.582 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000039", 2023-10-11T15:14:07.582 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.582 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.583 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.583 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:14:07.583 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.583 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.583 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.688083Z", 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.584 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.585 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.585 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.585 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.585 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.585 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.586 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.586 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.586 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.3148941605672295793", 2023-10-11T15:14:07.586 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.586 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.587 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.587 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.587 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.587 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000040", 2023-10-11T15:14:07.587 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.588 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.588 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.588 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.588 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.588 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.589 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.589 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.589 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.589 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.590 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.641229Z", 2023-10-11T15:14:07.590 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.590 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.590 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.590 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.591 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.591 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.591 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.591 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.591 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.592 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15056850825530727828", 2023-10-11T15:14:07.592 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.592 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.592 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.592 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.593 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.593 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000042", 2023-10-11T15:14:07.593 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.593 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.593 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.594 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:14:07.594 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.594 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.594 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.594 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.595 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.595 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.595 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.681262Z", 2023-10-11T15:14:07.595 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.596 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.597 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.597 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.597 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17914113993292642855", 2023-10-11T15:14:07.597 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.597 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.598 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.598 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.598 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.598 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000045", 2023-10-11T15:14:07.599 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.599 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.599 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.599 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.599 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.600 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.600 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.600 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.600 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.600 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.601 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.771122Z", 2023-10-11T15:14:07.601 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.601 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.601 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.601 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6616342004263211810", 2023-10-11T15:14:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.604 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.604 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000046", 2023-10-11T15:14:07.604 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.604 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.604 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.605 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.605 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.605 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.605 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.605 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.606 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.606 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.606 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.044242Z", 2023-10-11T15:14:07.606 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.606 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.607 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.607 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.607 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.607 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.607 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.608 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.608 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14328467819523892062", 2023-10-11T15:14:07.608 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.608 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.609 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.609 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.609 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.609 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000048", 2023-10-11T15:14:07.610 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.610 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.610 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 16 2023-10-11T15:14:07.610 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.611 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.611 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.611 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.611 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.611 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.612 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.535335Z", 2023-10-11T15:14:07.612 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.612 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.612 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.612 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.613 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.613 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.613 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.613 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.614 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17590327210947284905", 2023-10-11T15:14:07.614 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.614 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.614 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.615 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.615 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.615 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000049", 2023-10-11T15:14:07.615 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.615 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.616 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.616 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.616 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.616 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.617 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.617 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.617 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.617 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.944664Z", 2023-10-11T15:14:07.617 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.618 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.618 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.618 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.618 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.618 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.619 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.619 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.619 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.619 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.16787624205048617485", 2023-10-11T15:14:07.619 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.620 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.620 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.620 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.620 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000050", 2023-10-11T15:14:07.621 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.621 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.621 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.621 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:14:07.621 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.622 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.623 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.837579Z", 2023-10-11T15:14:07.623 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.623 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.623 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14399131998396818899", 2023-10-11T15:14:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000053", 2023-10-11T15:14:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.217608Z", 2023-10-11T15:14:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10426972888379707186", 2023-10-11T15:14:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000056", 2023-10-11T15:14:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.924438Z", 2023-10-11T15:14:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12553951286011005142", 2023-10-11T15:14:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000057", 2023-10-11T15:14:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.599951Z", 2023-10-11T15:14:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10827083030839609072", 2023-10-11T15:14:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000058", 2023-10-11T15:14:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.645 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.645 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.340678Z", 2023-10-11T15:14:07.645 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.645 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.645 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.646 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.646 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.646 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.646 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.646 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.647 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5188072161464836223", 2023-10-11T15:14:07.647 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.647 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.647 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.648 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.648 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000059", 2023-10-11T15:14:07.648 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.649 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.649 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.649 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.649 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.650 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.650 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.650 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.650 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.650 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.862268Z", 2023-10-11T15:14:07.651 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.651 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.651 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.651 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.651 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.652 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.652 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.652 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.652 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11203340778391555186", 2023-10-11T15:14:07.653 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.653 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.653 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.653 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.654 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000060", 2023-10-11T15:14:07.654 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.654 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.654 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.654 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:07.655 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.655 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.655 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.655 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.655 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.113916Z", 2023-10-11T15:14:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.656 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.657 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.658 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.658 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.658 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5315886021831595772", 2023-10-11T15:14:07.658 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.658 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.659 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.659 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.659 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000061", 2023-10-11T15:14:07.659 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.661 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.662 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.663 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:14:07.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.663 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.663 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.663 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.664 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.664 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.664 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.430765Z", 2023-10-11T15:14:07.664 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.665 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.665 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.665 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.665 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.665 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.666 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.666 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.666 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.666 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5628318352948623665", 2023-10-11T15:14:07.666 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.667 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.667 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.667 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.667 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000062", 2023-10-11T15:14:07.668 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.668 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.668 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.668 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.669 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.669 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.669 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.669 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.669 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.670 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.702427Z", 2023-10-11T15:14:07.670 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.670 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.670 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.671 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.671 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.671 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.671 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.671 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.672 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.672 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.7523838260528709362", 2023-10-11T15:14:07.672 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.672 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.672 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.673 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.673 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.673 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000065", 2023-10-11T15:14:07.673 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.673 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.674 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.674 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.674 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.674 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.675 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.675 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.675 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.675 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.056868Z", 2023-10-11T15:14:07.675 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.676 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.676 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.676 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.676 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.676 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.677 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.677 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.677 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.677 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14094122599233576459", 2023-10-11T15:14:07.677 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.678 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.678 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.678 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000076", 2023-10-11T15:14:07.679 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.679 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.679 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.679 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.679 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.680 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.680 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.680 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.680 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.680 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.681 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:13:59.090821Z", 2023-10-11T15:14:07.681 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.681 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.681 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.682 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.682 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.682 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.682 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.682 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.683 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.683 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14174141741055437830", 2023-10-11T15:14:07.683 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.683 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.683 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.684 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.684 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000133", 2023-10-11T15:14:07.684 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.684 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.685 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.685 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.685 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.685 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.685 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.686 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.686 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.686 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.686 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.508570Z", 2023-10-11T15:14:07.687 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.687 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.687 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.687 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.687 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.688 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.688 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.688 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.688 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.688 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5948672546947182954", 2023-10-11T15:14:07.689 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.689 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.689 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.689 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.689 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.690 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000162", 2023-10-11T15:14:07.690 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.690 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.690 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.690 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:07.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.691 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.691 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.691 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.691 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.692 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.692 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.943763Z", 2023-10-11T15:14:07.692 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.692 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.692 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.693 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.694 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8862936101971421327", 2023-10-11T15:14:07.694 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.694 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.694 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.695 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.695 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.695 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000271", 2023-10-11T15:14:07.695 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.695 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.696 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.696 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:14:07.696 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.696 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.696 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.697 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.697 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.697 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.697 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.272549Z", 2023-10-11T15:14:07.698 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.698 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.698 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.698 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.698 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.18389063203683903731", 2023-10-11T15:14:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000283", 2023-10-11T15:14:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.111758Z", 2023-10-11T15:14:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14658576276263148958", 2023-10-11T15:14:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000317", 2023-10-11T15:14:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.026360Z", 2023-10-11T15:14:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5669156325380591795", 2023-10-11T15:14:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.712 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.713 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.713 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.713 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000423", 2023-10-11T15:14:07.713 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.713 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.714 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:14:07.714 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.714 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.714 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.715 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.715 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.715 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.715 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.715 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.006914Z", 2023-10-11T15:14:07.716 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.716 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.716 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.716 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.716 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.717 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.717 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.717 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.717 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.717 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.3263190487366778934", 2023-10-11T15:14:07.718 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.718 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.718 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.718 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:07.718 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:14:07.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:07.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:07.720 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:07.772 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:07.772 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:07.803 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:14:07.804 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.804 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000000", 2023-10-11T15:14:07.804 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.805 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.805 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.805 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:14:07.805 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.806 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.806 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.806 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.806 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.806 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.807 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.867099Z", 2023-10-11T15:14:07.807 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.807 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.808 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.808 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.808 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.809 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.809 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.809 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.810 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.810 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_W4T9wpaxn9bLsro56KuipUKRPO85cvB", 2023-10-11T15:14:07.810 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.810 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.811 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.811 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.811 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.812 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000001", 2023-10-11T15:14:07.812 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.812 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.813 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.813 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.813 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.813 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.814 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.814 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.814 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.814 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.872986Z", 2023-10-11T15:14:07.815 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.815 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.815 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.815 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.815 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.816 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.816 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.816 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.817 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_FjFhG_sVDvsnVNcPvi3hXrYFnKZneMz", 2023-10-11T15:14:07.817 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.817 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.817 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.818 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.818 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000002", 2023-10-11T15:14:07.819 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.819 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.819 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.819 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:14:07.820 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.820 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.820 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.820 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.821 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.821 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.821 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.128132Z", 2023-10-11T15:14:07.821 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.821 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.822 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.822 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.822 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.822 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.823 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.823 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.823 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_iwqQSculbQqB5eIqLc8OX58Ae0HI73q", 2023-10-11T15:14:07.824 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.824 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.824 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.824 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.824 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.825 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000003", 2023-10-11T15:14:07.825 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.825 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.825 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.825 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.826 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.826 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.826 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.826 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.827 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.827 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.827 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.624738Z", 2023-10-11T15:14:07.828 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.828 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.828 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.829 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.829 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.829 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.830 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.830 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.830 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.831 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_ZuivMYKtG8vWFG57zBr6MxBl5g_owh_", 2023-10-11T15:14:07.831 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.831 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.832 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.832 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.832 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.833 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000004", 2023-10-11T15:14:07.833 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.833 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.834 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.834 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.834 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.835 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.835 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.835 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.836 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.836 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.836 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.520612Z", 2023-10-11T15:14:07.837 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.837 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.837 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.838 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.838 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.839 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.839 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.839 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.840 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.840 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_YN5P3B8kaLZa8K_-ZGZuwv6n4jdiZ6B", 2023-10-11T15:14:07.840 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.841 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.841 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.842 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.842 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000005", 2023-10-11T15:14:07.842 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.843 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.843 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.843 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.843 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.844 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.844 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.844 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.844 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.845 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.845 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.782685Z", 2023-10-11T15:14:07.845 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.845 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.846 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.846 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.846 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.846 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.846 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.847 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.847 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_aCfXbhaPhnYvaoX2rMSMgJcdD_NSDdu", 2023-10-11T15:14:07.847 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.848 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.848 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.848 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.848 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000006", 2023-10-11T15:14:07.849 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.849 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.849 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.849 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.849 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.850 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.850 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.850 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.850 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.851 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.851 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.461664Z", 2023-10-11T15:14:07.851 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.851 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.852 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.852 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.852 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.852 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.852 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.853 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.853 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.853 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_RQjWQ8IQzJiIftT8QF_CGbRiJLTCGuC", 2023-10-11T15:14:07.853 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.854 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.854 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.854 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.855 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.855 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000007", 2023-10-11T15:14:07.855 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.855 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.855 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.856 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.856 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.856 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.856 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.856 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.857 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.857 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.857 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.888646Z", 2023-10-11T15:14:07.857 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.858 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.858 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.858 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.858 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.859 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.859 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.859 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.859 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.859 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_HtUH-5dRVzzonPUpiOBiCh1zMGDnwC8", 2023-10-11T15:14:07.860 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.860 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.860 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.860 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.860 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.861 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000008", 2023-10-11T15:14:07.861 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.861 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.861 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.862 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.862 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.862 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.862 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.863 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.863 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.863 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.864 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.134751Z", 2023-10-11T15:14:07.864 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.864 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.866 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.866 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.867 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.867 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.867 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.867 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.868 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.868 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_OaHYBl7Qcnp_KCdAx4YcxrIL4F8c1vF", 2023-10-11T15:14:07.868 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.868 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.869 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.869 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.869 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.869 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000009", 2023-10-11T15:14:07.870 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.870 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.870 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.870 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.871 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.871 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.871 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.871 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.872 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.872 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.196140Z", 2023-10-11T15:14:07.872 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.872 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.873 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.873 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.873 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.873 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.874 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.874 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.874 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_pU1VBdEiXHTyCu_Q_38_Ncu5NTVxmti", 2023-10-11T15:14:07.875 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.875 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.875 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.875 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.875 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.876 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000010", 2023-10-11T15:14:07.876 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.876 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.876 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.876 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.877 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.877 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.877 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.877 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.878 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.878 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.878 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.619273Z", 2023-10-11T15:14:07.878 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.879 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.879 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.879 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.879 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.879 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.880 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.880 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.880 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.880 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_zoQFo-vte5deDvXVvYgl2_fQDM_IoZ3", 2023-10-11T15:14:07.880 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.881 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.881 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.881 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.882 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.882 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000011", 2023-10-11T15:14:07.883 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.883 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.883 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.884 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.884 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.884 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.885 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.885 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.885 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.886 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.886 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.710918Z", 2023-10-11T15:14:07.886 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.887 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.887 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.888 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.888 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.888 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.888 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.888 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.889 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.889 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_AF3O4SolHinf4e2IoyMeAIZjPP4f62W", 2023-10-11T15:14:07.889 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.890 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.890 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000012", 2023-10-11T15:14:07.890 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.891 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.891 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.891 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.891 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.892 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.892 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.892 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.892 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.892 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.893 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.106669Z", 2023-10-11T15:14:07.893 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.893 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.893 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.893 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.894 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.894 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.894 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.894 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.895 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.895 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_lHZ2Fvcekb-1onlzV7FsR5qndiR8Y3Q", 2023-10-11T15:14:07.895 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.895 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.895 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.896 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.896 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000013", 2023-10-11T15:14:07.896 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.897 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.897 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.897 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 34 2023-10-11T15:14:07.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.897 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.898 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.898 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.898 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.898 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.898 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.721417Z", 2023-10-11T15:14:07.899 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.899 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.899 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.899 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.899 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.900 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.900 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.900 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.901 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_MeR_vYFSHqS45TNE9vyv9b5PGf3f80Y", 2023-10-11T15:14:07.901 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.901 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.902 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000014", 2023-10-11T15:14:07.902 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.902 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.903 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.903 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:07.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.903 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.903 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.904 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.904 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.904 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.904 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.358929Z", 2023-10-11T15:14:07.904 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.905 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.905 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.905 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.905 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.906 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.906 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.906 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.907 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_nkmd6UkfWzXWwCZdsPig0LUhqiON0w-", 2023-10-11T15:14:07.907 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.907 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.907 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.907 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.908 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000015", 2023-10-11T15:14:07.908 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.908 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.909 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 35 2023-10-11T15:14:07.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.909 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.909 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.910 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.910 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.910 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.911 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.439809Z", 2023-10-11T15:14:07.911 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.911 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.911 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.912 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.912 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.912 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.913 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.913 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.913 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.913 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_02x-Ipxer2W9GZWT08W_z-MSwvj6uCe", 2023-10-11T15:14:07.913 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.914 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.914 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.914 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.915 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000016", 2023-10-11T15:14:07.915 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.915 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.915 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.915 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.916 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.916 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.916 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.916 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.917 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.917 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.917 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.554577Z", 2023-10-11T15:14:07.917 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.917 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.918 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.918 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.918 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.918 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.918 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.919 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.919 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_bdQ2s1dJv2NRzdfIM7ddbeCzHBwZZko", 2023-10-11T15:14:07.919 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.920 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.920 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.920 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.921 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000017", 2023-10-11T15:14:07.921 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.921 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.921 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.921 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.922 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.922 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.922 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.923 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.923 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.923 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.791403Z", 2023-10-11T15:14:07.923 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.923 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.924 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.924 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.924 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.924 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.924 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.925 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.925 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.925 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_6HLdUZcP5PtpxWCD_k3tqTT8sXidqP7", 2023-10-11T15:14:07.925 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.925 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.926 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.926 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.926 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.926 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000018", 2023-10-11T15:14:07.926 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.927 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.927 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.927 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.927 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.928 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.929 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.298694Z", 2023-10-11T15:14:07.929 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.929 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.929 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.929 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.930 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.930 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.930 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_2bAszb5bF1rcHDIWMtffczEzcqW3s-d", 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.931 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.932 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.932 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000019", 2023-10-11T15:14:07.932 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.933 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.933 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.933 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.933 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.933 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.934 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.934 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.934 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.934 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.934 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.805830Z", 2023-10-11T15:14:07.935 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.935 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.935 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.935 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.935 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.936 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.936 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.936 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.936 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.936 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_5oQhjADv7qt0hYnCxMVklmO1W7Acghe", 2023-10-11T15:14:07.937 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.937 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.937 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.937 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.937 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.938 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000020", 2023-10-11T15:14:07.938 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.938 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.938 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.938 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.939 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.940 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.940 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.940 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.941 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.941 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.851771Z", 2023-10-11T15:14:07.941 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.942 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.942 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.943 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.943 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.943 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.944 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.944 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.944 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_TA7JSx8_GJW3Vy_fQ-SqlPYeK2M2CON", 2023-10-11T15:14:07.945 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.945 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.945 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.946 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.946 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.946 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000021", 2023-10-11T15:14:07.947 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.947 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.947 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.948 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.949 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.949 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.277119Z", 2023-10-11T15:14:07.949 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.949 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.950 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.950 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.950 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.950 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.951 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.951 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.951 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.951 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Cdy_1rug733gDUpkZbtF_mmRCvE-6MI", 2023-10-11T15:14:07.951 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.952 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.952 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.952 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.952 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.952 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000022", 2023-10-11T15:14:07.953 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.953 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.953 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.954 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:07.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.954 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.954 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.955 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.955 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.955 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.955 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.865067Z", 2023-10-11T15:14:07.956 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.956 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.956 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.956 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.956 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.957 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.957 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.957 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.957 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.958 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_MQppqyzvwVNiEIoVpz8bkqNeFiwnbC1", 2023-10-11T15:14:07.958 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.958 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.959 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.959 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.959 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.959 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000023", 2023-10-11T15:14:07.959 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.960 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.960 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.960 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.960 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.960 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.961 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.961 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.961 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.961 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:07.962 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.777412Z", 2023-10-11T15:14:07.962 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.962 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.962 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.962 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.963 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.963 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.963 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.963 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.964 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.964 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "__c48DniDoJOPZ9qiYNHcb3WxXSJvok6", 2023-10-11T15:14:07.964 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.964 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.964 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000024", 2023-10-11T15:14:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.541424Z", 2023-10-11T15:14:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.969 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.969 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.969 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.969 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.970 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.970 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_7yF_qRMRx6jp9uLaGqI8x5tTgbGBvbC", 2023-10-11T15:14:07.970 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.970 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.970 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.971 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.971 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.971 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000025", 2023-10-11T15:14:07.971 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.972 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.972 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.972 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:07.972 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.973 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.973 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.973 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.973 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.974 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.974 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.720463Z", 2023-10-11T15:14:07.974 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.974 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.975 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.975 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.975 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.975 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.975 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.976 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.976 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.976 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_WZzc9AH2DlsWnduCDaZuyTwCB63R_b6", 2023-10-11T15:14:07.976 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.977 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.977 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.977 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.977 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000026", 2023-10-11T15:14:07.978 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.978 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.978 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.978 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:07.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.979 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.979 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.980 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.980 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.980 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:07.980 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:00.732689Z", 2023-10-11T15:14:07.981 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.981 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.981 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.981 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.981 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.982 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.982 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.982 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.982 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.983 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Ki0UIzVH-piWmvclMSd5ovNQsDSEE75", 2023-10-11T15:14:07.983 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.983 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.983 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.984 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000027", 2023-10-11T15:14:07.984 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.985 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.985 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.985 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 36 2023-10-11T15:14:07.985 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.985 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.986 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.986 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.986 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.986 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:07.987 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.951461Z", 2023-10-11T15:14:07.987 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.987 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.987 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.987 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.988 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.988 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.988 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.988 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.988 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.989 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_oZJ3_HklzFZBIC8Abb3QZWjzgcsOHrp", 2023-10-11T15:14:07.989 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.989 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.989 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.990 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.990 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000028", 2023-10-11T15:14:07.990 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.990 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.991 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.991 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:07.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.991 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.991 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.992 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.992 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.992 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.992 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:13:59.170304Z", 2023-10-11T15:14:07.992 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.993 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.993 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.993 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.993 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:07.993 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:07.994 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:07.994 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:07.994 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.994 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_oKlQMkFp2Tu5vx5ovR13B-f-UVRey11", 2023-10-11T15:14:07.995 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:07.995 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:07.995 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:07.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.996 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:07.996 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000029", 2023-10-11T15:14:07.996 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:07.996 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:07.997 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:07.997 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:07.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:07.997 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:07.997 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:07.998 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:07.998 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:07.998 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:07.998 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.187274Z", 2023-10-11T15:14:07.999 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:07.999 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:07.999 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:07.999 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:07.999 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.000 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.000 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.000 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.000 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.001 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_m9Mr_b6VCXf6pDdfK9WbId4EaJA76yQ", 2023-10-11T15:14:08.001 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.001 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.002 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.002 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000030", 2023-10-11T15:14:08.002 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.002 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.003 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.003 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:08.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.003 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.003 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.004 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.004 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.004 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:08.004 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.380646Z", 2023-10-11T15:14:08.005 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.005 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.005 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.005 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.006 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.006 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.006 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.006 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.007 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.007 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_22Xv4_cRT14lD0RU08NdYzvLbzsK-il", 2023-10-11T15:14:08.007 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.007 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.008 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.008 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.008 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000031", 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.009 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.010 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.010 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.010 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.010 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.011 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.779233Z", 2023-10-11T15:14:08.011 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.011 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.011 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.012 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.012 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.012 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.013 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.013 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.013 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.013 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_1LJlJ9ZGFUH4qM_CTnWrUDM-k1hL6CV", 2023-10-11T15:14:08.013 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.014 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.014 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.015 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000032", 2023-10-11T15:14:08.015 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.015 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.015 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.016 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:08.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.016 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.016 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.017 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.017 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.017 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.017 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.259150Z", 2023-10-11T15:14:08.017 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.018 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.018 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.018 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.018 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.019 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.019 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.019 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.020 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_TbfEakl0t4oez1VBSVjhxAF3CL3YgP5", 2023-10-11T15:14:08.020 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.020 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.020 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.021 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000033", 2023-10-11T15:14:08.021 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.021 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.022 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.022 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:08.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.022 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.022 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.023 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.023 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.024 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:08.024 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.035423Z", 2023-10-11T15:14:08.024 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.024 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.025 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.025 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.025 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.025 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.025 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.026 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.026 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.026 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Jnc-DpMzxK966-Z4MPmCq6qCvfYzQSw", 2023-10-11T15:14:08.027 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.027 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.027 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.028 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000036", 2023-10-11T15:14:08.028 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.028 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.028 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.028 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 37 2023-10-11T15:14:08.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.029 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.029 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.029 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.029 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.030 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.030 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.972589Z", 2023-10-11T15:14:08.030 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.031 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.031 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.031 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.031 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.031 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.032 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.032 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.032 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_okXLPDWLb2Hl-qcwxPyG9srsoiSYPGv", 2023-10-11T15:14:08.033 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.033 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.034 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000037", 2023-10-11T15:14:08.034 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.034 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.035 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.035 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:14:08.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.035 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.036 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.036 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.036 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.036 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.037 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.275683Z", 2023-10-11T15:14:08.037 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.037 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.037 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.038 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.038 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.038 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.038 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.039 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.039 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_tiYYtmwNueYDhG0F6Yl8-TUXQf2HV4_", 2023-10-11T15:14:08.039 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.039 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.040 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.040 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.040 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000039", 2023-10-11T15:14:08.040 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.041 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.041 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.041 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.042 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.042 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.042 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.042 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.042 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:08.043 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.688083Z", 2023-10-11T15:14:08.043 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.043 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.043 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.043 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.044 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.044 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.044 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.044 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.045 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.045 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_mycqRWDyD2mXUaN3iDUvaUJZNjnGzkI", 2023-10-11T15:14:08.045 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.045 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.046 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.046 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.046 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000040", 2023-10-11T15:14:08.046 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.047 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.047 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.047 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:08.048 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.048 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.048 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.048 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.048 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.049 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.049 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.641229Z", 2023-10-11T15:14:08.049 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.049 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.049 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.050 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.050 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.050 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.050 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.050 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.051 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.051 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_8s-D9lluaMf_KFHRFMuAX9W5NtBBy_w", 2023-10-11T15:14:08.051 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.051 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.052 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.052 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.052 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.052 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000042", 2023-10-11T15:14:08.052 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.053 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.053 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.053 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:08.053 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.053 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.054 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.054 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.054 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.054 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.055 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:02.681262Z", 2023-10-11T15:14:08.055 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.055 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.055 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.056 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.056 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.056 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.056 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.057 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.057 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.057 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_vM_qy4C5laNWN_JqKJsKRglcK69k8sW", 2023-10-11T15:14:08.057 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.057 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.058 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.058 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.058 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.058 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000045", 2023-10-11T15:14:08.059 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.059 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.059 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.059 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.060 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.061 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.061 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.771122Z", 2023-10-11T15:14:08.061 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.062 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.062 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.062 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.062 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.063 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.063 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.063 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.063 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.063 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_8_3X_GGwGJDgROL79zHxV0OJM54gzKP", 2023-10-11T15:14:08.064 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.064 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.064 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.065 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000046", 2023-10-11T15:14:08.065 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.065 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.066 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.066 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.066 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.067 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.067 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.070 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.071 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.071 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.071 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.044242Z", 2023-10-11T15:14:08.071 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.072 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.072 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.073 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.073 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.073 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.074 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.074 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.074 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_jJ_NY6Esl2JV2uje5Yss4I0sqinhVt_", 2023-10-11T15:14:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000048", 2023-10-11T15:14:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:14:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.535335Z", 2023-10-11T15:14:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_WxzvkmOE_SD_pegUtXF3cxnJuqAEnFZ", 2023-10-11T15:14:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000049", 2023-10-11T15:14:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.944664Z", 2023-10-11T15:14:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_YX9xtc6DB7-CHb66Y1a9prHa2QWEuEd", 2023-10-11T15:14:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.089 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.089 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.089 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000050", 2023-10-11T15:14:08.089 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.089 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.090 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.090 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:08.090 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.091 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.091 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.091 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.091 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.092 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.837579Z", 2023-10-11T15:14:08.092 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.092 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.092 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.093 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.093 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.093 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.093 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.093 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.094 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.094 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_2LM0qw0nHiNXRxqHf-FVjjcL8fa0S-U", 2023-10-11T15:14:08.094 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.094 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.094 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.095 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.095 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000053", 2023-10-11T15:14:08.095 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.096 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.096 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.096 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:08.096 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.096 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.097 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.097 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.097 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.097 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:08.098 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.217608Z", 2023-10-11T15:14:08.098 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.098 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.098 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.099 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.099 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.099 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.099 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.099 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.100 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.100 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_gQzhwgpJ3LETO990Z4Yo_J22MdqpPer", 2023-10-11T15:14:08.100 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.100 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.100 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.101 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.101 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.101 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000056", 2023-10-11T15:14:08.101 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.101 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.102 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.102 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.102 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.102 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.103 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.103 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.103 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.103 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:08.103 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.924438Z", 2023-10-11T15:14:08.104 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.104 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.104 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.104 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.104 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.105 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.105 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.105 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.105 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.106 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_t6x6ltVizb6_DsEettnEXuIMfExYzp7", 2023-10-11T15:14:08.106 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.106 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.106 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.106 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.107 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000057", 2023-10-11T15:14:08.107 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.107 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.108 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.108 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.108 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.108 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.108 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.109 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.109 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.109 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.109 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.599951Z", 2023-10-11T15:14:08.110 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.110 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.110 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.110 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.110 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.111 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.111 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.111 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.111 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.112 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_cJ4u2hbhv_r0vz5qCToYnQNBnSRXnKA", 2023-10-11T15:14:08.112 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.112 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.112 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.113 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.113 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000058", 2023-10-11T15:14:08.113 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.113 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.114 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.114 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.114 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.114 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.114 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.115 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.115 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.115 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.115 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.340678Z", 2023-10-11T15:14:08.116 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.116 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.116 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.116 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.116 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.117 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.117 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.117 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.117 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.117 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_YWDnmhMNwAfAJYGIhrR8w3eH2SBmHU8", 2023-10-11T15:14:08.118 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.118 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.118 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.119 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.119 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000059", 2023-10-11T15:14:08.119 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.119 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.120 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.120 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:08.120 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.120 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.120 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.121 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.121 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.121 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.121 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:01.862268Z", 2023-10-11T15:14:08.122 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.122 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.122 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.122 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.122 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.123 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.123 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.123 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.123 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_UKk72-A06FHLFRHXizFCsgkAvJgQUEP", 2023-10-11T15:14:08.124 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.124 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.124 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.124 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.125 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000060", 2023-10-11T15:14:08.125 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.125 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.125 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.125 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.126 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.126 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.126 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.127 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.127 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.127 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.113916Z", 2023-10-11T15:14:08.127 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.127 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.129 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.129 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.129 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_oYkvtzeeQVOWg9uSRtyXavUFP4yHNLb", 2023-10-11T15:14:08.129 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.129 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.130 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.130 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000061", 2023-10-11T15:14:08.131 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.131 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.131 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.131 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:14:08.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.132 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.132 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.132 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.132 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.132 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.133 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.430765Z", 2023-10-11T15:14:08.133 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.133 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.133 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.133 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.134 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.134 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.134 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.134 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.135 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.135 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_eYboPU_hBunU5rWkx8X9ANtaIU5xU9q", 2023-10-11T15:14:08.135 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.135 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.135 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.136 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000062", 2023-10-11T15:14:08.136 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.137 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.137 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.137 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:08.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.137 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.138 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.138 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.138 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.138 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:14:08.139 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.702427Z", 2023-10-11T15:14:08.139 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.139 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.140 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.140 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.140 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.140 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.141 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.141 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.141 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_MTObOQm-wjQcmAiBbB_ArHReBp4mMig", 2023-10-11T15:14:08.141 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.142 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.142 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.142 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.142 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000065", 2023-10-11T15:14:08.143 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.143 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.143 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.143 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:14:08.143 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.144 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.144 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.144 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.144 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.144 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.145 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:05.056868Z", 2023-10-11T15:14:08.145 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.145 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.145 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.146 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.146 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.146 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.146 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.147 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.147 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.147 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_X4vStrTw7bL057nJb1nOXIuiN3vhxMs", 2023-10-11T15:14:08.147 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.148 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.148 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.148 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000076", 2023-10-11T15:14:08.149 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.149 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.149 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.149 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:14:08.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.150 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.150 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.150 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.150 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.150 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.151 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:13:59.090821Z", 2023-10-11T15:14:08.151 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.151 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.151 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_1zRZS8DZFBSekaudgjwx6AjFi-QD_89", 2023-10-11T15:14:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000133", 2023-10-11T15:14:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:14:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.508570Z", 2023-10-11T15:14:08.157 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.157 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.157 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.157 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.157 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.158 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.158 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.158 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.159 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_X9zCnoF0__JnHAJ6GOWHzzrQs7iA_ZF", 2023-10-11T15:14:08.159 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.159 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.159 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.159 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.160 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.160 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000162", 2023-10-11T15:14:08.160 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.160 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.160 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:14:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.162 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.162 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:14:08.162 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.943763Z", 2023-10-11T15:14:08.162 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.162 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.163 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.163 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.163 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.163 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.164 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.164 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.164 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.164 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_dmnSDpSp0YCp5Bzc-EAWipD82Xl5amV", 2023-10-11T15:14:08.164 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.165 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.165 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.166 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.167 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000271", 2023-10-11T15:14:08.167 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.167 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.167 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.167 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:14:08.168 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.168 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.168 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.168 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.169 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.169 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.169 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:03.272549Z", 2023-10-11T15:14:08.169 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.170 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.170 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.170 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.171 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.171 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.171 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.171 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.172 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.172 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_HaW_6Y7B5Zc8HBSieiFS1Rfhsd-w9Bp", 2023-10-11T15:14:08.172 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.173 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.173 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.173 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.174 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000283", 2023-10-11T15:14:08.174 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.174 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.174 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.175 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:08.175 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.175 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.176 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.176 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.176 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.176 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.177 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.111758Z", 2023-10-11T15:14:08.177 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.177 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.177 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.178 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.178 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.178 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.178 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.179 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.179 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.179 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_U_W2IqXFnP0KG-W833Ga2idPT55sLB1", 2023-10-11T15:14:08.179 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.180 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.180 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.180 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.181 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.181 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000317", 2023-10-11T15:14:08.181 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.181 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.181 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.182 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:14:08.182 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.182 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.182 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.183 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.183 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.183 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:14:08.183 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.026360Z", 2023-10-11T15:14:08.183 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.184 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.184 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.184 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.184 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.185 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.185 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.185 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.186 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_9gqr5NI3Aol0yJiDHgjNkLzsOj7v6K4", 2023-10-11T15:14:08.186 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.186 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.186 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.187 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.187 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:08.187 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj00000000000000000423", 2023-10-11T15:14:08.187 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:14:08.187 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:14:08.188 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:14:08.188 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:14:08.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.188 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:14:08.189 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:14:08.189 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:14:08.189 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:14:08.189 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:14:08.189 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:14:04.006914Z", 2023-10-11T15:14:08.190 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:14:08.190 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:14:08.190 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:14:08.190 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:14:08.191 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:14:08.191 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:14:08.191 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:14:08.191 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:14:08.192 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:08.192 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_XbZcO8ydh3qICn1aMX3rZPcHcQuNqsM", 2023-10-11T15:14:08.192 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:14:08.192 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:14:08.193 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:14:08.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:08.193 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:14:08.194 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:14:08.195 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:14:08.197 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:14:08.198 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:14:08.222 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:14:08.223 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:14:08.223 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:14:09.224 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:09.250 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:14:09.250 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:14:09.250 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:14:09.251 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:14:09.252 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-58 2023-10-11T15:14:09.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:14:09.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:14:09.253 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:14:09.349 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:09.350 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:09.389 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:09.386+0000 7ff0f23a4780 1 bucket sync caught up with source: 2023-10-11T15:14:09.390 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.62.6, 00000000002.60.6, 00000000002.102.6, 00000000002.72.6, 00000000002.59.6, 00000000001.37.6, 00000000002.91.6, 00000000002.52.6, 00000000001.27.6, 00000000002.45.6, , 00000000002.53.6, 00000000001.41.6, 00000000002.64.6, 00000000002.62.6, 00000000002.29.6, , , 00000000002.68.6, 00000000002.71.6, 00000000002.78.6, 00000000002.54.6, 00000000002.57.6, 00000000002.84.6, 00000000002.65.6, 00000000002.70.6, 00000000002.59.6, 00000000002.56.6, 00000000002.81.6, 00000000002.100.6, 00000000001.67.6, 00000000002.50.6, 00000000002.61.6, 00000000002.84.6, 00000000001.69.6, 00000000002.43.6, 00000000002.59.6] 2023-10-11T15:14:09.390 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.62.6, 00000000002.60.6, 00000000002.102.6, 00000000002.72.6, 00000000002.59.6, 00000000001.37.6, 00000000002.91.6, 00000000002.52.6, 00000000001.27.6, 00000000002.45.6, , 00000000002.53.6, 00000000001.41.6, 00000000002.64.6, 00000000002.62.6, 00000000002.29.6, , , 00000000002.68.6, 00000000002.71.6, 00000000002.78.6, 00000000002.54.6, 00000000002.57.6, 00000000002.84.6, 00000000002.65.6, 00000000002.70.6, 00000000002.59.6, 00000000002.56.6, 00000000002.81.6, 00000000002.100.6, 00000000001.67.6, 00000000002.50.6, 00000000002.61.6, 00000000002.84.6, 00000000001.69.6, 00000000002.43.6, 00000000002.59.6] 2023-10-11T15:14:09.390 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:09.386+0000 7ff0f23a4780 0 bucket checkpoint complete 2023-10-11T15:14:09.402 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-58 2023-10-11T15:14:09.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:14:09.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:14:09.404 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:14:09.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:09.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:09.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:09.496+0000 7efee1414780 1 waiting for incremental sync to catch up: 2023-10-11T15:14:09.499 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.272.6, 00000000006.72.6, 00000000006.195.6, 00000000009.327.6, 00000000001.295.6, 00000000008.170.6, 00000000008.168.6, 00000000006.261.6, 00000000001.233.6, 00000000007.290.6, 00000000003.198.6] 2023-10-11T15:14:09.499 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000005.280.6, 00000000006.72.6, 00000000006.195.6, 00000000009.327.6, 00000000005.303.6, 00000000008.170.6, 00000000008.168.6, 00000000006.261.6, 00000000007.245.6, 00000000007.290.6, 00000000003.198.6] 2023-10-11T15:14:14.500 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:14.498+0000 7efee1414780 1 bucket sync caught up with source: 2023-10-11T15:14:14.501 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000005.280.6, 00000000006.72.6, 00000000006.195.6, 00000000009.327.6, 00000000005.303.6, 00000000008.170.6, 00000000008.168.6, 00000000006.261.6, 00000000007.245.6, 00000000007.290.6, 00000000003.198.6] 2023-10-11T15:14:14.501 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000005.280.6, 00000000006.72.6, 00000000006.195.6, 00000000009.327.6, 00000000005.303.6, 00000000008.170.6, 00000000008.168.6, 00000000006.261.6, 00000000007.245.6, 00000000007.290.6, 00000000003.198.6] 2023-10-11T15:14:14.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:14.498+0000 7efee1414780 0 bucket checkpoint complete 2023-10-11T15:14:14.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-58 zones={a2, a1} 2023-10-11T15:14:14.518 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.518 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.519 DEBUG:boto:Method: HEAD 2023-10-11T15:14:14.519 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:14.519 DEBUG:boto:Data: 2023-10-11T15:14:14.520 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.521 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.521 DEBUG:boto:Params: {} 2023-10-11T15:14:14.522 DEBUG:boto:Token: None 2023-10-11T15:14:14.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/ 2023-10-11T15:14:14.523 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY= 2023-10-11T15:14:14.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY=', 'Content-Length': '0'} 2023-10-11T15:14:14.524 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-11T15:14:14.525 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:14:14.776 DEBUG:boto:Token: None 2023-10-11T15:14:14.777 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/ 2023-10-11T15:14:14.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY= 2023-10-11T15:14:14.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY=', 'Content-Length': '0'} 2023-10-11T15:14:14.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073e14afc4880a198-006526bbc6-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.785 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.785 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.785 DEBUG:boto:Method: HEAD 2023-10-11T15:14:14.786 DEBUG:boto:Path: /dghqvd-58/ 2023-10-11T15:14:14.786 DEBUG:boto:Data: 2023-10-11T15:14:14.787 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.788 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.788 DEBUG:boto:Params: {} 2023-10-11T15:14:14.789 DEBUG:boto:Token: None 2023-10-11T15:14:14.789 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/ 2023-10-11T15:14:14.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY= 2023-10-11T15:14:14.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pQJbmXbvZU3fznxaUfX28nrquUY=', 'Content-Length': '0'} 2023-10-11T15:14:14.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e41345b6ca424655-006526bbc6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:14:14.795 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.796 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.796 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.797 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.797 DEBUG:boto:Method: GET 2023-10-11T15:14:14.797 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.798 DEBUG:boto:Data: 2023-10-11T15:14:14.798 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.799 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.800 DEBUG:boto:Params: {} 2023-10-11T15:14:14.800 DEBUG:boto:Token: None 2023-10-11T15:14:14.801 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/?versions 2023-10-11T15:14:14.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc= 2023-10-11T15:14:14.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc=', 'Content-Length': '0'} 2023-10-11T15:14:14.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9136a786f1b3b5c-006526bbc6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.807 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:14.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-11T15:14:14.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-11T15:14:14.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-11T15:14:14.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-11T15:14:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-11T15:14:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-11T15:14:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-11T15:14:14.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-11T15:14:14.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-11T15:14:14.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-11T15:14:14.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-11T15:14:14.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-11T15:14:14.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-11T15:14:14.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-11T15:14:14.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-11T15:14:14.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-11T15:14:14.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-11T15:14:14.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-11T15:14:14.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-11T15:14:14.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-11T15:14:14.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-11T15:14:14.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-11T15:14:14.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-11T15:14:14.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-11T15:14:14.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-11T15:14:14.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-11T15:14:14.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-11T15:14:14.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-11T15:14:14.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-11T15:14:14.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-11T15:14:14.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-11T15:14:14.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-11T15:14:14.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-11T15:14:14.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-11T15:14:14.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-11T15:14:14.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-11T15:14:14.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-11T15:14:14.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-11T15:14:14.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-11T15:14:14.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-11T15:14:14.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-11T15:14:14.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-11T15:14:14.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-11T15:14:14.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-11T15:14:14.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-11T15:14:14.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-10-11T15:14:14.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-11T15:14:14.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-11T15:14:14.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-11T15:14:14.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-11T15:14:14.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-11T15:14:14.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-11T15:14:14.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-11T15:14:14.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-11T15:14:14.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-11T15:14:14.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-11T15:14:14.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-11T15:14:14.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-11T15:14:14.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-11T15:14:14.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-11T15:14:14.832 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:14:14.833 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.833 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.833 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.834 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.834 DEBUG:boto:Method: GET 2023-10-11T15:14:14.835 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.835 DEBUG:boto:Data: 2023-10-11T15:14:14.835 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.836 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.837 DEBUG:boto:Params: {} 2023-10-11T15:14:14.837 DEBUG:boto:Token: None 2023-10-11T15:14:14.837 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/?versions 2023-10-11T15:14:14.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc= 2023-10-11T15:14:14.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc=', 'Content-Length': '0'} 2023-10-11T15:14:14.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4f0e3f3724b3bdb-006526bbc6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.845 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:14.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-10-11T15:14:14.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-10-11T15:14:14.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-10-11T15:14:14.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-10-11T15:14:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-10-11T15:14:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-10-11T15:14:14.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-10-11T15:14:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-10-11T15:14:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-10-11T15:14:14.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-10-11T15:14:14.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-10-11T15:14:14.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-10-11T15:14:14.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-10-11T15:14:14.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-10-11T15:14:14.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-10-11T15:14:14.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-10-11T15:14:14.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-10-11T15:14:14.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-10-11T15:14:14.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-10-11T15:14:14.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-10-11T15:14:14.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-10-11T15:14:14.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-10-11T15:14:14.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-10-11T15:14:14.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-10-11T15:14:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-10-11T15:14:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-10-11T15:14:14.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-10-11T15:14:14.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-10-11T15:14:14.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-10-11T15:14:14.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-10-11T15:14:14.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-10-11T15:14:14.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-10-11T15:14:14.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-10-11T15:14:14.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-10-11T15:14:14.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-10-11T15:14:14.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-10-11T15:14:14.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-10-11T15:14:14.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-10-11T15:14:14.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-10-11T15:14:14.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-10-11T15:14:14.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-10-11T15:14:14.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-10-11T15:14:14.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-10-11T15:14:14.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-10-11T15:14:14.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-10-11T15:14:14.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-10-11T15:14:14.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-10-11T15:14:14.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-10-11T15:14:14.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-10-11T15:14:14.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-10-11T15:14:14.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-10-11T15:14:14.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-10-11T15:14:14.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-10-11T15:14:14.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-10-11T15:14:14.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-10-11T15:14:14.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-10-11T15:14:14.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-10-11T15:14:14.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-10-11T15:14:14.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-10-11T15:14:14.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-10-11T15:14:14.870 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.871 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.871 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.872 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.872 DEBUG:boto:Method: GET 2023-10-11T15:14:14.872 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.873 DEBUG:boto:Data: 2023-10-11T15:14:14.873 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.874 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.874 DEBUG:boto:Params: {} 2023-10-11T15:14:14.875 DEBUG:boto:Token: None 2023-10-11T15:14:14.875 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/?versions 2023-10-11T15:14:14.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc= 2023-10-11T15:14:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc=', 'Content-Length': '0'} 2023-10-11T15:14:14.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1880c4b6fa0f39d-006526bbc6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.880 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:14.883 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.884 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.884 DEBUG:boto:path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.885 DEBUG:boto:auth_path=/dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.885 DEBUG:boto:Method: GET 2023-10-11T15:14:14.885 DEBUG:boto:Path: /dghqvd-58/?versions&max-keys=999 2023-10-11T15:14:14.886 DEBUG:boto:Data: 2023-10-11T15:14:14.886 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.887 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.888 DEBUG:boto:Params: {} 2023-10-11T15:14:14.888 DEBUG:boto:Token: None 2023-10-11T15:14:14.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/?versions 2023-10-11T15:14:14.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc= 2023-10-11T15:14:14.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/LALqErVrK4OqNQxhHscTp4g7Oc=', 'Content-Length': '0'} 2023-10-11T15:14:14.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f17b1027c6dc35a-006526bbc6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.894 DEBUG:boto:b'dghqvd-58999falseobj00000000000000000000nulltrue2023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:14.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:14:14.899 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.900 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.900 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.900 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.901 DEBUG:boto:Method: GET 2023-10-11T15:14:14.901 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.902 DEBUG:boto:Data: 2023-10-11T15:14:14.902 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.903 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.903 DEBUG:boto:Params: {} 2023-10-11T15:14:14.904 DEBUG:boto:Token: None 2023-10-11T15:14:14.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY= 2023-10-11T15:14:14.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY=', 'Content-Length': '0'} 2023-10-11T15:14:14.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f265cedc2a61f67-006526bbc6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.909 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.909 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.910 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.910 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.911 DEBUG:boto:Method: GET 2023-10-11T15:14:14.911 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.911 DEBUG:boto:Data: 2023-10-11T15:14:14.912 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.913 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.913 DEBUG:boto:Params: {} 2023-10-11T15:14:14.914 DEBUG:boto:Token: None 2023-10-11T15:14:14.914 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY= 2023-10-11T15:14:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY=', 'Content-Length': '0'} 2023-10-11T15:14:14.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30e1a45d987dd07-006526bbc6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.918 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.918 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.919 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.919 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.919 DEBUG:boto:Method: HEAD 2023-10-11T15:14:14.920 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.920 DEBUG:boto:Data: 2023-10-11T15:14:14.921 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.922 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.922 DEBUG:boto:Params: {} 2023-10-11T15:14:14.922 DEBUG:boto:Token: None 2023-10-11T15:14:14.923 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7DWeuKP9Oz34DnkWzUKaXA5OUUQ= 2023-10-11T15:14:14.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7DWeuKP9Oz34DnkWzUKaXA5OUUQ=', 'Content-Length': '0'} 2023-10-11T15:14:14.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c8a9bb5cdec746-006526bbc6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.926 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.927 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.927 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.927 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.928 DEBUG:boto:Method: HEAD 2023-10-11T15:14:14.928 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.929 DEBUG:boto:Data: 2023-10-11T15:14:14.929 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.930 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.930 DEBUG:boto:Params: {} 2023-10-11T15:14:14.931 DEBUG:boto:Token: None 2023-10-11T15:14:14.931 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7DWeuKP9Oz34DnkWzUKaXA5OUUQ= 2023-10-11T15:14:14.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7DWeuKP9Oz34DnkWzUKaXA5OUUQ=', 'Content-Length': '0'} 2023-10-11T15:14:14.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf07d8743c0b0050-006526bbc6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:14:14.935 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.935 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.936 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.936 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.937 DEBUG:boto:Method: GET 2023-10-11T15:14:14.937 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.938 DEBUG:boto:Data: 2023-10-11T15:14:14.938 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:14.939 DEBUG:boto:Port: 8001 2023-10-11T15:14:14.939 DEBUG:boto:Params: {} 2023-10-11T15:14:14.940 DEBUG:boto:Token: None 2023-10-11T15:14:14.940 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY= 2023-10-11T15:14:14.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY=', 'Content-Length': '0'} 2023-10-11T15:14:14.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079096c80cf4124af-006526bbc6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.986 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.987 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:14.987 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.988 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.988 DEBUG:boto:Method: GET 2023-10-11T15:14:14.989 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.989 DEBUG:boto:Data: 2023-10-11T15:14:14.990 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:14.991 DEBUG:boto:Port: 8000 2023-10-11T15:14:14.991 DEBUG:boto:Params: {} 2023-10-11T15:14:14.991 DEBUG:boto:Token: None 2023-10-11T15:14:14.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:14 GMT /dghqvd-58/obj00000000000000000000?versionId=null 2023-10-11T15:14:14.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY= 2023-10-11T15:14:14.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gzn+ic393lQyS9DoPwpu1pzqrAY=', 'Content-Length': '0'} 2023-10-11T15:14:14.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648f38325b2983d7-006526bbc6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:14.996 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:14.996 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:14.997 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:14.997 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:14.997 DEBUG:boto:Method: GET 2023-10-11T15:14:14.998 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:14.998 DEBUG:boto:Data: 2023-10-11T15:14:14.999 DEBUG:boto:Headers: {} 2023-10-11T15:14:14.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.000 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.000 DEBUG:boto:Params: {} 2023-10-11T15:14:15.000 DEBUG:boto:Token: None 2023-10-11T15:14:15.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc15f9bf9d2d536-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.005 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.006 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.007 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.007 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.008 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.008 DEBUG:boto:Method: GET 2023-10-11T15:14:15.008 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.009 DEBUG:boto:Data: 2023-10-11T15:14:15.009 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.010 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.010 DEBUG:boto:Params: {} 2023-10-11T15:14:15.011 DEBUG:boto:Token: None 2023-10-11T15:14:15.011 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002250aa99329e41b8-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.017 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.018 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.018 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.019 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.019 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.020 DEBUG:boto:Data: 2023-10-11T15:14:15.020 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.021 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.021 DEBUG:boto:Params: {} 2023-10-11T15:14:15.022 DEBUG:boto:Token: None 2023-10-11T15:14:15.022 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eo3oYOjh3hUOsf6Y79LXtJh+m08= 2023-10-11T15:14:15.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eo3oYOjh3hUOsf6Y79LXtJh+m08=', 'Content-Length': '0'} 2023-10-11T15:14:15.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039b488ba10277240-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.025 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.026 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.026 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.027 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.027 DEBUG:boto:Data: 2023-10-11T15:14:15.028 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.028 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.029 DEBUG:boto:Params: {} 2023-10-11T15:14:15.029 DEBUG:boto:Token: None 2023-10-11T15:14:15.030 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eo3oYOjh3hUOsf6Y79LXtJh+m08= 2023-10-11T15:14:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eo3oYOjh3hUOsf6Y79LXtJh+m08=', 'Content-Length': '0'} 2023-10-11T15:14:15.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009befe92657aa841f-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-10-11T15:14:15.034 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.034 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.035 DEBUG:boto:Method: GET 2023-10-11T15:14:15.035 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.035 DEBUG:boto:Data: 2023-10-11T15:14:15.036 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.037 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.037 DEBUG:boto:Params: {} 2023-10-11T15:14:15.038 DEBUG:boto:Token: None 2023-10-11T15:14:15.038 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z4rTD2SuqHOMaYmscPTMhRtvai8= 2023-10-11T15:14:15.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z4rTD2SuqHOMaYmscPTMhRtvai8=', 'Content-Length': '0'} 2023-10-11T15:14:15.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8e72fa6406d14c-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.042 DEBUG:boto:path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.042 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.043 DEBUG:boto:Method: GET 2023-10-11T15:14:15.043 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.044 DEBUG:boto:Data: 2023-10-11T15:14:15.044 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.045 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.046 DEBUG:boto:Params: {} 2023-10-11T15:14:15.046 DEBUG:boto:Token: None 2023-10-11T15:14:15.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000000 2023-10-11T15:14:15.047 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z4rTD2SuqHOMaYmscPTMhRtvai8= 2023-10-11T15:14:15.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z4rTD2SuqHOMaYmscPTMhRtvai8=', 'Content-Length': '0'} 2023-10-11T15:14:15.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a21c3ebd505c944-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.094 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.095 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.096 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.096 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.096 DEBUG:boto:Method: GET 2023-10-11T15:14:15.097 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.097 DEBUG:boto:Data: 2023-10-11T15:14:15.098 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.099 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.099 DEBUG:boto:Params: {} 2023-10-11T15:14:15.100 DEBUG:boto:Token: None 2023-10-11T15:14:15.100 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bd2fe9cc35020d-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.105 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.105 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.106 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.106 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.107 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.107 DEBUG:boto:Method: GET 2023-10-11T15:14:15.108 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000000 2023-10-11T15:14:15.108 DEBUG:boto:Data: 2023-10-11T15:14:15.108 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.109 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.110 DEBUG:boto:Params: {} 2023-10-11T15:14:15.110 DEBUG:boto:Token: None 2023-10-11T15:14:15.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad6595d19b0a421b-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.116 DEBUG:boto:b'dghqvd-58obj000000000000000000001000falseobj000000000000000000002023-10-11T15:14:04.867Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:14:15.121 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.121 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.121 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.122 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.122 DEBUG:boto:Method: GET 2023-10-11T15:14:15.123 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.123 DEBUG:boto:Data: 2023-10-11T15:14:15.124 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.125 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.125 DEBUG:boto:Params: {} 2023-10-11T15:14:15.126 DEBUG:boto:Token: None 2023-10-11T15:14:15.126 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.127 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw= 2023-10-11T15:14:15.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw=', 'Content-Length': '0'} 2023-10-11T15:14:15.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069709aa419b053cd-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.131 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.131 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.132 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.132 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.133 DEBUG:boto:Method: GET 2023-10-11T15:14:15.133 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.133 DEBUG:boto:Data: 2023-10-11T15:14:15.134 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.135 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.135 DEBUG:boto:Params: {} 2023-10-11T15:14:15.136 DEBUG:boto:Token: None 2023-10-11T15:14:15.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw= 2023-10-11T15:14:15.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw=', 'Content-Length': '0'} 2023-10-11T15:14:15.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009530ab0f079c83e8-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.140 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.140 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.141 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.141 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.142 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.142 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.143 DEBUG:boto:Data: 2023-10-11T15:14:15.143 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.144 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.144 DEBUG:boto:Params: {} 2023-10-11T15:14:15.145 DEBUG:boto:Token: None 2023-10-11T15:14:15.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2npTNMEOgn//XAYa6gyavK45FcM= 2023-10-11T15:14:15.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2npTNMEOgn//XAYa6gyavK45FcM=', 'Content-Length': '0'} 2023-10-11T15:14:15.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58c700af7f15f4d-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.149 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.149 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.150 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.150 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.151 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.151 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.151 DEBUG:boto:Data: 2023-10-11T15:14:15.152 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.153 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.153 DEBUG:boto:Params: {} 2023-10-11T15:14:15.153 DEBUG:boto:Token: None 2023-10-11T15:14:15.154 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2npTNMEOgn//XAYa6gyavK45FcM= 2023-10-11T15:14:15.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2npTNMEOgn//XAYa6gyavK45FcM=', 'Content-Length': '0'} 2023-10-11T15:14:15.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006784abb6d03d4fc-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:14:15.158 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.158 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.159 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.159 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.160 DEBUG:boto:Method: GET 2023-10-11T15:14:15.160 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.161 DEBUG:boto:Data: 2023-10-11T15:14:15.161 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.161 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.162 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.162 DEBUG:boto:Params: {} 2023-10-11T15:14:15.163 DEBUG:boto:Token: None 2023-10-11T15:14:15.163 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.164 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw= 2023-10-11T15:14:15.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw=', 'Content-Length': '0'} 2023-10-11T15:14:15.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b7bb7a620df2f0-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.167 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.167 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.168 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.168 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.169 DEBUG:boto:Method: GET 2023-10-11T15:14:15.169 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.169 DEBUG:boto:Data: 2023-10-11T15:14:15.170 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.171 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.171 DEBUG:boto:Params: {} 2023-10-11T15:14:15.172 DEBUG:boto:Token: None 2023-10-11T15:14:15.172 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001?versionId=null 2023-10-11T15:14:15.173 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw= 2023-10-11T15:14:15.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oeaHdfbSGhE9jy+p3fxDZ0gVFaw=', 'Content-Length': '0'} 2023-10-11T15:14:15.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60ab146b7d1ad81-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.218 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.219 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.219 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.220 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.220 DEBUG:boto:Method: GET 2023-10-11T15:14:15.221 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.221 DEBUG:boto:Data: 2023-10-11T15:14:15.222 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.223 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.223 DEBUG:boto:Params: {} 2023-10-11T15:14:15.224 DEBUG:boto:Token: None 2023-10-11T15:14:15.224 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.225 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003df49def1587f894-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.228 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.229 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.229 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.230 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.230 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.231 DEBUG:boto:Method: GET 2023-10-11T15:14:15.231 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.231 DEBUG:boto:Data: 2023-10-11T15:14:15.232 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.233 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.233 DEBUG:boto:Params: {} 2023-10-11T15:14:15.233 DEBUG:boto:Token: None 2023-10-11T15:14:15.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004016dc42b952fb0e-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.240 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.241 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.241 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.241 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.242 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.242 DEBUG:boto:Data: 2023-10-11T15:14:15.243 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.243 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.244 DEBUG:boto:Params: {} 2023-10-11T15:14:15.244 DEBUG:boto:Token: None 2023-10-11T15:14:15.245 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQgzw8ezloVKO2iiSO6F04MtyqI= 2023-10-11T15:14:15.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQgzw8ezloVKO2iiSO6F04MtyqI=', 'Content-Length': '0'} 2023-10-11T15:14:15.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b4f08bdf652f3f-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.248 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.249 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.249 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.249 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.250 DEBUG:boto:Data: 2023-10-11T15:14:15.250 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.253 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.254 DEBUG:boto:Params: {} 2023-10-11T15:14:15.255 DEBUG:boto:Token: None 2023-10-11T15:14:15.255 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQgzw8ezloVKO2iiSO6F04MtyqI= 2023-10-11T15:14:15.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQgzw8ezloVKO2iiSO6F04MtyqI=', 'Content-Length': '0'} 2023-10-11T15:14:15.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000027d6902e4fce66c-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-10-11T15:14:15.262 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.262 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.263 DEBUG:boto:Method: GET 2023-10-11T15:14:15.263 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.264 DEBUG:boto:Data: 2023-10-11T15:14:15.264 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.264 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.265 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.265 DEBUG:boto:Params: {} 2023-10-11T15:14:15.266 DEBUG:boto:Token: None 2023-10-11T15:14:15.266 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LvwCRjcjVmrRVktoK111qLs4u2c= 2023-10-11T15:14:15.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LvwCRjcjVmrRVktoK111qLs4u2c=', 'Content-Length': '0'} 2023-10-11T15:14:15.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3f1a2a74a2b500-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.310 DEBUG:boto:path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.311 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.311 DEBUG:boto:Method: GET 2023-10-11T15:14:15.312 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.312 DEBUG:boto:Data: 2023-10-11T15:14:15.313 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.314 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.314 DEBUG:boto:Params: {} 2023-10-11T15:14:15.315 DEBUG:boto:Token: None 2023-10-11T15:14:15.315 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000001 2023-10-11T15:14:15.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LvwCRjcjVmrRVktoK111qLs4u2c= 2023-10-11T15:14:15.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LvwCRjcjVmrRVktoK111qLs4u2c=', 'Content-Length': '0'} 2023-10-11T15:14:15.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8b54af2aa0217d-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.320 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.320 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.321 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.321 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.322 DEBUG:boto:Method: GET 2023-10-11T15:14:15.322 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.322 DEBUG:boto:Data: 2023-10-11T15:14:15.323 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.324 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.324 DEBUG:boto:Params: {} 2023-10-11T15:14:15.325 DEBUG:boto:Token: None 2023-10-11T15:14:15.325 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd58fa7c63e9809c-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.329 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.330 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.331 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.331 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.332 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.332 DEBUG:boto:Method: GET 2023-10-11T15:14:15.332 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000001 2023-10-11T15:14:15.333 DEBUG:boto:Data: 2023-10-11T15:14:15.333 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.334 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.335 DEBUG:boto:Params: {} 2023-10-11T15:14:15.335 DEBUG:boto:Token: None 2023-10-11T15:14:15.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee52a0412bf88ad-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.382 DEBUG:boto:b'dghqvd-58obj000000000000000000011000falseobj000000000000000000012023-10-11T15:14:05.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:14:15.384 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.384 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.385 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.385 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.386 DEBUG:boto:Method: GET 2023-10-11T15:14:15.386 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.387 DEBUG:boto:Data: 2023-10-11T15:14:15.387 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.388 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.389 DEBUG:boto:Params: {} 2023-10-11T15:14:15.389 DEBUG:boto:Token: None 2023-10-11T15:14:15.390 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc= 2023-10-11T15:14:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc=', 'Content-Length': '0'} 2023-10-11T15:14:15.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006babffa4238a8817-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.394 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.395 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.395 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.396 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.396 DEBUG:boto:Method: GET 2023-10-11T15:14:15.397 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.397 DEBUG:boto:Data: 2023-10-11T15:14:15.398 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.399 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.399 DEBUG:boto:Params: {} 2023-10-11T15:14:15.400 DEBUG:boto:Token: None 2023-10-11T15:14:15.401 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc= 2023-10-11T15:14:15.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc=', 'Content-Length': '0'} 2023-10-11T15:14:15.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5596ee32fc1751-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.404 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.405 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.405 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.406 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.406 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.407 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.407 DEBUG:boto:Data: 2023-10-11T15:14:15.408 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.409 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.409 DEBUG:boto:Params: {} 2023-10-11T15:14:15.410 DEBUG:boto:Token: None 2023-10-11T15:14:15.410 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sJMjCNZn7K5I0Q65axj9SzA1Q2g= 2023-10-11T15:14:15.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sJMjCNZn7K5I0Q65axj9SzA1Q2g=', 'Content-Length': '0'} 2023-10-11T15:14:15.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772a59829e0d69b2-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.414 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.415 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.415 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.415 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.416 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.416 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.417 DEBUG:boto:Data: 2023-10-11T15:14:15.417 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.418 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.419 DEBUG:boto:Params: {} 2023-10-11T15:14:15.419 DEBUG:boto:Token: None 2023-10-11T15:14:15.420 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sJMjCNZn7K5I0Q65axj9SzA1Q2g= 2023-10-11T15:14:15.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sJMjCNZn7K5I0Q65axj9SzA1Q2g=', 'Content-Length': '0'} 2023-10-11T15:14:15.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6a0e585c01caa7-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:14:15.424 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.425 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.425 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.425 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.426 DEBUG:boto:Method: GET 2023-10-11T15:14:15.426 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.427 DEBUG:boto:Data: 2023-10-11T15:14:15.427 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.428 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.428 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.428 DEBUG:boto:Params: {} 2023-10-11T15:14:15.429 DEBUG:boto:Token: None 2023-10-11T15:14:15.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc= 2023-10-11T15:14:15.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc=', 'Content-Length': '0'} 2023-10-11T15:14:15.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512aff04f6eea04c-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.433 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.434 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.434 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.435 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.435 DEBUG:boto:Method: GET 2023-10-11T15:14:15.436 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.436 DEBUG:boto:Data: 2023-10-11T15:14:15.436 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.437 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.438 DEBUG:boto:Params: {} 2023-10-11T15:14:15.438 DEBUG:boto:Token: None 2023-10-11T15:14:15.439 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002?versionId=null 2023-10-11T15:14:15.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc= 2023-10-11T15:14:15.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HripGFMDpc346Hu5xTwbZQ7glEc=', 'Content-Length': '0'} 2023-10-11T15:14:15.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1bfb85a5cc40b1d-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.442 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.443 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.443 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.444 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.444 DEBUG:boto:Method: GET 2023-10-11T15:14:15.445 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.445 DEBUG:boto:Data: 2023-10-11T15:14:15.445 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.446 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.447 DEBUG:boto:Params: {} 2023-10-11T15:14:15.447 DEBUG:boto:Token: None 2023-10-11T15:14:15.448 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8f4f7ba11a3049-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.453 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.453 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.454 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.454 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.455 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.458 DEBUG:boto:Method: GET 2023-10-11T15:14:15.459 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.459 DEBUG:boto:Data: 2023-10-11T15:14:15.460 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.461 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.461 DEBUG:boto:Params: {} 2023-10-11T15:14:15.462 DEBUG:boto:Token: None 2023-10-11T15:14:15.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85eced74f42db19-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.468 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.469 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.469 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.470 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.470 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.470 DEBUG:boto:Data: 2023-10-11T15:14:15.471 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.472 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.472 DEBUG:boto:Params: {} 2023-10-11T15:14:15.473 DEBUG:boto:Token: None 2023-10-11T15:14:15.473 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dl5HLdYom+KBtfPdsFDfIMZZAmU= 2023-10-11T15:14:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dl5HLdYom+KBtfPdsFDfIMZZAmU=', 'Content-Length': '0'} 2023-10-11T15:14:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fd177edc2f750d-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.477 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.477 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.478 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.478 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.479 DEBUG:boto:Data: 2023-10-11T15:14:15.479 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.480 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.480 DEBUG:boto:Params: {} 2023-10-11T15:14:15.481 DEBUG:boto:Token: None 2023-10-11T15:14:15.481 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.482 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dl5HLdYom+KBtfPdsFDfIMZZAmU= 2023-10-11T15:14:15.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dl5HLdYom+KBtfPdsFDfIMZZAmU=', 'Content-Length': '0'} 2023-10-11T15:14:15.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8ef9d2d72ea0aa-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-10-11T15:14:15.485 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.485 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.486 DEBUG:boto:Method: GET 2023-10-11T15:14:15.486 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.487 DEBUG:boto:Data: 2023-10-11T15:14:15.487 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.488 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.489 DEBUG:boto:Params: {} 2023-10-11T15:14:15.489 DEBUG:boto:Token: None 2023-10-11T15:14:15.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lzowWn3IZtn7e23oIVuIBm3bMjs= 2023-10-11T15:14:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lzowWn3IZtn7e23oIVuIBm3bMjs=', 'Content-Length': '0'} 2023-10-11T15:14:15.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce19394efba161ba-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.494 DEBUG:boto:path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.494 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.495 DEBUG:boto:Method: GET 2023-10-11T15:14:15.495 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.495 DEBUG:boto:Data: 2023-10-11T15:14:15.496 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.497 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.497 DEBUG:boto:Params: {} 2023-10-11T15:14:15.498 DEBUG:boto:Token: None 2023-10-11T15:14:15.498 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000002 2023-10-11T15:14:15.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lzowWn3IZtn7e23oIVuIBm3bMjs= 2023-10-11T15:14:15.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lzowWn3IZtn7e23oIVuIBm3bMjs=', 'Content-Length': '0'} 2023-10-11T15:14:15.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e89458a324843a-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.542 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.543 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.543 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.544 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.544 DEBUG:boto:Method: GET 2023-10-11T15:14:15.545 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.545 DEBUG:boto:Data: 2023-10-11T15:14:15.546 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.546 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.547 DEBUG:boto:Params: {} 2023-10-11T15:14:15.547 DEBUG:boto:Token: None 2023-10-11T15:14:15.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069fa50943998bbba-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.552 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.553 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.554 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.554 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.555 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.555 DEBUG:boto:Method: GET 2023-10-11T15:14:15.556 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000002 2023-10-11T15:14:15.556 DEBUG:boto:Data: 2023-10-11T15:14:15.557 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.558 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.558 DEBUG:boto:Params: {} 2023-10-11T15:14:15.559 DEBUG:boto:Token: None 2023-10-11T15:14:15.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae98043042188626-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.566 DEBUG:boto:b'dghqvd-58obj000000000000000000021000falseobj000000000000000000022023-10-11T15:14:01.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:14:15.567 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.568 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.568 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.568 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.569 DEBUG:boto:Method: GET 2023-10-11T15:14:15.570 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.570 DEBUG:boto:Data: 2023-10-11T15:14:15.571 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.571 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.572 DEBUG:boto:Params: {} 2023-10-11T15:14:15.572 DEBUG:boto:Token: None 2023-10-11T15:14:15.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk= 2023-10-11T15:14:15.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk=', 'Content-Length': '0'} 2023-10-11T15:14:15.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fad12ca98a76cf3-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.618 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.619 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.619 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.620 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.620 DEBUG:boto:Method: GET 2023-10-11T15:14:15.621 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.621 DEBUG:boto:Data: 2023-10-11T15:14:15.622 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.622 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.623 DEBUG:boto:Params: {} 2023-10-11T15:14:15.623 DEBUG:boto:Token: None 2023-10-11T15:14:15.624 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.625 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk= 2023-10-11T15:14:15.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk=', 'Content-Length': '0'} 2023-10-11T15:14:15.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dfec642ab1a0932-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.628 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.629 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.629 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.630 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.630 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.630 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.631 DEBUG:boto:Data: 2023-10-11T15:14:15.631 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.632 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.632 DEBUG:boto:Params: {} 2023-10-11T15:14:15.633 DEBUG:boto:Token: None 2023-10-11T15:14:15.633 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:poPFgv5OKeLuVe5GJpi70OvTWEM= 2023-10-11T15:14:15.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:poPFgv5OKeLuVe5GJpi70OvTWEM=', 'Content-Length': '0'} 2023-10-11T15:14:15.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b6262b9031b23b-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.637 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.637 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.638 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.638 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.638 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.639 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.639 DEBUG:boto:Data: 2023-10-11T15:14:15.640 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.640 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.641 DEBUG:boto:Params: {} 2023-10-11T15:14:15.641 DEBUG:boto:Token: None 2023-10-11T15:14:15.642 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.643 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:poPFgv5OKeLuVe5GJpi70OvTWEM= 2023-10-11T15:14:15.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:poPFgv5OKeLuVe5GJpi70OvTWEM=', 'Content-Length': '0'} 2023-10-11T15:14:15.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cabe25e1e85d7d1-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:14:15.646 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.647 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.647 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.647 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.648 DEBUG:boto:Method: GET 2023-10-11T15:14:15.648 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.649 DEBUG:boto:Data: 2023-10-11T15:14:15.649 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.650 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.650 DEBUG:boto:Params: {} 2023-10-11T15:14:15.651 DEBUG:boto:Token: None 2023-10-11T15:14:15.651 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk= 2023-10-11T15:14:15.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk=', 'Content-Length': '0'} 2023-10-11T15:14:15.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cab06b74de03699-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.655 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.655 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.656 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.656 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.657 DEBUG:boto:Method: GET 2023-10-11T15:14:15.657 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.658 DEBUG:boto:Data: 2023-10-11T15:14:15.658 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.659 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.659 DEBUG:boto:Params: {} 2023-10-11T15:14:15.660 DEBUG:boto:Token: None 2023-10-11T15:14:15.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003?versionId=null 2023-10-11T15:14:15.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk= 2023-10-11T15:14:15.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:utXbpUGMYUf8MEa0TTQZMMFzImk=', 'Content-Length': '0'} 2023-10-11T15:14:15.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78964bb5c46e0be-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.668 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.668 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.668 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.669 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.669 DEBUG:boto:Method: GET 2023-10-11T15:14:15.670 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.670 DEBUG:boto:Data: 2023-10-11T15:14:15.671 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.672 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.672 DEBUG:boto:Params: {} 2023-10-11T15:14:15.673 DEBUG:boto:Token: None 2023-10-11T15:14:15.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dca5531d615591d7-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.677 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.678 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.679 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.679 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.679 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.680 DEBUG:boto:Method: GET 2023-10-11T15:14:15.680 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.681 DEBUG:boto:Data: 2023-10-11T15:14:15.681 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.682 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.682 DEBUG:boto:Params: {} 2023-10-11T15:14:15.683 DEBUG:boto:Token: None 2023-10-11T15:14:15.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed6bcf8acc82763-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.689 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.690 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.690 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.691 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.691 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.691 DEBUG:boto:Data: 2023-10-11T15:14:15.692 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.692 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.693 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.693 DEBUG:boto:Params: {} 2023-10-11T15:14:15.694 DEBUG:boto:Token: None 2023-10-11T15:14:15.694 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.695 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gpjpVe9j4/JW1nXE11BisFumsN0= 2023-10-11T15:14:15.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gpjpVe9j4/JW1nXE11BisFumsN0=', 'Content-Length': '0'} 2023-10-11T15:14:15.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598b25e4969e43fe-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.698 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.698 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.699 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.699 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.700 DEBUG:boto:Data: 2023-10-11T15:14:15.700 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.700 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.701 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.701 DEBUG:boto:Params: {} 2023-10-11T15:14:15.701 DEBUG:boto:Token: None 2023-10-11T15:14:15.702 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.702 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gpjpVe9j4/JW1nXE11BisFumsN0= 2023-10-11T15:14:15.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gpjpVe9j4/JW1nXE11BisFumsN0=', 'Content-Length': '0'} 2023-10-11T15:14:15.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069bdccaab09a1d7d-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-10-11T15:14:15.706 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.706 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.707 DEBUG:boto:Method: GET 2023-10-11T15:14:15.708 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.708 DEBUG:boto:Data: 2023-10-11T15:14:15.708 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.709 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.710 DEBUG:boto:Params: {} 2023-10-11T15:14:15.710 DEBUG:boto:Token: None 2023-10-11T15:14:15.711 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YcMLPvSkJmwKmneEVYQfbSI5Qo4= 2023-10-11T15:14:15.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YcMLPvSkJmwKmneEVYQfbSI5Qo4=', 'Content-Length': '0'} 2023-10-11T15:14:15.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2ae900cda2fc3c3-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.715 DEBUG:boto:path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.715 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.715 DEBUG:boto:Method: GET 2023-10-11T15:14:15.716 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.716 DEBUG:boto:Data: 2023-10-11T15:14:15.717 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.718 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.718 DEBUG:boto:Params: {} 2023-10-11T15:14:15.718 DEBUG:boto:Token: None 2023-10-11T15:14:15.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000003 2023-10-11T15:14:15.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YcMLPvSkJmwKmneEVYQfbSI5Qo4= 2023-10-11T15:14:15.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YcMLPvSkJmwKmneEVYQfbSI5Qo4=', 'Content-Length': '0'} 2023-10-11T15:14:15.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009d8c0314300ece9-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.762 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.763 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.763 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.764 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.764 DEBUG:boto:Method: GET 2023-10-11T15:14:15.765 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.765 DEBUG:boto:Data: 2023-10-11T15:14:15.766 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.766 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.767 DEBUG:boto:Params: {} 2023-10-11T15:14:15.767 DEBUG:boto:Token: None 2023-10-11T15:14:15.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043ebad953bf53ba-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.773 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.773 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.774 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.774 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.775 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.775 DEBUG:boto:Method: GET 2023-10-11T15:14:15.776 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000003 2023-10-11T15:14:15.776 DEBUG:boto:Data: 2023-10-11T15:14:15.777 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.778 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.778 DEBUG:boto:Params: {} 2023-10-11T15:14:15.779 DEBUG:boto:Token: None 2023-10-11T15:14:15.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe320df517b37a5-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.785 DEBUG:boto:b'dghqvd-58obj000000000000000000031000falseobj000000000000000000032023-10-11T15:14:05.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:14:15.787 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.787 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.788 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.788 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.788 DEBUG:boto:Method: GET 2023-10-11T15:14:15.789 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.789 DEBUG:boto:Data: 2023-10-11T15:14:15.790 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.790 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.791 DEBUG:boto:Params: {} 2023-10-11T15:14:15.791 DEBUG:boto:Token: None 2023-10-11T15:14:15.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE= 2023-10-11T15:14:15.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE=', 'Content-Length': '0'} 2023-10-11T15:14:15.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcaa35c7e2bc56a5-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.838 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.839 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.839 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.840 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.840 DEBUG:boto:Method: GET 2023-10-11T15:14:15.841 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.841 DEBUG:boto:Data: 2023-10-11T15:14:15.841 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.842 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.843 DEBUG:boto:Params: {} 2023-10-11T15:14:15.844 DEBUG:boto:Token: None 2023-10-11T15:14:15.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE= 2023-10-11T15:14:15.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE=', 'Content-Length': '0'} 2023-10-11T15:14:15.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000769624211be7654-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.848 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.849 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.849 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.850 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.850 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.850 DEBUG:boto:Data: 2023-10-11T15:14:15.851 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.852 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.852 DEBUG:boto:Params: {} 2023-10-11T15:14:15.852 DEBUG:boto:Token: None 2023-10-11T15:14:15.853 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OVsOSMtftMsMBqamM8qlPj9LwJw= 2023-10-11T15:14:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OVsOSMtftMsMBqamM8qlPj9LwJw=', 'Content-Length': '0'} 2023-10-11T15:14:15.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009223f82aa665dd3a-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.857 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.857 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.857 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.858 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.858 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.859 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.859 DEBUG:boto:Data: 2023-10-11T15:14:15.859 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.860 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.861 DEBUG:boto:Params: {} 2023-10-11T15:14:15.861 DEBUG:boto:Token: None 2023-10-11T15:14:15.861 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OVsOSMtftMsMBqamM8qlPj9LwJw= 2023-10-11T15:14:15.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OVsOSMtftMsMBqamM8qlPj9LwJw=', 'Content-Length': '0'} 2023-10-11T15:14:15.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d85c42f7c0f6429-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:14:15.866 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.866 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.866 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.867 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.869 DEBUG:boto:Method: GET 2023-10-11T15:14:15.870 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.870 DEBUG:boto:Data: 2023-10-11T15:14:15.871 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.871 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.872 DEBUG:boto:Params: {} 2023-10-11T15:14:15.872 DEBUG:boto:Token: None 2023-10-11T15:14:15.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE= 2023-10-11T15:14:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE=', 'Content-Length': '0'} 2023-10-11T15:14:15.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006678e393fb683622-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.877 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.877 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.878 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.878 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.879 DEBUG:boto:Method: GET 2023-10-11T15:14:15.879 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.879 DEBUG:boto:Data: 2023-10-11T15:14:15.880 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.881 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.881 DEBUG:boto:Params: {} 2023-10-11T15:14:15.882 DEBUG:boto:Token: None 2023-10-11T15:14:15.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004?versionId=null 2023-10-11T15:14:15.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE= 2023-10-11T15:14:15.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zjrQuH2TltTTb4suuqK5DsafcBE=', 'Content-Length': '0'} 2023-10-11T15:14:15.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000185a4ecec06f0f77-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.886 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.886 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.887 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.887 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.887 DEBUG:boto:Method: GET 2023-10-11T15:14:15.888 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.888 DEBUG:boto:Data: 2023-10-11T15:14:15.889 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.890 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.890 DEBUG:boto:Params: {} 2023-10-11T15:14:15.890 DEBUG:boto:Token: None 2023-10-11T15:14:15.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000625fc68e724bbc31-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.895 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.896 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.896 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.897 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.897 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.898 DEBUG:boto:Method: GET 2023-10-11T15:14:15.898 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.899 DEBUG:boto:Data: 2023-10-11T15:14:15.899 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.900 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.900 DEBUG:boto:Params: {} 2023-10-11T15:14:15.901 DEBUG:boto:Token: None 2023-10-11T15:14:15.901 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2611c3ea1ac7e7-006526bbc7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.907 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.908 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.908 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.908 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.909 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.909 DEBUG:boto:Data: 2023-10-11T15:14:15.910 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.910 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.911 DEBUG:boto:Params: {} 2023-10-11T15:14:15.911 DEBUG:boto:Token: None 2023-10-11T15:14:15.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IqbjV4sdDzveNaj9W4X8Zm4nLNE= 2023-10-11T15:14:15.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IqbjV4sdDzveNaj9W4X8Zm4nLNE=', 'Content-Length': '0'} 2023-10-11T15:14:15.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002085bd16d66bd0a8-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.915 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.916 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.916 DEBUG:boto:Method: HEAD 2023-10-11T15:14:15.917 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.917 DEBUG:boto:Data: 2023-10-11T15:14:15.918 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.919 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.919 DEBUG:boto:Params: {} 2023-10-11T15:14:15.920 DEBUG:boto:Token: None 2023-10-11T15:14:15.920 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IqbjV4sdDzveNaj9W4X8Zm4nLNE= 2023-10-11T15:14:15.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IqbjV4sdDzveNaj9W4X8Zm4nLNE=', 'Content-Length': '0'} 2023-10-11T15:14:15.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f3c8c4c834e6423-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-10-11T15:14:15.924 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.924 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.925 DEBUG:boto:Method: GET 2023-10-11T15:14:15.925 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.926 DEBUG:boto:Data: 2023-10-11T15:14:15.926 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.927 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.927 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.928 DEBUG:boto:Params: {} 2023-10-11T15:14:15.928 DEBUG:boto:Token: None 2023-10-11T15:14:15.929 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.929 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7VM4rUtcHecJ1Lg2vr9Ffkjh39Q= 2023-10-11T15:14:15.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7VM4rUtcHecJ1Lg2vr9Ffkjh39Q=', 'Content-Length': '0'} 2023-10-11T15:14:15.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa03660c025daba-006526bbc7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.979 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.980 DEBUG:boto:Method: GET 2023-10-11T15:14:15.980 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.980 DEBUG:boto:Data: 2023-10-11T15:14:15.981 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:15.982 DEBUG:boto:Port: 8000 2023-10-11T15:14:15.982 DEBUG:boto:Params: {} 2023-10-11T15:14:15.983 DEBUG:boto:Token: None 2023-10-11T15:14:15.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/obj00000000000000000004 2023-10-11T15:14:15.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7VM4rUtcHecJ1Lg2vr9Ffkjh39Q= 2023-10-11T15:14:15.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7VM4rUtcHecJ1Lg2vr9Ffkjh39Q=', 'Content-Length': '0'} 2023-10-11T15:14:15.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823629935af08e98-006526bbc7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.987 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.987 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.988 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.988 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.989 DEBUG:boto:Method: GET 2023-10-11T15:14:15.989 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.990 DEBUG:boto:Data: 2023-10-11T15:14:15.990 DEBUG:boto:Headers: {} 2023-10-11T15:14:15.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:15.991 DEBUG:boto:Port: 8001 2023-10-11T15:14:15.991 DEBUG:boto:Params: {} 2023-10-11T15:14:15.992 DEBUG:boto:Token: None 2023-10-11T15:14:15.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:15 GMT /dghqvd-58/ 2023-10-11T15:14:15.993 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo= 2023-10-11T15:14:15.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hpeTbrlnyUvU+BmPvoDLF8VUFTo=', 'Content-Length': '0'} 2023-10-11T15:14:15.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379968b112eb927d-006526bbc7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:15.997 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:15.997 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:15.998 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:15.998 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.999 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:15.999 DEBUG:boto:Method: GET 2023-10-11T15:14:15.999 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000004 2023-10-11T15:14:16.000 DEBUG:boto:Data: 2023-10-11T15:14:16.000 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.001 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.002 DEBUG:boto:Params: {} 2023-10-11T15:14:16.002 DEBUG:boto:Token: None 2023-10-11T15:14:16.003 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.003 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4a6b9ce0984a61-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.008 DEBUG:boto:b'dghqvd-58obj000000000000000000041000falseobj000000000000000000042023-10-11T15:14:05.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:14:16.009 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.010 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.010 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.011 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.011 DEBUG:boto:Method: GET 2023-10-11T15:14:16.012 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.012 DEBUG:boto:Data: 2023-10-11T15:14:16.012 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.013 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.014 DEBUG:boto:Params: {} 2023-10-11T15:14:16.014 DEBUG:boto:Token: None 2023-10-11T15:14:16.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts= 2023-10-11T15:14:16.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts=', 'Content-Length': '0'} 2023-10-11T15:14:16.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006887a7e122205d80-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.019 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.019 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.020 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.020 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.020 DEBUG:boto:Method: GET 2023-10-11T15:14:16.021 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.021 DEBUG:boto:Data: 2023-10-11T15:14:16.022 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.023 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.023 DEBUG:boto:Params: {} 2023-10-11T15:14:16.023 DEBUG:boto:Token: None 2023-10-11T15:14:16.024 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts= 2023-10-11T15:14:16.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts=', 'Content-Length': '0'} 2023-10-11T15:14:16.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e7cfdc8062a08d-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.028 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.028 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.029 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.029 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.030 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.030 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.030 DEBUG:boto:Data: 2023-10-11T15:14:16.031 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.032 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.032 DEBUG:boto:Params: {} 2023-10-11T15:14:16.033 DEBUG:boto:Token: None 2023-10-11T15:14:16.033 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ffth/B7gefsNFh3GNhlFCfiCkSw= 2023-10-11T15:14:16.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ffth/B7gefsNFh3GNhlFCfiCkSw=', 'Content-Length': '0'} 2023-10-11T15:14:16.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071491f59aa2f98d0-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.037 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.037 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.037 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.038 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.038 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.039 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.039 DEBUG:boto:Data: 2023-10-11T15:14:16.039 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.040 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.041 DEBUG:boto:Params: {} 2023-10-11T15:14:16.041 DEBUG:boto:Token: None 2023-10-11T15:14:16.042 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ffth/B7gefsNFh3GNhlFCfiCkSw= 2023-10-11T15:14:16.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ffth/B7gefsNFh3GNhlFCfiCkSw=', 'Content-Length': '0'} 2023-10-11T15:14:16.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b30eccb6553000d-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:14:16.046 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.046 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.047 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.047 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.047 DEBUG:boto:Method: GET 2023-10-11T15:14:16.048 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.048 DEBUG:boto:Data: 2023-10-11T15:14:16.049 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.049 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.050 DEBUG:boto:Params: {} 2023-10-11T15:14:16.050 DEBUG:boto:Token: None 2023-10-11T15:14:16.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts= 2023-10-11T15:14:16.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts=', 'Content-Length': '0'} 2023-10-11T15:14:16.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ff902e0caaf660-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.054 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.055 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.055 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.056 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.056 DEBUG:boto:Method: GET 2023-10-11T15:14:16.057 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.057 DEBUG:boto:Data: 2023-10-11T15:14:16.057 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.058 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.059 DEBUG:boto:Params: {} 2023-10-11T15:14:16.059 DEBUG:boto:Token: None 2023-10-11T15:14:16.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005?versionId=null 2023-10-11T15:14:16.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts= 2023-10-11T15:14:16.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cOT/7RBmy/agGjqTzjkGmgeocts=', 'Content-Length': '0'} 2023-10-11T15:14:16.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad74f22cfbee805-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.106 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.107 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.107 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.108 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.108 DEBUG:boto:Method: GET 2023-10-11T15:14:16.109 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.109 DEBUG:boto:Data: 2023-10-11T15:14:16.109 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.110 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.111 DEBUG:boto:Params: {} 2023-10-11T15:14:16.111 DEBUG:boto:Token: None 2023-10-11T15:14:16.112 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e718cca2fde64f4c-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.116 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.117 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.117 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.118 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.118 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.119 DEBUG:boto:Method: GET 2023-10-11T15:14:16.119 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.119 DEBUG:boto:Data: 2023-10-11T15:14:16.120 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.121 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.121 DEBUG:boto:Params: {} 2023-10-11T15:14:16.122 DEBUG:boto:Token: None 2023-10-11T15:14:16.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008673ea42ae64142d-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.128 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.129 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.129 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.129 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.130 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.130 DEBUG:boto:Data: 2023-10-11T15:14:16.131 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.132 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.132 DEBUG:boto:Params: {} 2023-10-11T15:14:16.132 DEBUG:boto:Token: None 2023-10-11T15:14:16.133 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bxfB+qlKl8IPRsuw8UvUb/KjxgI= 2023-10-11T15:14:16.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bxfB+qlKl8IPRsuw8UvUb/KjxgI=', 'Content-Length': '0'} 2023-10-11T15:14:16.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c8e453e4e9002a-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.136 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.137 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.137 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.138 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.138 DEBUG:boto:Data: 2023-10-11T15:14:16.139 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.139 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.139 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.140 DEBUG:boto:Params: {} 2023-10-11T15:14:16.140 DEBUG:boto:Token: None 2023-10-11T15:14:16.141 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.141 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bxfB+qlKl8IPRsuw8UvUb/KjxgI= 2023-10-11T15:14:16.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bxfB+qlKl8IPRsuw8UvUb/KjxgI=', 'Content-Length': '0'} 2023-10-11T15:14:16.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008454169f47e43a3c-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-10-11T15:14:16.144 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.145 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.145 DEBUG:boto:Method: GET 2023-10-11T15:14:16.146 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.146 DEBUG:boto:Data: 2023-10-11T15:14:16.147 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.148 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.148 DEBUG:boto:Params: {} 2023-10-11T15:14:16.148 DEBUG:boto:Token: None 2023-10-11T15:14:16.149 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFSMbmkrqtSEEfVsVJp++sBhHRA= 2023-10-11T15:14:16.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFSMbmkrqtSEEfVsVJp++sBhHRA=', 'Content-Length': '0'} 2023-10-11T15:14:16.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d749c2f6fa23690-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.194 DEBUG:boto:path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.197 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.197 DEBUG:boto:Method: GET 2023-10-11T15:14:16.198 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.198 DEBUG:boto:Data: 2023-10-11T15:14:16.199 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.199 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.200 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.200 DEBUG:boto:Params: {} 2023-10-11T15:14:16.201 DEBUG:boto:Token: None 2023-10-11T15:14:16.201 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000005 2023-10-11T15:14:16.202 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFSMbmkrqtSEEfVsVJp++sBhHRA= 2023-10-11T15:14:16.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFSMbmkrqtSEEfVsVJp++sBhHRA=', 'Content-Length': '0'} 2023-10-11T15:14:16.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006889b3dfaedb1093-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.205 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.206 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.206 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.207 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.207 DEBUG:boto:Method: GET 2023-10-11T15:14:16.208 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.208 DEBUG:boto:Data: 2023-10-11T15:14:16.208 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.209 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.210 DEBUG:boto:Params: {} 2023-10-11T15:14:16.211 DEBUG:boto:Token: None 2023-10-11T15:14:16.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20a3b6d3360b851-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.215 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.216 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.217 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.217 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.218 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.218 DEBUG:boto:Method: GET 2023-10-11T15:14:16.219 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000005 2023-10-11T15:14:16.219 DEBUG:boto:Data: 2023-10-11T15:14:16.219 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.220 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.221 DEBUG:boto:Params: {} 2023-10-11T15:14:16.221 DEBUG:boto:Token: None 2023-10-11T15:14:16.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aecbb75345b5791f-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.227 DEBUG:boto:b'dghqvd-58obj000000000000000000051000falseobj000000000000000000052023-10-11T15:14:04.782Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-11T15:14:16.229 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.229 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.229 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.230 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.230 DEBUG:boto:Method: GET 2023-10-11T15:14:16.231 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.231 DEBUG:boto:Data: 2023-10-11T15:14:16.232 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.233 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.233 DEBUG:boto:Params: {} 2023-10-11T15:14:16.233 DEBUG:boto:Token: None 2023-10-11T15:14:16.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y= 2023-10-11T15:14:16.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd6a0c3384e9b775-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.238 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.239 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.239 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.240 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.240 DEBUG:boto:Method: GET 2023-10-11T15:14:16.241 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.241 DEBUG:boto:Data: 2023-10-11T15:14:16.241 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.242 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.243 DEBUG:boto:Params: {} 2023-10-11T15:14:16.243 DEBUG:boto:Token: None 2023-10-11T15:14:16.244 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y= 2023-10-11T15:14:16.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6c84d0555669bb-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.290 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.291 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.291 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.292 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.293 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.293 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.293 DEBUG:boto:Data: 2023-10-11T15:14:16.294 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.295 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.295 DEBUG:boto:Params: {} 2023-10-11T15:14:16.296 DEBUG:boto:Token: None 2023-10-11T15:14:16.296 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5EdRtNX7P9X9BqJydulJA3n2jBM= 2023-10-11T15:14:16.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5EdRtNX7P9X9BqJydulJA3n2jBM=', 'Content-Length': '0'} 2023-10-11T15:14:16.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059852b6905b6a9c5-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.300 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.301 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.301 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.302 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.302 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.302 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.303 DEBUG:boto:Data: 2023-10-11T15:14:16.303 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.304 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.305 DEBUG:boto:Params: {} 2023-10-11T15:14:16.305 DEBUG:boto:Token: None 2023-10-11T15:14:16.306 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5EdRtNX7P9X9BqJydulJA3n2jBM= 2023-10-11T15:14:16.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5EdRtNX7P9X9BqJydulJA3n2jBM=', 'Content-Length': '0'} 2023-10-11T15:14:16.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919bc3024bd3071f-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-11T15:14:16.310 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.310 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.311 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.311 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.312 DEBUG:boto:Method: GET 2023-10-11T15:14:16.312 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.313 DEBUG:boto:Data: 2023-10-11T15:14:16.314 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.315 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.315 DEBUG:boto:Params: {} 2023-10-11T15:14:16.316 DEBUG:boto:Token: None 2023-10-11T15:14:16.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y= 2023-10-11T15:14:16.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d88d1494ca8f1f-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.320 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.321 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.321 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.322 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.322 DEBUG:boto:Method: GET 2023-10-11T15:14:16.323 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.323 DEBUG:boto:Data: 2023-10-11T15:14:16.324 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.325 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.326 DEBUG:boto:Params: {} 2023-10-11T15:14:16.327 DEBUG:boto:Token: None 2023-10-11T15:14:16.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006?versionId=null 2023-10-11T15:14:16.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y= 2023-10-11T15:14:16.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5i8UiLQ14nDgnWZWocKPRFf/B5Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079dd13d4db0f7fd0-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.331 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.331 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.332 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.332 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.333 DEBUG:boto:Method: GET 2023-10-11T15:14:16.333 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.333 DEBUG:boto:Data: 2023-10-11T15:14:16.334 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.335 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.335 DEBUG:boto:Params: {} 2023-10-11T15:14:16.336 DEBUG:boto:Token: None 2023-10-11T15:14:16.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000898e5c039c3f3f7-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.340 DEBUG:boto:b'dghqvd-58obj000000000000000000061000falseobj000000000000000000062023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.341 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.342 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.342 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.343 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.343 DEBUG:boto:Method: GET 2023-10-11T15:14:16.344 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.344 DEBUG:boto:Data: 2023-10-11T15:14:16.345 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.345 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.346 DEBUG:boto:Params: {} 2023-10-11T15:14:16.346 DEBUG:boto:Token: None 2023-10-11T15:14:16.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe6b862890fce3e-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.354 DEBUG:boto:b'dghqvd-58obj000000000000000000061000falseobj000000000000000000062023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.355 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.355 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.356 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.356 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.357 DEBUG:boto:Data: 2023-10-11T15:14:16.357 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.358 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.359 DEBUG:boto:Params: {} 2023-10-11T15:14:16.359 DEBUG:boto:Token: None 2023-10-11T15:14:16.360 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yv3vbPOx4vR1xZ9oAJVZDPxPXDE= 2023-10-11T15:14:16.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yv3vbPOx4vR1xZ9oAJVZDPxPXDE=', 'Content-Length': '0'} 2023-10-11T15:14:16.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842416982a00a305-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.363 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.364 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.364 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.364 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.365 DEBUG:boto:Data: 2023-10-11T15:14:16.365 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.366 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.367 DEBUG:boto:Params: {} 2023-10-11T15:14:16.367 DEBUG:boto:Token: None 2023-10-11T15:14:16.367 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yv3vbPOx4vR1xZ9oAJVZDPxPXDE= 2023-10-11T15:14:16.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yv3vbPOx4vR1xZ9oAJVZDPxPXDE=', 'Content-Length': '0'} 2023-10-11T15:14:16.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b379cca5373366-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-10-11T15:14:16.372 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.372 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.372 DEBUG:boto:Method: GET 2023-10-11T15:14:16.373 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.373 DEBUG:boto:Data: 2023-10-11T15:14:16.374 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.375 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.375 DEBUG:boto:Params: {} 2023-10-11T15:14:16.376 DEBUG:boto:Token: None 2023-10-11T15:14:16.376 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:daddA04BRGtTOojuzRt4BvR9M/0= 2023-10-11T15:14:16.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:daddA04BRGtTOojuzRt4BvR9M/0=', 'Content-Length': '0'} 2023-10-11T15:14:16.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aec9632f94139a9-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.426 DEBUG:boto:path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.427 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.428 DEBUG:boto:Method: GET 2023-10-11T15:14:16.428 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.428 DEBUG:boto:Data: 2023-10-11T15:14:16.430 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.431 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.432 DEBUG:boto:Params: {} 2023-10-11T15:14:16.433 DEBUG:boto:Token: None 2023-10-11T15:14:16.433 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000006 2023-10-11T15:14:16.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:daddA04BRGtTOojuzRt4BvR9M/0= 2023-10-11T15:14:16.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:daddA04BRGtTOojuzRt4BvR9M/0=', 'Content-Length': '0'} 2023-10-11T15:14:16.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24278172aadd365-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.437 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.438 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.438 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.438 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.439 DEBUG:boto:Method: GET 2023-10-11T15:14:16.439 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.440 DEBUG:boto:Data: 2023-10-11T15:14:16.440 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.441 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.441 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.442 DEBUG:boto:Params: {} 2023-10-11T15:14:16.442 DEBUG:boto:Token: None 2023-10-11T15:14:16.443 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001505dc611377ae40-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.447 DEBUG:boto:b'dghqvd-58obj000000000000000000061000falseobj000000000000000000062023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.448 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.449 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.449 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.450 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.450 DEBUG:boto:Method: GET 2023-10-11T15:14:16.451 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000006 2023-10-11T15:14:16.451 DEBUG:boto:Data: 2023-10-11T15:14:16.452 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.452 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.453 DEBUG:boto:Params: {} 2023-10-11T15:14:16.453 DEBUG:boto:Token: None 2023-10-11T15:14:16.454 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c518802682af6eb-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.460 DEBUG:boto:b'dghqvd-58obj000000000000000000061000falseobj000000000000000000062023-10-11T15:14:05.461Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-11T15:14:16.461 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.461 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.462 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.462 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.463 DEBUG:boto:Method: GET 2023-10-11T15:14:16.463 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.464 DEBUG:boto:Data: 2023-10-11T15:14:16.464 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.465 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.465 DEBUG:boto:Params: {} 2023-10-11T15:14:16.466 DEBUG:boto:Token: None 2023-10-11T15:14:16.466 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E= 2023-10-11T15:14:16.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E=', 'Content-Length': '0'} 2023-10-11T15:14:16.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4fc0d6d4bbeb794-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.470 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.471 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.471 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.472 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.472 DEBUG:boto:Method: GET 2023-10-11T15:14:16.473 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.473 DEBUG:boto:Data: 2023-10-11T15:14:16.473 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.474 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.475 DEBUG:boto:Params: {} 2023-10-11T15:14:16.475 DEBUG:boto:Token: None 2023-10-11T15:14:16.476 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E= 2023-10-11T15:14:16.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E=', 'Content-Length': '0'} 2023-10-11T15:14:16.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014cf89d4378775d-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.482 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.482 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.482 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.483 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.483 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.484 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.484 DEBUG:boto:Data: 2023-10-11T15:14:16.485 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.485 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.486 DEBUG:boto:Params: {} 2023-10-11T15:14:16.486 DEBUG:boto:Token: None 2023-10-11T15:14:16.487 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZTlYaIkwVae0S0v692kYKu7IKnc= 2023-10-11T15:14:16.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZTlYaIkwVae0S0v692kYKu7IKnc=', 'Content-Length': '0'} 2023-10-11T15:14:16.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c0bece775c81c1-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.490 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.491 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.491 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.492 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.492 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.493 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.493 DEBUG:boto:Data: 2023-10-11T15:14:16.493 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.494 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.495 DEBUG:boto:Params: {} 2023-10-11T15:14:16.495 DEBUG:boto:Token: None 2023-10-11T15:14:16.496 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.496 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZTlYaIkwVae0S0v692kYKu7IKnc= 2023-10-11T15:14:16.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZTlYaIkwVae0S0v692kYKu7IKnc=', 'Content-Length': '0'} 2023-10-11T15:14:16.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99eb6e47f7b2ad9-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-11T15:14:16.499 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.500 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.500 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.501 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.501 DEBUG:boto:Method: GET 2023-10-11T15:14:16.501 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.503 DEBUG:boto:Data: 2023-10-11T15:14:16.503 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.503 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.504 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.504 DEBUG:boto:Params: {} 2023-10-11T15:14:16.505 DEBUG:boto:Token: None 2023-10-11T15:14:16.505 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E= 2023-10-11T15:14:16.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E=', 'Content-Length': '0'} 2023-10-11T15:14:16.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003991a2662a34237a-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.509 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.509 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.510 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.510 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.510 DEBUG:boto:Method: GET 2023-10-11T15:14:16.511 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.511 DEBUG:boto:Data: 2023-10-11T15:14:16.512 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.512 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.513 DEBUG:boto:Params: {} 2023-10-11T15:14:16.513 DEBUG:boto:Token: None 2023-10-11T15:14:16.514 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007?versionId=null 2023-10-11T15:14:16.514 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E= 2023-10-11T15:14:16.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RX+EUwPafrmwMosu51SwgsF/b7E=', 'Content-Length': '0'} 2023-10-11T15:14:16.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d3d300b236fa01-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.558 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.559 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.560 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.560 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.560 DEBUG:boto:Method: GET 2023-10-11T15:14:16.561 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.561 DEBUG:boto:Data: 2023-10-11T15:14:16.562 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.562 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.563 DEBUG:boto:Params: {} 2023-10-11T15:14:16.563 DEBUG:boto:Token: None 2023-10-11T15:14:16.564 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f601feaa21e8699d-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.570 DEBUG:boto:b'dghqvd-58obj000000000000000000071000falseobj000000000000000000072023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.571 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.571 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.572 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.572 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.573 DEBUG:boto:Method: GET 2023-10-11T15:14:16.573 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.574 DEBUG:boto:Data: 2023-10-11T15:14:16.574 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.575 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.575 DEBUG:boto:Params: {} 2023-10-11T15:14:16.576 DEBUG:boto:Token: None 2023-10-11T15:14:16.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4c5a44d646c7f7-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.582 DEBUG:boto:b'dghqvd-58obj000000000000000000071000falseobj000000000000000000072023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.583 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.583 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.584 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.584 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.585 DEBUG:boto:Data: 2023-10-11T15:14:16.585 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.586 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.586 DEBUG:boto:Params: {} 2023-10-11T15:14:16.587 DEBUG:boto:Token: None 2023-10-11T15:14:16.587 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ek02qkdvvlQKHHsGHuDJnOwt4+U= 2023-10-11T15:14:16.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ek02qkdvvlQKHHsGHuDJnOwt4+U=', 'Content-Length': '0'} 2023-10-11T15:14:16.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e5861c65fb1dfd-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.591 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.591 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.592 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.592 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.592 DEBUG:boto:Data: 2023-10-11T15:14:16.593 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.594 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.594 DEBUG:boto:Params: {} 2023-10-11T15:14:16.594 DEBUG:boto:Token: None 2023-10-11T15:14:16.595 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ek02qkdvvlQKHHsGHuDJnOwt4+U= 2023-10-11T15:14:16.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ek02qkdvvlQKHHsGHuDJnOwt4+U=', 'Content-Length': '0'} 2023-10-11T15:14:16.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8a4b28987015d4-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-10-11T15:14:16.599 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.599 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.600 DEBUG:boto:Method: GET 2023-10-11T15:14:16.600 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.600 DEBUG:boto:Data: 2023-10-11T15:14:16.601 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.602 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.602 DEBUG:boto:Params: {} 2023-10-11T15:14:16.603 DEBUG:boto:Token: None 2023-10-11T15:14:16.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jbvO96TmaGtiO+E1Ig1Fq48Mz1g= 2023-10-11T15:14:16.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jbvO96TmaGtiO+E1Ig1Fq48Mz1g=', 'Content-Length': '0'} 2023-10-11T15:14:16.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147ce391d48dd4de-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.646 DEBUG:boto:path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.647 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.647 DEBUG:boto:Method: GET 2023-10-11T15:14:16.648 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.648 DEBUG:boto:Data: 2023-10-11T15:14:16.649 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.650 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.650 DEBUG:boto:Params: {} 2023-10-11T15:14:16.651 DEBUG:boto:Token: None 2023-10-11T15:14:16.651 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000007 2023-10-11T15:14:16.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jbvO96TmaGtiO+E1Ig1Fq48Mz1g= 2023-10-11T15:14:16.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jbvO96TmaGtiO+E1Ig1Fq48Mz1g=', 'Content-Length': '0'} 2023-10-11T15:14:16.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a99b499780484c-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.655 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.656 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.656 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.656 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.657 DEBUG:boto:Method: GET 2023-10-11T15:14:16.657 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.658 DEBUG:boto:Data: 2023-10-11T15:14:16.658 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.659 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.660 DEBUG:boto:Params: {} 2023-10-11T15:14:16.660 DEBUG:boto:Token: None 2023-10-11T15:14:16.661 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000400a8b26e84234e9-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.665 DEBUG:boto:b'dghqvd-58obj000000000000000000071000falseobj000000000000000000072023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.666 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.667 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.667 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.668 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.668 DEBUG:boto:Method: GET 2023-10-11T15:14:16.668 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000007 2023-10-11T15:14:16.669 DEBUG:boto:Data: 2023-10-11T15:14:16.669 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.670 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.671 DEBUG:boto:Params: {} 2023-10-11T15:14:16.671 DEBUG:boto:Token: None 2023-10-11T15:14:16.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bfa5f0655018261-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.678 DEBUG:boto:b'dghqvd-58obj000000000000000000071000falseobj000000000000000000072023-10-11T15:14:04.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-11T15:14:16.679 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.679 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.680 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.680 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.680 DEBUG:boto:Method: GET 2023-10-11T15:14:16.681 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.682 DEBUG:boto:Data: 2023-10-11T15:14:16.682 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.686 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.686 DEBUG:boto:Params: {} 2023-10-11T15:14:16.687 DEBUG:boto:Token: None 2023-10-11T15:14:16.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y= 2023-10-11T15:14:16.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001226191bbddcea5c-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.691 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.692 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.692 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.692 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.693 DEBUG:boto:Method: GET 2023-10-11T15:14:16.693 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.694 DEBUG:boto:Data: 2023-10-11T15:14:16.694 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.695 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.696 DEBUG:boto:Params: {} 2023-10-11T15:14:16.696 DEBUG:boto:Token: None 2023-10-11T15:14:16.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y= 2023-10-11T15:14:16.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002977b23d3ccf6a76-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.742 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.743 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.744 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.744 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.745 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.745 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.746 DEBUG:boto:Data: 2023-10-11T15:14:16.746 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.747 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.747 DEBUG:boto:Params: {} 2023-10-11T15:14:16.748 DEBUG:boto:Token: None 2023-10-11T15:14:16.748 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7K+y1AqvDliqjaVX9KU6fN2hYo= 2023-10-11T15:14:16.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7K+y1AqvDliqjaVX9KU6fN2hYo=', 'Content-Length': '0'} 2023-10-11T15:14:16.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56fe93676b3b8ac-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.752 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.753 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.753 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.754 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.754 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.755 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.755 DEBUG:boto:Data: 2023-10-11T15:14:16.756 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.756 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.757 DEBUG:boto:Params: {} 2023-10-11T15:14:16.757 DEBUG:boto:Token: None 2023-10-11T15:14:16.758 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7K+y1AqvDliqjaVX9KU6fN2hYo= 2023-10-11T15:14:16.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7K+y1AqvDliqjaVX9KU6fN2hYo=', 'Content-Length': '0'} 2023-10-11T15:14:16.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f411b9559efa1fb3-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-11T15:14:16.762 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.763 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.763 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.764 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.764 DEBUG:boto:Method: GET 2023-10-11T15:14:16.764 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.765 DEBUG:boto:Data: 2023-10-11T15:14:16.765 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.766 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.767 DEBUG:boto:Params: {} 2023-10-11T15:14:16.767 DEBUG:boto:Token: None 2023-10-11T15:14:16.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y= 2023-10-11T15:14:16.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e19f5ccfc7c5cd-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.772 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.772 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.773 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.773 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.774 DEBUG:boto:Method: GET 2023-10-11T15:14:16.774 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.775 DEBUG:boto:Data: 2023-10-11T15:14:16.775 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.776 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.777 DEBUG:boto:Params: {} 2023-10-11T15:14:16.777 DEBUG:boto:Token: None 2023-10-11T15:14:16.778 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008?versionId=null 2023-10-11T15:14:16.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y= 2023-10-11T15:14:16.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KiD7YgTPn9adwYU15SGrKtfNw6Y=', 'Content-Length': '0'} 2023-10-11T15:14:16.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2328d0f40cf2d7d-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.782 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.782 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.783 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.783 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.784 DEBUG:boto:Method: GET 2023-10-11T15:14:16.784 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.785 DEBUG:boto:Data: 2023-10-11T15:14:16.785 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.786 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.786 DEBUG:boto:Params: {} 2023-10-11T15:14:16.787 DEBUG:boto:Token: None 2023-10-11T15:14:16.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca2485b51f01bf8-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.791 DEBUG:boto:b'dghqvd-58obj000000000000000000081000falseobj000000000000000000082023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.792 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.793 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.793 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.794 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.794 DEBUG:boto:Method: GET 2023-10-11T15:14:16.795 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.795 DEBUG:boto:Data: 2023-10-11T15:14:16.796 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.796 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.797 DEBUG:boto:Params: {} 2023-10-11T15:14:16.797 DEBUG:boto:Token: None 2023-10-11T15:14:16.798 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2c4005d58db6c9e-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.804 DEBUG:boto:b'dghqvd-58obj000000000000000000081000falseobj000000000000000000082023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.805 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.805 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.805 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.806 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.806 DEBUG:boto:Data: 2023-10-11T15:14:16.807 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.808 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.808 DEBUG:boto:Params: {} 2023-10-11T15:14:16.809 DEBUG:boto:Token: None 2023-10-11T15:14:16.809 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PyAjlFlmxzvqsuSaPcbvGsexSTQ= 2023-10-11T15:14:16.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PyAjlFlmxzvqsuSaPcbvGsexSTQ=', 'Content-Length': '0'} 2023-10-11T15:14:16.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99951421ae9504a-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.814 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.815 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.815 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.816 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.816 DEBUG:boto:Data: 2023-10-11T15:14:16.817 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.817 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.818 DEBUG:boto:Params: {} 2023-10-11T15:14:16.818 DEBUG:boto:Token: None 2023-10-11T15:14:16.819 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PyAjlFlmxzvqsuSaPcbvGsexSTQ= 2023-10-11T15:14:16.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PyAjlFlmxzvqsuSaPcbvGsexSTQ=', 'Content-Length': '0'} 2023-10-11T15:14:16.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a004be22aff758-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-10-11T15:14:16.823 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.823 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.824 DEBUG:boto:Method: GET 2023-10-11T15:14:16.824 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.825 DEBUG:boto:Data: 2023-10-11T15:14:16.825 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.826 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.826 DEBUG:boto:Params: {} 2023-10-11T15:14:16.827 DEBUG:boto:Token: None 2023-10-11T15:14:16.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2hLd2eLpZC0+pOPPtAAdAA/gXZs= 2023-10-11T15:14:16.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2hLd2eLpZC0+pOPPtAAdAA/gXZs=', 'Content-Length': '0'} 2023-10-11T15:14:16.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003027e0235339a974-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.878 DEBUG:boto:path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.879 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.879 DEBUG:boto:Method: GET 2023-10-11T15:14:16.880 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.880 DEBUG:boto:Data: 2023-10-11T15:14:16.881 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.881 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.882 DEBUG:boto:Params: {} 2023-10-11T15:14:16.882 DEBUG:boto:Token: None 2023-10-11T15:14:16.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000008 2023-10-11T15:14:16.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2hLd2eLpZC0+pOPPtAAdAA/gXZs= 2023-10-11T15:14:16.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2hLd2eLpZC0+pOPPtAAdAA/gXZs=', 'Content-Length': '0'} 2023-10-11T15:14:16.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74a7c4e9926e363-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.887 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.890 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.890 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.891 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.891 DEBUG:boto:Method: GET 2023-10-11T15:14:16.892 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.892 DEBUG:boto:Data: 2023-10-11T15:14:16.892 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.893 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.893 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.894 DEBUG:boto:Params: {} 2023-10-11T15:14:16.894 DEBUG:boto:Token: None 2023-10-11T15:14:16.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.895 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd2863e6f902c26-006526bbc8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.899 DEBUG:boto:b'dghqvd-58obj000000000000000000081000falseobj000000000000000000082023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.900 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:16.901 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:16.901 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.902 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.902 DEBUG:boto:Method: GET 2023-10-11T15:14:16.902 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000008 2023-10-11T15:14:16.903 DEBUG:boto:Data: 2023-10-11T15:14:16.903 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.904 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.904 DEBUG:boto:Params: {} 2023-10-11T15:14:16.905 DEBUG:boto:Token: None 2023-10-11T15:14:16.905 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/ 2023-10-11T15:14:16.906 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8= 2023-10-11T15:14:16.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZyVGUocRSnIbdtrO7hlH/rJHEL8=', 'Content-Length': '0'} 2023-10-11T15:14:16.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003590fdb72dbef7c9-006526bbc8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.911 DEBUG:boto:b'dghqvd-58obj000000000000000000081000falseobj000000000000000000082023-10-11T15:14:05.134Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:16.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-11T15:14:16.912 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.913 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.913 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.913 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.914 DEBUG:boto:Method: GET 2023-10-11T15:14:16.914 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.915 DEBUG:boto:Data: 2023-10-11T15:14:16.915 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.916 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.916 DEBUG:boto:Params: {} 2023-10-11T15:14:16.917 DEBUG:boto:Token: None 2023-10-11T15:14:16.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw= 2023-10-11T15:14:16.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw=', 'Content-Length': '0'} 2023-10-11T15:14:16.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4827680505d7d86-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.921 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.922 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.922 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.923 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.923 DEBUG:boto:Method: GET 2023-10-11T15:14:16.923 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.924 DEBUG:boto:Data: 2023-10-11T15:14:16.924 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.925 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.926 DEBUG:boto:Params: {} 2023-10-11T15:14:16.926 DEBUG:boto:Token: None 2023-10-11T15:14:16.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw= 2023-10-11T15:14:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw=', 'Content-Length': '0'} 2023-10-11T15:14:16.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0dbc928977ba866-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.970 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.971 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.971 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.972 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.972 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.972 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.973 DEBUG:boto:Data: 2023-10-11T15:14:16.973 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.974 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.974 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.975 DEBUG:boto:Params: {} 2023-10-11T15:14:16.975 DEBUG:boto:Token: None 2023-10-11T15:14:16.976 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oIKn+MpNp2Zdn15BoWge45p8cNs= 2023-10-11T15:14:16.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oIKn+MpNp2Zdn15BoWge45p8cNs=', 'Content-Length': '0'} 2023-10-11T15:14:16.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050be524d3b228aac-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.979 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.980 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.980 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.981 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.981 DEBUG:boto:Method: HEAD 2023-10-11T15:14:16.982 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.982 DEBUG:boto:Data: 2023-10-11T15:14:16.982 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:16.983 DEBUG:boto:Port: 8000 2023-10-11T15:14:16.984 DEBUG:boto:Params: {} 2023-10-11T15:14:16.984 DEBUG:boto:Token: None 2023-10-11T15:14:16.985 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.985 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oIKn+MpNp2Zdn15BoWge45p8cNs= 2023-10-11T15:14:16.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oIKn+MpNp2Zdn15BoWge45p8cNs=', 'Content-Length': '0'} 2023-10-11T15:14:16.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bea22f2e191d81e-006526bbc8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:16.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-11T15:14:16.989 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.989 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:16.989 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.990 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.990 DEBUG:boto:Method: GET 2023-10-11T15:14:16.991 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.991 DEBUG:boto:Data: 2023-10-11T15:14:16.992 DEBUG:boto:Headers: {} 2023-10-11T15:14:16.992 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:16.992 DEBUG:boto:Port: 8001 2023-10-11T15:14:16.993 DEBUG:boto:Params: {} 2023-10-11T15:14:16.993 DEBUG:boto:Token: None 2023-10-11T15:14:16.994 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:16 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:16.994 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw= 2023-10-11T15:14:16.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qetpuHbakIgtH8km7+suH9QI0pw=', 'Content-Length': '0'} 2023-10-11T15:14:16.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cacb191e8c9cbbc-006526bbc8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.038 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.039 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.039 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:17.040 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:17.040 DEBUG:boto:Method: GET 2023-10-11T15:14:17.041 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:17.041 DEBUG:boto:Data: 2023-10-11T15:14:17.042 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.043 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.043 DEBUG:boto:Params: {} 2023-10-11T15:14:17.044 DEBUG:boto:Token: None 2023-10-11T15:14:17.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000009?versionId=null 2023-10-11T15:14:17.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XNX19s6m7XeDJO4zd6gsg1TaKI8= 2023-10-11T15:14:17.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XNX19s6m7XeDJO4zd6gsg1TaKI8=', 'Content-Length': '0'} 2023-10-11T15:14:17.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c3274bbb86a096-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.048 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.048 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.049 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.049 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.050 DEBUG:boto:Method: GET 2023-10-11T15:14:17.050 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.050 DEBUG:boto:Data: 2023-10-11T15:14:17.051 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.052 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.052 DEBUG:boto:Params: {} 2023-10-11T15:14:17.053 DEBUG:boto:Token: None 2023-10-11T15:14:17.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b025390ff6fb844-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.057 DEBUG:boto:b'dghqvd-58obj000000000000000000091000falseobj000000000000000000092023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.058 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.058 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.059 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.059 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.060 DEBUG:boto:Method: GET 2023-10-11T15:14:17.060 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.060 DEBUG:boto:Data: 2023-10-11T15:14:17.061 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.062 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.062 DEBUG:boto:Params: {} 2023-10-11T15:14:17.063 DEBUG:boto:Token: None 2023-10-11T15:14:17.063 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057ad2280f0b55cd1-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.069 DEBUG:boto:b'dghqvd-58obj000000000000000000091000falseobj000000000000000000092023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.070 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.070 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.071 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.071 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.071 DEBUG:boto:Data: 2023-10-11T15:14:17.072 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.073 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.073 DEBUG:boto:Params: {} 2023-10-11T15:14:17.073 DEBUG:boto:Token: None 2023-10-11T15:14:17.074 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ghB97fht5io4dpqWLf09zZtVQ48= 2023-10-11T15:14:17.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ghB97fht5io4dpqWLf09zZtVQ48=', 'Content-Length': '0'} 2023-10-11T15:14:17.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5341166062b50d-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.078 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.078 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.079 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.079 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.079 DEBUG:boto:Data: 2023-10-11T15:14:17.080 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.080 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.081 DEBUG:boto:Params: {} 2023-10-11T15:14:17.081 DEBUG:boto:Token: None 2023-10-11T15:14:17.082 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ghB97fht5io4dpqWLf09zZtVQ48= 2023-10-11T15:14:17.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ghB97fht5io4dpqWLf09zZtVQ48=', 'Content-Length': '0'} 2023-10-11T15:14:17.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002407c81465a2ba08-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-10-11T15:14:17.086 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.086 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.087 DEBUG:boto:Method: GET 2023-10-11T15:14:17.087 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.088 DEBUG:boto:Data: 2023-10-11T15:14:17.088 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.089 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.089 DEBUG:boto:Params: {} 2023-10-11T15:14:17.090 DEBUG:boto:Token: None 2023-10-11T15:14:17.090 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nAXt4tlxY+wU2oNDzWtbRoLor28= 2023-10-11T15:14:17.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nAXt4tlxY+wU2oNDzWtbRoLor28=', 'Content-Length': '0'} 2023-10-11T15:14:17.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b43f5c96890d63-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.094 DEBUG:boto:path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.094 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.095 DEBUG:boto:Method: GET 2023-10-11T15:14:17.095 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.097 DEBUG:boto:Data: 2023-10-11T15:14:17.097 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.098 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.099 DEBUG:boto:Params: {} 2023-10-11T15:14:17.099 DEBUG:boto:Token: None 2023-10-11T15:14:17.100 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000009 2023-10-11T15:14:17.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nAXt4tlxY+wU2oNDzWtbRoLor28= 2023-10-11T15:14:17.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nAXt4tlxY+wU2oNDzWtbRoLor28=', 'Content-Length': '0'} 2023-10-11T15:14:17.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0982da76b86364-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.150 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.151 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.151 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.152 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.152 DEBUG:boto:Method: GET 2023-10-11T15:14:17.153 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.153 DEBUG:boto:Data: 2023-10-11T15:14:17.154 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.155 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.155 DEBUG:boto:Params: {} 2023-10-11T15:14:17.155 DEBUG:boto:Token: None 2023-10-11T15:14:17.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c356a7f1cf17cf2-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.160 DEBUG:boto:b'dghqvd-58obj000000000000000000091000falseobj000000000000000000092023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.161 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.161 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.162 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.162 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.162 DEBUG:boto:Method: GET 2023-10-11T15:14:17.163 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000009 2023-10-11T15:14:17.163 DEBUG:boto:Data: 2023-10-11T15:14:17.164 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.165 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.165 DEBUG:boto:Params: {} 2023-10-11T15:14:17.165 DEBUG:boto:Token: None 2023-10-11T15:14:17.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6b23ba3840cb059-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.172 DEBUG:boto:b'dghqvd-58obj000000000000000000091000falseobj000000000000000000092023-10-11T15:14:05.196Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:14:17.173 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.174 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.174 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.174 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.175 DEBUG:boto:Method: GET 2023-10-11T15:14:17.175 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.176 DEBUG:boto:Data: 2023-10-11T15:14:17.176 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.177 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.177 DEBUG:boto:Params: {} 2023-10-11T15:14:17.178 DEBUG:boto:Token: None 2023-10-11T15:14:17.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0= 2023-10-11T15:14:17.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e982d0b1711e9e-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.183 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.183 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.183 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.184 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.184 DEBUG:boto:Method: GET 2023-10-11T15:14:17.185 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.185 DEBUG:boto:Data: 2023-10-11T15:14:17.186 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.187 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.187 DEBUG:boto:Params: {} 2023-10-11T15:14:17.187 DEBUG:boto:Token: None 2023-10-11T15:14:17.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0= 2023-10-11T15:14:17.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37d01e27a1ccbfb-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.192 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.192 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.193 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.193 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.194 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.194 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.195 DEBUG:boto:Data: 2023-10-11T15:14:17.195 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.196 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.196 DEBUG:boto:Params: {} 2023-10-11T15:14:17.197 DEBUG:boto:Token: None 2023-10-11T15:14:17.197 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MZCJDbqNKqUXqbIUP3eHn0ZHmQ4= 2023-10-11T15:14:17.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MZCJDbqNKqUXqbIUP3eHn0ZHmQ4=', 'Content-Length': '0'} 2023-10-11T15:14:17.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b94384f51ab609b-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.200 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.201 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.201 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.202 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.202 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.202 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.203 DEBUG:boto:Data: 2023-10-11T15:14:17.203 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.204 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.204 DEBUG:boto:Params: {} 2023-10-11T15:14:17.205 DEBUG:boto:Token: None 2023-10-11T15:14:17.205 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MZCJDbqNKqUXqbIUP3eHn0ZHmQ4= 2023-10-11T15:14:17.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MZCJDbqNKqUXqbIUP3eHn0ZHmQ4=', 'Content-Length': '0'} 2023-10-11T15:14:17.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc7e8b1a7df294f-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:14:17.209 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.210 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.210 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.211 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.211 DEBUG:boto:Method: GET 2023-10-11T15:14:17.212 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.212 DEBUG:boto:Data: 2023-10-11T15:14:17.212 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.213 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.214 DEBUG:boto:Params: {} 2023-10-11T15:14:17.214 DEBUG:boto:Token: None 2023-10-11T15:14:17.215 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0= 2023-10-11T15:14:17.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1508dcb39db44de-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.218 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.219 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.219 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.220 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.220 DEBUG:boto:Method: GET 2023-10-11T15:14:17.221 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.221 DEBUG:boto:Data: 2023-10-11T15:14:17.221 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.222 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.223 DEBUG:boto:Params: {} 2023-10-11T15:14:17.223 DEBUG:boto:Token: None 2023-10-11T15:14:17.224 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010?versionId=null 2023-10-11T15:14:17.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0= 2023-10-11T15:14:17.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9KPTVFMRh/WkiARhK1fDg80/wC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075aefadc0ef4d88c-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.270 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.271 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.271 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.272 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.272 DEBUG:boto:Method: GET 2023-10-11T15:14:17.273 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.273 DEBUG:boto:Data: 2023-10-11T15:14:17.274 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.274 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.275 DEBUG:boto:Params: {} 2023-10-11T15:14:17.275 DEBUG:boto:Token: None 2023-10-11T15:14:17.276 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.276 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf9c83301a62e151-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.280 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.281 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.281 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.282 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.282 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.282 DEBUG:boto:Method: GET 2023-10-11T15:14:17.283 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.283 DEBUG:boto:Data: 2023-10-11T15:14:17.284 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.285 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.285 DEBUG:boto:Params: {} 2023-10-11T15:14:17.285 DEBUG:boto:Token: None 2023-10-11T15:14:17.286 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1d1cf15c3db59c-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.292 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.293 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.293 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.294 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.294 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.294 DEBUG:boto:Data: 2023-10-11T15:14:17.295 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.296 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.296 DEBUG:boto:Params: {} 2023-10-11T15:14:17.296 DEBUG:boto:Token: None 2023-10-11T15:14:17.297 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wraq7JdboWtyqHivH0KEDL6Wcz4= 2023-10-11T15:14:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wraq7JdboWtyqHivH0KEDL6Wcz4=', 'Content-Length': '0'} 2023-10-11T15:14:17.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096110e051b1d01ea-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.300 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.301 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.301 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.302 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.302 DEBUG:boto:Data: 2023-10-11T15:14:17.302 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.303 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.304 DEBUG:boto:Params: {} 2023-10-11T15:14:17.304 DEBUG:boto:Token: None 2023-10-11T15:14:17.305 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wraq7JdboWtyqHivH0KEDL6Wcz4= 2023-10-11T15:14:17.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wraq7JdboWtyqHivH0KEDL6Wcz4=', 'Content-Length': '0'} 2023-10-11T15:14:17.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b9d853016cb194-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-10-11T15:14:17.309 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.309 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.309 DEBUG:boto:Method: GET 2023-10-11T15:14:17.310 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.310 DEBUG:boto:Data: 2023-10-11T15:14:17.311 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.312 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.312 DEBUG:boto:Params: {} 2023-10-11T15:14:17.312 DEBUG:boto:Token: None 2023-10-11T15:14:17.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jeQsXMPYuFf+4nvwV79QGQDp1N4= 2023-10-11T15:14:17.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jeQsXMPYuFf+4nvwV79QGQDp1N4=', 'Content-Length': '0'} 2023-10-11T15:14:17.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b671c628f54d0b2e-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.358 DEBUG:boto:path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.359 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.359 DEBUG:boto:Method: GET 2023-10-11T15:14:17.360 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.360 DEBUG:boto:Data: 2023-10-11T15:14:17.361 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.362 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.362 DEBUG:boto:Params: {} 2023-10-11T15:14:17.363 DEBUG:boto:Token: None 2023-10-11T15:14:17.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000010 2023-10-11T15:14:17.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jeQsXMPYuFf+4nvwV79QGQDp1N4= 2023-10-11T15:14:17.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jeQsXMPYuFf+4nvwV79QGQDp1N4=', 'Content-Length': '0'} 2023-10-11T15:14:17.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcc7f9182682b06-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.367 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.368 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.368 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.368 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.369 DEBUG:boto:Method: GET 2023-10-11T15:14:17.369 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.370 DEBUG:boto:Data: 2023-10-11T15:14:17.370 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.371 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.372 DEBUG:boto:Params: {} 2023-10-11T15:14:17.372 DEBUG:boto:Token: None 2023-10-11T15:14:17.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc91e749ab8b8e99-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.377 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.377 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.378 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.378 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.379 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.379 DEBUG:boto:Method: GET 2023-10-11T15:14:17.380 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000010 2023-10-11T15:14:17.380 DEBUG:boto:Data: 2023-10-11T15:14:17.381 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.383 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.383 DEBUG:boto:Params: {} 2023-10-11T15:14:17.384 DEBUG:boto:Token: None 2023-10-11T15:14:17.385 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b4d341c85fc5573-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.434 DEBUG:boto:b'dghqvd-58obj000000000000000000101000falseobj000000000000000000102023-10-11T15:14:04.619Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:14:17.436 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.436 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.437 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.437 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.437 DEBUG:boto:Method: GET 2023-10-11T15:14:17.438 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.438 DEBUG:boto:Data: 2023-10-11T15:14:17.439 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.440 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.440 DEBUG:boto:Params: {} 2023-10-11T15:14:17.441 DEBUG:boto:Token: None 2023-10-11T15:14:17.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw= 2023-10-11T15:14:17.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw=', 'Content-Length': '0'} 2023-10-11T15:14:17.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a4815a6dc0673d-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.445 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.446 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.446 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.447 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.447 DEBUG:boto:Method: GET 2023-10-11T15:14:17.447 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.448 DEBUG:boto:Data: 2023-10-11T15:14:17.448 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.449 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.450 DEBUG:boto:Params: {} 2023-10-11T15:14:17.450 DEBUG:boto:Token: None 2023-10-11T15:14:17.451 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.451 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw= 2023-10-11T15:14:17.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw=', 'Content-Length': '0'} 2023-10-11T15:14:17.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a4b06467df76ef-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.455 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.456 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.457 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.457 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.457 DEBUG:boto:Data: 2023-10-11T15:14:17.458 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.459 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.459 DEBUG:boto:Params: {} 2023-10-11T15:14:17.460 DEBUG:boto:Token: None 2023-10-11T15:14:17.460 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EHnvzxwsolEgJlZ86TdCyqDg9xk= 2023-10-11T15:14:17.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EHnvzxwsolEgJlZ86TdCyqDg9xk=', 'Content-Length': '0'} 2023-10-11T15:14:17.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfedf626ffc29274-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.464 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.464 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.465 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.465 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.466 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.467 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.467 DEBUG:boto:Data: 2023-10-11T15:14:17.467 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.468 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.469 DEBUG:boto:Params: {} 2023-10-11T15:14:17.469 DEBUG:boto:Token: None 2023-10-11T15:14:17.470 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EHnvzxwsolEgJlZ86TdCyqDg9xk= 2023-10-11T15:14:17.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EHnvzxwsolEgJlZ86TdCyqDg9xk=', 'Content-Length': '0'} 2023-10-11T15:14:17.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011817c2dd1f9c2e7-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:14:17.475 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.475 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.476 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.476 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.477 DEBUG:boto:Method: GET 2023-10-11T15:14:17.477 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.478 DEBUG:boto:Data: 2023-10-11T15:14:17.478 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.479 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.480 DEBUG:boto:Params: {} 2023-10-11T15:14:17.480 DEBUG:boto:Token: None 2023-10-11T15:14:17.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw= 2023-10-11T15:14:17.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw=', 'Content-Length': '0'} 2023-10-11T15:14:17.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a22dce3452a27a-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.485 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.485 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.486 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.486 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.487 DEBUG:boto:Method: GET 2023-10-11T15:14:17.487 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.488 DEBUG:boto:Data: 2023-10-11T15:14:17.488 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.489 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.490 DEBUG:boto:Params: {} 2023-10-11T15:14:17.490 DEBUG:boto:Token: None 2023-10-11T15:14:17.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011?versionId=null 2023-10-11T15:14:17.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw= 2023-10-11T15:14:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BgByzJUV8mIX38NrhhRG9RWrKdw=', 'Content-Length': '0'} 2023-10-11T15:14:17.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2087c146df5714b-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.495 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.495 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.496 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.496 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.497 DEBUG:boto:Method: GET 2023-10-11T15:14:17.497 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.498 DEBUG:boto:Data: 2023-10-11T15:14:17.498 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.499 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.499 DEBUG:boto:Params: {} 2023-10-11T15:14:17.500 DEBUG:boto:Token: None 2023-10-11T15:14:17.501 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d18aa94286bbc55-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.508 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.509 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.509 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.510 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.510 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.511 DEBUG:boto:Method: GET 2023-10-11T15:14:17.511 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.512 DEBUG:boto:Data: 2023-10-11T15:14:17.512 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.513 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.514 DEBUG:boto:Params: {} 2023-10-11T15:14:17.514 DEBUG:boto:Token: None 2023-10-11T15:14:17.515 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008050c181ea5f8317-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.521 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.521 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.522 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.522 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.523 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.523 DEBUG:boto:Data: 2023-10-11T15:14:17.524 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.524 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.525 DEBUG:boto:Params: {} 2023-10-11T15:14:17.525 DEBUG:boto:Token: None 2023-10-11T15:14:17.526 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ri1vLeF7+hJCn6F/mm8C9VXyRf8= 2023-10-11T15:14:17.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ri1vLeF7+hJCn6F/mm8C9VXyRf8=', 'Content-Length': '0'} 2023-10-11T15:14:17.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a2cd1e08075110-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.529 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.529 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.530 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.530 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.531 DEBUG:boto:Data: 2023-10-11T15:14:17.531 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.532 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.532 DEBUG:boto:Params: {} 2023-10-11T15:14:17.533 DEBUG:boto:Token: None 2023-10-11T15:14:17.533 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.534 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ri1vLeF7+hJCn6F/mm8C9VXyRf8= 2023-10-11T15:14:17.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ri1vLeF7+hJCn6F/mm8C9VXyRf8=', 'Content-Length': '0'} 2023-10-11T15:14:17.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee092f6801edb25c-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-10-11T15:14:17.537 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.538 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.538 DEBUG:boto:Method: GET 2023-10-11T15:14:17.539 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.539 DEBUG:boto:Data: 2023-10-11T15:14:17.539 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.540 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.541 DEBUG:boto:Params: {} 2023-10-11T15:14:17.541 DEBUG:boto:Token: None 2023-10-11T15:14:17.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RPE5PCirPiMR+Oxi024AK6zjsoo= 2023-10-11T15:14:17.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RPE5PCirPiMR+Oxi024AK6zjsoo=', 'Content-Length': '0'} 2023-10-11T15:14:17.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013164f99ef2f25be-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.545 DEBUG:boto:path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.546 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.546 DEBUG:boto:Method: GET 2023-10-11T15:14:17.547 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.547 DEBUG:boto:Data: 2023-10-11T15:14:17.548 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.549 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.549 DEBUG:boto:Params: {} 2023-10-11T15:14:17.550 DEBUG:boto:Token: None 2023-10-11T15:14:17.550 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000011 2023-10-11T15:14:17.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RPE5PCirPiMR+Oxi024AK6zjsoo= 2023-10-11T15:14:17.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RPE5PCirPiMR+Oxi024AK6zjsoo=', 'Content-Length': '0'} 2023-10-11T15:14:17.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c87537e66d807a3c-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.594 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.595 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.596 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.596 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.596 DEBUG:boto:Method: GET 2023-10-11T15:14:17.597 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.597 DEBUG:boto:Data: 2023-10-11T15:14:17.598 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.598 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.599 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.599 DEBUG:boto:Params: {} 2023-10-11T15:14:17.599 DEBUG:boto:Token: None 2023-10-11T15:14:17.600 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef37d658b8d70697-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.604 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.605 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.605 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.606 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.606 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.607 DEBUG:boto:Method: GET 2023-10-11T15:14:17.607 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000011 2023-10-11T15:14:17.608 DEBUG:boto:Data: 2023-10-11T15:14:17.608 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.608 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.609 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.609 DEBUG:boto:Params: {} 2023-10-11T15:14:17.610 DEBUG:boto:Token: None 2023-10-11T15:14:17.610 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad58ba9583232eb6-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.616 DEBUG:boto:b'dghqvd-58obj000000000000000000111000falseobj000000000000000000112023-10-11T15:14:05.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:14:17.617 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.618 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.618 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.619 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.619 DEBUG:boto:Method: GET 2023-10-11T15:14:17.619 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.620 DEBUG:boto:Data: 2023-10-11T15:14:17.620 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.621 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.621 DEBUG:boto:Params: {} 2023-10-11T15:14:17.622 DEBUG:boto:Token: None 2023-10-11T15:14:17.622 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0= 2023-10-11T15:14:17.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c555c655ef4cb1c3-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.670 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.671 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.671 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.672 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.672 DEBUG:boto:Method: GET 2023-10-11T15:14:17.673 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.673 DEBUG:boto:Data: 2023-10-11T15:14:17.673 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.674 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.675 DEBUG:boto:Params: {} 2023-10-11T15:14:17.675 DEBUG:boto:Token: None 2023-10-11T15:14:17.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0= 2023-10-11T15:14:17.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8674232c4f3fdd-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.680 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.680 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.681 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.681 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.682 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.682 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.682 DEBUG:boto:Data: 2023-10-11T15:14:17.683 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.684 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.684 DEBUG:boto:Params: {} 2023-10-11T15:14:17.685 DEBUG:boto:Token: None 2023-10-11T15:14:17.685 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AdeP0iUqv4ioi1KySey9bugcHWg= 2023-10-11T15:14:17.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AdeP0iUqv4ioi1KySey9bugcHWg=', 'Content-Length': '0'} 2023-10-11T15:14:17.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240bf434896c1cf2-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.689 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.689 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.689 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.690 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.690 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.691 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.691 DEBUG:boto:Data: 2023-10-11T15:14:17.691 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.692 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.692 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.693 DEBUG:boto:Params: {} 2023-10-11T15:14:17.693 DEBUG:boto:Token: None 2023-10-11T15:14:17.693 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.694 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AdeP0iUqv4ioi1KySey9bugcHWg= 2023-10-11T15:14:17.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AdeP0iUqv4ioi1KySey9bugcHWg=', 'Content-Length': '0'} 2023-10-11T15:14:17.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eedadb1a09656aa7-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:14:17.698 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.698 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.698 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.699 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.699 DEBUG:boto:Method: GET 2023-10-11T15:14:17.700 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.700 DEBUG:boto:Data: 2023-10-11T15:14:17.701 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.701 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.702 DEBUG:boto:Params: {} 2023-10-11T15:14:17.702 DEBUG:boto:Token: None 2023-10-11T15:14:17.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0= 2023-10-11T15:14:17.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d1d1d381069e30-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.706 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.709 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.710 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.710 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.711 DEBUG:boto:Method: GET 2023-10-11T15:14:17.711 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.711 DEBUG:boto:Data: 2023-10-11T15:14:17.712 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.713 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.713 DEBUG:boto:Params: {} 2023-10-11T15:14:17.714 DEBUG:boto:Token: None 2023-10-11T15:14:17.714 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012?versionId=null 2023-10-11T15:14:17.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0= 2023-10-11T15:14:17.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z07UupM5MTg/jDBU9sTKAvFqDC0=', 'Content-Length': '0'} 2023-10-11T15:14:17.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098a3f66c0680e754-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.718 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.718 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.719 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.719 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.720 DEBUG:boto:Method: GET 2023-10-11T15:14:17.720 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.720 DEBUG:boto:Data: 2023-10-11T15:14:17.721 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.721 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.722 DEBUG:boto:Params: {} 2023-10-11T15:14:17.722 DEBUG:boto:Token: None 2023-10-11T15:14:17.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b97e61564a5a31f6-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.727 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.728 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.728 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.728 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.729 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.729 DEBUG:boto:Method: GET 2023-10-11T15:14:17.730 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.730 DEBUG:boto:Data: 2023-10-11T15:14:17.731 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.731 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.732 DEBUG:boto:Params: {} 2023-10-11T15:14:17.732 DEBUG:boto:Token: None 2023-10-11T15:14:17.733 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005703d2259ba54f30-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.738 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.739 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.740 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.740 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.740 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.741 DEBUG:boto:Data: 2023-10-11T15:14:17.741 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.742 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.742 DEBUG:boto:Params: {} 2023-10-11T15:14:17.743 DEBUG:boto:Token: None 2023-10-11T15:14:17.743 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rZdVXSjRXgAOf08DRuZIQL/QpuE= 2023-10-11T15:14:17.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rZdVXSjRXgAOf08DRuZIQL/QpuE=', 'Content-Length': '0'} 2023-10-11T15:14:17.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094099bf035958869-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.747 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.747 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.748 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.748 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.749 DEBUG:boto:Data: 2023-10-11T15:14:17.749 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.750 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.750 DEBUG:boto:Params: {} 2023-10-11T15:14:17.751 DEBUG:boto:Token: None 2023-10-11T15:14:17.751 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rZdVXSjRXgAOf08DRuZIQL/QpuE= 2023-10-11T15:14:17.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rZdVXSjRXgAOf08DRuZIQL/QpuE=', 'Content-Length': '0'} 2023-10-11T15:14:17.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007553f46d605a70be-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-10-11T15:14:17.755 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.755 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.756 DEBUG:boto:Method: GET 2023-10-11T15:14:17.756 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.757 DEBUG:boto:Data: 2023-10-11T15:14:17.757 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.758 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.758 DEBUG:boto:Params: {} 2023-10-11T15:14:17.759 DEBUG:boto:Token: None 2023-10-11T15:14:17.759 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LypMDXz6E3fHjgSmlWbuuW00kDc= 2023-10-11T15:14:17.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LypMDXz6E3fHjgSmlWbuuW00kDc=', 'Content-Length': '0'} 2023-10-11T15:14:17.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004958e3ed30557b56-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.810 DEBUG:boto:path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.811 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.811 DEBUG:boto:Method: GET 2023-10-11T15:14:17.812 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.812 DEBUG:boto:Data: 2023-10-11T15:14:17.813 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.813 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.814 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.814 DEBUG:boto:Params: {} 2023-10-11T15:14:17.815 DEBUG:boto:Token: None 2023-10-11T15:14:17.815 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000012 2023-10-11T15:14:17.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LypMDXz6E3fHjgSmlWbuuW00kDc= 2023-10-11T15:14:17.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LypMDXz6E3fHjgSmlWbuuW00kDc=', 'Content-Length': '0'} 2023-10-11T15:14:17.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d303aca6606c8490-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.819 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.819 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.820 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.820 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.820 DEBUG:boto:Method: GET 2023-10-11T15:14:17.821 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.821 DEBUG:boto:Data: 2023-10-11T15:14:17.822 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.822 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.822 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.823 DEBUG:boto:Params: {} 2023-10-11T15:14:17.823 DEBUG:boto:Token: None 2023-10-11T15:14:17.824 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.824 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009af31fddb3983aa7-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.828 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.828 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.829 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.829 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.830 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.830 DEBUG:boto:Method: GET 2023-10-11T15:14:17.831 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000012 2023-10-11T15:14:17.831 DEBUG:boto:Data: 2023-10-11T15:14:17.831 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.832 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.833 DEBUG:boto:Params: {} 2023-10-11T15:14:17.833 DEBUG:boto:Token: None 2023-10-11T15:14:17.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000674087425ddf913a-006526bbc9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.839 DEBUG:boto:b'dghqvd-58obj000000000000000000121000falseobj000000000000000000122023-10-11T15:14:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:17.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:14:17.841 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.841 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.842 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.842 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.842 DEBUG:boto:Method: GET 2023-10-11T15:14:17.843 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.843 DEBUG:boto:Data: 2023-10-11T15:14:17.844 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.845 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.845 DEBUG:boto:Params: {} 2023-10-11T15:14:17.846 DEBUG:boto:Token: None 2023-10-11T15:14:17.846 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o= 2023-10-11T15:14:17.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o=', 'Content-Length': '0'} 2023-10-11T15:14:17.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aeaafe23fd7ca2c-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.850 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.850 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.851 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.851 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.852 DEBUG:boto:Method: GET 2023-10-11T15:14:17.852 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.852 DEBUG:boto:Data: 2023-10-11T15:14:17.853 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.854 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.854 DEBUG:boto:Params: {} 2023-10-11T15:14:17.855 DEBUG:boto:Token: None 2023-10-11T15:14:17.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o= 2023-10-11T15:14:17.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o=', 'Content-Length': '0'} 2023-10-11T15:14:17.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba01fca0c8f4507-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.907 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.907 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.908 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.908 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.909 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.909 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.910 DEBUG:boto:Data: 2023-10-11T15:14:17.910 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.911 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.911 DEBUG:boto:Params: {} 2023-10-11T15:14:17.912 DEBUG:boto:Token: None 2023-10-11T15:14:17.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AYfUgrOdGNb4GXbbnWQga4mjLRI= 2023-10-11T15:14:17.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AYfUgrOdGNb4GXbbnWQga4mjLRI=', 'Content-Length': '0'} 2023-10-11T15:14:17.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343d7c68106fe6f1-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.921 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.921 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.921 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.922 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.922 DEBUG:boto:Method: HEAD 2023-10-11T15:14:17.923 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.923 DEBUG:boto:Data: 2023-10-11T15:14:17.923 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.924 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.925 DEBUG:boto:Params: {} 2023-10-11T15:14:17.925 DEBUG:boto:Token: None 2023-10-11T15:14:17.926 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AYfUgrOdGNb4GXbbnWQga4mjLRI= 2023-10-11T15:14:17.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AYfUgrOdGNb4GXbbnWQga4mjLRI=', 'Content-Length': '0'} 2023-10-11T15:14:17.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac80dd81cea2734-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:14:17.931 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.931 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.932 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.932 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.932 DEBUG:boto:Method: GET 2023-10-11T15:14:17.933 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.933 DEBUG:boto:Data: 2023-10-11T15:14:17.934 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.935 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.935 DEBUG:boto:Params: {} 2023-10-11T15:14:17.936 DEBUG:boto:Token: None 2023-10-11T15:14:17.936 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o= 2023-10-11T15:14:17.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o=', 'Content-Length': '0'} 2023-10-11T15:14:17.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e99429b2563e905-006526bbc9-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.940 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.940 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:17.941 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.941 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.941 DEBUG:boto:Method: GET 2023-10-11T15:14:17.942 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.942 DEBUG:boto:Data: 2023-10-11T15:14:17.943 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:17.944 DEBUG:boto:Port: 8000 2023-10-11T15:14:17.944 DEBUG:boto:Params: {} 2023-10-11T15:14:17.945 DEBUG:boto:Token: None 2023-10-11T15:14:17.945 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/obj00000000000000000013?versionId=null 2023-10-11T15:14:17.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o= 2023-10-11T15:14:17.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTB3GxIq+lEZhkWQ8kS1WhZbv2o=', 'Content-Length': '0'} 2023-10-11T15:14:17.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e047f7c6a89beb11-006526bbc9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:17.950 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:17.951 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:17.951 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:17.952 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:17.952 DEBUG:boto:Method: GET 2023-10-11T15:14:17.953 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:17.953 DEBUG:boto:Data: 2023-10-11T15:14:17.954 DEBUG:boto:Headers: {} 2023-10-11T15:14:17.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:17.954 DEBUG:boto:Port: 8001 2023-10-11T15:14:17.955 DEBUG:boto:Params: {} 2023-10-11T15:14:17.955 DEBUG:boto:Token: None 2023-10-11T15:14:17.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:17 GMT /dghqvd-58/ 2023-10-11T15:14:17.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o= 2023-10-11T15:14:17.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QQCUq2Fopk26wyt1wYTxvpUd74o=', 'Content-Length': '0'} 2023-10-11T15:14:17.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa9194dfcd6009b-006526bbc9-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.002 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.003 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.004 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.004 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.005 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.005 DEBUG:boto:Method: GET 2023-10-11T15:14:18.006 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.006 DEBUG:boto:Data: 2023-10-11T15:14:18.007 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.007 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.008 DEBUG:boto:Params: {} 2023-10-11T15:14:18.008 DEBUG:boto:Token: None 2023-10-11T15:14:18.009 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c70c4f10d9bab0-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.015 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.015 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.016 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.016 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.017 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.017 DEBUG:boto:Data: 2023-10-11T15:14:18.017 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.018 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.019 DEBUG:boto:Params: {} 2023-10-11T15:14:18.019 DEBUG:boto:Token: None 2023-10-11T15:14:18.020 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0LZV1nUMMYmz21HB7K6yGSCQNo8= 2023-10-11T15:14:18.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0LZV1nUMMYmz21HB7K6yGSCQNo8=', 'Content-Length': '0'} 2023-10-11T15:14:18.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095abfc8545f1b428-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.023 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.024 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.024 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.025 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.025 DEBUG:boto:Data: 2023-10-11T15:14:18.025 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.026 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.027 DEBUG:boto:Params: {} 2023-10-11T15:14:18.027 DEBUG:boto:Token: None 2023-10-11T15:14:18.027 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.028 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0LZV1nUMMYmz21HB7K6yGSCQNo8= 2023-10-11T15:14:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0LZV1nUMMYmz21HB7K6yGSCQNo8=', 'Content-Length': '0'} 2023-10-11T15:14:18.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520d2d9a762698eb-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-10-11T15:14:18.031 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.032 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.032 DEBUG:boto:Method: GET 2023-10-11T15:14:18.033 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.033 DEBUG:boto:Data: 2023-10-11T15:14:18.034 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.034 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.035 DEBUG:boto:Params: {} 2023-10-11T15:14:18.035 DEBUG:boto:Token: None 2023-10-11T15:14:18.036 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uyjlNHH6XKFBPDgwBi6j677wfr8= 2023-10-11T15:14:18.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uyjlNHH6XKFBPDgwBi6j677wfr8=', 'Content-Length': '0'} 2023-10-11T15:14:18.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6e714a39e757ee-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.039 DEBUG:boto:path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.040 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.040 DEBUG:boto:Method: GET 2023-10-11T15:14:18.041 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.041 DEBUG:boto:Data: 2023-10-11T15:14:18.042 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.042 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.043 DEBUG:boto:Params: {} 2023-10-11T15:14:18.043 DEBUG:boto:Token: None 2023-10-11T15:14:18.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000013 2023-10-11T15:14:18.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uyjlNHH6XKFBPDgwBi6j677wfr8= 2023-10-11T15:14:18.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uyjlNHH6XKFBPDgwBi6j677wfr8=', 'Content-Length': '0'} 2023-10-11T15:14:18.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b5d19a84d35f60-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.090 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.091 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.091 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.092 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.092 DEBUG:boto:Method: GET 2023-10-11T15:14:18.093 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.093 DEBUG:boto:Data: 2023-10-11T15:14:18.094 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.094 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.095 DEBUG:boto:Params: {} 2023-10-11T15:14:18.095 DEBUG:boto:Token: None 2023-10-11T15:14:18.096 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f90c897567faae0f-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.100 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.101 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.101 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.102 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.102 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.102 DEBUG:boto:Method: GET 2023-10-11T15:14:18.103 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000013 2023-10-11T15:14:18.103 DEBUG:boto:Data: 2023-10-11T15:14:18.104 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.105 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.105 DEBUG:boto:Params: {} 2023-10-11T15:14:18.105 DEBUG:boto:Token: None 2023-10-11T15:14:18.106 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f628fe28253b46-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.112 DEBUG:boto:b'dghqvd-58obj000000000000000000131000falseobj000000000000000000132023-10-11T15:14:04.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-11T15:14:18.113 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.113 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.114 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.114 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.115 DEBUG:boto:Method: GET 2023-10-11T15:14:18.115 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.115 DEBUG:boto:Data: 2023-10-11T15:14:18.116 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.117 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.117 DEBUG:boto:Params: {} 2023-10-11T15:14:18.118 DEBUG:boto:Token: None 2023-10-11T15:14:18.118 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw= 2023-10-11T15:14:18.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw=', 'Content-Length': '0'} 2023-10-11T15:14:18.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b6f5c84ea8a31d4-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.162 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.163 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.164 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.164 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.164 DEBUG:boto:Method: GET 2023-10-11T15:14:18.165 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.165 DEBUG:boto:Data: 2023-10-11T15:14:18.166 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.166 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.167 DEBUG:boto:Params: {} 2023-10-11T15:14:18.167 DEBUG:boto:Token: None 2023-10-11T15:14:18.168 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw= 2023-10-11T15:14:18.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw=', 'Content-Length': '0'} 2023-10-11T15:14:18.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c365db509aaeefaf-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.172 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.173 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.173 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.173 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.174 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.174 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.175 DEBUG:boto:Data: 2023-10-11T15:14:18.175 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.176 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.176 DEBUG:boto:Params: {} 2023-10-11T15:14:18.177 DEBUG:boto:Token: None 2023-10-11T15:14:18.177 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8KaABJrifGn+NYtSVy4WQy5l0ns= 2023-10-11T15:14:18.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8KaABJrifGn+NYtSVy4WQy5l0ns=', 'Content-Length': '0'} 2023-10-11T15:14:18.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001049691a4603d40-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.181 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.181 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.181 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.182 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.182 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.183 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.183 DEBUG:boto:Data: 2023-10-11T15:14:18.183 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.184 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.184 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.185 DEBUG:boto:Params: {} 2023-10-11T15:14:18.185 DEBUG:boto:Token: None 2023-10-11T15:14:18.186 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.186 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8KaABJrifGn+NYtSVy4WQy5l0ns= 2023-10-11T15:14:18.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8KaABJrifGn+NYtSVy4WQy5l0ns=', 'Content-Length': '0'} 2023-10-11T15:14:18.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000988ab74313e36f5-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-11T15:14:18.190 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.190 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.191 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.191 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.192 DEBUG:boto:Method: GET 2023-10-11T15:14:18.192 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.192 DEBUG:boto:Data: 2023-10-11T15:14:18.193 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.194 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.194 DEBUG:boto:Params: {} 2023-10-11T15:14:18.195 DEBUG:boto:Token: None 2023-10-11T15:14:18.195 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw= 2023-10-11T15:14:18.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw=', 'Content-Length': '0'} 2023-10-11T15:14:18.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454c04aaee5e281b-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.199 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.199 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.200 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.200 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.201 DEBUG:boto:Method: GET 2023-10-11T15:14:18.201 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.202 DEBUG:boto:Data: 2023-10-11T15:14:18.202 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.203 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.203 DEBUG:boto:Params: {} 2023-10-11T15:14:18.204 DEBUG:boto:Token: None 2023-10-11T15:14:18.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014?versionId=null 2023-10-11T15:14:18.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw= 2023-10-11T15:14:18.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fZtYIUUSSMlsDwe7dJ+3hmzXKAw=', 'Content-Length': '0'} 2023-10-11T15:14:18.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae7b82f7a3c3b89-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.208 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.208 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.209 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.209 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.210 DEBUG:boto:Method: GET 2023-10-11T15:14:18.210 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.211 DEBUG:boto:Data: 2023-10-11T15:14:18.211 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.212 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.212 DEBUG:boto:Params: {} 2023-10-11T15:14:18.213 DEBUG:boto:Token: None 2023-10-11T15:14:18.213 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.214 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c6793d796b4867-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.217 DEBUG:boto:b'dghqvd-58obj000000000000000000141000falseobj000000000000000000142023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.218 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.218 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.219 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.219 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.220 DEBUG:boto:Method: GET 2023-10-11T15:14:18.220 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.220 DEBUG:boto:Data: 2023-10-11T15:14:18.221 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.222 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.222 DEBUG:boto:Params: {} 2023-10-11T15:14:18.223 DEBUG:boto:Token: None 2023-10-11T15:14:18.223 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4f9a39a6ee5172a-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.270 DEBUG:boto:b'dghqvd-58obj000000000000000000141000falseobj000000000000000000142023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.271 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.272 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.272 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.273 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.273 DEBUG:boto:Data: 2023-10-11T15:14:18.273 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.274 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.275 DEBUG:boto:Params: {} 2023-10-11T15:14:18.275 DEBUG:boto:Token: None 2023-10-11T15:14:18.276 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.276 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XRxkjNG8Hv+6p/2TWGzgCzrvdJQ= 2023-10-11T15:14:18.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XRxkjNG8Hv+6p/2TWGzgCzrvdJQ=', 'Content-Length': '0'} 2023-10-11T15:14:18.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d6be9e9b9e4bb78-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.279 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.280 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.280 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.280 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.281 DEBUG:boto:Data: 2023-10-11T15:14:18.281 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.282 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.282 DEBUG:boto:Params: {} 2023-10-11T15:14:18.283 DEBUG:boto:Token: None 2023-10-11T15:14:18.283 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XRxkjNG8Hv+6p/2TWGzgCzrvdJQ= 2023-10-11T15:14:18.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XRxkjNG8Hv+6p/2TWGzgCzrvdJQ=', 'Content-Length': '0'} 2023-10-11T15:14:18.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20a0974d9c102c1-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-10-11T15:14:18.287 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.288 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.288 DEBUG:boto:Method: GET 2023-10-11T15:14:18.289 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.289 DEBUG:boto:Data: 2023-10-11T15:14:18.290 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.290 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.291 DEBUG:boto:Params: {} 2023-10-11T15:14:18.291 DEBUG:boto:Token: None 2023-10-11T15:14:18.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PELyeUfKVKCmKeR9PfZ561LM8qg= 2023-10-11T15:14:18.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PELyeUfKVKCmKeR9PfZ561LM8qg=', 'Content-Length': '0'} 2023-10-11T15:14:18.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda0f8d16fde94e1-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.342 DEBUG:boto:path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.343 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.343 DEBUG:boto:Method: GET 2023-10-11T15:14:18.344 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.344 DEBUG:boto:Data: 2023-10-11T15:14:18.345 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.345 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.346 DEBUG:boto:Params: {} 2023-10-11T15:14:18.346 DEBUG:boto:Token: None 2023-10-11T15:14:18.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000014 2023-10-11T15:14:18.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PELyeUfKVKCmKeR9PfZ561LM8qg= 2023-10-11T15:14:18.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PELyeUfKVKCmKeR9PfZ561LM8qg=', 'Content-Length': '0'} 2023-10-11T15:14:18.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4fe84210f5aa17-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.350 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.351 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.351 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.352 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.352 DEBUG:boto:Method: GET 2023-10-11T15:14:18.353 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.353 DEBUG:boto:Data: 2023-10-11T15:14:18.353 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.354 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.355 DEBUG:boto:Params: {} 2023-10-11T15:14:18.355 DEBUG:boto:Token: None 2023-10-11T15:14:18.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000886bbd8591227090-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.360 DEBUG:boto:b'dghqvd-58obj000000000000000000141000falseobj000000000000000000142023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.361 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.361 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.362 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.362 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.362 DEBUG:boto:Method: GET 2023-10-11T15:14:18.363 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000014 2023-10-11T15:14:18.363 DEBUG:boto:Data: 2023-10-11T15:14:18.364 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.364 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.365 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.365 DEBUG:boto:Params: {} 2023-10-11T15:14:18.365 DEBUG:boto:Token: None 2023-10-11T15:14:18.366 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000179f00f5b17482fd-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.371 DEBUG:boto:b'dghqvd-58obj000000000000000000141000falseobj000000000000000000142023-10-11T15:14:05.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-11T15:14:18.372 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.373 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.373 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.374 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.374 DEBUG:boto:Method: GET 2023-10-11T15:14:18.374 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.375 DEBUG:boto:Data: 2023-10-11T15:14:18.375 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.376 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.376 DEBUG:boto:Params: {} 2023-10-11T15:14:18.377 DEBUG:boto:Token: None 2023-10-11T15:14:18.377 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck= 2023-10-11T15:14:18.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck=', 'Content-Length': '0'} 2023-10-11T15:14:18.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff71511f534869c2-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.381 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.381 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.382 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.382 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.383 DEBUG:boto:Method: GET 2023-10-11T15:14:18.383 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.383 DEBUG:boto:Data: 2023-10-11T15:14:18.384 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.385 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.385 DEBUG:boto:Params: {} 2023-10-11T15:14:18.385 DEBUG:boto:Token: None 2023-10-11T15:14:18.386 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck= 2023-10-11T15:14:18.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck=', 'Content-Length': '0'} 2023-10-11T15:14:18.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b0c844e797f873-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.389 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.390 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.390 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.391 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.391 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.392 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.392 DEBUG:boto:Data: 2023-10-11T15:14:18.392 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.393 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.393 DEBUG:boto:Params: {} 2023-10-11T15:14:18.394 DEBUG:boto:Token: None 2023-10-11T15:14:18.394 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2P/q+Wi7+5We8IrWxACsPuWIYGM= 2023-10-11T15:14:18.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2P/q+Wi7+5We8IrWxACsPuWIYGM=', 'Content-Length': '0'} 2023-10-11T15:14:18.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d92642f1595b4a-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.398 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.398 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.399 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.399 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.399 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.400 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.400 DEBUG:boto:Data: 2023-10-11T15:14:18.401 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.401 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.402 DEBUG:boto:Params: {} 2023-10-11T15:14:18.402 DEBUG:boto:Token: None 2023-10-11T15:14:18.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2P/q+Wi7+5We8IrWxACsPuWIYGM= 2023-10-11T15:14:18.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2P/q+Wi7+5We8IrWxACsPuWIYGM=', 'Content-Length': '0'} 2023-10-11T15:14:18.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a458efe1048a233-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-11T15:14:18.406 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.407 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.407 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.408 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.408 DEBUG:boto:Method: GET 2023-10-11T15:14:18.409 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.409 DEBUG:boto:Data: 2023-10-11T15:14:18.409 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.410 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.411 DEBUG:boto:Params: {} 2023-10-11T15:14:18.411 DEBUG:boto:Token: None 2023-10-11T15:14:18.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck= 2023-10-11T15:14:18.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck=', 'Content-Length': '0'} 2023-10-11T15:14:18.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178819d0f28dc333-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.415 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.415 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.416 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.416 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.417 DEBUG:boto:Method: GET 2023-10-11T15:14:18.417 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.418 DEBUG:boto:Data: 2023-10-11T15:14:18.418 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.419 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.419 DEBUG:boto:Params: {} 2023-10-11T15:14:18.420 DEBUG:boto:Token: None 2023-10-11T15:14:18.420 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015?versionId=null 2023-10-11T15:14:18.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck= 2023-10-11T15:14:18.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4IFrtmG7LtDKwhJ13FvDPk+E/ck=', 'Content-Length': '0'} 2023-10-11T15:14:18.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012aff4e7f12fd7e1-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.424 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.424 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.425 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.425 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.425 DEBUG:boto:Method: GET 2023-10-11T15:14:18.426 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.426 DEBUG:boto:Data: 2023-10-11T15:14:18.427 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.427 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.428 DEBUG:boto:Params: {} 2023-10-11T15:14:18.428 DEBUG:boto:Token: None 2023-10-11T15:14:18.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096e2955f8ff1726a-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.474 DEBUG:boto:b'dghqvd-58obj000000000000000000151000falseobj000000000000000000152023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.475 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.476 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.476 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.477 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.477 DEBUG:boto:Method: GET 2023-10-11T15:14:18.477 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.478 DEBUG:boto:Data: 2023-10-11T15:14:18.478 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.480 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.481 DEBUG:boto:Params: {} 2023-10-11T15:14:18.482 DEBUG:boto:Token: None 2023-10-11T15:14:18.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003d1c41571fafae-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.488 DEBUG:boto:b'dghqvd-58obj000000000000000000151000falseobj000000000000000000152023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.489 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.490 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.490 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.491 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.492 DEBUG:boto:Data: 2023-10-11T15:14:18.492 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.493 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.493 DEBUG:boto:Params: {} 2023-10-11T15:14:18.494 DEBUG:boto:Token: None 2023-10-11T15:14:18.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UH9pPdffBopic6DMhgfFGe/Jl0I= 2023-10-11T15:14:18.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UH9pPdffBopic6DMhgfFGe/Jl0I=', 'Content-Length': '0'} 2023-10-11T15:14:18.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87a83cec4e07560-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.498 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.499 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.499 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.499 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.500 DEBUG:boto:Data: 2023-10-11T15:14:18.500 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.501 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.502 DEBUG:boto:Params: {} 2023-10-11T15:14:18.502 DEBUG:boto:Token: None 2023-10-11T15:14:18.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.503 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UH9pPdffBopic6DMhgfFGe/Jl0I= 2023-10-11T15:14:18.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UH9pPdffBopic6DMhgfFGe/Jl0I=', 'Content-Length': '0'} 2023-10-11T15:14:18.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113800f6a2aebe10-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-10-11T15:14:18.507 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.507 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.508 DEBUG:boto:Method: GET 2023-10-11T15:14:18.509 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.509 DEBUG:boto:Data: 2023-10-11T15:14:18.509 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.511 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.511 DEBUG:boto:Params: {} 2023-10-11T15:14:18.511 DEBUG:boto:Token: None 2023-10-11T15:14:18.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YEdeMyUUwoWWSDc155OOiMkkF/Y= 2023-10-11T15:14:18.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YEdeMyUUwoWWSDc155OOiMkkF/Y=', 'Content-Length': '0'} 2023-10-11T15:14:18.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5a74b8efe2c609-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.516 DEBUG:boto:path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.516 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.517 DEBUG:boto:Method: GET 2023-10-11T15:14:18.517 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.518 DEBUG:boto:Data: 2023-10-11T15:14:18.518 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.519 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.519 DEBUG:boto:Params: {} 2023-10-11T15:14:18.520 DEBUG:boto:Token: None 2023-10-11T15:14:18.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000015 2023-10-11T15:14:18.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YEdeMyUUwoWWSDc155OOiMkkF/Y= 2023-10-11T15:14:18.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YEdeMyUUwoWWSDc155OOiMkkF/Y=', 'Content-Length': '0'} 2023-10-11T15:14:18.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009697d162380a53fc-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.570 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.571 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.571 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.572 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.572 DEBUG:boto:Method: GET 2023-10-11T15:14:18.573 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.573 DEBUG:boto:Data: 2023-10-11T15:14:18.574 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.575 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.576 DEBUG:boto:Params: {} 2023-10-11T15:14:18.576 DEBUG:boto:Token: None 2023-10-11T15:14:18.577 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca7eb27dddf9729e-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.581 DEBUG:boto:b'dghqvd-58obj000000000000000000151000falseobj000000000000000000152023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.582 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.582 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.583 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.583 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.584 DEBUG:boto:Method: GET 2023-10-11T15:14:18.584 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000015 2023-10-11T15:14:18.585 DEBUG:boto:Data: 2023-10-11T15:14:18.585 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.586 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.587 DEBUG:boto:Params: {} 2023-10-11T15:14:18.587 DEBUG:boto:Token: None 2023-10-11T15:14:18.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000813e5fe3c71b725f-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.593 DEBUG:boto:b'dghqvd-58obj000000000000000000151000falseobj000000000000000000152023-10-11T15:14:05.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-11T15:14:18.595 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.595 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.596 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.596 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.597 DEBUG:boto:Method: GET 2023-10-11T15:14:18.597 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.598 DEBUG:boto:Data: 2023-10-11T15:14:18.598 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.599 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.600 DEBUG:boto:Params: {} 2023-10-11T15:14:18.600 DEBUG:boto:Token: None 2023-10-11T15:14:18.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk= 2023-10-11T15:14:18.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk=', 'Content-Length': '0'} 2023-10-11T15:14:18.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f280eb08123b75-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.605 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.606 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.606 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.607 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.607 DEBUG:boto:Method: GET 2023-10-11T15:14:18.608 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.608 DEBUG:boto:Data: 2023-10-11T15:14:18.609 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.609 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.610 DEBUG:boto:Params: {} 2023-10-11T15:14:18.610 DEBUG:boto:Token: None 2023-10-11T15:14:18.611 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.611 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk= 2023-10-11T15:14:18.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk=', 'Content-Length': '0'} 2023-10-11T15:14:18.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de706195ce93d61-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.615 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.615 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.615 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.616 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.616 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.617 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.617 DEBUG:boto:Data: 2023-10-11T15:14:18.618 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.620 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.620 DEBUG:boto:Params: {} 2023-10-11T15:14:18.620 DEBUG:boto:Token: None 2023-10-11T15:14:18.621 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ljBCdmDfY5oHM4kpUz5B2bwG+yg= 2023-10-11T15:14:18.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ljBCdmDfY5oHM4kpUz5B2bwG+yg=', 'Content-Length': '0'} 2023-10-11T15:14:18.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab004336107acc33-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.624 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.625 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.625 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.626 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.626 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.626 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.627 DEBUG:boto:Data: 2023-10-11T15:14:18.627 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.628 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.628 DEBUG:boto:Params: {} 2023-10-11T15:14:18.629 DEBUG:boto:Token: None 2023-10-11T15:14:18.629 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ljBCdmDfY5oHM4kpUz5B2bwG+yg= 2023-10-11T15:14:18.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ljBCdmDfY5oHM4kpUz5B2bwG+yg=', 'Content-Length': '0'} 2023-10-11T15:14:18.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db59fb7873d7eda-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-11T15:14:18.634 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.634 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.635 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.635 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.636 DEBUG:boto:Method: GET 2023-10-11T15:14:18.636 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.637 DEBUG:boto:Data: 2023-10-11T15:14:18.637 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.638 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.638 DEBUG:boto:Params: {} 2023-10-11T15:14:18.639 DEBUG:boto:Token: None 2023-10-11T15:14:18.639 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk= 2023-10-11T15:14:18.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk=', 'Content-Length': '0'} 2023-10-11T15:14:18.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8850a1f6aa6dca8-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.643 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.643 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.644 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.644 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.645 DEBUG:boto:Method: GET 2023-10-11T15:14:18.645 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.646 DEBUG:boto:Data: 2023-10-11T15:14:18.646 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.647 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.647 DEBUG:boto:Params: {} 2023-10-11T15:14:18.648 DEBUG:boto:Token: None 2023-10-11T15:14:18.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016?versionId=null 2023-10-11T15:14:18.649 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk= 2023-10-11T15:14:18.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fJMIPW9pu9YhC0vMTL0g5QcI5Lk=', 'Content-Length': '0'} 2023-10-11T15:14:18.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e116a10824f55d54-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.652 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.653 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.653 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.653 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.654 DEBUG:boto:Method: GET 2023-10-11T15:14:18.654 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.655 DEBUG:boto:Data: 2023-10-11T15:14:18.655 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.656 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.656 DEBUG:boto:Params: {} 2023-10-11T15:14:18.657 DEBUG:boto:Token: None 2023-10-11T15:14:18.657 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfc9f43668969a5-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.662 DEBUG:boto:b'dghqvd-58obj000000000000000000161000falseobj000000000000000000162023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.662 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.663 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.663 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.664 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.664 DEBUG:boto:Method: GET 2023-10-11T15:14:18.664 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.665 DEBUG:boto:Data: 2023-10-11T15:14:18.666 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.667 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.667 DEBUG:boto:Params: {} 2023-10-11T15:14:18.667 DEBUG:boto:Token: None 2023-10-11T15:14:18.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004688ed94f5293b04-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.714 DEBUG:boto:b'dghqvd-58obj000000000000000000161000falseobj000000000000000000162023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.716 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.716 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.717 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.718 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.718 DEBUG:boto:Data: 2023-10-11T15:14:18.719 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.721 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.721 DEBUG:boto:Params: {} 2023-10-11T15:14:18.722 DEBUG:boto:Token: None 2023-10-11T15:14:18.723 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1Vp5brzbaA1Tc49ZvcBnWQJiSg= 2023-10-11T15:14:18.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1Vp5brzbaA1Tc49ZvcBnWQJiSg=', 'Content-Length': '0'} 2023-10-11T15:14:18.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1b141f2f216724-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.727 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.727 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.728 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.728 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.729 DEBUG:boto:Data: 2023-10-11T15:14:18.729 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.730 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.730 DEBUG:boto:Params: {} 2023-10-11T15:14:18.731 DEBUG:boto:Token: None 2023-10-11T15:14:18.731 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1Vp5brzbaA1Tc49ZvcBnWQJiSg= 2023-10-11T15:14:18.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1Vp5brzbaA1Tc49ZvcBnWQJiSg=', 'Content-Length': '0'} 2023-10-11T15:14:18.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003270ed5cde1beaeb-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-10-11T15:14:18.739 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.740 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.741 DEBUG:boto:Method: GET 2023-10-11T15:14:18.741 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.742 DEBUG:boto:Data: 2023-10-11T15:14:18.743 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.744 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.745 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.745 DEBUG:boto:Params: {} 2023-10-11T15:14:18.746 DEBUG:boto:Token: None 2023-10-11T15:14:18.747 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HopacTcCusTyEp2Zgbr/sBk6psM= 2023-10-11T15:14:18.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HopacTcCusTyEp2Zgbr/sBk6psM=', 'Content-Length': '0'} 2023-10-11T15:14:18.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5eec95048300ac-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.794 DEBUG:boto:path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.795 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.795 DEBUG:boto:Method: GET 2023-10-11T15:14:18.796 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.796 DEBUG:boto:Data: 2023-10-11T15:14:18.797 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.798 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.798 DEBUG:boto:Params: {} 2023-10-11T15:14:18.799 DEBUG:boto:Token: None 2023-10-11T15:14:18.799 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000016 2023-10-11T15:14:18.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HopacTcCusTyEp2Zgbr/sBk6psM= 2023-10-11T15:14:18.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HopacTcCusTyEp2Zgbr/sBk6psM=', 'Content-Length': '0'} 2023-10-11T15:14:18.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc402a358029065f-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.803 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.804 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.805 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.805 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.806 DEBUG:boto:Method: GET 2023-10-11T15:14:18.807 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.808 DEBUG:boto:Data: 2023-10-11T15:14:18.809 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.810 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.811 DEBUG:boto:Params: {} 2023-10-11T15:14:18.812 DEBUG:boto:Token: None 2023-10-11T15:14:18.813 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77d9e9b527c34da-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.818 DEBUG:boto:b'dghqvd-58obj000000000000000000161000falseobj000000000000000000162023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.820 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.820 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.821 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.822 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.823 DEBUG:boto:Method: GET 2023-10-11T15:14:18.823 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000016 2023-10-11T15:14:18.824 DEBUG:boto:Data: 2023-10-11T15:14:18.824 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.826 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.826 DEBUG:boto:Params: {} 2023-10-11T15:14:18.827 DEBUG:boto:Token: None 2023-10-11T15:14:18.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea258db9e664e813-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.835 DEBUG:boto:b'dghqvd-58obj000000000000000000161000falseobj000000000000000000162023-10-11T15:14:04.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-11T15:14:18.836 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.837 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.837 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.838 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.838 DEBUG:boto:Method: GET 2023-10-11T15:14:18.839 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.839 DEBUG:boto:Data: 2023-10-11T15:14:18.840 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.840 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.840 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.841 DEBUG:boto:Params: {} 2023-10-11T15:14:18.841 DEBUG:boto:Token: None 2023-10-11T15:14:18.842 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM= 2023-10-11T15:14:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM=', 'Content-Length': '0'} 2023-10-11T15:14:18.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d010c741bad38357-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.849 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.849 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.850 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.851 DEBUG:boto:Method: GET 2023-10-11T15:14:18.852 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.852 DEBUG:boto:Data: 2023-10-11T15:14:18.853 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.855 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.855 DEBUG:boto:Params: {} 2023-10-11T15:14:18.856 DEBUG:boto:Token: None 2023-10-11T15:14:18.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM= 2023-10-11T15:14:18.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM=', 'Content-Length': '0'} 2023-10-11T15:14:18.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b175de4f1cd7730d-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.902 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.903 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.903 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.904 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.904 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.905 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.905 DEBUG:boto:Data: 2023-10-11T15:14:18.905 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.906 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.907 DEBUG:boto:Params: {} 2023-10-11T15:14:18.907 DEBUG:boto:Token: None 2023-10-11T15:14:18.908 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5OvGIqEH5QJstgSui9Ue4uSZ1eg= 2023-10-11T15:14:18.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5OvGIqEH5QJstgSui9Ue4uSZ1eg=', 'Content-Length': '0'} 2023-10-11T15:14:18.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28af5e117ffe9f4-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.913 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.914 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.915 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.916 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.916 DEBUG:boto:Method: HEAD 2023-10-11T15:14:18.917 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.918 DEBUG:boto:Data: 2023-10-11T15:14:18.918 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.920 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.921 DEBUG:boto:Params: {} 2023-10-11T15:14:18.921 DEBUG:boto:Token: None 2023-10-11T15:14:18.922 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5OvGIqEH5QJstgSui9Ue4uSZ1eg= 2023-10-11T15:14:18.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5OvGIqEH5QJstgSui9Ue4uSZ1eg=', 'Content-Length': '0'} 2023-10-11T15:14:18.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f0d8728063e106-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-11T15:14:18.928 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.928 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.929 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.930 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.931 DEBUG:boto:Method: GET 2023-10-11T15:14:18.931 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.932 DEBUG:boto:Data: 2023-10-11T15:14:18.933 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.934 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.935 DEBUG:boto:Params: {} 2023-10-11T15:14:18.936 DEBUG:boto:Token: None 2023-10-11T15:14:18.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM= 2023-10-11T15:14:18.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM=', 'Content-Length': '0'} 2023-10-11T15:14:18.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453b0d6bbaf1312e-006526bbca-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.941 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.941 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:18.942 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.942 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.943 DEBUG:boto:Method: GET 2023-10-11T15:14:18.943 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.945 DEBUG:boto:Data: 2023-10-11T15:14:18.946 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.947 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.948 DEBUG:boto:Params: {} 2023-10-11T15:14:18.948 DEBUG:boto:Token: None 2023-10-11T15:14:18.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/obj00000000000000000017?versionId=null 2023-10-11T15:14:18.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM= 2023-10-11T15:14:18.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pY5JXGVQ68o1jqyofOplLCXhoeM=', 'Content-Length': '0'} 2023-10-11T15:14:18.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b79c65c19b4a37f5-006526bbca-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.955 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.956 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.957 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.958 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.958 DEBUG:boto:Method: GET 2023-10-11T15:14:18.959 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.960 DEBUG:boto:Data: 2023-10-11T15:14:18.961 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:18.962 DEBUG:boto:Port: 8001 2023-10-11T15:14:18.963 DEBUG:boto:Params: {} 2023-10-11T15:14:18.964 DEBUG:boto:Token: None 2023-10-11T15:14:18.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009634630eb9d7b4f3-006526bbca-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:18.970 DEBUG:boto:b'dghqvd-58obj000000000000000000171000falseobj000000000000000000172023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:18.971 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:18.972 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:18.973 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.974 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.974 DEBUG:boto:Method: GET 2023-10-11T15:14:18.975 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:18.976 DEBUG:boto:Data: 2023-10-11T15:14:18.976 DEBUG:boto:Headers: {} 2023-10-11T15:14:18.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:18.977 DEBUG:boto:Port: 8000 2023-10-11T15:14:18.977 DEBUG:boto:Params: {} 2023-10-11T15:14:18.978 DEBUG:boto:Token: None 2023-10-11T15:14:18.978 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:18 GMT /dghqvd-58/ 2023-10-11T15:14:18.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc= 2023-10-11T15:14:18.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eSi3utBfUO3AjqBlXWNYcCK93Sc=', 'Content-Length': '0'} 2023-10-11T15:14:18.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000389674b9d939a91b-006526bbca-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.026 DEBUG:boto:b'dghqvd-58obj000000000000000000171000falseobj000000000000000000172023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.027 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.028 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.028 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.028 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.029 DEBUG:boto:Data: 2023-10-11T15:14:19.029 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.030 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.030 DEBUG:boto:Params: {} 2023-10-11T15:14:19.031 DEBUG:boto:Token: None 2023-10-11T15:14:19.031 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/+1lE4ewjO9JBOQIlt1oikbQCic= 2023-10-11T15:14:19.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/+1lE4ewjO9JBOQIlt1oikbQCic=', 'Content-Length': '0'} 2023-10-11T15:14:19.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9d59ca20710f9c-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.035 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.035 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.036 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.036 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.036 DEBUG:boto:Data: 2023-10-11T15:14:19.037 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.037 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.038 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.038 DEBUG:boto:Params: {} 2023-10-11T15:14:19.038 DEBUG:boto:Token: None 2023-10-11T15:14:19.039 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/+1lE4ewjO9JBOQIlt1oikbQCic= 2023-10-11T15:14:19.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/+1lE4ewjO9JBOQIlt1oikbQCic=', 'Content-Length': '0'} 2023-10-11T15:14:19.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c8378dde4e7696-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-10-11T15:14:19.043 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.043 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.044 DEBUG:boto:Method: GET 2023-10-11T15:14:19.044 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.045 DEBUG:boto:Data: 2023-10-11T15:14:19.045 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.046 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.046 DEBUG:boto:Params: {} 2023-10-11T15:14:19.047 DEBUG:boto:Token: None 2023-10-11T15:14:19.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ddwMky3nMOihnE0YgoKJOatElWw= 2023-10-11T15:14:19.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ddwMky3nMOihnE0YgoKJOatElWw=', 'Content-Length': '0'} 2023-10-11T15:14:19.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e418f1bcb30195f-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.051 DEBUG:boto:path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.051 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.052 DEBUG:boto:Method: GET 2023-10-11T15:14:19.052 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.052 DEBUG:boto:Data: 2023-10-11T15:14:19.053 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.054 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.054 DEBUG:boto:Params: {} 2023-10-11T15:14:19.055 DEBUG:boto:Token: None 2023-10-11T15:14:19.055 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000017 2023-10-11T15:14:19.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ddwMky3nMOihnE0YgoKJOatElWw= 2023-10-11T15:14:19.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ddwMky3nMOihnE0YgoKJOatElWw=', 'Content-Length': '0'} 2023-10-11T15:14:19.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f242e6d043e3ef-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.058 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.059 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.059 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.060 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.060 DEBUG:boto:Method: GET 2023-10-11T15:14:19.061 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.061 DEBUG:boto:Data: 2023-10-11T15:14:19.061 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.062 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.062 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.063 DEBUG:boto:Params: {} 2023-10-11T15:14:19.063 DEBUG:boto:Token: None 2023-10-11T15:14:19.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e14e55eab69f8e86-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.068 DEBUG:boto:b'dghqvd-58obj000000000000000000171000falseobj000000000000000000172023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.068 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.069 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.069 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.070 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.070 DEBUG:boto:Method: GET 2023-10-11T15:14:19.071 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000017 2023-10-11T15:14:19.071 DEBUG:boto:Data: 2023-10-11T15:14:19.071 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.072 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.073 DEBUG:boto:Params: {} 2023-10-11T15:14:19.073 DEBUG:boto:Token: None 2023-10-11T15:14:19.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe533e1f9d1696e-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.079 DEBUG:boto:b'dghqvd-58obj000000000000000000171000falseobj000000000000000000172023-10-11T15:14:05.791Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-11T15:14:19.080 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.081 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.081 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.082 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.082 DEBUG:boto:Method: GET 2023-10-11T15:14:19.082 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.083 DEBUG:boto:Data: 2023-10-11T15:14:19.083 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.084 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.084 DEBUG:boto:Params: {} 2023-10-11T15:14:19.085 DEBUG:boto:Token: None 2023-10-11T15:14:19.085 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.086 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk= 2023-10-11T15:14:19.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk=', 'Content-Length': '0'} 2023-10-11T15:14:19.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d36afaecdd6b97-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.130 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.131 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.131 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.132 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.132 DEBUG:boto:Method: GET 2023-10-11T15:14:19.133 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.133 DEBUG:boto:Data: 2023-10-11T15:14:19.133 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.134 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.135 DEBUG:boto:Params: {} 2023-10-11T15:14:19.135 DEBUG:boto:Token: None 2023-10-11T15:14:19.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk= 2023-10-11T15:14:19.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk=', 'Content-Length': '0'} 2023-10-11T15:14:19.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000391132844bd46de-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.139 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.140 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.140 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.141 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.141 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.141 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.142 DEBUG:boto:Data: 2023-10-11T15:14:19.142 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.143 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.143 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.143 DEBUG:boto:Params: {} 2023-10-11T15:14:19.144 DEBUG:boto:Token: None 2023-10-11T15:14:19.144 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.145 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NUe5QGCo5vpOTcfKnmpuMF/3NuI= 2023-10-11T15:14:19.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NUe5QGCo5vpOTcfKnmpuMF/3NuI=', 'Content-Length': '0'} 2023-10-11T15:14:19.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477d1973ae30c5cf-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.148 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.148 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.149 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.149 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.150 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.150 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.153 DEBUG:boto:Data: 2023-10-11T15:14:19.153 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.154 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.154 DEBUG:boto:Params: {} 2023-10-11T15:14:19.155 DEBUG:boto:Token: None 2023-10-11T15:14:19.155 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NUe5QGCo5vpOTcfKnmpuMF/3NuI= 2023-10-11T15:14:19.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NUe5QGCo5vpOTcfKnmpuMF/3NuI=', 'Content-Length': '0'} 2023-10-11T15:14:19.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194c6a856a9976b0-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-11T15:14:19.159 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.160 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.160 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.161 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.161 DEBUG:boto:Method: GET 2023-10-11T15:14:19.161 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.162 DEBUG:boto:Data: 2023-10-11T15:14:19.162 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.163 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.164 DEBUG:boto:Params: {} 2023-10-11T15:14:19.164 DEBUG:boto:Token: None 2023-10-11T15:14:19.164 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk= 2023-10-11T15:14:19.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk=', 'Content-Length': '0'} 2023-10-11T15:14:19.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbee952369714a5-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.168 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.168 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.169 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.169 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.170 DEBUG:boto:Method: GET 2023-10-11T15:14:19.170 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.170 DEBUG:boto:Data: 2023-10-11T15:14:19.171 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.171 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.172 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.172 DEBUG:boto:Params: {} 2023-10-11T15:14:19.173 DEBUG:boto:Token: None 2023-10-11T15:14:19.173 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018?versionId=null 2023-10-11T15:14:19.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk= 2023-10-11T15:14:19.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+B+52cYQUxd2esDULtnoLgaEJZk=', 'Content-Length': '0'} 2023-10-11T15:14:19.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325e708262e0042f-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.222 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.223 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.223 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.224 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.224 DEBUG:boto:Method: GET 2023-10-11T15:14:19.225 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.225 DEBUG:boto:Data: 2023-10-11T15:14:19.225 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.226 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.227 DEBUG:boto:Params: {} 2023-10-11T15:14:19.227 DEBUG:boto:Token: None 2023-10-11T15:14:19.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006968df20a27fd542-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.232 DEBUG:boto:b'dghqvd-58obj000000000000000000181000falseobj000000000000000000182023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.233 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.233 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.234 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.234 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.234 DEBUG:boto:Method: GET 2023-10-11T15:14:19.235 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.235 DEBUG:boto:Data: 2023-10-11T15:14:19.236 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.236 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.237 DEBUG:boto:Params: {} 2023-10-11T15:14:19.237 DEBUG:boto:Token: None 2023-10-11T15:14:19.238 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab63530d5e6d395-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.244 DEBUG:boto:b'dghqvd-58obj000000000000000000181000falseobj000000000000000000182023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.244 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.245 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.245 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.245 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.246 DEBUG:boto:Data: 2023-10-11T15:14:19.246 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.247 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.247 DEBUG:boto:Params: {} 2023-10-11T15:14:19.248 DEBUG:boto:Token: None 2023-10-11T15:14:19.248 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:30dcQzNtUSXYtgbKur3stWAnFHg= 2023-10-11T15:14:19.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:30dcQzNtUSXYtgbKur3stWAnFHg=', 'Content-Length': '0'} 2023-10-11T15:14:19.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99a14be5f2b3c51-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.252 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.252 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.253 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.253 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.253 DEBUG:boto:Data: 2023-10-11T15:14:19.254 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.255 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.255 DEBUG:boto:Params: {} 2023-10-11T15:14:19.255 DEBUG:boto:Token: None 2023-10-11T15:14:19.256 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:30dcQzNtUSXYtgbKur3stWAnFHg= 2023-10-11T15:14:19.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:30dcQzNtUSXYtgbKur3stWAnFHg=', 'Content-Length': '0'} 2023-10-11T15:14:19.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1d73b262b7ee24d-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-10-11T15:14:19.260 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.260 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.261 DEBUG:boto:Method: GET 2023-10-11T15:14:19.261 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.261 DEBUG:boto:Data: 2023-10-11T15:14:19.262 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.263 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.263 DEBUG:boto:Params: {} 2023-10-11T15:14:19.264 DEBUG:boto:Token: None 2023-10-11T15:14:19.264 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pvil/ZHc4tPrUBHhRMkTFe1BMI4= 2023-10-11T15:14:19.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pvil/ZHc4tPrUBHhRMkTFe1BMI4=', 'Content-Length': '0'} 2023-10-11T15:14:19.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b1244156c46278-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.310 DEBUG:boto:path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.311 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.311 DEBUG:boto:Method: GET 2023-10-11T15:14:19.312 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.312 DEBUG:boto:Data: 2023-10-11T15:14:19.313 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.314 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.314 DEBUG:boto:Params: {} 2023-10-11T15:14:19.314 DEBUG:boto:Token: None 2023-10-11T15:14:19.315 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000018 2023-10-11T15:14:19.315 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pvil/ZHc4tPrUBHhRMkTFe1BMI4= 2023-10-11T15:14:19.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pvil/ZHc4tPrUBHhRMkTFe1BMI4=', 'Content-Length': '0'} 2023-10-11T15:14:19.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c2755eee459db44-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.319 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.319 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.320 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.320 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.321 DEBUG:boto:Method: GET 2023-10-11T15:14:19.321 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.321 DEBUG:boto:Data: 2023-10-11T15:14:19.322 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.323 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.323 DEBUG:boto:Params: {} 2023-10-11T15:14:19.324 DEBUG:boto:Token: None 2023-10-11T15:14:19.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9232dc2e499c4a4-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.328 DEBUG:boto:b'dghqvd-58obj000000000000000000181000falseobj000000000000000000182023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.329 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.330 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.330 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.330 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.331 DEBUG:boto:Method: GET 2023-10-11T15:14:19.331 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000018 2023-10-11T15:14:19.332 DEBUG:boto:Data: 2023-10-11T15:14:19.332 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.333 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.333 DEBUG:boto:Params: {} 2023-10-11T15:14:19.334 DEBUG:boto:Token: None 2023-10-11T15:14:19.334 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0517d0dea8fb637-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.382 DEBUG:boto:b'dghqvd-58obj000000000000000000181000falseobj000000000000000000182023-10-11T15:14:05.298Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-11T15:14:19.384 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.384 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.385 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.385 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.385 DEBUG:boto:Method: GET 2023-10-11T15:14:19.386 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.386 DEBUG:boto:Data: 2023-10-11T15:14:19.387 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.388 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.388 DEBUG:boto:Params: {} 2023-10-11T15:14:19.388 DEBUG:boto:Token: None 2023-10-11T15:14:19.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64= 2023-10-11T15:14:19.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64=', 'Content-Length': '0'} 2023-10-11T15:14:19.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b54d54d6fad00a5-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.393 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.393 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.394 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.394 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.395 DEBUG:boto:Method: GET 2023-10-11T15:14:19.395 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.396 DEBUG:boto:Data: 2023-10-11T15:14:19.396 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.397 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.397 DEBUG:boto:Params: {} 2023-10-11T15:14:19.398 DEBUG:boto:Token: None 2023-10-11T15:14:19.398 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64= 2023-10-11T15:14:19.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64=', 'Content-Length': '0'} 2023-10-11T15:14:19.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f42aa50d66cd6afd-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.402 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.402 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.402 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.403 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.404 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.404 DEBUG:boto:Data: 2023-10-11T15:14:19.404 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.405 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.406 DEBUG:boto:Params: {} 2023-10-11T15:14:19.406 DEBUG:boto:Token: None 2023-10-11T15:14:19.407 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.407 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F9rUhE/Izhi/95Ft/xIhBQiST9o= 2023-10-11T15:14:19.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F9rUhE/Izhi/95Ft/xIhBQiST9o=', 'Content-Length': '0'} 2023-10-11T15:14:19.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000755316da6eb82b3f-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.410 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.410 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.411 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.411 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.411 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.412 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.412 DEBUG:boto:Data: 2023-10-11T15:14:19.413 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.413 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.413 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.414 DEBUG:boto:Params: {} 2023-10-11T15:14:19.414 DEBUG:boto:Token: None 2023-10-11T15:14:19.415 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F9rUhE/Izhi/95Ft/xIhBQiST9o= 2023-10-11T15:14:19.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F9rUhE/Izhi/95Ft/xIhBQiST9o=', 'Content-Length': '0'} 2023-10-11T15:14:19.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c958addc94b23bc-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-11T15:14:19.419 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.419 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.420 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.420 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.420 DEBUG:boto:Method: GET 2023-10-11T15:14:19.421 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.421 DEBUG:boto:Data: 2023-10-11T15:14:19.422 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.423 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.423 DEBUG:boto:Params: {} 2023-10-11T15:14:19.423 DEBUG:boto:Token: None 2023-10-11T15:14:19.424 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64= 2023-10-11T15:14:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64=', 'Content-Length': '0'} 2023-10-11T15:14:19.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e5750a213ab794-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.428 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.428 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.429 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.429 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.429 DEBUG:boto:Method: GET 2023-10-11T15:14:19.430 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.430 DEBUG:boto:Data: 2023-10-11T15:14:19.431 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.431 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.432 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.432 DEBUG:boto:Params: {} 2023-10-11T15:14:19.432 DEBUG:boto:Token: None 2023-10-11T15:14:19.433 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019?versionId=null 2023-10-11T15:14:19.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64= 2023-10-11T15:14:19.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+9ToF4Ec4Rkr6lCqqP6jBEMLZ64=', 'Content-Length': '0'} 2023-10-11T15:14:19.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b60417c4b5611a1-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.437 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.437 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.437 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.438 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.438 DEBUG:boto:Method: GET 2023-10-11T15:14:19.439 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.439 DEBUG:boto:Data: 2023-10-11T15:14:19.440 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.440 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.441 DEBUG:boto:Params: {} 2023-10-11T15:14:19.441 DEBUG:boto:Token: None 2023-10-11T15:14:19.442 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9e1dde39004d81-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.446 DEBUG:boto:b'dghqvd-58obj000000000000000000191000falseobj000000000000000000192023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.447 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.447 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.448 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.448 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.449 DEBUG:boto:Method: GET 2023-10-11T15:14:19.449 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.449 DEBUG:boto:Data: 2023-10-11T15:14:19.450 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.450 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.451 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.451 DEBUG:boto:Params: {} 2023-10-11T15:14:19.452 DEBUG:boto:Token: None 2023-10-11T15:14:19.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083304cb7a33cf62-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.459 DEBUG:boto:b'dghqvd-58obj000000000000000000191000falseobj000000000000000000192023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.459 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.460 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.460 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.461 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.461 DEBUG:boto:Data: 2023-10-11T15:14:19.461 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.462 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.463 DEBUG:boto:Params: {} 2023-10-11T15:14:19.463 DEBUG:boto:Token: None 2023-10-11T15:14:19.464 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Axj7RF1AVUYJzDl8FlARN2m24qg= 2023-10-11T15:14:19.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Axj7RF1AVUYJzDl8FlARN2m24qg=', 'Content-Length': '0'} 2023-10-11T15:14:19.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9e60ac2743b989-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.467 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.468 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.468 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.469 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.469 DEBUG:boto:Data: 2023-10-11T15:14:19.469 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.470 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.471 DEBUG:boto:Params: {} 2023-10-11T15:14:19.471 DEBUG:boto:Token: None 2023-10-11T15:14:19.472 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Axj7RF1AVUYJzDl8FlARN2m24qg= 2023-10-11T15:14:19.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Axj7RF1AVUYJzDl8FlARN2m24qg=', 'Content-Length': '0'} 2023-10-11T15:14:19.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045214190802b32c6-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-10-11T15:14:19.475 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.476 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.476 DEBUG:boto:Method: GET 2023-10-11T15:14:19.477 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.477 DEBUG:boto:Data: 2023-10-11T15:14:19.478 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.479 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.479 DEBUG:boto:Params: {} 2023-10-11T15:14:19.480 DEBUG:boto:Token: None 2023-10-11T15:14:19.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yw0JNFNNGpC8zwGGb7Eu7gbVbyQ= 2023-10-11T15:14:19.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yw0JNFNNGpC8zwGGb7Eu7gbVbyQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c1635e25bfec905-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.530 DEBUG:boto:path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.531 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.531 DEBUG:boto:Method: GET 2023-10-11T15:14:19.532 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.532 DEBUG:boto:Data: 2023-10-11T15:14:19.533 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.534 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.534 DEBUG:boto:Params: {} 2023-10-11T15:14:19.535 DEBUG:boto:Token: None 2023-10-11T15:14:19.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000019 2023-10-11T15:14:19.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yw0JNFNNGpC8zwGGb7Eu7gbVbyQ= 2023-10-11T15:14:19.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yw0JNFNNGpC8zwGGb7Eu7gbVbyQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061170b95db98858b-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.539 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.540 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.540 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.540 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.541 DEBUG:boto:Method: GET 2023-10-11T15:14:19.541 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.542 DEBUG:boto:Data: 2023-10-11T15:14:19.542 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.543 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.543 DEBUG:boto:Params: {} 2023-10-11T15:14:19.544 DEBUG:boto:Token: None 2023-10-11T15:14:19.544 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b776acf0c38318-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.549 DEBUG:boto:b'dghqvd-58obj000000000000000000191000falseobj000000000000000000192023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.549 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.550 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.550 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.551 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.551 DEBUG:boto:Method: GET 2023-10-11T15:14:19.552 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000019 2023-10-11T15:14:19.552 DEBUG:boto:Data: 2023-10-11T15:14:19.553 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.553 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.554 DEBUG:boto:Params: {} 2023-10-11T15:14:19.554 DEBUG:boto:Token: None 2023-10-11T15:14:19.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eac7d2f99ebc6b94-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.560 DEBUG:boto:b'dghqvd-58obj000000000000000000191000falseobj000000000000000000192023-10-11T15:14:04.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-11T15:14:19.561 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.562 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.562 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.563 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.563 DEBUG:boto:Method: GET 2023-10-11T15:14:19.564 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.564 DEBUG:boto:Data: 2023-10-11T15:14:19.564 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.565 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.565 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.566 DEBUG:boto:Params: {} 2023-10-11T15:14:19.566 DEBUG:boto:Token: None 2023-10-11T15:14:19.567 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs= 2023-10-11T15:14:19.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs=', 'Content-Length': '0'} 2023-10-11T15:14:19.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b92e19d61e9906e-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.570 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.571 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.571 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.572 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.572 DEBUG:boto:Method: GET 2023-10-11T15:14:19.573 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.573 DEBUG:boto:Data: 2023-10-11T15:14:19.574 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.575 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.575 DEBUG:boto:Params: {} 2023-10-11T15:14:19.576 DEBUG:boto:Token: None 2023-10-11T15:14:19.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs= 2023-10-11T15:14:19.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs=', 'Content-Length': '0'} 2023-10-11T15:14:19.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048eb90251edbeb16-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.626 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.627 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.627 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.628 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.628 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.629 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.629 DEBUG:boto:Data: 2023-10-11T15:14:19.630 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.631 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.631 DEBUG:boto:Params: {} 2023-10-11T15:14:19.632 DEBUG:boto:Token: None 2023-10-11T15:14:19.632 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.633 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6UHcZrWL67i1Os9mhfgzNicQTZ8= 2023-10-11T15:14:19.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6UHcZrWL67i1Os9mhfgzNicQTZ8=', 'Content-Length': '0'} 2023-10-11T15:14:19.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c82a7b7999908ad-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.636 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.637 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.637 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.638 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.638 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.639 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.639 DEBUG:boto:Data: 2023-10-11T15:14:19.640 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.640 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.641 DEBUG:boto:Params: {} 2023-10-11T15:14:19.641 DEBUG:boto:Token: None 2023-10-11T15:14:19.642 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.642 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6UHcZrWL67i1Os9mhfgzNicQTZ8= 2023-10-11T15:14:19.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6UHcZrWL67i1Os9mhfgzNicQTZ8=', 'Content-Length': '0'} 2023-10-11T15:14:19.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca2d1a1b856ee43-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-11T15:14:19.646 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.647 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.647 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.648 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.648 DEBUG:boto:Method: GET 2023-10-11T15:14:19.649 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.650 DEBUG:boto:Data: 2023-10-11T15:14:19.650 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.651 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.652 DEBUG:boto:Params: {} 2023-10-11T15:14:19.653 DEBUG:boto:Token: None 2023-10-11T15:14:19.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs= 2023-10-11T15:14:19.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs=', 'Content-Length': '0'} 2023-10-11T15:14:19.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8d1043fc1d4261-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.658 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.658 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.659 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.660 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.661 DEBUG:boto:Method: GET 2023-10-11T15:14:19.661 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.662 DEBUG:boto:Data: 2023-10-11T15:14:19.662 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.663 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.663 DEBUG:boto:Params: {} 2023-10-11T15:14:19.664 DEBUG:boto:Token: None 2023-10-11T15:14:19.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020?versionId=null 2023-10-11T15:14:19.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs= 2023-10-11T15:14:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IArPTw1LRKmhMVf2EpjtDtFgUDs=', 'Content-Length': '0'} 2023-10-11T15:14:19.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e66ddef2d4e7909-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.668 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.668 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.669 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.669 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.670 DEBUG:boto:Method: GET 2023-10-11T15:14:19.670 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.670 DEBUG:boto:Data: 2023-10-11T15:14:19.671 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.672 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.672 DEBUG:boto:Params: {} 2023-10-11T15:14:19.673 DEBUG:boto:Token: None 2023-10-11T15:14:19.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008278a644f3f76eab-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.678 DEBUG:boto:b'dghqvd-58obj000000000000000000201000falseobj000000000000000000202023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.678 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.679 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.679 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.680 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.680 DEBUG:boto:Method: GET 2023-10-11T15:14:19.681 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.681 DEBUG:boto:Data: 2023-10-11T15:14:19.681 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.682 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.683 DEBUG:boto:Params: {} 2023-10-11T15:14:19.683 DEBUG:boto:Token: None 2023-10-11T15:14:19.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580d854bdf1fcce2-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.689 DEBUG:boto:b'dghqvd-58obj000000000000000000201000falseobj000000000000000000202023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.690 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.690 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.691 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.691 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.692 DEBUG:boto:Data: 2023-10-11T15:14:19.692 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.693 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.693 DEBUG:boto:Params: {} 2023-10-11T15:14:19.694 DEBUG:boto:Token: None 2023-10-11T15:14:19.694 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.695 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8b9WchYrAANGRHALFa34cmFN9M= 2023-10-11T15:14:19.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8b9WchYrAANGRHALFa34cmFN9M=', 'Content-Length': '0'} 2023-10-11T15:14:19.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96fb91806063875-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.698 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.698 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.699 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.699 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.700 DEBUG:boto:Data: 2023-10-11T15:14:19.700 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.700 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.701 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.701 DEBUG:boto:Params: {} 2023-10-11T15:14:19.702 DEBUG:boto:Token: None 2023-10-11T15:14:19.702 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8b9WchYrAANGRHALFa34cmFN9M= 2023-10-11T15:14:19.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8b9WchYrAANGRHALFa34cmFN9M=', 'Content-Length': '0'} 2023-10-11T15:14:19.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078220b9625c72079-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-10-11T15:14:19.706 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.707 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.707 DEBUG:boto:Method: GET 2023-10-11T15:14:19.707 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.708 DEBUG:boto:Data: 2023-10-11T15:14:19.708 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.709 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.710 DEBUG:boto:Params: {} 2023-10-11T15:14:19.710 DEBUG:boto:Token: None 2023-10-11T15:14:19.711 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:khTvRLIg6e1OIfB6WK3nn4+Zp24= 2023-10-11T15:14:19.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:khTvRLIg6e1OIfB6WK3nn4+Zp24=', 'Content-Length': '0'} 2023-10-11T15:14:19.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093459f9032c9323a-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.714 DEBUG:boto:path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.715 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.715 DEBUG:boto:Method: GET 2023-10-11T15:14:19.716 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.716 DEBUG:boto:Data: 2023-10-11T15:14:19.716 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.717 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.718 DEBUG:boto:Params: {} 2023-10-11T15:14:19.718 DEBUG:boto:Token: None 2023-10-11T15:14:19.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000020 2023-10-11T15:14:19.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:khTvRLIg6e1OIfB6WK3nn4+Zp24= 2023-10-11T15:14:19.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:khTvRLIg6e1OIfB6WK3nn4+Zp24=', 'Content-Length': '0'} 2023-10-11T15:14:19.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057fc67a217ff3bac-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.762 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.763 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.763 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.764 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.764 DEBUG:boto:Method: GET 2023-10-11T15:14:19.765 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.765 DEBUG:boto:Data: 2023-10-11T15:14:19.766 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.769 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.770 DEBUG:boto:Params: {} 2023-10-11T15:14:19.770 DEBUG:boto:Token: None 2023-10-11T15:14:19.771 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067fb31a79906c3d1-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.776 DEBUG:boto:b'dghqvd-58obj000000000000000000201000falseobj000000000000000000202023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.777 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.777 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.778 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.778 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.778 DEBUG:boto:Method: GET 2023-10-11T15:14:19.779 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000020 2023-10-11T15:14:19.779 DEBUG:boto:Data: 2023-10-11T15:14:19.780 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.781 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.781 DEBUG:boto:Params: {} 2023-10-11T15:14:19.781 DEBUG:boto:Token: None 2023-10-11T15:14:19.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dc0ff821c8b6c75-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.788 DEBUG:boto:b'dghqvd-58obj000000000000000000201000falseobj000000000000000000202023-10-11T15:14:05.851Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-11T15:14:19.789 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.790 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.790 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.791 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.791 DEBUG:boto:Method: GET 2023-10-11T15:14:19.791 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.792 DEBUG:boto:Data: 2023-10-11T15:14:19.792 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.793 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.793 DEBUG:boto:Params: {} 2023-10-11T15:14:19.794 DEBUG:boto:Token: None 2023-10-11T15:14:19.794 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI= 2023-10-11T15:14:19.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI=', 'Content-Length': '0'} 2023-10-11T15:14:19.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446ad52ff1fa7ca6-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.838 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.839 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.839 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.840 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.840 DEBUG:boto:Method: GET 2023-10-11T15:14:19.841 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.841 DEBUG:boto:Data: 2023-10-11T15:14:19.842 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.842 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.843 DEBUG:boto:Params: {} 2023-10-11T15:14:19.843 DEBUG:boto:Token: None 2023-10-11T15:14:19.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI= 2023-10-11T15:14:19.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI=', 'Content-Length': '0'} 2023-10-11T15:14:19.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5125b5f9c140535-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.848 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.849 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.849 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.850 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.850 DEBUG:boto:Data: 2023-10-11T15:14:19.850 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.851 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.852 DEBUG:boto:Params: {} 2023-10-11T15:14:19.852 DEBUG:boto:Token: None 2023-10-11T15:14:19.853 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J0HdtUOYIw6G/XxY1KjXr8jVX/A= 2023-10-11T15:14:19.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J0HdtUOYIw6G/XxY1KjXr8jVX/A=', 'Content-Length': '0'} 2023-10-11T15:14:19.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f28319ff3038972-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.856 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.857 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.857 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.858 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.858 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.859 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.859 DEBUG:boto:Data: 2023-10-11T15:14:19.859 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.860 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.861 DEBUG:boto:Params: {} 2023-10-11T15:14:19.861 DEBUG:boto:Token: None 2023-10-11T15:14:19.862 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J0HdtUOYIw6G/XxY1KjXr8jVX/A= 2023-10-11T15:14:19.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J0HdtUOYIw6G/XxY1KjXr8jVX/A=', 'Content-Length': '0'} 2023-10-11T15:14:19.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2f2a5e9e545e30-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-11T15:14:19.866 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.866 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.867 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.867 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.867 DEBUG:boto:Method: GET 2023-10-11T15:14:19.868 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.868 DEBUG:boto:Data: 2023-10-11T15:14:19.869 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.870 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.870 DEBUG:boto:Params: {} 2023-10-11T15:14:19.871 DEBUG:boto:Token: None 2023-10-11T15:14:19.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI= 2023-10-11T15:14:19.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI=', 'Content-Length': '0'} 2023-10-11T15:14:19.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ca76b8fb4a6a4f-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.875 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.876 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.876 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.876 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.877 DEBUG:boto:Method: GET 2023-10-11T15:14:19.877 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.878 DEBUG:boto:Data: 2023-10-11T15:14:19.878 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.879 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.879 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.879 DEBUG:boto:Params: {} 2023-10-11T15:14:19.880 DEBUG:boto:Token: None 2023-10-11T15:14:19.880 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021?versionId=null 2023-10-11T15:14:19.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI= 2023-10-11T15:14:19.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:odiROEa2IfahUg7em/B+HdyQyvI=', 'Content-Length': '0'} 2023-10-11T15:14:19.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b4f2888a6bb3e2-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.884 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.884 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.885 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.886 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.886 DEBUG:boto:Method: GET 2023-10-11T15:14:19.887 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.887 DEBUG:boto:Data: 2023-10-11T15:14:19.887 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.888 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.888 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.889 DEBUG:boto:Params: {} 2023-10-11T15:14:19.889 DEBUG:boto:Token: None 2023-10-11T15:14:19.890 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.890 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000834a0d74d84cd36-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.894 DEBUG:boto:b'dghqvd-58obj000000000000000000211000falseobj000000000000000000212023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.894 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.895 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.895 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.896 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.896 DEBUG:boto:Method: GET 2023-10-11T15:14:19.897 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.897 DEBUG:boto:Data: 2023-10-11T15:14:19.898 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.899 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.899 DEBUG:boto:Params: {} 2023-10-11T15:14:19.900 DEBUG:boto:Token: None 2023-10-11T15:14:19.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e1cdcb77986991-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.906 DEBUG:boto:b'dghqvd-58obj000000000000000000211000falseobj000000000000000000212023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.907 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.907 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.907 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.908 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.908 DEBUG:boto:Data: 2023-10-11T15:14:19.909 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.909 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.910 DEBUG:boto:Params: {} 2023-10-11T15:14:19.910 DEBUG:boto:Token: None 2023-10-11T15:14:19.911 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lFEidHofzVlpZc8GpvHqC+bxU50= 2023-10-11T15:14:19.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lFEidHofzVlpZc8GpvHqC+bxU50=', 'Content-Length': '0'} 2023-10-11T15:14:19.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd54ffc1d98cab8-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.914 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.915 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.915 DEBUG:boto:Method: HEAD 2023-10-11T15:14:19.916 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.916 DEBUG:boto:Data: 2023-10-11T15:14:19.916 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.917 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.918 DEBUG:boto:Params: {} 2023-10-11T15:14:19.918 DEBUG:boto:Token: None 2023-10-11T15:14:19.919 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.919 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lFEidHofzVlpZc8GpvHqC+bxU50= 2023-10-11T15:14:19.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lFEidHofzVlpZc8GpvHqC+bxU50=', 'Content-Length': '0'} 2023-10-11T15:14:19.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb2f5e5bd9235d6-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-10-11T15:14:19.923 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.923 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.923 DEBUG:boto:Method: GET 2023-10-11T15:14:19.924 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.924 DEBUG:boto:Data: 2023-10-11T15:14:19.925 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.926 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.926 DEBUG:boto:Params: {} 2023-10-11T15:14:19.927 DEBUG:boto:Token: None 2023-10-11T15:14:19.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t56FAEyiaudk4lXtgU1aBrBRtoI= 2023-10-11T15:14:19.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t56FAEyiaudk4lXtgU1aBrBRtoI=', 'Content-Length': '0'} 2023-10-11T15:14:19.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd265c814e8182e-006526bbcb-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.931 DEBUG:boto:path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.931 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.932 DEBUG:boto:Method: GET 2023-10-11T15:14:19.932 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.932 DEBUG:boto:Data: 2023-10-11T15:14:19.933 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.934 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.934 DEBUG:boto:Params: {} 2023-10-11T15:14:19.935 DEBUG:boto:Token: None 2023-10-11T15:14:19.935 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/obj00000000000000000021 2023-10-11T15:14:19.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t56FAEyiaudk4lXtgU1aBrBRtoI= 2023-10-11T15:14:19.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t56FAEyiaudk4lXtgU1aBrBRtoI=', 'Content-Length': '0'} 2023-10-11T15:14:19.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94292743e331075-006526bbcb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.978 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.979 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.980 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.980 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.980 DEBUG:boto:Method: GET 2023-10-11T15:14:19.981 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.981 DEBUG:boto:Data: 2023-10-11T15:14:19.982 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:19.983 DEBUG:boto:Port: 8001 2023-10-11T15:14:19.983 DEBUG:boto:Params: {} 2023-10-11T15:14:19.983 DEBUG:boto:Token: None 2023-10-11T15:14:19.984 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:19.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e3698a66be616bd-006526bbcb-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:19.988 DEBUG:boto:b'dghqvd-58obj000000000000000000211000falseobj000000000000000000212023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:19.989 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:19.989 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:19.990 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.990 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.991 DEBUG:boto:Method: GET 2023-10-11T15:14:19.991 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000021 2023-10-11T15:14:19.992 DEBUG:boto:Data: 2023-10-11T15:14:19.992 DEBUG:boto:Headers: {} 2023-10-11T15:14:19.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:19.993 DEBUG:boto:Port: 8000 2023-10-11T15:14:19.993 DEBUG:boto:Params: {} 2023-10-11T15:14:19.994 DEBUG:boto:Token: None 2023-10-11T15:14:19.994 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:19 GMT /dghqvd-58/ 2023-10-11T15:14:19.995 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ= 2023-10-11T15:14:19.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cn+vVrQGs8u3vV9Un8WNO8bDZpQ=', 'Content-Length': '0'} 2023-10-11T15:14:20.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1774e92b9bb4d2-006526bbcb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.000 DEBUG:boto:b'dghqvd-58obj000000000000000000211000falseobj000000000000000000212023-10-11T15:14:05.277Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-11T15:14:20.001 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.002 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.002 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.003 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.003 DEBUG:boto:Method: GET 2023-10-11T15:14:20.003 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.004 DEBUG:boto:Data: 2023-10-11T15:14:20.004 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.005 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.006 DEBUG:boto:Params: {} 2023-10-11T15:14:20.006 DEBUG:boto:Token: None 2023-10-11T15:14:20.007 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM= 2023-10-11T15:14:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM=', 'Content-Length': '0'} 2023-10-11T15:14:20.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005efcbcfc49c53e84-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.051 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.051 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.052 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.052 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.053 DEBUG:boto:Method: GET 2023-10-11T15:14:20.053 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.054 DEBUG:boto:Data: 2023-10-11T15:14:20.054 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.055 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.056 DEBUG:boto:Params: {} 2023-10-11T15:14:20.056 DEBUG:boto:Token: None 2023-10-11T15:14:20.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM= 2023-10-11T15:14:20.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM=', 'Content-Length': '0'} 2023-10-11T15:14:20.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fed4beba88a753-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.061 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.061 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.062 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.062 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.062 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.063 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.063 DEBUG:boto:Data: 2023-10-11T15:14:20.064 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.064 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.065 DEBUG:boto:Params: {} 2023-10-11T15:14:20.065 DEBUG:boto:Token: None 2023-10-11T15:14:20.065 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0o08kxGXSSwy4UwOH0XCTfbGGFc= 2023-10-11T15:14:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0o08kxGXSSwy4UwOH0XCTfbGGFc=', 'Content-Length': '0'} 2023-10-11T15:14:20.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e671ea20e7c634f-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.069 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.069 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.070 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.070 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.070 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.071 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.071 DEBUG:boto:Data: 2023-10-11T15:14:20.072 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.073 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.073 DEBUG:boto:Params: {} 2023-10-11T15:14:20.073 DEBUG:boto:Token: None 2023-10-11T15:14:20.074 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0o08kxGXSSwy4UwOH0XCTfbGGFc= 2023-10-11T15:14:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0o08kxGXSSwy4UwOH0XCTfbGGFc=', 'Content-Length': '0'} 2023-10-11T15:14:20.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073eba2df96773240-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-11T15:14:20.077 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.078 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.078 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.079 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.079 DEBUG:boto:Method: GET 2023-10-11T15:14:20.080 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.080 DEBUG:boto:Data: 2023-10-11T15:14:20.080 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.081 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.082 DEBUG:boto:Params: {} 2023-10-11T15:14:20.082 DEBUG:boto:Token: None 2023-10-11T15:14:20.083 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM= 2023-10-11T15:14:20.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM=', 'Content-Length': '0'} 2023-10-11T15:14:20.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000519cbffbcd8ecd27-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.087 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.087 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.087 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.088 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.088 DEBUG:boto:Method: GET 2023-10-11T15:14:20.089 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.089 DEBUG:boto:Data: 2023-10-11T15:14:20.089 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.090 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.091 DEBUG:boto:Params: {} 2023-10-11T15:14:20.091 DEBUG:boto:Token: None 2023-10-11T15:14:20.092 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022?versionId=null 2023-10-11T15:14:20.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM= 2023-10-11T15:14:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SFbn6FuAm8SBRFicf3oIav9sucM=', 'Content-Length': '0'} 2023-10-11T15:14:20.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9052ac920a412f-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.095 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.095 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.096 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.096 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.097 DEBUG:boto:Method: GET 2023-10-11T15:14:20.097 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.098 DEBUG:boto:Data: 2023-10-11T15:14:20.098 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.099 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.099 DEBUG:boto:Params: {} 2023-10-11T15:14:20.100 DEBUG:boto:Token: None 2023-10-11T15:14:20.100 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f062045ba4a5d1-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.104 DEBUG:boto:b'dghqvd-58obj000000000000000000221000falseobj000000000000000000222023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.105 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.105 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.106 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.106 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.107 DEBUG:boto:Method: GET 2023-10-11T15:14:20.107 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.108 DEBUG:boto:Data: 2023-10-11T15:14:20.108 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.109 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.109 DEBUG:boto:Params: {} 2023-10-11T15:14:20.110 DEBUG:boto:Token: None 2023-10-11T15:14:20.110 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6302aebd00c29e-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.115 DEBUG:boto:b'dghqvd-58obj000000000000000000221000falseobj000000000000000000222023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.116 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.116 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.117 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.117 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.118 DEBUG:boto:Data: 2023-10-11T15:14:20.118 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.119 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.119 DEBUG:boto:Params: {} 2023-10-11T15:14:20.120 DEBUG:boto:Token: None 2023-10-11T15:14:20.120 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y3tQQD4DhGS7pCbNXIdZzA6qv6M= 2023-10-11T15:14:20.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y3tQQD4DhGS7pCbNXIdZzA6qv6M=', 'Content-Length': '0'} 2023-10-11T15:14:20.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba22b6b7918ed91-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.124 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.124 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.125 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.125 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.126 DEBUG:boto:Data: 2023-10-11T15:14:20.126 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.127 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.127 DEBUG:boto:Params: {} 2023-10-11T15:14:20.128 DEBUG:boto:Token: None 2023-10-11T15:14:20.128 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.129 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y3tQQD4DhGS7pCbNXIdZzA6qv6M= 2023-10-11T15:14:20.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y3tQQD4DhGS7pCbNXIdZzA6qv6M=', 'Content-Length': '0'} 2023-10-11T15:14:20.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2949188458722e-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-10-11T15:14:20.132 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.132 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.133 DEBUG:boto:Method: GET 2023-10-11T15:14:20.133 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.134 DEBUG:boto:Data: 2023-10-11T15:14:20.134 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.135 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.135 DEBUG:boto:Params: {} 2023-10-11T15:14:20.136 DEBUG:boto:Token: None 2023-10-11T15:14:20.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6jpSjT+u0IVmOUrlwBr7QPXtH6Y= 2023-10-11T15:14:20.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6jpSjT+u0IVmOUrlwBr7QPXtH6Y=', 'Content-Length': '0'} 2023-10-11T15:14:20.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139f64e33cb3ea0a-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.182 DEBUG:boto:path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.183 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.184 DEBUG:boto:Method: GET 2023-10-11T15:14:20.184 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.184 DEBUG:boto:Data: 2023-10-11T15:14:20.185 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.186 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.186 DEBUG:boto:Params: {} 2023-10-11T15:14:20.187 DEBUG:boto:Token: None 2023-10-11T15:14:20.187 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000022 2023-10-11T15:14:20.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6jpSjT+u0IVmOUrlwBr7QPXtH6Y= 2023-10-11T15:14:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6jpSjT+u0IVmOUrlwBr7QPXtH6Y=', 'Content-Length': '0'} 2023-10-11T15:14:20.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe812ac3aa67d66-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.191 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.192 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.192 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.192 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.193 DEBUG:boto:Method: GET 2023-10-11T15:14:20.193 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.194 DEBUG:boto:Data: 2023-10-11T15:14:20.194 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.195 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.195 DEBUG:boto:Params: {} 2023-10-11T15:14:20.196 DEBUG:boto:Token: None 2023-10-11T15:14:20.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005791cb016b07d35a-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.201 DEBUG:boto:b'dghqvd-58obj000000000000000000221000falseobj000000000000000000222023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.201 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.202 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.202 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.203 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.203 DEBUG:boto:Method: GET 2023-10-11T15:14:20.204 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000022 2023-10-11T15:14:20.204 DEBUG:boto:Data: 2023-10-11T15:14:20.204 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.205 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.205 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.206 DEBUG:boto:Params: {} 2023-10-11T15:14:20.206 DEBUG:boto:Token: None 2023-10-11T15:14:20.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afd9bfd1c21caa45-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.212 DEBUG:boto:b'dghqvd-58obj000000000000000000221000falseobj000000000000000000222023-10-11T15:14:03.865Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-11T15:14:20.213 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.214 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.214 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.215 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.215 DEBUG:boto:Method: GET 2023-10-11T15:14:20.216 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.216 DEBUG:boto:Data: 2023-10-11T15:14:20.216 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.217 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.217 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.218 DEBUG:boto:Params: {} 2023-10-11T15:14:20.218 DEBUG:boto:Token: None 2023-10-11T15:14:20.219 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.219 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE= 2023-10-11T15:14:20.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE=', 'Content-Length': '0'} 2023-10-11T15:14:20.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bc57003ef66c811-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.223 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.223 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.223 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.224 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.224 DEBUG:boto:Method: GET 2023-10-11T15:14:20.225 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.225 DEBUG:boto:Data: 2023-10-11T15:14:20.226 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.226 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.227 DEBUG:boto:Params: {} 2023-10-11T15:14:20.227 DEBUG:boto:Token: None 2023-10-11T15:14:20.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE= 2023-10-11T15:14:20.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE=', 'Content-Length': '0'} 2023-10-11T15:14:20.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69a8f9d602de485-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.232 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.232 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.232 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.233 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.233 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.234 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.234 DEBUG:boto:Data: 2023-10-11T15:14:20.234 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.235 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.236 DEBUG:boto:Params: {} 2023-10-11T15:14:20.236 DEBUG:boto:Token: None 2023-10-11T15:14:20.237 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzWvlE3grRja3bty/Ur0qho7PsA= 2023-10-11T15:14:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzWvlE3grRja3bty/Ur0qho7PsA=', 'Content-Length': '0'} 2023-10-11T15:14:20.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b50d64fa308eebc-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.241 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.241 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.241 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.242 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.242 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.243 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.243 DEBUG:boto:Data: 2023-10-11T15:14:20.243 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.244 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.245 DEBUG:boto:Params: {} 2023-10-11T15:14:20.245 DEBUG:boto:Token: None 2023-10-11T15:14:20.246 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzWvlE3grRja3bty/Ur0qho7PsA= 2023-10-11T15:14:20.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzWvlE3grRja3bty/Ur0qho7PsA=', 'Content-Length': '0'} 2023-10-11T15:14:20.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35954868e160788-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-11T15:14:20.249 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.250 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.250 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.251 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.252 DEBUG:boto:Method: GET 2023-10-11T15:14:20.252 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.252 DEBUG:boto:Data: 2023-10-11T15:14:20.253 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.254 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.254 DEBUG:boto:Params: {} 2023-10-11T15:14:20.255 DEBUG:boto:Token: None 2023-10-11T15:14:20.255 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE= 2023-10-11T15:14:20.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE=', 'Content-Length': '0'} 2023-10-11T15:14:20.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cda771e4a1b23cfc-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.259 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.259 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.260 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.260 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.261 DEBUG:boto:Method: GET 2023-10-11T15:14:20.261 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.262 DEBUG:boto:Data: 2023-10-11T15:14:20.262 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.263 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.263 DEBUG:boto:Params: {} 2023-10-11T15:14:20.264 DEBUG:boto:Token: None 2023-10-11T15:14:20.264 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023?versionId=null 2023-10-11T15:14:20.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE= 2023-10-11T15:14:20.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctPbhv3ZzP0X/iMgQrEVlC+/mVE=', 'Content-Length': '0'} 2023-10-11T15:14:20.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e766f10e9122ecf-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.314 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.315 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.315 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.316 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.316 DEBUG:boto:Method: GET 2023-10-11T15:14:20.317 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.317 DEBUG:boto:Data: 2023-10-11T15:14:20.318 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.319 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.319 DEBUG:boto:Params: {} 2023-10-11T15:14:20.319 DEBUG:boto:Token: None 2023-10-11T15:14:20.320 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc3b896d208c348b-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.324 DEBUG:boto:b'dghqvd-58obj000000000000000000231000falseobj000000000000000000232023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.325 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.325 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.326 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.326 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.327 DEBUG:boto:Method: GET 2023-10-11T15:14:20.327 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.327 DEBUG:boto:Data: 2023-10-11T15:14:20.328 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.329 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.329 DEBUG:boto:Params: {} 2023-10-11T15:14:20.330 DEBUG:boto:Token: None 2023-10-11T15:14:20.330 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2fd459d97d920ef-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.336 DEBUG:boto:b'dghqvd-58obj000000000000000000231000falseobj000000000000000000232023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.337 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.337 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.337 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.338 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.338 DEBUG:boto:Data: 2023-10-11T15:14:20.339 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.339 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.340 DEBUG:boto:Params: {} 2023-10-11T15:14:20.340 DEBUG:boto:Token: None 2023-10-11T15:14:20.341 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BkPXVKmaqbN+qGAIBiNcwRUiJqU= 2023-10-11T15:14:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BkPXVKmaqbN+qGAIBiNcwRUiJqU=', 'Content-Length': '0'} 2023-10-11T15:14:20.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b84b66705b39f83f-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.344 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.345 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.345 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.346 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.346 DEBUG:boto:Data: 2023-10-11T15:14:20.346 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.347 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.347 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.348 DEBUG:boto:Params: {} 2023-10-11T15:14:20.348 DEBUG:boto:Token: None 2023-10-11T15:14:20.349 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BkPXVKmaqbN+qGAIBiNcwRUiJqU= 2023-10-11T15:14:20.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BkPXVKmaqbN+qGAIBiNcwRUiJqU=', 'Content-Length': '0'} 2023-10-11T15:14:20.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a93d4fbb4c0412-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-10-11T15:14:20.353 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.353 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.354 DEBUG:boto:Method: GET 2023-10-11T15:14:20.354 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.354 DEBUG:boto:Data: 2023-10-11T15:14:20.355 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.356 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.356 DEBUG:boto:Params: {} 2023-10-11T15:14:20.357 DEBUG:boto:Token: None 2023-10-11T15:14:20.357 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+fMxKWyKQCjO/zYk77K7NYaaahk= 2023-10-11T15:14:20.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+fMxKWyKQCjO/zYk77K7NYaaahk=', 'Content-Length': '0'} 2023-10-11T15:14:20.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037cecf0ff5bdc72a-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.402 DEBUG:boto:path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.403 DEBUG:boto:Method: GET 2023-10-11T15:14:20.404 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.404 DEBUG:boto:Data: 2023-10-11T15:14:20.405 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.406 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.406 DEBUG:boto:Params: {} 2023-10-11T15:14:20.407 DEBUG:boto:Token: None 2023-10-11T15:14:20.407 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000023 2023-10-11T15:14:20.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+fMxKWyKQCjO/zYk77K7NYaaahk= 2023-10-11T15:14:20.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+fMxKWyKQCjO/zYk77K7NYaaahk=', 'Content-Length': '0'} 2023-10-11T15:14:20.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc406501de7849a-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.411 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.412 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.412 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.413 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.413 DEBUG:boto:Method: GET 2023-10-11T15:14:20.414 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.414 DEBUG:boto:Data: 2023-10-11T15:14:20.414 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.415 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.416 DEBUG:boto:Params: {} 2023-10-11T15:14:20.416 DEBUG:boto:Token: None 2023-10-11T15:14:20.417 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000152f922077873646-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.422 DEBUG:boto:b'dghqvd-58obj000000000000000000231000falseobj000000000000000000232023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.423 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.424 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.424 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.424 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.425 DEBUG:boto:Method: GET 2023-10-11T15:14:20.425 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000023 2023-10-11T15:14:20.426 DEBUG:boto:Data: 2023-10-11T15:14:20.426 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.427 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.427 DEBUG:boto:Params: {} 2023-10-11T15:14:20.428 DEBUG:boto:Token: None 2023-10-11T15:14:20.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585b08631f9cb87d-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.482 DEBUG:boto:b'dghqvd-58obj000000000000000000231000falseobj000000000000000000232023-10-11T15:14:03.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-11T15:14:20.484 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.484 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.485 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.485 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.485 DEBUG:boto:Method: GET 2023-10-11T15:14:20.486 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.486 DEBUG:boto:Data: 2023-10-11T15:14:20.487 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.488 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.488 DEBUG:boto:Params: {} 2023-10-11T15:14:20.488 DEBUG:boto:Token: None 2023-10-11T15:14:20.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE= 2023-10-11T15:14:20.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE=', 'Content-Length': '0'} 2023-10-11T15:14:20.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cfd251a8b2b0132-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.493 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.494 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.494 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.494 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.495 DEBUG:boto:Method: GET 2023-10-11T15:14:20.495 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.496 DEBUG:boto:Data: 2023-10-11T15:14:20.496 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.497 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.497 DEBUG:boto:Params: {} 2023-10-11T15:14:20.498 DEBUG:boto:Token: None 2023-10-11T15:14:20.498 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE= 2023-10-11T15:14:20.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE=', 'Content-Length': '0'} 2023-10-11T15:14:20.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37b4c1e199ab3ac-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.502 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.502 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.503 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.503 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.504 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.504 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.504 DEBUG:boto:Data: 2023-10-11T15:14:20.505 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.505 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.506 DEBUG:boto:Params: {} 2023-10-11T15:14:20.506 DEBUG:boto:Token: None 2023-10-11T15:14:20.507 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zciJcYfc8LnWrVb+Z8DFQ5CI5kQ= 2023-10-11T15:14:20.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zciJcYfc8LnWrVb+Z8DFQ5CI5kQ=', 'Content-Length': '0'} 2023-10-11T15:14:20.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b59e82dc9cdb989-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.510 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.511 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.511 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.512 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.512 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.512 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.513 DEBUG:boto:Data: 2023-10-11T15:14:20.513 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.514 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.514 DEBUG:boto:Params: {} 2023-10-11T15:14:20.515 DEBUG:boto:Token: None 2023-10-11T15:14:20.515 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zciJcYfc8LnWrVb+Z8DFQ5CI5kQ= 2023-10-11T15:14:20.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zciJcYfc8LnWrVb+Z8DFQ5CI5kQ=', 'Content-Length': '0'} 2023-10-11T15:14:20.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba81bff94ec0f71e-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-11T15:14:20.519 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.520 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.521 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.521 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.521 DEBUG:boto:Method: GET 2023-10-11T15:14:20.522 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.522 DEBUG:boto:Data: 2023-10-11T15:14:20.523 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.523 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.523 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.524 DEBUG:boto:Params: {} 2023-10-11T15:14:20.524 DEBUG:boto:Token: None 2023-10-11T15:14:20.525 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE= 2023-10-11T15:14:20.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE=', 'Content-Length': '0'} 2023-10-11T15:14:20.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7ab0cba1f149f9-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.528 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.529 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.529 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.530 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.530 DEBUG:boto:Method: GET 2023-10-11T15:14:20.530 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.531 DEBUG:boto:Data: 2023-10-11T15:14:20.531 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.532 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.533 DEBUG:boto:Params: {} 2023-10-11T15:14:20.533 DEBUG:boto:Token: None 2023-10-11T15:14:20.534 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024?versionId=null 2023-10-11T15:14:20.534 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE= 2023-10-11T15:14:20.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+sm6hQq49bbweFzlA4a6RUXdRsE=', 'Content-Length': '0'} 2023-10-11T15:14:20.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc9bbf58d7f964a-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.537 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.538 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.538 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.539 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.539 DEBUG:boto:Method: GET 2023-10-11T15:14:20.539 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.540 DEBUG:boto:Data: 2023-10-11T15:14:20.540 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.541 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.541 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.542 DEBUG:boto:Params: {} 2023-10-11T15:14:20.542 DEBUG:boto:Token: None 2023-10-11T15:14:20.543 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f453d43785da1bc-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.547 DEBUG:boto:b'dghqvd-58obj000000000000000000241000falseobj000000000000000000242023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.547 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.548 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.548 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.549 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.549 DEBUG:boto:Method: GET 2023-10-11T15:14:20.550 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.550 DEBUG:boto:Data: 2023-10-11T15:14:20.550 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.551 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.552 DEBUG:boto:Params: {} 2023-10-11T15:14:20.552 DEBUG:boto:Token: None 2023-10-11T15:14:20.553 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040f71d65bb7ab912-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.558 DEBUG:boto:b'dghqvd-58obj000000000000000000241000falseobj000000000000000000242023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.559 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.559 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.560 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.560 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.561 DEBUG:boto:Data: 2023-10-11T15:14:20.561 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.562 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.562 DEBUG:boto:Params: {} 2023-10-11T15:14:20.563 DEBUG:boto:Token: None 2023-10-11T15:14:20.563 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:83zJuVjE+8Tyuxq0FIi5CmxQU9s= 2023-10-11T15:14:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:83zJuVjE+8Tyuxq0FIi5CmxQU9s=', 'Content-Length': '0'} 2023-10-11T15:14:20.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726f40df825cdcba-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.567 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.567 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.567 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.568 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.568 DEBUG:boto:Data: 2023-10-11T15:14:20.569 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.569 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.570 DEBUG:boto:Params: {} 2023-10-11T15:14:20.570 DEBUG:boto:Token: None 2023-10-11T15:14:20.571 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:83zJuVjE+8Tyuxq0FIi5CmxQU9s= 2023-10-11T15:14:20.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:83zJuVjE+8Tyuxq0FIi5CmxQU9s=', 'Content-Length': '0'} 2023-10-11T15:14:20.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9b9b55f17897dc-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-10-11T15:14:20.575 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.575 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.576 DEBUG:boto:Method: GET 2023-10-11T15:14:20.576 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.577 DEBUG:boto:Data: 2023-10-11T15:14:20.577 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.578 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.578 DEBUG:boto:Params: {} 2023-10-11T15:14:20.579 DEBUG:boto:Token: None 2023-10-11T15:14:20.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:C+VZWCmj1w1gQBX2faFpO/m3PPw= 2023-10-11T15:14:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C+VZWCmj1w1gQBX2faFpO/m3PPw=', 'Content-Length': '0'} 2023-10-11T15:14:20.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4f03515d8a6b37-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.585 DEBUG:boto:path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.586 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.586 DEBUG:boto:Method: GET 2023-10-11T15:14:20.586 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.587 DEBUG:boto:Data: 2023-10-11T15:14:20.587 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.588 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.589 DEBUG:boto:Params: {} 2023-10-11T15:14:20.589 DEBUG:boto:Token: None 2023-10-11T15:14:20.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000024 2023-10-11T15:14:20.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:C+VZWCmj1w1gQBX2faFpO/m3PPw= 2023-10-11T15:14:20.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C+VZWCmj1w1gQBX2faFpO/m3PPw=', 'Content-Length': '0'} 2023-10-11T15:14:20.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8fb734f24b3ce6a-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.593 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.594 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.594 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.595 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.595 DEBUG:boto:Method: GET 2023-10-11T15:14:20.596 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.596 DEBUG:boto:Data: 2023-10-11T15:14:20.596 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.597 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.598 DEBUG:boto:Params: {} 2023-10-11T15:14:20.598 DEBUG:boto:Token: None 2023-10-11T15:14:20.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000188975ea6f13701a-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.603 DEBUG:boto:b'dghqvd-58obj000000000000000000241000falseobj000000000000000000242023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.604 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.604 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.604 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.605 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.605 DEBUG:boto:Method: GET 2023-10-11T15:14:20.606 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000024 2023-10-11T15:14:20.606 DEBUG:boto:Data: 2023-10-11T15:14:20.607 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.607 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.608 DEBUG:boto:Params: {} 2023-10-11T15:14:20.608 DEBUG:boto:Token: None 2023-10-11T15:14:20.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b7f9bafff93b08-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.614 DEBUG:boto:b'dghqvd-58obj000000000000000000241000falseobj000000000000000000242023-10-11T15:14:05.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-11T15:14:20.615 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.616 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.616 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.616 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.617 DEBUG:boto:Method: GET 2023-10-11T15:14:20.617 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.618 DEBUG:boto:Data: 2023-10-11T15:14:20.618 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.619 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.619 DEBUG:boto:Params: {} 2023-10-11T15:14:20.620 DEBUG:boto:Token: None 2023-10-11T15:14:20.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw= 2023-10-11T15:14:20.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw=', 'Content-Length': '0'} 2023-10-11T15:14:20.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd361665df1b7ac1-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.666 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.667 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.668 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.668 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.668 DEBUG:boto:Method: GET 2023-10-11T15:14:20.669 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.669 DEBUG:boto:Data: 2023-10-11T15:14:20.670 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.671 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.671 DEBUG:boto:Params: {} 2023-10-11T15:14:20.672 DEBUG:boto:Token: None 2023-10-11T15:14:20.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw= 2023-10-11T15:14:20.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw=', 'Content-Length': '0'} 2023-10-11T15:14:20.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031962babadc2183b-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.676 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.677 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.677 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.677 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.678 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.678 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.679 DEBUG:boto:Data: 2023-10-11T15:14:20.679 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.680 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.680 DEBUG:boto:Params: {} 2023-10-11T15:14:20.681 DEBUG:boto:Token: None 2023-10-11T15:14:20.681 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WbOUoghxgefj9Fgl8g8+pMPzOeg= 2023-10-11T15:14:20.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WbOUoghxgefj9Fgl8g8+pMPzOeg=', 'Content-Length': '0'} 2023-10-11T15:14:20.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b271e45e5ccb230b-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.685 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.685 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.685 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.686 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.686 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.687 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.687 DEBUG:boto:Data: 2023-10-11T15:14:20.688 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.688 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.689 DEBUG:boto:Params: {} 2023-10-11T15:14:20.689 DEBUG:boto:Token: None 2023-10-11T15:14:20.690 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WbOUoghxgefj9Fgl8g8+pMPzOeg= 2023-10-11T15:14:20.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WbOUoghxgefj9Fgl8g8+pMPzOeg=', 'Content-Length': '0'} 2023-10-11T15:14:20.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b554df18dafe41-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-11T15:14:20.694 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.694 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.695 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.695 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.696 DEBUG:boto:Method: GET 2023-10-11T15:14:20.696 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.696 DEBUG:boto:Data: 2023-10-11T15:14:20.697 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.698 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.698 DEBUG:boto:Params: {} 2023-10-11T15:14:20.699 DEBUG:boto:Token: None 2023-10-11T15:14:20.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw= 2023-10-11T15:14:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw=', 'Content-Length': '0'} 2023-10-11T15:14:20.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b963c9e127298b09-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.703 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.703 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.704 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.704 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.705 DEBUG:boto:Method: GET 2023-10-11T15:14:20.705 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.705 DEBUG:boto:Data: 2023-10-11T15:14:20.706 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.707 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.707 DEBUG:boto:Params: {} 2023-10-11T15:14:20.708 DEBUG:boto:Token: None 2023-10-11T15:14:20.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025?versionId=null 2023-10-11T15:14:20.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw= 2023-10-11T15:14:20.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UkcUPQyigfOgeYPQQUdGfpuqDxw=', 'Content-Length': '0'} 2023-10-11T15:14:20.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df912d8aea48c0fd-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.758 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.759 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.759 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.760 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.760 DEBUG:boto:Method: GET 2023-10-11T15:14:20.761 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.762 DEBUG:boto:Data: 2023-10-11T15:14:20.762 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.763 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.763 DEBUG:boto:Params: {} 2023-10-11T15:14:20.764 DEBUG:boto:Token: None 2023-10-11T15:14:20.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103118044d7b910f-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.769 DEBUG:boto:b'dghqvd-58obj000000000000000000251000falseobj000000000000000000252023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.769 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.770 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.770 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.771 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.771 DEBUG:boto:Method: GET 2023-10-11T15:14:20.772 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.772 DEBUG:boto:Data: 2023-10-11T15:14:20.773 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.773 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.774 DEBUG:boto:Params: {} 2023-10-11T15:14:20.774 DEBUG:boto:Token: None 2023-10-11T15:14:20.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.775 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b70aeddb8262c0-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.781 DEBUG:boto:b'dghqvd-58obj000000000000000000251000falseobj000000000000000000252023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.781 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.782 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.782 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.783 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.783 DEBUG:boto:Data: 2023-10-11T15:14:20.783 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.784 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.785 DEBUG:boto:Params: {} 2023-10-11T15:14:20.785 DEBUG:boto:Token: None 2023-10-11T15:14:20.786 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pAx8jg4CPOtAuI7bc2BYG4QIGYQ= 2023-10-11T15:14:20.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pAx8jg4CPOtAuI7bc2BYG4QIGYQ=', 'Content-Length': '0'} 2023-10-11T15:14:20.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543192d534790f2c-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.789 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.790 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.790 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.791 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.791 DEBUG:boto:Data: 2023-10-11T15:14:20.791 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.792 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.793 DEBUG:boto:Params: {} 2023-10-11T15:14:20.793 DEBUG:boto:Token: None 2023-10-11T15:14:20.794 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pAx8jg4CPOtAuI7bc2BYG4QIGYQ= 2023-10-11T15:14:20.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pAx8jg4CPOtAuI7bc2BYG4QIGYQ=', 'Content-Length': '0'} 2023-10-11T15:14:20.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218959717a799501-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-10-11T15:14:20.797 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.798 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.798 DEBUG:boto:Method: GET 2023-10-11T15:14:20.799 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.799 DEBUG:boto:Data: 2023-10-11T15:14:20.800 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.800 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.801 DEBUG:boto:Params: {} 2023-10-11T15:14:20.801 DEBUG:boto:Token: None 2023-10-11T15:14:20.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFibake/QqhNdVGrfsBinOxJIf8= 2023-10-11T15:14:20.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFibake/QqhNdVGrfsBinOxJIf8=', 'Content-Length': '0'} 2023-10-11T15:14:20.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2fd597b422f3bd-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.805 DEBUG:boto:path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.805 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.806 DEBUG:boto:Method: GET 2023-10-11T15:14:20.806 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.807 DEBUG:boto:Data: 2023-10-11T15:14:20.807 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.808 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.809 DEBUG:boto:Params: {} 2023-10-11T15:14:20.809 DEBUG:boto:Token: None 2023-10-11T15:14:20.809 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000025 2023-10-11T15:14:20.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zFibake/QqhNdVGrfsBinOxJIf8= 2023-10-11T15:14:20.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zFibake/QqhNdVGrfsBinOxJIf8=', 'Content-Length': '0'} 2023-10-11T15:14:20.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab73ee65e2a7bf5-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.813 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.813 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.814 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.814 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.815 DEBUG:boto:Method: GET 2023-10-11T15:14:20.815 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.815 DEBUG:boto:Data: 2023-10-11T15:14:20.816 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.817 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.817 DEBUG:boto:Params: {} 2023-10-11T15:14:20.818 DEBUG:boto:Token: None 2023-10-11T15:14:20.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771298adef82eead-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.822 DEBUG:boto:b'dghqvd-58obj000000000000000000251000falseobj000000000000000000252023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.823 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.823 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.824 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.824 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.825 DEBUG:boto:Method: GET 2023-10-11T15:14:20.825 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000025 2023-10-11T15:14:20.826 DEBUG:boto:Data: 2023-10-11T15:14:20.826 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.827 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.827 DEBUG:boto:Params: {} 2023-10-11T15:14:20.828 DEBUG:boto:Token: None 2023-10-11T15:14:20.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000811e7d61ecfb72aa-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.834 DEBUG:boto:b'dghqvd-58obj000000000000000000251000falseobj000000000000000000252023-10-11T15:14:02.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-11T15:14:20.835 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.835 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.836 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.836 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.837 DEBUG:boto:Method: GET 2023-10-11T15:14:20.837 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.837 DEBUG:boto:Data: 2023-10-11T15:14:20.838 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.839 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.839 DEBUG:boto:Params: {} 2023-10-11T15:14:20.840 DEBUG:boto:Token: None 2023-10-11T15:14:20.840 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.841 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I= 2023-10-11T15:14:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I=', 'Content-Length': '0'} 2023-10-11T15:14:20.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000534bb1834fb8f68c-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.844 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.845 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.845 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.845 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.846 DEBUG:boto:Method: GET 2023-10-11T15:14:20.846 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.847 DEBUG:boto:Data: 2023-10-11T15:14:20.847 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.848 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.848 DEBUG:boto:Params: {} 2023-10-11T15:14:20.849 DEBUG:boto:Token: None 2023-10-11T15:14:20.849 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I= 2023-10-11T15:14:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I=', 'Content-Length': '0'} 2023-10-11T15:14:20.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3d4e0766a684b67-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.898 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.899 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.899 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.900 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.900 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.900 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.901 DEBUG:boto:Data: 2023-10-11T15:14:20.901 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.902 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.902 DEBUG:boto:Params: {} 2023-10-11T15:14:20.903 DEBUG:boto:Token: None 2023-10-11T15:14:20.903 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yPAX7VW5EjAaWnRSNmsRRbrC3mY= 2023-10-11T15:14:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yPAX7VW5EjAaWnRSNmsRRbrC3mY=', 'Content-Length': '0'} 2023-10-11T15:14:20.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e788aa57dcafc36b-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.907 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.907 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.908 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.908 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.909 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.909 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.909 DEBUG:boto:Data: 2023-10-11T15:14:20.910 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.911 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.911 DEBUG:boto:Params: {} 2023-10-11T15:14:20.911 DEBUG:boto:Token: None 2023-10-11T15:14:20.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yPAX7VW5EjAaWnRSNmsRRbrC3mY= 2023-10-11T15:14:20.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yPAX7VW5EjAaWnRSNmsRRbrC3mY=', 'Content-Length': '0'} 2023-10-11T15:14:20.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db12e49bfe79f7f-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-11T15:14:20.916 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.916 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.916 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.917 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.917 DEBUG:boto:Method: GET 2023-10-11T15:14:20.918 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.918 DEBUG:boto:Data: 2023-10-11T15:14:20.919 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.919 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.920 DEBUG:boto:Params: {} 2023-10-11T15:14:20.920 DEBUG:boto:Token: None 2023-10-11T15:14:20.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I= 2023-10-11T15:14:20.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I=', 'Content-Length': '0'} 2023-10-11T15:14:20.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d40028b95f14e4-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.924 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.925 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.925 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.926 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.926 DEBUG:boto:Method: GET 2023-10-11T15:14:20.926 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.927 DEBUG:boto:Data: 2023-10-11T15:14:20.927 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.928 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.929 DEBUG:boto:Params: {} 2023-10-11T15:14:20.929 DEBUG:boto:Token: None 2023-10-11T15:14:20.930 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026?versionId=null 2023-10-11T15:14:20.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I= 2023-10-11T15:14:20.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yyG7xmQE/Dn2QgIA8FFLqP2B09I=', 'Content-Length': '0'} 2023-10-11T15:14:20.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703021630f010aef-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.933 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.934 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.934 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.934 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.935 DEBUG:boto:Method: GET 2023-10-11T15:14:20.935 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.936 DEBUG:boto:Data: 2023-10-11T15:14:20.936 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.937 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.937 DEBUG:boto:Params: {} 2023-10-11T15:14:20.938 DEBUG:boto:Token: None 2023-10-11T15:14:20.938 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaac054a5c7269e2-006526bbcc-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.942 DEBUG:boto:b'dghqvd-58obj000000000000000000261000falseobj000000000000000000262023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.943 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:20.943 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:20.944 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.944 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.945 DEBUG:boto:Method: GET 2023-10-11T15:14:20.945 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:20.945 DEBUG:boto:Data: 2023-10-11T15:14:20.946 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.947 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.947 DEBUG:boto:Params: {} 2023-10-11T15:14:20.948 DEBUG:boto:Token: None 2023-10-11T15:14:20.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/ 2023-10-11T15:14:20.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU= 2023-10-11T15:14:20.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hHH7ysXRL8/inStlD/KKu4V0xRU=', 'Content-Length': '0'} 2023-10-11T15:14:20.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4b7e92659aeedd-006526bbcc-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.953 DEBUG:boto:b'dghqvd-58obj000000000000000000261000falseobj000000000000000000262023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:20.954 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.955 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.955 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.955 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.956 DEBUG:boto:Data: 2023-10-11T15:14:20.956 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.956 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.957 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.957 DEBUG:boto:Params: {} 2023-10-11T15:14:20.958 DEBUG:boto:Token: None 2023-10-11T15:14:20.958 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG12P7I8KMcDbU7BilAFf+ZC54M= 2023-10-11T15:14:20.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG12P7I8KMcDbU7BilAFf+ZC54M=', 'Content-Length': '0'} 2023-10-11T15:14:20.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d2f443c3fc7e56-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.962 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.962 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.963 DEBUG:boto:Method: HEAD 2023-10-11T15:14:20.963 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.963 DEBUG:boto:Data: 2023-10-11T15:14:20.964 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.965 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.965 DEBUG:boto:Params: {} 2023-10-11T15:14:20.965 DEBUG:boto:Token: None 2023-10-11T15:14:20.966 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FG12P7I8KMcDbU7BilAFf+ZC54M= 2023-10-11T15:14:20.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FG12P7I8KMcDbU7BilAFf+ZC54M=', 'Content-Length': '0'} 2023-10-11T15:14:20.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68c5ba7f77d352e-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-10-11T15:14:20.970 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.970 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.971 DEBUG:boto:Method: GET 2023-10-11T15:14:20.971 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.972 DEBUG:boto:Data: 2023-10-11T15:14:20.972 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:20.973 DEBUG:boto:Port: 8001 2023-10-11T15:14:20.973 DEBUG:boto:Params: {} 2023-10-11T15:14:20.974 DEBUG:boto:Token: None 2023-10-11T15:14:20.974 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZLLoxEufAUe5OvOZaytw7aZyy4s= 2023-10-11T15:14:20.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZLLoxEufAUe5OvOZaytw7aZyy4s=', 'Content-Length': '0'} 2023-10-11T15:14:20.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2e243fa1e65ac7-006526bbcc-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:20.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.978 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.979 DEBUG:boto:Method: GET 2023-10-11T15:14:20.979 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.980 DEBUG:boto:Data: 2023-10-11T15:14:20.980 DEBUG:boto:Headers: {} 2023-10-11T15:14:20.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:20.981 DEBUG:boto:Port: 8000 2023-10-11T15:14:20.981 DEBUG:boto:Params: {} 2023-10-11T15:14:20.982 DEBUG:boto:Token: None 2023-10-11T15:14:20.982 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:20 GMT /dghqvd-58/obj00000000000000000026 2023-10-11T15:14:20.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZLLoxEufAUe5OvOZaytw7aZyy4s= 2023-10-11T15:14:20.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZLLoxEufAUe5OvOZaytw7aZyy4s=', 'Content-Length': '0'} 2023-10-11T15:14:20.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071186a8daf96f1a6-006526bbcc-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.030 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.031 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.031 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.032 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.032 DEBUG:boto:Method: GET 2023-10-11T15:14:21.033 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.033 DEBUG:boto:Data: 2023-10-11T15:14:21.033 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.034 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.035 DEBUG:boto:Params: {} 2023-10-11T15:14:21.035 DEBUG:boto:Token: None 2023-10-11T15:14:21.036 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca813e112c22041e-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.040 DEBUG:boto:b'dghqvd-58obj000000000000000000261000falseobj000000000000000000262023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.041 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.041 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.041 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.042 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.042 DEBUG:boto:Method: GET 2023-10-11T15:14:21.043 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000026 2023-10-11T15:14:21.043 DEBUG:boto:Data: 2023-10-11T15:14:21.044 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.044 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.045 DEBUG:boto:Params: {} 2023-10-11T15:14:21.045 DEBUG:boto:Token: None 2023-10-11T15:14:21.046 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa833f8a7398919-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.051 DEBUG:boto:b'dghqvd-58obj000000000000000000261000falseobj000000000000000000262023-10-11T15:14:00.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-11T15:14:21.052 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.053 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.053 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.054 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.054 DEBUG:boto:Method: GET 2023-10-11T15:14:21.055 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.055 DEBUG:boto:Data: 2023-10-11T15:14:21.056 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.056 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.057 DEBUG:boto:Params: {} 2023-10-11T15:14:21.057 DEBUG:boto:Token: None 2023-10-11T15:14:21.058 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac= 2023-10-11T15:14:21.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac=', 'Content-Length': '0'} 2023-10-11T15:14:21.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d645298154e28d-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.062 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.062 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.062 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.063 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.063 DEBUG:boto:Method: GET 2023-10-11T15:14:21.064 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.064 DEBUG:boto:Data: 2023-10-11T15:14:21.064 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.065 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.065 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.066 DEBUG:boto:Params: {} 2023-10-11T15:14:21.066 DEBUG:boto:Token: None 2023-10-11T15:14:21.067 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac= 2023-10-11T15:14:21.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac=', 'Content-Length': '0'} 2023-10-11T15:14:21.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d0098fed09c973-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.071 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.071 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.071 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.072 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.072 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.073 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.073 DEBUG:boto:Data: 2023-10-11T15:14:21.073 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.074 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.075 DEBUG:boto:Params: {} 2023-10-11T15:14:21.075 DEBUG:boto:Token: None 2023-10-11T15:14:21.076 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bMSJDYdj0hMw+Nrq7GK9O5aBAT4= 2023-10-11T15:14:21.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bMSJDYdj0hMw+Nrq7GK9O5aBAT4=', 'Content-Length': '0'} 2023-10-11T15:14:21.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330510a256faae1a-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.079 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.080 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.080 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.080 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.081 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.081 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.081 DEBUG:boto:Data: 2023-10-11T15:14:21.082 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.083 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.083 DEBUG:boto:Params: {} 2023-10-11T15:14:21.084 DEBUG:boto:Token: None 2023-10-11T15:14:21.084 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bMSJDYdj0hMw+Nrq7GK9O5aBAT4= 2023-10-11T15:14:21.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bMSJDYdj0hMw+Nrq7GK9O5aBAT4=', 'Content-Length': '0'} 2023-10-11T15:14:21.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe674ef3674743f9-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-11T15:14:21.088 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.088 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.089 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.089 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.090 DEBUG:boto:Method: GET 2023-10-11T15:14:21.090 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.090 DEBUG:boto:Data: 2023-10-11T15:14:21.091 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.092 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.092 DEBUG:boto:Params: {} 2023-10-11T15:14:21.093 DEBUG:boto:Token: None 2023-10-11T15:14:21.093 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac= 2023-10-11T15:14:21.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac=', 'Content-Length': '0'} 2023-10-11T15:14:21.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae7f7d34adeeceda-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.138 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.139 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.139 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.140 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.140 DEBUG:boto:Method: GET 2023-10-11T15:14:21.141 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.141 DEBUG:boto:Data: 2023-10-11T15:14:21.141 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.143 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.143 DEBUG:boto:Params: {} 2023-10-11T15:14:21.144 DEBUG:boto:Token: None 2023-10-11T15:14:21.144 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027?versionId=null 2023-10-11T15:14:21.145 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac= 2023-10-11T15:14:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vnwqNlUeSjDEDZHcZ0WJCR6oXac=', 'Content-Length': '0'} 2023-10-11T15:14:21.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f138b533bf9b12-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.148 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.148 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.149 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.149 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.150 DEBUG:boto:Method: GET 2023-10-11T15:14:21.150 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.151 DEBUG:boto:Data: 2023-10-11T15:14:21.151 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.152 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.152 DEBUG:boto:Params: {} 2023-10-11T15:14:21.153 DEBUG:boto:Token: None 2023-10-11T15:14:21.153 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e025bce41ab50c0-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.157 DEBUG:boto:b'dghqvd-58obj000000000000000000271000falseobj000000000000000000272023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.158 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.158 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.159 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.159 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.159 DEBUG:boto:Method: GET 2023-10-11T15:14:21.160 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.160 DEBUG:boto:Data: 2023-10-11T15:14:21.161 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.161 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.161 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.162 DEBUG:boto:Params: {} 2023-10-11T15:14:21.162 DEBUG:boto:Token: None 2023-10-11T15:14:21.163 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748276c770802beb-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.214 DEBUG:boto:b'dghqvd-58obj000000000000000000271000falseobj000000000000000000272023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.215 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.216 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.216 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.217 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.217 DEBUG:boto:Data: 2023-10-11T15:14:21.217 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.218 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.219 DEBUG:boto:Params: {} 2023-10-11T15:14:21.219 DEBUG:boto:Token: None 2023-10-11T15:14:21.220 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cE+cAlZgkNOo+dPlX40jfGqh4Zo= 2023-10-11T15:14:21.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cE+cAlZgkNOo+dPlX40jfGqh4Zo=', 'Content-Length': '0'} 2023-10-11T15:14:21.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8418d8f6e8f7df-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.223 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.224 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.224 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.225 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.225 DEBUG:boto:Data: 2023-10-11T15:14:21.225 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.226 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.227 DEBUG:boto:Params: {} 2023-10-11T15:14:21.227 DEBUG:boto:Token: None 2023-10-11T15:14:21.227 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cE+cAlZgkNOo+dPlX40jfGqh4Zo= 2023-10-11T15:14:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cE+cAlZgkNOo+dPlX40jfGqh4Zo=', 'Content-Length': '0'} 2023-10-11T15:14:21.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8dd4597aa95b0c0-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-10-11T15:14:21.231 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.232 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.232 DEBUG:boto:Method: GET 2023-10-11T15:14:21.233 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.233 DEBUG:boto:Data: 2023-10-11T15:14:21.236 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.237 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.237 DEBUG:boto:Params: {} 2023-10-11T15:14:21.239 DEBUG:boto:Token: None 2023-10-11T15:14:21.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K4EDdDV9rwrTORmWTvkPEeHWcVE= 2023-10-11T15:14:21.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K4EDdDV9rwrTORmWTvkPEeHWcVE=', 'Content-Length': '0'} 2023-10-11T15:14:21.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562dd18b5b7a53d1-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.243 DEBUG:boto:path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.244 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.244 DEBUG:boto:Method: GET 2023-10-11T15:14:21.244 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.245 DEBUG:boto:Data: 2023-10-11T15:14:21.245 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.246 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.247 DEBUG:boto:Params: {} 2023-10-11T15:14:21.247 DEBUG:boto:Token: None 2023-10-11T15:14:21.248 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000027 2023-10-11T15:14:21.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K4EDdDV9rwrTORmWTvkPEeHWcVE= 2023-10-11T15:14:21.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K4EDdDV9rwrTORmWTvkPEeHWcVE=', 'Content-Length': '0'} 2023-10-11T15:14:21.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238b7772247084b5-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.251 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.252 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.252 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.253 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.253 DEBUG:boto:Method: GET 2023-10-11T15:14:21.254 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.254 DEBUG:boto:Data: 2023-10-11T15:14:21.254 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.255 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.256 DEBUG:boto:Params: {} 2023-10-11T15:14:21.256 DEBUG:boto:Token: None 2023-10-11T15:14:21.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc092e272d48ae0c-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.261 DEBUG:boto:b'dghqvd-58obj000000000000000000271000falseobj000000000000000000272023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.261 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.262 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.262 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.263 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.263 DEBUG:boto:Method: GET 2023-10-11T15:14:21.264 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000027 2023-10-11T15:14:21.264 DEBUG:boto:Data: 2023-10-11T15:14:21.264 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.266 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.266 DEBUG:boto:Params: {} 2023-10-11T15:14:21.267 DEBUG:boto:Token: None 2023-10-11T15:14:21.267 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b27b6364bf01f9-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.318 DEBUG:boto:b'dghqvd-58obj000000000000000000271000falseobj000000000000000000272023-10-11T15:14:04.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-11T15:14:21.320 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.320 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.321 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.321 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.321 DEBUG:boto:Method: GET 2023-10-11T15:14:21.322 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.322 DEBUG:boto:Data: 2023-10-11T15:14:21.323 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.324 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.324 DEBUG:boto:Params: {} 2023-10-11T15:14:21.324 DEBUG:boto:Token: None 2023-10-11T15:14:21.325 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo= 2023-10-11T15:14:21.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo=', 'Content-Length': '0'} 2023-10-11T15:14:21.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70452969d86b283-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.329 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.330 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.330 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.330 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.331 DEBUG:boto:Method: GET 2023-10-11T15:14:21.331 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.332 DEBUG:boto:Data: 2023-10-11T15:14:21.332 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.333 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.333 DEBUG:boto:Params: {} 2023-10-11T15:14:21.334 DEBUG:boto:Token: None 2023-10-11T15:14:21.334 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo= 2023-10-11T15:14:21.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo=', 'Content-Length': '0'} 2023-10-11T15:14:21.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f32854e9ed3c08d-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.338 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.339 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.339 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.339 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.340 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.340 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.341 DEBUG:boto:Data: 2023-10-11T15:14:21.341 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.342 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.342 DEBUG:boto:Params: {} 2023-10-11T15:14:21.343 DEBUG:boto:Token: None 2023-10-11T15:14:21.343 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:04QcyewPemTDCtkDAS6FaotbSjo= 2023-10-11T15:14:21.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:04QcyewPemTDCtkDAS6FaotbSjo=', 'Content-Length': '0'} 2023-10-11T15:14:21.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006930fd1879ae66b9-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.347 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.347 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.348 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.348 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.349 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.349 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.349 DEBUG:boto:Data: 2023-10-11T15:14:21.350 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.351 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.351 DEBUG:boto:Params: {} 2023-10-11T15:14:21.351 DEBUG:boto:Token: None 2023-10-11T15:14:21.352 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:04QcyewPemTDCtkDAS6FaotbSjo= 2023-10-11T15:14:21.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:04QcyewPemTDCtkDAS6FaotbSjo=', 'Content-Length': '0'} 2023-10-11T15:14:21.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3480dcf31e7afeb-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-11T15:14:21.355 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.356 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.356 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.357 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.357 DEBUG:boto:Method: GET 2023-10-11T15:14:21.357 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.358 DEBUG:boto:Data: 2023-10-11T15:14:21.358 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.359 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.360 DEBUG:boto:Params: {} 2023-10-11T15:14:21.360 DEBUG:boto:Token: None 2023-10-11T15:14:21.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo= 2023-10-11T15:14:21.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo=', 'Content-Length': '0'} 2023-10-11T15:14:21.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41c73cebf7f7ed6-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.364 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.364 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.365 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.365 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.366 DEBUG:boto:Method: GET 2023-10-11T15:14:21.366 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.366 DEBUG:boto:Data: 2023-10-11T15:14:21.367 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.368 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.368 DEBUG:boto:Params: {} 2023-10-11T15:14:21.369 DEBUG:boto:Token: None 2023-10-11T15:14:21.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028?versionId=null 2023-10-11T15:14:21.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo= 2023-10-11T15:14:21.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LzAnIcZKMsqvQIrRIOuVwoHSfvo=', 'Content-Length': '0'} 2023-10-11T15:14:21.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb662c69f46e24d-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.373 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.373 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.374 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.374 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.374 DEBUG:boto:Method: GET 2023-10-11T15:14:21.375 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.375 DEBUG:boto:Data: 2023-10-11T15:14:21.376 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.377 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.377 DEBUG:boto:Params: {} 2023-10-11T15:14:21.378 DEBUG:boto:Token: None 2023-10-11T15:14:21.378 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08ab50b02bfb874-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.382 DEBUG:boto:b'dghqvd-58obj000000000000000000281000falseobj000000000000000000282023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.383 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.383 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.384 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.384 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.385 DEBUG:boto:Method: GET 2023-10-11T15:14:21.385 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.385 DEBUG:boto:Data: 2023-10-11T15:14:21.386 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.387 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.387 DEBUG:boto:Params: {} 2023-10-11T15:14:21.388 DEBUG:boto:Token: None 2023-10-11T15:14:21.388 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007433fa403019a81-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.394 DEBUG:boto:b'dghqvd-58obj000000000000000000281000falseobj000000000000000000282023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.395 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.395 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.396 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.396 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.396 DEBUG:boto:Data: 2023-10-11T15:14:21.397 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.398 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.398 DEBUG:boto:Params: {} 2023-10-11T15:14:21.398 DEBUG:boto:Token: None 2023-10-11T15:14:21.399 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEuc0qlRdb7sgekQcL2w97798Go= 2023-10-11T15:14:21.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEuc0qlRdb7sgekQcL2w97798Go=', 'Content-Length': '0'} 2023-10-11T15:14:21.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052218b1fe7e4f217-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.402 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.403 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.404 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.404 DEBUG:boto:Data: 2023-10-11T15:14:21.404 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.405 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.406 DEBUG:boto:Params: {} 2023-10-11T15:14:21.406 DEBUG:boto:Token: None 2023-10-11T15:14:21.407 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.407 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEuc0qlRdb7sgekQcL2w97798Go= 2023-10-11T15:14:21.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEuc0qlRdb7sgekQcL2w97798Go=', 'Content-Length': '0'} 2023-10-11T15:14:21.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb89ccd399a9335f-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-10-11T15:14:21.411 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.411 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.411 DEBUG:boto:Method: GET 2023-10-11T15:14:21.412 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.412 DEBUG:boto:Data: 2023-10-11T15:14:21.413 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.413 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.414 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.414 DEBUG:boto:Params: {} 2023-10-11T15:14:21.415 DEBUG:boto:Token: None 2023-10-11T15:14:21.415 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R2iZM98iX3426ZEp3BmkQZUIBt0= 2023-10-11T15:14:21.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R2iZM98iX3426ZEp3BmkQZUIBt0=', 'Content-Length': '0'} 2023-10-11T15:14:21.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f503331676151575-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.419 DEBUG:boto:path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.419 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.419 DEBUG:boto:Method: GET 2023-10-11T15:14:21.420 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.420 DEBUG:boto:Data: 2023-10-11T15:14:21.421 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.422 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.422 DEBUG:boto:Params: {} 2023-10-11T15:14:21.423 DEBUG:boto:Token: None 2023-10-11T15:14:21.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000028 2023-10-11T15:14:21.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R2iZM98iX3426ZEp3BmkQZUIBt0= 2023-10-11T15:14:21.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R2iZM98iX3426ZEp3BmkQZUIBt0=', 'Content-Length': '0'} 2023-10-11T15:14:21.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3d20e44f80d13e-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.474 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.475 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.476 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.476 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.476 DEBUG:boto:Method: GET 2023-10-11T15:14:21.477 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.477 DEBUG:boto:Data: 2023-10-11T15:14:21.478 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.478 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.479 DEBUG:boto:Params: {} 2023-10-11T15:14:21.479 DEBUG:boto:Token: None 2023-10-11T15:14:21.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5087ce2ec5fad9f-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.484 DEBUG:boto:b'dghqvd-58obj000000000000000000281000falseobj000000000000000000282023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.485 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.485 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.486 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.486 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.487 DEBUG:boto:Method: GET 2023-10-11T15:14:21.487 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000028 2023-10-11T15:14:21.487 DEBUG:boto:Data: 2023-10-11T15:14:21.488 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.489 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.489 DEBUG:boto:Params: {} 2023-10-11T15:14:21.490 DEBUG:boto:Token: None 2023-10-11T15:14:21.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9c78e49d182d1bc-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.496 DEBUG:boto:b'dghqvd-58obj000000000000000000281000falseobj000000000000000000282023-10-11T15:13:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-11T15:14:21.497 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.498 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.498 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.499 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.499 DEBUG:boto:Method: GET 2023-10-11T15:14:21.500 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.500 DEBUG:boto:Data: 2023-10-11T15:14:21.501 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.501 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.502 DEBUG:boto:Params: {} 2023-10-11T15:14:21.502 DEBUG:boto:Token: None 2023-10-11T15:14:21.503 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.503 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E= 2023-10-11T15:14:21.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005c175ffa94e21e8-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.554 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.555 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.556 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.556 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.556 DEBUG:boto:Method: GET 2023-10-11T15:14:21.557 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.557 DEBUG:boto:Data: 2023-10-11T15:14:21.558 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.559 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.559 DEBUG:boto:Params: {} 2023-10-11T15:14:21.560 DEBUG:boto:Token: None 2023-10-11T15:14:21.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E= 2023-10-11T15:14:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd022025f93ec1c-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.565 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.565 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.566 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.566 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.567 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.567 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.568 DEBUG:boto:Data: 2023-10-11T15:14:21.568 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.568 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.569 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.569 DEBUG:boto:Params: {} 2023-10-11T15:14:21.570 DEBUG:boto:Token: None 2023-10-11T15:14:21.570 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+halOb9JeElrGNq+OIMTxwyXzlQ= 2023-10-11T15:14:21.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+halOb9JeElrGNq+OIMTxwyXzlQ=', 'Content-Length': '0'} 2023-10-11T15:14:21.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea37f2ca55948af-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.574 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.575 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.575 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.575 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.576 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.577 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.577 DEBUG:boto:Data: 2023-10-11T15:14:21.577 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.578 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.579 DEBUG:boto:Params: {} 2023-10-11T15:14:21.579 DEBUG:boto:Token: None 2023-10-11T15:14:21.579 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+halOb9JeElrGNq+OIMTxwyXzlQ= 2023-10-11T15:14:21.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+halOb9JeElrGNq+OIMTxwyXzlQ=', 'Content-Length': '0'} 2023-10-11T15:14:21.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad23cea95f4a9c0-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-11T15:14:21.583 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.584 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.584 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.585 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.585 DEBUG:boto:Method: GET 2023-10-11T15:14:21.586 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.586 DEBUG:boto:Data: 2023-10-11T15:14:21.586 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.587 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.588 DEBUG:boto:Params: {} 2023-10-11T15:14:21.588 DEBUG:boto:Token: None 2023-10-11T15:14:21.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E= 2023-10-11T15:14:21.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fda6f6a6427e55f2-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.592 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.593 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.593 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.594 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.594 DEBUG:boto:Method: GET 2023-10-11T15:14:21.595 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.595 DEBUG:boto:Data: 2023-10-11T15:14:21.596 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.597 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.597 DEBUG:boto:Params: {} 2023-10-11T15:14:21.597 DEBUG:boto:Token: None 2023-10-11T15:14:21.598 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029?versionId=null 2023-10-11T15:14:21.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E= 2023-10-11T15:14:21.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6adL/SKeLQUlwNvQRv5fLnZ3S1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088aeb9e9eba6d362-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.605 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.606 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.607 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.607 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.607 DEBUG:boto:Method: GET 2023-10-11T15:14:21.608 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.608 DEBUG:boto:Data: 2023-10-11T15:14:21.609 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.610 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.610 DEBUG:boto:Params: {} 2023-10-11T15:14:21.610 DEBUG:boto:Token: None 2023-10-11T15:14:21.611 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa28b835a60cb458-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.615 DEBUG:boto:b'dghqvd-58obj000000000000000000291000falseobj000000000000000000292023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.616 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.617 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.617 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.617 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.618 DEBUG:boto:Method: GET 2023-10-11T15:14:21.618 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.619 DEBUG:boto:Data: 2023-10-11T15:14:21.619 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.620 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.620 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.620 DEBUG:boto:Params: {} 2023-10-11T15:14:21.621 DEBUG:boto:Token: None 2023-10-11T15:14:21.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.622 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d6af3dd5bb68f0-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.627 DEBUG:boto:b'dghqvd-58obj000000000000000000291000falseobj000000000000000000292023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.628 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.628 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.629 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.629 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.630 DEBUG:boto:Data: 2023-10-11T15:14:21.630 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.631 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.631 DEBUG:boto:Params: {} 2023-10-11T15:14:21.632 DEBUG:boto:Token: None 2023-10-11T15:14:21.632 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.633 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kouSeDqweB4nKz+Fu7XeeYJjm1E= 2023-10-11T15:14:21.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kouSeDqweB4nKz+Fu7XeeYJjm1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b81882bdc49151-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.636 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.636 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.637 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.637 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.637 DEBUG:boto:Data: 2023-10-11T15:14:21.638 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.639 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.639 DEBUG:boto:Params: {} 2023-10-11T15:14:21.640 DEBUG:boto:Token: None 2023-10-11T15:14:21.640 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kouSeDqweB4nKz+Fu7XeeYJjm1E= 2023-10-11T15:14:21.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kouSeDqweB4nKz+Fu7XeeYJjm1E=', 'Content-Length': '0'} 2023-10-11T15:14:21.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c026e14771af34-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-10-11T15:14:21.644 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.645 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.645 DEBUG:boto:Method: GET 2023-10-11T15:14:21.645 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.646 DEBUG:boto:Data: 2023-10-11T15:14:21.646 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.647 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.647 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.648 DEBUG:boto:Params: {} 2023-10-11T15:14:21.648 DEBUG:boto:Token: None 2023-10-11T15:14:21.649 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.649 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mSG4Y3xP9fJbCTq5LvKMmx+kj+0= 2023-10-11T15:14:21.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mSG4Y3xP9fJbCTq5LvKMmx+kj+0=', 'Content-Length': '0'} 2023-10-11T15:14:21.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dcc3a401ed8385c-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.694 DEBUG:boto:path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.695 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.695 DEBUG:boto:Method: GET 2023-10-11T15:14:21.696 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.696 DEBUG:boto:Data: 2023-10-11T15:14:21.696 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.697 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.698 DEBUG:boto:Params: {} 2023-10-11T15:14:21.698 DEBUG:boto:Token: None 2023-10-11T15:14:21.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000029 2023-10-11T15:14:21.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mSG4Y3xP9fJbCTq5LvKMmx+kj+0= 2023-10-11T15:14:21.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mSG4Y3xP9fJbCTq5LvKMmx+kj+0=', 'Content-Length': '0'} 2023-10-11T15:14:21.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b843993ddddc68-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.703 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.703 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.704 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.704 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.704 DEBUG:boto:Method: GET 2023-10-11T15:14:21.705 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.705 DEBUG:boto:Data: 2023-10-11T15:14:21.706 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.707 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.707 DEBUG:boto:Params: {} 2023-10-11T15:14:21.707 DEBUG:boto:Token: None 2023-10-11T15:14:21.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c18a85c7d48c12d-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.712 DEBUG:boto:b'dghqvd-58obj000000000000000000291000falseobj000000000000000000292023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.712 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.713 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.713 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.714 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.714 DEBUG:boto:Method: GET 2023-10-11T15:14:21.715 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000029 2023-10-11T15:14:21.715 DEBUG:boto:Data: 2023-10-11T15:14:21.715 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.716 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.716 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.717 DEBUG:boto:Params: {} 2023-10-11T15:14:21.717 DEBUG:boto:Token: None 2023-10-11T15:14:21.718 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.718 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000afab6fa781b60d2-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.723 DEBUG:boto:b'dghqvd-58obj000000000000000000291000falseobj000000000000000000292023-10-11T15:14:03.187Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-11T15:14:21.724 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.725 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.725 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.726 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.726 DEBUG:boto:Method: GET 2023-10-11T15:14:21.726 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.727 DEBUG:boto:Data: 2023-10-11T15:14:21.727 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.728 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.728 DEBUG:boto:Params: {} 2023-10-11T15:14:21.729 DEBUG:boto:Token: None 2023-10-11T15:14:21.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g= 2023-10-11T15:14:21.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g=', 'Content-Length': '0'} 2023-10-11T15:14:21.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45f4e6d48b22ee4-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.733 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.733 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.734 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.734 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.735 DEBUG:boto:Method: GET 2023-10-11T15:14:21.735 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.736 DEBUG:boto:Data: 2023-10-11T15:14:21.736 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.737 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.737 DEBUG:boto:Params: {} 2023-10-11T15:14:21.738 DEBUG:boto:Token: None 2023-10-11T15:14:21.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g= 2023-10-11T15:14:21.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g=', 'Content-Length': '0'} 2023-10-11T15:14:21.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f333703fa28bf5-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.742 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.743 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.743 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.743 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.744 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.744 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.745 DEBUG:boto:Data: 2023-10-11T15:14:21.745 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.746 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.746 DEBUG:boto:Params: {} 2023-10-11T15:14:21.747 DEBUG:boto:Token: None 2023-10-11T15:14:21.747 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oa7kZRvn9SpRU8BscDManwsFnWs= 2023-10-11T15:14:21.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oa7kZRvn9SpRU8BscDManwsFnWs=', 'Content-Length': '0'} 2023-10-11T15:14:21.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c30882cdbde300-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.751 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.751 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.751 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.752 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.752 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.752 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.753 DEBUG:boto:Data: 2023-10-11T15:14:21.753 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.754 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.754 DEBUG:boto:Params: {} 2023-10-11T15:14:21.755 DEBUG:boto:Token: None 2023-10-11T15:14:21.755 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oa7kZRvn9SpRU8BscDManwsFnWs= 2023-10-11T15:14:21.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oa7kZRvn9SpRU8BscDManwsFnWs=', 'Content-Length': '0'} 2023-10-11T15:14:21.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e94abd3eae9957c-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-11T15:14:21.759 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.760 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.760 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.760 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.761 DEBUG:boto:Method: GET 2023-10-11T15:14:21.761 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.762 DEBUG:boto:Data: 2023-10-11T15:14:21.762 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.763 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.763 DEBUG:boto:Params: {} 2023-10-11T15:14:21.764 DEBUG:boto:Token: None 2023-10-11T15:14:21.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g= 2023-10-11T15:14:21.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g=', 'Content-Length': '0'} 2023-10-11T15:14:21.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b384bf88d266c7d-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.768 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.769 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.769 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.769 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.770 DEBUG:boto:Method: GET 2023-10-11T15:14:21.770 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.771 DEBUG:boto:Data: 2023-10-11T15:14:21.771 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.772 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.772 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.772 DEBUG:boto:Params: {} 2023-10-11T15:14:21.773 DEBUG:boto:Token: None 2023-10-11T15:14:21.773 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030?versionId=null 2023-10-11T15:14:21.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g= 2023-10-11T15:14:21.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syDwwgEMiOkzCAcabsK2dGH0P0g=', 'Content-Length': '0'} 2023-10-11T15:14:21.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df2d1d2a99062d5-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.777 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.778 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.778 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.778 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.779 DEBUG:boto:Method: GET 2023-10-11T15:14:21.779 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.780 DEBUG:boto:Data: 2023-10-11T15:14:21.780 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.781 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.781 DEBUG:boto:Params: {} 2023-10-11T15:14:21.782 DEBUG:boto:Token: None 2023-10-11T15:14:21.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000117917e104395d9c-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.830 DEBUG:boto:b'dghqvd-58obj000000000000000000301000falseobj000000000000000000302023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.831 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.832 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.833 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.833 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.834 DEBUG:boto:Method: GET 2023-10-11T15:14:21.834 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.835 DEBUG:boto:Data: 2023-10-11T15:14:21.835 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.836 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.836 DEBUG:boto:Params: {} 2023-10-11T15:14:21.837 DEBUG:boto:Token: None 2023-10-11T15:14:21.837 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da454cba8e64d9e-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.843 DEBUG:boto:b'dghqvd-58obj000000000000000000301000falseobj000000000000000000302023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.844 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.845 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.845 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.846 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.846 DEBUG:boto:Data: 2023-10-11T15:14:21.846 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.847 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.848 DEBUG:boto:Params: {} 2023-10-11T15:14:21.848 DEBUG:boto:Token: None 2023-10-11T15:14:21.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zk4F4EIeae3bjkOxRx+DagLiU78= 2023-10-11T15:14:21.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zk4F4EIeae3bjkOxRx+DagLiU78=', 'Content-Length': '0'} 2023-10-11T15:14:21.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c13a2e40a3b59a-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.852 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.852 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.853 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.853 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.854 DEBUG:boto:Data: 2023-10-11T15:14:21.854 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.855 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.855 DEBUG:boto:Params: {} 2023-10-11T15:14:21.856 DEBUG:boto:Token: None 2023-10-11T15:14:21.856 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zk4F4EIeae3bjkOxRx+DagLiU78= 2023-10-11T15:14:21.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zk4F4EIeae3bjkOxRx+DagLiU78=', 'Content-Length': '0'} 2023-10-11T15:14:21.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031aaedf0d32ee4b-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-10-11T15:14:21.860 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.861 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.861 DEBUG:boto:Method: GET 2023-10-11T15:14:21.862 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.862 DEBUG:boto:Data: 2023-10-11T15:14:21.863 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.863 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.864 DEBUG:boto:Params: {} 2023-10-11T15:14:21.864 DEBUG:boto:Token: None 2023-10-11T15:14:21.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4cpa/2R13pCaHFLoHB+xoVGMb6E= 2023-10-11T15:14:21.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4cpa/2R13pCaHFLoHB+xoVGMb6E=', 'Content-Length': '0'} 2023-10-11T15:14:21.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf00e2baaa76aaf-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.869 DEBUG:boto:path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.869 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.869 DEBUG:boto:Method: GET 2023-10-11T15:14:21.870 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.870 DEBUG:boto:Data: 2023-10-11T15:14:21.871 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.872 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.872 DEBUG:boto:Params: {} 2023-10-11T15:14:21.873 DEBUG:boto:Token: None 2023-10-11T15:14:21.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000030 2023-10-11T15:14:21.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4cpa/2R13pCaHFLoHB+xoVGMb6E= 2023-10-11T15:14:21.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4cpa/2R13pCaHFLoHB+xoVGMb6E=', 'Content-Length': '0'} 2023-10-11T15:14:21.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465cb85c4925ad5c-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.918 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.919 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.919 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.920 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.920 DEBUG:boto:Method: GET 2023-10-11T15:14:21.921 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.921 DEBUG:boto:Data: 2023-10-11T15:14:21.922 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.922 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.922 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.923 DEBUG:boto:Params: {} 2023-10-11T15:14:21.923 DEBUG:boto:Token: None 2023-10-11T15:14:21.924 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.924 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c01c5c400b7dba0-006526bbcd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.928 DEBUG:boto:b'dghqvd-58obj000000000000000000301000falseobj000000000000000000302023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.929 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:21.929 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:21.929 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.930 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.930 DEBUG:boto:Method: GET 2023-10-11T15:14:21.931 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000030 2023-10-11T15:14:21.931 DEBUG:boto:Data: 2023-10-11T15:14:21.932 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.932 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.932 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.933 DEBUG:boto:Params: {} 2023-10-11T15:14:21.933 DEBUG:boto:Token: None 2023-10-11T15:14:21.934 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/ 2023-10-11T15:14:21.934 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q= 2023-10-11T15:14:21.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XgkwhTQXSwWYo2hywsCcbjqQW7Q=', 'Content-Length': '0'} 2023-10-11T15:14:21.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e2e740cded29024-006526bbcd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.940 DEBUG:boto:b'dghqvd-58obj000000000000000000301000falseobj000000000000000000302023-10-11T15:14:05.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:21.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-11T15:14:21.941 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.941 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.942 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.942 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.942 DEBUG:boto:Method: GET 2023-10-11T15:14:21.943 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.943 DEBUG:boto:Data: 2023-10-11T15:14:21.944 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.945 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.945 DEBUG:boto:Params: {} 2023-10-11T15:14:21.945 DEBUG:boto:Token: None 2023-10-11T15:14:21.946 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U= 2023-10-11T15:14:21.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U=', 'Content-Length': '0'} 2023-10-11T15:14:21.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8eab7906fcbc251-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.950 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.950 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.951 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.951 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.952 DEBUG:boto:Method: GET 2023-10-11T15:14:21.952 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.953 DEBUG:boto:Data: 2023-10-11T15:14:21.953 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.954 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.954 DEBUG:boto:Params: {} 2023-10-11T15:14:21.955 DEBUG:boto:Token: None 2023-10-11T15:14:21.955 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U= 2023-10-11T15:14:21.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U=', 'Content-Length': '0'} 2023-10-11T15:14:21.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022745aa4800507b0-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.959 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.959 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.960 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.960 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.960 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.961 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.961 DEBUG:boto:Data: 2023-10-11T15:14:21.962 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.962 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.963 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.963 DEBUG:boto:Params: {} 2023-10-11T15:14:21.963 DEBUG:boto:Token: None 2023-10-11T15:14:21.964 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kNrepr0GI8fOhxU/1QITQtbh+BU= 2023-10-11T15:14:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kNrepr0GI8fOhxU/1QITQtbh+BU=', 'Content-Length': '0'} 2023-10-11T15:14:21.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d922e5a1a346ca-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.967 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.968 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.968 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.969 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.969 DEBUG:boto:Method: HEAD 2023-10-11T15:14:21.969 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.970 DEBUG:boto:Data: 2023-10-11T15:14:21.970 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:21.971 DEBUG:boto:Port: 8000 2023-10-11T15:14:21.972 DEBUG:boto:Params: {} 2023-10-11T15:14:21.972 DEBUG:boto:Token: None 2023-10-11T15:14:21.972 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kNrepr0GI8fOhxU/1QITQtbh+BU= 2023-10-11T15:14:21.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kNrepr0GI8fOhxU/1QITQtbh+BU=', 'Content-Length': '0'} 2023-10-11T15:14:21.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0de9c6e8a68860d-006526bbcd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:21.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-11T15:14:21.976 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.977 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:21.977 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.978 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.978 DEBUG:boto:Method: GET 2023-10-11T15:14:21.979 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.979 DEBUG:boto:Data: 2023-10-11T15:14:21.979 DEBUG:boto:Headers: {} 2023-10-11T15:14:21.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:21.980 DEBUG:boto:Port: 8001 2023-10-11T15:14:21.981 DEBUG:boto:Params: {} 2023-10-11T15:14:21.981 DEBUG:boto:Token: None 2023-10-11T15:14:21.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:21 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:21.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U= 2023-10-11T15:14:21.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:21 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqQXBotADLXVu0ZRQm7gHkLzs5U=', 'Content-Length': '0'} 2023-10-11T15:14:21.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091a92193a715136-006526bbcd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.030 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.031 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.031 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:22.032 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:22.032 DEBUG:boto:Method: GET 2023-10-11T15:14:22.033 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:22.033 DEBUG:boto:Data: 2023-10-11T15:14:22.034 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.035 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.035 DEBUG:boto:Params: {} 2023-10-11T15:14:22.036 DEBUG:boto:Token: None 2023-10-11T15:14:22.036 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000031?versionId=null 2023-10-11T15:14:22.037 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jb82QbhcPJrlmAiLmxUfTPLa+p4= 2023-10-11T15:14:22.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jb82QbhcPJrlmAiLmxUfTPLa+p4=', 'Content-Length': '0'} 2023-10-11T15:14:22.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098fc8152ef2d4375-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.040 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.040 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.041 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.041 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.042 DEBUG:boto:Method: GET 2023-10-11T15:14:22.042 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.042 DEBUG:boto:Data: 2023-10-11T15:14:22.043 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.044 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.044 DEBUG:boto:Params: {} 2023-10-11T15:14:22.044 DEBUG:boto:Token: None 2023-10-11T15:14:22.045 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ea99d7a48b6c7f-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.049 DEBUG:boto:b'dghqvd-58obj000000000000000000311000falseobj000000000000000000312023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.050 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.050 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.051 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.051 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.052 DEBUG:boto:Method: GET 2023-10-11T15:14:22.052 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.052 DEBUG:boto:Data: 2023-10-11T15:14:22.053 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.054 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.054 DEBUG:boto:Params: {} 2023-10-11T15:14:22.055 DEBUG:boto:Token: None 2023-10-11T15:14:22.055 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad7d41d13ef7abf-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.061 DEBUG:boto:b'dghqvd-58obj000000000000000000311000falseobj000000000000000000312023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.062 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.062 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.063 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.063 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.063 DEBUG:boto:Data: 2023-10-11T15:14:22.064 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.065 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.065 DEBUG:boto:Params: {} 2023-10-11T15:14:22.066 DEBUG:boto:Token: None 2023-10-11T15:14:22.066 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tJ5AGplQx+dsoJZhLcC/pfW66EU= 2023-10-11T15:14:22.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tJ5AGplQx+dsoJZhLcC/pfW66EU=', 'Content-Length': '0'} 2023-10-11T15:14:22.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369bd160052f610f-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.070 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.070 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.070 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.071 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.071 DEBUG:boto:Data: 2023-10-11T15:14:22.072 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.073 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.073 DEBUG:boto:Params: {} 2023-10-11T15:14:22.073 DEBUG:boto:Token: None 2023-10-11T15:14:22.074 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tJ5AGplQx+dsoJZhLcC/pfW66EU= 2023-10-11T15:14:22.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tJ5AGplQx+dsoJZhLcC/pfW66EU=', 'Content-Length': '0'} 2023-10-11T15:14:22.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4bc1dd201a5db47-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-10-11T15:14:22.078 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.078 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.079 DEBUG:boto:Method: GET 2023-10-11T15:14:22.079 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.079 DEBUG:boto:Data: 2023-10-11T15:14:22.080 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.081 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.081 DEBUG:boto:Params: {} 2023-10-11T15:14:22.082 DEBUG:boto:Token: None 2023-10-11T15:14:22.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s+lyO9qJszIOzkOP1Ficdu0T1HQ= 2023-10-11T15:14:22.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s+lyO9qJszIOzkOP1Ficdu0T1HQ=', 'Content-Length': '0'} 2023-10-11T15:14:22.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c739a9801a150f-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.085 DEBUG:boto:path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.086 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.086 DEBUG:boto:Method: GET 2023-10-11T15:14:22.087 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.087 DEBUG:boto:Data: 2023-10-11T15:14:22.088 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.088 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.089 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.089 DEBUG:boto:Params: {} 2023-10-11T15:14:22.089 DEBUG:boto:Token: None 2023-10-11T15:14:22.090 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000031 2023-10-11T15:14:22.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s+lyO9qJszIOzkOP1Ficdu0T1HQ= 2023-10-11T15:14:22.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s+lyO9qJszIOzkOP1Ficdu0T1HQ=', 'Content-Length': '0'} 2023-10-11T15:14:22.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008208eb1726e30a92-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.094 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.094 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.095 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.095 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.095 DEBUG:boto:Method: GET 2023-10-11T15:14:22.096 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.096 DEBUG:boto:Data: 2023-10-11T15:14:22.097 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.097 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.098 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.098 DEBUG:boto:Params: {} 2023-10-11T15:14:22.099 DEBUG:boto:Token: None 2023-10-11T15:14:22.099 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041824c131c990048-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.103 DEBUG:boto:b'dghqvd-58obj000000000000000000311000falseobj000000000000000000312023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.104 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.104 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.104 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.105 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.105 DEBUG:boto:Method: GET 2023-10-11T15:14:22.106 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000031 2023-10-11T15:14:22.106 DEBUG:boto:Data: 2023-10-11T15:14:22.107 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.107 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.108 DEBUG:boto:Params: {} 2023-10-11T15:14:22.108 DEBUG:boto:Token: None 2023-10-11T15:14:22.109 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5a0338f5aee019-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.114 DEBUG:boto:b'dghqvd-58obj000000000000000000311000falseobj000000000000000000312023-10-11T15:14:02.779Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-11T15:14:22.115 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.116 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.116 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.116 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.117 DEBUG:boto:Method: GET 2023-10-11T15:14:22.117 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.118 DEBUG:boto:Data: 2023-10-11T15:14:22.118 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.119 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.120 DEBUG:boto:Params: {} 2023-10-11T15:14:22.120 DEBUG:boto:Token: None 2023-10-11T15:14:22.120 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E= 2023-10-11T15:14:22.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E=', 'Content-Length': '0'} 2023-10-11T15:14:22.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8fc64dfe098c1b-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.124 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.124 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.125 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.125 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.126 DEBUG:boto:Method: GET 2023-10-11T15:14:22.126 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.127 DEBUG:boto:Data: 2023-10-11T15:14:22.127 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.128 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.128 DEBUG:boto:Params: {} 2023-10-11T15:14:22.129 DEBUG:boto:Token: None 2023-10-11T15:14:22.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E= 2023-10-11T15:14:22.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E=', 'Content-Length': '0'} 2023-10-11T15:14:22.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c535017f216ff0-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.133 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.133 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.133 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.134 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.134 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.135 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.135 DEBUG:boto:Data: 2023-10-11T15:14:22.136 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.137 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.137 DEBUG:boto:Params: {} 2023-10-11T15:14:22.137 DEBUG:boto:Token: None 2023-10-11T15:14:22.138 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nPsvsopxPD0NTl/VH3157h3iFRk= 2023-10-11T15:14:22.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nPsvsopxPD0NTl/VH3157h3iFRk=', 'Content-Length': '0'} 2023-10-11T15:14:22.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2c0faa84c189da-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.141 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.142 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.142 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.142 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.143 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.143 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.144 DEBUG:boto:Data: 2023-10-11T15:14:22.144 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.145 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.145 DEBUG:boto:Params: {} 2023-10-11T15:14:22.146 DEBUG:boto:Token: None 2023-10-11T15:14:22.146 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nPsvsopxPD0NTl/VH3157h3iFRk= 2023-10-11T15:14:22.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nPsvsopxPD0NTl/VH3157h3iFRk=', 'Content-Length': '0'} 2023-10-11T15:14:22.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cf8ce1af1652d3c-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-11T15:14:22.150 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.150 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.151 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.151 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.152 DEBUG:boto:Method: GET 2023-10-11T15:14:22.152 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.152 DEBUG:boto:Data: 2023-10-11T15:14:22.153 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.153 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.154 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.154 DEBUG:boto:Params: {} 2023-10-11T15:14:22.155 DEBUG:boto:Token: None 2023-10-11T15:14:22.155 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E= 2023-10-11T15:14:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E=', 'Content-Length': '0'} 2023-10-11T15:14:22.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aac9fb2ab9ec1628-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.159 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.159 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.159 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.160 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.160 DEBUG:boto:Method: GET 2023-10-11T15:14:22.161 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.161 DEBUG:boto:Data: 2023-10-11T15:14:22.161 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.162 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.163 DEBUG:boto:Params: {} 2023-10-11T15:14:22.163 DEBUG:boto:Token: None 2023-10-11T15:14:22.163 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032?versionId=null 2023-10-11T15:14:22.164 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E= 2023-10-11T15:14:22.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WAgCJqp4OpymXETuEgF4X0Z8Y0E=', 'Content-Length': '0'} 2023-10-11T15:14:22.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd95d7df1aedc70-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.167 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.168 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.168 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.169 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.169 DEBUG:boto:Method: GET 2023-10-11T15:14:22.169 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.170 DEBUG:boto:Data: 2023-10-11T15:14:22.170 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.171 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.171 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.172 DEBUG:boto:Params: {} 2023-10-11T15:14:22.172 DEBUG:boto:Token: None 2023-10-11T15:14:22.172 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.173 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082bc68bab410070d-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.176 DEBUG:boto:b'dghqvd-58obj000000000000000000321000falseobj000000000000000000322023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.177 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.178 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.178 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.178 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.179 DEBUG:boto:Method: GET 2023-10-11T15:14:22.179 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.180 DEBUG:boto:Data: 2023-10-11T15:14:22.180 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.180 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.181 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.181 DEBUG:boto:Params: {} 2023-10-11T15:14:22.182 DEBUG:boto:Token: None 2023-10-11T15:14:22.182 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.182 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c62dcd94b998f53-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.230 DEBUG:boto:b'dghqvd-58obj000000000000000000321000falseobj000000000000000000322023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.231 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.232 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.232 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.232 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.233 DEBUG:boto:Data: 2023-10-11T15:14:22.233 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.234 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.234 DEBUG:boto:Params: {} 2023-10-11T15:14:22.235 DEBUG:boto:Token: None 2023-10-11T15:14:22.235 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:in8h2uK3SkgutiLT/uB6sqaz4wg= 2023-10-11T15:14:22.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:in8h2uK3SkgutiLT/uB6sqaz4wg=', 'Content-Length': '0'} 2023-10-11T15:14:22.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bcdf7fd8f721e99-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.239 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.239 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.240 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.240 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.241 DEBUG:boto:Data: 2023-10-11T15:14:22.241 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.242 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.242 DEBUG:boto:Params: {} 2023-10-11T15:14:22.243 DEBUG:boto:Token: None 2023-10-11T15:14:22.243 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:in8h2uK3SkgutiLT/uB6sqaz4wg= 2023-10-11T15:14:22.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:in8h2uK3SkgutiLT/uB6sqaz4wg=', 'Content-Length': '0'} 2023-10-11T15:14:22.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51d85dff60e204f-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-10-11T15:14:22.247 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.248 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.248 DEBUG:boto:Method: GET 2023-10-11T15:14:22.248 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.249 DEBUG:boto:Data: 2023-10-11T15:14:22.249 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.250 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.251 DEBUG:boto:Params: {} 2023-10-11T15:14:22.251 DEBUG:boto:Token: None 2023-10-11T15:14:22.252 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1nn/1lABdM4Le1NrFdSDTfAMHk= 2023-10-11T15:14:22.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1nn/1lABdM4Le1NrFdSDTfAMHk=', 'Content-Length': '0'} 2023-10-11T15:14:22.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d627f71adb03aaf6-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.298 DEBUG:boto:path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.299 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.300 DEBUG:boto:Method: GET 2023-10-11T15:14:22.300 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.300 DEBUG:boto:Data: 2023-10-11T15:14:22.301 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.302 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.302 DEBUG:boto:Params: {} 2023-10-11T15:14:22.303 DEBUG:boto:Token: None 2023-10-11T15:14:22.303 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000032 2023-10-11T15:14:22.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1nn/1lABdM4Le1NrFdSDTfAMHk= 2023-10-11T15:14:22.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1nn/1lABdM4Le1NrFdSDTfAMHk=', 'Content-Length': '0'} 2023-10-11T15:14:22.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a41cfabe44bba5c-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.307 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.307 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.308 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.308 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.308 DEBUG:boto:Method: GET 2023-10-11T15:14:22.309 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.309 DEBUG:boto:Data: 2023-10-11T15:14:22.310 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.311 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.311 DEBUG:boto:Params: {} 2023-10-11T15:14:22.311 DEBUG:boto:Token: None 2023-10-11T15:14:22.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1af49b40f6b550-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.316 DEBUG:boto:b'dghqvd-58obj000000000000000000321000falseobj000000000000000000322023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.317 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.317 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.318 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.318 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.319 DEBUG:boto:Method: GET 2023-10-11T15:14:22.319 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000032 2023-10-11T15:14:22.320 DEBUG:boto:Data: 2023-10-11T15:14:22.320 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.321 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.321 DEBUG:boto:Params: {} 2023-10-11T15:14:22.322 DEBUG:boto:Token: None 2023-10-11T15:14:22.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5985937f3b439a5-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.327 DEBUG:boto:b'dghqvd-58obj000000000000000000321000falseobj000000000000000000322023-10-11T15:14:01.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-11T15:14:22.329 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.329 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.329 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.330 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.330 DEBUG:boto:Method: GET 2023-10-11T15:14:22.331 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.331 DEBUG:boto:Data: 2023-10-11T15:14:22.332 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.332 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.333 DEBUG:boto:Params: {} 2023-10-11T15:14:22.333 DEBUG:boto:Token: None 2023-10-11T15:14:22.334 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU= 2023-10-11T15:14:22.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU=', 'Content-Length': '0'} 2023-10-11T15:14:22.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b73beb28a298a4-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.338 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.338 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.339 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.339 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.339 DEBUG:boto:Method: GET 2023-10-11T15:14:22.340 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.340 DEBUG:boto:Data: 2023-10-11T15:14:22.341 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.342 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.342 DEBUG:boto:Params: {} 2023-10-11T15:14:22.342 DEBUG:boto:Token: None 2023-10-11T15:14:22.343 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.343 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU= 2023-10-11T15:14:22.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU=', 'Content-Length': '0'} 2023-10-11T15:14:22.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac92498ebe2b29a-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.346 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.347 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.347 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.348 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.348 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.348 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.349 DEBUG:boto:Data: 2023-10-11T15:14:22.349 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.350 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.351 DEBUG:boto:Params: {} 2023-10-11T15:14:22.351 DEBUG:boto:Token: None 2023-10-11T15:14:22.351 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7iIc4YN0o/LFkf9F5FylrWqmAIw= 2023-10-11T15:14:22.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7iIc4YN0o/LFkf9F5FylrWqmAIw=', 'Content-Length': '0'} 2023-10-11T15:14:22.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70cd40f5a3bb1df-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.354 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.355 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.355 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.356 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.356 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.356 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.357 DEBUG:boto:Data: 2023-10-11T15:14:22.357 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.358 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.358 DEBUG:boto:Params: {} 2023-10-11T15:14:22.359 DEBUG:boto:Token: None 2023-10-11T15:14:22.359 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7iIc4YN0o/LFkf9F5FylrWqmAIw= 2023-10-11T15:14:22.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7iIc4YN0o/LFkf9F5FylrWqmAIw=', 'Content-Length': '0'} 2023-10-11T15:14:22.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a8c8d6f99066e9-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-11T15:14:22.363 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.364 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.364 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.364 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.365 DEBUG:boto:Method: GET 2023-10-11T15:14:22.365 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.366 DEBUG:boto:Data: 2023-10-11T15:14:22.366 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.367 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.367 DEBUG:boto:Params: {} 2023-10-11T15:14:22.368 DEBUG:boto:Token: None 2023-10-11T15:14:22.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU= 2023-10-11T15:14:22.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU=', 'Content-Length': '0'} 2023-10-11T15:14:22.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4a14c1200acd31-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.372 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.372 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.373 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.373 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.373 DEBUG:boto:Method: GET 2023-10-11T15:14:22.374 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.374 DEBUG:boto:Data: 2023-10-11T15:14:22.375 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.375 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.375 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.376 DEBUG:boto:Params: {} 2023-10-11T15:14:22.376 DEBUG:boto:Token: None 2023-10-11T15:14:22.377 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033?versionId=null 2023-10-11T15:14:22.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU= 2023-10-11T15:14:22.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tl7MSEYUN2fUTmW0QYecW3usyXU=', 'Content-Length': '0'} 2023-10-11T15:14:22.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd687747d53fb5ff-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.380 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.381 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.381 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.382 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.382 DEBUG:boto:Method: GET 2023-10-11T15:14:22.382 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.383 DEBUG:boto:Data: 2023-10-11T15:14:22.383 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.384 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.384 DEBUG:boto:Params: {} 2023-10-11T15:14:22.385 DEBUG:boto:Token: None 2023-10-11T15:14:22.385 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d35b40a14a76bc78-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.430 DEBUG:boto:b'dghqvd-58obj000000000000000000331000falseobj000000000000000000332023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.431 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.432 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.432 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.433 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.433 DEBUG:boto:Method: GET 2023-10-11T15:14:22.433 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.434 DEBUG:boto:Data: 2023-10-11T15:14:22.434 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.435 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.435 DEBUG:boto:Params: {} 2023-10-11T15:14:22.436 DEBUG:boto:Token: None 2023-10-11T15:14:22.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b7c20bb56a1b621-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.442 DEBUG:boto:b'dghqvd-58obj000000000000000000331000falseobj000000000000000000332023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.443 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.443 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.444 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.444 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.444 DEBUG:boto:Data: 2023-10-11T15:14:22.445 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.445 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.446 DEBUG:boto:Params: {} 2023-10-11T15:14:22.446 DEBUG:boto:Token: None 2023-10-11T15:14:22.447 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.447 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ES3cKHnFqAeuuhzo5vSdeqcUPpw= 2023-10-11T15:14:22.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ES3cKHnFqAeuuhzo5vSdeqcUPpw=', 'Content-Length': '0'} 2023-10-11T15:14:22.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a83121bbc66c0c-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.450 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.450 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.451 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.451 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.452 DEBUG:boto:Data: 2023-10-11T15:14:22.452 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.453 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.453 DEBUG:boto:Params: {} 2023-10-11T15:14:22.454 DEBUG:boto:Token: None 2023-10-11T15:14:22.454 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ES3cKHnFqAeuuhzo5vSdeqcUPpw= 2023-10-11T15:14:22.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ES3cKHnFqAeuuhzo5vSdeqcUPpw=', 'Content-Length': '0'} 2023-10-11T15:14:22.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004076845047f877d7-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-10-11T15:14:22.458 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.458 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.459 DEBUG:boto:Method: GET 2023-10-11T15:14:22.459 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.460 DEBUG:boto:Data: 2023-10-11T15:14:22.460 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.461 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.461 DEBUG:boto:Params: {} 2023-10-11T15:14:22.462 DEBUG:boto:Token: None 2023-10-11T15:14:22.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uvHXlySl8cv9alWidSfhoNqWyAc= 2023-10-11T15:14:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uvHXlySl8cv9alWidSfhoNqWyAc=', 'Content-Length': '0'} 2023-10-11T15:14:22.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c3c42eeffe3aa6-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.466 DEBUG:boto:path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.466 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.467 DEBUG:boto:Method: GET 2023-10-11T15:14:22.467 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.467 DEBUG:boto:Data: 2023-10-11T15:14:22.468 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.469 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.469 DEBUG:boto:Params: {} 2023-10-11T15:14:22.470 DEBUG:boto:Token: None 2023-10-11T15:14:22.470 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000033 2023-10-11T15:14:22.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uvHXlySl8cv9alWidSfhoNqWyAc= 2023-10-11T15:14:22.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uvHXlySl8cv9alWidSfhoNqWyAc=', 'Content-Length': '0'} 2023-10-11T15:14:22.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43d9d7461658b5e-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.474 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.474 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.475 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.475 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.476 DEBUG:boto:Method: GET 2023-10-11T15:14:22.476 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.476 DEBUG:boto:Data: 2023-10-11T15:14:22.477 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.478 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.478 DEBUG:boto:Params: {} 2023-10-11T15:14:22.478 DEBUG:boto:Token: None 2023-10-11T15:14:22.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f5985eedfa7afd-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.483 DEBUG:boto:b'dghqvd-58obj000000000000000000331000falseobj000000000000000000332023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.483 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.484 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.484 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.485 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.485 DEBUG:boto:Method: GET 2023-10-11T15:14:22.486 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000033 2023-10-11T15:14:22.486 DEBUG:boto:Data: 2023-10-11T15:14:22.486 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.487 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.488 DEBUG:boto:Params: {} 2023-10-11T15:14:22.488 DEBUG:boto:Token: None 2023-10-11T15:14:22.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b1b3cbd2612103-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.494 DEBUG:boto:b'dghqvd-58obj000000000000000000331000falseobj000000000000000000332023-10-11T15:14:05.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-11T15:14:22.495 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.495 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.496 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.496 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.497 DEBUG:boto:Method: GET 2023-10-11T15:14:22.497 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.498 DEBUG:boto:Data: 2023-10-11T15:14:22.498 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.499 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.499 DEBUG:boto:Params: {} 2023-10-11T15:14:22.500 DEBUG:boto:Token: None 2023-10-11T15:14:22.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE= 2023-10-11T15:14:22.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE=', 'Content-Length': '0'} 2023-10-11T15:14:22.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df0f8fbdf1113c36-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.504 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.504 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.505 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.505 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.506 DEBUG:boto:Method: GET 2023-10-11T15:14:22.506 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.506 DEBUG:boto:Data: 2023-10-11T15:14:22.507 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.508 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.508 DEBUG:boto:Params: {} 2023-10-11T15:14:22.509 DEBUG:boto:Token: None 2023-10-11T15:14:22.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE= 2023-10-11T15:14:22.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE=', 'Content-Length': '0'} 2023-10-11T15:14:22.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012727e7e6bd3a0a3-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.513 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.513 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.514 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.514 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.514 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.515 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.515 DEBUG:boto:Data: 2023-10-11T15:14:22.516 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.516 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.517 DEBUG:boto:Params: {} 2023-10-11T15:14:22.517 DEBUG:boto:Token: None 2023-10-11T15:14:22.518 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDFKFJjnkv/9v7Qj+zA8aXv5uGQ= 2023-10-11T15:14:22.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDFKFJjnkv/9v7Qj+zA8aXv5uGQ=', 'Content-Length': '0'} 2023-10-11T15:14:22.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f798b9c0b953bbd-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.521 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.522 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.522 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.523 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.523 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.523 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.524 DEBUG:boto:Data: 2023-10-11T15:14:22.524 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.525 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.525 DEBUG:boto:Params: {} 2023-10-11T15:14:22.526 DEBUG:boto:Token: None 2023-10-11T15:14:22.526 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDFKFJjnkv/9v7Qj+zA8aXv5uGQ= 2023-10-11T15:14:22.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDFKFJjnkv/9v7Qj+zA8aXv5uGQ=', 'Content-Length': '0'} 2023-10-11T15:14:22.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67b488235c13a20-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-11T15:14:22.530 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.530 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.531 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.531 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.532 DEBUG:boto:Method: GET 2023-10-11T15:14:22.532 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.532 DEBUG:boto:Data: 2023-10-11T15:14:22.533 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.534 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.534 DEBUG:boto:Params: {} 2023-10-11T15:14:22.534 DEBUG:boto:Token: None 2023-10-11T15:14:22.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE= 2023-10-11T15:14:22.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE=', 'Content-Length': '0'} 2023-10-11T15:14:22.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ecdf9cbae76466-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.578 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.579 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.579 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.580 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.580 DEBUG:boto:Method: GET 2023-10-11T15:14:22.581 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.581 DEBUG:boto:Data: 2023-10-11T15:14:22.582 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.582 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.583 DEBUG:boto:Params: {} 2023-10-11T15:14:22.583 DEBUG:boto:Token: None 2023-10-11T15:14:22.584 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036?versionId=null 2023-10-11T15:14:22.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE= 2023-10-11T15:14:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ayPu8HJc1rs3wJCrAoQKY5haFCE=', 'Content-Length': '0'} 2023-10-11T15:14:22.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000583c906e7f6bac8b-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.588 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.588 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.588 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.589 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.589 DEBUG:boto:Method: GET 2023-10-11T15:14:22.590 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.590 DEBUG:boto:Data: 2023-10-11T15:14:22.591 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.591 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.592 DEBUG:boto:Params: {} 2023-10-11T15:14:22.592 DEBUG:boto:Token: None 2023-10-11T15:14:22.593 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7f9b613d664007-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.597 DEBUG:boto:b'dghqvd-58obj000000000000000000361000falseobj000000000000000000362023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.598 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.598 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.599 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.599 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.599 DEBUG:boto:Method: GET 2023-10-11T15:14:22.600 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.600 DEBUG:boto:Data: 2023-10-11T15:14:22.601 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.602 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.602 DEBUG:boto:Params: {} 2023-10-11T15:14:22.603 DEBUG:boto:Token: None 2023-10-11T15:14:22.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c8288fe09ecc50-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.654 DEBUG:boto:b'dghqvd-58obj000000000000000000361000falseobj000000000000000000362023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.655 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.656 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.656 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.657 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.657 DEBUG:boto:Data: 2023-10-11T15:14:22.657 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.658 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.659 DEBUG:boto:Params: {} 2023-10-11T15:14:22.659 DEBUG:boto:Token: None 2023-10-11T15:14:22.660 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEy904VBAqsj0Kti4MI5eYatuzA= 2023-10-11T15:14:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEy904VBAqsj0Kti4MI5eYatuzA=', 'Content-Length': '0'} 2023-10-11T15:14:22.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6b93aab9a9f0ad-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.663 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.664 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.664 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.665 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.665 DEBUG:boto:Data: 2023-10-11T15:14:22.665 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.666 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.666 DEBUG:boto:Params: {} 2023-10-11T15:14:22.667 DEBUG:boto:Token: None 2023-10-11T15:14:22.667 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEy904VBAqsj0Kti4MI5eYatuzA= 2023-10-11T15:14:22.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEy904VBAqsj0Kti4MI5eYatuzA=', 'Content-Length': '0'} 2023-10-11T15:14:22.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ea4721181e9553b-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-10-11T15:14:22.671 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.672 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.672 DEBUG:boto:Method: GET 2023-10-11T15:14:22.672 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.673 DEBUG:boto:Data: 2023-10-11T15:14:22.673 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.674 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.675 DEBUG:boto:Params: {} 2023-10-11T15:14:22.675 DEBUG:boto:Token: None 2023-10-11T15:14:22.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPuStb1u1uELY9TUu/0jzv7h45I= 2023-10-11T15:14:22.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPuStb1u1uELY9TUu/0jzv7h45I=', 'Content-Length': '0'} 2023-10-11T15:14:22.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25d5fc657040d9e-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.680 DEBUG:boto:path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.680 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.681 DEBUG:boto:Method: GET 2023-10-11T15:14:22.681 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.682 DEBUG:boto:Data: 2023-10-11T15:14:22.682 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.683 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.683 DEBUG:boto:Params: {} 2023-10-11T15:14:22.684 DEBUG:boto:Token: None 2023-10-11T15:14:22.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000036 2023-10-11T15:14:22.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPuStb1u1uELY9TUu/0jzv7h45I= 2023-10-11T15:14:22.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPuStb1u1uELY9TUu/0jzv7h45I=', 'Content-Length': '0'} 2023-10-11T15:14:22.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4eb15d8f8251214-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.688 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.688 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.689 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.689 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.689 DEBUG:boto:Method: GET 2023-10-11T15:14:22.690 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.690 DEBUG:boto:Data: 2023-10-11T15:14:22.691 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.691 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.692 DEBUG:boto:Params: {} 2023-10-11T15:14:22.692 DEBUG:boto:Token: None 2023-10-11T15:14:22.693 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c672b2fa101c3e-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.738 DEBUG:boto:b'dghqvd-58obj000000000000000000361000falseobj000000000000000000362023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.739 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.740 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.740 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.740 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.741 DEBUG:boto:Method: GET 2023-10-11T15:14:22.741 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000036 2023-10-11T15:14:22.742 DEBUG:boto:Data: 2023-10-11T15:14:22.742 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.743 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.743 DEBUG:boto:Params: {} 2023-10-11T15:14:22.744 DEBUG:boto:Token: None 2023-10-11T15:14:22.744 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6dc7ff43139e2ef-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.750 DEBUG:boto:b'dghqvd-58obj000000000000000000361000falseobj000000000000000000362023-10-11T15:14:04.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-11T15:14:22.751 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.752 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.752 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.752 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.753 DEBUG:boto:Method: GET 2023-10-11T15:14:22.753 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.754 DEBUG:boto:Data: 2023-10-11T15:14:22.754 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.755 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.755 DEBUG:boto:Params: {} 2023-10-11T15:14:22.756 DEBUG:boto:Token: None 2023-10-11T15:14:22.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os= 2023-10-11T15:14:22.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os=', 'Content-Length': '0'} 2023-10-11T15:14:22.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098adba378884a2b1-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.760 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.761 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.761 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.761 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.762 DEBUG:boto:Method: GET 2023-10-11T15:14:22.762 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.763 DEBUG:boto:Data: 2023-10-11T15:14:22.763 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.764 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.764 DEBUG:boto:Params: {} 2023-10-11T15:14:22.765 DEBUG:boto:Token: None 2023-10-11T15:14:22.765 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os= 2023-10-11T15:14:22.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os=', 'Content-Length': '0'} 2023-10-11T15:14:22.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b008d2d968eb13-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.769 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.769 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.770 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.770 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.771 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.771 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.771 DEBUG:boto:Data: 2023-10-11T15:14:22.772 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.772 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.772 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.773 DEBUG:boto:Params: {} 2023-10-11T15:14:22.773 DEBUG:boto:Token: None 2023-10-11T15:14:22.774 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/GUmkAQtiK3xRILrCfG5UbPJ8= 2023-10-11T15:14:22.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/GUmkAQtiK3xRILrCfG5UbPJ8=', 'Content-Length': '0'} 2023-10-11T15:14:22.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194941c2cbd426f4-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.777 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.777 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.778 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.778 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.778 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.779 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.779 DEBUG:boto:Data: 2023-10-11T15:14:22.780 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.780 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.781 DEBUG:boto:Params: {} 2023-10-11T15:14:22.781 DEBUG:boto:Token: None 2023-10-11T15:14:22.782 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/GUmkAQtiK3xRILrCfG5UbPJ8= 2023-10-11T15:14:22.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/GUmkAQtiK3xRILrCfG5UbPJ8=', 'Content-Length': '0'} 2023-10-11T15:14:22.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794003164e172c9e-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-11T15:14:22.786 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.786 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.787 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.787 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.787 DEBUG:boto:Method: GET 2023-10-11T15:14:22.788 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.788 DEBUG:boto:Data: 2023-10-11T15:14:22.789 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.790 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.790 DEBUG:boto:Params: {} 2023-10-11T15:14:22.791 DEBUG:boto:Token: None 2023-10-11T15:14:22.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os= 2023-10-11T15:14:22.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os=', 'Content-Length': '0'} 2023-10-11T15:14:22.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002765286c26ed9eec-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.794 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.795 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.795 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.796 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.796 DEBUG:boto:Method: GET 2023-10-11T15:14:22.797 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.797 DEBUG:boto:Data: 2023-10-11T15:14:22.797 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.798 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.799 DEBUG:boto:Params: {} 2023-10-11T15:14:22.799 DEBUG:boto:Token: None 2023-10-11T15:14:22.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037?versionId=null 2023-10-11T15:14:22.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os= 2023-10-11T15:14:22.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y8Su8FD9cYnf4SsZvxfosy5o9os=', 'Content-Length': '0'} 2023-10-11T15:14:22.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030bc0e56d1b35f6f-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.803 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.803 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.804 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.804 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.805 DEBUG:boto:Method: GET 2023-10-11T15:14:22.805 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.806 DEBUG:boto:Data: 2023-10-11T15:14:22.806 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.807 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.807 DEBUG:boto:Params: {} 2023-10-11T15:14:22.808 DEBUG:boto:Token: None 2023-10-11T15:14:22.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091642d90ca48bf97-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.812 DEBUG:boto:b'dghqvd-58obj000000000000000000371000falseobj000000000000000000372023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.813 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.813 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.814 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.814 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.815 DEBUG:boto:Method: GET 2023-10-11T15:14:22.815 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.815 DEBUG:boto:Data: 2023-10-11T15:14:22.816 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.817 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.817 DEBUG:boto:Params: {} 2023-10-11T15:14:22.817 DEBUG:boto:Token: None 2023-10-11T15:14:22.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89f553606aca1e2-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.870 DEBUG:boto:b'dghqvd-58obj000000000000000000371000falseobj000000000000000000372023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.872 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.872 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.873 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.873 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.874 DEBUG:boto:Data: 2023-10-11T15:14:22.874 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.875 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.875 DEBUG:boto:Params: {} 2023-10-11T15:14:22.876 DEBUG:boto:Token: None 2023-10-11T15:14:22.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GFeckEAcQ0LzJmiN6xYHOrKT20A= 2023-10-11T15:14:22.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GFeckEAcQ0LzJmiN6xYHOrKT20A=', 'Content-Length': '0'} 2023-10-11T15:14:22.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9e60d6fd58cfc2-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.880 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.880 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.881 DEBUG:boto:Method: HEAD 2023-10-11T15:14:22.881 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.881 DEBUG:boto:Data: 2023-10-11T15:14:22.882 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.883 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.883 DEBUG:boto:Params: {} 2023-10-11T15:14:22.883 DEBUG:boto:Token: None 2023-10-11T15:14:22.884 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GFeckEAcQ0LzJmiN6xYHOrKT20A= 2023-10-11T15:14:22.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GFeckEAcQ0LzJmiN6xYHOrKT20A=', 'Content-Length': '0'} 2023-10-11T15:14:22.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c54388eb14992fc2-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-10-11T15:14:22.888 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.888 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.889 DEBUG:boto:Method: GET 2023-10-11T15:14:22.889 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.889 DEBUG:boto:Data: 2023-10-11T15:14:22.890 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.891 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.891 DEBUG:boto:Params: {} 2023-10-11T15:14:22.892 DEBUG:boto:Token: None 2023-10-11T15:14:22.892 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZRsr+C9Tl2ZwSMY+4IBDD7qfPEg= 2023-10-11T15:14:22.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZRsr+C9Tl2ZwSMY+4IBDD7qfPEg=', 'Content-Length': '0'} 2023-10-11T15:14:22.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681924dc75233042-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.938 DEBUG:boto:path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.939 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.939 DEBUG:boto:Method: GET 2023-10-11T15:14:22.940 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.940 DEBUG:boto:Data: 2023-10-11T15:14:22.941 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.942 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.942 DEBUG:boto:Params: {} 2023-10-11T15:14:22.943 DEBUG:boto:Token: None 2023-10-11T15:14:22.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000037 2023-10-11T15:14:22.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZRsr+C9Tl2ZwSMY+4IBDD7qfPEg= 2023-10-11T15:14:22.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZRsr+C9Tl2ZwSMY+4IBDD7qfPEg=', 'Content-Length': '0'} 2023-10-11T15:14:22.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811c7202e0928c32-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.947 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.947 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.948 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.948 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.949 DEBUG:boto:Method: GET 2023-10-11T15:14:22.949 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.949 DEBUG:boto:Data: 2023-10-11T15:14:22.950 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.950 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.950 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.951 DEBUG:boto:Params: {} 2023-10-11T15:14:22.951 DEBUG:boto:Token: None 2023-10-11T15:14:22.952 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.952 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309a2564edb23358-006526bbce-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.956 DEBUG:boto:b'dghqvd-58obj000000000000000000371000falseobj000000000000000000372023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.957 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:22.957 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:22.958 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.958 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.958 DEBUG:boto:Method: GET 2023-10-11T15:14:22.959 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000037 2023-10-11T15:14:22.959 DEBUG:boto:Data: 2023-10-11T15:14:22.959 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.960 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.961 DEBUG:boto:Params: {} 2023-10-11T15:14:22.961 DEBUG:boto:Token: None 2023-10-11T15:14:22.962 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/ 2023-10-11T15:14:22.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI= 2023-10-11T15:14:22.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pgI96VRs96Nd5MnQhc9PoYhb/WI=', 'Content-Length': '0'} 2023-10-11T15:14:22.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50386faa11b015d-006526bbce-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.967 DEBUG:boto:b'dghqvd-58obj000000000000000000371000falseobj000000000000000000372023-10-11T15:14:02.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:22.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-11T15:14:22.968 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:22.969 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:22.969 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.969 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.970 DEBUG:boto:Method: GET 2023-10-11T15:14:22.970 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.971 DEBUG:boto:Data: 2023-10-11T15:14:22.971 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:22.972 DEBUG:boto:Port: 8001 2023-10-11T15:14:22.973 DEBUG:boto:Params: {} 2023-10-11T15:14:22.973 DEBUG:boto:Token: None 2023-10-11T15:14:22.974 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:heTNWheKzS/CHG1ujSOxGV+rNEI= 2023-10-11T15:14:22.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:heTNWheKzS/CHG1ujSOxGV+rNEI=', 'Content-Length': '0'} 2023-10-11T15:14:22.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000defdf7edeb4bea04-006526bbce-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:22.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:22.978 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:22.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.979 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.979 DEBUG:boto:Method: GET 2023-10-11T15:14:22.980 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.980 DEBUG:boto:Data: 2023-10-11T15:14:22.980 DEBUG:boto:Headers: {} 2023-10-11T15:14:22.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:22.981 DEBUG:boto:Port: 8000 2023-10-11T15:14:22.982 DEBUG:boto:Params: {} 2023-10-11T15:14:22.982 DEBUG:boto:Token: None 2023-10-11T15:14:22.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:22 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:22.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:heTNWheKzS/CHG1ujSOxGV+rNEI= 2023-10-11T15:14:22.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:heTNWheKzS/CHG1ujSOxGV+rNEI=', 'Content-Length': '0'} 2023-10-11T15:14:22.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5c74e2228c2d50-006526bbce-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.026 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.027 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.027 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.028 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.028 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.029 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.029 DEBUG:boto:Data: 2023-10-11T15:14:23.029 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.030 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.031 DEBUG:boto:Params: {} 2023-10-11T15:14:23.031 DEBUG:boto:Token: None 2023-10-11T15:14:23.032 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:He+nmZgRtABSIuwJrN3wLR2w8LA= 2023-10-11T15:14:23.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:He+nmZgRtABSIuwJrN3wLR2w8LA=', 'Content-Length': '0'} 2023-10-11T15:14:23.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004207303c43c3efaa-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.035 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.036 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.036 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.036 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.037 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.037 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.038 DEBUG:boto:Data: 2023-10-11T15:14:23.038 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.042 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.042 DEBUG:boto:Params: {} 2023-10-11T15:14:23.043 DEBUG:boto:Token: None 2023-10-11T15:14:23.043 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:He+nmZgRtABSIuwJrN3wLR2w8LA= 2023-10-11T15:14:23.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:He+nmZgRtABSIuwJrN3wLR2w8LA=', 'Content-Length': '0'} 2023-10-11T15:14:23.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a78d7133d3756a-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-11T15:14:23.047 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.048 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.048 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.048 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.049 DEBUG:boto:Method: GET 2023-10-11T15:14:23.049 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.050 DEBUG:boto:Data: 2023-10-11T15:14:23.050 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.051 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.051 DEBUG:boto:Params: {} 2023-10-11T15:14:23.052 DEBUG:boto:Token: None 2023-10-11T15:14:23.052 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dupjbi898bjnYn7SJUAg8jk7AVA= 2023-10-11T15:14:23.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dupjbi898bjnYn7SJUAg8jk7AVA=', 'Content-Length': '0'} 2023-10-11T15:14:23.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bc512b91fc22bf-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.102 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.103 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.103 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.104 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.104 DEBUG:boto:Method: GET 2023-10-11T15:14:23.105 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.105 DEBUG:boto:Data: 2023-10-11T15:14:23.106 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.107 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.107 DEBUG:boto:Params: {} 2023-10-11T15:14:23.108 DEBUG:boto:Token: None 2023-10-11T15:14:23.108 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039?versionId=null 2023-10-11T15:14:23.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dupjbi898bjnYn7SJUAg8jk7AVA= 2023-10-11T15:14:23.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dupjbi898bjnYn7SJUAg8jk7AVA=', 'Content-Length': '0'} 2023-10-11T15:14:23.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056da9d5b79b8aea2-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.112 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.113 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.113 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.113 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.114 DEBUG:boto:Method: GET 2023-10-11T15:14:23.114 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.115 DEBUG:boto:Data: 2023-10-11T15:14:23.115 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.116 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.117 DEBUG:boto:Params: {} 2023-10-11T15:14:23.117 DEBUG:boto:Token: None 2023-10-11T15:14:23.118 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a838e529bdb9343c-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.122 DEBUG:boto:b'dghqvd-58obj000000000000000000391000falseobj000000000000000000392023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.123 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.123 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.124 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.124 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.125 DEBUG:boto:Method: GET 2023-10-11T15:14:23.125 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.125 DEBUG:boto:Data: 2023-10-11T15:14:23.126 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.127 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.127 DEBUG:boto:Params: {} 2023-10-11T15:14:23.128 DEBUG:boto:Token: None 2023-10-11T15:14:23.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.129 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d3570682a0d4e1-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.134 DEBUG:boto:b'dghqvd-58obj000000000000000000391000falseobj000000000000000000392023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.135 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.135 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.135 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.136 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.136 DEBUG:boto:Data: 2023-10-11T15:14:23.137 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.138 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.138 DEBUG:boto:Params: {} 2023-10-11T15:14:23.139 DEBUG:boto:Token: None 2023-10-11T15:14:23.139 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lEsfR3AVh/4/VgLLf5fnXxu9C0M= 2023-10-11T15:14:23.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lEsfR3AVh/4/VgLLf5fnXxu9C0M=', 'Content-Length': '0'} 2023-10-11T15:14:23.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e8ad59a7f78f79-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.143 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.144 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.144 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.145 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.145 DEBUG:boto:Data: 2023-10-11T15:14:23.145 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.146 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.147 DEBUG:boto:Params: {} 2023-10-11T15:14:23.147 DEBUG:boto:Token: None 2023-10-11T15:14:23.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lEsfR3AVh/4/VgLLf5fnXxu9C0M= 2023-10-11T15:14:23.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lEsfR3AVh/4/VgLLf5fnXxu9C0M=', 'Content-Length': '0'} 2023-10-11T15:14:23.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8d032901611ac18-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-10-11T15:14:23.152 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.152 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.153 DEBUG:boto:Method: GET 2023-10-11T15:14:23.153 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.154 DEBUG:boto:Data: 2023-10-11T15:14:23.154 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.155 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.155 DEBUG:boto:Params: {} 2023-10-11T15:14:23.156 DEBUG:boto:Token: None 2023-10-11T15:14:23.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXfW0fwfpy+ZxfdINF0o/D6cQh8= 2023-10-11T15:14:23.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXfW0fwfpy+ZxfdINF0o/D6cQh8=', 'Content-Length': '0'} 2023-10-11T15:14:23.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b99d61db30aa33a-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.160 DEBUG:boto:path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.160 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.161 DEBUG:boto:Method: GET 2023-10-11T15:14:23.161 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.162 DEBUG:boto:Data: 2023-10-11T15:14:23.162 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.163 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.163 DEBUG:boto:Params: {} 2023-10-11T15:14:23.164 DEBUG:boto:Token: None 2023-10-11T15:14:23.165 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000039 2023-10-11T15:14:23.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hXfW0fwfpy+ZxfdINF0o/D6cQh8= 2023-10-11T15:14:23.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXfW0fwfpy+ZxfdINF0o/D6cQh8=', 'Content-Length': '0'} 2023-10-11T15:14:23.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6509013ed2da48-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.214 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.215 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.216 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.216 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.216 DEBUG:boto:Method: GET 2023-10-11T15:14:23.217 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.217 DEBUG:boto:Data: 2023-10-11T15:14:23.218 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.219 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.219 DEBUG:boto:Params: {} 2023-10-11T15:14:23.220 DEBUG:boto:Token: None 2023-10-11T15:14:23.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003108f1193ff67377-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.224 DEBUG:boto:b'dghqvd-58obj000000000000000000391000falseobj000000000000000000392023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.225 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.226 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.226 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.226 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.227 DEBUG:boto:Method: GET 2023-10-11T15:14:23.227 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000039 2023-10-11T15:14:23.228 DEBUG:boto:Data: 2023-10-11T15:14:23.228 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.229 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.229 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.229 DEBUG:boto:Params: {} 2023-10-11T15:14:23.230 DEBUG:boto:Token: None 2023-10-11T15:14:23.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.231 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015294ee1c14ecaaf-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.236 DEBUG:boto:b'dghqvd-58obj000000000000000000391000falseobj000000000000000000392023-10-11T15:14:05.688Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-11T15:14:23.237 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.238 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.238 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.239 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.239 DEBUG:boto:Method: GET 2023-10-11T15:14:23.239 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.240 DEBUG:boto:Data: 2023-10-11T15:14:23.240 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.241 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.242 DEBUG:boto:Params: {} 2023-10-11T15:14:23.242 DEBUG:boto:Token: None 2023-10-11T15:14:23.245 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8= 2023-10-11T15:14:23.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8=', 'Content-Length': '0'} 2023-10-11T15:14:23.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021cfc7b419417c02-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.290 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.291 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.291 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.292 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.292 DEBUG:boto:Method: GET 2023-10-11T15:14:23.293 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.293 DEBUG:boto:Data: 2023-10-11T15:14:23.294 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.294 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.295 DEBUG:boto:Params: {} 2023-10-11T15:14:23.295 DEBUG:boto:Token: None 2023-10-11T15:14:23.296 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8= 2023-10-11T15:14:23.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8=', 'Content-Length': '0'} 2023-10-11T15:14:23.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab63fecd364d4b3f-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.300 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.301 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.301 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.301 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.302 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.302 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.303 DEBUG:boto:Data: 2023-10-11T15:14:23.303 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.304 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.304 DEBUG:boto:Params: {} 2023-10-11T15:14:23.305 DEBUG:boto:Token: None 2023-10-11T15:14:23.305 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VLPlFsakUyT3FKQHpv+JDjnqVr4= 2023-10-11T15:14:23.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VLPlFsakUyT3FKQHpv+JDjnqVr4=', 'Content-Length': '0'} 2023-10-11T15:14:23.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001966aca9e20942d8-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.309 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.309 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.310 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.310 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.310 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.311 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.311 DEBUG:boto:Data: 2023-10-11T15:14:23.312 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.313 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.313 DEBUG:boto:Params: {} 2023-10-11T15:14:23.313 DEBUG:boto:Token: None 2023-10-11T15:14:23.314 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VLPlFsakUyT3FKQHpv+JDjnqVr4= 2023-10-11T15:14:23.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VLPlFsakUyT3FKQHpv+JDjnqVr4=', 'Content-Length': '0'} 2023-10-11T15:14:23.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a370c6f5f485bec-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-11T15:14:23.318 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.319 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.319 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.319 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.320 DEBUG:boto:Method: GET 2023-10-11T15:14:23.320 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.321 DEBUG:boto:Data: 2023-10-11T15:14:23.321 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.322 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.322 DEBUG:boto:Params: {} 2023-10-11T15:14:23.323 DEBUG:boto:Token: None 2023-10-11T15:14:23.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8= 2023-10-11T15:14:23.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8=', 'Content-Length': '0'} 2023-10-11T15:14:23.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f286f12b2883cb2-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.327 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.328 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.328 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.329 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.329 DEBUG:boto:Method: GET 2023-10-11T15:14:23.330 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.330 DEBUG:boto:Data: 2023-10-11T15:14:23.330 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.331 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.332 DEBUG:boto:Params: {} 2023-10-11T15:14:23.332 DEBUG:boto:Token: None 2023-10-11T15:14:23.332 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040?versionId=null 2023-10-11T15:14:23.333 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8= 2023-10-11T15:14:23.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vNNVPvAdjm67NA+FpPMgKlO5C8=', 'Content-Length': '0'} 2023-10-11T15:14:23.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824b24d21d926be4-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.336 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.336 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.337 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.337 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.338 DEBUG:boto:Method: GET 2023-10-11T15:14:23.338 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.338 DEBUG:boto:Data: 2023-10-11T15:14:23.339 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.340 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.340 DEBUG:boto:Params: {} 2023-10-11T15:14:23.340 DEBUG:boto:Token: None 2023-10-11T15:14:23.341 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042235c8936942509-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.345 DEBUG:boto:b'dghqvd-58obj000000000000000000401000falseobj000000000000000000402023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.346 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.346 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.347 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.347 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.348 DEBUG:boto:Method: GET 2023-10-11T15:14:23.348 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.348 DEBUG:boto:Data: 2023-10-11T15:14:23.349 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.350 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.350 DEBUG:boto:Params: {} 2023-10-11T15:14:23.351 DEBUG:boto:Token: None 2023-10-11T15:14:23.351 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd2423e36af8efe-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.398 DEBUG:boto:b'dghqvd-58obj000000000000000000401000falseobj000000000000000000402023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.399 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.399 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.400 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.400 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.401 DEBUG:boto:Data: 2023-10-11T15:14:23.401 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.402 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.402 DEBUG:boto:Params: {} 2023-10-11T15:14:23.402 DEBUG:boto:Token: None 2023-10-11T15:14:23.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ej6+U6pJEzDO3aZStMY8sla4UD0= 2023-10-11T15:14:23.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ej6+U6pJEzDO3aZStMY8sla4UD0=', 'Content-Length': '0'} 2023-10-11T15:14:23.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf8d4480a8ee536-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.406 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.407 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.407 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.408 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.408 DEBUG:boto:Data: 2023-10-11T15:14:23.409 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.409 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.410 DEBUG:boto:Params: {} 2023-10-11T15:14:23.410 DEBUG:boto:Token: None 2023-10-11T15:14:23.411 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ej6+U6pJEzDO3aZStMY8sla4UD0= 2023-10-11T15:14:23.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ej6+U6pJEzDO3aZStMY8sla4UD0=', 'Content-Length': '0'} 2023-10-11T15:14:23.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf1713da0bedad4-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-10-11T15:14:23.415 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.415 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.416 DEBUG:boto:Method: GET 2023-10-11T15:14:23.416 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.416 DEBUG:boto:Data: 2023-10-11T15:14:23.417 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.418 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.418 DEBUG:boto:Params: {} 2023-10-11T15:14:23.419 DEBUG:boto:Token: None 2023-10-11T15:14:23.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:649k341QS8tv4dtS+mMhWCGlgGQ= 2023-10-11T15:14:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:649k341QS8tv4dtS+mMhWCGlgGQ=', 'Content-Length': '0'} 2023-10-11T15:14:23.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f3644254a915231-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.423 DEBUG:boto:path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.423 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.423 DEBUG:boto:Method: GET 2023-10-11T15:14:23.424 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.424 DEBUG:boto:Data: 2023-10-11T15:14:23.425 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.426 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.426 DEBUG:boto:Params: {} 2023-10-11T15:14:23.426 DEBUG:boto:Token: None 2023-10-11T15:14:23.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000040 2023-10-11T15:14:23.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:649k341QS8tv4dtS+mMhWCGlgGQ= 2023-10-11T15:14:23.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:649k341QS8tv4dtS+mMhWCGlgGQ=', 'Content-Length': '0'} 2023-10-11T15:14:23.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec64205b91836e4-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.430 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.431 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.431 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.431 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.432 DEBUG:boto:Method: GET 2023-10-11T15:14:23.432 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.433 DEBUG:boto:Data: 2023-10-11T15:14:23.433 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.434 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.434 DEBUG:boto:Params: {} 2023-10-11T15:14:23.435 DEBUG:boto:Token: None 2023-10-11T15:14:23.435 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.436 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf0df48b5e4ff8a-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.439 DEBUG:boto:b'dghqvd-58obj000000000000000000401000falseobj000000000000000000402023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.440 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.440 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.441 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.441 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.442 DEBUG:boto:Method: GET 2023-10-11T15:14:23.442 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000040 2023-10-11T15:14:23.443 DEBUG:boto:Data: 2023-10-11T15:14:23.443 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.444 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.444 DEBUG:boto:Params: {} 2023-10-11T15:14:23.445 DEBUG:boto:Token: None 2023-10-11T15:14:23.445 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.446 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f700c114c90c30b-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.453 DEBUG:boto:b'dghqvd-58obj000000000000000000401000falseobj000000000000000000402023-10-11T15:14:04.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-11T15:14:23.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.455 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.456 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.456 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.456 DEBUG:boto:Method: GET 2023-10-11T15:14:23.457 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.457 DEBUG:boto:Data: 2023-10-11T15:14:23.458 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.459 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.459 DEBUG:boto:Params: {} 2023-10-11T15:14:23.460 DEBUG:boto:Token: None 2023-10-11T15:14:23.460 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc= 2023-10-11T15:14:23.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc=', 'Content-Length': '0'} 2023-10-11T15:14:23.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097dd40951e035162-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.510 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.511 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.511 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.512 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.512 DEBUG:boto:Method: GET 2023-10-11T15:14:23.513 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.513 DEBUG:boto:Data: 2023-10-11T15:14:23.514 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.514 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.515 DEBUG:boto:Params: {} 2023-10-11T15:14:23.515 DEBUG:boto:Token: None 2023-10-11T15:14:23.516 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc= 2023-10-11T15:14:23.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc=', 'Content-Length': '0'} 2023-10-11T15:14:23.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237ddfec2bf925eb-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.520 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.521 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.521 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.521 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.522 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.522 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.523 DEBUG:boto:Data: 2023-10-11T15:14:23.523 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.524 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.525 DEBUG:boto:Params: {} 2023-10-11T15:14:23.525 DEBUG:boto:Token: None 2023-10-11T15:14:23.526 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6aps9LZ0hxVyfIm6kC4YLOAXisY= 2023-10-11T15:14:23.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6aps9LZ0hxVyfIm6kC4YLOAXisY=', 'Content-Length': '0'} 2023-10-11T15:14:23.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726e8a4124b4f768-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.529 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.530 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.530 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.531 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.531 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.532 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.532 DEBUG:boto:Data: 2023-10-11T15:14:23.533 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.533 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.534 DEBUG:boto:Params: {} 2023-10-11T15:14:23.534 DEBUG:boto:Token: None 2023-10-11T15:14:23.535 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6aps9LZ0hxVyfIm6kC4YLOAXisY= 2023-10-11T15:14:23.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6aps9LZ0hxVyfIm6kC4YLOAXisY=', 'Content-Length': '0'} 2023-10-11T15:14:23.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512492daabb99c7c-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-11T15:14:23.539 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.540 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.540 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.540 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.541 DEBUG:boto:Method: GET 2023-10-11T15:14:23.541 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.542 DEBUG:boto:Data: 2023-10-11T15:14:23.542 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.543 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.544 DEBUG:boto:Params: {} 2023-10-11T15:14:23.544 DEBUG:boto:Token: None 2023-10-11T15:14:23.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc= 2023-10-11T15:14:23.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc=', 'Content-Length': '0'} 2023-10-11T15:14:23.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064796a9caca8960a-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.549 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.550 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.550 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.551 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.551 DEBUG:boto:Method: GET 2023-10-11T15:14:23.552 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.552 DEBUG:boto:Data: 2023-10-11T15:14:23.552 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.553 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.554 DEBUG:boto:Params: {} 2023-10-11T15:14:23.554 DEBUG:boto:Token: None 2023-10-11T15:14:23.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042?versionId=null 2023-10-11T15:14:23.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc= 2023-10-11T15:14:23.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:phUufOgNwQC7sZ2DsdVzKCU4dqc=', 'Content-Length': '0'} 2023-10-11T15:14:23.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021331676bd4408c-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.558 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.559 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.559 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.560 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.560 DEBUG:boto:Method: GET 2023-10-11T15:14:23.561 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.561 DEBUG:boto:Data: 2023-10-11T15:14:23.561 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.562 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.563 DEBUG:boto:Params: {} 2023-10-11T15:14:23.563 DEBUG:boto:Token: None 2023-10-11T15:14:23.564 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb6c6750a64a17d-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.568 DEBUG:boto:b'dghqvd-58obj000000000000000000421000falseobj000000000000000000422023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.568 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.569 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.569 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.570 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.570 DEBUG:boto:Method: GET 2023-10-11T15:14:23.571 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.571 DEBUG:boto:Data: 2023-10-11T15:14:23.572 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.572 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.573 DEBUG:boto:Params: {} 2023-10-11T15:14:23.573 DEBUG:boto:Token: None 2023-10-11T15:14:23.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004592710b8c2eddc8-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.579 DEBUG:boto:b'dghqvd-58obj000000000000000000421000falseobj000000000000000000422023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.580 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.580 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.581 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.581 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.582 DEBUG:boto:Data: 2023-10-11T15:14:23.582 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.583 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.583 DEBUG:boto:Params: {} 2023-10-11T15:14:23.584 DEBUG:boto:Token: None 2023-10-11T15:14:23.584 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqxiWoVu5Ks+iEGJSG24fLQS1Ck= 2023-10-11T15:14:23.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqxiWoVu5Ks+iEGJSG24fLQS1Ck=', 'Content-Length': '0'} 2023-10-11T15:14:23.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3bfe8ffa9b6ddeb-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.588 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.588 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.589 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.589 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.589 DEBUG:boto:Data: 2023-10-11T15:14:23.590 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.591 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.591 DEBUG:boto:Params: {} 2023-10-11T15:14:23.592 DEBUG:boto:Token: None 2023-10-11T15:14:23.592 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqxiWoVu5Ks+iEGJSG24fLQS1Ck= 2023-10-11T15:14:23.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqxiWoVu5Ks+iEGJSG24fLQS1Ck=', 'Content-Length': '0'} 2023-10-11T15:14:23.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ee1a5b9a52d43a-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-10-11T15:14:23.596 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.597 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.597 DEBUG:boto:Method: GET 2023-10-11T15:14:23.598 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.598 DEBUG:boto:Data: 2023-10-11T15:14:23.599 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.599 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.600 DEBUG:boto:Params: {} 2023-10-11T15:14:23.600 DEBUG:boto:Token: None 2023-10-11T15:14:23.601 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N7CtYoz+X3tB1ctTG6T6i6WYNGQ= 2023-10-11T15:14:23.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N7CtYoz+X3tB1ctTG6T6i6WYNGQ=', 'Content-Length': '0'} 2023-10-11T15:14:23.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005183e71c8f65bdf2-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.650 DEBUG:boto:path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.651 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.651 DEBUG:boto:Method: GET 2023-10-11T15:14:23.652 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.652 DEBUG:boto:Data: 2023-10-11T15:14:23.653 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.653 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.653 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.654 DEBUG:boto:Params: {} 2023-10-11T15:14:23.654 DEBUG:boto:Token: None 2023-10-11T15:14:23.655 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000042 2023-10-11T15:14:23.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N7CtYoz+X3tB1ctTG6T6i6WYNGQ= 2023-10-11T15:14:23.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N7CtYoz+X3tB1ctTG6T6i6WYNGQ=', 'Content-Length': '0'} 2023-10-11T15:14:23.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11cc6b66c03971d-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.661 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.662 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.662 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.662 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.663 DEBUG:boto:Method: GET 2023-10-11T15:14:23.663 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.664 DEBUG:boto:Data: 2023-10-11T15:14:23.664 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.665 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.666 DEBUG:boto:Params: {} 2023-10-11T15:14:23.667 DEBUG:boto:Token: None 2023-10-11T15:14:23.667 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f287ea3528269d3a-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.671 DEBUG:boto:b'dghqvd-58obj000000000000000000421000falseobj000000000000000000422023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.672 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.672 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.673 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.673 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.674 DEBUG:boto:Method: GET 2023-10-11T15:14:23.675 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000042 2023-10-11T15:14:23.675 DEBUG:boto:Data: 2023-10-11T15:14:23.675 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.676 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.677 DEBUG:boto:Params: {} 2023-10-11T15:14:23.677 DEBUG:boto:Token: None 2023-10-11T15:14:23.678 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.678 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e2497cff079099-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.683 DEBUG:boto:b'dghqvd-58obj000000000000000000421000falseobj000000000000000000422023-10-11T15:14:02.681Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-11T15:14:23.685 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.685 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.686 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.686 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.687 DEBUG:boto:Method: GET 2023-10-11T15:14:23.687 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.687 DEBUG:boto:Data: 2023-10-11T15:14:23.688 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.689 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.689 DEBUG:boto:Params: {} 2023-10-11T15:14:23.690 DEBUG:boto:Token: None 2023-10-11T15:14:23.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q= 2023-10-11T15:14:23.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q=', 'Content-Length': '0'} 2023-10-11T15:14:23.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9850334b11ade66-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.694 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.695 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.695 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.695 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.696 DEBUG:boto:Method: GET 2023-10-11T15:14:23.696 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.697 DEBUG:boto:Data: 2023-10-11T15:14:23.697 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.698 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.698 DEBUG:boto:Params: {} 2023-10-11T15:14:23.699 DEBUG:boto:Token: None 2023-10-11T15:14:23.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q= 2023-10-11T15:14:23.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q=', 'Content-Length': '0'} 2023-10-11T15:14:23.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006190a7133950b0cf-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.703 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.705 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.705 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.705 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.706 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.706 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.707 DEBUG:boto:Data: 2023-10-11T15:14:23.707 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.708 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.708 DEBUG:boto:Params: {} 2023-10-11T15:14:23.709 DEBUG:boto:Token: None 2023-10-11T15:14:23.709 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1qa7gNvT/Z21D/KADR8iSjcZ/7A= 2023-10-11T15:14:23.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1qa7gNvT/Z21D/KADR8iSjcZ/7A=', 'Content-Length': '0'} 2023-10-11T15:14:23.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a3795da05c03b8e-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.713 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.713 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.714 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.714 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.714 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.715 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.715 DEBUG:boto:Data: 2023-10-11T15:14:23.716 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.716 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.716 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.717 DEBUG:boto:Params: {} 2023-10-11T15:14:23.717 DEBUG:boto:Token: None 2023-10-11T15:14:23.717 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.718 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1qa7gNvT/Z21D/KADR8iSjcZ/7A= 2023-10-11T15:14:23.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1qa7gNvT/Z21D/KADR8iSjcZ/7A=', 'Content-Length': '0'} 2023-10-11T15:14:23.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aa6f99085fb6f08-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-11T15:14:23.722 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.722 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.722 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.723 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.723 DEBUG:boto:Method: GET 2023-10-11T15:14:23.724 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.724 DEBUG:boto:Data: 2023-10-11T15:14:23.725 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.725 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.726 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.726 DEBUG:boto:Params: {} 2023-10-11T15:14:23.726 DEBUG:boto:Token: None 2023-10-11T15:14:23.727 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.727 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q= 2023-10-11T15:14:23.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q=', 'Content-Length': '0'} 2023-10-11T15:14:23.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d15bd686849cb0-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.731 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.731 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.732 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.732 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.732 DEBUG:boto:Method: GET 2023-10-11T15:14:23.733 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.733 DEBUG:boto:Data: 2023-10-11T15:14:23.734 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.735 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.735 DEBUG:boto:Params: {} 2023-10-11T15:14:23.735 DEBUG:boto:Token: None 2023-10-11T15:14:23.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045?versionId=null 2023-10-11T15:14:23.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q= 2023-10-11T15:14:23.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6VWoJ2E48xTqBr8Q2QInmymh1Q=', 'Content-Length': '0'} 2023-10-11T15:14:23.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a21049c32e93041-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.739 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.740 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.740 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.741 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.741 DEBUG:boto:Method: GET 2023-10-11T15:14:23.742 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.742 DEBUG:boto:Data: 2023-10-11T15:14:23.742 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.743 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.744 DEBUG:boto:Params: {} 2023-10-11T15:14:23.744 DEBUG:boto:Token: None 2023-10-11T15:14:23.745 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001a02395ea800d07-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.748 DEBUG:boto:b'dghqvd-58obj000000000000000000451000falseobj000000000000000000452023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.749 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.750 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.750 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.750 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.751 DEBUG:boto:Method: GET 2023-10-11T15:14:23.751 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.752 DEBUG:boto:Data: 2023-10-11T15:14:23.752 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.753 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.753 DEBUG:boto:Params: {} 2023-10-11T15:14:23.754 DEBUG:boto:Token: None 2023-10-11T15:14:23.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006116ba6ab1f9980-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.806 DEBUG:boto:b'dghqvd-58obj000000000000000000451000falseobj000000000000000000452023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.807 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.808 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.808 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.808 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.809 DEBUG:boto:Data: 2023-10-11T15:14:23.809 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.810 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.810 DEBUG:boto:Params: {} 2023-10-11T15:14:23.811 DEBUG:boto:Token: None 2023-10-11T15:14:23.811 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:svFaVJDskE9A3OG8GizUH+wDNf0= 2023-10-11T15:14:23.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:svFaVJDskE9A3OG8GizUH+wDNf0=', 'Content-Length': '0'} 2023-10-11T15:14:23.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d09de6d31ad160f-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.815 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.816 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.816 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.816 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.817 DEBUG:boto:Data: 2023-10-11T15:14:23.817 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.818 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.818 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.819 DEBUG:boto:Params: {} 2023-10-11T15:14:23.819 DEBUG:boto:Token: None 2023-10-11T15:14:23.819 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:svFaVJDskE9A3OG8GizUH+wDNf0= 2023-10-11T15:14:23.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:svFaVJDskE9A3OG8GizUH+wDNf0=', 'Content-Length': '0'} 2023-10-11T15:14:23.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c8833af31fd53e-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-10-11T15:14:23.824 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.824 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.824 DEBUG:boto:Method: GET 2023-10-11T15:14:23.825 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.825 DEBUG:boto:Data: 2023-10-11T15:14:23.826 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.827 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.827 DEBUG:boto:Params: {} 2023-10-11T15:14:23.828 DEBUG:boto:Token: None 2023-10-11T15:14:23.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GSsSMrXN8Y61qo+98d+8MGUHxYc= 2023-10-11T15:14:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GSsSMrXN8Y61qo+98d+8MGUHxYc=', 'Content-Length': '0'} 2023-10-11T15:14:23.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f77d57c4ae3c399-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.875 DEBUG:boto:path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.876 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.877 DEBUG:boto:Method: GET 2023-10-11T15:14:23.878 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.879 DEBUG:boto:Data: 2023-10-11T15:14:23.879 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.880 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.881 DEBUG:boto:Params: {} 2023-10-11T15:14:23.882 DEBUG:boto:Token: None 2023-10-11T15:14:23.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000045 2023-10-11T15:14:23.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GSsSMrXN8Y61qo+98d+8MGUHxYc= 2023-10-11T15:14:23.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GSsSMrXN8Y61qo+98d+8MGUHxYc=', 'Content-Length': '0'} 2023-10-11T15:14:23.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0655da4dc3c496a-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.887 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.888 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.888 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.889 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.889 DEBUG:boto:Method: GET 2023-10-11T15:14:23.890 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.890 DEBUG:boto:Data: 2023-10-11T15:14:23.891 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.892 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.892 DEBUG:boto:Params: {} 2023-10-11T15:14:23.893 DEBUG:boto:Token: None 2023-10-11T15:14:23.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643354b1bb2efed4-006526bbcf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.898 DEBUG:boto:b'dghqvd-58obj000000000000000000451000falseobj000000000000000000452023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.899 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:23.899 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:23.900 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.901 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.901 DEBUG:boto:Method: GET 2023-10-11T15:14:23.901 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000045 2023-10-11T15:14:23.902 DEBUG:boto:Data: 2023-10-11T15:14:23.902 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.903 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.904 DEBUG:boto:Params: {} 2023-10-11T15:14:23.905 DEBUG:boto:Token: None 2023-10-11T15:14:23.906 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/ 2023-10-11T15:14:23.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc= 2023-10-11T15:14:23.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5O/rc8qTDwJb4ktVhLej6sciUVc=', 'Content-Length': '0'} 2023-10-11T15:14:23.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000671b88c3e4f386a9-006526bbcf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.913 DEBUG:boto:b'dghqvd-58obj000000000000000000451000falseobj000000000000000000452023-10-11T15:14:05.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:23.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-11T15:14:23.915 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.915 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.916 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.916 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.917 DEBUG:boto:Method: GET 2023-10-11T15:14:23.917 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.918 DEBUG:boto:Data: 2023-10-11T15:14:23.918 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.919 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.920 DEBUG:boto:Params: {} 2023-10-11T15:14:23.920 DEBUG:boto:Token: None 2023-10-11T15:14:23.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lfiYwj2aeLsuFHWo/2suHpWNCrw= 2023-10-11T15:14:23.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lfiYwj2aeLsuFHWo/2suHpWNCrw=', 'Content-Length': '0'} 2023-10-11T15:14:23.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b96c4d8584fc8479-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.925 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.925 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.926 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.926 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.927 DEBUG:boto:Method: GET 2023-10-11T15:14:23.927 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.928 DEBUG:boto:Data: 2023-10-11T15:14:23.928 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.929 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.930 DEBUG:boto:Params: {} 2023-10-11T15:14:23.930 DEBUG:boto:Token: None 2023-10-11T15:14:23.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lfiYwj2aeLsuFHWo/2suHpWNCrw= 2023-10-11T15:14:23.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lfiYwj2aeLsuFHWo/2suHpWNCrw=', 'Content-Length': '0'} 2023-10-11T15:14:23.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd8cfbc68fb39e1-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.979 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.980 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.981 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.982 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.982 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.983 DEBUG:boto:Data: 2023-10-11T15:14:23.984 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.984 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:23.984 DEBUG:boto:Port: 8001 2023-10-11T15:14:23.985 DEBUG:boto:Params: {} 2023-10-11T15:14:23.985 DEBUG:boto:Token: None 2023-10-11T15:14:23.986 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.986 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MOQ2Yvs3jUx2T/x/flYi+8p/j9o= 2023-10-11T15:14:23.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MOQ2Yvs3jUx2T/x/flYi+8p/j9o=', 'Content-Length': '0'} 2023-10-11T15:14:23.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1f2d2acb21737e-006526bbcf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.990 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.990 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:23.991 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.991 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.991 DEBUG:boto:Method: HEAD 2023-10-11T15:14:23.992 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.992 DEBUG:boto:Data: 2023-10-11T15:14:23.993 DEBUG:boto:Headers: {} 2023-10-11T15:14:23.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:23.994 DEBUG:boto:Port: 8000 2023-10-11T15:14:23.994 DEBUG:boto:Params: {} 2023-10-11T15:14:23.995 DEBUG:boto:Token: None 2023-10-11T15:14:23.995 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:23 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:23.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MOQ2Yvs3jUx2T/x/flYi+8p/j9o= 2023-10-11T15:14:23.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:23 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MOQ2Yvs3jUx2T/x/flYi+8p/j9o=', 'Content-Length': '0'} 2023-10-11T15:14:23.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66aaad23042952f-006526bbcf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:23 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:23.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-11T15:14:24.000 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.000 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.001 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.002 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.003 DEBUG:boto:Method: GET 2023-10-11T15:14:24.003 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.004 DEBUG:boto:Data: 2023-10-11T15:14:24.005 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.006 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.007 DEBUG:boto:Params: {} 2023-10-11T15:14:24.008 DEBUG:boto:Token: None 2023-10-11T15:14:24.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZNqNOq3gn4Cef3pxLuUTphYKhwg= 2023-10-11T15:14:24.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZNqNOq3gn4Cef3pxLuUTphYKhwg=', 'Content-Length': '0'} 2023-10-11T15:14:24.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee10f15f7c6d6a6f-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.012 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.013 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.013 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.014 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.014 DEBUG:boto:Method: GET 2023-10-11T15:14:24.015 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.015 DEBUG:boto:Data: 2023-10-11T15:14:24.016 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.016 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.017 DEBUG:boto:Params: {} 2023-10-11T15:14:24.017 DEBUG:boto:Token: None 2023-10-11T15:14:24.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046?versionId=null 2023-10-11T15:14:24.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZNqNOq3gn4Cef3pxLuUTphYKhwg= 2023-10-11T15:14:24.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZNqNOq3gn4Cef3pxLuUTphYKhwg=', 'Content-Length': '0'} 2023-10-11T15:14:24.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cacb246a93c08b-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.022 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.022 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.023 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.023 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.024 DEBUG:boto:Method: GET 2023-10-11T15:14:24.024 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.025 DEBUG:boto:Data: 2023-10-11T15:14:24.026 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.027 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.028 DEBUG:boto:Params: {} 2023-10-11T15:14:24.029 DEBUG:boto:Token: None 2023-10-11T15:14:24.030 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed8c2e05663a4df-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.034 DEBUG:boto:b'dghqvd-58obj000000000000000000461000falseobj000000000000000000462023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.035 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.035 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.036 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.036 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.037 DEBUG:boto:Method: GET 2023-10-11T15:14:24.037 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.038 DEBUG:boto:Data: 2023-10-11T15:14:24.038 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.039 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.039 DEBUG:boto:Params: {} 2023-10-11T15:14:24.040 DEBUG:boto:Token: None 2023-10-11T15:14:24.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f697edad868e4bba-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.047 DEBUG:boto:b'dghqvd-58obj000000000000000000461000falseobj000000000000000000462023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.048 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.049 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.050 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.050 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.051 DEBUG:boto:Data: 2023-10-11T15:14:24.051 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.052 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.053 DEBUG:boto:Params: {} 2023-10-11T15:14:24.053 DEBUG:boto:Token: None 2023-10-11T15:14:24.054 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k1sXaUkKtlGXlC53G58V4DorTWM= 2023-10-11T15:14:24.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k1sXaUkKtlGXlC53G58V4DorTWM=', 'Content-Length': '0'} 2023-10-11T15:14:24.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b3819edf3a2cb9-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.058 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.058 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.059 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.059 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.060 DEBUG:boto:Data: 2023-10-11T15:14:24.060 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.061 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.061 DEBUG:boto:Params: {} 2023-10-11T15:14:24.062 DEBUG:boto:Token: None 2023-10-11T15:14:24.062 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k1sXaUkKtlGXlC53G58V4DorTWM= 2023-10-11T15:14:24.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k1sXaUkKtlGXlC53G58V4DorTWM=', 'Content-Length': '0'} 2023-10-11T15:14:24.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72a43e712faa56f-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-10-11T15:14:24.067 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.067 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.068 DEBUG:boto:Method: GET 2023-10-11T15:14:24.068 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.068 DEBUG:boto:Data: 2023-10-11T15:14:24.069 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.070 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.070 DEBUG:boto:Params: {} 2023-10-11T15:14:24.074 DEBUG:boto:Token: None 2023-10-11T15:14:24.075 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CsNOxKoJsJngEhPI5/Wa4nuihn8= 2023-10-11T15:14:24.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CsNOxKoJsJngEhPI5/Wa4nuihn8=', 'Content-Length': '0'} 2023-10-11T15:14:24.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ad7779c849b837-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.079 DEBUG:boto:path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.079 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.079 DEBUG:boto:Method: GET 2023-10-11T15:14:24.080 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.080 DEBUG:boto:Data: 2023-10-11T15:14:24.081 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.082 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.082 DEBUG:boto:Params: {} 2023-10-11T15:14:24.082 DEBUG:boto:Token: None 2023-10-11T15:14:24.083 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000046 2023-10-11T15:14:24.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CsNOxKoJsJngEhPI5/Wa4nuihn8= 2023-10-11T15:14:24.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CsNOxKoJsJngEhPI5/Wa4nuihn8=', 'Content-Length': '0'} 2023-10-11T15:14:24.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec880b66fd2cc587-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.126 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.127 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.128 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.128 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.128 DEBUG:boto:Method: GET 2023-10-11T15:14:24.129 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.129 DEBUG:boto:Data: 2023-10-11T15:14:24.130 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.131 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.131 DEBUG:boto:Params: {} 2023-10-11T15:14:24.132 DEBUG:boto:Token: None 2023-10-11T15:14:24.132 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf02a46e10eae48f-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.136 DEBUG:boto:b'dghqvd-58obj000000000000000000461000falseobj000000000000000000462023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.137 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.137 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.138 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.138 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.139 DEBUG:boto:Method: GET 2023-10-11T15:14:24.139 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000046 2023-10-11T15:14:24.139 DEBUG:boto:Data: 2023-10-11T15:14:24.140 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.140 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.141 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.141 DEBUG:boto:Params: {} 2023-10-11T15:14:24.141 DEBUG:boto:Token: None 2023-10-11T15:14:24.142 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.142 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002fcbf4232ee53ca-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.148 DEBUG:boto:b'dghqvd-58obj000000000000000000461000falseobj000000000000000000462023-10-11T15:14:01.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-11T15:14:24.149 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.149 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.150 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.150 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.151 DEBUG:boto:Method: GET 2023-10-11T15:14:24.151 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.152 DEBUG:boto:Data: 2023-10-11T15:14:24.152 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.152 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.153 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.153 DEBUG:boto:Params: {} 2023-10-11T15:14:24.154 DEBUG:boto:Token: None 2023-10-11T15:14:24.154 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc= 2023-10-11T15:14:24.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc=', 'Content-Length': '0'} 2023-10-11T15:14:24.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31d3d6afce4eaf7-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.198 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.199 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.199 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.200 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.200 DEBUG:boto:Method: GET 2023-10-11T15:14:24.200 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.201 DEBUG:boto:Data: 2023-10-11T15:14:24.201 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.202 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.203 DEBUG:boto:Params: {} 2023-10-11T15:14:24.203 DEBUG:boto:Token: None 2023-10-11T15:14:24.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.204 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc= 2023-10-11T15:14:24.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc=', 'Content-Length': '0'} 2023-10-11T15:14:24.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f874a778836ee351-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.208 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.208 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.208 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.209 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.209 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.210 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.210 DEBUG:boto:Data: 2023-10-11T15:14:24.211 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.211 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.212 DEBUG:boto:Params: {} 2023-10-11T15:14:24.212 DEBUG:boto:Token: None 2023-10-11T15:14:24.213 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+Qm+t++gam8J7xFKKXgAKFsWfsQ= 2023-10-11T15:14:24.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+Qm+t++gam8J7xFKKXgAKFsWfsQ=', 'Content-Length': '0'} 2023-10-11T15:14:24.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449ad3dc0f2c1e8f-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.216 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.216 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.217 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.217 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.218 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.218 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.218 DEBUG:boto:Data: 2023-10-11T15:14:24.219 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.221 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.222 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.222 DEBUG:boto:Params: {} 2023-10-11T15:14:24.223 DEBUG:boto:Token: None 2023-10-11T15:14:24.224 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+Qm+t++gam8J7xFKKXgAKFsWfsQ= 2023-10-11T15:14:24.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+Qm+t++gam8J7xFKKXgAKFsWfsQ=', 'Content-Length': '0'} 2023-10-11T15:14:24.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c4ae115ded4659-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-11T15:14:24.228 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.229 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.229 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.230 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.230 DEBUG:boto:Method: GET 2023-10-11T15:14:24.230 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.231 DEBUG:boto:Data: 2023-10-11T15:14:24.231 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.232 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.232 DEBUG:boto:Params: {} 2023-10-11T15:14:24.233 DEBUG:boto:Token: None 2023-10-11T15:14:24.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc= 2023-10-11T15:14:24.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc=', 'Content-Length': '0'} 2023-10-11T15:14:24.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68998c86cf83001-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.237 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.238 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.238 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.239 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.239 DEBUG:boto:Method: GET 2023-10-11T15:14:24.240 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.240 DEBUG:boto:Data: 2023-10-11T15:14:24.240 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.241 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.242 DEBUG:boto:Params: {} 2023-10-11T15:14:24.242 DEBUG:boto:Token: None 2023-10-11T15:14:24.243 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048?versionId=null 2023-10-11T15:14:24.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc= 2023-10-11T15:14:24.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xt80MyAUXI/qgYIWeqtuwBw6wNc=', 'Content-Length': '0'} 2023-10-11T15:14:24.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f409348540a773a-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.286 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.287 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.287 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.288 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.288 DEBUG:boto:Method: GET 2023-10-11T15:14:24.288 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.289 DEBUG:boto:Data: 2023-10-11T15:14:24.289 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.290 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.291 DEBUG:boto:Params: {} 2023-10-11T15:14:24.291 DEBUG:boto:Token: None 2023-10-11T15:14:24.291 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b46cb041c5c679e-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.296 DEBUG:boto:b'dghqvd-58obj000000000000000000481000falseobj000000000000000000482023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.296 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.297 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.297 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.298 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.298 DEBUG:boto:Method: GET 2023-10-11T15:14:24.299 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.299 DEBUG:boto:Data: 2023-10-11T15:14:24.299 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.300 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.300 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.301 DEBUG:boto:Params: {} 2023-10-11T15:14:24.301 DEBUG:boto:Token: None 2023-10-11T15:14:24.302 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085dc62d2f848ded7-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.307 DEBUG:boto:b'dghqvd-58obj000000000000000000481000falseobj000000000000000000482023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.308 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.308 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.309 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.309 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.310 DEBUG:boto:Data: 2023-10-11T15:14:24.310 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.311 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.311 DEBUG:boto:Params: {} 2023-10-11T15:14:24.312 DEBUG:boto:Token: None 2023-10-11T15:14:24.312 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CZLNtOe0DlPeocClHC6NTgvA7sI= 2023-10-11T15:14:24.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CZLNtOe0DlPeocClHC6NTgvA7sI=', 'Content-Length': '0'} 2023-10-11T15:14:24.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a57ea36ba4445fb-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.315 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.316 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.316 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.317 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.317 DEBUG:boto:Data: 2023-10-11T15:14:24.317 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.318 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.319 DEBUG:boto:Params: {} 2023-10-11T15:14:24.319 DEBUG:boto:Token: None 2023-10-11T15:14:24.319 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CZLNtOe0DlPeocClHC6NTgvA7sI= 2023-10-11T15:14:24.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CZLNtOe0DlPeocClHC6NTgvA7sI=', 'Content-Length': '0'} 2023-10-11T15:14:24.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df118737f346337c-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-10-11T15:14:24.323 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.324 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.324 DEBUG:boto:Method: GET 2023-10-11T15:14:24.325 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.325 DEBUG:boto:Data: 2023-10-11T15:14:24.326 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.327 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.327 DEBUG:boto:Params: {} 2023-10-11T15:14:24.327 DEBUG:boto:Token: None 2023-10-11T15:14:24.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P20eNJ10rleNL42o9hmho5iiszc= 2023-10-11T15:14:24.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P20eNJ10rleNL42o9hmho5iiszc=', 'Content-Length': '0'} 2023-10-11T15:14:24.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fb3aeb07c91a74-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.374 DEBUG:boto:path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.375 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.375 DEBUG:boto:Method: GET 2023-10-11T15:14:24.376 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.376 DEBUG:boto:Data: 2023-10-11T15:14:24.377 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.378 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.378 DEBUG:boto:Params: {} 2023-10-11T15:14:24.378 DEBUG:boto:Token: None 2023-10-11T15:14:24.379 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000048 2023-10-11T15:14:24.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P20eNJ10rleNL42o9hmho5iiszc= 2023-10-11T15:14:24.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P20eNJ10rleNL42o9hmho5iiszc=', 'Content-Length': '0'} 2023-10-11T15:14:24.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359453d663c7b9c2-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.383 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.383 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.384 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.384 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.385 DEBUG:boto:Method: GET 2023-10-11T15:14:24.385 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.386 DEBUG:boto:Data: 2023-10-11T15:14:24.386 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.387 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.387 DEBUG:boto:Params: {} 2023-10-11T15:14:24.388 DEBUG:boto:Token: None 2023-10-11T15:14:24.388 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf97e6839ce0ffdf-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.392 DEBUG:boto:b'dghqvd-58obj000000000000000000481000falseobj000000000000000000482023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.393 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.393 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.394 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.394 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.395 DEBUG:boto:Method: GET 2023-10-11T15:14:24.395 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000048 2023-10-11T15:14:24.396 DEBUG:boto:Data: 2023-10-11T15:14:24.396 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.397 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.397 DEBUG:boto:Params: {} 2023-10-11T15:14:24.398 DEBUG:boto:Token: None 2023-10-11T15:14:24.398 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2cc77e0f075e0b1-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.450 DEBUG:boto:b'dghqvd-58obj000000000000000000481000falseobj000000000000000000482023-10-11T15:14:04.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-11T15:14:24.452 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.452 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.453 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.453 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.453 DEBUG:boto:Method: GET 2023-10-11T15:14:24.454 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.454 DEBUG:boto:Data: 2023-10-11T15:14:24.455 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.456 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.456 DEBUG:boto:Params: {} 2023-10-11T15:14:24.456 DEBUG:boto:Token: None 2023-10-11T15:14:24.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4= 2023-10-11T15:14:24.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4=', 'Content-Length': '0'} 2023-10-11T15:14:24.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4bbda575f0aaa5e-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.461 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.461 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.462 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.462 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.463 DEBUG:boto:Method: GET 2023-10-11T15:14:24.463 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.464 DEBUG:boto:Data: 2023-10-11T15:14:24.464 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.465 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.465 DEBUG:boto:Params: {} 2023-10-11T15:14:24.466 DEBUG:boto:Token: None 2023-10-11T15:14:24.466 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4= 2023-10-11T15:14:24.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4=', 'Content-Length': '0'} 2023-10-11T15:14:24.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013489159b55b6772-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.470 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.470 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.471 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.471 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.472 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.472 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.472 DEBUG:boto:Data: 2023-10-11T15:14:24.473 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.474 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.474 DEBUG:boto:Params: {} 2023-10-11T15:14:24.475 DEBUG:boto:Token: None 2023-10-11T15:14:24.475 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gwgsu9X+3Xd3yVYiGFt/WjA3LMs= 2023-10-11T15:14:24.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gwgsu9X+3Xd3yVYiGFt/WjA3LMs=', 'Content-Length': '0'} 2023-10-11T15:14:24.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000324fe52d6d60038-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.479 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.479 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.480 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.480 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.480 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.481 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.481 DEBUG:boto:Data: 2023-10-11T15:14:24.482 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.482 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.483 DEBUG:boto:Params: {} 2023-10-11T15:14:24.483 DEBUG:boto:Token: None 2023-10-11T15:14:24.484 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gwgsu9X+3Xd3yVYiGFt/WjA3LMs= 2023-10-11T15:14:24.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gwgsu9X+3Xd3yVYiGFt/WjA3LMs=', 'Content-Length': '0'} 2023-10-11T15:14:24.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022d20bc20cbc5317-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-11T15:14:24.488 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.488 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.489 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.489 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.489 DEBUG:boto:Method: GET 2023-10-11T15:14:24.490 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.490 DEBUG:boto:Data: 2023-10-11T15:14:24.491 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.491 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.492 DEBUG:boto:Params: {} 2023-10-11T15:14:24.492 DEBUG:boto:Token: None 2023-10-11T15:14:24.493 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4= 2023-10-11T15:14:24.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4=', 'Content-Length': '0'} 2023-10-11T15:14:24.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06fc35db14b1a49-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.497 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.497 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.498 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.498 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.499 DEBUG:boto:Method: GET 2023-10-11T15:14:24.499 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.499 DEBUG:boto:Data: 2023-10-11T15:14:24.500 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.501 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.501 DEBUG:boto:Params: {} 2023-10-11T15:14:24.501 DEBUG:boto:Token: None 2023-10-11T15:14:24.502 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049?versionId=null 2023-10-11T15:14:24.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4= 2023-10-11T15:14:24.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgB07o0cW6awL/WTyPyPa8C7sO4=', 'Content-Length': '0'} 2023-10-11T15:14:24.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c7c8031f1991082-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.506 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.506 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.507 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.507 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.508 DEBUG:boto:Method: GET 2023-10-11T15:14:24.508 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.509 DEBUG:boto:Data: 2023-10-11T15:14:24.509 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.510 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.510 DEBUG:boto:Params: {} 2023-10-11T15:14:24.511 DEBUG:boto:Token: None 2023-10-11T15:14:24.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3404761151df9ae-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.516 DEBUG:boto:b'dghqvd-58obj000000000000000000491000falseobj000000000000000000492023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.516 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.517 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.517 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.518 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.518 DEBUG:boto:Method: GET 2023-10-11T15:14:24.518 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.519 DEBUG:boto:Data: 2023-10-11T15:14:24.519 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.520 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.520 DEBUG:boto:Params: {} 2023-10-11T15:14:24.521 DEBUG:boto:Token: None 2023-10-11T15:14:24.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce17c5c8bd182957-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.528 DEBUG:boto:b'dghqvd-58obj000000000000000000491000falseobj000000000000000000492023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.528 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.529 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.529 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.530 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.530 DEBUG:boto:Data: 2023-10-11T15:14:24.530 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.531 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.532 DEBUG:boto:Params: {} 2023-10-11T15:14:24.532 DEBUG:boto:Token: None 2023-10-11T15:14:24.532 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZRGVDOZwjcCkIXuXRdZDF6ooqbw= 2023-10-11T15:14:24.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZRGVDOZwjcCkIXuXRdZDF6ooqbw=', 'Content-Length': '0'} 2023-10-11T15:14:24.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003beee18c1078d8e3-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.536 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.536 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.537 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.537 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.537 DEBUG:boto:Data: 2023-10-11T15:14:24.538 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.539 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.539 DEBUG:boto:Params: {} 2023-10-11T15:14:24.540 DEBUG:boto:Token: None 2023-10-11T15:14:24.540 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZRGVDOZwjcCkIXuXRdZDF6ooqbw= 2023-10-11T15:14:24.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZRGVDOZwjcCkIXuXRdZDF6ooqbw=', 'Content-Length': '0'} 2023-10-11T15:14:24.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4e64bf2a3a9366-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-10-11T15:14:24.544 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.544 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.545 DEBUG:boto:Method: GET 2023-10-11T15:14:24.545 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.546 DEBUG:boto:Data: 2023-10-11T15:14:24.546 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.547 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.547 DEBUG:boto:Params: {} 2023-10-11T15:14:24.548 DEBUG:boto:Token: None 2023-10-11T15:14:24.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/WHU65/3SPIL3OGu1heQUNIiorM= 2023-10-11T15:14:24.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/WHU65/3SPIL3OGu1heQUNIiorM=', 'Content-Length': '0'} 2023-10-11T15:14:24.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f18e06a9809bc9b-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.595 DEBUG:boto:path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.595 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.596 DEBUG:boto:Method: GET 2023-10-11T15:14:24.596 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.597 DEBUG:boto:Data: 2023-10-11T15:14:24.597 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.598 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.598 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.598 DEBUG:boto:Params: {} 2023-10-11T15:14:24.599 DEBUG:boto:Token: None 2023-10-11T15:14:24.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000049 2023-10-11T15:14:24.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/WHU65/3SPIL3OGu1heQUNIiorM= 2023-10-11T15:14:24.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/WHU65/3SPIL3OGu1heQUNIiorM=', 'Content-Length': '0'} 2023-10-11T15:14:24.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20e717edc438f8a-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.603 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.604 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.604 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.605 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.605 DEBUG:boto:Method: GET 2023-10-11T15:14:24.605 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.606 DEBUG:boto:Data: 2023-10-11T15:14:24.606 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.607 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.608 DEBUG:boto:Params: {} 2023-10-11T15:14:24.608 DEBUG:boto:Token: None 2023-10-11T15:14:24.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad5fa2940dde9ac-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.613 DEBUG:boto:b'dghqvd-58obj000000000000000000491000falseobj000000000000000000492023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.614 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.614 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.615 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.616 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.616 DEBUG:boto:Method: GET 2023-10-11T15:14:24.617 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000049 2023-10-11T15:14:24.617 DEBUG:boto:Data: 2023-10-11T15:14:24.617 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.618 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.619 DEBUG:boto:Params: {} 2023-10-11T15:14:24.619 DEBUG:boto:Token: None 2023-10-11T15:14:24.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dac93ab43758d4a-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.625 DEBUG:boto:b'dghqvd-58obj000000000000000000491000falseobj000000000000000000492023-10-11T15:14:01.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-11T15:14:24.626 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.627 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.627 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.628 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.628 DEBUG:boto:Method: GET 2023-10-11T15:14:24.628 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.629 DEBUG:boto:Data: 2023-10-11T15:14:24.629 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.630 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.631 DEBUG:boto:Params: {} 2023-10-11T15:14:24.631 DEBUG:boto:Token: None 2023-10-11T15:14:24.631 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw= 2023-10-11T15:14:24.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw=', 'Content-Length': '0'} 2023-10-11T15:14:24.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c36a2361cccc779d-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.635 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.636 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.636 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.637 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.637 DEBUG:boto:Method: GET 2023-10-11T15:14:24.637 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.638 DEBUG:boto:Data: 2023-10-11T15:14:24.638 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.639 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.639 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.640 DEBUG:boto:Params: {} 2023-10-11T15:14:24.640 DEBUG:boto:Token: None 2023-10-11T15:14:24.641 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.641 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw= 2023-10-11T15:14:24.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw=', 'Content-Length': '0'} 2023-10-11T15:14:24.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000046536a9903acfa7-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.644 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.645 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.645 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.646 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.646 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.646 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.647 DEBUG:boto:Data: 2023-10-11T15:14:24.647 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.648 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.648 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.648 DEBUG:boto:Params: {} 2023-10-11T15:14:24.649 DEBUG:boto:Token: None 2023-10-11T15:14:24.649 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.650 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6y70UnwQRFPXpvkGxn+uAETwdZg= 2023-10-11T15:14:24.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6y70UnwQRFPXpvkGxn+uAETwdZg=', 'Content-Length': '0'} 2023-10-11T15:14:24.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008424be6a8011dd5f-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.653 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.653 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.654 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.654 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.655 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.655 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.655 DEBUG:boto:Data: 2023-10-11T15:14:24.656 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.657 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.657 DEBUG:boto:Params: {} 2023-10-11T15:14:24.657 DEBUG:boto:Token: None 2023-10-11T15:14:24.658 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6y70UnwQRFPXpvkGxn+uAETwdZg= 2023-10-11T15:14:24.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6y70UnwQRFPXpvkGxn+uAETwdZg=', 'Content-Length': '0'} 2023-10-11T15:14:24.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816d2906c58e7fad-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-11T15:14:24.662 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.663 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.663 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.663 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.664 DEBUG:boto:Method: GET 2023-10-11T15:14:24.664 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.665 DEBUG:boto:Data: 2023-10-11T15:14:24.665 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.666 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.666 DEBUG:boto:Params: {} 2023-10-11T15:14:24.667 DEBUG:boto:Token: None 2023-10-11T15:14:24.667 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw= 2023-10-11T15:14:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw=', 'Content-Length': '0'} 2023-10-11T15:14:24.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028640fc6e100114-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.671 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.671 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.672 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.672 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.673 DEBUG:boto:Method: GET 2023-10-11T15:14:24.673 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.673 DEBUG:boto:Data: 2023-10-11T15:14:24.674 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.675 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.675 DEBUG:boto:Params: {} 2023-10-11T15:14:24.676 DEBUG:boto:Token: None 2023-10-11T15:14:24.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050?versionId=null 2023-10-11T15:14:24.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw= 2023-10-11T15:14:24.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pCC4+AWnnemsp9H6f023iUEGBYw=', 'Content-Length': '0'} 2023-10-11T15:14:24.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267e4ec942d44be1-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.680 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.680 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.681 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.681 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.682 DEBUG:boto:Method: GET 2023-10-11T15:14:24.682 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.682 DEBUG:boto:Data: 2023-10-11T15:14:24.685 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.686 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.686 DEBUG:boto:Params: {} 2023-10-11T15:14:24.687 DEBUG:boto:Token: None 2023-10-11T15:14:24.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea04dacda1622a14-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.738 DEBUG:boto:b'dghqvd-58obj000000000000000000501000falseobj000000000000000000502023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.739 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.740 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.740 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.741 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.741 DEBUG:boto:Method: GET 2023-10-11T15:14:24.741 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.742 DEBUG:boto:Data: 2023-10-11T15:14:24.742 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.743 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.744 DEBUG:boto:Params: {} 2023-10-11T15:14:24.744 DEBUG:boto:Token: None 2023-10-11T15:14:24.745 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf83a4e659730379-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.752 DEBUG:boto:b'dghqvd-58obj000000000000000000501000falseobj000000000000000000502023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.752 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.753 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.753 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.754 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.754 DEBUG:boto:Data: 2023-10-11T15:14:24.755 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.756 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.756 DEBUG:boto:Params: {} 2023-10-11T15:14:24.756 DEBUG:boto:Token: None 2023-10-11T15:14:24.757 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDAgW1DAq5IcLhcRt4/szgXIY6U= 2023-10-11T15:14:24.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDAgW1DAq5IcLhcRt4/szgXIY6U=', 'Content-Length': '0'} 2023-10-11T15:14:24.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae75c68eb2b3111-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.761 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.761 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.762 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.762 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.762 DEBUG:boto:Data: 2023-10-11T15:14:24.763 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.764 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.764 DEBUG:boto:Params: {} 2023-10-11T15:14:24.765 DEBUG:boto:Token: None 2023-10-11T15:14:24.765 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.766 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDAgW1DAq5IcLhcRt4/szgXIY6U= 2023-10-11T15:14:24.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDAgW1DAq5IcLhcRt4/szgXIY6U=', 'Content-Length': '0'} 2023-10-11T15:14:24.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd979d0a0332e643-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-10-11T15:14:24.769 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.769 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.770 DEBUG:boto:Method: GET 2023-10-11T15:14:24.770 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.771 DEBUG:boto:Data: 2023-10-11T15:14:24.771 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.772 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.772 DEBUG:boto:Params: {} 2023-10-11T15:14:24.773 DEBUG:boto:Token: None 2023-10-11T15:14:24.773 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jDmiKohYae+jZvHsK3cmgdd6910= 2023-10-11T15:14:24.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jDmiKohYae+jZvHsK3cmgdd6910=', 'Content-Length': '0'} 2023-10-11T15:14:24.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38c1e36d9fa4a61-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.777 DEBUG:boto:path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.777 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.778 DEBUG:boto:Method: GET 2023-10-11T15:14:24.778 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.779 DEBUG:boto:Data: 2023-10-11T15:14:24.779 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.780 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.781 DEBUG:boto:Params: {} 2023-10-11T15:14:24.781 DEBUG:boto:Token: None 2023-10-11T15:14:24.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000050 2023-10-11T15:14:24.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jDmiKohYae+jZvHsK3cmgdd6910= 2023-10-11T15:14:24.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jDmiKohYae+jZvHsK3cmgdd6910=', 'Content-Length': '0'} 2023-10-11T15:14:24.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2994cc612d7c60-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.826 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.827 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.828 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.828 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.828 DEBUG:boto:Method: GET 2023-10-11T15:14:24.829 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.829 DEBUG:boto:Data: 2023-10-11T15:14:24.830 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.831 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.831 DEBUG:boto:Params: {} 2023-10-11T15:14:24.831 DEBUG:boto:Token: None 2023-10-11T15:14:24.832 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000147107f13fbd36ed-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.836 DEBUG:boto:b'dghqvd-58obj000000000000000000501000falseobj000000000000000000502023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.837 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.838 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.838 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.838 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.839 DEBUG:boto:Method: GET 2023-10-11T15:14:24.839 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000050 2023-10-11T15:14:24.840 DEBUG:boto:Data: 2023-10-11T15:14:24.840 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.841 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.841 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.841 DEBUG:boto:Params: {} 2023-10-11T15:14:24.842 DEBUG:boto:Token: None 2023-10-11T15:14:24.842 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007f7a240859c31f7-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.848 DEBUG:boto:b'dghqvd-58obj000000000000000000501000falseobj000000000000000000502023-10-11T15:14:03.837Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-11T15:14:24.849 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.850 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.851 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.851 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.853 DEBUG:boto:Method: GET 2023-10-11T15:14:24.853 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.854 DEBUG:boto:Data: 2023-10-11T15:14:24.854 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.855 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.855 DEBUG:boto:Params: {} 2023-10-11T15:14:24.856 DEBUG:boto:Token: None 2023-10-11T15:14:24.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI= 2023-10-11T15:14:24.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI=', 'Content-Length': '0'} 2023-10-11T15:14:24.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006722c7009be479ad-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.860 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.861 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.861 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.861 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.862 DEBUG:boto:Method: GET 2023-10-11T15:14:24.862 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.863 DEBUG:boto:Data: 2023-10-11T15:14:24.863 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.864 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.864 DEBUG:boto:Params: {} 2023-10-11T15:14:24.865 DEBUG:boto:Token: None 2023-10-11T15:14:24.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI= 2023-10-11T15:14:24.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI=', 'Content-Length': '0'} 2023-10-11T15:14:24.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a41c6f6ffd4a773-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.869 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.869 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.870 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.870 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.870 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.871 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.871 DEBUG:boto:Data: 2023-10-11T15:14:24.872 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.872 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.873 DEBUG:boto:Params: {} 2023-10-11T15:14:24.873 DEBUG:boto:Token: None 2023-10-11T15:14:24.874 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n+NB00xzhGHKBgMBDroCHq+9eMM= 2023-10-11T15:14:24.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n+NB00xzhGHKBgMBDroCHq+9eMM=', 'Content-Length': '0'} 2023-10-11T15:14:24.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075f5b634d5458b8f-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.877 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.877 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.878 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.878 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.879 DEBUG:boto:Method: HEAD 2023-10-11T15:14:24.879 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.879 DEBUG:boto:Data: 2023-10-11T15:14:24.880 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.880 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.881 DEBUG:boto:Params: {} 2023-10-11T15:14:24.881 DEBUG:boto:Token: None 2023-10-11T15:14:24.882 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n+NB00xzhGHKBgMBDroCHq+9eMM= 2023-10-11T15:14:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n+NB00xzhGHKBgMBDroCHq+9eMM=', 'Content-Length': '0'} 2023-10-11T15:14:24.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dac87c5bd5aa5db-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-11T15:14:24.886 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.886 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.886 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.887 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.887 DEBUG:boto:Method: GET 2023-10-11T15:14:24.888 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.888 DEBUG:boto:Data: 2023-10-11T15:14:24.889 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.889 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.890 DEBUG:boto:Params: {} 2023-10-11T15:14:24.890 DEBUG:boto:Token: None 2023-10-11T15:14:24.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI= 2023-10-11T15:14:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI=', 'Content-Length': '0'} 2023-10-11T15:14:24.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e461dbe617d3fee1-006526bbd0-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.938 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.939 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:24.939 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.940 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.940 DEBUG:boto:Method: GET 2023-10-11T15:14:24.941 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.941 DEBUG:boto:Data: 2023-10-11T15:14:24.942 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.942 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.943 DEBUG:boto:Params: {} 2023-10-11T15:14:24.943 DEBUG:boto:Token: None 2023-10-11T15:14:24.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/obj00000000000000000053?versionId=null 2023-10-11T15:14:24.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI= 2023-10-11T15:14:24.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PwEAiVlWAWV05xljI5fcWQ3AzGI=', 'Content-Length': '0'} 2023-10-11T15:14:24.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a580783095dbcc-006526bbd0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.947 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.948 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.948 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.948 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.949 DEBUG:boto:Method: GET 2023-10-11T15:14:24.949 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.950 DEBUG:boto:Data: 2023-10-11T15:14:24.950 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:24.951 DEBUG:boto:Port: 8001 2023-10-11T15:14:24.951 DEBUG:boto:Params: {} 2023-10-11T15:14:24.952 DEBUG:boto:Token: None 2023-10-11T15:14:24.952 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca18ec60ac2d152e-006526bbd0-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:24.956 DEBUG:boto:b'dghqvd-58obj000000000000000000531000falseobj000000000000000000532023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:24.957 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:24.957 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:24.958 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.958 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.958 DEBUG:boto:Method: GET 2023-10-11T15:14:24.959 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:24.959 DEBUG:boto:Data: 2023-10-11T15:14:24.960 DEBUG:boto:Headers: {} 2023-10-11T15:14:24.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:24.961 DEBUG:boto:Port: 8000 2023-10-11T15:14:24.961 DEBUG:boto:Params: {} 2023-10-11T15:14:24.961 DEBUG:boto:Token: None 2023-10-11T15:14:24.962 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:24 GMT /dghqvd-58/ 2023-10-11T15:14:24.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM= 2023-10-11T15:14:24.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X7v50kKk2qKf7ruRY0s1tAVsOEM=', 'Content-Length': '0'} 2023-10-11T15:14:24.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc55d41c9c5d99e1-006526bbd0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.010 DEBUG:boto:b'dghqvd-58obj000000000000000000531000falseobj000000000000000000532023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.011 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.012 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.012 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.013 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.013 DEBUG:boto:Data: 2023-10-11T15:14:25.013 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.014 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.015 DEBUG:boto:Params: {} 2023-10-11T15:14:25.015 DEBUG:boto:Token: None 2023-10-11T15:14:25.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:17UJvODfyWCeO+zKThpHLL8KQ5g= 2023-10-11T15:14:25.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:17UJvODfyWCeO+zKThpHLL8KQ5g=', 'Content-Length': '0'} 2023-10-11T15:14:25.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fd627c46a5999f-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.019 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.019 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.020 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.020 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.021 DEBUG:boto:Data: 2023-10-11T15:14:25.021 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.022 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.022 DEBUG:boto:Params: {} 2023-10-11T15:14:25.023 DEBUG:boto:Token: None 2023-10-11T15:14:25.023 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:17UJvODfyWCeO+zKThpHLL8KQ5g= 2023-10-11T15:14:25.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:17UJvODfyWCeO+zKThpHLL8KQ5g=', 'Content-Length': '0'} 2023-10-11T15:14:25.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca7643cf0b09cdc-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-10-11T15:14:25.027 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.027 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.028 DEBUG:boto:Method: GET 2023-10-11T15:14:25.028 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.029 DEBUG:boto:Data: 2023-10-11T15:14:25.029 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.030 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.030 DEBUG:boto:Params: {} 2023-10-11T15:14:25.031 DEBUG:boto:Token: None 2023-10-11T15:14:25.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M6TKOtlx5gws7vOEtdtRJCb4ni4= 2023-10-11T15:14:25.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M6TKOtlx5gws7vOEtdtRJCb4ni4=', 'Content-Length': '0'} 2023-10-11T15:14:25.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003458c15d8c9b7684-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.035 DEBUG:boto:path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.035 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.036 DEBUG:boto:Method: GET 2023-10-11T15:14:25.036 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.036 DEBUG:boto:Data: 2023-10-11T15:14:25.037 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.037 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.037 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.038 DEBUG:boto:Params: {} 2023-10-11T15:14:25.038 DEBUG:boto:Token: None 2023-10-11T15:14:25.039 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000053 2023-10-11T15:14:25.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M6TKOtlx5gws7vOEtdtRJCb4ni4= 2023-10-11T15:14:25.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M6TKOtlx5gws7vOEtdtRJCb4ni4=', 'Content-Length': '0'} 2023-10-11T15:14:25.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f8ece8fd31641d-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.042 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.043 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.043 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.044 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.044 DEBUG:boto:Method: GET 2023-10-11T15:14:25.044 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.045 DEBUG:boto:Data: 2023-10-11T15:14:25.045 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.046 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.046 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.046 DEBUG:boto:Params: {} 2023-10-11T15:14:25.047 DEBUG:boto:Token: None 2023-10-11T15:14:25.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd3d7257255c620-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.098 DEBUG:boto:b'dghqvd-58obj000000000000000000531000falseobj000000000000000000532023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.099 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.099 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.100 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.100 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.101 DEBUG:boto:Method: GET 2023-10-11T15:14:25.101 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000053 2023-10-11T15:14:25.101 DEBUG:boto:Data: 2023-10-11T15:14:25.102 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.102 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.103 DEBUG:boto:Params: {} 2023-10-11T15:14:25.103 DEBUG:boto:Token: None 2023-10-11T15:14:25.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b8e6693f5a91959-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.109 DEBUG:boto:b'dghqvd-58obj000000000000000000531000falseobj000000000000000000532023-10-11T15:14:05.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-11T15:14:25.110 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.111 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.111 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.112 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.112 DEBUG:boto:Method: GET 2023-10-11T15:14:25.112 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.113 DEBUG:boto:Data: 2023-10-11T15:14:25.113 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.113 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.114 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.114 DEBUG:boto:Params: {} 2023-10-11T15:14:25.115 DEBUG:boto:Token: None 2023-10-11T15:14:25.115 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.116 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA= 2023-10-11T15:14:25.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA=', 'Content-Length': '0'} 2023-10-11T15:14:25.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab93d4f83edd1ba-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.119 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.119 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.120 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.120 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.121 DEBUG:boto:Method: GET 2023-10-11T15:14:25.121 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.121 DEBUG:boto:Data: 2023-10-11T15:14:25.122 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.123 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.123 DEBUG:boto:Params: {} 2023-10-11T15:14:25.123 DEBUG:boto:Token: None 2023-10-11T15:14:25.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA= 2023-10-11T15:14:25.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA=', 'Content-Length': '0'} 2023-10-11T15:14:25.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f1afbe7ace0f5b-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.128 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.128 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.129 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.129 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.129 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.130 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.130 DEBUG:boto:Data: 2023-10-11T15:14:25.130 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.131 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.132 DEBUG:boto:Params: {} 2023-10-11T15:14:25.132 DEBUG:boto:Token: None 2023-10-11T15:14:25.133 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0+AAn7Yqxd5USfP95ZiKSxeXto= 2023-10-11T15:14:25.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0+AAn7Yqxd5USfP95ZiKSxeXto=', 'Content-Length': '0'} 2023-10-11T15:14:25.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0ef2eb96a3cc5c-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.136 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.136 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.137 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.137 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.138 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.138 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.138 DEBUG:boto:Data: 2023-10-11T15:14:25.139 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.139 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.140 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.140 DEBUG:boto:Params: {} 2023-10-11T15:14:25.140 DEBUG:boto:Token: None 2023-10-11T15:14:25.141 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.141 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U0+AAn7Yqxd5USfP95ZiKSxeXto= 2023-10-11T15:14:25.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U0+AAn7Yqxd5USfP95ZiKSxeXto=', 'Content-Length': '0'} 2023-10-11T15:14:25.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af58fa2b37bf127a-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-11T15:14:25.145 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.145 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.146 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.146 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.147 DEBUG:boto:Method: GET 2023-10-11T15:14:25.147 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.147 DEBUG:boto:Data: 2023-10-11T15:14:25.148 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.149 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.149 DEBUG:boto:Params: {} 2023-10-11T15:14:25.150 DEBUG:boto:Token: None 2023-10-11T15:14:25.150 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA= 2023-10-11T15:14:25.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA=', 'Content-Length': '0'} 2023-10-11T15:14:25.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8931e3a033eb07-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.154 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.155 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.155 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.156 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.156 DEBUG:boto:Method: GET 2023-10-11T15:14:25.157 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.157 DEBUG:boto:Data: 2023-10-11T15:14:25.157 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.159 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.160 DEBUG:boto:Params: {} 2023-10-11T15:14:25.160 DEBUG:boto:Token: None 2023-10-11T15:14:25.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056?versionId=null 2023-10-11T15:14:25.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA= 2023-10-11T15:14:25.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zbwLreAZDXRaVOQQswZEFStnxhA=', 'Content-Length': '0'} 2023-10-11T15:14:25.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da83246c45a5499-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.210 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.211 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.211 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.212 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.212 DEBUG:boto:Method: GET 2023-10-11T15:14:25.212 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.213 DEBUG:boto:Data: 2023-10-11T15:14:25.213 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.214 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.215 DEBUG:boto:Params: {} 2023-10-11T15:14:25.215 DEBUG:boto:Token: None 2023-10-11T15:14:25.216 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda84f0ef31f5da8-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.220 DEBUG:boto:b'dghqvd-58obj000000000000000000561000falseobj000000000000000000562023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.220 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.221 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.221 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.222 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.222 DEBUG:boto:Method: GET 2023-10-11T15:14:25.223 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.223 DEBUG:boto:Data: 2023-10-11T15:14:25.224 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.224 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.225 DEBUG:boto:Params: {} 2023-10-11T15:14:25.225 DEBUG:boto:Token: None 2023-10-11T15:14:25.226 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c4556044731f6c-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.231 DEBUG:boto:b'dghqvd-58obj000000000000000000561000falseobj000000000000000000562023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.232 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.233 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.233 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.233 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.234 DEBUG:boto:Data: 2023-10-11T15:14:25.234 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.235 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.235 DEBUG:boto:Params: {} 2023-10-11T15:14:25.236 DEBUG:boto:Token: None 2023-10-11T15:14:25.236 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yGUyzAZ8M7FgdAaxhK4BeZMJrn0= 2023-10-11T15:14:25.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yGUyzAZ8M7FgdAaxhK4BeZMJrn0=', 'Content-Length': '0'} 2023-10-11T15:14:25.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900c95db1454c19a-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.240 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.240 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.241 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.241 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.241 DEBUG:boto:Data: 2023-10-11T15:14:25.242 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.243 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.243 DEBUG:boto:Params: {} 2023-10-11T15:14:25.244 DEBUG:boto:Token: None 2023-10-11T15:14:25.244 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yGUyzAZ8M7FgdAaxhK4BeZMJrn0= 2023-10-11T15:14:25.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yGUyzAZ8M7FgdAaxhK4BeZMJrn0=', 'Content-Length': '0'} 2023-10-11T15:14:25.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da2b24c3244bd9a-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-10-11T15:14:25.248 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.248 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.249 DEBUG:boto:Method: GET 2023-10-11T15:14:25.249 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.249 DEBUG:boto:Data: 2023-10-11T15:14:25.250 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.251 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.251 DEBUG:boto:Params: {} 2023-10-11T15:14:25.252 DEBUG:boto:Token: None 2023-10-11T15:14:25.252 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oFwr2RkLy+6KQKf5xJnGuEqft7E= 2023-10-11T15:14:25.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oFwr2RkLy+6KQKf5xJnGuEqft7E=', 'Content-Length': '0'} 2023-10-11T15:14:25.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7fbbcdc6098009-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.298 DEBUG:boto:path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.299 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.300 DEBUG:boto:Method: GET 2023-10-11T15:14:25.300 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.301 DEBUG:boto:Data: 2023-10-11T15:14:25.301 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.302 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.302 DEBUG:boto:Params: {} 2023-10-11T15:14:25.303 DEBUG:boto:Token: None 2023-10-11T15:14:25.303 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000056 2023-10-11T15:14:25.304 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oFwr2RkLy+6KQKf5xJnGuEqft7E= 2023-10-11T15:14:25.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oFwr2RkLy+6KQKf5xJnGuEqft7E=', 'Content-Length': '0'} 2023-10-11T15:14:25.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477fd8e996f57fee-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.307 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.308 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.308 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.308 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.309 DEBUG:boto:Method: GET 2023-10-11T15:14:25.310 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.310 DEBUG:boto:Data: 2023-10-11T15:14:25.310 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.311 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.312 DEBUG:boto:Params: {} 2023-10-11T15:14:25.312 DEBUG:boto:Token: None 2023-10-11T15:14:25.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e490eeae3c25ab5-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.317 DEBUG:boto:b'dghqvd-58obj000000000000000000561000falseobj000000000000000000562023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.318 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.318 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.319 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.319 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.320 DEBUG:boto:Method: GET 2023-10-11T15:14:25.320 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000056 2023-10-11T15:14:25.321 DEBUG:boto:Data: 2023-10-11T15:14:25.321 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.321 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.322 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.322 DEBUG:boto:Params: {} 2023-10-11T15:14:25.323 DEBUG:boto:Token: None 2023-10-11T15:14:25.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000124c7283e8b2fa93-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.374 DEBUG:boto:b'dghqvd-58obj000000000000000000561000falseobj000000000000000000562023-10-11T15:14:03.924Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-11T15:14:25.376 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.376 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.376 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.377 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.377 DEBUG:boto:Method: GET 2023-10-11T15:14:25.378 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.378 DEBUG:boto:Data: 2023-10-11T15:14:25.379 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.379 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.380 DEBUG:boto:Params: {} 2023-10-11T15:14:25.380 DEBUG:boto:Token: None 2023-10-11T15:14:25.381 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM= 2023-10-11T15:14:25.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM=', 'Content-Length': '0'} 2023-10-11T15:14:25.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a14602a773d8d5-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.384 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.385 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.385 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.386 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.386 DEBUG:boto:Method: GET 2023-10-11T15:14:25.386 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.387 DEBUG:boto:Data: 2023-10-11T15:14:25.387 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.388 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.389 DEBUG:boto:Params: {} 2023-10-11T15:14:25.389 DEBUG:boto:Token: None 2023-10-11T15:14:25.390 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.390 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM= 2023-10-11T15:14:25.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM=', 'Content-Length': '0'} 2023-10-11T15:14:25.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bef0e7a87a6f2c-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.393 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.394 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.394 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.394 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.395 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.395 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.396 DEBUG:boto:Data: 2023-10-11T15:14:25.396 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.397 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.397 DEBUG:boto:Params: {} 2023-10-11T15:14:25.397 DEBUG:boto:Token: None 2023-10-11T15:14:25.398 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZK97bR5PNuRMyASzEIR6Fc4sa8= 2023-10-11T15:14:25.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZK97bR5PNuRMyASzEIR6Fc4sa8=', 'Content-Length': '0'} 2023-10-11T15:14:25.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f5ce67bfaea661-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.401 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.403 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.403 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.404 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.404 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.405 DEBUG:boto:Data: 2023-10-11T15:14:25.405 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.405 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.406 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.406 DEBUG:boto:Params: {} 2023-10-11T15:14:25.407 DEBUG:boto:Token: None 2023-10-11T15:14:25.407 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.407 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZK97bR5PNuRMyASzEIR6Fc4sa8= 2023-10-11T15:14:25.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZK97bR5PNuRMyASzEIR6Fc4sa8=', 'Content-Length': '0'} 2023-10-11T15:14:25.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004650e552b8bcd011-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-11T15:14:25.411 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.411 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.412 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.412 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.412 DEBUG:boto:Method: GET 2023-10-11T15:14:25.413 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.413 DEBUG:boto:Data: 2023-10-11T15:14:25.414 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.415 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.415 DEBUG:boto:Params: {} 2023-10-11T15:14:25.415 DEBUG:boto:Token: None 2023-10-11T15:14:25.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM= 2023-10-11T15:14:25.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM=', 'Content-Length': '0'} 2023-10-11T15:14:25.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad96b1b04a823fe-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.419 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.420 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.420 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.421 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.421 DEBUG:boto:Method: GET 2023-10-11T15:14:25.421 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.422 DEBUG:boto:Data: 2023-10-11T15:14:25.422 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.423 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.424 DEBUG:boto:Params: {} 2023-10-11T15:14:25.424 DEBUG:boto:Token: None 2023-10-11T15:14:25.424 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057?versionId=null 2023-10-11T15:14:25.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM= 2023-10-11T15:14:25.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPIJLzT/EPCuupbb3BS7whGPZbM=', 'Content-Length': '0'} 2023-10-11T15:14:25.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db8abcbaf4b1b0bc-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.428 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.428 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.429 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.429 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.430 DEBUG:boto:Method: GET 2023-10-11T15:14:25.430 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.431 DEBUG:boto:Data: 2023-10-11T15:14:25.431 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.431 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.432 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.432 DEBUG:boto:Params: {} 2023-10-11T15:14:25.433 DEBUG:boto:Token: None 2023-10-11T15:14:25.433 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042964820a65ac5ca-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.437 DEBUG:boto:b'dghqvd-58obj000000000000000000571000falseobj000000000000000000572023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.438 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.438 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.439 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.439 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.439 DEBUG:boto:Method: GET 2023-10-11T15:14:25.440 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.440 DEBUG:boto:Data: 2023-10-11T15:14:25.441 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.441 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.441 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.442 DEBUG:boto:Params: {} 2023-10-11T15:14:25.442 DEBUG:boto:Token: None 2023-10-11T15:14:25.443 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005147a2dfa70ecd12-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.448 DEBUG:boto:b'dghqvd-58obj000000000000000000571000falseobj000000000000000000572023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.449 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.449 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.450 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.450 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.450 DEBUG:boto:Data: 2023-10-11T15:14:25.451 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.452 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.452 DEBUG:boto:Params: {} 2023-10-11T15:14:25.452 DEBUG:boto:Token: None 2023-10-11T15:14:25.453 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZDAn3z4lTjCdbF6Jsd5icv2JjhQ= 2023-10-11T15:14:25.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZDAn3z4lTjCdbF6Jsd5icv2JjhQ=', 'Content-Length': '0'} 2023-10-11T15:14:25.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4be8eab2246136-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.456 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.457 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.457 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.458 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.458 DEBUG:boto:Data: 2023-10-11T15:14:25.458 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.459 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.460 DEBUG:boto:Params: {} 2023-10-11T15:14:25.460 DEBUG:boto:Token: None 2023-10-11T15:14:25.460 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZDAn3z4lTjCdbF6Jsd5icv2JjhQ= 2023-10-11T15:14:25.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZDAn3z4lTjCdbF6Jsd5icv2JjhQ=', 'Content-Length': '0'} 2023-10-11T15:14:25.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047fc47fa67c34e58-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-10-11T15:14:25.464 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.465 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.465 DEBUG:boto:Method: GET 2023-10-11T15:14:25.465 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.466 DEBUG:boto:Data: 2023-10-11T15:14:25.466 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.467 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.467 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.468 DEBUG:boto:Params: {} 2023-10-11T15:14:25.468 DEBUG:boto:Token: None 2023-10-11T15:14:25.469 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.469 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2RsX8Ag42RrgCXWqQ7jlMC92PA= 2023-10-11T15:14:25.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2RsX8Ag42RrgCXWqQ7jlMC92PA=', 'Content-Length': '0'} 2023-10-11T15:14:25.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021877efd0726d1d2-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.472 DEBUG:boto:path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.472 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.473 DEBUG:boto:Method: GET 2023-10-11T15:14:25.473 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.474 DEBUG:boto:Data: 2023-10-11T15:14:25.474 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.475 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.475 DEBUG:boto:Params: {} 2023-10-11T15:14:25.476 DEBUG:boto:Token: None 2023-10-11T15:14:25.476 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000057 2023-10-11T15:14:25.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2RsX8Ag42RrgCXWqQ7jlMC92PA= 2023-10-11T15:14:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2RsX8Ag42RrgCXWqQ7jlMC92PA=', 'Content-Length': '0'} 2023-10-11T15:14:25.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31d67427259f7b3-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.526 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.527 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.528 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.528 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.528 DEBUG:boto:Method: GET 2023-10-11T15:14:25.529 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.529 DEBUG:boto:Data: 2023-10-11T15:14:25.530 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.531 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.531 DEBUG:boto:Params: {} 2023-10-11T15:14:25.531 DEBUG:boto:Token: None 2023-10-11T15:14:25.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a1cd56e4dc47aa-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.536 DEBUG:boto:b'dghqvd-58obj000000000000000000571000falseobj000000000000000000572023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.537 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.537 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.538 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.538 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.539 DEBUG:boto:Method: GET 2023-10-11T15:14:25.539 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000057 2023-10-11T15:14:25.540 DEBUG:boto:Data: 2023-10-11T15:14:25.540 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.540 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.541 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.541 DEBUG:boto:Params: {} 2023-10-11T15:14:25.542 DEBUG:boto:Token: None 2023-10-11T15:14:25.542 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e148589524419349-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.548 DEBUG:boto:b'dghqvd-58obj000000000000000000571000falseobj000000000000000000572023-10-11T15:14:05.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-11T15:14:25.550 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.550 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.550 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.551 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.551 DEBUG:boto:Method: GET 2023-10-11T15:14:25.552 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.552 DEBUG:boto:Data: 2023-10-11T15:14:25.553 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.553 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.554 DEBUG:boto:Params: {} 2023-10-11T15:14:25.554 DEBUG:boto:Token: None 2023-10-11T15:14:25.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8= 2023-10-11T15:14:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8=', 'Content-Length': '0'} 2023-10-11T15:14:25.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13ab72426b30ef9-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.598 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.599 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.600 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.600 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.601 DEBUG:boto:Method: GET 2023-10-11T15:14:25.601 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.601 DEBUG:boto:Data: 2023-10-11T15:14:25.602 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.603 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.603 DEBUG:boto:Params: {} 2023-10-11T15:14:25.604 DEBUG:boto:Token: None 2023-10-11T15:14:25.604 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8= 2023-10-11T15:14:25.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8=', 'Content-Length': '0'} 2023-10-11T15:14:25.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003b85c736a99d7b4-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.609 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.609 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.609 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.610 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.610 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.611 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.611 DEBUG:boto:Data: 2023-10-11T15:14:25.612 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.613 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.613 DEBUG:boto:Params: {} 2023-10-11T15:14:25.614 DEBUG:boto:Token: None 2023-10-11T15:14:25.614 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LuXNTFQGk9u+Zj/B2uT2QSWC74M= 2023-10-11T15:14:25.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LuXNTFQGk9u+Zj/B2uT2QSWC74M=', 'Content-Length': '0'} 2023-10-11T15:14:25.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa355d6ba264470-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.618 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.618 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.619 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.619 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.620 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.620 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.620 DEBUG:boto:Data: 2023-10-11T15:14:25.621 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.622 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.622 DEBUG:boto:Params: {} 2023-10-11T15:14:25.623 DEBUG:boto:Token: None 2023-10-11T15:14:25.623 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LuXNTFQGk9u+Zj/B2uT2QSWC74M= 2023-10-11T15:14:25.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LuXNTFQGk9u+Zj/B2uT2QSWC74M=', 'Content-Length': '0'} 2023-10-11T15:14:25.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5c1110a35a6bc5-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-11T15:14:25.628 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.628 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.629 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.629 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.629 DEBUG:boto:Method: GET 2023-10-11T15:14:25.630 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.630 DEBUG:boto:Data: 2023-10-11T15:14:25.631 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.631 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.632 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.632 DEBUG:boto:Params: {} 2023-10-11T15:14:25.633 DEBUG:boto:Token: None 2023-10-11T15:14:25.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8= 2023-10-11T15:14:25.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8=', 'Content-Length': '0'} 2023-10-11T15:14:25.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c452ecbb2f3609f-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.637 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.638 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.638 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.639 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.639 DEBUG:boto:Method: GET 2023-10-11T15:14:25.640 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.640 DEBUG:boto:Data: 2023-10-11T15:14:25.641 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.641 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.642 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.642 DEBUG:boto:Params: {} 2023-10-11T15:14:25.643 DEBUG:boto:Token: None 2023-10-11T15:14:25.643 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058?versionId=null 2023-10-11T15:14:25.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8= 2023-10-11T15:14:25.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:syohmhuH0940vsprmzeq6bH68j8=', 'Content-Length': '0'} 2023-10-11T15:14:25.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442953db59dc9ad0-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.647 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.647 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.648 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.648 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.649 DEBUG:boto:Method: GET 2023-10-11T15:14:25.649 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.649 DEBUG:boto:Data: 2023-10-11T15:14:25.650 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.651 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.651 DEBUG:boto:Params: {} 2023-10-11T15:14:25.651 DEBUG:boto:Token: None 2023-10-11T15:14:25.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef94ce2266a9ffe-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.656 DEBUG:boto:b'dghqvd-58obj000000000000000000581000falseobj000000000000000000582023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.656 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.657 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.657 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.658 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.658 DEBUG:boto:Method: GET 2023-10-11T15:14:25.659 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.659 DEBUG:boto:Data: 2023-10-11T15:14:25.659 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.660 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.661 DEBUG:boto:Params: {} 2023-10-11T15:14:25.661 DEBUG:boto:Token: None 2023-10-11T15:14:25.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f8039c518a1da16-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.710 DEBUG:boto:b'dghqvd-58obj000000000000000000581000falseobj000000000000000000582023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.711 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.711 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.712 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.712 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.713 DEBUG:boto:Data: 2023-10-11T15:14:25.713 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.714 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.714 DEBUG:boto:Params: {} 2023-10-11T15:14:25.715 DEBUG:boto:Token: None 2023-10-11T15:14:25.715 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BUgs4lxp9P8rsa0+nB5xY4KRHS4= 2023-10-11T15:14:25.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BUgs4lxp9P8rsa0+nB5xY4KRHS4=', 'Content-Length': '0'} 2023-10-11T15:14:25.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3774ef52e0c889-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.719 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.719 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.720 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.720 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.721 DEBUG:boto:Data: 2023-10-11T15:14:25.721 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.722 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.722 DEBUG:boto:Params: {} 2023-10-11T15:14:25.723 DEBUG:boto:Token: None 2023-10-11T15:14:25.723 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BUgs4lxp9P8rsa0+nB5xY4KRHS4= 2023-10-11T15:14:25.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BUgs4lxp9P8rsa0+nB5xY4KRHS4=', 'Content-Length': '0'} 2023-10-11T15:14:25.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031928537ce86e055-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-10-11T15:14:25.727 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.728 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.728 DEBUG:boto:Method: GET 2023-10-11T15:14:25.728 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.729 DEBUG:boto:Data: 2023-10-11T15:14:25.730 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.730 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.731 DEBUG:boto:Params: {} 2023-10-11T15:14:25.731 DEBUG:boto:Token: None 2023-10-11T15:14:25.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GwmjnZYj/IC2e9bHd8ZREKHwMXc= 2023-10-11T15:14:25.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GwmjnZYj/IC2e9bHd8ZREKHwMXc=', 'Content-Length': '0'} 2023-10-11T15:14:25.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692eb7334f41c1df-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.736 DEBUG:boto:path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.736 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.736 DEBUG:boto:Method: GET 2023-10-11T15:14:25.737 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.737 DEBUG:boto:Data: 2023-10-11T15:14:25.738 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.739 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.739 DEBUG:boto:Params: {} 2023-10-11T15:14:25.740 DEBUG:boto:Token: None 2023-10-11T15:14:25.740 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000058 2023-10-11T15:14:25.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GwmjnZYj/IC2e9bHd8ZREKHwMXc= 2023-10-11T15:14:25.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GwmjnZYj/IC2e9bHd8ZREKHwMXc=', 'Content-Length': '0'} 2023-10-11T15:14:25.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43c5839fa9afb6c-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.744 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.744 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.744 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.745 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.745 DEBUG:boto:Method: GET 2023-10-11T15:14:25.746 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.746 DEBUG:boto:Data: 2023-10-11T15:14:25.747 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.747 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.747 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.748 DEBUG:boto:Params: {} 2023-10-11T15:14:25.748 DEBUG:boto:Token: None 2023-10-11T15:14:25.749 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f84b20ba3780ddc-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.798 DEBUG:boto:b'dghqvd-58obj000000000000000000581000falseobj000000000000000000582023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.799 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.800 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.800 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.801 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.801 DEBUG:boto:Method: GET 2023-10-11T15:14:25.801 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000058 2023-10-11T15:14:25.802 DEBUG:boto:Data: 2023-10-11T15:14:25.802 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.803 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.804 DEBUG:boto:Params: {} 2023-10-11T15:14:25.804 DEBUG:boto:Token: None 2023-10-11T15:14:25.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.805 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa49f9f5344eb455-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.810 DEBUG:boto:b'dghqvd-58obj000000000000000000581000falseobj000000000000000000582023-10-11T15:14:01.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-11T15:14:25.812 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.812 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.812 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.813 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.813 DEBUG:boto:Method: GET 2023-10-11T15:14:25.814 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.814 DEBUG:boto:Data: 2023-10-11T15:14:25.815 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.815 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.816 DEBUG:boto:Params: {} 2023-10-11T15:14:25.816 DEBUG:boto:Token: None 2023-10-11T15:14:25.817 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY= 2023-10-11T15:14:25.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY=', 'Content-Length': '0'} 2023-10-11T15:14:25.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225d901ccfcae448-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.821 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.821 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.822 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.822 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.822 DEBUG:boto:Method: GET 2023-10-11T15:14:25.823 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.823 DEBUG:boto:Data: 2023-10-11T15:14:25.824 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.824 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.825 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.825 DEBUG:boto:Params: {} 2023-10-11T15:14:25.825 DEBUG:boto:Token: None 2023-10-11T15:14:25.826 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY= 2023-10-11T15:14:25.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY=', 'Content-Length': '0'} 2023-10-11T15:14:25.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db14d2befbe0196b-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.830 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.830 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.831 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.831 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.832 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.832 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.832 DEBUG:boto:Data: 2023-10-11T15:14:25.833 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.834 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.834 DEBUG:boto:Params: {} 2023-10-11T15:14:25.835 DEBUG:boto:Token: None 2023-10-11T15:14:25.835 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dp7WDkRXgv8stBXhDaKXVvEJvxs= 2023-10-11T15:14:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dp7WDkRXgv8stBXhDaKXVvEJvxs=', 'Content-Length': '0'} 2023-10-11T15:14:25.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20c82271e3afc37-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.838 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.839 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.839 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.840 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.840 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.841 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.841 DEBUG:boto:Data: 2023-10-11T15:14:25.841 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.842 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.843 DEBUG:boto:Params: {} 2023-10-11T15:14:25.843 DEBUG:boto:Token: None 2023-10-11T15:14:25.844 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dp7WDkRXgv8stBXhDaKXVvEJvxs= 2023-10-11T15:14:25.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dp7WDkRXgv8stBXhDaKXVvEJvxs=', 'Content-Length': '0'} 2023-10-11T15:14:25.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a9980475d3fd89-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-11T15:14:25.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.848 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.848 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.849 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.849 DEBUG:boto:Method: GET 2023-10-11T15:14:25.850 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.850 DEBUG:boto:Data: 2023-10-11T15:14:25.851 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.851 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.852 DEBUG:boto:Params: {} 2023-10-11T15:14:25.852 DEBUG:boto:Token: None 2023-10-11T15:14:25.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY= 2023-10-11T15:14:25.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY=', 'Content-Length': '0'} 2023-10-11T15:14:25.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081ac1f110fa8fe77-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.857 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.857 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.857 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.858 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.858 DEBUG:boto:Method: GET 2023-10-11T15:14:25.859 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.859 DEBUG:boto:Data: 2023-10-11T15:14:25.860 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.860 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.861 DEBUG:boto:Params: {} 2023-10-11T15:14:25.861 DEBUG:boto:Token: None 2023-10-11T15:14:25.862 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059?versionId=null 2023-10-11T15:14:25.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY= 2023-10-11T15:14:25.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z536w79NwycCEUkneC1xQVaghHY=', 'Content-Length': '0'} 2023-10-11T15:14:25.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099ddadc6ea0379e4-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.866 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.866 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.867 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.867 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.868 DEBUG:boto:Method: GET 2023-10-11T15:14:25.868 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.868 DEBUG:boto:Data: 2023-10-11T15:14:25.869 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.870 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.870 DEBUG:boto:Params: {} 2023-10-11T15:14:25.871 DEBUG:boto:Token: None 2023-10-11T15:14:25.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.872 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000000336b9a5f2139-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.875 DEBUG:boto:b'dghqvd-58obj000000000000000000591000falseobj000000000000000000592023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.876 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.876 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.877 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.877 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.878 DEBUG:boto:Method: GET 2023-10-11T15:14:25.878 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.879 DEBUG:boto:Data: 2023-10-11T15:14:25.879 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.879 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.880 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.880 DEBUG:boto:Params: {} 2023-10-11T15:14:25.881 DEBUG:boto:Token: None 2023-10-11T15:14:25.881 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d614905467d06201-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.887 DEBUG:boto:b'dghqvd-58obj000000000000000000591000falseobj000000000000000000592023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.887 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.888 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.888 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.889 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.889 DEBUG:boto:Data: 2023-10-11T15:14:25.889 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.890 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.891 DEBUG:boto:Params: {} 2023-10-11T15:14:25.891 DEBUG:boto:Token: None 2023-10-11T15:14:25.892 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjeeRyTDXcIxs1ru9WX3U39uc6U= 2023-10-11T15:14:25.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjeeRyTDXcIxs1ru9WX3U39uc6U=', 'Content-Length': '0'} 2023-10-11T15:14:25.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0a6f0e45d8a0679-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.895 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.895 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.896 DEBUG:boto:Method: HEAD 2023-10-11T15:14:25.896 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.897 DEBUG:boto:Data: 2023-10-11T15:14:25.897 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.898 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.898 DEBUG:boto:Params: {} 2023-10-11T15:14:25.899 DEBUG:boto:Token: None 2023-10-11T15:14:25.899 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjeeRyTDXcIxs1ru9WX3U39uc6U= 2023-10-11T15:14:25.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjeeRyTDXcIxs1ru9WX3U39uc6U=', 'Content-Length': '0'} 2023-10-11T15:14:25.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40b81c1eebd948a-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-10-11T15:14:25.903 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.904 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.904 DEBUG:boto:Method: GET 2023-10-11T15:14:25.905 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.905 DEBUG:boto:Data: 2023-10-11T15:14:25.905 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.906 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.907 DEBUG:boto:Params: {} 2023-10-11T15:14:25.907 DEBUG:boto:Token: None 2023-10-11T15:14:25.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DVoXP3ykCo6DQZGx7h0LOUQmB64= 2023-10-11T15:14:25.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVoXP3ykCo6DQZGx7h0LOUQmB64=', 'Content-Length': '0'} 2023-10-11T15:14:25.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a0395215a7e89e-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.954 DEBUG:boto:path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.955 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.955 DEBUG:boto:Method: GET 2023-10-11T15:14:25.956 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.956 DEBUG:boto:Data: 2023-10-11T15:14:25.957 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.957 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.958 DEBUG:boto:Params: {} 2023-10-11T15:14:25.958 DEBUG:boto:Token: None 2023-10-11T15:14:25.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000059 2023-10-11T15:14:25.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DVoXP3ykCo6DQZGx7h0LOUQmB64= 2023-10-11T15:14:25.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVoXP3ykCo6DQZGx7h0LOUQmB64=', 'Content-Length': '0'} 2023-10-11T15:14:25.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7079997cbd10aa-006526bbd1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.963 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.963 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.964 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.964 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.965 DEBUG:boto:Method: GET 2023-10-11T15:14:25.965 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.966 DEBUG:boto:Data: 2023-10-11T15:14:25.966 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.967 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.967 DEBUG:boto:Params: {} 2023-10-11T15:14:25.968 DEBUG:boto:Token: None 2023-10-11T15:14:25.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646d95829b30ae7a-006526bbd1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.972 DEBUG:boto:b'dghqvd-58obj000000000000000000591000falseobj000000000000000000592023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.973 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:25.974 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:25.974 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.975 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.975 DEBUG:boto:Method: GET 2023-10-11T15:14:25.975 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000059 2023-10-11T15:14:25.976 DEBUG:boto:Data: 2023-10-11T15:14:25.976 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.977 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.977 DEBUG:boto:Params: {} 2023-10-11T15:14:25.978 DEBUG:boto:Token: None 2023-10-11T15:14:25.978 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/ 2023-10-11T15:14:25.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY= 2023-10-11T15:14:25.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h/ie2Lgt79WWxydpkpoQIDsyPAY=', 'Content-Length': '0'} 2023-10-11T15:14:25.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f1fa90192a2f018-006526bbd1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.984 DEBUG:boto:b'dghqvd-58obj000000000000000000591000falseobj000000000000000000592023-10-11T15:14:01.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:25.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-11T15:14:25.985 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:25.985 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:25.986 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.986 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.987 DEBUG:boto:Method: GET 2023-10-11T15:14:25.987 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.987 DEBUG:boto:Data: 2023-10-11T15:14:25.988 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:25.989 DEBUG:boto:Port: 8001 2023-10-11T15:14:25.989 DEBUG:boto:Params: {} 2023-10-11T15:14:25.989 DEBUG:boto:Token: None 2023-10-11T15:14:25.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R3xWzb5BLm46QlS/+0o9DKBVmOM= 2023-10-11T15:14:25.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R3xWzb5BLm46QlS/+0o9DKBVmOM=', 'Content-Length': '0'} 2023-10-11T15:14:25.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f8b1ba68d5e3e58-006526bbd1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:25 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:25.994 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:25.994 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:25.995 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.995 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.996 DEBUG:boto:Method: GET 2023-10-11T15:14:25.996 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:25.997 DEBUG:boto:Data: 2023-10-11T15:14:25.997 DEBUG:boto:Headers: {} 2023-10-11T15:14:25.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:25.998 DEBUG:boto:Port: 8000 2023-10-11T15:14:25.998 DEBUG:boto:Params: {} 2023-10-11T15:14:25.999 DEBUG:boto:Token: None 2023-10-11T15:14:25.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:25 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R3xWzb5BLm46QlS/+0o9DKBVmOM= 2023-10-11T15:14:26.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:25 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R3xWzb5BLm46QlS/+0o9DKBVmOM=', 'Content-Length': '0'} 2023-10-11T15:14:26.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf22df8378ae467-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.003 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.004 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.004 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.004 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.005 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.005 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.006 DEBUG:boto:Data: 2023-10-11T15:14:26.006 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.007 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.007 DEBUG:boto:Params: {} 2023-10-11T15:14:26.008 DEBUG:boto:Token: None 2023-10-11T15:14:26.008 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T8HloInXy53JLPXjepBhUZpSDeY= 2023-10-11T15:14:26.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T8HloInXy53JLPXjepBhUZpSDeY=', 'Content-Length': '0'} 2023-10-11T15:14:26.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda22457097abb9f-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.011 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.012 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.012 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.013 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.013 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.013 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.014 DEBUG:boto:Data: 2023-10-11T15:14:26.014 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.015 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.015 DEBUG:boto:Params: {} 2023-10-11T15:14:26.016 DEBUG:boto:Token: None 2023-10-11T15:14:26.016 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T8HloInXy53JLPXjepBhUZpSDeY= 2023-10-11T15:14:26.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T8HloInXy53JLPXjepBhUZpSDeY=', 'Content-Length': '0'} 2023-10-11T15:14:26.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef26bc3af388617f-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-11T15:14:26.020 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.021 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.021 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.022 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.022 DEBUG:boto:Method: GET 2023-10-11T15:14:26.023 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.023 DEBUG:boto:Data: 2023-10-11T15:14:26.023 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.024 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.024 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.024 DEBUG:boto:Params: {} 2023-10-11T15:14:26.025 DEBUG:boto:Token: None 2023-10-11T15:14:26.025 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IqEnhPb/dNuGa6wQmkw0CdsV00s= 2023-10-11T15:14:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IqEnhPb/dNuGa6wQmkw0CdsV00s=', 'Content-Length': '0'} 2023-10-11T15:14:26.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9d9096908c3266-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.070 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.071 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.071 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.072 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.072 DEBUG:boto:Method: GET 2023-10-11T15:14:26.072 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.073 DEBUG:boto:Data: 2023-10-11T15:14:26.073 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.074 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.075 DEBUG:boto:Params: {} 2023-10-11T15:14:26.075 DEBUG:boto:Token: None 2023-10-11T15:14:26.076 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060?versionId=null 2023-10-11T15:14:26.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IqEnhPb/dNuGa6wQmkw0CdsV00s= 2023-10-11T15:14:26.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IqEnhPb/dNuGa6wQmkw0CdsV00s=', 'Content-Length': '0'} 2023-10-11T15:14:26.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d31c7ac09916d5f-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.080 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.080 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.081 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.081 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.081 DEBUG:boto:Method: GET 2023-10-11T15:14:26.082 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.082 DEBUG:boto:Data: 2023-10-11T15:14:26.083 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.084 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.084 DEBUG:boto:Params: {} 2023-10-11T15:14:26.085 DEBUG:boto:Token: None 2023-10-11T15:14:26.085 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.086 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d9ea60ffee1b85-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.089 DEBUG:boto:b'dghqvd-58obj000000000000000000601000falseobj000000000000000000602023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.090 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.091 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.091 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.091 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.092 DEBUG:boto:Method: GET 2023-10-11T15:14:26.092 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.093 DEBUG:boto:Data: 2023-10-11T15:14:26.093 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.094 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.094 DEBUG:boto:Params: {} 2023-10-11T15:14:26.095 DEBUG:boto:Token: None 2023-10-11T15:14:26.095 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd7ecb7548c71217-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.101 DEBUG:boto:b'dghqvd-58obj000000000000000000601000falseobj000000000000000000602023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.101 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.102 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.102 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.103 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.103 DEBUG:boto:Data: 2023-10-11T15:14:26.103 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.104 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.105 DEBUG:boto:Params: {} 2023-10-11T15:14:26.105 DEBUG:boto:Token: None 2023-10-11T15:14:26.106 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u5ddKNuU74rkzZaasyinGyho0ck= 2023-10-11T15:14:26.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u5ddKNuU74rkzZaasyinGyho0ck=', 'Content-Length': '0'} 2023-10-11T15:14:26.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c5f97743e35096-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.109 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.110 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.110 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.110 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.111 DEBUG:boto:Data: 2023-10-11T15:14:26.111 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.112 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.113 DEBUG:boto:Params: {} 2023-10-11T15:14:26.113 DEBUG:boto:Token: None 2023-10-11T15:14:26.113 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u5ddKNuU74rkzZaasyinGyho0ck= 2023-10-11T15:14:26.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u5ddKNuU74rkzZaasyinGyho0ck=', 'Content-Length': '0'} 2023-10-11T15:14:26.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abae320543f5c1bb-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-10-11T15:14:26.119 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.119 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.120 DEBUG:boto:Method: GET 2023-10-11T15:14:26.120 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.121 DEBUG:boto:Data: 2023-10-11T15:14:26.121 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.122 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.122 DEBUG:boto:Params: {} 2023-10-11T15:14:26.123 DEBUG:boto:Token: None 2023-10-11T15:14:26.123 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YN0yy2r9t/mVLnDUjuSisIyhQdM= 2023-10-11T15:14:26.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YN0yy2r9t/mVLnDUjuSisIyhQdM=', 'Content-Length': '0'} 2023-10-11T15:14:26.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008007b29df2a288e4-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.127 DEBUG:boto:path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.127 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.128 DEBUG:boto:Method: GET 2023-10-11T15:14:26.128 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.129 DEBUG:boto:Data: 2023-10-11T15:14:26.129 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.130 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.130 DEBUG:boto:Params: {} 2023-10-11T15:14:26.131 DEBUG:boto:Token: None 2023-10-11T15:14:26.131 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000060 2023-10-11T15:14:26.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YN0yy2r9t/mVLnDUjuSisIyhQdM= 2023-10-11T15:14:26.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YN0yy2r9t/mVLnDUjuSisIyhQdM=', 'Content-Length': '0'} 2023-10-11T15:14:26.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba1feb8e22fbe40-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.135 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.135 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.136 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.136 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.136 DEBUG:boto:Method: GET 2023-10-11T15:14:26.137 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.137 DEBUG:boto:Data: 2023-10-11T15:14:26.138 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.139 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.139 DEBUG:boto:Params: {} 2023-10-11T15:14:26.139 DEBUG:boto:Token: None 2023-10-11T15:14:26.140 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f702282184e2ea-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.144 DEBUG:boto:b'dghqvd-58obj000000000000000000601000falseobj000000000000000000602023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.145 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.145 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.146 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.146 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.147 DEBUG:boto:Method: GET 2023-10-11T15:14:26.147 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000060 2023-10-11T15:14:26.148 DEBUG:boto:Data: 2023-10-11T15:14:26.148 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.149 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.149 DEBUG:boto:Params: {} 2023-10-11T15:14:26.150 DEBUG:boto:Token: None 2023-10-11T15:14:26.150 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759ae4db41e43ad3-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.198 DEBUG:boto:b'dghqvd-58obj000000000000000000601000falseobj000000000000000000602023-10-11T15:14:05.113Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-11T15:14:26.199 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.200 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.200 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.200 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.201 DEBUG:boto:Method: GET 2023-10-11T15:14:26.201 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.202 DEBUG:boto:Data: 2023-10-11T15:14:26.202 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.203 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.203 DEBUG:boto:Params: {} 2023-10-11T15:14:26.204 DEBUG:boto:Token: None 2023-10-11T15:14:26.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw= 2023-10-11T15:14:26.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw=', 'Content-Length': '0'} 2023-10-11T15:14:26.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ecbb3e9a3f52e11-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.208 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.209 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.209 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.209 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.210 DEBUG:boto:Method: GET 2023-10-11T15:14:26.210 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.211 DEBUG:boto:Data: 2023-10-11T15:14:26.211 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.212 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.212 DEBUG:boto:Params: {} 2023-10-11T15:14:26.213 DEBUG:boto:Token: None 2023-10-11T15:14:26.213 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw= 2023-10-11T15:14:26.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw=', 'Content-Length': '0'} 2023-10-11T15:14:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0a040f98f9be1d-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.216 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.217 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.217 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.218 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.218 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.219 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.219 DEBUG:boto:Data: 2023-10-11T15:14:26.219 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.220 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.220 DEBUG:boto:Params: {} 2023-10-11T15:14:26.221 DEBUG:boto:Token: None 2023-10-11T15:14:26.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8P7vnegCVvhxWvwqoyDDQsNF+8= 2023-10-11T15:14:26.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8P7vnegCVvhxWvwqoyDDQsNF+8=', 'Content-Length': '0'} 2023-10-11T15:14:26.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e41fedd04bd327-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.225 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.225 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.226 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.226 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.226 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.227 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.227 DEBUG:boto:Data: 2023-10-11T15:14:26.228 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.228 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.229 DEBUG:boto:Params: {} 2023-10-11T15:14:26.229 DEBUG:boto:Token: None 2023-10-11T15:14:26.230 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G8P7vnegCVvhxWvwqoyDDQsNF+8= 2023-10-11T15:14:26.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G8P7vnegCVvhxWvwqoyDDQsNF+8=', 'Content-Length': '0'} 2023-10-11T15:14:26.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5b7cb5bf4349a5c-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-11T15:14:26.233 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.234 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.234 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.235 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.237 DEBUG:boto:Method: GET 2023-10-11T15:14:26.237 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.238 DEBUG:boto:Data: 2023-10-11T15:14:26.238 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.239 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.239 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.239 DEBUG:boto:Params: {} 2023-10-11T15:14:26.240 DEBUG:boto:Token: None 2023-10-11T15:14:26.240 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.241 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw= 2023-10-11T15:14:26.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw=', 'Content-Length': '0'} 2023-10-11T15:14:26.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b111b48b3e8abed-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.286 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.287 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.287 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.288 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.288 DEBUG:boto:Method: GET 2023-10-11T15:14:26.289 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.289 DEBUG:boto:Data: 2023-10-11T15:14:26.289 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.290 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.291 DEBUG:boto:Params: {} 2023-10-11T15:14:26.291 DEBUG:boto:Token: None 2023-10-11T15:14:26.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061?versionId=null 2023-10-11T15:14:26.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw= 2023-10-11T15:14:26.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+n07yNZUikX4mRGH+CQzVq7TWTw=', 'Content-Length': '0'} 2023-10-11T15:14:26.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74ca7b1b5fc5f41-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.296 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.296 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.296 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.297 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.297 DEBUG:boto:Method: GET 2023-10-11T15:14:26.298 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.298 DEBUG:boto:Data: 2023-10-11T15:14:26.299 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.300 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.300 DEBUG:boto:Params: {} 2023-10-11T15:14:26.301 DEBUG:boto:Token: None 2023-10-11T15:14:26.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7d4314eb8b7db1-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.305 DEBUG:boto:b'dghqvd-58obj000000000000000000611000falseobj000000000000000000612023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.306 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.307 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.307 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.307 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.308 DEBUG:boto:Method: GET 2023-10-11T15:14:26.308 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.309 DEBUG:boto:Data: 2023-10-11T15:14:26.309 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.310 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.310 DEBUG:boto:Params: {} 2023-10-11T15:14:26.311 DEBUG:boto:Token: None 2023-10-11T15:14:26.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc93dba56c26a115-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.317 DEBUG:boto:b'dghqvd-58obj000000000000000000611000falseobj000000000000000000612023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.317 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.318 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.318 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.318 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.319 DEBUG:boto:Data: 2023-10-11T15:14:26.319 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.320 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.320 DEBUG:boto:Params: {} 2023-10-11T15:14:26.321 DEBUG:boto:Token: None 2023-10-11T15:14:26.321 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur4hVmi2UiXe5pIwyce469CahrE= 2023-10-11T15:14:26.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur4hVmi2UiXe5pIwyce469CahrE=', 'Content-Length': '0'} 2023-10-11T15:14:26.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f117bce55689d446-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.325 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.325 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.326 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.326 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.326 DEBUG:boto:Data: 2023-10-11T15:14:26.327 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.327 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.328 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.328 DEBUG:boto:Params: {} 2023-10-11T15:14:26.328 DEBUG:boto:Token: None 2023-10-11T15:14:26.329 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur4hVmi2UiXe5pIwyce469CahrE= 2023-10-11T15:14:26.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur4hVmi2UiXe5pIwyce469CahrE=', 'Content-Length': '0'} 2023-10-11T15:14:26.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ffc55c5bf133c8-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-10-11T15:14:26.333 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.333 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.334 DEBUG:boto:Method: GET 2023-10-11T15:14:26.334 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.335 DEBUG:boto:Data: 2023-10-11T15:14:26.335 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.336 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.336 DEBUG:boto:Params: {} 2023-10-11T15:14:26.337 DEBUG:boto:Token: None 2023-10-11T15:14:26.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqgUYZ+YB67V6mMKVhpB6/0KxUY= 2023-10-11T15:14:26.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqgUYZ+YB67V6mMKVhpB6/0KxUY=', 'Content-Length': '0'} 2023-10-11T15:14:26.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b43c9c851dbb423-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.341 DEBUG:boto:path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.341 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.342 DEBUG:boto:Method: GET 2023-10-11T15:14:26.342 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.343 DEBUG:boto:Data: 2023-10-11T15:14:26.343 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.344 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.344 DEBUG:boto:Params: {} 2023-10-11T15:14:26.345 DEBUG:boto:Token: None 2023-10-11T15:14:26.345 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000061 2023-10-11T15:14:26.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqgUYZ+YB67V6mMKVhpB6/0KxUY= 2023-10-11T15:14:26.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqgUYZ+YB67V6mMKVhpB6/0KxUY=', 'Content-Length': '0'} 2023-10-11T15:14:26.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a8a870f2f64eb4-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.390 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.391 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.391 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.392 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.392 DEBUG:boto:Method: GET 2023-10-11T15:14:26.393 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.393 DEBUG:boto:Data: 2023-10-11T15:14:26.393 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.394 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.395 DEBUG:boto:Params: {} 2023-10-11T15:14:26.395 DEBUG:boto:Token: None 2023-10-11T15:14:26.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042dd431a8d4d4057-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.400 DEBUG:boto:b'dghqvd-58obj000000000000000000611000falseobj000000000000000000612023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.400 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.401 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.401 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.402 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.402 DEBUG:boto:Method: GET 2023-10-11T15:14:26.403 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000061 2023-10-11T15:14:26.403 DEBUG:boto:Data: 2023-10-11T15:14:26.403 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.404 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.405 DEBUG:boto:Params: {} 2023-10-11T15:14:26.405 DEBUG:boto:Token: None 2023-10-11T15:14:26.405 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361ea992d5a3f84d-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.411 DEBUG:boto:b'dghqvd-58obj000000000000000000611000falseobj000000000000000000612023-10-11T15:14:03.430Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-11T15:14:26.412 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.412 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.413 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.413 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.413 DEBUG:boto:Method: GET 2023-10-11T15:14:26.414 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.414 DEBUG:boto:Data: 2023-10-11T15:14:26.415 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.415 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.416 DEBUG:boto:Params: {} 2023-10-11T15:14:26.416 DEBUG:boto:Token: None 2023-10-11T15:14:26.417 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E= 2023-10-11T15:14:26.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E=', 'Content-Length': '0'} 2023-10-11T15:14:26.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002991b0c711939c75-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.421 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.421 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.421 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.422 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.422 DEBUG:boto:Method: GET 2023-10-11T15:14:26.422 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.423 DEBUG:boto:Data: 2023-10-11T15:14:26.423 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.424 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.425 DEBUG:boto:Params: {} 2023-10-11T15:14:26.425 DEBUG:boto:Token: None 2023-10-11T15:14:26.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E= 2023-10-11T15:14:26.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E=', 'Content-Length': '0'} 2023-10-11T15:14:26.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7df3d1f22d23786-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.429 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.429 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.430 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.430 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.431 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.431 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.431 DEBUG:boto:Data: 2023-10-11T15:14:26.432 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.432 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.433 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.433 DEBUG:boto:Params: {} 2023-10-11T15:14:26.433 DEBUG:boto:Token: None 2023-10-11T15:14:26.434 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KHrvdDq3wCAYS4V8ENcM5P2m8Ik= 2023-10-11T15:14:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KHrvdDq3wCAYS4V8ENcM5P2m8Ik=', 'Content-Length': '0'} 2023-10-11T15:14:26.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec8d2cca9ec377c-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.437 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.438 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.438 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.438 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.439 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.439 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.440 DEBUG:boto:Data: 2023-10-11T15:14:26.440 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.441 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.441 DEBUG:boto:Params: {} 2023-10-11T15:14:26.442 DEBUG:boto:Token: None 2023-10-11T15:14:26.442 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KHrvdDq3wCAYS4V8ENcM5P2m8Ik= 2023-10-11T15:14:26.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KHrvdDq3wCAYS4V8ENcM5P2m8Ik=', 'Content-Length': '0'} 2023-10-11T15:14:26.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017743c43c270915d-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-11T15:14:26.446 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.446 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.447 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.447 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.447 DEBUG:boto:Method: GET 2023-10-11T15:14:26.448 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.448 DEBUG:boto:Data: 2023-10-11T15:14:26.449 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.450 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.450 DEBUG:boto:Params: {} 2023-10-11T15:14:26.450 DEBUG:boto:Token: None 2023-10-11T15:14:26.451 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.451 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E= 2023-10-11T15:14:26.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E=', 'Content-Length': '0'} 2023-10-11T15:14:26.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c8b8e470a2605b-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.454 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.455 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.456 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.456 DEBUG:boto:Method: GET 2023-10-11T15:14:26.457 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.457 DEBUG:boto:Data: 2023-10-11T15:14:26.457 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.458 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.459 DEBUG:boto:Params: {} 2023-10-11T15:14:26.459 DEBUG:boto:Token: None 2023-10-11T15:14:26.460 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062?versionId=null 2023-10-11T15:14:26.460 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E= 2023-10-11T15:14:26.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t/xU3ZLTmFeQSaJox36znHUSv3E=', 'Content-Length': '0'} 2023-10-11T15:14:26.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e25fde6a815992-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.463 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.463 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.464 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.464 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.464 DEBUG:boto:Method: GET 2023-10-11T15:14:26.465 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.465 DEBUG:boto:Data: 2023-10-11T15:14:26.466 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.466 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.467 DEBUG:boto:Params: {} 2023-10-11T15:14:26.467 DEBUG:boto:Token: None 2023-10-11T15:14:26.468 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c776b31865f8abab-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.472 DEBUG:boto:b'dghqvd-58obj000000000000000000621000falseobj000000000000000000622023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.472 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.473 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.473 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.474 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.474 DEBUG:boto:Method: GET 2023-10-11T15:14:26.474 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.475 DEBUG:boto:Data: 2023-10-11T15:14:26.475 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.476 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.476 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.477 DEBUG:boto:Params: {} 2023-10-11T15:14:26.477 DEBUG:boto:Token: None 2023-10-11T15:14:26.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5076645bd7c7b2d-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.526 DEBUG:boto:b'dghqvd-58obj000000000000000000621000falseobj000000000000000000622023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.527 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.528 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.528 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.529 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.529 DEBUG:boto:Data: 2023-10-11T15:14:26.529 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.530 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.531 DEBUG:boto:Params: {} 2023-10-11T15:14:26.531 DEBUG:boto:Token: None 2023-10-11T15:14:26.531 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.532 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+klYhOUW8lBJh32pAzwvfLCIjao= 2023-10-11T15:14:26.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+klYhOUW8lBJh32pAzwvfLCIjao=', 'Content-Length': '0'} 2023-10-11T15:14:26.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c7cccd898e6a099-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.535 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.535 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.536 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.536 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.537 DEBUG:boto:Data: 2023-10-11T15:14:26.537 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.538 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.538 DEBUG:boto:Params: {} 2023-10-11T15:14:26.539 DEBUG:boto:Token: None 2023-10-11T15:14:26.539 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+klYhOUW8lBJh32pAzwvfLCIjao= 2023-10-11T15:14:26.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+klYhOUW8lBJh32pAzwvfLCIjao=', 'Content-Length': '0'} 2023-10-11T15:14:26.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003806c205a73de480-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-10-11T15:14:26.543 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.544 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.544 DEBUG:boto:Method: GET 2023-10-11T15:14:26.544 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.545 DEBUG:boto:Data: 2023-10-11T15:14:26.545 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.546 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.547 DEBUG:boto:Params: {} 2023-10-11T15:14:26.547 DEBUG:boto:Token: None 2023-10-11T15:14:26.547 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IeHBoyTmKAVJdPnLJ0Tem9mN2x8= 2023-10-11T15:14:26.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IeHBoyTmKAVJdPnLJ0Tem9mN2x8=', 'Content-Length': '0'} 2023-10-11T15:14:26.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be6048324cca4b8-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.594 DEBUG:boto:path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.595 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.595 DEBUG:boto:Method: GET 2023-10-11T15:14:26.596 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.596 DEBUG:boto:Data: 2023-10-11T15:14:26.597 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.597 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.598 DEBUG:boto:Params: {} 2023-10-11T15:14:26.598 DEBUG:boto:Token: None 2023-10-11T15:14:26.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000062 2023-10-11T15:14:26.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IeHBoyTmKAVJdPnLJ0Tem9mN2x8= 2023-10-11T15:14:26.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IeHBoyTmKAVJdPnLJ0Tem9mN2x8=', 'Content-Length': '0'} 2023-10-11T15:14:26.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5afaf894b9e8fbd-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.602 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.603 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.603 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.603 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.604 DEBUG:boto:Method: GET 2023-10-11T15:14:26.604 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.605 DEBUG:boto:Data: 2023-10-11T15:14:26.605 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.605 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.606 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.606 DEBUG:boto:Params: {} 2023-10-11T15:14:26.607 DEBUG:boto:Token: None 2023-10-11T15:14:26.607 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dac640bcb076f8e-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.611 DEBUG:boto:b'dghqvd-58obj000000000000000000621000falseobj000000000000000000622023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.612 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.612 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.612 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.613 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.613 DEBUG:boto:Method: GET 2023-10-11T15:14:26.614 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000062 2023-10-11T15:14:26.614 DEBUG:boto:Data: 2023-10-11T15:14:26.614 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.615 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.615 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.616 DEBUG:boto:Params: {} 2023-10-11T15:14:26.616 DEBUG:boto:Token: None 2023-10-11T15:14:26.616 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.617 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e0b4244922bee6b-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.622 DEBUG:boto:b'dghqvd-58obj000000000000000000621000falseobj000000000000000000622023-10-11T15:14:04.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-11T15:14:26.623 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.623 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.624 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.624 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.624 DEBUG:boto:Method: GET 2023-10-11T15:14:26.625 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.625 DEBUG:boto:Data: 2023-10-11T15:14:26.626 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.626 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.626 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.627 DEBUG:boto:Params: {} 2023-10-11T15:14:26.627 DEBUG:boto:Token: None 2023-10-11T15:14:26.628 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.628 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw= 2023-10-11T15:14:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw=', 'Content-Length': '0'} 2023-10-11T15:14:26.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003fb0f5a9f6e0564-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.632 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.632 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.632 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.633 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.633 DEBUG:boto:Method: GET 2023-10-11T15:14:26.633 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.634 DEBUG:boto:Data: 2023-10-11T15:14:26.634 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.635 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.635 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.635 DEBUG:boto:Params: {} 2023-10-11T15:14:26.636 DEBUG:boto:Token: None 2023-10-11T15:14:26.636 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.637 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw= 2023-10-11T15:14:26.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw=', 'Content-Length': '0'} 2023-10-11T15:14:26.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71fba76eb9a388c-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.640 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.640 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.640 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.641 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.641 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.641 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.642 DEBUG:boto:Data: 2023-10-11T15:14:26.642 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.643 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.643 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.643 DEBUG:boto:Params: {} 2023-10-11T15:14:26.644 DEBUG:boto:Token: None 2023-10-11T15:14:26.644 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wfYbK4T14NXeU/wM+ln/Y+MGyzU= 2023-10-11T15:14:26.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wfYbK4T14NXeU/wM+ln/Y+MGyzU=', 'Content-Length': '0'} 2023-10-11T15:14:26.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d452617807b3fef6-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.647 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.648 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.648 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.649 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.649 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.649 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.650 DEBUG:boto:Data: 2023-10-11T15:14:26.650 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.651 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.651 DEBUG:boto:Params: {} 2023-10-11T15:14:26.652 DEBUG:boto:Token: None 2023-10-11T15:14:26.652 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wfYbK4T14NXeU/wM+ln/Y+MGyzU= 2023-10-11T15:14:26.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wfYbK4T14NXeU/wM+ln/Y+MGyzU=', 'Content-Length': '0'} 2023-10-11T15:14:26.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056aaaa2a626a4f47-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-11T15:14:26.656 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.656 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.656 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.657 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.657 DEBUG:boto:Method: GET 2023-10-11T15:14:26.658 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.658 DEBUG:boto:Data: 2023-10-11T15:14:26.658 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.659 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.660 DEBUG:boto:Params: {} 2023-10-11T15:14:26.660 DEBUG:boto:Token: None 2023-10-11T15:14:26.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw= 2023-10-11T15:14:26.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw=', 'Content-Length': '0'} 2023-10-11T15:14:26.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f41828a4ebe664f-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.710 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.711 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.711 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.712 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.712 DEBUG:boto:Method: GET 2023-10-11T15:14:26.713 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.713 DEBUG:boto:Data: 2023-10-11T15:14:26.713 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.714 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.715 DEBUG:boto:Params: {} 2023-10-11T15:14:26.715 DEBUG:boto:Token: None 2023-10-11T15:14:26.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065?versionId=null 2023-10-11T15:14:26.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw= 2023-10-11T15:14:26.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/0JfrxOk/wG17oyBBX/nHoH3Bw=', 'Content-Length': '0'} 2023-10-11T15:14:26.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf4827f5375a7bd-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.720 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.720 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.720 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.721 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.721 DEBUG:boto:Method: GET 2023-10-11T15:14:26.722 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.722 DEBUG:boto:Data: 2023-10-11T15:14:26.723 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.723 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.724 DEBUG:boto:Params: {} 2023-10-11T15:14:26.724 DEBUG:boto:Token: None 2023-10-11T15:14:26.725 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b525943897bb5af-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.730 DEBUG:boto:b'dghqvd-58obj000000000000000000651000falseobj000000000000000000652023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.731 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.731 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.731 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.732 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.732 DEBUG:boto:Method: GET 2023-10-11T15:14:26.733 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.733 DEBUG:boto:Data: 2023-10-11T15:14:26.734 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.734 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.735 DEBUG:boto:Params: {} 2023-10-11T15:14:26.735 DEBUG:boto:Token: None 2023-10-11T15:14:26.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c58d3ab75fe1f905-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.741 DEBUG:boto:b'dghqvd-58obj000000000000000000651000falseobj000000000000000000652023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.742 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.743 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.743 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.743 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.744 DEBUG:boto:Data: 2023-10-11T15:14:26.744 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.745 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.745 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.745 DEBUG:boto:Params: {} 2023-10-11T15:14:26.746 DEBUG:boto:Token: None 2023-10-11T15:14:26.746 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6mN0jjDjGtoiujb6PrkuxHai2+A= 2023-10-11T15:14:26.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6mN0jjDjGtoiujb6PrkuxHai2+A=', 'Content-Length': '0'} 2023-10-11T15:14:26.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa375d8788e01b74-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.750 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.750 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.751 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.751 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.751 DEBUG:boto:Data: 2023-10-11T15:14:26.752 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.753 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.753 DEBUG:boto:Params: {} 2023-10-11T15:14:26.753 DEBUG:boto:Token: None 2023-10-11T15:14:26.754 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6mN0jjDjGtoiujb6PrkuxHai2+A= 2023-10-11T15:14:26.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6mN0jjDjGtoiujb6PrkuxHai2+A=', 'Content-Length': '0'} 2023-10-11T15:14:26.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8cff4d9168d1a3-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-10-11T15:14:26.758 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.758 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.759 DEBUG:boto:Method: GET 2023-10-11T15:14:26.759 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.759 DEBUG:boto:Data: 2023-10-11T15:14:26.760 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.761 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.761 DEBUG:boto:Params: {} 2023-10-11T15:14:26.762 DEBUG:boto:Token: None 2023-10-11T15:14:26.762 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mzhvtYSiJQ/hNM8wVc0yJmDBjKM= 2023-10-11T15:14:26.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mzhvtYSiJQ/hNM8wVc0yJmDBjKM=', 'Content-Length': '0'} 2023-10-11T15:14:26.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507918efbac51cde-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.766 DEBUG:boto:path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.766 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.766 DEBUG:boto:Method: GET 2023-10-11T15:14:26.767 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.767 DEBUG:boto:Data: 2023-10-11T15:14:26.768 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.769 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.769 DEBUG:boto:Params: {} 2023-10-11T15:14:26.769 DEBUG:boto:Token: None 2023-10-11T15:14:26.770 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000065 2023-10-11T15:14:26.770 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mzhvtYSiJQ/hNM8wVc0yJmDBjKM= 2023-10-11T15:14:26.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mzhvtYSiJQ/hNM8wVc0yJmDBjKM=', 'Content-Length': '0'} 2023-10-11T15:14:26.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4af5d43f76d882e-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.774 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.774 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.775 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.775 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.776 DEBUG:boto:Method: GET 2023-10-11T15:14:26.776 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.777 DEBUG:boto:Data: 2023-10-11T15:14:26.777 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.778 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.778 DEBUG:boto:Params: {} 2023-10-11T15:14:26.779 DEBUG:boto:Token: None 2023-10-11T15:14:26.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056774027044ddf8f-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.783 DEBUG:boto:b'dghqvd-58obj000000000000000000651000falseobj000000000000000000652023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.784 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.784 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.784 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.785 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.785 DEBUG:boto:Method: GET 2023-10-11T15:14:26.786 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000065 2023-10-11T15:14:26.786 DEBUG:boto:Data: 2023-10-11T15:14:26.786 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.787 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.788 DEBUG:boto:Params: {} 2023-10-11T15:14:26.788 DEBUG:boto:Token: None 2023-10-11T15:14:26.789 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057cc2512d7fcd125-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.794 DEBUG:boto:b'dghqvd-58obj000000000000000000651000falseobj000000000000000000652023-10-11T15:14:05.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-11T15:14:26.795 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.796 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.796 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.797 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.797 DEBUG:boto:Method: GET 2023-10-11T15:14:26.798 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.798 DEBUG:boto:Data: 2023-10-11T15:14:26.798 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.799 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.800 DEBUG:boto:Params: {} 2023-10-11T15:14:26.800 DEBUG:boto:Token: None 2023-10-11T15:14:26.801 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY= 2023-10-11T15:14:26.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY=', 'Content-Length': '0'} 2023-10-11T15:14:26.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e82518ef167f4257-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.805 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.805 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.806 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.806 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.807 DEBUG:boto:Method: GET 2023-10-11T15:14:26.807 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.807 DEBUG:boto:Data: 2023-10-11T15:14:26.808 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.809 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.809 DEBUG:boto:Params: {} 2023-10-11T15:14:26.810 DEBUG:boto:Token: None 2023-10-11T15:14:26.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY= 2023-10-11T15:14:26.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY=', 'Content-Length': '0'} 2023-10-11T15:14:26.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065c834715eb6ac4e-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.814 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.814 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.815 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.815 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.816 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.816 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.817 DEBUG:boto:Data: 2023-10-11T15:14:26.817 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.818 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.818 DEBUG:boto:Params: {} 2023-10-11T15:14:26.819 DEBUG:boto:Token: None 2023-10-11T15:14:26.819 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lnaZ1aKxGNIAGIXTFHplfAwlStg= 2023-10-11T15:14:26.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lnaZ1aKxGNIAGIXTFHplfAwlStg=', 'Content-Length': '0'} 2023-10-11T15:14:26.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032dcea113fa35507-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.823 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.823 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.823 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.824 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.824 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.825 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.825 DEBUG:boto:Data: 2023-10-11T15:14:26.825 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.826 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.827 DEBUG:boto:Params: {} 2023-10-11T15:14:26.827 DEBUG:boto:Token: None 2023-10-11T15:14:26.828 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lnaZ1aKxGNIAGIXTFHplfAwlStg= 2023-10-11T15:14:26.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lnaZ1aKxGNIAGIXTFHplfAwlStg=', 'Content-Length': '0'} 2023-10-11T15:14:26.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97c24b6c4af5f69-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-11T15:14:26.831 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.832 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.832 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.833 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.833 DEBUG:boto:Method: GET 2023-10-11T15:14:26.833 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.834 DEBUG:boto:Data: 2023-10-11T15:14:26.834 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.835 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.835 DEBUG:boto:Params: {} 2023-10-11T15:14:26.836 DEBUG:boto:Token: None 2023-10-11T15:14:26.836 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY= 2023-10-11T15:14:26.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY=', 'Content-Length': '0'} 2023-10-11T15:14:26.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020713a4a518f5bf6-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.840 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.840 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.841 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.841 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.841 DEBUG:boto:Method: GET 2023-10-11T15:14:26.842 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.842 DEBUG:boto:Data: 2023-10-11T15:14:26.843 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.843 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.844 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.844 DEBUG:boto:Params: {} 2023-10-11T15:14:26.844 DEBUG:boto:Token: None 2023-10-11T15:14:26.845 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076?versionId=null 2023-10-11T15:14:26.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY= 2023-10-11T15:14:26.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ryklM1S+ubHtPxD+PLedRDMhOvY=', 'Content-Length': '0'} 2023-10-11T15:14:26.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001438ad8dd9322b70-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.890 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.891 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.891 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.892 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.892 DEBUG:boto:Method: GET 2023-10-11T15:14:26.892 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.893 DEBUG:boto:Data: 2023-10-11T15:14:26.893 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.893 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.894 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.894 DEBUG:boto:Params: {} 2023-10-11T15:14:26.895 DEBUG:boto:Token: None 2023-10-11T15:14:26.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d142678d78710f-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.899 DEBUG:boto:b'dghqvd-58obj000000000000000000761000falseobj000000000000000000762023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.900 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.900 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.901 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.901 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.902 DEBUG:boto:Method: GET 2023-10-11T15:14:26.902 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.902 DEBUG:boto:Data: 2023-10-11T15:14:26.903 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.904 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.904 DEBUG:boto:Params: {} 2023-10-11T15:14:26.904 DEBUG:boto:Token: None 2023-10-11T15:14:26.905 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000856fb0a9b9b221e8-006526bbd2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.910 DEBUG:boto:b'dghqvd-58obj000000000000000000761000falseobj000000000000000000762023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.911 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.911 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.912 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.912 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.913 DEBUG:boto:Data: 2023-10-11T15:14:26.913 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.914 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.914 DEBUG:boto:Params: {} 2023-10-11T15:14:26.914 DEBUG:boto:Token: None 2023-10-11T15:14:26.915 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b3EqjorAK2VA1nfOo8dPsTHqSJU= 2023-10-11T15:14:26.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b3EqjorAK2VA1nfOo8dPsTHqSJU=', 'Content-Length': '0'} 2023-10-11T15:14:26.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6c99bfde1318f3-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.918 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.918 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.919 DEBUG:boto:Method: HEAD 2023-10-11T15:14:26.919 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.920 DEBUG:boto:Data: 2023-10-11T15:14:26.920 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.921 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.921 DEBUG:boto:Params: {} 2023-10-11T15:14:26.922 DEBUG:boto:Token: None 2023-10-11T15:14:26.922 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b3EqjorAK2VA1nfOo8dPsTHqSJU= 2023-10-11T15:14:26.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b3EqjorAK2VA1nfOo8dPsTHqSJU=', 'Content-Length': '0'} 2023-10-11T15:14:26.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898f8a03371fc682-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-10-11T15:14:26.926 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.926 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.927 DEBUG:boto:Method: GET 2023-10-11T15:14:26.927 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.928 DEBUG:boto:Data: 2023-10-11T15:14:26.928 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.929 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.929 DEBUG:boto:Params: {} 2023-10-11T15:14:26.930 DEBUG:boto:Token: None 2023-10-11T15:14:26.930 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9v4wrZDDlc+yvecOQXCHDFRcmxo= 2023-10-11T15:14:26.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9v4wrZDDlc+yvecOQXCHDFRcmxo=', 'Content-Length': '0'} 2023-10-11T15:14:26.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399b1c0d06318124-006526bbd2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.978 DEBUG:boto:path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.979 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.979 DEBUG:boto:Method: GET 2023-10-11T15:14:26.979 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.980 DEBUG:boto:Data: 2023-10-11T15:14:26.980 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.981 DEBUG:boto:Port: 8000 2023-10-11T15:14:26.981 DEBUG:boto:Params: {} 2023-10-11T15:14:26.982 DEBUG:boto:Token: None 2023-10-11T15:14:26.982 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/obj00000000000000000076 2023-10-11T15:14:26.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9v4wrZDDlc+yvecOQXCHDFRcmxo= 2023-10-11T15:14:26.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9v4wrZDDlc+yvecOQXCHDFRcmxo=', 'Content-Length': '0'} 2023-10-11T15:14:26.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:13:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8d3aa21dd41f250-006526bbd2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.986 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.986 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.987 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.987 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.988 DEBUG:boto:Method: GET 2023-10-11T15:14:26.988 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.988 DEBUG:boto:Data: 2023-10-11T15:14:26.989 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:26.990 DEBUG:boto:Port: 8001 2023-10-11T15:14:26.990 DEBUG:boto:Params: {} 2023-10-11T15:14:26.991 DEBUG:boto:Token: None 2023-10-11T15:14:26.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:26 GMT /dghqvd-58/ 2023-10-11T15:14:26.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8= 2023-10-11T15:14:26.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDsHK7OtQ5bfOJI9mNioXCowiA8=', 'Content-Length': '0'} 2023-10-11T15:14:26.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdcfdde8bca836d5-006526bbd2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:26.995 DEBUG:boto:b'dghqvd-58obj000000000000000000761000falseobj000000000000000000762023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:26.995 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:26.996 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:26.996 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.997 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.997 DEBUG:boto:Method: GET 2023-10-11T15:14:26.998 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000076 2023-10-11T15:14:26.998 DEBUG:boto:Data: 2023-10-11T15:14:26.998 DEBUG:boto:Headers: {} 2023-10-11T15:14:26.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:26.999 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.000 DEBUG:boto:Params: {} 2023-10-11T15:14:27.000 DEBUG:boto:Token: None 2023-10-11T15:14:27.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ffcb72cf0cfb54-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.006 DEBUG:boto:b'dghqvd-58obj000000000000000000761000falseobj000000000000000000762023-10-11T15:13:59.090Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-11T15:14:27.007 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.007 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.008 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.008 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.009 DEBUG:boto:Method: GET 2023-10-11T15:14:27.009 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.009 DEBUG:boto:Data: 2023-10-11T15:14:27.010 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.011 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.011 DEBUG:boto:Params: {} 2023-10-11T15:14:27.011 DEBUG:boto:Token: None 2023-10-11T15:14:27.012 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE= 2023-10-11T15:14:27.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE=', 'Content-Length': '0'} 2023-10-11T15:14:27.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1e4c7adfe76ae6-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.015 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.016 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.016 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.016 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.017 DEBUG:boto:Method: GET 2023-10-11T15:14:27.017 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.018 DEBUG:boto:Data: 2023-10-11T15:14:27.018 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.019 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.019 DEBUG:boto:Params: {} 2023-10-11T15:14:27.020 DEBUG:boto:Token: None 2023-10-11T15:14:27.020 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE= 2023-10-11T15:14:27.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE=', 'Content-Length': '0'} 2023-10-11T15:14:27.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472a407db1382c5e-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.023 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.024 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.024 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.024 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.025 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.025 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.026 DEBUG:boto:Data: 2023-10-11T15:14:27.026 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.027 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.027 DEBUG:boto:Params: {} 2023-10-11T15:14:27.028 DEBUG:boto:Token: None 2023-10-11T15:14:27.028 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.028 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hKJ6MrNgZBkvppfm0d87MngBa0E= 2023-10-11T15:14:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hKJ6MrNgZBkvppfm0d87MngBa0E=', 'Content-Length': '0'} 2023-10-11T15:14:27.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b1a129aed0f71b9-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.031 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.032 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.032 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.032 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.033 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.033 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.034 DEBUG:boto:Data: 2023-10-11T15:14:27.034 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.035 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.035 DEBUG:boto:Params: {} 2023-10-11T15:14:27.035 DEBUG:boto:Token: None 2023-10-11T15:14:27.036 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hKJ6MrNgZBkvppfm0d87MngBa0E= 2023-10-11T15:14:27.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hKJ6MrNgZBkvppfm0d87MngBa0E=', 'Content-Length': '0'} 2023-10-11T15:14:27.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab57f9338c1d5c8-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-11T15:14:27.040 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.040 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.041 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.041 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.041 DEBUG:boto:Method: GET 2023-10-11T15:14:27.042 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.042 DEBUG:boto:Data: 2023-10-11T15:14:27.043 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.043 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.044 DEBUG:boto:Params: {} 2023-10-11T15:14:27.044 DEBUG:boto:Token: None 2023-10-11T15:14:27.045 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE= 2023-10-11T15:14:27.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE=', 'Content-Length': '0'} 2023-10-11T15:14:27.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a343a672cfce97b-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.090 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.091 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.091 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.092 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.092 DEBUG:boto:Method: GET 2023-10-11T15:14:27.092 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.093 DEBUG:boto:Data: 2023-10-11T15:14:27.093 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.094 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.095 DEBUG:boto:Params: {} 2023-10-11T15:14:27.095 DEBUG:boto:Token: None 2023-10-11T15:14:27.095 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133?versionId=null 2023-10-11T15:14:27.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE= 2023-10-11T15:14:27.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2I5jcmAJ7ZitByhoJnLGA1dWArE=', 'Content-Length': '0'} 2023-10-11T15:14:27.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b7e01d06d90f31-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.099 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.100 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.100 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.101 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.101 DEBUG:boto:Method: GET 2023-10-11T15:14:27.101 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.102 DEBUG:boto:Data: 2023-10-11T15:14:27.102 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.103 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.103 DEBUG:boto:Params: {} 2023-10-11T15:14:27.104 DEBUG:boto:Token: None 2023-10-11T15:14:27.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f66edcd4f710a37b-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.108 DEBUG:boto:b'dghqvd-58obj000000000000000001331000falseobj000000000000000001332023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.109 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.109 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.109 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.110 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.110 DEBUG:boto:Method: GET 2023-10-11T15:14:27.111 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.111 DEBUG:boto:Data: 2023-10-11T15:14:27.111 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.112 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.113 DEBUG:boto:Params: {} 2023-10-11T15:14:27.113 DEBUG:boto:Token: None 2023-10-11T15:14:27.113 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b65c4023ed1905-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.162 DEBUG:boto:b'dghqvd-58obj000000000000000001331000falseobj000000000000000001332023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.163 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.163 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.164 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.164 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.165 DEBUG:boto:Data: 2023-10-11T15:14:27.165 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.166 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.166 DEBUG:boto:Params: {} 2023-10-11T15:14:27.167 DEBUG:boto:Token: None 2023-10-11T15:14:27.167 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:86jqaVM6/mKhNttAOWwe9398hOU= 2023-10-11T15:14:27.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86jqaVM6/mKhNttAOWwe9398hOU=', 'Content-Length': '0'} 2023-10-11T15:14:27.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665926163443f595-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.170 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.171 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.171 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.172 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.172 DEBUG:boto:Data: 2023-10-11T15:14:27.172 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.173 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.173 DEBUG:boto:Params: {} 2023-10-11T15:14:27.174 DEBUG:boto:Token: None 2023-10-11T15:14:27.174 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:86jqaVM6/mKhNttAOWwe9398hOU= 2023-10-11T15:14:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86jqaVM6/mKhNttAOWwe9398hOU=', 'Content-Length': '0'} 2023-10-11T15:14:27.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343d79510eda3d4b-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-10-11T15:14:27.178 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.179 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.179 DEBUG:boto:Method: GET 2023-10-11T15:14:27.180 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.180 DEBUG:boto:Data: 2023-10-11T15:14:27.180 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.181 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.182 DEBUG:boto:Params: {} 2023-10-11T15:14:27.182 DEBUG:boto:Token: None 2023-10-11T15:14:27.183 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.183 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CK1E6W3XUuEgpK8sGPre/qXl/is= 2023-10-11T15:14:27.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CK1E6W3XUuEgpK8sGPre/qXl/is=', 'Content-Length': '0'} 2023-10-11T15:14:27.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b378645f7c2ca762-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.186 DEBUG:boto:path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.187 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.187 DEBUG:boto:Method: GET 2023-10-11T15:14:27.188 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.188 DEBUG:boto:Data: 2023-10-11T15:14:27.188 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.189 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.190 DEBUG:boto:Params: {} 2023-10-11T15:14:27.190 DEBUG:boto:Token: None 2023-10-11T15:14:27.191 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000133 2023-10-11T15:14:27.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CK1E6W3XUuEgpK8sGPre/qXl/is= 2023-10-11T15:14:27.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CK1E6W3XUuEgpK8sGPre/qXl/is=', 'Content-Length': '0'} 2023-10-11T15:14:27.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fed7ecd22a0b2f-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.194 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.194 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.195 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.195 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.196 DEBUG:boto:Method: GET 2023-10-11T15:14:27.196 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.196 DEBUG:boto:Data: 2023-10-11T15:14:27.197 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.198 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.198 DEBUG:boto:Params: {} 2023-10-11T15:14:27.199 DEBUG:boto:Token: None 2023-10-11T15:14:27.199 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ede299b1b8b712-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.203 DEBUG:boto:b'dghqvd-58obj000000000000000001331000falseobj000000000000000001332023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.204 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.204 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.205 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.205 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.205 DEBUG:boto:Method: GET 2023-10-11T15:14:27.206 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000133 2023-10-11T15:14:27.206 DEBUG:boto:Data: 2023-10-11T15:14:27.207 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.207 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.208 DEBUG:boto:Params: {} 2023-10-11T15:14:27.208 DEBUG:boto:Token: None 2023-10-11T15:14:27.209 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.209 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f42dda740f80b6-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.214 DEBUG:boto:b'dghqvd-58obj000000000000000001331000falseobj000000000000000001332023-10-11T15:14:03.508Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-11T15:14:27.215 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.216 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.216 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.217 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.217 DEBUG:boto:Method: GET 2023-10-11T15:14:27.217 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.218 DEBUG:boto:Data: 2023-10-11T15:14:27.218 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.219 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.219 DEBUG:boto:Params: {} 2023-10-11T15:14:27.220 DEBUG:boto:Token: None 2023-10-11T15:14:27.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU= 2023-10-11T15:14:27.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU=', 'Content-Length': '0'} 2023-10-11T15:14:27.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f5a3f659ec520d-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.224 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.225 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.225 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.226 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.226 DEBUG:boto:Method: GET 2023-10-11T15:14:27.226 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.227 DEBUG:boto:Data: 2023-10-11T15:14:27.227 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.228 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.228 DEBUG:boto:Params: {} 2023-10-11T15:14:27.229 DEBUG:boto:Token: None 2023-10-11T15:14:27.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU= 2023-10-11T15:14:27.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU=', 'Content-Length': '0'} 2023-10-11T15:14:27.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d70c225a5f19aabb-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.233 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.233 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.234 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.234 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.235 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.235 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.235 DEBUG:boto:Data: 2023-10-11T15:14:27.236 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.236 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.237 DEBUG:boto:Params: {} 2023-10-11T15:14:27.237 DEBUG:boto:Token: None 2023-10-11T15:14:27.238 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zMo7XWakRBFEPSU/XwAZoTxiJe4= 2023-10-11T15:14:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zMo7XWakRBFEPSU/XwAZoTxiJe4=', 'Content-Length': '0'} 2023-10-11T15:14:27.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b8ef2bbfd90e09-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.241 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.241 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.242 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.242 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.243 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.243 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.244 DEBUG:boto:Data: 2023-10-11T15:14:27.244 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.245 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.245 DEBUG:boto:Params: {} 2023-10-11T15:14:27.245 DEBUG:boto:Token: None 2023-10-11T15:14:27.246 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zMo7XWakRBFEPSU/XwAZoTxiJe4= 2023-10-11T15:14:27.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zMo7XWakRBFEPSU/XwAZoTxiJe4=', 'Content-Length': '0'} 2023-10-11T15:14:27.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041840c76028d45e8-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-11T15:14:27.250 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.251 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.251 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.251 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.252 DEBUG:boto:Method: GET 2023-10-11T15:14:27.252 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.253 DEBUG:boto:Data: 2023-10-11T15:14:27.253 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.254 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.254 DEBUG:boto:Params: {} 2023-10-11T15:14:27.255 DEBUG:boto:Token: None 2023-10-11T15:14:27.255 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU= 2023-10-11T15:14:27.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU=', 'Content-Length': '0'} 2023-10-11T15:14:27.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05e04a45790eb45-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.306 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.307 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.308 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.308 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.308 DEBUG:boto:Method: GET 2023-10-11T15:14:27.309 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.309 DEBUG:boto:Data: 2023-10-11T15:14:27.310 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.311 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.312 DEBUG:boto:Params: {} 2023-10-11T15:14:27.312 DEBUG:boto:Token: None 2023-10-11T15:14:27.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162?versionId=null 2023-10-11T15:14:27.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU= 2023-10-11T15:14:27.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:344ZxQMCkngJiYgFNkh+rRNJmAU=', 'Content-Length': '0'} 2023-10-11T15:14:27.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dea7887a178a86-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.317 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.317 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.318 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.318 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.318 DEBUG:boto:Method: GET 2023-10-11T15:14:27.319 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.319 DEBUG:boto:Data: 2023-10-11T15:14:27.320 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.320 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.321 DEBUG:boto:Params: {} 2023-10-11T15:14:27.321 DEBUG:boto:Token: None 2023-10-11T15:14:27.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8092aa1037a1336-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.326 DEBUG:boto:b'dghqvd-58obj000000000000000001621000falseobj000000000000000001622023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.326 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.327 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.327 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.328 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.328 DEBUG:boto:Method: GET 2023-10-11T15:14:27.329 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.329 DEBUG:boto:Data: 2023-10-11T15:14:27.329 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.330 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.331 DEBUG:boto:Params: {} 2023-10-11T15:14:27.331 DEBUG:boto:Token: None 2023-10-11T15:14:27.332 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a7bc249fc66f62-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.337 DEBUG:boto:b'dghqvd-58obj000000000000000001621000falseobj000000000000000001622023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.338 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.341 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.341 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.342 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.342 DEBUG:boto:Data: 2023-10-11T15:14:27.343 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.344 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.344 DEBUG:boto:Params: {} 2023-10-11T15:14:27.344 DEBUG:boto:Token: None 2023-10-11T15:14:27.345 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rOcmlP2ndIIwxmuCypmTdAWGKPU= 2023-10-11T15:14:27.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rOcmlP2ndIIwxmuCypmTdAWGKPU=', 'Content-Length': '0'} 2023-10-11T15:14:27.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0b4a87e069e40d-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.349 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.349 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.349 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.350 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.350 DEBUG:boto:Data: 2023-10-11T15:14:27.351 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.351 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.352 DEBUG:boto:Params: {} 2023-10-11T15:14:27.352 DEBUG:boto:Token: None 2023-10-11T15:14:27.352 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rOcmlP2ndIIwxmuCypmTdAWGKPU= 2023-10-11T15:14:27.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rOcmlP2ndIIwxmuCypmTdAWGKPU=', 'Content-Length': '0'} 2023-10-11T15:14:27.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8097d102dc0b44c-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-10-11T15:14:27.357 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.358 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.358 DEBUG:boto:Method: GET 2023-10-11T15:14:27.359 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.359 DEBUG:boto:Data: 2023-10-11T15:14:27.359 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.360 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.360 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.361 DEBUG:boto:Params: {} 2023-10-11T15:14:27.361 DEBUG:boto:Token: None 2023-10-11T15:14:27.362 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.362 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ADBlPR9RFCf57ZhxqeiALcKDfcA= 2023-10-11T15:14:27.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ADBlPR9RFCf57ZhxqeiALcKDfcA=', 'Content-Length': '0'} 2023-10-11T15:14:27.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3329c02d279b8ee-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.366 DEBUG:boto:path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.366 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.366 DEBUG:boto:Method: GET 2023-10-11T15:14:27.367 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.367 DEBUG:boto:Data: 2023-10-11T15:14:27.368 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.369 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.369 DEBUG:boto:Params: {} 2023-10-11T15:14:27.370 DEBUG:boto:Token: None 2023-10-11T15:14:27.370 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000162 2023-10-11T15:14:27.371 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ADBlPR9RFCf57ZhxqeiALcKDfcA= 2023-10-11T15:14:27.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ADBlPR9RFCf57ZhxqeiALcKDfcA=', 'Content-Length': '0'} 2023-10-11T15:14:27.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5db53f74d60b3c5-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.418 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.419 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.420 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.420 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.421 DEBUG:boto:Method: GET 2023-10-11T15:14:27.421 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.421 DEBUG:boto:Data: 2023-10-11T15:14:27.422 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.423 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.423 DEBUG:boto:Params: {} 2023-10-11T15:14:27.424 DEBUG:boto:Token: None 2023-10-11T15:14:27.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4de1493ccbc236f-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.429 DEBUG:boto:b'dghqvd-58obj000000000000000001621000falseobj000000000000000001622023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.430 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.431 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.431 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.432 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.432 DEBUG:boto:Method: GET 2023-10-11T15:14:27.433 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000162 2023-10-11T15:14:27.433 DEBUG:boto:Data: 2023-10-11T15:14:27.434 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.435 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.435 DEBUG:boto:Params: {} 2023-10-11T15:14:27.436 DEBUG:boto:Token: None 2023-10-11T15:14:27.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d2fd80b6b63bc01-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.442 DEBUG:boto:b'dghqvd-58obj000000000000000001621000falseobj000000000000000001622023-10-11T15:14:03.943Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-11T15:14:27.443 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.444 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.444 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.445 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.445 DEBUG:boto:Method: GET 2023-10-11T15:14:27.446 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.446 DEBUG:boto:Data: 2023-10-11T15:14:27.447 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.448 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.448 DEBUG:boto:Params: {} 2023-10-11T15:14:27.449 DEBUG:boto:Token: None 2023-10-11T15:14:27.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA= 2023-10-11T15:14:27.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA=', 'Content-Length': '0'} 2023-10-11T15:14:27.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af65acf0bae5d01e-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.455 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.456 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.457 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.458 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.459 DEBUG:boto:Method: GET 2023-10-11T15:14:27.459 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.460 DEBUG:boto:Data: 2023-10-11T15:14:27.461 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.462 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.463 DEBUG:boto:Params: {} 2023-10-11T15:14:27.464 DEBUG:boto:Token: None 2023-10-11T15:14:27.465 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA= 2023-10-11T15:14:27.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA=', 'Content-Length': '0'} 2023-10-11T15:14:27.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011edcc26de354c50-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.469 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.469 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.469 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.470 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.470 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.471 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.471 DEBUG:boto:Data: 2023-10-11T15:14:27.471 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.472 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.472 DEBUG:boto:Params: {} 2023-10-11T15:14:27.473 DEBUG:boto:Token: None 2023-10-11T15:14:27.473 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wiZEIYw2SAN4p6T9WcACa44i1Ds= 2023-10-11T15:14:27.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wiZEIYw2SAN4p6T9WcACa44i1Ds=', 'Content-Length': '0'} 2023-10-11T15:14:27.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd3130717c21ef3-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.477 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.477 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.478 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.478 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.479 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.479 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.479 DEBUG:boto:Data: 2023-10-11T15:14:27.480 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.481 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.481 DEBUG:boto:Params: {} 2023-10-11T15:14:27.481 DEBUG:boto:Token: None 2023-10-11T15:14:27.482 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.482 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wiZEIYw2SAN4p6T9WcACa44i1Ds= 2023-10-11T15:14:27.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wiZEIYw2SAN4p6T9WcACa44i1Ds=', 'Content-Length': '0'} 2023-10-11T15:14:27.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db9b000d4bd85e26-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-11T15:14:27.486 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.486 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.487 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.487 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.487 DEBUG:boto:Method: GET 2023-10-11T15:14:27.488 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.488 DEBUG:boto:Data: 2023-10-11T15:14:27.489 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.490 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.490 DEBUG:boto:Params: {} 2023-10-11T15:14:27.491 DEBUG:boto:Token: None 2023-10-11T15:14:27.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA= 2023-10-11T15:14:27.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA=', 'Content-Length': '0'} 2023-10-11T15:14:27.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e53d3c209d285e-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.495 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.495 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.496 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.496 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.496 DEBUG:boto:Method: GET 2023-10-11T15:14:27.497 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.497 DEBUG:boto:Data: 2023-10-11T15:14:27.498 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.499 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.499 DEBUG:boto:Params: {} 2023-10-11T15:14:27.499 DEBUG:boto:Token: None 2023-10-11T15:14:27.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271?versionId=null 2023-10-11T15:14:27.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA= 2023-10-11T15:14:27.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T2KKJmSNt2vWwHBO5njmoIc0yfA=', 'Content-Length': '0'} 2023-10-11T15:14:27.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa12342319f50f68-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.503 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.504 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.504 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.505 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.505 DEBUG:boto:Method: GET 2023-10-11T15:14:27.506 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.506 DEBUG:boto:Data: 2023-10-11T15:14:27.506 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.507 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.508 DEBUG:boto:Params: {} 2023-10-11T15:14:27.508 DEBUG:boto:Token: None 2023-10-11T15:14:27.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1f8a4e44e79878-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.513 DEBUG:boto:b'dghqvd-58obj000000000000000002711000falseobj000000000000000002712023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.513 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.514 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.514 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.515 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.515 DEBUG:boto:Method: GET 2023-10-11T15:14:27.515 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.516 DEBUG:boto:Data: 2023-10-11T15:14:27.516 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.517 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.518 DEBUG:boto:Params: {} 2023-10-11T15:14:27.518 DEBUG:boto:Token: None 2023-10-11T15:14:27.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c983e51abe75f47-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.566 DEBUG:boto:b'dghqvd-58obj000000000000000002711000falseobj000000000000000002712023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.567 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.568 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.568 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.569 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.569 DEBUG:boto:Data: 2023-10-11T15:14:27.569 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.570 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.571 DEBUG:boto:Params: {} 2023-10-11T15:14:27.571 DEBUG:boto:Token: None 2023-10-11T15:14:27.571 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MuR6b/GwxfZqQQgPfv20dekQ1TQ= 2023-10-11T15:14:27.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MuR6b/GwxfZqQQgPfv20dekQ1TQ=', 'Content-Length': '0'} 2023-10-11T15:14:27.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6c06dafebc6b1b-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.575 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.576 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.576 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.576 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.577 DEBUG:boto:Data: 2023-10-11T15:14:27.577 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.578 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.578 DEBUG:boto:Params: {} 2023-10-11T15:14:27.579 DEBUG:boto:Token: None 2023-10-11T15:14:27.579 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MuR6b/GwxfZqQQgPfv20dekQ1TQ= 2023-10-11T15:14:27.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MuR6b/GwxfZqQQgPfv20dekQ1TQ=', 'Content-Length': '0'} 2023-10-11T15:14:27.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d760520a8bf2779-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-10-11T15:14:27.583 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.584 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.584 DEBUG:boto:Method: GET 2023-10-11T15:14:27.585 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.585 DEBUG:boto:Data: 2023-10-11T15:14:27.585 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.586 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.587 DEBUG:boto:Params: {} 2023-10-11T15:14:27.587 DEBUG:boto:Token: None 2023-10-11T15:14:27.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D6HFUknHhUqTNKFHpEli3apg90w= 2023-10-11T15:14:27.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6HFUknHhUqTNKFHpEli3apg90w=', 'Content-Length': '0'} 2023-10-11T15:14:27.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000259e66b16e572db1-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.634 DEBUG:boto:path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.635 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.635 DEBUG:boto:Method: GET 2023-10-11T15:14:27.636 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.636 DEBUG:boto:Data: 2023-10-11T15:14:27.637 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.637 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.638 DEBUG:boto:Params: {} 2023-10-11T15:14:27.638 DEBUG:boto:Token: None 2023-10-11T15:14:27.639 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000271 2023-10-11T15:14:27.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D6HFUknHhUqTNKFHpEli3apg90w= 2023-10-11T15:14:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6HFUknHhUqTNKFHpEli3apg90w=', 'Content-Length': '0'} 2023-10-11T15:14:27.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed187ad08feff71a-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.643 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.643 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.643 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.644 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.644 DEBUG:boto:Method: GET 2023-10-11T15:14:27.645 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.645 DEBUG:boto:Data: 2023-10-11T15:14:27.646 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.646 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.647 DEBUG:boto:Params: {} 2023-10-11T15:14:27.647 DEBUG:boto:Token: None 2023-10-11T15:14:27.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb90bf3349c7391-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.652 DEBUG:boto:b'dghqvd-58obj000000000000000002711000falseobj000000000000000002712023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.652 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.653 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.653 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.654 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.654 DEBUG:boto:Method: GET 2023-10-11T15:14:27.655 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000271 2023-10-11T15:14:27.655 DEBUG:boto:Data: 2023-10-11T15:14:27.655 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.656 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.657 DEBUG:boto:Params: {} 2023-10-11T15:14:27.657 DEBUG:boto:Token: None 2023-10-11T15:14:27.658 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fb7139159ef7ebd-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.663 DEBUG:boto:b'dghqvd-58obj000000000000000002711000falseobj000000000000000002712023-10-11T15:14:03.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-11T15:14:27.664 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.664 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.665 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.665 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.666 DEBUG:boto:Method: GET 2023-10-11T15:14:27.666 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.667 DEBUG:boto:Data: 2023-10-11T15:14:27.667 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.668 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.668 DEBUG:boto:Params: {} 2023-10-11T15:14:27.669 DEBUG:boto:Token: None 2023-10-11T15:14:27.669 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I= 2023-10-11T15:14:27.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I=', 'Content-Length': '0'} 2023-10-11T15:14:27.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c926ae05f03ebe8d-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.673 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.674 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.674 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.675 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.675 DEBUG:boto:Method: GET 2023-10-11T15:14:27.676 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.676 DEBUG:boto:Data: 2023-10-11T15:14:27.676 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.677 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.678 DEBUG:boto:Params: {} 2023-10-11T15:14:27.678 DEBUG:boto:Token: None 2023-10-11T15:14:27.679 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I= 2023-10-11T15:14:27.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I=', 'Content-Length': '0'} 2023-10-11T15:14:27.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c748bf98d515c87b-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.683 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.683 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.683 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.684 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.684 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.685 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.685 DEBUG:boto:Data: 2023-10-11T15:14:27.685 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.686 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.687 DEBUG:boto:Params: {} 2023-10-11T15:14:27.687 DEBUG:boto:Token: None 2023-10-11T15:14:27.688 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgJREzgZ8wXnrj7eDPFEBgNtC+M= 2023-10-11T15:14:27.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgJREzgZ8wXnrj7eDPFEBgNtC+M=', 'Content-Length': '0'} 2023-10-11T15:14:27.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea64b7d54b4620c-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.691 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.691 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.692 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.692 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.693 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.693 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.693 DEBUG:boto:Data: 2023-10-11T15:14:27.694 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.695 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.695 DEBUG:boto:Params: {} 2023-10-11T15:14:27.695 DEBUG:boto:Token: None 2023-10-11T15:14:27.696 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgJREzgZ8wXnrj7eDPFEBgNtC+M= 2023-10-11T15:14:27.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgJREzgZ8wXnrj7eDPFEBgNtC+M=', 'Content-Length': '0'} 2023-10-11T15:14:27.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a888fc39d0ad0f2e-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-11T15:14:27.700 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.700 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.701 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.701 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.702 DEBUG:boto:Method: GET 2023-10-11T15:14:27.702 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.702 DEBUG:boto:Data: 2023-10-11T15:14:27.703 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.704 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.704 DEBUG:boto:Params: {} 2023-10-11T15:14:27.705 DEBUG:boto:Token: None 2023-10-11T15:14:27.705 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.706 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I= 2023-10-11T15:14:27.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I=', 'Content-Length': '0'} 2023-10-11T15:14:27.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c463d5a5db4d360-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.750 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.753 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.753 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.754 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.754 DEBUG:boto:Method: GET 2023-10-11T15:14:27.755 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.755 DEBUG:boto:Data: 2023-10-11T15:14:27.755 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.756 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.757 DEBUG:boto:Params: {} 2023-10-11T15:14:27.757 DEBUG:boto:Token: None 2023-10-11T15:14:27.758 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283?versionId=null 2023-10-11T15:14:27.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I= 2023-10-11T15:14:27.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n4F4mKNUbzBOlSKD4IVMwjWIw6I=', 'Content-Length': '0'} 2023-10-11T15:14:27.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977c15a0c8f20a77-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.762 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.762 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.763 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.763 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.764 DEBUG:boto:Method: GET 2023-10-11T15:14:27.764 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.765 DEBUG:boto:Data: 2023-10-11T15:14:27.765 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.766 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.766 DEBUG:boto:Params: {} 2023-10-11T15:14:27.767 DEBUG:boto:Token: None 2023-10-11T15:14:27.767 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad69782b18fd7c7e-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.771 DEBUG:boto:b'dghqvd-58obj000000000000000002831000falseobj000000000000000002832023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.772 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.772 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.773 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.773 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.774 DEBUG:boto:Method: GET 2023-10-11T15:14:27.774 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.775 DEBUG:boto:Data: 2023-10-11T15:14:27.775 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.776 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.777 DEBUG:boto:Params: {} 2023-10-11T15:14:27.777 DEBUG:boto:Token: None 2023-10-11T15:14:27.777 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a493031ccd34a21-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.783 DEBUG:boto:b'dghqvd-58obj000000000000000002831000falseobj000000000000000002832023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.784 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.784 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.785 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.785 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.786 DEBUG:boto:Data: 2023-10-11T15:14:27.786 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.787 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.787 DEBUG:boto:Params: {} 2023-10-11T15:14:27.788 DEBUG:boto:Token: None 2023-10-11T15:14:27.788 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gV7nOCCrBc7aV+F0bQG6wJ90zF4= 2023-10-11T15:14:27.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gV7nOCCrBc7aV+F0bQG6wJ90zF4=', 'Content-Length': '0'} 2023-10-11T15:14:27.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b5a95c262d38815-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.792 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.792 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.792 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.793 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.793 DEBUG:boto:Data: 2023-10-11T15:14:27.794 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.794 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.795 DEBUG:boto:Params: {} 2023-10-11T15:14:27.795 DEBUG:boto:Token: None 2023-10-11T15:14:27.796 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gV7nOCCrBc7aV+F0bQG6wJ90zF4= 2023-10-11T15:14:27.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gV7nOCCrBc7aV+F0bQG6wJ90zF4=', 'Content-Length': '0'} 2023-10-11T15:14:27.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8487550d8fe2d39-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-10-11T15:14:27.800 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.801 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.802 DEBUG:boto:Method: GET 2023-10-11T15:14:27.802 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.803 DEBUG:boto:Data: 2023-10-11T15:14:27.803 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.804 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.804 DEBUG:boto:Params: {} 2023-10-11T15:14:27.805 DEBUG:boto:Token: None 2023-10-11T15:14:27.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ugZyWryndZNYDFyIZiGUYZXtzF8= 2023-10-11T15:14:27.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ugZyWryndZNYDFyIZiGUYZXtzF8=', 'Content-Length': '0'} 2023-10-11T15:14:27.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dbf10b53ec8c9a9-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.850 DEBUG:boto:path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.851 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.851 DEBUG:boto:Method: GET 2023-10-11T15:14:27.852 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.852 DEBUG:boto:Data: 2023-10-11T15:14:27.853 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.854 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.854 DEBUG:boto:Params: {} 2023-10-11T15:14:27.855 DEBUG:boto:Token: None 2023-10-11T15:14:27.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000283 2023-10-11T15:14:27.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ugZyWryndZNYDFyIZiGUYZXtzF8= 2023-10-11T15:14:27.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ugZyWryndZNYDFyIZiGUYZXtzF8=', 'Content-Length': '0'} 2023-10-11T15:14:27.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007764e22822e5dbeb-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.859 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.859 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.860 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.860 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.861 DEBUG:boto:Method: GET 2023-10-11T15:14:27.861 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.861 DEBUG:boto:Data: 2023-10-11T15:14:27.862 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.863 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.863 DEBUG:boto:Params: {} 2023-10-11T15:14:27.864 DEBUG:boto:Token: None 2023-10-11T15:14:27.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96f31e1c0ae80d9-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.868 DEBUG:boto:b'dghqvd-58obj000000000000000002831000falseobj000000000000000002832023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.869 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.870 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.870 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.870 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.871 DEBUG:boto:Method: GET 2023-10-11T15:14:27.871 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000283 2023-10-11T15:14:27.872 DEBUG:boto:Data: 2023-10-11T15:14:27.872 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.873 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.874 DEBUG:boto:Params: {} 2023-10-11T15:14:27.874 DEBUG:boto:Token: None 2023-10-11T15:14:27.875 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904162c72874c208-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.880 DEBUG:boto:b'dghqvd-58obj000000000000000002831000falseobj000000000000000002832023-10-11T15:14:04.111Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-11T15:14:27.881 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.881 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.882 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.882 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.883 DEBUG:boto:Method: GET 2023-10-11T15:14:27.883 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.884 DEBUG:boto:Data: 2023-10-11T15:14:27.884 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.885 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.885 DEBUG:boto:Params: {} 2023-10-11T15:14:27.886 DEBUG:boto:Token: None 2023-10-11T15:14:27.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI= 2023-10-11T15:14:27.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI=', 'Content-Length': '0'} 2023-10-11T15:14:27.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f46e6dc52b9075-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.890 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.891 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.891 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.891 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.892 DEBUG:boto:Method: GET 2023-10-11T15:14:27.892 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.893 DEBUG:boto:Data: 2023-10-11T15:14:27.893 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.893 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.894 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.894 DEBUG:boto:Params: {} 2023-10-11T15:14:27.895 DEBUG:boto:Token: None 2023-10-11T15:14:27.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI= 2023-10-11T15:14:27.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI=', 'Content-Length': '0'} 2023-10-11T15:14:27.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df70e6455c85968-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.899 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.899 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.900 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.900 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.901 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.901 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.901 DEBUG:boto:Data: 2023-10-11T15:14:27.902 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.903 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.903 DEBUG:boto:Params: {} 2023-10-11T15:14:27.903 DEBUG:boto:Token: None 2023-10-11T15:14:27.904 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6VkphDGDYjNomF0kQ3FNQstNZ4M= 2023-10-11T15:14:27.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6VkphDGDYjNomF0kQ3FNQstNZ4M=', 'Content-Length': '0'} 2023-10-11T15:14:27.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4de22b67410b6d-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.907 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.908 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.908 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.908 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.909 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.909 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.909 DEBUG:boto:Data: 2023-10-11T15:14:27.910 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.911 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.911 DEBUG:boto:Params: {} 2023-10-11T15:14:27.912 DEBUG:boto:Token: None 2023-10-11T15:14:27.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6VkphDGDYjNomF0kQ3FNQstNZ4M= 2023-10-11T15:14:27.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6VkphDGDYjNomF0kQ3FNQstNZ4M=', 'Content-Length': '0'} 2023-10-11T15:14:27.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e3e748973b0cf63-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-11T15:14:27.916 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.916 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.917 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.917 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.918 DEBUG:boto:Method: GET 2023-10-11T15:14:27.918 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.918 DEBUG:boto:Data: 2023-10-11T15:14:27.919 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.920 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.920 DEBUG:boto:Params: {} 2023-10-11T15:14:27.921 DEBUG:boto:Token: None 2023-10-11T15:14:27.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI= 2023-10-11T15:14:27.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI=', 'Content-Length': '0'} 2023-10-11T15:14:27.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003825ced513c58575-006526bbd3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.966 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.967 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.967 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.968 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.968 DEBUG:boto:Method: GET 2023-10-11T15:14:27.969 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.969 DEBUG:boto:Data: 2023-10-11T15:14:27.970 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.970 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.971 DEBUG:boto:Params: {} 2023-10-11T15:14:27.971 DEBUG:boto:Token: None 2023-10-11T15:14:27.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/obj00000000000000000317?versionId=null 2023-10-11T15:14:27.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI= 2023-10-11T15:14:27.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smF9LcksJXrFqC2igA5rUedzfnI=', 'Content-Length': '0'} 2023-10-11T15:14:27.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b940a075a19468-006526bbd3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.976 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.976 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.977 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.977 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.978 DEBUG:boto:Method: GET 2023-10-11T15:14:27.978 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.978 DEBUG:boto:Data: 2023-10-11T15:14:27.979 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:27.980 DEBUG:boto:Port: 8001 2023-10-11T15:14:27.980 DEBUG:boto:Params: {} 2023-10-11T15:14:27.981 DEBUG:boto:Token: None 2023-10-11T15:14:27.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47d6bb8ba6342be-006526bbd3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.985 DEBUG:boto:b'dghqvd-58obj000000000000000003171000falseobj000000000000000003172023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.986 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:27.987 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:27.987 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.988 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.988 DEBUG:boto:Method: GET 2023-10-11T15:14:27.988 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:27.989 DEBUG:boto:Data: 2023-10-11T15:14:27.989 DEBUG:boto:Headers: {} 2023-10-11T15:14:27.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:27.990 DEBUG:boto:Port: 8000 2023-10-11T15:14:27.991 DEBUG:boto:Params: {} 2023-10-11T15:14:27.991 DEBUG:boto:Token: None 2023-10-11T15:14:27.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:27 GMT /dghqvd-58/ 2023-10-11T15:14:27.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8= 2023-10-11T15:14:27.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+HHVSfyPDjNcNwWXS+bg00wE9k8=', 'Content-Length': '0'} 2023-10-11T15:14:27.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000766e7605c254f175-006526bbd3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:27.997 DEBUG:boto:b'dghqvd-58obj000000000000000003171000falseobj000000000000000003172023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:27.998 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.998 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:27.999 DEBUG:boto:Method: HEAD 2023-10-11T15:14:27.999 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.000 DEBUG:boto:Data: 2023-10-11T15:14:28.000 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.001 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.001 DEBUG:boto:Params: {} 2023-10-11T15:14:28.002 DEBUG:boto:Token: None 2023-10-11T15:14:28.002 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.003 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8hrWZirPFPJUvSzzeXuekC/md0g= 2023-10-11T15:14:28.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8hrWZirPFPJUvSzzeXuekC/md0g=', 'Content-Length': '0'} 2023-10-11T15:14:28.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2483a695ec85797-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.006 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.007 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.007 DEBUG:boto:Method: HEAD 2023-10-11T15:14:28.008 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.008 DEBUG:boto:Data: 2023-10-11T15:14:28.008 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.009 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.010 DEBUG:boto:Params: {} 2023-10-11T15:14:28.010 DEBUG:boto:Token: None 2023-10-11T15:14:28.010 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8hrWZirPFPJUvSzzeXuekC/md0g= 2023-10-11T15:14:28.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8hrWZirPFPJUvSzzeXuekC/md0g=', 'Content-Length': '0'} 2023-10-11T15:14:28.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bcb60befb81dc2-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-10-11T15:14:28.014 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.015 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.015 DEBUG:boto:Method: GET 2023-10-11T15:14:28.016 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.016 DEBUG:boto:Data: 2023-10-11T15:14:28.017 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.017 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.018 DEBUG:boto:Params: {} 2023-10-11T15:14:28.018 DEBUG:boto:Token: None 2023-10-11T15:14:28.019 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4gcAybg7SOYdRvw4Zb/5pNQnMKQ= 2023-10-11T15:14:28.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4gcAybg7SOYdRvw4Zb/5pNQnMKQ=', 'Content-Length': '0'} 2023-10-11T15:14:28.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d715803d2cd9f763-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.023 DEBUG:boto:path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.023 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.024 DEBUG:boto:Method: GET 2023-10-11T15:14:28.024 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.024 DEBUG:boto:Data: 2023-10-11T15:14:28.025 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.026 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.026 DEBUG:boto:Params: {} 2023-10-11T15:14:28.027 DEBUG:boto:Token: None 2023-10-11T15:14:28.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000317 2023-10-11T15:14:28.028 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4gcAybg7SOYdRvw4Zb/5pNQnMKQ= 2023-10-11T15:14:28.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4gcAybg7SOYdRvw4Zb/5pNQnMKQ=', 'Content-Length': '0'} 2023-10-11T15:14:28.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063835066761b5f72-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.031 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.031 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.032 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.032 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.033 DEBUG:boto:Method: GET 2023-10-11T15:14:28.033 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.033 DEBUG:boto:Data: 2023-10-11T15:14:28.034 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.035 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.035 DEBUG:boto:Params: {} 2023-10-11T15:14:28.036 DEBUG:boto:Token: None 2023-10-11T15:14:28.036 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.037 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003801ed71ad10b5b3-006526bbd4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.040 DEBUG:boto:b'dghqvd-58obj000000000000000003171000falseobj000000000000000003172023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.041 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.042 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.042 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.042 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.043 DEBUG:boto:Method: GET 2023-10-11T15:14:28.043 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000317 2023-10-11T15:14:28.044 DEBUG:boto:Data: 2023-10-11T15:14:28.044 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.045 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.045 DEBUG:boto:Params: {} 2023-10-11T15:14:28.046 DEBUG:boto:Token: None 2023-10-11T15:14:28.046 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.047 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ec413d6dad5746a-006526bbd4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.098 DEBUG:boto:b'dghqvd-58obj000000000000000003171000falseobj000000000000000003172023-10-11T15:14:04.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-11T15:14:28.100 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.100 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.101 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.101 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.101 DEBUG:boto:Method: GET 2023-10-11T15:14:28.102 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.102 DEBUG:boto:Data: 2023-10-11T15:14:28.103 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.103 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.104 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.104 DEBUG:boto:Params: {} 2023-10-11T15:14:28.104 DEBUG:boto:Token: None 2023-10-11T15:14:28.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis= 2023-10-11T15:14:28.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis=', 'Content-Length': '0'} 2023-10-11T15:14:28.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f9f7b69c045257-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.109 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.109 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.110 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.110 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.111 DEBUG:boto:Method: GET 2023-10-11T15:14:28.111 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.111 DEBUG:boto:Data: 2023-10-11T15:14:28.112 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.113 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.113 DEBUG:boto:Params: {} 2023-10-11T15:14:28.114 DEBUG:boto:Token: None 2023-10-11T15:14:28.114 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.115 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis= 2023-10-11T15:14:28.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis=', 'Content-Length': '0'} 2023-10-11T15:14:28.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000225786b5373c35d7-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.118 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.118 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.119 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.119 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.120 DEBUG:boto:Method: HEAD 2023-10-11T15:14:28.120 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.120 DEBUG:boto:Data: 2023-10-11T15:14:28.121 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.122 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.122 DEBUG:boto:Params: {} 2023-10-11T15:14:28.122 DEBUG:boto:Token: None 2023-10-11T15:14:28.123 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zARU9rBPZTAkMdKrRc6TNT1Crj4= 2023-10-11T15:14:28.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zARU9rBPZTAkMdKrRc6TNT1Crj4=', 'Content-Length': '0'} 2023-10-11T15:14:28.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c0a291c8e2174d-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.127 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.127 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.128 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.128 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.128 DEBUG:boto:Method: HEAD 2023-10-11T15:14:28.129 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.129 DEBUG:boto:Data: 2023-10-11T15:14:28.130 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.130 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.131 DEBUG:boto:Params: {} 2023-10-11T15:14:28.131 DEBUG:boto:Token: None 2023-10-11T15:14:28.132 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zARU9rBPZTAkMdKrRc6TNT1Crj4= 2023-10-11T15:14:28.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zARU9rBPZTAkMdKrRc6TNT1Crj4=', 'Content-Length': '0'} 2023-10-11T15:14:28.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e109e8fb217da1-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-11T15:14:28.136 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.137 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.137 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.137 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.138 DEBUG:boto:Method: GET 2023-10-11T15:14:28.138 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.139 DEBUG:boto:Data: 2023-10-11T15:14:28.139 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.140 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.140 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.140 DEBUG:boto:Params: {} 2023-10-11T15:14:28.141 DEBUG:boto:Token: None 2023-10-11T15:14:28.141 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.142 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis= 2023-10-11T15:14:28.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis=', 'Content-Length': '0'} 2023-10-11T15:14:28.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f57e4ea09c5fe5-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.145 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.146 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.146 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.146 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.147 DEBUG:boto:Method: GET 2023-10-11T15:14:28.147 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.148 DEBUG:boto:Data: 2023-10-11T15:14:28.148 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.149 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.149 DEBUG:boto:Params: {} 2023-10-11T15:14:28.150 DEBUG:boto:Token: None 2023-10-11T15:14:28.150 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423?versionId=null 2023-10-11T15:14:28.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis= 2023-10-11T15:14:28.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Uh2I44bkyN5iVbQeKUQ8TvfzDis=', 'Content-Length': '0'} 2023-10-11T15:14:28.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61b533e7f82c9bb-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.154 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.155 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.155 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.155 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.156 DEBUG:boto:Method: GET 2023-10-11T15:14:28.156 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.157 DEBUG:boto:Data: 2023-10-11T15:14:28.157 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.157 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.158 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.158 DEBUG:boto:Params: {} 2023-10-11T15:14:28.161 DEBUG:boto:Token: None 2023-10-11T15:14:28.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000058098e36568a1e8-006526bbd4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.166 DEBUG:boto:b'dghqvd-58obj000000000000000004231000falseobj000000000000000004232023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.167 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.168 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.168 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.168 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.169 DEBUG:boto:Method: GET 2023-10-11T15:14:28.169 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.170 DEBUG:boto:Data: 2023-10-11T15:14:28.170 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.171 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.171 DEBUG:boto:Params: {} 2023-10-11T15:14:28.172 DEBUG:boto:Token: None 2023-10-11T15:14:28.172 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.173 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0b2d0b41480390f-006526bbd4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.178 DEBUG:boto:b'dghqvd-58obj000000000000000004231000falseobj000000000000000004232023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.179 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.179 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.179 DEBUG:boto:Method: HEAD 2023-10-11T15:14:28.180 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.180 DEBUG:boto:Data: 2023-10-11T15:14:28.181 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.181 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.182 DEBUG:boto:Params: {} 2023-10-11T15:14:28.182 DEBUG:boto:Token: None 2023-10-11T15:14:28.183 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.183 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fh+G1FNdqcKPd9rJy9XVkE20DUA= 2023-10-11T15:14:28.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fh+G1FNdqcKPd9rJy9XVkE20DUA=', 'Content-Length': '0'} 2023-10-11T15:14:28.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae00714886238b3-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.187 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.187 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.187 DEBUG:boto:Method: HEAD 2023-10-11T15:14:28.188 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.188 DEBUG:boto:Data: 2023-10-11T15:14:28.189 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.189 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.190 DEBUG:boto:Params: {} 2023-10-11T15:14:28.190 DEBUG:boto:Token: None 2023-10-11T15:14:28.191 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fh+G1FNdqcKPd9rJy9XVkE20DUA= 2023-10-11T15:14:28.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fh+G1FNdqcKPd9rJy9XVkE20DUA=', 'Content-Length': '0'} 2023-10-11T15:14:28.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000810e203608934187-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-10-11T15:14:28.195 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.195 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.196 DEBUG:boto:Method: GET 2023-10-11T15:14:28.196 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.196 DEBUG:boto:Data: 2023-10-11T15:14:28.197 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.198 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.198 DEBUG:boto:Params: {} 2023-10-11T15:14:28.199 DEBUG:boto:Token: None 2023-10-11T15:14:28.199 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2chik9GXZeEuiAauhiXGygNkmoA= 2023-10-11T15:14:28.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2chik9GXZeEuiAauhiXGygNkmoA=', 'Content-Length': '0'} 2023-10-11T15:14:28.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-58:daa316e9-869c-4074-a756-a8adfb4a3573.5751.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083b974a4fa4e795a-006526bbd4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.203 DEBUG:boto:path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.203 DEBUG:boto:auth_path=/dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.204 DEBUG:boto:Method: GET 2023-10-11T15:14:28.204 DEBUG:boto:Path: /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.205 DEBUG:boto:Data: 2023-10-11T15:14:28.205 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.205 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.206 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.206 DEBUG:boto:Params: {} 2023-10-11T15:14:28.207 DEBUG:boto:Token: None 2023-10-11T15:14:28.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/obj00000000000000000423 2023-10-11T15:14:28.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2chik9GXZeEuiAauhiXGygNkmoA= 2023-10-11T15:14:28.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2chik9GXZeEuiAauhiXGygNkmoA=', 'Content-Length': '0'} 2023-10-11T15:14:28.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:14:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a016146b94a052-006526bbd4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.254 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.255 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.255 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.256 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.256 DEBUG:boto:Method: GET 2023-10-11T15:14:28.257 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.257 DEBUG:boto:Data: 2023-10-11T15:14:28.258 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.258 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:14:28.258 DEBUG:boto:Port: 8001 2023-10-11T15:14:28.259 DEBUG:boto:Params: {} 2023-10-11T15:14:28.259 DEBUG:boto:Token: None 2023-10-11T15:14:28.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b0e5d2a2bf8754-006526bbd4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.264 DEBUG:boto:b'dghqvd-58obj000000000000000004231000falseobj000000000000000004232023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.265 DEBUG:boto:path=/dghqvd-58/ 2023-10-11T15:14:28.265 DEBUG:boto:auth_path=/dghqvd-58/ 2023-10-11T15:14:28.266 DEBUG:boto:path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.266 DEBUG:boto:auth_path=/dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.266 DEBUG:boto:Method: GET 2023-10-11T15:14:28.267 DEBUG:boto:Path: /dghqvd-58/?prefix=obj00000000000000000423 2023-10-11T15:14:28.267 DEBUG:boto:Data: 2023-10-11T15:14:28.268 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.268 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.269 DEBUG:boto:Params: {} 2023-10-11T15:14:28.269 DEBUG:boto:Token: None 2023-10-11T15:14:28.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-58/ 2023-10-11T15:14:28.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms= 2023-10-11T15:14:28.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U20WwWjIr41zq1MQXFH++8kLxms=', 'Content-Length': '0'} 2023-10-11T15:14:28.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77ffb3d8892e848-006526bbd4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.275 DEBUG:boto:b'dghqvd-58obj000000000000000004231000falseobj000000000000000004232023-10-11T15:14:04.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:14:28.276 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-58 zones={a2, a1} 2023-10-11T15:14:28.277 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-10-11T15:14:28.280 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:14:28.280 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:14:28.280 DEBUG:boto:Method: PUT 2023-10-11T15:14:28.281 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:14:28.281 DEBUG:boto:Data: 2023-10-11T15:14:28.282 DEBUG:boto:Headers: {} 2023-10-11T15:14:28.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:14:28.282 DEBUG:boto:Port: 8000 2023-10-11T15:14:28.283 DEBUG:boto:Params: {} 2023-10-11T15:14:28.283 DEBUG:boto:Token: None 2023-10-11T15:14:28.284 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:14:28 GMT /dghqvd-59/ 2023-10-11T15:14:28.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MJylSMbsChoBKdbEJ/JazMK+eds= 2023-10-11T15:14:28.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:14:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MJylSMbsChoBKdbEJ/JazMK+eds=', 'Content-Length': '0'} 2023-10-11T15:14:28.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e82a945ce63e3a29-006526bbd4-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:14:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:14:28.303 DEBUG:rgw_multi.tests:created bucket=dghqvd-59 2023-10-11T15:14:28.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:28.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:28.304 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:28.358 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:28.358 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:28.390 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:14:28.403 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:14:28.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.404 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:28.404 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:14:28.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.405 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:28.405 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:14:28.405 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.405 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.406 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:28.406 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T15:14:28.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.406 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.406 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037268.300154_465.1", 2023-10-11T15:14:28.407 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:14:28.300154Z" 2023-10-11T15:14:28.407 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.407 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:14:28.408 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:28.409 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:14:28.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:28.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:28.410 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:28.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:28.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:28.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:28.489+0000 7fa2d3453780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:28.491 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:28.490+0000 7fa2d3453780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:28.504 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:28.504 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:28.504 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:28.504 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:28.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.506 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:28.507 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:28.507 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:28.507 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:28.508 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:28.508 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:28.508 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:28.508 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:28.508 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:28.509 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.509 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:28.510 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:28.510 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:28.510 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:28.510 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:28.510 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:28.511 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:28.511 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:28.511 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:28.512 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.512 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.512 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.512 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:28.512 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:28.513 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:28.514 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:28.514 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:28.514 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:28.514 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:28.514 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:28.515 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:28.515 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:28.516 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:28.516 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:28.516 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:28.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:28.516 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:28.517 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:28.517 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:28.517 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:28.517 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:28.517 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:28.518 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:28.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.518 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:28.518 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:28.519 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:28.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:28.519 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:28Z" 2023-10-11T15:14:28.519 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:28.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:28Z" } 2023-10-11T15:14:28.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:28.521 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:28.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:28.522 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:33.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:33.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:33.525 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:33.582 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:33.583 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:33.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:33.614+0000 7f5e929dc780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:33.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:33.615+0000 7f5e929dc780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:33.629 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:33.630 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:33.630 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:33.630 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:33.630 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:33.631 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:33.631 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:33.631 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.631 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:33.632 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:33.632 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:33.632 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:33.632 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:33.633 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:33.633 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:33.633 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:33.633 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:33.633 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:33.634 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:33.634 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:33.634 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.634 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:33.634 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:33.635 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:33.635 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:33.635 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:33.635 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:33.636 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:33.636 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:33.636 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:33.636 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:33.636 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:33.637 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.637 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:33.637 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:33.637 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:33.637 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:33.638 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:33.638 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:33.638 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:33.638 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:33.639 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:33.639 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:33.639 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:33.639 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.639 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:33.640 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:33.640 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:33.640 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:33.640 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:33.640 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:33.641 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:33.641 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:33.641 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:33.641 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:33.641 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:33.642 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:33.642 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:33.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.642 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:33.643 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:33.643 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:33.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:33.643 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:33Z" 2023-10-11T15:14:33.643 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:33.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:33Z" } 2023-10-11T15:14:33.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:33.645 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:33.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:33.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:38.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:38.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:38.648 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:38.707 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:38.707 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:38.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:38.737+0000 7f827af16780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:38.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:38.738+0000 7f827af16780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:38.753 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:38.753 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:38.754 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:38.754 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:38.754 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:38.754 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:38.755 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:38.755 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.755 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:38.755 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:38.756 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:38.756 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:38.756 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:38.756 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:38.756 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:38.757 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:38.757 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:38.757 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:38.757 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:38.757 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:38.758 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.758 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:38.758 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:38.758 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:38.758 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:38.759 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:38.759 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:38.759 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:38.759 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:38.760 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:38.760 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:38.760 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:38.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.760 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:38.761 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:38.761 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:38.761 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:38.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:38.761 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:38.762 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:38.762 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:38.762 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:38.762 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:38.763 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:38.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.763 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:38.763 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:38.763 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:38.764 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:38.764 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:38.764 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:38.764 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:38.765 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:38.765 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:38.765 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:38.765 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:38.765 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:38.766 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:38.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.766 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:38.766 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:38.766 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:38.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:38.767 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:38Z" 2023-10-11T15:14:38.767 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:38.768 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:38Z" } 2023-10-11T15:14:38.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:38.769 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:38.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:38.770 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:43.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:43.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:43.772 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:43.829 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:43.830 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:43.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:43.859+0000 7f0d2ea35780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:43.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:43.860+0000 7f0d2ea35780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:43.875 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:43.875 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:43.876 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:43.876 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:43.876 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:43.876 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:43.876 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:43.877 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.877 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:43.877 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:43.877 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:43.878 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:43.878 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:43.878 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:43.878 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:43.878 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:43.879 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:43.879 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:43.879 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:43.879 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:43.880 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.880 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:43.880 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:43.880 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:43.880 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:43.881 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:43.881 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:43.881 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:43.881 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:43.881 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:43.882 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:43.882 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:43.882 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.882 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:43.883 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:43.883 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:43.883 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:43.883 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:43.883 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:43.884 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:43.884 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:43.884 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:43.884 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:43.884 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:43.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.885 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:43.885 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:43.885 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:43.885 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:43.886 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:43.886 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:43.886 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:43.886 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:43.886 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:43.887 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:43.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:43.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:43.887 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:43.887 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.888 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:43.888 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:43.888 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:43.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:43.888 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:43Z" 2023-10-11T15:14:43.889 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:43.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:43Z" } 2023-10-11T15:14:43.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:43.891 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:43.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:43.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:48.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:48.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:48.894 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:48.951 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:48.952 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:48.983 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:48.982+0000 7f6da1ec2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:48.984 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:48.983+0000 7f6da1ec2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:48.997 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:48.997 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:48.998 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:48.998 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:48.998 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:48.998 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:48.999 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:48.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:48.999 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:48.999 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:48.999 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:49.000 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:49.000 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:49.000 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:49.001 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:49.001 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:49.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:49.002 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:49.002 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:49.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:49.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:49.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:49.003 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:49.003 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:49.003 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:49.004 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:49.005 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:49.005 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:49.005 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:49.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:49.005 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:49.006 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:49.006 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:49.006 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:49.006 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:49.007 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:49.007 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:49.007 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:49.007 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:49.007 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:49.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:49.008 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:49.008 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:49.008 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:49.008 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:49.009 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:49.009 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:49.009 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:49.009 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:49.009 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:49.010 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:49.010 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:49.010 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:49.010 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:49.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:49.011 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:49.011 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:49.012 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:49.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:49.012 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:48Z" 2023-10-11T15:14:49.013 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:49.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:48Z" } 2023-10-11T15:14:49.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:49.015 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:49.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:49.016 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:54.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:54.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:54.018 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:54.076 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:54.077 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:54.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:54.107+0000 7f567d625780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:54.109 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:54.108+0000 7f567d625780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:54.123 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:54.124 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:54.124 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:54.125 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:54.125 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:54.125 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:54.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:54.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.126 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:54.127 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:54.127 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:54.128 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:54.128 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:54.128 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:54.129 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:54.129 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:54.129 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:54.130 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:54.130 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:54.131 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:54.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.131 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:54.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:54.132 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:54.133 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:54.133 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:54.133 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:54.133 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:54.134 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:54.134 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:54.134 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:54.134 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:54.135 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.135 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:54.135 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:54.135 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:54.136 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:54.136 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:54.136 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:54.136 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:54.137 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:54.137 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:54.137 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:54.137 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:54.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.138 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:54.138 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:54.138 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:54.138 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:54.139 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:54.139 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:54.139 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:54.139 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:54.139 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:54.140 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:54.140 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:54.140 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:54.140 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:54.140 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.141 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:54.141 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:54.141 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:54.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:54.142 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:54Z" 2023-10-11T15:14:54.142 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:54.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:54Z" } 2023-10-11T15:14:54.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:54.146 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:54.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:54.147 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:14:59.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:59.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:14:59.151 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:14:59.207 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:14:59.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:14:59.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:59.240+0000 7fb4ec83a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:59.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:14:59.242+0000 7fb4ec83a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:14:59.256 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:14:59.256 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:14:59.257 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:14:59.257 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:14:59.257 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:14:59.257 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:14:59.257 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:59.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.258 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:14:59.258 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:59.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:14:59.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:59.260 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:14:59.260 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:59.260 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:59.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:59.261 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:14:59.261 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:59.261 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:59.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:14:59.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:59.262 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:59.262 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:59.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:14:59.262 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:59.263 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:59.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:59.263 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:14:59.263 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:59.264 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:59.264 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:14:59.264 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:59.264 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:59.265 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:59.265 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:14:59.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:59.265 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:59.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.266 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:14:59.266 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:14:59.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:14:59.266 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:14:59.267 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:14:59.267 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:14:59.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:14:59.267 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:14:59.267 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:14:59.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:14:59.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:59.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:14:59.268 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:14:59.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.269 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:14:59.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:14:59.269 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:14:59.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:14:59.270 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:14:59Z" 2023-10-11T15:14:59.270 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:14:59.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:14:59Z" } 2023-10-11T15:14:59.272 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:14:59.272 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:14:59.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:14:59.273 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:04.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:04.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:04.276 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:04.336 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:04.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:04.366+0000 7fc0ab6a4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:04.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:04.367+0000 7fc0ab6a4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:04.382 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:04.383 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:04.383 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:04.383 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:04.384 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:04.384 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:04.384 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:04.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.385 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:04.385 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:04.386 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:04.386 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:04.387 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:04.387 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:04.387 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:04.388 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:04.388 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:04.388 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:04.389 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:04.389 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:04.389 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.389 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:04.389 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:04.390 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:04.390 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:04.390 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:04.391 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:04.391 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:04.391 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:04.391 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:04.391 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:04.392 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:04.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.392 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:04.392 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:04.392 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:04.393 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:04.393 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:04.393 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:04.393 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:04.394 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:04.394 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:04.394 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:04.394 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:04.394 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.395 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:04.395 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:04.395 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:04.395 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:04.395 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:04.396 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:04.396 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:04.396 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:04.396 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:04.396 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:04.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:04.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:04.397 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:04.397 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.398 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:04.398 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:04.398 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:04.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:04.399 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:04Z" 2023-10-11T15:15:04.399 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:04.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:04Z" } 2023-10-11T15:15:04.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:04.401 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:04.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:04.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:09.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:09.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:09.405 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:09.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:09.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:09.494 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:09.493+0000 7fa99888f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:09.495 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:09.494+0000 7fa99888f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:09.509 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:09.509 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:09.509 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:09.509 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:09.510 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:09.510 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:09.510 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:09.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.511 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:09.511 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:09.511 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:09.511 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:09.511 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:09.512 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:09.512 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:09.512 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:09.512 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.513 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:09.514 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:09.514 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:09.514 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:09.514 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:09.515 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:09.515 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:09.515 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:09.515 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:09.515 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:09.516 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:09.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.516 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:09.516 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:09.516 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:09.517 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:09.517 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:09.517 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:09.517 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:09.518 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:09.518 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:09.518 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:09.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:09.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:09.519 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:09.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:09.519 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:09.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:09.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:09.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:09.520 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:09.520 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:09.520 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:09.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:09.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:09.521 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:09.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.521 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:09.522 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:09.522 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:09.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:09.522 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:09Z" 2023-10-11T15:15:09.523 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:09.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:09Z" } 2023-10-11T15:15:09.524 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:09.525 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:09.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:09.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:14.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:14.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:14.529 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:14.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:14.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:14.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:14.617+0000 7f9a1c778780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:14.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:14.618+0000 7f9a1c778780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:14.633 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:14.634 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:14.634 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:14.634 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:14.634 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:14.634 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:14.635 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:14.635 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.635 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:14.635 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:14.636 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:14.636 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:14.636 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:14.636 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:14.636 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:14.637 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:14.637 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:14.637 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:14.637 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:14.637 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:14.638 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.638 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:14.638 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:14.638 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:14.639 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:14.639 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:14.639 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:14.639 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:14.639 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:14.640 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:14.640 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:14.640 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:14.640 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:14.641 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:14.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:14.641 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:14.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:14.641 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:14.642 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:14.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:14.642 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:14.642 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:14.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:14.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.643 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:14.643 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:14.643 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:14.644 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:14.644 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:14.644 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:14.644 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:14.644 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:14.645 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:14.645 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:14.645 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:14.645 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:14.645 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:14.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.646 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:14.646 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:14.646 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:14.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:14.647 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:14Z" 2023-10-11T15:15:14.647 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:14.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:14Z" } 2023-10-11T15:15:14.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:14.649 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:14.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:14.650 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:19.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:19.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:19.652 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:19.709 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:19.709 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:19.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:19.738+0000 7fe121432780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:19.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:19.739+0000 7fe121432780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:19.755 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:19.755 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:19.756 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:19.757 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:19.758 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:19.759 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:19.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:19.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:19.762 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.763 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:19.764 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:19.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:19.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:19.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:19.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:19.768 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:19.768 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:19.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:19.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:19.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:19.769 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:19.769 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:19.769 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:19.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:19.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:19.770 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:19.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.770 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:19.771 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:19.771 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:19.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:19.771 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:19Z" 2023-10-11T15:15:19.772 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:19.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:19Z" } 2023-10-11T15:15:19.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:19.774 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:19.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:19.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:24.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:24.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:24.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:24.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:24.868 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:24.865+0000 7f7b69fc7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:24.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:24.866+0000 7f7b69fc7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:24.882 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:24.883 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:24.883 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:24.883 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:24.883 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:24.884 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:24.884 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:24.884 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.884 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:24.884 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:24.885 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:24.885 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:24.885 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:24.885 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:24.885 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:24.886 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:24.886 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:24.886 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:24.886 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:24.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:24.887 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.887 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:24.887 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:24.887 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:24.888 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:24.888 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:24.888 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:24.888 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:24.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:24.889 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:24.889 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:24.889 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:24.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:24.890 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:24.890 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:24.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:24.891 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:24.891 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:24.891 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:24.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:24.892 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:24.892 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:24.892 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:24.892 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.893 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:24.893 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:24.893 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:24.893 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:24.894 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:24.894 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:24.894 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:24.894 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:24.895 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:24.895 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:24.895 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:24.895 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:24.896 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:24.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.896 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:24.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:24.897 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:24.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:24.897 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:24Z" 2023-10-11T15:15:24.897 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:24.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:24Z" } 2023-10-11T15:15:24.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:24.900 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:24.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:24.901 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:29.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:29.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:29.904 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:29.961 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:29.961 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:29.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:29.990+0000 7f4c327c9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:29.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:29.991+0000 7f4c327c9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:30.007 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:30.007 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:30.007 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:30.008 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:30.008 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:30.008 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:30.008 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:30.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.009 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:30.009 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:30.009 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:30.009 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:30.010 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:30.010 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:30.010 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:30.010 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:30.010 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:30.011 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:30.011 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:30.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:30.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.011 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:30.012 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:30.012 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:30.012 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:30.012 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:30.012 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:30.013 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:30.013 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:30.013 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:30.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:30.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:30.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:30.014 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:30.014 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:30.015 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:30.015 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:30.015 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:30.015 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:30.015 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:30.016 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:30.016 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:30.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:30.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:30.017 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:30.017 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:30.017 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:30.017 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:30.017 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:30.018 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:30.018 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:30.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:30.018 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:30.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:30.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:30.019 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:30.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.019 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:30.020 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:30.020 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:30.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:30.020 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:29Z" 2023-10-11T15:15:30.020 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:30.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:29Z" } 2023-10-11T15:15:30.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:30.022 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:30.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:30.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:35.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:35.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:35.026 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:35.083 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:35.084 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:35.115 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:35.112+0000 7fc7aaef9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:35.116 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:35.114+0000 7fc7aaef9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:35.129 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:35.130 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:35.130 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:35.130 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.131 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:35.132 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:35.133 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:35.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:35.135 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:35.136 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:35.136 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:35.136 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:35.136 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:35.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.137 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:35.137 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:35.137 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:35.137 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:35.138 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:35.138 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:35.138 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:35.138 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:35.138 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:35.139 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:35.139 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:35.139 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.139 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:35.140 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:35.140 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:35.140 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:35.140 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:35.141 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:35.141 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:35.141 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:35.141 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:35.141 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:35.142 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:35.142 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:35.142 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:35.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.142 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:35.143 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:35.143 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:35.143 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:35.143 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:35Z" 2023-10-11T15:15:35.143 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:35.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:35Z" } 2023-10-11T15:15:35.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:35.146 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:35.146 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:35.146 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:40.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:40.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:40.149 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:40.202 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:40.202 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:40.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:40.230+0000 7fc88a8ea780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:40.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:40.231+0000 7fc88a8ea780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:40.246 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:40.247 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:40.247 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:40.247 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:40.248 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:40.248 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:40.248 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:40.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.248 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:40.249 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:40.249 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:40.249 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:40.249 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:40.249 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:40.250 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:40.250 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:40.250 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:40.250 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:40.251 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:40.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:40.251 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.251 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:40.251 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:40.252 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:40.252 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:40.252 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:40.252 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:40.253 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:40.253 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:40.253 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:40.253 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:40.253 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:40.254 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.254 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:40.254 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:40.254 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:40.255 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:40.255 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:40.255 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:40.255 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:40.256 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:40.256 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:40.256 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:40.256 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:40.256 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:40.257 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:40.257 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:40.257 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:40.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:40.258 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:40.258 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:40.258 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:40.258 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:40.259 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:40.259 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:40.259 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:40.259 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:40.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.260 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:40.260 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:40.260 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:40.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:40.261 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:40Z" 2023-10-11T15:15:40.261 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:40.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:40Z" } 2023-10-11T15:15:40.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:40.263 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:40.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:40.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:45.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:45.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:45.266 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:45.323 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:45.324 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:45.356 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:45.354+0000 7f3059e54780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:45.357 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:45.355+0000 7f3059e54780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:45.370 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:45.371 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:45.371 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:45.371 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:45.371 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:45.372 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:45.372 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:45.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.372 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:45.373 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:45.373 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:45.373 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:45.373 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:45.373 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:45.374 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:45.374 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:45.374 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:45.374 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:45.374 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:45.375 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:45.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:45.375 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:45.375 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:45.376 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:45.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:45.376 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:45.376 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:45.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:45.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:45.377 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:45.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:45.377 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:45.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:45.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:45.378 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:45.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:45.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:45.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:45.379 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:45.379 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:45.380 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:45.380 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:45.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.380 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:45.380 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:45.381 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:45.381 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:45.381 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:45.381 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:45.381 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:45.382 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:45.382 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:45.382 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:45.382 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:45.382 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:45.383 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:45.384 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:45.384 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:45Z" 2023-10-11T15:15:45.384 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:45.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:45Z" } 2023-10-11T15:15:45.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:45.386 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:45.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:45.387 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:50.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:50.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:50.390 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:50.448 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:50.449 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:50.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:50.478+0000 7f6ea02c6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:50.481 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:50.479+0000 7f6ea02c6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:50.495 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:50.495 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:50.495 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:50.496 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:50.496 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:50.496 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:50.497 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:50.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.497 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:50.498 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:50.498 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:50.499 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:50.499 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:50.499 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:50.500 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:50.500 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:50.500 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:50.501 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:50.501 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:50.501 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:50.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.502 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:50.502 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:50.502 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:50.503 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:50.503 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:50.503 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:50.504 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:50.504 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:50.504 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:50.504 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:50.504 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:50.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:50.505 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:50.505 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:50.506 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:50.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:50.506 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:50.506 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:50.507 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:50.507 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:50.507 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:50.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:50.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:50.508 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:50.508 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:50.508 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:50.508 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:50.509 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:50.509 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:50.509 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:50.509 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:50.510 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:50.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:50.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:50.510 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:50.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.511 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:50.511 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:50.511 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:50.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:50.511 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:50Z" 2023-10-11T15:15:50.512 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:50.513 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:50Z" } 2023-10-11T15:15:50.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:50.514 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:50.514 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:50.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:15:55.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:55.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:15:55.517 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:15:55.577 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:15:55.577 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:15:55.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:55.606+0000 7f0c4bc1c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:55.609 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:15:55.607+0000 7f0c4bc1c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:15:55.623 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:15:55.623 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:15:55.623 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:15:55.624 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:15:55.624 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:15:55.624 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:15:55.624 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:55.624 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.625 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:15:55.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:55.625 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:15:55.625 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:55.625 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:55.626 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:15:55.626 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:55.626 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:15:55.626 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:55.627 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:15:55.627 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:55.627 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:55.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.627 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:55.628 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:15:55.628 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:55.628 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:55.628 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:15:55.629 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:55.629 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:55.629 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:55.629 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:15:55.630 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:55.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:55.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.630 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:55.630 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:15:55.631 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:55.631 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:55.631 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:15:55.631 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:55.631 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.632 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:15:55.633 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:15:55.633 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:15:55.633 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:15:55.633 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037214.528226_437.1", 2023-10-11T15:15:55.634 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:15:55.634 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:15:55.634 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:15:55.634 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:34.528226Z", 2023-10-11T15:15:55.634 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:15:55.635 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:55.635 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:15:55.635 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:15:55.635 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.636 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:15:55.636 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:15:55.636 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:15:55.636 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:15:55.637 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:15:55Z" 2023-10-11T15:15:55.637 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:15:55.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037214.528226_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:34.528226Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:15:55Z" } 2023-10-11T15:15:55.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037214.528226_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:34.528226Z', 'realm_epoch': 2}}] 2023-10-11T15:15:55.639 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:15:55.639 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037214.528226_437.1'} 2023-10-11T15:15:55.640 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037214.528226_437.1 2023-10-11T15:16:00.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:16:00.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:16:00.642 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:16:00.707 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:00.708 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:00.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:00.737+0000 7f59745bd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:16:00.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:00.739+0000 7f59745bd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:16:00.753 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:16:00.753 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:16:00.753 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:16:00.754 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:16:00.754 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:16:00.754 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:16:00.754 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:00.755 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.755 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:16:00.755 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:00.755 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:16:00.756 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:00.756 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:00.756 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:16:00.756 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:00.757 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:16:00.758 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:00.758 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:16:00.758 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:00.758 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:00.759 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.759 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:00.759 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:16:00.759 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:00.760 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:00.760 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:16:00.760 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:00.760 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:00.760 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:00.761 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:16:00.761 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:00.761 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:00.761 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.762 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:00.762 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:16:00.762 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:00.762 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:00.762 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:16:00.763 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:00.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:00.763 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:00.763 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:16:00.763 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:00.764 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:00.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:00.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:16:00.765 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:00.765 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:00.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037239.629875_442.1", 2023-10-11T15:16:00.765 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:00.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:00.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:00.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:13:59.629875Z", 2023-10-11T15:16:00.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:00.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:00.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:00.767 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:16:00.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.767 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:16:00.767 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:16:00.767 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:16:00.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:00.768 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:16:00Z" 2023-10-11T15:16:00.768 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:16:00.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037239.629875_442.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:13:59.629875Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:16:00Z" } 2023-10-11T15:16:00.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037239.629875_442.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:13:59.629875Z', 'realm_epoch': 2}}] 2023-10-11T15:16:00.770 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:16:00.771 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037239.629875_442.1'} 2023-10-11T15:16:00.771 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697037268.300154_465.1 target=1_1697037239.629875_442.1 2023-10-11T15:16:05.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:16:05.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:16:05.774 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:16:05.833 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:05.833 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:05.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:05.863+0000 7fccb2c20780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:16:05.866 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:05.864+0000 7fccb2c20780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:16:05.879 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:16:05.880 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:16:05.880 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:16:05.880 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:16:05.880 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:16:05.881 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:16:05.881 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:05.881 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.881 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:16:05.882 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:05.882 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:16:05.882 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:05.882 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:05.882 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:16:05.883 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:05.883 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:16:05.883 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:05.883 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:16:05.884 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:05.884 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:05.884 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.884 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:05.884 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:16:05.885 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:05.885 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:05.885 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:16:05.885 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:05.886 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:05.886 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:05.886 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:16:05.887 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:05.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:05.887 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.887 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:05.887 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:16:05.888 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:05.888 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:05.888 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:16:05.888 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:05.889 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:05.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:05.889 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:16:05.889 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:05.890 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:05.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:16:05.890 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:16:05.890 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:16:05.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:16:05.891 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037268.300154_465.1", 2023-10-11T15:16:05.891 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:16:05.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:16:05.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:16:05.892 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:14:28.300154Z", 2023-10-11T15:16:05.892 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:16:05.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:05.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:16:05.893 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:16:05.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.893 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:16:05.894 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:16:05.894 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:16:05.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:16:05.894 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:16:05Z" 2023-10-11T15:16:05.895 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:16:05.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037268.300154_465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:14:28.300154Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:16:05Z" } 2023-10-11T15:16:05.896 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037268.300154_465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:14:28.300154Z', 'realm_epoch': 2}}] 2023-10-11T15:16:05.897 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:16:05.897 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037268.300154_465.1'} 2023-10-11T15:16:05.898 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:16:05.898 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:05.899 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:05.899 DEBUG:boto:Method: HEAD 2023-10-11T15:16:05.900 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:05.900 DEBUG:boto:Data: 2023-10-11T15:16:05.900 DEBUG:boto:Headers: {} 2023-10-11T15:16:05.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.901 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.902 DEBUG:boto:Params: {} 2023-10-11T15:16:05.902 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:16:05.903 DEBUG:boto:Token: None 2023-10-11T15:16:05.904 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/ 2023-10-11T15:16:05.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0= 2023-10-11T15:16:05.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0=', 'Content-Length': '0'} 2023-10-11T15:16:05.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5de3fd0ace9e1b0-006526bc35-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.910 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:05.910 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:05.911 DEBUG:boto:Method: PUT 2023-10-11T15:16:05.911 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:16:05.912 DEBUG:boto:Data: 2023-10-11T15:16:05.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:05.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.913 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.914 DEBUG:boto:Params: {} 2023-10-11T15:16:05.914 DEBUG:boto:Token: None 2023-10-11T15:16:05.915 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/obj11 2023-10-11T15:16:05.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Cb/qrrOU1hqZR47wT9Ws1cPiRzQ= 2023-10-11T15:16:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Cb/qrrOU1hqZR47wT9Ws1cPiRzQ='} 2023-10-11T15:16:05.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8e5b7df1f81a204-006526bc35-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.923 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:05.923 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:05.924 DEBUG:boto:Method: HEAD 2023-10-11T15:16:05.924 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:05.924 DEBUG:boto:Data: 2023-10-11T15:16:05.925 DEBUG:boto:Headers: {} 2023-10-11T15:16:05.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.926 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.926 DEBUG:boto:Params: {} 2023-10-11T15:16:05.927 DEBUG:boto:Token: None 2023-10-11T15:16:05.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/ 2023-10-11T15:16:05.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0= 2023-10-11T15:16:05.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0=', 'Content-Length': '0'} 2023-10-11T15:16:05.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000e7a5e0ccd816f8-006526bc35-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.931 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:05.932 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:05.932 DEBUG:boto:Method: PUT 2023-10-11T15:16:05.933 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:16:05.933 DEBUG:boto:Data: 2023-10-11T15:16:05.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:05.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.934 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.935 DEBUG:boto:Params: {} 2023-10-11T15:16:05.935 DEBUG:boto:Token: None 2023-10-11T15:16:05.936 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/obj22 2023-10-11T15:16:05.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m0TT4KcVIrkpxHQ4OBJ8oBK3m2U= 2023-10-11T15:16:05.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m0TT4KcVIrkpxHQ4OBJ8oBK3m2U='} 2023-10-11T15:16:05.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000594fff7ec0df39eb-006526bc35-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.942 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:05.942 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:05.943 DEBUG:boto:Method: HEAD 2023-10-11T15:16:05.943 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:05.944 DEBUG:boto:Data: 2023-10-11T15:16:05.944 DEBUG:boto:Headers: {} 2023-10-11T15:16:05.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.945 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.945 DEBUG:boto:Params: {} 2023-10-11T15:16:05.946 DEBUG:boto:Token: None 2023-10-11T15:16:05.946 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/ 2023-10-11T15:16:05.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0= 2023-10-11T15:16:05.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TwkpyKlJ+Wk7vYRXXHnDfS4pwA0=', 'Content-Length': '0'} 2023-10-11T15:16:05.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000376778f5f6cf953c-006526bc35-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.950 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:05.951 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:05.951 DEBUG:boto:Method: PUT 2023-10-11T15:16:05.952 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:16:05.952 DEBUG:boto:Data: 2023-10-11T15:16:05.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:05.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:05.953 DEBUG:boto:Port: 8000 2023-10-11T15:16:05.954 DEBUG:boto:Params: {} 2023-10-11T15:16:05.954 DEBUG:boto:Token: None 2023-10-11T15:16:05.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:05 GMT /dghqvd-59/obj33 2023-10-11T15:16:05.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u0FhqnS1rkhaCopSsg+Z/Qp0smk= 2023-10-11T15:16:05.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u0FhqnS1rkhaCopSsg+Z/Qp0smk='} 2023-10-11T15:16:05.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f76414cc39967882-006526bc35-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:05.999 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:06.000 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:06.000 DEBUG:boto:Method: HEAD 2023-10-11T15:16:06.000 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:06.001 DEBUG:boto:Data: 2023-10-11T15:16:06.001 DEBUG:boto:Headers: {} 2023-10-11T15:16:06.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:06.002 DEBUG:boto:Port: 8000 2023-10-11T15:16:06.003 DEBUG:boto:Params: {} 2023-10-11T15:16:06.003 DEBUG:boto:Token: None 2023-10-11T15:16:06.003 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:06 GMT /dghqvd-59/ 2023-10-11T15:16:06.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V88gU3bLrUWcdyNJP7w4QIAZasg= 2023-10-11T15:16:06.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V88gU3bLrUWcdyNJP7w4QIAZasg=', 'Content-Length': '0'} 2023-10-11T15:16:06.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012a433bfe9f28a45-006526bc36-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:06.007 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:06.008 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:06.008 DEBUG:boto:Method: PUT 2023-10-11T15:16:06.009 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:16:06.009 DEBUG:boto:Data: 2023-10-11T15:16:06.009 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:06.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:06.010 DEBUG:boto:Port: 8000 2023-10-11T15:16:06.011 DEBUG:boto:Params: {} 2023-10-11T15:16:06.011 DEBUG:boto:Token: None 2023-10-11T15:16:06.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:06 GMT /dghqvd-59/obj44 2023-10-11T15:16:06.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mC3XHWTwKQ2zc73ycrm4c+H0A2k= 2023-10-11T15:16:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mC3XHWTwKQ2zc73ycrm4c+H0A2k='} 2023-10-11T15:16:06.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013c407f006ba714c-006526bc36-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:06.018 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:06.019 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:06.019 DEBUG:boto:Method: HEAD 2023-10-11T15:16:06.020 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:06.020 DEBUG:boto:Data: 2023-10-11T15:16:06.020 DEBUG:boto:Headers: {} 2023-10-11T15:16:06.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:06.021 DEBUG:boto:Port: 8000 2023-10-11T15:16:06.021 DEBUG:boto:Params: {} 2023-10-11T15:16:06.022 DEBUG:boto:Token: None 2023-10-11T15:16:06.022 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:06 GMT /dghqvd-59/ 2023-10-11T15:16:06.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V88gU3bLrUWcdyNJP7w4QIAZasg= 2023-10-11T15:16:06.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V88gU3bLrUWcdyNJP7w4QIAZasg=', 'Content-Length': '0'} 2023-10-11T15:16:06.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017ed06d0350103f6-006526bc36-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:06.026 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:06.026 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:06.027 DEBUG:boto:Method: PUT 2023-10-11T15:16:06.027 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:16:06.028 DEBUG:boto:Data: 2023-10-11T15:16:06.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:06.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:06.029 DEBUG:boto:Port: 8000 2023-10-11T15:16:06.029 DEBUG:boto:Params: {} 2023-10-11T15:16:06.030 DEBUG:boto:Token: None 2023-10-11T15:16:06.030 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:06 GMT /dghqvd-59/obj55 2023-10-11T15:16:06.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lE3yUjffAHJdMGD7eBwZLBz/Cm4= 2023-10-11T15:16:06.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lE3yUjffAHJdMGD7eBwZLBz/Cm4='} 2023-10-11T15:16:06.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f3c43985e050275-006526bc36-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:06.075 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:16:06.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:06.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:06.077 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:16:06.134 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:06.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:06.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:06.170+0000 7f86b1b4e780 1 waiting to reach incremental sync.. 2023-10-11T15:16:11.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:11.171+0000 7f86b1b4e780 1 waiting to reach incremental sync.. 2023-10-11T15:16:16.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:16.171+0000 7f86b1b4e780 1 waiting to reach incremental sync.. 2023-10-11T15:16:21.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:21.173+0000 7f86b1b4e780 1 bucket sync caught up with source: 2023-10-11T15:16:21.175 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.88.6, 00000000002.66.6, 00000000002.89.6] 2023-10-11T15:16:21.176 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.88.6, 00000000002.66.6, 00000000002.89.6] 2023-10-11T15:16:21.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:21.173+0000 7f86b1b4e780 0 bucket checkpoint complete 2023-10-11T15:16:21.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:16:21.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:21.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:21.195 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:16:21.258 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:21.258 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:21.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:21.293+0000 7fa6fa6ba780 1 waiting to reach incremental sync.. 2023-10-11T15:16:26.296 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:26.294+0000 7fa6fa6ba780 1 waiting to reach incremental sync.. 2023-10-11T15:16:31.297 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:31.296+0000 7fa6fa6ba780 1 bucket sync caught up with source: 2023-10-11T15:16:31.298 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.220.6, 00000000002.163.6, 00000000002.280.6] 2023-10-11T15:16:31.298 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.220.6, 00000000002.163.6, 00000000002.280.6] 2023-10-11T15:16:31.298 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:31.296+0000 7fa6fa6ba780 0 bucket checkpoint complete 2023-10-11T15:16:31.314 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:16:31.315 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.315 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.316 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.316 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:31.317 DEBUG:boto:Data: 2023-10-11T15:16:31.317 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.318 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.318 DEBUG:boto:Params: {} 2023-10-11T15:16:31.319 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:16:31.319 DEBUG:boto:Token: None 2023-10-11T15:16:31.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3sKivgQMlxRXetk+LO0Wo8fsvY= 2023-10-11T15:16:31.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3sKivgQMlxRXetk+LO0Wo8fsvY=', 'Content-Length': '0'} 2023-10-11T15:16:31.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004574a9be469698c2-006526bc4f-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.326 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.327 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.327 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.327 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:31.328 DEBUG:boto:Data: 2023-10-11T15:16:31.328 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.329 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.329 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.329 DEBUG:boto:Params: {} 2023-10-11T15:16:31.330 DEBUG:boto:Token: None 2023-10-11T15:16:31.330 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3sKivgQMlxRXetk+LO0Wo8fsvY= 2023-10-11T15:16:31.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3sKivgQMlxRXetk+LO0Wo8fsvY=', 'Content-Length': '0'} 2023-10-11T15:16:31.334 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb06263c355bd69c-006526bc4f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.334 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:16:31.335 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.335 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.336 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.336 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.337 DEBUG:boto:Method: GET 2023-10-11T15:16:31.337 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.337 DEBUG:boto:Data: 2023-10-11T15:16:31.338 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.339 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.339 DEBUG:boto:Params: {} 2023-10-11T15:16:31.340 DEBUG:boto:Token: None 2023-10-11T15:16:31.340 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/?versions 2023-10-11T15:16:31.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y= 2023-10-11T15:16:31.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y=', 'Content-Length': '0'} 2023-10-11T15:16:31.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97e5ef7a922d516-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.344 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:16:31.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:16:31.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:16:31.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:16:31.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:16:31.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:16:31.348 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.348 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.348 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.349 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.349 DEBUG:boto:Method: GET 2023-10-11T15:16:31.350 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.350 DEBUG:boto:Data: 2023-10-11T15:16:31.350 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.351 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.352 DEBUG:boto:Params: {} 2023-10-11T15:16:31.352 DEBUG:boto:Token: None 2023-10-11T15:16:31.353 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/?versions 2023-10-11T15:16:31.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y= 2023-10-11T15:16:31.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y=', 'Content-Length': '0'} 2023-10-11T15:16:31.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782bd1c7a905bf52-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.357 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:16:31.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:16:31.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:16:31.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:16:31.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:16:31.360 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.360 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.361 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.361 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.362 DEBUG:boto:Method: GET 2023-10-11T15:16:31.362 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.363 DEBUG:boto:Data: 2023-10-11T15:16:31.363 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.364 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.364 DEBUG:boto:Params: {} 2023-10-11T15:16:31.365 DEBUG:boto:Token: None 2023-10-11T15:16:31.365 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/?versions 2023-10-11T15:16:31.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y= 2023-10-11T15:16:31.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y=', 'Content-Length': '0'} 2023-10-11T15:16:31.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138ecfa13b87fee5-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.369 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.370 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.371 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.371 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.371 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.372 DEBUG:boto:Method: GET 2023-10-11T15:16:31.372 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:16:31.373 DEBUG:boto:Data: 2023-10-11T15:16:31.373 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.374 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.374 DEBUG:boto:Params: {} 2023-10-11T15:16:31.375 DEBUG:boto:Token: None 2023-10-11T15:16:31.375 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/?versions 2023-10-11T15:16:31.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y= 2023-10-11T15:16:31.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WHSh25hIX7BN+XbAs0EbDqeDj8Y=', 'Content-Length': '0'} 2023-10-11T15:16:31.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6123937548f6d06-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.379 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:16:31.381 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.381 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.382 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.382 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.382 DEBUG:boto:Method: GET 2023-10-11T15:16:31.383 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.383 DEBUG:boto:Data: 2023-10-11T15:16:31.384 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.385 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.385 DEBUG:boto:Params: {} 2023-10-11T15:16:31.385 DEBUG:boto:Token: None 2023-10-11T15:16:31.386 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50= 2023-10-11T15:16:31.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50=', 'Content-Length': '0'} 2023-10-11T15:16:31.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b345bba88144725-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.390 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.390 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.391 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.391 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.392 DEBUG:boto:Method: GET 2023-10-11T15:16:31.392 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.393 DEBUG:boto:Data: 2023-10-11T15:16:31.393 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.394 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.395 DEBUG:boto:Params: {} 2023-10-11T15:16:31.395 DEBUG:boto:Token: None 2023-10-11T15:16:31.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50= 2023-10-11T15:16:31.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50=', 'Content-Length': '0'} 2023-10-11T15:16:31.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d59cd987046895-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.399 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.400 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.400 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.401 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.401 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.401 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.402 DEBUG:boto:Data: 2023-10-11T15:16:31.402 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.403 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.404 DEBUG:boto:Params: {} 2023-10-11T15:16:31.404 DEBUG:boto:Token: None 2023-10-11T15:16:31.405 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FoiOJPgqr14thRns5q/mCNtsDTA= 2023-10-11T15:16:31.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FoiOJPgqr14thRns5q/mCNtsDTA=', 'Content-Length': '0'} 2023-10-11T15:16:31.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af603f1af21fcfe2-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.408 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.409 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.409 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.410 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.410 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.410 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.411 DEBUG:boto:Data: 2023-10-11T15:16:31.411 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.412 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.413 DEBUG:boto:Params: {} 2023-10-11T15:16:31.413 DEBUG:boto:Token: None 2023-10-11T15:16:31.414 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FoiOJPgqr14thRns5q/mCNtsDTA= 2023-10-11T15:16:31.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FoiOJPgqr14thRns5q/mCNtsDTA=', 'Content-Length': '0'} 2023-10-11T15:16:31.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a08941cea63814-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:16:31.417 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.418 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.418 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.419 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.419 DEBUG:boto:Method: GET 2023-10-11T15:16:31.420 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.420 DEBUG:boto:Data: 2023-10-11T15:16:31.421 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.422 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.422 DEBUG:boto:Params: {} 2023-10-11T15:16:31.423 DEBUG:boto:Token: None 2023-10-11T15:16:31.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.424 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50= 2023-10-11T15:16:31.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50=', 'Content-Length': '0'} 2023-10-11T15:16:31.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ce673cda801ca6-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.426 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.427 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.427 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.428 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.428 DEBUG:boto:Method: GET 2023-10-11T15:16:31.429 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.429 DEBUG:boto:Data: 2023-10-11T15:16:31.429 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.430 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.431 DEBUG:boto:Params: {} 2023-10-11T15:16:31.431 DEBUG:boto:Token: None 2023-10-11T15:16:31.432 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:16:31.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50= 2023-10-11T15:16:31.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uldth8g690sC06PSkSTF3Kfqy50=', 'Content-Length': '0'} 2023-10-11T15:16:31.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d805f49d808a66-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.436 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.436 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.436 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.437 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.437 DEBUG:boto:Method: GET 2023-10-11T15:16:31.438 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.438 DEBUG:boto:Data: 2023-10-11T15:16:31.439 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.440 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.440 DEBUG:boto:Params: {} 2023-10-11T15:16:31.440 DEBUG:boto:Token: None 2023-10-11T15:16:31.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c209945cfdc8554-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.445 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.446 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.446 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.447 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.447 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.448 DEBUG:boto:Method: GET 2023-10-11T15:16:31.448 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.449 DEBUG:boto:Data: 2023-10-11T15:16:31.449 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.450 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.450 DEBUG:boto:Params: {} 2023-10-11T15:16:31.451 DEBUG:boto:Token: None 2023-10-11T15:16:31.451 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.452 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776d2112dd7a0b3c-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.498 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.499 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.500 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.500 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.501 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:16:31.501 DEBUG:boto:Data: 2023-10-11T15:16:31.501 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.502 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.502 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.503 DEBUG:boto:Params: {} 2023-10-11T15:16:31.503 DEBUG:boto:Token: None 2023-10-11T15:16:31.504 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11 2023-10-11T15:16:31.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hc2hOT60SH0FqnA9WEPSOUHUfg= 2023-10-11T15:16:31.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hc2hOT60SH0FqnA9WEPSOUHUfg=', 'Content-Length': '0'} 2023-10-11T15:16:31.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fc5c249e19876d1-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.507 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.508 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.508 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.509 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:16:31.509 DEBUG:boto:Data: 2023-10-11T15:16:31.509 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.510 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.511 DEBUG:boto:Params: {} 2023-10-11T15:16:31.511 DEBUG:boto:Token: None 2023-10-11T15:16:31.512 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11 2023-10-11T15:16:31.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3hc2hOT60SH0FqnA9WEPSOUHUfg= 2023-10-11T15:16:31.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3hc2hOT60SH0FqnA9WEPSOUHUfg=', 'Content-Length': '0'} 2023-10-11T15:16:31.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3019cd811bca33d-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:16:31.516 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.516 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.517 DEBUG:boto:Method: GET 2023-10-11T15:16:31.517 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:16:31.517 DEBUG:boto:Data: 2023-10-11T15:16:31.518 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.519 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.519 DEBUG:boto:Params: {} 2023-10-11T15:16:31.520 DEBUG:boto:Token: None 2023-10-11T15:16:31.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11 2023-10-11T15:16:31.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhCKntK/JS1Ngdu7xulU9aUXnVA= 2023-10-11T15:16:31.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhCKntK/JS1Ngdu7xulU9aUXnVA=', 'Content-Length': '0'} 2023-10-11T15:16:31.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5a9bb74fe2b7ef-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.524 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:16:31.525 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:16:31.525 DEBUG:boto:Method: GET 2023-10-11T15:16:31.526 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:16:31.526 DEBUG:boto:Data: 2023-10-11T15:16:31.527 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.528 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.528 DEBUG:boto:Params: {} 2023-10-11T15:16:31.529 DEBUG:boto:Token: None 2023-10-11T15:16:31.529 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj11 2023-10-11T15:16:31.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EhCKntK/JS1Ngdu7xulU9aUXnVA= 2023-10-11T15:16:31.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EhCKntK/JS1Ngdu7xulU9aUXnVA=', 'Content-Length': '0'} 2023-10-11T15:16:31.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d4931d4abc1020-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.533 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.534 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.534 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.535 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.535 DEBUG:boto:Method: GET 2023-10-11T15:16:31.535 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.536 DEBUG:boto:Data: 2023-10-11T15:16:31.536 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.537 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.537 DEBUG:boto:Params: {} 2023-10-11T15:16:31.538 DEBUG:boto:Token: None 2023-10-11T15:16:31.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9d5ffea59d458ce-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.542 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.543 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.544 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.544 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.544 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.545 DEBUG:boto:Method: GET 2023-10-11T15:16:31.545 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:16:31.546 DEBUG:boto:Data: 2023-10-11T15:16:31.546 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.547 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.548 DEBUG:boto:Params: {} 2023-10-11T15:16:31.548 DEBUG:boto:Token: None 2023-10-11T15:16:31.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e145e53be29b7ca-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.553 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:16:31.554 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.554 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.555 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.555 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.556 DEBUG:boto:Method: GET 2023-10-11T15:16:31.556 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.556 DEBUG:boto:Data: 2023-10-11T15:16:31.557 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.558 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.558 DEBUG:boto:Params: {} 2023-10-11T15:16:31.559 DEBUG:boto:Token: None 2023-10-11T15:16:31.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE= 2023-10-11T15:16:31.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE=', 'Content-Length': '0'} 2023-10-11T15:16:31.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008727bf5ae86b377b-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.563 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.563 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.564 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.564 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.564 DEBUG:boto:Method: GET 2023-10-11T15:16:31.565 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.565 DEBUG:boto:Data: 2023-10-11T15:16:31.566 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.567 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.567 DEBUG:boto:Params: {} 2023-10-11T15:16:31.568 DEBUG:boto:Token: None 2023-10-11T15:16:31.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE= 2023-10-11T15:16:31.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE=', 'Content-Length': '0'} 2023-10-11T15:16:31.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40540c559ace17f-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.572 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.573 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.573 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.573 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.574 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.574 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.575 DEBUG:boto:Data: 2023-10-11T15:16:31.575 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.576 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.576 DEBUG:boto:Params: {} 2023-10-11T15:16:31.577 DEBUG:boto:Token: None 2023-10-11T15:16:31.577 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TlRu9IZf0WfuQcmyqfIzhZL4Jc4= 2023-10-11T15:16:31.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TlRu9IZf0WfuQcmyqfIzhZL4Jc4=', 'Content-Length': '0'} 2023-10-11T15:16:31.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008864b463f56da85a-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.580 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.581 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.581 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.582 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.582 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.582 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.583 DEBUG:boto:Data: 2023-10-11T15:16:31.583 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.584 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.585 DEBUG:boto:Params: {} 2023-10-11T15:16:31.585 DEBUG:boto:Token: None 2023-10-11T15:16:31.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TlRu9IZf0WfuQcmyqfIzhZL4Jc4= 2023-10-11T15:16:31.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TlRu9IZf0WfuQcmyqfIzhZL4Jc4=', 'Content-Length': '0'} 2023-10-11T15:16:31.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fd19b64951c0ce5-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:16:31.589 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.590 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.590 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.590 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.591 DEBUG:boto:Method: GET 2023-10-11T15:16:31.591 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.592 DEBUG:boto:Data: 2023-10-11T15:16:31.592 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.593 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.594 DEBUG:boto:Params: {} 2023-10-11T15:16:31.594 DEBUG:boto:Token: None 2023-10-11T15:16:31.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE= 2023-10-11T15:16:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE=', 'Content-Length': '0'} 2023-10-11T15:16:31.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d9b5ebb50641730-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.642 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.643 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.644 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.644 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.645 DEBUG:boto:Method: GET 2023-10-11T15:16:31.645 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.646 DEBUG:boto:Data: 2023-10-11T15:16:31.646 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.647 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.648 DEBUG:boto:Params: {} 2023-10-11T15:16:31.649 DEBUG:boto:Token: None 2023-10-11T15:16:31.649 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:16:31.650 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE= 2023-10-11T15:16:31.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vMxYcIJkbgT/bLeQmpOmfSWGCE=', 'Content-Length': '0'} 2023-10-11T15:16:31.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba3dcf2b76f49c7-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.653 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.654 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.654 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.655 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.655 DEBUG:boto:Method: GET 2023-10-11T15:16:31.656 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.656 DEBUG:boto:Data: 2023-10-11T15:16:31.656 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.657 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.657 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.658 DEBUG:boto:Params: {} 2023-10-11T15:16:31.659 DEBUG:boto:Token: None 2023-10-11T15:16:31.659 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.659 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c30e1cf19ae74adf-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.664 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.665 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.665 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.666 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.667 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.667 DEBUG:boto:Method: GET 2023-10-11T15:16:31.668 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.669 DEBUG:boto:Data: 2023-10-11T15:16:31.669 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.670 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.671 DEBUG:boto:Params: {} 2023-10-11T15:16:31.672 DEBUG:boto:Token: None 2023-10-11T15:16:31.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9cb9bf3e5acc2e4-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.676 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.678 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.678 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.679 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.680 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:16:31.681 DEBUG:boto:Data: 2023-10-11T15:16:31.681 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.683 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.684 DEBUG:boto:Params: {} 2023-10-11T15:16:31.684 DEBUG:boto:Token: None 2023-10-11T15:16:31.685 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22 2023-10-11T15:16:31.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hh16TMuq+SsuwlXUaqPmX6/ODk4= 2023-10-11T15:16:31.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hh16TMuq+SsuwlXUaqPmX6/ODk4=', 'Content-Length': '0'} 2023-10-11T15:16:31.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003334b493c5a4a69a-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.690 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.691 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.691 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.692 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:16:31.692 DEBUG:boto:Data: 2023-10-11T15:16:31.693 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.693 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.694 DEBUG:boto:Params: {} 2023-10-11T15:16:31.694 DEBUG:boto:Token: None 2023-10-11T15:16:31.695 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22 2023-10-11T15:16:31.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hh16TMuq+SsuwlXUaqPmX6/ODk4= 2023-10-11T15:16:31.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hh16TMuq+SsuwlXUaqPmX6/ODk4=', 'Content-Length': '0'} 2023-10-11T15:16:31.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252f4c56f3bd3f96-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:16:31.700 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.701 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.701 DEBUG:boto:Method: GET 2023-10-11T15:16:31.702 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:16:31.702 DEBUG:boto:Data: 2023-10-11T15:16:31.703 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.704 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.704 DEBUG:boto:Params: {} 2023-10-11T15:16:31.705 DEBUG:boto:Token: None 2023-10-11T15:16:31.705 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22 2023-10-11T15:16:31.706 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t31x9Mp3nx4QvCliIjKFJpYV2zg= 2023-10-11T15:16:31.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t31x9Mp3nx4QvCliIjKFJpYV2zg=', 'Content-Length': '0'} 2023-10-11T15:16:31.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e060c2e3dfb6788c-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.710 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:16:31.710 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:16:31.711 DEBUG:boto:Method: GET 2023-10-11T15:16:31.711 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:16:31.712 DEBUG:boto:Data: 2023-10-11T15:16:31.712 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.713 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.714 DEBUG:boto:Params: {} 2023-10-11T15:16:31.714 DEBUG:boto:Token: None 2023-10-11T15:16:31.715 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj22 2023-10-11T15:16:31.715 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t31x9Mp3nx4QvCliIjKFJpYV2zg= 2023-10-11T15:16:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t31x9Mp3nx4QvCliIjKFJpYV2zg=', 'Content-Length': '0'} 2023-10-11T15:16:31.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e419db002a8e0a1b-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.718 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.719 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.719 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.719 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.720 DEBUG:boto:Method: GET 2023-10-11T15:16:31.720 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.721 DEBUG:boto:Data: 2023-10-11T15:16:31.721 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.722 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.723 DEBUG:boto:Params: {} 2023-10-11T15:16:31.723 DEBUG:boto:Token: None 2023-10-11T15:16:31.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172d6ff4736cd399-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.728 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.729 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.730 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.730 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.731 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.731 DEBUG:boto:Method: GET 2023-10-11T15:16:31.732 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:16:31.732 DEBUG:boto:Data: 2023-10-11T15:16:31.732 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.733 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.733 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.734 DEBUG:boto:Params: {} 2023-10-11T15:16:31.734 DEBUG:boto:Token: None 2023-10-11T15:16:31.735 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045ca20c29b34197-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.739 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:16:31.740 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.740 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.740 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.741 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.741 DEBUG:boto:Method: GET 2023-10-11T15:16:31.742 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.742 DEBUG:boto:Data: 2023-10-11T15:16:31.743 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.743 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.744 DEBUG:boto:Params: {} 2023-10-11T15:16:31.744 DEBUG:boto:Token: None 2023-10-11T15:16:31.745 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko= 2023-10-11T15:16:31.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko=', 'Content-Length': '0'} 2023-10-11T15:16:31.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2409534931d0b8-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.749 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.749 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.750 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.750 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.750 DEBUG:boto:Method: GET 2023-10-11T15:16:31.751 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.751 DEBUG:boto:Data: 2023-10-11T15:16:31.752 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.753 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.753 DEBUG:boto:Params: {} 2023-10-11T15:16:31.754 DEBUG:boto:Token: None 2023-10-11T15:16:31.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko= 2023-10-11T15:16:31.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko=', 'Content-Length': '0'} 2023-10-11T15:16:31.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5fed29a617065f8-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.758 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.758 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.759 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.759 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.760 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.760 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.760 DEBUG:boto:Data: 2023-10-11T15:16:31.761 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.761 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.762 DEBUG:boto:Params: {} 2023-10-11T15:16:31.762 DEBUG:boto:Token: None 2023-10-11T15:16:31.763 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mw+boIGfNaGhPNIbEribJHlFIxY= 2023-10-11T15:16:31.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mw+boIGfNaGhPNIbEribJHlFIxY=', 'Content-Length': '0'} 2023-10-11T15:16:31.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c981b03f9f4317-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.766 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.767 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.767 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.767 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.768 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.768 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.768 DEBUG:boto:Data: 2023-10-11T15:16:31.769 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.770 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.770 DEBUG:boto:Params: {} 2023-10-11T15:16:31.770 DEBUG:boto:Token: None 2023-10-11T15:16:31.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mw+boIGfNaGhPNIbEribJHlFIxY= 2023-10-11T15:16:31.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mw+boIGfNaGhPNIbEribJHlFIxY=', 'Content-Length': '0'} 2023-10-11T15:16:31.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df0ee2b9186d416-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:16:31.775 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.775 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.776 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.776 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.777 DEBUG:boto:Method: GET 2023-10-11T15:16:31.777 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.778 DEBUG:boto:Data: 2023-10-11T15:16:31.778 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.779 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.779 DEBUG:boto:Params: {} 2023-10-11T15:16:31.780 DEBUG:boto:Token: None 2023-10-11T15:16:31.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko= 2023-10-11T15:16:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko=', 'Content-Length': '0'} 2023-10-11T15:16:31.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379c5d5396f570f9-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.784 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.784 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.785 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.785 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.786 DEBUG:boto:Method: GET 2023-10-11T15:16:31.786 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.787 DEBUG:boto:Data: 2023-10-11T15:16:31.787 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.788 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.788 DEBUG:boto:Params: {} 2023-10-11T15:16:31.789 DEBUG:boto:Token: None 2023-10-11T15:16:31.789 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:16:31.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko= 2023-10-11T15:16:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aEun7FoELf5SB8opMLa7jCluJko=', 'Content-Length': '0'} 2023-10-11T15:16:31.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047eb211b9be9f4d0-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.793 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.793 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.794 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.794 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.795 DEBUG:boto:Method: GET 2023-10-11T15:16:31.795 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.795 DEBUG:boto:Data: 2023-10-11T15:16:31.796 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.797 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.797 DEBUG:boto:Params: {} 2023-10-11T15:16:31.798 DEBUG:boto:Token: None 2023-10-11T15:16:31.798 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794c6a7a2a953b93-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.846 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.847 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.848 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.848 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.849 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.849 DEBUG:boto:Method: GET 2023-10-11T15:16:31.850 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.850 DEBUG:boto:Data: 2023-10-11T15:16:31.850 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.851 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.852 DEBUG:boto:Params: {} 2023-10-11T15:16:31.852 DEBUG:boto:Token: None 2023-10-11T15:16:31.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d666fb6aac7b22d-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.857 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.858 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.858 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.859 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.859 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:16:31.859 DEBUG:boto:Data: 2023-10-11T15:16:31.860 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.861 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.861 DEBUG:boto:Params: {} 2023-10-11T15:16:31.861 DEBUG:boto:Token: None 2023-10-11T15:16:31.862 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33 2023-10-11T15:16:31.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rr6jsrK6h1t0wDiIVnOxaxYMYV8= 2023-10-11T15:16:31.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rr6jsrK6h1t0wDiIVnOxaxYMYV8=', 'Content-Length': '0'} 2023-10-11T15:16:31.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7e39e15e1def3e-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.866 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.866 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.867 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.867 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:16:31.867 DEBUG:boto:Data: 2023-10-11T15:16:31.868 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.869 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.869 DEBUG:boto:Params: {} 2023-10-11T15:16:31.869 DEBUG:boto:Token: None 2023-10-11T15:16:31.870 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33 2023-10-11T15:16:31.870 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rr6jsrK6h1t0wDiIVnOxaxYMYV8= 2023-10-11T15:16:31.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rr6jsrK6h1t0wDiIVnOxaxYMYV8=', 'Content-Length': '0'} 2023-10-11T15:16:31.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ceb53bca7e20613-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:16:31.874 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.875 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.875 DEBUG:boto:Method: GET 2023-10-11T15:16:31.876 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:16:31.876 DEBUG:boto:Data: 2023-10-11T15:16:31.876 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.877 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.878 DEBUG:boto:Params: {} 2023-10-11T15:16:31.878 DEBUG:boto:Token: None 2023-10-11T15:16:31.879 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33 2023-10-11T15:16:31.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5vnNl64cVtfYt7paldfEEOpTxFU= 2023-10-11T15:16:31.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5vnNl64cVtfYt7paldfEEOpTxFU=', 'Content-Length': '0'} 2023-10-11T15:16:31.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f011961664f9251-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.882 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:16:31.883 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:16:31.883 DEBUG:boto:Method: GET 2023-10-11T15:16:31.884 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:16:31.884 DEBUG:boto:Data: 2023-10-11T15:16:31.885 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.886 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.886 DEBUG:boto:Params: {} 2023-10-11T15:16:31.886 DEBUG:boto:Token: None 2023-10-11T15:16:31.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj33 2023-10-11T15:16:31.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5vnNl64cVtfYt7paldfEEOpTxFU= 2023-10-11T15:16:31.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5vnNl64cVtfYt7paldfEEOpTxFU=', 'Content-Length': '0'} 2023-10-11T15:16:31.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff291191c61d68b-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.891 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.891 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.892 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.892 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.892 DEBUG:boto:Method: GET 2023-10-11T15:16:31.893 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.893 DEBUG:boto:Data: 2023-10-11T15:16:31.894 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.894 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.895 DEBUG:boto:Params: {} 2023-10-11T15:16:31.895 DEBUG:boto:Token: None 2023-10-11T15:16:31.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.897 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071aac6adbf0a3522-006526bc4f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.900 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.901 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:31.901 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:31.902 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.902 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.903 DEBUG:boto:Method: GET 2023-10-11T15:16:31.903 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:16:31.904 DEBUG:boto:Data: 2023-10-11T15:16:31.904 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.905 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.905 DEBUG:boto:Params: {} 2023-10-11T15:16:31.906 DEBUG:boto:Token: None 2023-10-11T15:16:31.906 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/ 2023-10-11T15:16:31.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM= 2023-10-11T15:16:31.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rpl583ggQlIAjBM8CrwDStPH6VM=', 'Content-Length': '0'} 2023-10-11T15:16:31.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed1f2b9d7e57f963-006526bc4f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.911 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:31.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:16:31.912 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:31.912 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:31.913 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.913 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.914 DEBUG:boto:Method: GET 2023-10-11T15:16:31.914 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.914 DEBUG:boto:Data: 2023-10-11T15:16:31.915 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.916 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.916 DEBUG:boto:Params: {} 2023-10-11T15:16:31.917 DEBUG:boto:Token: None 2023-10-11T15:16:31.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40= 2023-10-11T15:16:31.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40=', 'Content-Length': '0'} 2023-10-11T15:16:31.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f61c969fc5f036-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.921 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:31.922 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:31.922 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.923 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.923 DEBUG:boto:Method: GET 2023-10-11T15:16:31.924 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.924 DEBUG:boto:Data: 2023-10-11T15:16:31.924 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.925 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.926 DEBUG:boto:Params: {} 2023-10-11T15:16:31.926 DEBUG:boto:Token: None 2023-10-11T15:16:31.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40= 2023-10-11T15:16:31.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40=', 'Content-Length': '0'} 2023-10-11T15:16:31.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c562cc6a9df224-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.974 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:31.975 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:31.975 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.976 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.976 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.977 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.977 DEBUG:boto:Data: 2023-10-11T15:16:31.977 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.978 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.979 DEBUG:boto:Params: {} 2023-10-11T15:16:31.979 DEBUG:boto:Token: None 2023-10-11T15:16:31.980 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GluVgZF232k0CXtNCUP0qSFdg8Q= 2023-10-11T15:16:31.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GluVgZF232k0CXtNCUP0qSFdg8Q=', 'Content-Length': '0'} 2023-10-11T15:16:31.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a560051ff0659f4-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.983 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:31.984 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:31.984 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.985 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.985 DEBUG:boto:Method: HEAD 2023-10-11T15:16:31.985 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.986 DEBUG:boto:Data: 2023-10-11T15:16:31.986 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:31.987 DEBUG:boto:Port: 8000 2023-10-11T15:16:31.987 DEBUG:boto:Params: {} 2023-10-11T15:16:31.988 DEBUG:boto:Token: None 2023-10-11T15:16:31.988 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GluVgZF232k0CXtNCUP0qSFdg8Q= 2023-10-11T15:16:31.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GluVgZF232k0CXtNCUP0qSFdg8Q=', 'Content-Length': '0'} 2023-10-11T15:16:31.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a14a332f97a91bb-006526bc4f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:31.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:16:31.993 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:31.993 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:31.993 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.994 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.994 DEBUG:boto:Method: GET 2023-10-11T15:16:31.995 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.995 DEBUG:boto:Data: 2023-10-11T15:16:31.996 DEBUG:boto:Headers: {} 2023-10-11T15:16:31.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:31.997 DEBUG:boto:Port: 8001 2023-10-11T15:16:31.997 DEBUG:boto:Params: {} 2023-10-11T15:16:31.997 DEBUG:boto:Token: None 2023-10-11T15:16:31.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:31 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:31.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40= 2023-10-11T15:16:31.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vyQtQUgujWdwzBHC6+yeTGVlP40=', 'Content-Length': '0'} 2023-10-11T15:16:32.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004381d148719fa2a6-006526bc4f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.046 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:32.047 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:32.047 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:32.048 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:16:32.048 DEBUG:boto:Method: GET 2023-10-11T15:16:32.049 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:32.049 DEBUG:boto:Data: 2023-10-11T15:16:32.050 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.050 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.051 DEBUG:boto:Params: {} 2023-10-11T15:16:32.051 DEBUG:boto:Token: None 2023-10-11T15:16:32.052 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:16:32.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DwyAXGuBi8aK+kaSGCyveDZPqZM= 2023-10-11T15:16:32.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DwyAXGuBi8aK+kaSGCyveDZPqZM=', 'Content-Length': '0'} 2023-10-11T15:16:32.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c8002cedea80ce-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.056 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.056 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.057 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.057 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.058 DEBUG:boto:Method: GET 2023-10-11T15:16:32.058 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.059 DEBUG:boto:Data: 2023-10-11T15:16:32.059 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.060 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.060 DEBUG:boto:Params: {} 2023-10-11T15:16:32.061 DEBUG:boto:Token: None 2023-10-11T15:16:32.061 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.062 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2d480fbcfa74e8-006526bc50-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.065 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.066 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.067 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.067 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.068 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.068 DEBUG:boto:Method: GET 2023-10-11T15:16:32.068 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.069 DEBUG:boto:Data: 2023-10-11T15:16:32.069 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.070 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.071 DEBUG:boto:Params: {} 2023-10-11T15:16:32.071 DEBUG:boto:Token: None 2023-10-11T15:16:32.072 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3aa18355991d9fa-006526bc50-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.076 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.077 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:32.077 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:32.078 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.078 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:16:32.079 DEBUG:boto:Data: 2023-10-11T15:16:32.079 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.080 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.080 DEBUG:boto:Params: {} 2023-10-11T15:16:32.081 DEBUG:boto:Token: None 2023-10-11T15:16:32.081 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj44 2023-10-11T15:16:32.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WWZ2ATSsDzhpRpN7umHEQ1GH8PU= 2023-10-11T15:16:32.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WWZ2ATSsDzhpRpN7umHEQ1GH8PU=', 'Content-Length': '0'} 2023-10-11T15:16:32.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf0b44315d520e5-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.085 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:32.085 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:32.086 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.086 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:16:32.087 DEBUG:boto:Data: 2023-10-11T15:16:32.087 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.088 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.088 DEBUG:boto:Params: {} 2023-10-11T15:16:32.089 DEBUG:boto:Token: None 2023-10-11T15:16:32.089 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj44 2023-10-11T15:16:32.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WWZ2ATSsDzhpRpN7umHEQ1GH8PU= 2023-10-11T15:16:32.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WWZ2ATSsDzhpRpN7umHEQ1GH8PU=', 'Content-Length': '0'} 2023-10-11T15:16:32.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000865dbfeb5051104b-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:16:32.093 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:32.093 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:32.094 DEBUG:boto:Method: GET 2023-10-11T15:16:32.094 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:16:32.095 DEBUG:boto:Data: 2023-10-11T15:16:32.095 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.096 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.097 DEBUG:boto:Params: {} 2023-10-11T15:16:32.097 DEBUG:boto:Token: None 2023-10-11T15:16:32.098 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj44 2023-10-11T15:16:32.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O4WgCMCJJArFtqfT7AlxDJJuSHI= 2023-10-11T15:16:32.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O4WgCMCJJArFtqfT7AlxDJJuSHI=', 'Content-Length': '0'} 2023-10-11T15:16:32.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e496dc2fc64bb0d6-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.101 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:16:32.102 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:16:32.102 DEBUG:boto:Method: GET 2023-10-11T15:16:32.102 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:16:32.103 DEBUG:boto:Data: 2023-10-11T15:16:32.103 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.104 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.105 DEBUG:boto:Params: {} 2023-10-11T15:16:32.105 DEBUG:boto:Token: None 2023-10-11T15:16:32.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj44 2023-10-11T15:16:32.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O4WgCMCJJArFtqfT7AlxDJJuSHI= 2023-10-11T15:16:32.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O4WgCMCJJArFtqfT7AlxDJJuSHI=', 'Content-Length': '0'} 2023-10-11T15:16:32.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619b709e41911408-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.154 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.155 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.156 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.156 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.156 DEBUG:boto:Method: GET 2023-10-11T15:16:32.157 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.157 DEBUG:boto:Data: 2023-10-11T15:16:32.158 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.158 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.159 DEBUG:boto:Params: {} 2023-10-11T15:16:32.159 DEBUG:boto:Token: None 2023-10-11T15:16:32.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464b4395d5585869-006526bc50-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.165 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.165 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.166 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.166 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.167 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.167 DEBUG:boto:Method: GET 2023-10-11T15:16:32.168 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:16:32.168 DEBUG:boto:Data: 2023-10-11T15:16:32.168 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.169 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.170 DEBUG:boto:Params: {} 2023-10-11T15:16:32.170 DEBUG:boto:Token: None 2023-10-11T15:16:32.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f80dac68b347b15-006526bc50-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.175 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:16:32.176 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.177 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.177 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.178 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.178 DEBUG:boto:Method: GET 2023-10-11T15:16:32.179 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.179 DEBUG:boto:Data: 2023-10-11T15:16:32.179 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.180 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.180 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.181 DEBUG:boto:Params: {} 2023-10-11T15:16:32.181 DEBUG:boto:Token: None 2023-10-11T15:16:32.182 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.182 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8= 2023-10-11T15:16:32.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8=', 'Content-Length': '0'} 2023-10-11T15:16:32.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c3426f7f705e2eb-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.186 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.186 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.187 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.187 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.187 DEBUG:boto:Method: GET 2023-10-11T15:16:32.188 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.188 DEBUG:boto:Data: 2023-10-11T15:16:32.189 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.190 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.190 DEBUG:boto:Params: {} 2023-10-11T15:16:32.191 DEBUG:boto:Token: None 2023-10-11T15:16:32.191 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.192 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8= 2023-10-11T15:16:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8=', 'Content-Length': '0'} 2023-10-11T15:16:32.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f93890c3ff51af-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.195 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.195 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.195 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.196 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.196 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.197 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.197 DEBUG:boto:Data: 2023-10-11T15:16:32.197 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.198 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.199 DEBUG:boto:Params: {} 2023-10-11T15:16:32.199 DEBUG:boto:Token: None 2023-10-11T15:16:32.200 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dow/npxinUTC/loY4KwApOF2qgY= 2023-10-11T15:16:32.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dow/npxinUTC/loY4KwApOF2qgY=', 'Content-Length': '0'} 2023-10-11T15:16:32.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e3249cdd65a1d3-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.203 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.203 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.204 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.204 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.205 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.205 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.205 DEBUG:boto:Data: 2023-10-11T15:16:32.206 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.206 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.207 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.207 DEBUG:boto:Params: {} 2023-10-11T15:16:32.208 DEBUG:boto:Token: None 2023-10-11T15:16:32.208 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dow/npxinUTC/loY4KwApOF2qgY= 2023-10-11T15:16:32.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dow/npxinUTC/loY4KwApOF2qgY=', 'Content-Length': '0'} 2023-10-11T15:16:32.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8658f2ba5a15efe-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:16:32.212 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.212 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.213 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.213 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.213 DEBUG:boto:Method: GET 2023-10-11T15:16:32.214 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.214 DEBUG:boto:Data: 2023-10-11T15:16:32.215 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.215 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.216 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.216 DEBUG:boto:Params: {} 2023-10-11T15:16:32.216 DEBUG:boto:Token: None 2023-10-11T15:16:32.217 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.217 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8= 2023-10-11T15:16:32.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8=', 'Content-Length': '0'} 2023-10-11T15:16:32.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e7267c8182f6367-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.262 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.263 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.263 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.264 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.264 DEBUG:boto:Method: GET 2023-10-11T15:16:32.265 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.265 DEBUG:boto:Data: 2023-10-11T15:16:32.266 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.267 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.267 DEBUG:boto:Params: {} 2023-10-11T15:16:32.267 DEBUG:boto:Token: None 2023-10-11T15:16:32.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:16:32.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8= 2023-10-11T15:16:32.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5u5Yq6sSPGgmLuLe+OSn/hwGVt8=', 'Content-Length': '0'} 2023-10-11T15:16:32.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80bba9813e2717f-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.272 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.272 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.273 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.273 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.273 DEBUG:boto:Method: GET 2023-10-11T15:16:32.274 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.274 DEBUG:boto:Data: 2023-10-11T15:16:32.275 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.276 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.276 DEBUG:boto:Params: {} 2023-10-11T15:16:32.276 DEBUG:boto:Token: None 2023-10-11T15:16:32.277 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a441a9d714e6d2d2-006526bc50-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.281 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.282 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.282 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.283 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.283 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.284 DEBUG:boto:Method: GET 2023-10-11T15:16:32.284 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.284 DEBUG:boto:Data: 2023-10-11T15:16:32.285 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.286 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.286 DEBUG:boto:Params: {} 2023-10-11T15:16:32.287 DEBUG:boto:Token: None 2023-10-11T15:16:32.287 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006669998da41b7cb1-006526bc50-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.334 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.335 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.336 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.336 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.336 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:16:32.337 DEBUG:boto:Data: 2023-10-11T15:16:32.337 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.337 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.338 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.338 DEBUG:boto:Params: {} 2023-10-11T15:16:32.339 DEBUG:boto:Token: None 2023-10-11T15:16:32.339 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55 2023-10-11T15:16:32.340 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VQxTSETyDPXPaL0tFIFSjHh6+pk= 2023-10-11T15:16:32.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VQxTSETyDPXPaL0tFIFSjHh6+pk=', 'Content-Length': '0'} 2023-10-11T15:16:32.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714f1e3f3d5af953-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.343 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.344 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.344 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.344 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:16:32.345 DEBUG:boto:Data: 2023-10-11T15:16:32.345 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.346 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.346 DEBUG:boto:Params: {} 2023-10-11T15:16:32.347 DEBUG:boto:Token: None 2023-10-11T15:16:32.347 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55 2023-10-11T15:16:32.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VQxTSETyDPXPaL0tFIFSjHh6+pk= 2023-10-11T15:16:32.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VQxTSETyDPXPaL0tFIFSjHh6+pk=', 'Content-Length': '0'} 2023-10-11T15:16:32.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecffa70c54657459-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:16:32.352 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.352 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.353 DEBUG:boto:Method: GET 2023-10-11T15:16:32.353 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:16:32.353 DEBUG:boto:Data: 2023-10-11T15:16:32.354 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.355 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.355 DEBUG:boto:Params: {} 2023-10-11T15:16:32.356 DEBUG:boto:Token: None 2023-10-11T15:16:32.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55 2023-10-11T15:16:32.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hAxPtCmc4F/Zrw3+UZDk1SBdOo0= 2023-10-11T15:16:32.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hAxPtCmc4F/Zrw3+UZDk1SBdOo0=', 'Content-Length': '0'} 2023-10-11T15:16:32.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aab47b5bb34e78e-006526bc50-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.360 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:16:32.360 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:16:32.361 DEBUG:boto:Method: GET 2023-10-11T15:16:32.361 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:16:32.361 DEBUG:boto:Data: 2023-10-11T15:16:32.362 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.363 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.363 DEBUG:boto:Params: {} 2023-10-11T15:16:32.364 DEBUG:boto:Token: None 2023-10-11T15:16:32.364 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj55 2023-10-11T15:16:32.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hAxPtCmc4F/Zrw3+UZDk1SBdOo0= 2023-10-11T15:16:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hAxPtCmc4F/Zrw3+UZDk1SBdOo0=', 'Content-Length': '0'} 2023-10-11T15:16:32.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979c7ac9e6134372-006526bc50-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.368 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.368 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.369 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.369 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.370 DEBUG:boto:Method: GET 2023-10-11T15:16:32.370 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.371 DEBUG:boto:Data: 2023-10-11T15:16:32.371 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:16:32.372 DEBUG:boto:Port: 8001 2023-10-11T15:16:32.372 DEBUG:boto:Params: {} 2023-10-11T15:16:32.373 DEBUG:boto:Token: None 2023-10-11T15:16:32.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e060cf84664505f-006526bc50-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.422 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.423 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.424 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.424 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.425 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.425 DEBUG:boto:Method: GET 2023-10-11T15:16:32.426 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:16:32.426 DEBUG:boto:Data: 2023-10-11T15:16:32.427 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.428 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.428 DEBUG:boto:Params: {} 2023-10-11T15:16:32.429 DEBUG:boto:Token: None 2023-10-11T15:16:32.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU= 2023-10-11T15:16:32.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jQ2fhgG8F9nJ6losR962kqsh6aU=', 'Content-Length': '0'} 2023-10-11T15:16:32.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd52db6ce03fa804-006526bc50-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.434 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:16:32.435 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:16:32.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:16:32.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:16:32.436 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:16:32.496 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:32.496 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:32.543 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:16:32.543 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:16:32.547 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 5 2023-10-11T15:16:32.583 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.583 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.584 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.584 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:32.585 DEBUG:boto:Data: 2023-10-11T15:16:32.585 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.585 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.586 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.586 DEBUG:boto:Params: {} 2023-10-11T15:16:32.587 DEBUG:boto:Token: None 2023-10-11T15:16:32.587 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA= 2023-10-11T15:16:32.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA=', 'Content-Length': '0'} 2023-10-11T15:16:32.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058933b0171c23462-006526bc50-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.592 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:16:32.592 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:16:32.593 DEBUG:boto:Method: PUT 2023-10-11T15:16:32.593 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:16:32.593 DEBUG:boto:Data: 2023-10-11T15:16:32.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:32.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.595 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.595 DEBUG:boto:Params: {} 2023-10-11T15:16:32.596 DEBUG:boto:Token: None 2023-10-11T15:16:32.596 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj17 2023-10-11T15:16:32.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qZKT49lSPUDzTMDrW2KiUd6z1qk= 2023-10-11T15:16:32.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qZKT49lSPUDzTMDrW2KiUd6z1qk='} 2023-10-11T15:16:32.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087d5eade3b07d50b-006526bc50-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.643 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.644 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.644 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.645 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:32.645 DEBUG:boto:Data: 2023-10-11T15:16:32.646 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.646 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.647 DEBUG:boto:Params: {} 2023-10-11T15:16:32.647 DEBUG:boto:Token: None 2023-10-11T15:16:32.648 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.649 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA= 2023-10-11T15:16:32.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA=', 'Content-Length': '0'} 2023-10-11T15:16:32.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b36598f6d8c189b-006526bc50-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.653 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:16:32.653 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:16:32.654 DEBUG:boto:Method: PUT 2023-10-11T15:16:32.654 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:16:32.655 DEBUG:boto:Data: 2023-10-11T15:16:32.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:32.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.656 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.656 DEBUG:boto:Params: {} 2023-10-11T15:16:32.657 DEBUG:boto:Token: None 2023-10-11T15:16:32.658 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj34 2023-10-11T15:16:32.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bwd3/BdRrdznzvIRs/xswIXavr8= 2023-10-11T15:16:32.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bwd3/BdRrdznzvIRs/xswIXavr8='} 2023-10-11T15:16:32.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db363ab19db97ecc-006526bc50-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.665 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.665 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.666 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.666 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:32.667 DEBUG:boto:Data: 2023-10-11T15:16:32.667 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.668 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.669 DEBUG:boto:Params: {} 2023-10-11T15:16:32.669 DEBUG:boto:Token: None 2023-10-11T15:16:32.670 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA= 2023-10-11T15:16:32.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA=', 'Content-Length': '0'} 2023-10-11T15:16:32.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5c3a0ee85ea4e41-006526bc50-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.674 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:16:32.674 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:16:32.675 DEBUG:boto:Method: PUT 2023-10-11T15:16:32.675 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:16:32.676 DEBUG:boto:Data: 2023-10-11T15:16:32.676 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:32.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.677 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.677 DEBUG:boto:Params: {} 2023-10-11T15:16:32.678 DEBUG:boto:Token: None 2023-10-11T15:16:32.679 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj51 2023-10-11T15:16:32.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dyLObZ4yxFye50Wt5CT+A3EN3Io= 2023-10-11T15:16:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dyLObZ4yxFye50Wt5CT+A3EN3Io='} 2023-10-11T15:16:32.731 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074201d3b003f6369-006526bc50-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.731 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.732 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.732 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.733 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:32.733 DEBUG:boto:Data: 2023-10-11T15:16:32.733 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.734 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.735 DEBUG:boto:Params: {} 2023-10-11T15:16:32.735 DEBUG:boto:Token: None 2023-10-11T15:16:32.736 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA= 2023-10-11T15:16:32.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA=', 'Content-Length': '0'} 2023-10-11T15:16:32.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f5501c78e87447-006526bc50-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.740 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:16:32.740 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:16:32.741 DEBUG:boto:Method: PUT 2023-10-11T15:16:32.741 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:16:32.741 DEBUG:boto:Data: 2023-10-11T15:16:32.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:32.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.743 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.743 DEBUG:boto:Params: {} 2023-10-11T15:16:32.743 DEBUG:boto:Token: None 2023-10-11T15:16:32.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj68 2023-10-11T15:16:32.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qI6SgDLzSL0ASnohhM+NppNmuZ0= 2023-10-11T15:16:32.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qI6SgDLzSL0ASnohhM+NppNmuZ0='} 2023-10-11T15:16:32.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a6e3b5cac3e2d0e-006526bc50-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.751 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:16:32.751 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:16:32.751 DEBUG:boto:Method: HEAD 2023-10-11T15:16:32.752 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:16:32.752 DEBUG:boto:Data: 2023-10-11T15:16:32.753 DEBUG:boto:Headers: {} 2023-10-11T15:16:32.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.753 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.754 DEBUG:boto:Params: {} 2023-10-11T15:16:32.754 DEBUG:boto:Token: None 2023-10-11T15:16:32.755 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/ 2023-10-11T15:16:32.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA= 2023-10-11T15:16:32.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ae9fuMQfVXzGa+D+0MKpvnpwUSA=', 'Content-Length': '0'} 2023-10-11T15:16:32.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082bc3f2a9ebf536d-006526bc50-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.759 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:16:32.759 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:16:32.760 DEBUG:boto:Method: PUT 2023-10-11T15:16:32.760 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:16:32.760 DEBUG:boto:Data: 2023-10-11T15:16:32.761 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:16:32.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:16:32.762 DEBUG:boto:Port: 8000 2023-10-11T15:16:32.762 DEBUG:boto:Params: {} 2023-10-11T15:16:32.763 DEBUG:boto:Token: None 2023-10-11T15:16:32.763 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:16:32 GMT /dghqvd-59/obj85 2023-10-11T15:16:32.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mhY0gXTmsSlMm1i57DkKZMcUIaQ= 2023-10-11T15:16:32.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:16:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mhY0gXTmsSlMm1i57DkKZMcUIaQ='} 2023-10-11T15:16:32.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f6ce97070f1c68e-006526bc50-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:16:32.811 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:16:32.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:32.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:16:32.813 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:16:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:16:32.871 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:16:32.906 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:32.904+0000 7f6db3abd780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T15:16:37.907 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:37.905+0000 7f6db3abd780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T15:16:42.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:42.907+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:16:42.909 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-11T15:16:42.909 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:16:47.910 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:47.908+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:16:47.910 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-11T15:16:47.911 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:16:52.912 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:52.909+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:16:52.912 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-11T15:16:52.912 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:16:57.913 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:16:57.910+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:16:57.913 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-10-11T15:16:57.914 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:02.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:02.912+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:02.915 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:02.915 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:07.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:07.913+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:07.918 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:07.918 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:12.917 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:12.915+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:12.918 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:12.918 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:17.916+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:17.919 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:17.920 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:22.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:22.918+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:22.921 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:22.921 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:27.922 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:27.919+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:27.922 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:27.922 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:32.923 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:32.922+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:32.924 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:32.924 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:37.925 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:37.923+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:37.925 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:37.925 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:42.926 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:42.925+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:42.927 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:42.927 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:47.928 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:47.926+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:47.928 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:47.928 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:52.929 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:52.928+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:52.930 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:52.930 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:57.931 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:17:57.929+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:17:57.931 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:17:57.931 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:02.932 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:02.930+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:02.932 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:02.933 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:07.933 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:07.932+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:07.934 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:07.934 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:12.935 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:12.933+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:12.936 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:12.936 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:17.937 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:17.935+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:17.937 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:17.937 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:22.938 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:22.935+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:22.938 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:22.939 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:27.939 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:27.937+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:27.940 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:27.940 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:32.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:32.938+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:32.941 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:32.942 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:37.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:37.940+0000 7f6db3abd780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:37.943 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , , , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:37.943 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:42.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:42.941+0000 7f6db3abd780 1 bucket sync caught up with source: 2023-10-11T15:18:42.945 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:42.945 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000001.33.6, , , , , , 00000000001.49.6, , 00000000001.45.6, , , , 00000000001.57.6, 00000000001.106.6, , , ] 2023-10-11T15:18:42.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:42.941+0000 7f6db3abd780 0 bucket checkpoint complete 2023-10-11T15:18:42.960 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:18:42.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:18:42.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:18:42.962 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:18:43.022 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:18:43.022 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:18:43.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:43.057+0000 7f59b7935780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:43.060 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000002.280.6] 2023-10-11T15:18:43.060 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000003.284.6] 2023-10-11T15:18:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:48.058+0000 7f59b7935780 1 waiting for incremental sync to catch up: 2023-10-11T15:18:48.061 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000002.280.6] 2023-10-11T15:18:48.062 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000003.284.6] 2023-10-11T15:18:53.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:53.059+0000 7f59b7935780 1 bucket sync caught up with source: 2023-10-11T15:18:53.062 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000003.284.6] 2023-10-11T15:18:53.062 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000001.215.6, 00000000001.226.6, , , 00000000001.284.6, , , 00000000001.220.6, 00000000003.167.6, 00000000003.284.6] 2023-10-11T15:18:53.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:53.059+0000 7f59b7935780 0 bucket checkpoint complete 2023-10-11T15:18:53.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:18:53.080 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.080 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.080 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.081 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:53.081 DEBUG:boto:Data: 2023-10-11T15:18:53.082 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.082 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.083 DEBUG:boto:Params: {} 2023-10-11T15:18:53.084 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:18:53.084 DEBUG:boto:Token: None 2023-10-11T15:18:53.085 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uRbOwfBVasNOyjBwcxMArZXCZ2Y= 2023-10-11T15:18:53.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRbOwfBVasNOyjBwcxMArZXCZ2Y=', 'Content-Length': '0'} 2023-10-11T15:18:53.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fccc7d3537326768-006526bcdd-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.090 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.091 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.091 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.092 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:53.092 DEBUG:boto:Data: 2023-10-11T15:18:53.092 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.093 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.093 DEBUG:boto:Params: {} 2023-10-11T15:18:53.094 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:18:53.094 DEBUG:boto:Token: None 2023-10-11T15:18:53.095 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uRbOwfBVasNOyjBwcxMArZXCZ2Y= 2023-10-11T15:18:53.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRbOwfBVasNOyjBwcxMArZXCZ2Y=', 'Content-Length': '0'} 2023-10-11T15:18:53.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073bd52cd561ad6dd-006526bcdd-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.100 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:18:53.101 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.101 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.102 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.102 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.102 DEBUG:boto:Method: GET 2023-10-11T15:18:53.103 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.103 DEBUG:boto:Data: 2023-10-11T15:18:53.104 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.104 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.105 DEBUG:boto:Params: {} 2023-10-11T15:18:53.105 DEBUG:boto:Token: None 2023-10-11T15:18:53.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/?versions 2023-10-11T15:18:53.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk= 2023-10-11T15:18:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk=', 'Content-Length': '0'} 2023-10-11T15:18:53.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88f97ffd44dc118-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.110 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:18:53.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:18:53.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:18:53.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:18:53.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:18:53.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:18:53.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:18:53.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:18:53.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:18:53.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:18:53.114 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:18:53.115 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.115 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.115 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.116 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.116 DEBUG:boto:Method: GET 2023-10-11T15:18:53.116 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.117 DEBUG:boto:Data: 2023-10-11T15:18:53.117 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.118 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.119 DEBUG:boto:Params: {} 2023-10-11T15:18:53.119 DEBUG:boto:Token: None 2023-10-11T15:18:53.119 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/?versions 2023-10-11T15:18:53.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk= 2023-10-11T15:18:53.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk=', 'Content-Length': '0'} 2023-10-11T15:18:53.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2771eb8ab2aa2ca-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.124 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:18:53.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:18:53.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:18:53.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:18:53.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:18:53.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:18:53.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:18:53.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:18:53.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:18:53.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:18:53.128 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.129 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.129 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.129 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.130 DEBUG:boto:Method: GET 2023-10-11T15:18:53.130 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.131 DEBUG:boto:Data: 2023-10-11T15:18:53.131 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.132 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.132 DEBUG:boto:Params: {} 2023-10-11T15:18:53.133 DEBUG:boto:Token: None 2023-10-11T15:18:53.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/?versions 2023-10-11T15:18:53.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk= 2023-10-11T15:18:53.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk=', 'Content-Length': '0'} 2023-10-11T15:18:53.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003746a831a27eaecd-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.137 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.138 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.139 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.139 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.140 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.140 DEBUG:boto:Method: GET 2023-10-11T15:18:53.140 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:18:53.141 DEBUG:boto:Data: 2023-10-11T15:18:53.141 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.142 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.142 DEBUG:boto:Params: {} 2023-10-11T15:18:53.143 DEBUG:boto:Token: None 2023-10-11T15:18:53.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/?versions 2023-10-11T15:18:53.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk= 2023-10-11T15:18:53.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uk9qfA7ki/UWj28ocJbNyY38qvk=', 'Content-Length': '0'} 2023-10-11T15:18:53.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e66b3e837c216ab-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.148 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:18:53.149 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.150 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.150 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.150 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.151 DEBUG:boto:Method: GET 2023-10-11T15:18:53.151 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.152 DEBUG:boto:Data: 2023-10-11T15:18:53.152 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.153 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.153 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.153 DEBUG:boto:Params: {} 2023-10-11T15:18:53.154 DEBUG:boto:Token: None 2023-10-11T15:18:53.154 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.155 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw= 2023-10-11T15:18:53.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw=', 'Content-Length': '0'} 2023-10-11T15:18:53.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1aa5f5a2a0df527-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.158 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.158 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.159 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.159 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.159 DEBUG:boto:Method: GET 2023-10-11T15:18:53.160 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.160 DEBUG:boto:Data: 2023-10-11T15:18:53.161 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.161 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.161 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.162 DEBUG:boto:Params: {} 2023-10-11T15:18:53.162 DEBUG:boto:Token: None 2023-10-11T15:18:53.163 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw= 2023-10-11T15:18:53.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw=', 'Content-Length': '0'} 2023-10-11T15:18:53.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7b70c85068f930-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.206 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.207 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.207 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.208 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.208 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.208 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.209 DEBUG:boto:Data: 2023-10-11T15:18:53.209 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.210 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.210 DEBUG:boto:Params: {} 2023-10-11T15:18:53.211 DEBUG:boto:Token: None 2023-10-11T15:18:53.211 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QLL8u/czipme4ClCuVHiljbn+sk= 2023-10-11T15:18:53.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QLL8u/czipme4ClCuVHiljbn+sk=', 'Content-Length': '0'} 2023-10-11T15:18:53.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454b29c7fe1cb97c-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.215 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.215 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.216 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.216 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.216 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.217 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.217 DEBUG:boto:Data: 2023-10-11T15:18:53.218 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.218 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.219 DEBUG:boto:Params: {} 2023-10-11T15:18:53.219 DEBUG:boto:Token: None 2023-10-11T15:18:53.220 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QLL8u/czipme4ClCuVHiljbn+sk= 2023-10-11T15:18:53.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QLL8u/czipme4ClCuVHiljbn+sk=', 'Content-Length': '0'} 2023-10-11T15:18:53.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072215f4111658919-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:18:53.223 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.224 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.224 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.225 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.225 DEBUG:boto:Method: GET 2023-10-11T15:18:53.225 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.226 DEBUG:boto:Data: 2023-10-11T15:18:53.226 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.227 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.228 DEBUG:boto:Params: {} 2023-10-11T15:18:53.228 DEBUG:boto:Token: None 2023-10-11T15:18:53.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw= 2023-10-11T15:18:53.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw=', 'Content-Length': '0'} 2023-10-11T15:18:53.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de64e9f5ed0b5b3-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.232 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.233 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.233 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.234 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.234 DEBUG:boto:Method: GET 2023-10-11T15:18:53.234 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.235 DEBUG:boto:Data: 2023-10-11T15:18:53.235 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.236 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.237 DEBUG:boto:Params: {} 2023-10-11T15:18:53.237 DEBUG:boto:Token: None 2023-10-11T15:18:53.238 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:18:53.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw= 2023-10-11T15:18:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cA77ipKiJPgxq4e/tZ8SIWSt1Yw=', 'Content-Length': '0'} 2023-10-11T15:18:53.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f6c44cb7caa758-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.242 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.242 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.242 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.243 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.243 DEBUG:boto:Method: GET 2023-10-11T15:18:53.244 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.244 DEBUG:boto:Data: 2023-10-11T15:18:53.245 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.245 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.246 DEBUG:boto:Params: {} 2023-10-11T15:18:53.246 DEBUG:boto:Token: None 2023-10-11T15:18:53.247 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ede8e3f6f2b571-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.251 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.252 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.252 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.252 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.253 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.253 DEBUG:boto:Method: GET 2023-10-11T15:18:53.254 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.254 DEBUG:boto:Data: 2023-10-11T15:18:53.255 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.255 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.256 DEBUG:boto:Params: {} 2023-10-11T15:18:53.256 DEBUG:boto:Token: None 2023-10-11T15:18:53.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e22ed28557610d5-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.261 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.262 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.262 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.263 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.263 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:18:53.264 DEBUG:boto:Data: 2023-10-11T15:18:53.264 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.264 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.265 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.265 DEBUG:boto:Params: {} 2023-10-11T15:18:53.266 DEBUG:boto:Token: None 2023-10-11T15:18:53.266 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11 2023-10-11T15:18:53.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MSoCqJgS+1f+AoP8u9GUvlQMSpQ= 2023-10-11T15:18:53.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MSoCqJgS+1f+AoP8u9GUvlQMSpQ=', 'Content-Length': '0'} 2023-10-11T15:18:53.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008073450f1daaf2a2-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.269 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.270 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.270 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.273 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:18:53.273 DEBUG:boto:Data: 2023-10-11T15:18:53.274 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.275 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.275 DEBUG:boto:Params: {} 2023-10-11T15:18:53.275 DEBUG:boto:Token: None 2023-10-11T15:18:53.276 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11 2023-10-11T15:18:53.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MSoCqJgS+1f+AoP8u9GUvlQMSpQ= 2023-10-11T15:18:53.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MSoCqJgS+1f+AoP8u9GUvlQMSpQ=', 'Content-Length': '0'} 2023-10-11T15:18:53.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c62cbd5a2df87c1-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:18:53.280 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.280 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.281 DEBUG:boto:Method: GET 2023-10-11T15:18:53.281 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:18:53.282 DEBUG:boto:Data: 2023-10-11T15:18:53.282 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.283 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.283 DEBUG:boto:Params: {} 2023-10-11T15:18:53.284 DEBUG:boto:Token: None 2023-10-11T15:18:53.284 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11 2023-10-11T15:18:53.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:93b5lIT0ihBu0RcsgnOoD3YHcZQ= 2023-10-11T15:18:53.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:93b5lIT0ihBu0RcsgnOoD3YHcZQ=', 'Content-Length': '0'} 2023-10-11T15:18:53.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa430cafa6e38db7-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.288 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:18:53.289 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:18:53.289 DEBUG:boto:Method: GET 2023-10-11T15:18:53.289 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:18:53.290 DEBUG:boto:Data: 2023-10-11T15:18:53.290 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.291 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.292 DEBUG:boto:Params: {} 2023-10-11T15:18:53.292 DEBUG:boto:Token: None 2023-10-11T15:18:53.293 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj11 2023-10-11T15:18:53.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:93b5lIT0ihBu0RcsgnOoD3YHcZQ= 2023-10-11T15:18:53.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:93b5lIT0ihBu0RcsgnOoD3YHcZQ=', 'Content-Length': '0'} 2023-10-11T15:18:53.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acdd0dcd95f8e467-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.342 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.343 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.344 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.344 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.344 DEBUG:boto:Method: GET 2023-10-11T15:18:53.345 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.345 DEBUG:boto:Data: 2023-10-11T15:18:53.346 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.347 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.347 DEBUG:boto:Params: {} 2023-10-11T15:18:53.348 DEBUG:boto:Token: None 2023-10-11T15:18:53.348 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001249ae4afd7d1716-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.352 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.353 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.353 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.353 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.354 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.354 DEBUG:boto:Method: GET 2023-10-11T15:18:53.355 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:18:53.355 DEBUG:boto:Data: 2023-10-11T15:18:53.356 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.356 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.357 DEBUG:boto:Params: {} 2023-10-11T15:18:53.357 DEBUG:boto:Token: None 2023-10-11T15:18:53.358 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1e20da09854dc23-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.362 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:18:53.363 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.363 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.364 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.364 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.364 DEBUG:boto:Method: GET 2023-10-11T15:18:53.365 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.365 DEBUG:boto:Data: 2023-10-11T15:18:53.366 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.367 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.367 DEBUG:boto:Params: {} 2023-10-11T15:18:53.367 DEBUG:boto:Token: None 2023-10-11T15:18:53.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc= 2023-10-11T15:18:53.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc=', 'Content-Length': '0'} 2023-10-11T15:18:53.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870d18d03d7df231-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.372 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.372 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.373 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.373 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.373 DEBUG:boto:Method: GET 2023-10-11T15:18:53.374 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.374 DEBUG:boto:Data: 2023-10-11T15:18:53.375 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.375 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.376 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.376 DEBUG:boto:Params: {} 2023-10-11T15:18:53.376 DEBUG:boto:Token: None 2023-10-11T15:18:53.377 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc= 2023-10-11T15:18:53.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc=', 'Content-Length': '0'} 2023-10-11T15:18:53.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a815e9284ca79729-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.381 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.381 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.382 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.382 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.383 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.383 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.383 DEBUG:boto:Data: 2023-10-11T15:18:53.384 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.385 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.385 DEBUG:boto:Params: {} 2023-10-11T15:18:53.386 DEBUG:boto:Token: None 2023-10-11T15:18:53.386 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AxxyQHBa697iwuwFEKu8Eo9gNCs= 2023-10-11T15:18:53.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AxxyQHBa697iwuwFEKu8Eo9gNCs=', 'Content-Length': '0'} 2023-10-11T15:18:53.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01e86d3c7d4db3f-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.389 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.390 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.390 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.390 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.391 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.391 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.392 DEBUG:boto:Data: 2023-10-11T15:18:53.392 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.393 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.393 DEBUG:boto:Params: {} 2023-10-11T15:18:53.394 DEBUG:boto:Token: None 2023-10-11T15:18:53.394 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AxxyQHBa697iwuwFEKu8Eo9gNCs= 2023-10-11T15:18:53.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AxxyQHBa697iwuwFEKu8Eo9gNCs=', 'Content-Length': '0'} 2023-10-11T15:18:53.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8aa1db48e6351fe-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:18:53.398 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.399 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.399 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.400 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.400 DEBUG:boto:Method: GET 2023-10-11T15:18:53.400 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.401 DEBUG:boto:Data: 2023-10-11T15:18:53.401 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.402 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.403 DEBUG:boto:Params: {} 2023-10-11T15:18:53.403 DEBUG:boto:Token: None 2023-10-11T15:18:53.404 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc= 2023-10-11T15:18:53.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc=', 'Content-Length': '0'} 2023-10-11T15:18:53.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150c367e556ad34e-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.450 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.451 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.451 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.452 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.452 DEBUG:boto:Method: GET 2023-10-11T15:18:53.452 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.453 DEBUG:boto:Data: 2023-10-11T15:18:53.453 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.454 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.455 DEBUG:boto:Params: {} 2023-10-11T15:18:53.455 DEBUG:boto:Token: None 2023-10-11T15:18:53.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:18:53.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc= 2023-10-11T15:18:53.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9pQxB4Iu62UxAUyK7U++hTrN8qc=', 'Content-Length': '0'} 2023-10-11T15:18:53.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006640b6dbac48df13-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.459 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.460 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.460 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.461 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.461 DEBUG:boto:Method: GET 2023-10-11T15:18:53.462 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.462 DEBUG:boto:Data: 2023-10-11T15:18:53.462 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.463 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.464 DEBUG:boto:Params: {} 2023-10-11T15:18:53.464 DEBUG:boto:Token: None 2023-10-11T15:18:53.465 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbdb4b6c04fedf19-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.469 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.469 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.470 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.470 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.471 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.471 DEBUG:boto:Method: GET 2023-10-11T15:18:53.472 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.472 DEBUG:boto:Data: 2023-10-11T15:18:53.472 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.473 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.474 DEBUG:boto:Params: {} 2023-10-11T15:18:53.474 DEBUG:boto:Token: None 2023-10-11T15:18:53.477 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1aecdcbb005f12-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.482 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.483 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.483 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.484 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.484 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:18:53.485 DEBUG:boto:Data: 2023-10-11T15:18:53.485 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.485 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.486 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.486 DEBUG:boto:Params: {} 2023-10-11T15:18:53.487 DEBUG:boto:Token: None 2023-10-11T15:18:53.487 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17 2023-10-11T15:18:53.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HnmB3qcPHhhyU2iVKIiO3AcjjhU= 2023-10-11T15:18:53.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HnmB3qcPHhhyU2iVKIiO3AcjjhU=', 'Content-Length': '0'} 2023-10-11T15:18:53.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022aecab0f83fd109-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.491 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.491 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.492 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.492 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:18:53.492 DEBUG:boto:Data: 2023-10-11T15:18:53.493 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.494 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.494 DEBUG:boto:Params: {} 2023-10-11T15:18:53.494 DEBUG:boto:Token: None 2023-10-11T15:18:53.495 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17 2023-10-11T15:18:53.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HnmB3qcPHhhyU2iVKIiO3AcjjhU= 2023-10-11T15:18:53.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HnmB3qcPHhhyU2iVKIiO3AcjjhU=', 'Content-Length': '0'} 2023-10-11T15:18:53.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003748d13e3e203392-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:18:53.499 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.499 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.500 DEBUG:boto:Method: GET 2023-10-11T15:18:53.500 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:18:53.501 DEBUG:boto:Data: 2023-10-11T15:18:53.501 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.502 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.502 DEBUG:boto:Params: {} 2023-10-11T15:18:53.503 DEBUG:boto:Token: None 2023-10-11T15:18:53.503 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17 2023-10-11T15:18:53.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CP1/L+tOJHKjDU5OK8vtk7h7gdg= 2023-10-11T15:18:53.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CP1/L+tOJHKjDU5OK8vtk7h7gdg=', 'Content-Length': '0'} 2023-10-11T15:18:53.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cbff9b44c4753e1-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.507 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:18:53.507 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:18:53.508 DEBUG:boto:Method: GET 2023-10-11T15:18:53.508 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:18:53.508 DEBUG:boto:Data: 2023-10-11T15:18:53.509 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.510 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.510 DEBUG:boto:Params: {} 2023-10-11T15:18:53.511 DEBUG:boto:Token: None 2023-10-11T15:18:53.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj17 2023-10-11T15:18:53.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CP1/L+tOJHKjDU5OK8vtk7h7gdg= 2023-10-11T15:18:53.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CP1/L+tOJHKjDU5OK8vtk7h7gdg=', 'Content-Length': '0'} 2023-10-11T15:18:53.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009971cb1295d193c8-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.515 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.515 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.516 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.516 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.516 DEBUG:boto:Method: GET 2023-10-11T15:18:53.517 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.517 DEBUG:boto:Data: 2023-10-11T15:18:53.518 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.519 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.519 DEBUG:boto:Params: {} 2023-10-11T15:18:53.520 DEBUG:boto:Token: None 2023-10-11T15:18:53.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e673a951cd176493-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.566 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.567 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.568 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.568 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.569 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.569 DEBUG:boto:Method: GET 2023-10-11T15:18:53.570 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:18:53.570 DEBUG:boto:Data: 2023-10-11T15:18:53.570 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.571 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.572 DEBUG:boto:Params: {} 2023-10-11T15:18:53.572 DEBUG:boto:Token: None 2023-10-11T15:18:53.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ed0fb883945d3c-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.577 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:18:53.578 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.579 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.579 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.580 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.580 DEBUG:boto:Method: GET 2023-10-11T15:18:53.580 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.581 DEBUG:boto:Data: 2023-10-11T15:18:53.581 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.582 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.583 DEBUG:boto:Params: {} 2023-10-11T15:18:53.583 DEBUG:boto:Token: None 2023-10-11T15:18:53.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0= 2023-10-11T15:18:53.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0=', 'Content-Length': '0'} 2023-10-11T15:18:53.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e67732fb9ed7ef-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.587 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.587 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.588 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.588 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.589 DEBUG:boto:Method: GET 2023-10-11T15:18:53.589 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.590 DEBUG:boto:Data: 2023-10-11T15:18:53.590 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.591 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.591 DEBUG:boto:Params: {} 2023-10-11T15:18:53.592 DEBUG:boto:Token: None 2023-10-11T15:18:53.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0= 2023-10-11T15:18:53.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0=', 'Content-Length': '0'} 2023-10-11T15:18:53.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa706950ca655ec8-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.596 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.596 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.597 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.597 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.597 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.598 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.598 DEBUG:boto:Data: 2023-10-11T15:18:53.599 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.600 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.600 DEBUG:boto:Params: {} 2023-10-11T15:18:53.600 DEBUG:boto:Token: None 2023-10-11T15:18:53.601 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4EUtDwwxXdb++XjIt5HYbnALL1U= 2023-10-11T15:18:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4EUtDwwxXdb++XjIt5HYbnALL1U=', 'Content-Length': '0'} 2023-10-11T15:18:53.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf25e3d444ad0bf-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.604 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.605 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.605 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.605 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.606 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.606 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.607 DEBUG:boto:Data: 2023-10-11T15:18:53.607 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.608 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.608 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.608 DEBUG:boto:Params: {} 2023-10-11T15:18:53.609 DEBUG:boto:Token: None 2023-10-11T15:18:53.609 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4EUtDwwxXdb++XjIt5HYbnALL1U= 2023-10-11T15:18:53.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4EUtDwwxXdb++XjIt5HYbnALL1U=', 'Content-Length': '0'} 2023-10-11T15:18:53.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad26871cc47a5d6-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:18:53.613 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.614 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.614 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.615 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.615 DEBUG:boto:Method: GET 2023-10-11T15:18:53.615 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.616 DEBUG:boto:Data: 2023-10-11T15:18:53.616 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.617 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.617 DEBUG:boto:Params: {} 2023-10-11T15:18:53.618 DEBUG:boto:Token: None 2023-10-11T15:18:53.618 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.619 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0= 2023-10-11T15:18:53.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0=', 'Content-Length': '0'} 2023-10-11T15:18:53.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f20f059ba839789-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.622 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.623 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.623 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.624 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.624 DEBUG:boto:Method: GET 2023-10-11T15:18:53.624 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.625 DEBUG:boto:Data: 2023-10-11T15:18:53.625 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.626 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.626 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.627 DEBUG:boto:Params: {} 2023-10-11T15:18:53.627 DEBUG:boto:Token: None 2023-10-11T15:18:53.628 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:18:53.628 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0= 2023-10-11T15:18:53.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHSperx9jzOyrP2Dd7uwLWWALA0=', 'Content-Length': '0'} 2023-10-11T15:18:53.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93fc0e68c277784-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.678 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.682 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.682 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.683 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.683 DEBUG:boto:Method: GET 2023-10-11T15:18:53.684 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.684 DEBUG:boto:Data: 2023-10-11T15:18:53.684 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.685 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.685 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.686 DEBUG:boto:Params: {} 2023-10-11T15:18:53.686 DEBUG:boto:Token: None 2023-10-11T15:18:53.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f994c2ef44e716c-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.691 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.692 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.692 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.693 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.693 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.693 DEBUG:boto:Method: GET 2023-10-11T15:18:53.694 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.694 DEBUG:boto:Data: 2023-10-11T15:18:53.695 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.695 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.696 DEBUG:boto:Params: {} 2023-10-11T15:18:53.696 DEBUG:boto:Token: None 2023-10-11T15:18:53.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4b7d852f0eac4c-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.701 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.702 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.702 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.703 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.703 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:18:53.704 DEBUG:boto:Data: 2023-10-11T15:18:53.704 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.705 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.705 DEBUG:boto:Params: {} 2023-10-11T15:18:53.706 DEBUG:boto:Token: None 2023-10-11T15:18:53.706 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22 2023-10-11T15:18:53.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91pGecZQ6jRL5HL5VLmLUGrvnik= 2023-10-11T15:18:53.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91pGecZQ6jRL5HL5VLmLUGrvnik=', 'Content-Length': '0'} 2023-10-11T15:18:53.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051ee8beac08f0d91-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.710 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.710 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.711 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.711 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:18:53.711 DEBUG:boto:Data: 2023-10-11T15:18:53.712 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.713 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.713 DEBUG:boto:Params: {} 2023-10-11T15:18:53.714 DEBUG:boto:Token: None 2023-10-11T15:18:53.714 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22 2023-10-11T15:18:53.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:91pGecZQ6jRL5HL5VLmLUGrvnik= 2023-10-11T15:18:53.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91pGecZQ6jRL5HL5VLmLUGrvnik=', 'Content-Length': '0'} 2023-10-11T15:18:53.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009191192753d56d4-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:18:53.718 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.718 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.719 DEBUG:boto:Method: GET 2023-10-11T15:18:53.719 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:18:53.720 DEBUG:boto:Data: 2023-10-11T15:18:53.720 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.721 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.721 DEBUG:boto:Params: {} 2023-10-11T15:18:53.722 DEBUG:boto:Token: None 2023-10-11T15:18:53.722 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22 2023-10-11T15:18:53.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTUwf7Z8iVCwJxtWZWPRCWW0Fbc= 2023-10-11T15:18:53.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTUwf7Z8iVCwJxtWZWPRCWW0Fbc=', 'Content-Length': '0'} 2023-10-11T15:18:53.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba6d5e8edb034ec0-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.766 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:18:53.767 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:18:53.767 DEBUG:boto:Method: GET 2023-10-11T15:18:53.768 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:18:53.768 DEBUG:boto:Data: 2023-10-11T15:18:53.769 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.769 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.770 DEBUG:boto:Params: {} 2023-10-11T15:18:53.770 DEBUG:boto:Token: None 2023-10-11T15:18:53.771 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj22 2023-10-11T15:18:53.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OTUwf7Z8iVCwJxtWZWPRCWW0Fbc= 2023-10-11T15:18:53.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OTUwf7Z8iVCwJxtWZWPRCWW0Fbc=', 'Content-Length': '0'} 2023-10-11T15:18:53.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac2f0661d5e5ea4-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.775 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.775 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.775 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.776 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.776 DEBUG:boto:Method: GET 2023-10-11T15:18:53.777 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.777 DEBUG:boto:Data: 2023-10-11T15:18:53.778 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.779 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.779 DEBUG:boto:Params: {} 2023-10-11T15:18:53.779 DEBUG:boto:Token: None 2023-10-11T15:18:53.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f3a7b1513ea700-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.784 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.785 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.785 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.785 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.786 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.786 DEBUG:boto:Method: GET 2023-10-11T15:18:53.787 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:18:53.787 DEBUG:boto:Data: 2023-10-11T15:18:53.787 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.788 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.789 DEBUG:boto:Params: {} 2023-10-11T15:18:53.789 DEBUG:boto:Token: None 2023-10-11T15:18:53.790 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68fb1032d21727f-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.794 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:18:53.795 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.796 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.796 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.796 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.797 DEBUG:boto:Method: GET 2023-10-11T15:18:53.797 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.798 DEBUG:boto:Data: 2023-10-11T15:18:53.798 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.799 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.799 DEBUG:boto:Params: {} 2023-10-11T15:18:53.800 DEBUG:boto:Token: None 2023-10-11T15:18:53.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8= 2023-10-11T15:18:53.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8=', 'Content-Length': '0'} 2023-10-11T15:18:53.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe19d888ad6c215-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.804 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.805 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.805 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.805 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.806 DEBUG:boto:Method: GET 2023-10-11T15:18:53.806 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.807 DEBUG:boto:Data: 2023-10-11T15:18:53.807 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.808 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.809 DEBUG:boto:Params: {} 2023-10-11T15:18:53.809 DEBUG:boto:Token: None 2023-10-11T15:18:53.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8= 2023-10-11T15:18:53.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8=', 'Content-Length': '0'} 2023-10-11T15:18:53.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c304d76dd9bae8e-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.813 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.814 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.814 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.814 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.815 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.815 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.816 DEBUG:boto:Data: 2023-10-11T15:18:53.816 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.817 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.817 DEBUG:boto:Params: {} 2023-10-11T15:18:53.818 DEBUG:boto:Token: None 2023-10-11T15:18:53.818 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T7lWW6BMf/smnZeKzSIv2WubZKo= 2023-10-11T15:18:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T7lWW6BMf/smnZeKzSIv2WubZKo=', 'Content-Length': '0'} 2023-10-11T15:18:53.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004940a5dbc19095ae-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.822 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.822 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.823 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.823 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.823 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.824 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.824 DEBUG:boto:Data: 2023-10-11T15:18:53.825 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.825 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.826 DEBUG:boto:Params: {} 2023-10-11T15:18:53.826 DEBUG:boto:Token: None 2023-10-11T15:18:53.827 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T7lWW6BMf/smnZeKzSIv2WubZKo= 2023-10-11T15:18:53.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T7lWW6BMf/smnZeKzSIv2WubZKo=', 'Content-Length': '0'} 2023-10-11T15:18:53.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b596be04a0e851a-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:18:53.831 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.831 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.831 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.832 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.832 DEBUG:boto:Method: GET 2023-10-11T15:18:53.833 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.833 DEBUG:boto:Data: 2023-10-11T15:18:53.834 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.834 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.834 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.835 DEBUG:boto:Params: {} 2023-10-11T15:18:53.835 DEBUG:boto:Token: None 2023-10-11T15:18:53.836 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8= 2023-10-11T15:18:53.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8=', 'Content-Length': '0'} 2023-10-11T15:18:53.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b117c7885c06437-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.839 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.840 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.840 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.841 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.841 DEBUG:boto:Method: GET 2023-10-11T15:18:53.841 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.842 DEBUG:boto:Data: 2023-10-11T15:18:53.842 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.843 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.843 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.844 DEBUG:boto:Params: {} 2023-10-11T15:18:53.844 DEBUG:boto:Token: None 2023-10-11T15:18:53.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:18:53.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8= 2023-10-11T15:18:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UXrR/vzAhVT//Ffh4jlcdYy7HT8=', 'Content-Length': '0'} 2023-10-11T15:18:53.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9818d624ae6570-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.848 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.849 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.849 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.849 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.850 DEBUG:boto:Method: GET 2023-10-11T15:18:53.850 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.851 DEBUG:boto:Data: 2023-10-11T15:18:53.851 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.852 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.852 DEBUG:boto:Params: {} 2023-10-11T15:18:53.853 DEBUG:boto:Token: None 2023-10-11T15:18:53.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e1324ae7d85273-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.898 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.899 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.899 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.900 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.900 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.900 DEBUG:boto:Method: GET 2023-10-11T15:18:53.901 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.901 DEBUG:boto:Data: 2023-10-11T15:18:53.902 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.903 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.903 DEBUG:boto:Params: {} 2023-10-11T15:18:53.903 DEBUG:boto:Token: None 2023-10-11T15:18:53.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092196992bc611ee5-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.908 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.909 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.909 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.910 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.910 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:18:53.910 DEBUG:boto:Data: 2023-10-11T15:18:53.911 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.912 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.912 DEBUG:boto:Params: {} 2023-10-11T15:18:53.912 DEBUG:boto:Token: None 2023-10-11T15:18:53.913 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33 2023-10-11T15:18:53.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bEWV+bW7zPHh688imLNj3OxVgTM= 2023-10-11T15:18:53.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bEWV+bW7zPHh688imLNj3OxVgTM=', 'Content-Length': '0'} 2023-10-11T15:18:53.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae7eddab5c5a5ce-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.916 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.917 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.917 DEBUG:boto:Method: HEAD 2023-10-11T15:18:53.918 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:18:53.918 DEBUG:boto:Data: 2023-10-11T15:18:53.919 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.919 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.920 DEBUG:boto:Params: {} 2023-10-11T15:18:53.920 DEBUG:boto:Token: None 2023-10-11T15:18:53.920 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33 2023-10-11T15:18:53.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bEWV+bW7zPHh688imLNj3OxVgTM= 2023-10-11T15:18:53.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bEWV+bW7zPHh688imLNj3OxVgTM=', 'Content-Length': '0'} 2023-10-11T15:18:53.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068a3e1d39bf03536-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:18:53.924 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.924 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.925 DEBUG:boto:Method: GET 2023-10-11T15:18:53.925 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:18:53.926 DEBUG:boto:Data: 2023-10-11T15:18:53.926 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.927 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.927 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.927 DEBUG:boto:Params: {} 2023-10-11T15:18:53.928 DEBUG:boto:Token: None 2023-10-11T15:18:53.928 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33 2023-10-11T15:18:53.929 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3YKyJFiSqpnIDytWoHKBjIq31DY= 2023-10-11T15:18:53.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3YKyJFiSqpnIDytWoHKBjIq31DY=', 'Content-Length': '0'} 2023-10-11T15:18:53.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028e450a5a744ad94-006526bcdd-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.932 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:18:53.933 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:18:53.933 DEBUG:boto:Method: GET 2023-10-11T15:18:53.934 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:18:53.934 DEBUG:boto:Data: 2023-10-11T15:18:53.934 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.935 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.936 DEBUG:boto:Params: {} 2023-10-11T15:18:53.936 DEBUG:boto:Token: None 2023-10-11T15:18:53.936 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/obj33 2023-10-11T15:18:53.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3YKyJFiSqpnIDytWoHKBjIq31DY= 2023-10-11T15:18:53.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3YKyJFiSqpnIDytWoHKBjIq31DY=', 'Content-Length': '0'} 2023-10-11T15:18:53.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3f90bcdd714cad-006526bcdd-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.982 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.983 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.983 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.984 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.984 DEBUG:boto:Method: GET 2023-10-11T15:18:53.985 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.985 DEBUG:boto:Data: 2023-10-11T15:18:53.986 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:53.986 DEBUG:boto:Port: 8001 2023-10-11T15:18:53.987 DEBUG:boto:Params: {} 2023-10-11T15:18:53.987 DEBUG:boto:Token: None 2023-10-11T15:18:53.988 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:53.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad30f73e9eaf3267-006526bcdd-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:53.992 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:53.993 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:53.993 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:53.994 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.994 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.995 DEBUG:boto:Method: GET 2023-10-11T15:18:53.995 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:18:53.996 DEBUG:boto:Data: 2023-10-11T15:18:53.996 DEBUG:boto:Headers: {} 2023-10-11T15:18:53.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:53.997 DEBUG:boto:Port: 8000 2023-10-11T15:18:53.997 DEBUG:boto:Params: {} 2023-10-11T15:18:53.998 DEBUG:boto:Token: None 2023-10-11T15:18:53.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:53 GMT /dghqvd-59/ 2023-10-11T15:18:53.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4= 2023-10-11T15:18:53.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELi0Z+LGBh0Bxv8DaA8yuCNhXQ4=', 'Content-Length': '0'} 2023-10-11T15:18:54.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e7e21047b6236a-006526bcdd-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.003 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:18:54.004 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.004 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.005 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.005 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.006 DEBUG:boto:Method: GET 2023-10-11T15:18:54.006 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.006 DEBUG:boto:Data: 2023-10-11T15:18:54.007 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.008 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.008 DEBUG:boto:Params: {} 2023-10-11T15:18:54.009 DEBUG:boto:Token: None 2023-10-11T15:18:54.009 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.010 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM= 2023-10-11T15:18:54.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM=', 'Content-Length': '0'} 2023-10-11T15:18:54.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686a072c70538bb5-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.013 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.013 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.014 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.014 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.015 DEBUG:boto:Method: GET 2023-10-11T15:18:54.015 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.015 DEBUG:boto:Data: 2023-10-11T15:18:54.016 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.017 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.017 DEBUG:boto:Params: {} 2023-10-11T15:18:54.018 DEBUG:boto:Token: None 2023-10-11T15:18:54.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM= 2023-10-11T15:18:54.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM=', 'Content-Length': '0'} 2023-10-11T15:18:54.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f12ad28da0230ee9-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.022 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.022 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.023 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.023 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.023 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.024 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.024 DEBUG:boto:Data: 2023-10-11T15:18:54.025 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.026 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.026 DEBUG:boto:Params: {} 2023-10-11T15:18:54.027 DEBUG:boto:Token: None 2023-10-11T15:18:54.027 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w3MKVyf8aNhuwldC+Zy6kdazkIs= 2023-10-11T15:18:54.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w3MKVyf8aNhuwldC+Zy6kdazkIs=', 'Content-Length': '0'} 2023-10-11T15:18:54.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6cdafb0185ea192-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.030 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.031 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.031 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.032 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.032 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.032 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.033 DEBUG:boto:Data: 2023-10-11T15:18:54.033 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.034 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.034 DEBUG:boto:Params: {} 2023-10-11T15:18:54.035 DEBUG:boto:Token: None 2023-10-11T15:18:54.035 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w3MKVyf8aNhuwldC+Zy6kdazkIs= 2023-10-11T15:18:54.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w3MKVyf8aNhuwldC+Zy6kdazkIs=', 'Content-Length': '0'} 2023-10-11T15:18:54.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1cc5452546b4bf-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:18:54.039 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.040 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.040 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.041 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.041 DEBUG:boto:Method: GET 2023-10-11T15:18:54.041 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.042 DEBUG:boto:Data: 2023-10-11T15:18:54.042 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.043 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.044 DEBUG:boto:Params: {} 2023-10-11T15:18:54.044 DEBUG:boto:Token: None 2023-10-11T15:18:54.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM= 2023-10-11T15:18:54.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM=', 'Content-Length': '0'} 2023-10-11T15:18:54.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc31fb2bf9fcecf2-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.048 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.049 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.049 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.050 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.050 DEBUG:boto:Method: GET 2023-10-11T15:18:54.051 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.051 DEBUG:boto:Data: 2023-10-11T15:18:54.051 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.052 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.053 DEBUG:boto:Params: {} 2023-10-11T15:18:54.053 DEBUG:boto:Token: None 2023-10-11T15:18:54.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:18:54.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM= 2023-10-11T15:18:54.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qsY7ohHbxWdvfglClz8lwtQ8noM=', 'Content-Length': '0'} 2023-10-11T15:18:54.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ca15aa01818d45-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.102 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.103 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.103 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.104 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.104 DEBUG:boto:Method: GET 2023-10-11T15:18:54.105 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.105 DEBUG:boto:Data: 2023-10-11T15:18:54.106 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.106 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.107 DEBUG:boto:Params: {} 2023-10-11T15:18:54.107 DEBUG:boto:Token: None 2023-10-11T15:18:54.108 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f09a366c793e0a-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.112 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.113 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.113 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.114 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.114 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.115 DEBUG:boto:Method: GET 2023-10-11T15:18:54.115 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.115 DEBUG:boto:Data: 2023-10-11T15:18:54.116 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.117 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.117 DEBUG:boto:Params: {} 2023-10-11T15:18:54.118 DEBUG:boto:Token: None 2023-10-11T15:18:54.118 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051916413fa3e840b-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.123 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.123 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.124 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.124 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.124 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:18:54.125 DEBUG:boto:Data: 2023-10-11T15:18:54.125 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.126 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.126 DEBUG:boto:Params: {} 2023-10-11T15:18:54.127 DEBUG:boto:Token: None 2023-10-11T15:18:54.127 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34 2023-10-11T15:18:54.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z5KSNt4PCb72vTj93VBCT+SKLS8= 2023-10-11T15:18:54.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z5KSNt4PCb72vTj93VBCT+SKLS8=', 'Content-Length': '0'} 2023-10-11T15:18:54.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d03e4dbb93ccd0ca-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.131 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.131 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.132 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.132 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:18:54.132 DEBUG:boto:Data: 2023-10-11T15:18:54.133 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.134 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.134 DEBUG:boto:Params: {} 2023-10-11T15:18:54.135 DEBUG:boto:Token: None 2023-10-11T15:18:54.135 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34 2023-10-11T15:18:54.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z5KSNt4PCb72vTj93VBCT+SKLS8= 2023-10-11T15:18:54.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z5KSNt4PCb72vTj93VBCT+SKLS8=', 'Content-Length': '0'} 2023-10-11T15:18:54.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246e800f13795a00-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:18:54.139 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.139 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.140 DEBUG:boto:Method: GET 2023-10-11T15:18:54.140 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:18:54.140 DEBUG:boto:Data: 2023-10-11T15:18:54.141 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.142 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.142 DEBUG:boto:Params: {} 2023-10-11T15:18:54.143 DEBUG:boto:Token: None 2023-10-11T15:18:54.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34 2023-10-11T15:18:54.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3DGXlahtqWKjavmrVXdgCJiJNdU= 2023-10-11T15:18:54.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3DGXlahtqWKjavmrVXdgCJiJNdU=', 'Content-Length': '0'} 2023-10-11T15:18:54.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b0bcb453a8adb7-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.186 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:18:54.187 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:18:54.187 DEBUG:boto:Method: GET 2023-10-11T15:18:54.188 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:18:54.188 DEBUG:boto:Data: 2023-10-11T15:18:54.188 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.189 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.190 DEBUG:boto:Params: {} 2023-10-11T15:18:54.190 DEBUG:boto:Token: None 2023-10-11T15:18:54.191 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj34 2023-10-11T15:18:54.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3DGXlahtqWKjavmrVXdgCJiJNdU= 2023-10-11T15:18:54.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3DGXlahtqWKjavmrVXdgCJiJNdU=', 'Content-Length': '0'} 2023-10-11T15:18:54.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030958dba92c46b6-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.194 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.195 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.195 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.196 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.196 DEBUG:boto:Method: GET 2023-10-11T15:18:54.196 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.197 DEBUG:boto:Data: 2023-10-11T15:18:54.197 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.198 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.199 DEBUG:boto:Params: {} 2023-10-11T15:18:54.199 DEBUG:boto:Token: None 2023-10-11T15:18:54.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ae0c758aafd3c7b-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.204 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.204 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.205 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.205 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.206 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.206 DEBUG:boto:Method: GET 2023-10-11T15:18:54.206 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:18:54.207 DEBUG:boto:Data: 2023-10-11T15:18:54.207 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.208 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.208 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.209 DEBUG:boto:Params: {} 2023-10-11T15:18:54.209 DEBUG:boto:Token: None 2023-10-11T15:18:54.210 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.210 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d00405ae33505a-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.214 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:18:54.215 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.216 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.216 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.217 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.217 DEBUG:boto:Method: GET 2023-10-11T15:18:54.218 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.218 DEBUG:boto:Data: 2023-10-11T15:18:54.218 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.219 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.220 DEBUG:boto:Params: {} 2023-10-11T15:18:54.220 DEBUG:boto:Token: None 2023-10-11T15:18:54.221 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0= 2023-10-11T15:18:54.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0=', 'Content-Length': '0'} 2023-10-11T15:18:54.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021271044929e669a-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.224 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.225 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.225 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.226 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.226 DEBUG:boto:Method: GET 2023-10-11T15:18:54.227 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.227 DEBUG:boto:Data: 2023-10-11T15:18:54.227 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.228 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.229 DEBUG:boto:Params: {} 2023-10-11T15:18:54.229 DEBUG:boto:Token: None 2023-10-11T15:18:54.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0= 2023-10-11T15:18:54.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0=', 'Content-Length': '0'} 2023-10-11T15:18:54.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f334f3ca69a9f7-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.234 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.234 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.234 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.235 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.235 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.236 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.236 DEBUG:boto:Data: 2023-10-11T15:18:54.236 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.237 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.238 DEBUG:boto:Params: {} 2023-10-11T15:18:54.238 DEBUG:boto:Token: None 2023-10-11T15:18:54.239 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zq+RHV/znOBBHdiPSpuWadEbqQw= 2023-10-11T15:18:54.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zq+RHV/znOBBHdiPSpuWadEbqQw=', 'Content-Length': '0'} 2023-10-11T15:18:54.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343eb5c5b125961f-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.242 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.243 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.243 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.243 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.244 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.244 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.245 DEBUG:boto:Data: 2023-10-11T15:18:54.245 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.246 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.246 DEBUG:boto:Params: {} 2023-10-11T15:18:54.247 DEBUG:boto:Token: None 2023-10-11T15:18:54.247 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zq+RHV/znOBBHdiPSpuWadEbqQw= 2023-10-11T15:18:54.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zq+RHV/znOBBHdiPSpuWadEbqQw=', 'Content-Length': '0'} 2023-10-11T15:18:54.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5d2ce1d4e6efb6-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:18:54.251 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.251 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.252 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.252 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.253 DEBUG:boto:Method: GET 2023-10-11T15:18:54.253 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.253 DEBUG:boto:Data: 2023-10-11T15:18:54.254 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.255 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.255 DEBUG:boto:Params: {} 2023-10-11T15:18:54.256 DEBUG:boto:Token: None 2023-10-11T15:18:54.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0= 2023-10-11T15:18:54.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0=', 'Content-Length': '0'} 2023-10-11T15:18:54.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c190bf4405ba6cb1-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.260 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.260 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.260 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.261 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.261 DEBUG:boto:Method: GET 2023-10-11T15:18:54.262 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.262 DEBUG:boto:Data: 2023-10-11T15:18:54.263 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.263 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.263 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.264 DEBUG:boto:Params: {} 2023-10-11T15:18:54.264 DEBUG:boto:Token: None 2023-10-11T15:18:54.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:18:54.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0= 2023-10-11T15:18:54.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BAMql7Rj4PxARQQj6Rlt1yIsdn0=', 'Content-Length': '0'} 2023-10-11T15:18:54.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006336158cdaaace1e-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.269 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.269 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.269 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.270 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.270 DEBUG:boto:Method: GET 2023-10-11T15:18:54.271 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.271 DEBUG:boto:Data: 2023-10-11T15:18:54.272 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.272 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.273 DEBUG:boto:Params: {} 2023-10-11T15:18:54.273 DEBUG:boto:Token: None 2023-10-11T15:18:54.274 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e0a9636ba03cde-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.318 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.319 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.320 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.320 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.320 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.321 DEBUG:boto:Method: GET 2023-10-11T15:18:54.321 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.322 DEBUG:boto:Data: 2023-10-11T15:18:54.322 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.323 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.323 DEBUG:boto:Params: {} 2023-10-11T15:18:54.324 DEBUG:boto:Token: None 2023-10-11T15:18:54.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b960e653abe25b9f-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.329 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.330 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.330 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.331 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.331 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:18:54.332 DEBUG:boto:Data: 2023-10-11T15:18:54.332 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.333 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.333 DEBUG:boto:Params: {} 2023-10-11T15:18:54.334 DEBUG:boto:Token: None 2023-10-11T15:18:54.334 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44 2023-10-11T15:18:54.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pFPhYEUOeTQ4N6U4S8cWhdy771g= 2023-10-11T15:18:54.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pFPhYEUOeTQ4N6U4S8cWhdy771g=', 'Content-Length': '0'} 2023-10-11T15:18:54.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b48edd614cb9c45-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.338 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.338 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.339 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.339 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:18:54.339 DEBUG:boto:Data: 2023-10-11T15:18:54.340 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.340 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.341 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.341 DEBUG:boto:Params: {} 2023-10-11T15:18:54.341 DEBUG:boto:Token: None 2023-10-11T15:18:54.342 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44 2023-10-11T15:18:54.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pFPhYEUOeTQ4N6U4S8cWhdy771g= 2023-10-11T15:18:54.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pFPhYEUOeTQ4N6U4S8cWhdy771g=', 'Content-Length': '0'} 2023-10-11T15:18:54.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a13251835865a1b-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:18:54.346 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.346 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.347 DEBUG:boto:Method: GET 2023-10-11T15:18:54.347 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:18:54.348 DEBUG:boto:Data: 2023-10-11T15:18:54.348 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.349 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.349 DEBUG:boto:Params: {} 2023-10-11T15:18:54.350 DEBUG:boto:Token: None 2023-10-11T15:18:54.350 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44 2023-10-11T15:18:54.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yhhwx58WzJVxAiGvVZx+C/uqRmg= 2023-10-11T15:18:54.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yhhwx58WzJVxAiGvVZx+C/uqRmg=', 'Content-Length': '0'} 2023-10-11T15:18:54.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d52791765ce31ae2-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.354 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:18:54.354 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:18:54.355 DEBUG:boto:Method: GET 2023-10-11T15:18:54.355 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:18:54.356 DEBUG:boto:Data: 2023-10-11T15:18:54.356 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.357 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.357 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.357 DEBUG:boto:Params: {} 2023-10-11T15:18:54.358 DEBUG:boto:Token: None 2023-10-11T15:18:54.358 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj44 2023-10-11T15:18:54.359 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yhhwx58WzJVxAiGvVZx+C/uqRmg= 2023-10-11T15:18:54.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yhhwx58WzJVxAiGvVZx+C/uqRmg=', 'Content-Length': '0'} 2023-10-11T15:18:54.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000864d938604ec8c09-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.362 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.362 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.363 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.363 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.364 DEBUG:boto:Method: GET 2023-10-11T15:18:54.364 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.365 DEBUG:boto:Data: 2023-10-11T15:18:54.365 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.365 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.366 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.366 DEBUG:boto:Params: {} 2023-10-11T15:18:54.367 DEBUG:boto:Token: None 2023-10-11T15:18:54.367 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000232b65a95f94990-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.371 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.372 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.373 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.373 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.373 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.374 DEBUG:boto:Method: GET 2023-10-11T15:18:54.374 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:18:54.375 DEBUG:boto:Data: 2023-10-11T15:18:54.375 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.376 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.376 DEBUG:boto:Params: {} 2023-10-11T15:18:54.377 DEBUG:boto:Token: None 2023-10-11T15:18:54.377 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f922a21f3cfccd4a-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.382 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:18:54.383 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.383 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.384 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.384 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.384 DEBUG:boto:Method: GET 2023-10-11T15:18:54.385 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.385 DEBUG:boto:Data: 2023-10-11T15:18:54.386 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.387 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.387 DEBUG:boto:Params: {} 2023-10-11T15:18:54.388 DEBUG:boto:Token: None 2023-10-11T15:18:54.388 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA= 2023-10-11T15:18:54.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA=', 'Content-Length': '0'} 2023-10-11T15:18:54.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017b924c29957123-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.392 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.392 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.393 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.393 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.393 DEBUG:boto:Method: GET 2023-10-11T15:18:54.394 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.394 DEBUG:boto:Data: 2023-10-11T15:18:54.395 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.396 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.396 DEBUG:boto:Params: {} 2023-10-11T15:18:54.397 DEBUG:boto:Token: None 2023-10-11T15:18:54.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA= 2023-10-11T15:18:54.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA=', 'Content-Length': '0'} 2023-10-11T15:18:54.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4047ab034163c4d-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.401 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.401 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.401 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.402 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.402 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.403 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.403 DEBUG:boto:Data: 2023-10-11T15:18:54.403 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.404 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.404 DEBUG:boto:Params: {} 2023-10-11T15:18:54.405 DEBUG:boto:Token: None 2023-10-11T15:18:54.405 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9QZZLZ3uOaTE1+fNiXEMEUogKdI= 2023-10-11T15:18:54.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9QZZLZ3uOaTE1+fNiXEMEUogKdI=', 'Content-Length': '0'} 2023-10-11T15:18:54.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000489f41ddfcbef542-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.409 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.409 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.409 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.410 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.410 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.411 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.411 DEBUG:boto:Data: 2023-10-11T15:18:54.411 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.412 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.413 DEBUG:boto:Params: {} 2023-10-11T15:18:54.413 DEBUG:boto:Token: None 2023-10-11T15:18:54.413 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9QZZLZ3uOaTE1+fNiXEMEUogKdI= 2023-10-11T15:18:54.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9QZZLZ3uOaTE1+fNiXEMEUogKdI=', 'Content-Length': '0'} 2023-10-11T15:18:54.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000332a8d7590b0bc3e-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:18:54.417 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.418 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.418 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.419 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.419 DEBUG:boto:Method: GET 2023-10-11T15:18:54.420 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.420 DEBUG:boto:Data: 2023-10-11T15:18:54.420 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.421 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.422 DEBUG:boto:Params: {} 2023-10-11T15:18:54.422 DEBUG:boto:Token: None 2023-10-11T15:18:54.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA= 2023-10-11T15:18:54.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA=', 'Content-Length': '0'} 2023-10-11T15:18:54.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007594cb432107cb37-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.466 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.467 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.467 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.468 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.468 DEBUG:boto:Method: GET 2023-10-11T15:18:54.469 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.469 DEBUG:boto:Data: 2023-10-11T15:18:54.470 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.470 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.471 DEBUG:boto:Params: {} 2023-10-11T15:18:54.471 DEBUG:boto:Token: None 2023-10-11T15:18:54.472 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:18:54.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA= 2023-10-11T15:18:54.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jk/m9B0fhI8C+IvbaN6QHgaDqMA=', 'Content-Length': '0'} 2023-10-11T15:18:54.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f2f8ebc59f7bf0-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.476 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.476 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.476 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.477 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.477 DEBUG:boto:Method: GET 2023-10-11T15:18:54.478 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.478 DEBUG:boto:Data: 2023-10-11T15:18:54.479 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.479 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.480 DEBUG:boto:Params: {} 2023-10-11T15:18:54.480 DEBUG:boto:Token: None 2023-10-11T15:18:54.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd872acef3fe2306-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.485 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.486 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.486 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.487 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.487 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.487 DEBUG:boto:Method: GET 2023-10-11T15:18:54.488 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.488 DEBUG:boto:Data: 2023-10-11T15:18:54.489 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.490 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.490 DEBUG:boto:Params: {} 2023-10-11T15:18:54.490 DEBUG:boto:Token: None 2023-10-11T15:18:54.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c07adeda83919095-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.542 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.543 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.544 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.544 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.544 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:18:54.545 DEBUG:boto:Data: 2023-10-11T15:18:54.545 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.546 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.547 DEBUG:boto:Params: {} 2023-10-11T15:18:54.547 DEBUG:boto:Token: None 2023-10-11T15:18:54.547 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51 2023-10-11T15:18:54.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VxjP7TzYva+RGI2EiiStQJcTOpw= 2023-10-11T15:18:54.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VxjP7TzYva+RGI2EiiStQJcTOpw=', 'Content-Length': '0'} 2023-10-11T15:18:54.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000479122daf4cb744f-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.551 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.551 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.552 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.552 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:18:54.553 DEBUG:boto:Data: 2023-10-11T15:18:54.553 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.554 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.554 DEBUG:boto:Params: {} 2023-10-11T15:18:54.555 DEBUG:boto:Token: None 2023-10-11T15:18:54.555 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51 2023-10-11T15:18:54.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VxjP7TzYva+RGI2EiiStQJcTOpw= 2023-10-11T15:18:54.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VxjP7TzYva+RGI2EiiStQJcTOpw=', 'Content-Length': '0'} 2023-10-11T15:18:54.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071cca6911e6615e2-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:18:54.559 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.559 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.560 DEBUG:boto:Method: GET 2023-10-11T15:18:54.560 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:18:54.560 DEBUG:boto:Data: 2023-10-11T15:18:54.561 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.562 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.562 DEBUG:boto:Params: {} 2023-10-11T15:18:54.563 DEBUG:boto:Token: None 2023-10-11T15:18:54.563 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51 2023-10-11T15:18:54.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NWwRStSnR7ZUxgC34fmW4irLws8= 2023-10-11T15:18:54.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NWwRStSnR7ZUxgC34fmW4irLws8=', 'Content-Length': '0'} 2023-10-11T15:18:54.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006cb86081c6b1eee-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.567 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:18:54.567 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:18:54.567 DEBUG:boto:Method: GET 2023-10-11T15:18:54.568 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:18:54.568 DEBUG:boto:Data: 2023-10-11T15:18:54.569 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.570 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.570 DEBUG:boto:Params: {} 2023-10-11T15:18:54.571 DEBUG:boto:Token: None 2023-10-11T15:18:54.571 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj51 2023-10-11T15:18:54.571 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NWwRStSnR7ZUxgC34fmW4irLws8= 2023-10-11T15:18:54.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NWwRStSnR7ZUxgC34fmW4irLws8=', 'Content-Length': '0'} 2023-10-11T15:18:54.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4460b52740d4373-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.575 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.575 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.575 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.576 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.576 DEBUG:boto:Method: GET 2023-10-11T15:18:54.577 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.577 DEBUG:boto:Data: 2023-10-11T15:18:54.578 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.578 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.579 DEBUG:boto:Params: {} 2023-10-11T15:18:54.579 DEBUG:boto:Token: None 2023-10-11T15:18:54.580 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2447328eeb9b6d6-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.584 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.584 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.585 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.585 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.586 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.586 DEBUG:boto:Method: GET 2023-10-11T15:18:54.587 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:18:54.587 DEBUG:boto:Data: 2023-10-11T15:18:54.588 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.588 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.589 DEBUG:boto:Params: {} 2023-10-11T15:18:54.589 DEBUG:boto:Token: None 2023-10-11T15:18:54.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.590 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ca11bb717c303b0-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.594 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:18:54.595 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.596 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.596 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.597 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.597 DEBUG:boto:Method: GET 2023-10-11T15:18:54.597 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.598 DEBUG:boto:Data: 2023-10-11T15:18:54.598 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.599 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.599 DEBUG:boto:Params: {} 2023-10-11T15:18:54.600 DEBUG:boto:Token: None 2023-10-11T15:18:54.600 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ= 2023-10-11T15:18:54.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c764eb884a67dc-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.646 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.647 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.647 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.648 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.648 DEBUG:boto:Method: GET 2023-10-11T15:18:54.649 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.649 DEBUG:boto:Data: 2023-10-11T15:18:54.650 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.651 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.651 DEBUG:boto:Params: {} 2023-10-11T15:18:54.652 DEBUG:boto:Token: None 2023-10-11T15:18:54.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.653 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ= 2023-10-11T15:18:54.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4176425670f5c6-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.656 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.656 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.657 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.657 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.657 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.658 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.658 DEBUG:boto:Data: 2023-10-11T15:18:54.659 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.660 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.660 DEBUG:boto:Params: {} 2023-10-11T15:18:54.660 DEBUG:boto:Token: None 2023-10-11T15:18:54.661 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fMS2IldYWX3cYSlm1+tJPOty3sA= 2023-10-11T15:18:54.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fMS2IldYWX3cYSlm1+tJPOty3sA=', 'Content-Length': '0'} 2023-10-11T15:18:54.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1aa2924ce56947-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.664 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.665 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.665 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.666 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.666 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.666 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.667 DEBUG:boto:Data: 2023-10-11T15:18:54.667 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.668 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.668 DEBUG:boto:Params: {} 2023-10-11T15:18:54.669 DEBUG:boto:Token: None 2023-10-11T15:18:54.669 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fMS2IldYWX3cYSlm1+tJPOty3sA= 2023-10-11T15:18:54.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fMS2IldYWX3cYSlm1+tJPOty3sA=', 'Content-Length': '0'} 2023-10-11T15:18:54.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f88e18c891f49e-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:18:54.673 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.674 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.674 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.674 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.675 DEBUG:boto:Method: GET 2023-10-11T15:18:54.675 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.676 DEBUG:boto:Data: 2023-10-11T15:18:54.676 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.677 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.677 DEBUG:boto:Params: {} 2023-10-11T15:18:54.678 DEBUG:boto:Token: None 2023-10-11T15:18:54.678 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ= 2023-10-11T15:18:54.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f816ce51624b5df-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.682 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.682 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.682 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.683 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.683 DEBUG:boto:Method: GET 2023-10-11T15:18:54.684 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.684 DEBUG:boto:Data: 2023-10-11T15:18:54.685 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.685 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.685 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.686 DEBUG:boto:Params: {} 2023-10-11T15:18:54.686 DEBUG:boto:Token: None 2023-10-11T15:18:54.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:18:54.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ= 2023-10-11T15:18:54.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BhFZxohcrOA3HjvMWyU0QYLL1wQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7319ca8f78520a9-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.690 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.691 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.691 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.691 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.692 DEBUG:boto:Method: GET 2023-10-11T15:18:54.692 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.693 DEBUG:boto:Data: 2023-10-11T15:18:54.693 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.694 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.695 DEBUG:boto:Params: {} 2023-10-11T15:18:54.695 DEBUG:boto:Token: None 2023-10-11T15:18:54.695 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000775978bdc8a7e396-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.699 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.700 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.700 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.701 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.701 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.702 DEBUG:boto:Method: GET 2023-10-11T15:18:54.702 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.703 DEBUG:boto:Data: 2023-10-11T15:18:54.703 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.704 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.705 DEBUG:boto:Params: {} 2023-10-11T15:18:54.705 DEBUG:boto:Token: None 2023-10-11T15:18:54.706 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618ee6b3535df50d-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.710 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.711 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.712 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.712 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.713 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:18:54.713 DEBUG:boto:Data: 2023-10-11T15:18:54.713 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.714 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.715 DEBUG:boto:Params: {} 2023-10-11T15:18:54.715 DEBUG:boto:Token: None 2023-10-11T15:18:54.716 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55 2023-10-11T15:18:54.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW+Lb3RNOdbVN3BNKcLPtpGldkA= 2023-10-11T15:18:54.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW+Lb3RNOdbVN3BNKcLPtpGldkA=', 'Content-Length': '0'} 2023-10-11T15:18:54.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0e946bd22cf94e-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.721 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.721 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.722 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.722 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:18:54.722 DEBUG:boto:Data: 2023-10-11T15:18:54.723 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.724 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.724 DEBUG:boto:Params: {} 2023-10-11T15:18:54.725 DEBUG:boto:Token: None 2023-10-11T15:18:54.725 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55 2023-10-11T15:18:54.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IW+Lb3RNOdbVN3BNKcLPtpGldkA= 2023-10-11T15:18:54.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IW+Lb3RNOdbVN3BNKcLPtpGldkA=', 'Content-Length': '0'} 2023-10-11T15:18:54.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76de5d142616c66-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:18:54.729 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.729 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.730 DEBUG:boto:Method: GET 2023-10-11T15:18:54.730 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:18:54.731 DEBUG:boto:Data: 2023-10-11T15:18:54.731 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.732 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.732 DEBUG:boto:Params: {} 2023-10-11T15:18:54.733 DEBUG:boto:Token: None 2023-10-11T15:18:54.733 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55 2023-10-11T15:18:54.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZYagNhptLRF6B1sdpWiAwx9vsg= 2023-10-11T15:18:54.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZYagNhptLRF6B1sdpWiAwx9vsg=', 'Content-Length': '0'} 2023-10-11T15:18:54.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043911831c7539eb5-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.778 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:18:54.779 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:18:54.780 DEBUG:boto:Method: GET 2023-10-11T15:18:54.780 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:18:54.780 DEBUG:boto:Data: 2023-10-11T15:18:54.781 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.782 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.782 DEBUG:boto:Params: {} 2023-10-11T15:18:54.783 DEBUG:boto:Token: None 2023-10-11T15:18:54.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj55 2023-10-11T15:18:54.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZYagNhptLRF6B1sdpWiAwx9vsg= 2023-10-11T15:18:54.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZYagNhptLRF6B1sdpWiAwx9vsg=', 'Content-Length': '0'} 2023-10-11T15:18:54.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054206262e3b67681-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.787 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.787 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.788 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.788 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.789 DEBUG:boto:Method: GET 2023-10-11T15:18:54.789 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.789 DEBUG:boto:Data: 2023-10-11T15:18:54.790 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.791 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.791 DEBUG:boto:Params: {} 2023-10-11T15:18:54.792 DEBUG:boto:Token: None 2023-10-11T15:18:54.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ce9c2869a2949e-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.796 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.797 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.798 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.798 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.798 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.799 DEBUG:boto:Method: GET 2023-10-11T15:18:54.799 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:18:54.800 DEBUG:boto:Data: 2023-10-11T15:18:54.800 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.801 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.801 DEBUG:boto:Params: {} 2023-10-11T15:18:54.802 DEBUG:boto:Token: None 2023-10-11T15:18:54.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ae73145553d8fc-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.850 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:18:54.852 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.852 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.853 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.853 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.853 DEBUG:boto:Method: GET 2023-10-11T15:18:54.854 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.854 DEBUG:boto:Data: 2023-10-11T15:18:54.855 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.856 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.856 DEBUG:boto:Params: {} 2023-10-11T15:18:54.856 DEBUG:boto:Token: None 2023-10-11T15:18:54.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E= 2023-10-11T15:18:54.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E=', 'Content-Length': '0'} 2023-10-11T15:18:54.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055cc45a054aea07f-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.861 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.861 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.862 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.862 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.863 DEBUG:boto:Method: GET 2023-10-11T15:18:54.863 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.863 DEBUG:boto:Data: 2023-10-11T15:18:54.864 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.865 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.865 DEBUG:boto:Params: {} 2023-10-11T15:18:54.866 DEBUG:boto:Token: None 2023-10-11T15:18:54.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E= 2023-10-11T15:18:54.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E=', 'Content-Length': '0'} 2023-10-11T15:18:54.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb75487c34027d8e-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.870 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.870 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.870 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.871 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.871 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.872 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.872 DEBUG:boto:Data: 2023-10-11T15:18:54.872 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.873 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.874 DEBUG:boto:Params: {} 2023-10-11T15:18:54.874 DEBUG:boto:Token: None 2023-10-11T15:18:54.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R+Ceo4YeIvsBK09NS77wE4+1cyQ= 2023-10-11T15:18:54.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R+Ceo4YeIvsBK09NS77wE4+1cyQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b8338b57a104b0-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.878 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.878 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.879 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.879 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.879 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.880 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.880 DEBUG:boto:Data: 2023-10-11T15:18:54.880 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.881 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.882 DEBUG:boto:Params: {} 2023-10-11T15:18:54.882 DEBUG:boto:Token: None 2023-10-11T15:18:54.882 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R+Ceo4YeIvsBK09NS77wE4+1cyQ= 2023-10-11T15:18:54.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R+Ceo4YeIvsBK09NS77wE4+1cyQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dbf45d2da067e56-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:18:54.886 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.887 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.887 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.887 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.888 DEBUG:boto:Method: GET 2023-10-11T15:18:54.888 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.889 DEBUG:boto:Data: 2023-10-11T15:18:54.889 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.890 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.890 DEBUG:boto:Params: {} 2023-10-11T15:18:54.890 DEBUG:boto:Token: None 2023-10-11T15:18:54.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E= 2023-10-11T15:18:54.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E=', 'Content-Length': '0'} 2023-10-11T15:18:54.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ac559c07a300ce-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.894 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.895 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.895 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.896 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.896 DEBUG:boto:Method: GET 2023-10-11T15:18:54.897 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.897 DEBUG:boto:Data: 2023-10-11T15:18:54.898 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.898 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.899 DEBUG:boto:Params: {} 2023-10-11T15:18:54.899 DEBUG:boto:Token: None 2023-10-11T15:18:54.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:18:54.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E= 2023-10-11T15:18:54.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q4TRwKJLKik/K9oBZqemGNbY15E=', 'Content-Length': '0'} 2023-10-11T15:18:54.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ff342379733dc8d-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.903 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.904 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.904 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.904 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.905 DEBUG:boto:Method: GET 2023-10-11T15:18:54.905 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.906 DEBUG:boto:Data: 2023-10-11T15:18:54.906 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.907 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.907 DEBUG:boto:Params: {} 2023-10-11T15:18:54.908 DEBUG:boto:Token: None 2023-10-11T15:18:54.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128709fbf5252b7f-006526bcde-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.912 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.913 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.913 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.914 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.914 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.914 DEBUG:boto:Method: GET 2023-10-11T15:18:54.915 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.915 DEBUG:boto:Data: 2023-10-11T15:18:54.917 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.918 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.918 DEBUG:boto:Params: {} 2023-10-11T15:18:54.919 DEBUG:boto:Token: None 2023-10-11T15:18:54.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/ 2023-10-11T15:18:54.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w= 2023-10-11T15:18:54.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fes4pcqozBij+Jy3aI+BRGkJz+w=', 'Content-Length': '0'} 2023-10-11T15:18:54.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000234982e1d139e12c-006526bcde-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.923 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:54.924 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.924 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.925 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.925 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:18:54.926 DEBUG:boto:Data: 2023-10-11T15:18:54.926 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.927 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.927 DEBUG:boto:Params: {} 2023-10-11T15:18:54.928 DEBUG:boto:Token: None 2023-10-11T15:18:54.928 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68 2023-10-11T15:18:54.929 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:33CDLNzxiQRMkCCg16nTvmi3NAA= 2023-10-11T15:18:54.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:33CDLNzxiQRMkCCg16nTvmi3NAA=', 'Content-Length': '0'} 2023-10-11T15:18:54.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003685c94e38b0591-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.931 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.932 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.932 DEBUG:boto:Method: HEAD 2023-10-11T15:18:54.933 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:18:54.933 DEBUG:boto:Data: 2023-10-11T15:18:54.933 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.934 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.935 DEBUG:boto:Params: {} 2023-10-11T15:18:54.935 DEBUG:boto:Token: None 2023-10-11T15:18:54.935 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68 2023-10-11T15:18:54.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:33CDLNzxiQRMkCCg16nTvmi3NAA= 2023-10-11T15:18:54.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:33CDLNzxiQRMkCCg16nTvmi3NAA=', 'Content-Length': '0'} 2023-10-11T15:18:54.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e9e418fd55604f-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:18:54.939 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.939 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.940 DEBUG:boto:Method: GET 2023-10-11T15:18:54.940 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:18:54.941 DEBUG:boto:Data: 2023-10-11T15:18:54.941 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.942 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.942 DEBUG:boto:Params: {} 2023-10-11T15:18:54.943 DEBUG:boto:Token: None 2023-10-11T15:18:54.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68 2023-10-11T15:18:54.943 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dQPHmHV6CgnyvaBwzQrA6EYhBQQ= 2023-10-11T15:18:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dQPHmHV6CgnyvaBwzQrA6EYhBQQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb341384230df38c-006526bcde-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.986 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:18:54.987 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:18:54.987 DEBUG:boto:Method: GET 2023-10-11T15:18:54.988 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:18:54.988 DEBUG:boto:Data: 2023-10-11T15:18:54.989 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:54.990 DEBUG:boto:Port: 8000 2023-10-11T15:18:54.990 DEBUG:boto:Params: {} 2023-10-11T15:18:54.991 DEBUG:boto:Token: None 2023-10-11T15:18:54.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:54 GMT /dghqvd-59/obj68 2023-10-11T15:18:54.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dQPHmHV6CgnyvaBwzQrA6EYhBQQ= 2023-10-11T15:18:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dQPHmHV6CgnyvaBwzQrA6EYhBQQ=', 'Content-Length': '0'} 2023-10-11T15:18:54.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db4e74bb8a6d2223-006526bcde-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:54.995 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:54.995 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:54.996 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.996 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.997 DEBUG:boto:Method: GET 2023-10-11T15:18:54.997 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:18:54.997 DEBUG:boto:Data: 2023-10-11T15:18:54.998 DEBUG:boto:Headers: {} 2023-10-11T15:18:54.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:54.999 DEBUG:boto:Port: 8001 2023-10-11T15:18:54.999 DEBUG:boto:Params: {} 2023-10-11T15:18:55.000 DEBUG:boto:Token: None 2023-10-11T15:18:55.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e639fa06396e5d-006526bcdf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.004 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.005 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.005 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.006 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:55.006 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:18:55.007 DEBUG:boto:Method: GET 2023-10-11T15:18:55.007 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:18:55.007 DEBUG:boto:Data: 2023-10-11T15:18:55.008 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.008 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.009 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.009 DEBUG:boto:Params: {} 2023-10-11T15:18:55.010 DEBUG:boto:Token: None 2023-10-11T15:18:55.010 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000703cf536231b3efa-006526bcdf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.014 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:18:55.015 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.016 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.016 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.017 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.017 DEBUG:boto:Method: GET 2023-10-11T15:18:55.018 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.018 DEBUG:boto:Data: 2023-10-11T15:18:55.019 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.019 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.020 DEBUG:boto:Params: {} 2023-10-11T15:18:55.020 DEBUG:boto:Token: None 2023-10-11T15:18:55.021 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4= 2023-10-11T15:18:55.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4=', 'Content-Length': '0'} 2023-10-11T15:18:55.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4dd56476b26cdf-006526bcdf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.025 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.025 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.025 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.026 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.026 DEBUG:boto:Method: GET 2023-10-11T15:18:55.027 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.027 DEBUG:boto:Data: 2023-10-11T15:18:55.028 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.029 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.029 DEBUG:boto:Params: {} 2023-10-11T15:18:55.029 DEBUG:boto:Token: None 2023-10-11T15:18:55.030 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.030 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4= 2023-10-11T15:18:55.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4=', 'Content-Length': '0'} 2023-10-11T15:18:55.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647c6f2ab8dc9119-006526bcdf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.034 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.034 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.035 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.035 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.035 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.036 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.036 DEBUG:boto:Data: 2023-10-11T15:18:55.037 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.037 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.037 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.038 DEBUG:boto:Params: {} 2023-10-11T15:18:55.038 DEBUG:boto:Token: None 2023-10-11T15:18:55.039 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F5evqJyTsSVW2g27L+PYCABbQo8= 2023-10-11T15:18:55.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F5evqJyTsSVW2g27L+PYCABbQo8=', 'Content-Length': '0'} 2023-10-11T15:18:55.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075cb8e4db6aacd9e-006526bcdf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.042 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.043 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.043 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.043 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.044 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.044 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.045 DEBUG:boto:Data: 2023-10-11T15:18:55.045 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.046 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.046 DEBUG:boto:Params: {} 2023-10-11T15:18:55.047 DEBUG:boto:Token: None 2023-10-11T15:18:55.047 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F5evqJyTsSVW2g27L+PYCABbQo8= 2023-10-11T15:18:55.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F5evqJyTsSVW2g27L+PYCABbQo8=', 'Content-Length': '0'} 2023-10-11T15:18:55.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c8c458fe82620bd-006526bcdf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:18:55.051 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.052 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.052 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.052 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.053 DEBUG:boto:Method: GET 2023-10-11T15:18:55.053 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.054 DEBUG:boto:Data: 2023-10-11T15:18:55.054 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.055 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.055 DEBUG:boto:Params: {} 2023-10-11T15:18:55.056 DEBUG:boto:Token: None 2023-10-11T15:18:55.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4= 2023-10-11T15:18:55.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4=', 'Content-Length': '0'} 2023-10-11T15:18:55.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f6953747a55061-006526bcdf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.060 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.061 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.061 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.061 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.062 DEBUG:boto:Method: GET 2023-10-11T15:18:55.062 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.063 DEBUG:boto:Data: 2023-10-11T15:18:55.063 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.064 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.064 DEBUG:boto:Params: {} 2023-10-11T15:18:55.065 DEBUG:boto:Token: None 2023-10-11T15:18:55.065 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:18:55.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4= 2023-10-11T15:18:55.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lRZI6DnaEzgIPmDHmb6UZ2rLaN4=', 'Content-Length': '0'} 2023-10-11T15:18:55.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fac867650aeda13-006526bcdf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.069 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.070 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.070 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.070 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.071 DEBUG:boto:Method: GET 2023-10-11T15:18:55.071 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.072 DEBUG:boto:Data: 2023-10-11T15:18:55.072 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.073 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.073 DEBUG:boto:Params: {} 2023-10-11T15:18:55.074 DEBUG:boto:Token: None 2023-10-11T15:18:55.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fed847b4575edc9-006526bcdf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.078 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.079 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.080 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.080 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.080 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.081 DEBUG:boto:Method: GET 2023-10-11T15:18:55.081 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.082 DEBUG:boto:Data: 2023-10-11T15:18:55.082 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.083 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.083 DEBUG:boto:Params: {} 2023-10-11T15:18:55.084 DEBUG:boto:Token: None 2023-10-11T15:18:55.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ac74c32d9ce5a4-006526bcdf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.089 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.090 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.090 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.091 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.091 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:18:55.091 DEBUG:boto:Data: 2023-10-11T15:18:55.092 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.092 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.093 DEBUG:boto:Params: {} 2023-10-11T15:18:55.093 DEBUG:boto:Token: None 2023-10-11T15:18:55.094 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85 2023-10-11T15:18:55.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O+nZ2GmdII+D/yeTU0M0ehS4mYs= 2023-10-11T15:18:55.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O+nZ2GmdII+D/yeTU0M0ehS4mYs=', 'Content-Length': '0'} 2023-10-11T15:18:55.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e269b68e0afd296c-006526bcdf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.097 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.097 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.098 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.098 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:18:55.099 DEBUG:boto:Data: 2023-10-11T15:18:55.099 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.100 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.100 DEBUG:boto:Params: {} 2023-10-11T15:18:55.101 DEBUG:boto:Token: None 2023-10-11T15:18:55.101 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85 2023-10-11T15:18:55.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O+nZ2GmdII+D/yeTU0M0ehS4mYs= 2023-10-11T15:18:55.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O+nZ2GmdII+D/yeTU0M0ehS4mYs=', 'Content-Length': '0'} 2023-10-11T15:18:55.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003586f96d0aa79ab5-006526bcdf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:18:55.105 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.105 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.106 DEBUG:boto:Method: GET 2023-10-11T15:18:55.106 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:18:55.106 DEBUG:boto:Data: 2023-10-11T15:18:55.107 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.108 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.108 DEBUG:boto:Params: {} 2023-10-11T15:18:55.109 DEBUG:boto:Token: None 2023-10-11T15:18:55.109 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85 2023-10-11T15:18:55.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AB8OqHgo//DFT/1wx4xgQh/2jvY= 2023-10-11T15:18:55.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AB8OqHgo//DFT/1wx4xgQh/2jvY=', 'Content-Length': '0'} 2023-10-11T15:18:55.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300dd5db786fd889-006526bcdf-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.113 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:18:55.113 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:18:55.114 DEBUG:boto:Method: GET 2023-10-11T15:18:55.114 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:18:55.114 DEBUG:boto:Data: 2023-10-11T15:18:55.115 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.116 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.116 DEBUG:boto:Params: {} 2023-10-11T15:18:55.117 DEBUG:boto:Token: None 2023-10-11T15:18:55.117 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj85 2023-10-11T15:18:55.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AB8OqHgo//DFT/1wx4xgQh/2jvY= 2023-10-11T15:18:55.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AB8OqHgo//DFT/1wx4xgQh/2jvY=', 'Content-Length': '0'} 2023-10-11T15:18:55.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40e3a8dd4165ced-006526bcdf-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.162 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.163 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.163 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.164 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.164 DEBUG:boto:Method: GET 2023-10-11T15:18:55.164 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.165 DEBUG:boto:Data: 2023-10-11T15:18:55.165 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:18:55.166 DEBUG:boto:Port: 8001 2023-10-11T15:18:55.166 DEBUG:boto:Params: {} 2023-10-11T15:18:55.167 DEBUG:boto:Token: None 2023-10-11T15:18:55.167 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ac2482147126bb1-006526bcdf-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.171 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.172 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.172 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.173 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.173 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.173 DEBUG:boto:Method: GET 2023-10-11T15:18:55.174 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:18:55.174 DEBUG:boto:Data: 2023-10-11T15:18:55.174 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.175 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.176 DEBUG:boto:Params: {} 2023-10-11T15:18:55.176 DEBUG:boto:Token: None 2023-10-11T15:18:55.176 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI= 2023-10-11T15:18:55.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K9LEdj+6W5vhRHiwXJHW5rjF2gI=', 'Content-Length': '0'} 2023-10-11T15:18:55.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6ce098b867970f-006526bcdf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.181 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:18:55.182 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:18:55.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:18:55.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:18:55.183 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:18:55.240 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:18:55.241 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:18:55.284 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:18:55.285 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:18:55.290 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 10 2023-10-11T15:18:55.333 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.334 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.334 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.335 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:55.335 DEBUG:boto:Data: 2023-10-11T15:18:55.335 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.336 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.336 DEBUG:boto:Params: {} 2023-10-11T15:18:55.337 DEBUG:boto:Token: None 2023-10-11T15:18:55.337 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE= 2023-10-11T15:18:55.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE=', 'Content-Length': '0'} 2023-10-11T15:18:55.341 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d41a4d004504edb2-006526bcdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.342 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:18:55.342 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:18:55.343 DEBUG:boto:Method: PUT 2023-10-11T15:18:55.343 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:18:55.344 DEBUG:boto:Data: 2023-10-11T15:18:55.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:18:55.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.345 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.345 DEBUG:boto:Params: {} 2023-10-11T15:18:55.346 DEBUG:boto:Token: None 2023-10-11T15:18:55.346 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj19 2023-10-11T15:18:55.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GkuP+yaUxNIfRcJUhpn8vrfwP3I= 2023-10-11T15:18:55.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GkuP+yaUxNIfRcJUhpn8vrfwP3I='} 2023-10-11T15:18:55.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cc3b1d81494a83e-006526bcdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.352 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.353 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.353 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.354 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:55.354 DEBUG:boto:Data: 2023-10-11T15:18:55.355 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.355 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.356 DEBUG:boto:Params: {} 2023-10-11T15:18:55.356 DEBUG:boto:Token: None 2023-10-11T15:18:55.357 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE= 2023-10-11T15:18:55.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE=', 'Content-Length': '0'} 2023-10-11T15:18:55.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045b51a15d2b3d8cf-006526bcdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.361 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:18:55.361 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:18:55.362 DEBUG:boto:Method: PUT 2023-10-11T15:18:55.362 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:18:55.363 DEBUG:boto:Data: 2023-10-11T15:18:55.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:18:55.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.364 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.364 DEBUG:boto:Params: {} 2023-10-11T15:18:55.365 DEBUG:boto:Token: None 2023-10-11T15:18:55.365 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj38 2023-10-11T15:18:55.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fEPMBw3VHFOwylySBqFJOWuxbg0= 2023-10-11T15:18:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fEPMBw3VHFOwylySBqFJOWuxbg0='} 2023-10-11T15:18:55.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6875ca2078b7bd6-006526bcdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.371 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.371 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.372 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.372 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:55.372 DEBUG:boto:Data: 2023-10-11T15:18:55.373 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.373 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.374 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.374 DEBUG:boto:Params: {} 2023-10-11T15:18:55.374 DEBUG:boto:Token: None 2023-10-11T15:18:55.375 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.375 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE= 2023-10-11T15:18:55.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE=', 'Content-Length': '0'} 2023-10-11T15:18:55.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff2c6ff5805aa25-006526bcdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.379 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:18:55.379 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:18:55.380 DEBUG:boto:Method: PUT 2023-10-11T15:18:55.380 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:18:55.380 DEBUG:boto:Data: 2023-10-11T15:18:55.381 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:18:55.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.382 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.382 DEBUG:boto:Params: {} 2023-10-11T15:18:55.383 DEBUG:boto:Token: None 2023-10-11T15:18:55.383 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj57 2023-10-11T15:18:55.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y0mkczJnqVi0J6zfKyIcQdcOrWg= 2023-10-11T15:18:55.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y0mkczJnqVi0J6zfKyIcQdcOrWg='} 2023-10-11T15:18:55.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027e7b4841a5a6662-006526bcdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.431 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.431 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.432 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.432 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:55.433 DEBUG:boto:Data: 2023-10-11T15:18:55.433 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.434 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.434 DEBUG:boto:Params: {} 2023-10-11T15:18:55.435 DEBUG:boto:Token: None 2023-10-11T15:18:55.435 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.436 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE= 2023-10-11T15:18:55.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE=', 'Content-Length': '0'} 2023-10-11T15:18:55.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e28db20fff397e6-006526bcdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.440 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:18:55.440 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:18:55.441 DEBUG:boto:Method: PUT 2023-10-11T15:18:55.441 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:18:55.441 DEBUG:boto:Data: 2023-10-11T15:18:55.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:18:55.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.443 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.443 DEBUG:boto:Params: {} 2023-10-11T15:18:55.443 DEBUG:boto:Token: None 2023-10-11T15:18:55.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj76 2023-10-11T15:18:55.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qyra6YfnwukmeFUThJyJ9OG16FU= 2023-10-11T15:18:55.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qyra6YfnwukmeFUThJyJ9OG16FU='} 2023-10-11T15:18:55.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001604063d6c5b9e82-006526bcdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.450 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:18:55.451 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:18:55.451 DEBUG:boto:Method: HEAD 2023-10-11T15:18:55.451 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:18:55.452 DEBUG:boto:Data: 2023-10-11T15:18:55.452 DEBUG:boto:Headers: {} 2023-10-11T15:18:55.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.453 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.453 DEBUG:boto:Params: {} 2023-10-11T15:18:55.454 DEBUG:boto:Token: None 2023-10-11T15:18:55.454 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/ 2023-10-11T15:18:55.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE= 2023-10-11T15:18:55.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6q6iXDG1zQeNX54heFZPnhUvdAE=', 'Content-Length': '0'} 2023-10-11T15:18:55.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009943055f19219335-006526bcdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.458 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:18:55.458 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:18:55.459 DEBUG:boto:Method: PUT 2023-10-11T15:18:55.459 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:18:55.459 DEBUG:boto:Data: 2023-10-11T15:18:55.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:18:55.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:18:55.461 DEBUG:boto:Port: 8000 2023-10-11T15:18:55.461 DEBUG:boto:Params: {} 2023-10-11T15:18:55.461 DEBUG:boto:Token: None 2023-10-11T15:18:55.462 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:18:55 GMT /dghqvd-59/obj95 2023-10-11T15:18:55.462 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:noc+oB7j4AD4tK2HxMsufuBtAPQ= 2023-10-11T15:18:55.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:18:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:noc+oB7j4AD4tK2HxMsufuBtAPQ='} 2023-10-11T15:18:55.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f42dd91c0b8311b-006526bcdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:18:55.507 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:18:55.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:18:55.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:18:55.509 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:18:55.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:18:55.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:18:55.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:18:55.597+0000 7fda583f9780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T15:19:00.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:00.599+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:00.602 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-11T15:19:00.602 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:05.603 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:05.602+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:05.604 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-11T15:19:05.604 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:10.605 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:10.603+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:10.605 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-11T15:19:10.606 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:15.605+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:15.607 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-10-11T15:19:15.607 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:20.608 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:20.607+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:20.609 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:20.609 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:25.610 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:25.608+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:25.611 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:25.611 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:30.612 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:30.610+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:30.612 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:30.612 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:35.613 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:35.612+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:35.614 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:35.614 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:40.615 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:40.613+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:40.615 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:40.616 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:45.616 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:45.615+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:45.617 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:45.617 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:50.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:50.616+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:50.618 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:50.619 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:19:55.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:19:55.617+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:19:55.620 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000002.79.6, , , , , , , , , , , ] 2023-10-11T15:19:55.620 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:00.621 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:00.619+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:00.621 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:00.622 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:05.620+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:05.623 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:05.623 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:10.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:10.622+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:10.625 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:10.625 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:15.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:15.623+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:15.626 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:15.627 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:20.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:20.625+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:20.628 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:20.628 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:25.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:25.627+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:25.630 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:25.630 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:30.631 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:30.628+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:30.631 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:30.632 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:35.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:35.630+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:35.633 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:35.633 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:40.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:40.633+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:40.635 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:40.635 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:45.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:45.634+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:45.636 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:45.637 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:50.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:50.636+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:50.638 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:50.638 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:55.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:20:55.638+0000 7fda583f9780 1 waiting for incremental sync to catch up: 2023-10-11T15:20:55.640 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:20:55.640 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:00.639+0000 7fda583f9780 1 bucket sync caught up with source: 2023-10-11T15:21:00.641 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.63.6, 00000000004.83.6, , , , , , , , , , , ] 2023-10-11T15:21:00.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:00.639+0000 7fda583f9780 0 bucket checkpoint complete 2023-10-11T15:21:00.658 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:21:00.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:00.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:00.660 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:21:00.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:00.723 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:00.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:00.758+0000 7fd64bb7b780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:00.760 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, , 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:00.760 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, 00000000001.99.6, 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:05.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:05.759+0000 7fd64bb7b780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:05.761 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, , 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:05.761 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, 00000000001.99.6, 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:10.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:10.760+0000 7fd64bb7b780 1 bucket sync caught up with source: 2023-10-11T15:21:10.762 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, 00000000001.99.6, 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:10.763 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000001.226.6, 00000000001.287.6, 00000000001.99.6, 00000000001.284.6, , , 00000000002.232.6, 00000000004.171.6, 00000000003.284.6] 2023-10-11T15:21:10.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:10.760+0000 7fd64bb7b780 0 bucket checkpoint complete 2023-10-11T15:21:10.780 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:21:10.780 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.781 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.781 DEBUG:boto:Method: HEAD 2023-10-11T15:21:10.782 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:10.782 DEBUG:boto:Data: 2023-10-11T15:21:10.782 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.783 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.783 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.784 DEBUG:boto:Params: {} 2023-10-11T15:21:10.784 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:21:10.785 DEBUG:boto:Token: None 2023-10-11T15:21:10.785 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/ 2023-10-11T15:21:10.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PIPU+S0M2RgUQH8PA0Vr9TCYoeI= 2023-10-11T15:21:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PIPU+S0M2RgUQH8PA0Vr9TCYoeI=', 'Content-Length': '0'} 2023-10-11T15:21:10.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056775f3d143f1deb-006526bd66-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.791 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.792 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.792 DEBUG:boto:Method: HEAD 2023-10-11T15:21:10.793 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:10.793 DEBUG:boto:Data: 2023-10-11T15:21:10.793 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.794 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.797 DEBUG:boto:Params: {} 2023-10-11T15:21:10.798 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:21:10.798 DEBUG:boto:Token: None 2023-10-11T15:21:10.799 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/ 2023-10-11T15:21:10.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PIPU+S0M2RgUQH8PA0Vr9TCYoeI= 2023-10-11T15:21:10.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PIPU+S0M2RgUQH8PA0Vr9TCYoeI=', 'Content-Length': '0'} 2023-10-11T15:21:10.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd78955d794cd510-006526bd66-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:21:10.805 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.806 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.806 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.806 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.807 DEBUG:boto:Method: GET 2023-10-11T15:21:10.807 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.808 DEBUG:boto:Data: 2023-10-11T15:21:10.808 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.809 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.809 DEBUG:boto:Params: {} 2023-10-11T15:21:10.810 DEBUG:boto:Token: None 2023-10-11T15:21:10.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/?versions 2023-10-11T15:21:10.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0= 2023-10-11T15:21:10.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0=', 'Content-Length': '0'} 2023-10-11T15:21:10.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6827b3626eaff3-006526bd66-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.814 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:10.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:21:10.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:21:10.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:21:10.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:21:10.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:21:10.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:21:10.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:21:10.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:21:10.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:21:10.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:21:10.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:21:10.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:21:10.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:21:10.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:21:10.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:21:10.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:21:10.822 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.822 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.823 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.823 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.824 DEBUG:boto:Method: GET 2023-10-11T15:21:10.824 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.824 DEBUG:boto:Data: 2023-10-11T15:21:10.825 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.826 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.826 DEBUG:boto:Params: {} 2023-10-11T15:21:10.827 DEBUG:boto:Token: None 2023-10-11T15:21:10.827 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/?versions 2023-10-11T15:21:10.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0= 2023-10-11T15:21:10.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0=', 'Content-Length': '0'} 2023-10-11T15:21:10.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83abeddb0aa44de-006526bd66-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.832 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:10.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:21:10.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:21:10.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:21:10.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:21:10.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:21:10.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:21:10.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:21:10.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:21:10.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:21:10.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:21:10.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:21:10.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:21:10.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:21:10.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:21:10.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:21:10.839 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.839 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.839 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.840 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.840 DEBUG:boto:Method: GET 2023-10-11T15:21:10.841 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.841 DEBUG:boto:Data: 2023-10-11T15:21:10.841 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.842 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.843 DEBUG:boto:Params: {} 2023-10-11T15:21:10.843 DEBUG:boto:Token: None 2023-10-11T15:21:10.844 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/?versions 2023-10-11T15:21:10.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0= 2023-10-11T15:21:10.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0=', 'Content-Length': '0'} 2023-10-11T15:21:10.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb18c0685fa1c0c-006526bd66-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.848 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:10.849 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:10.850 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:10.850 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.850 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.851 DEBUG:boto:Method: GET 2023-10-11T15:21:10.851 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:10.852 DEBUG:boto:Data: 2023-10-11T15:21:10.852 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.853 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.853 DEBUG:boto:Params: {} 2023-10-11T15:21:10.854 DEBUG:boto:Token: None 2023-10-11T15:21:10.854 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/?versions 2023-10-11T15:21:10.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0= 2023-10-11T15:21:10.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fo6lYdO4Ff9Hhz3xUGZy0hyjNJ0=', 'Content-Length': '0'} 2023-10-11T15:21:10.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b67ab0878b23cf-006526bd66-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.859 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:10.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:10.861 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.861 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.862 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.862 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.862 DEBUG:boto:Method: GET 2023-10-11T15:21:10.863 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.863 DEBUG:boto:Data: 2023-10-11T15:21:10.864 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.865 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.865 DEBUG:boto:Params: {} 2023-10-11T15:21:10.866 DEBUG:boto:Token: None 2023-10-11T15:21:10.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c= 2023-10-11T15:21:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c=', 'Content-Length': '0'} 2023-10-11T15:21:10.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c051c3cc6ed752b9-006526bd66-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.870 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.871 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.871 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.871 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.872 DEBUG:boto:Method: GET 2023-10-11T15:21:10.872 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.873 DEBUG:boto:Data: 2023-10-11T15:21:10.873 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.874 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.874 DEBUG:boto:Params: {} 2023-10-11T15:21:10.875 DEBUG:boto:Token: None 2023-10-11T15:21:10.875 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c= 2023-10-11T15:21:10.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c=', 'Content-Length': '0'} 2023-10-11T15:21:10.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead9bd81ceedcef6-006526bd66-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.922 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.923 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.923 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.924 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.924 DEBUG:boto:Method: HEAD 2023-10-11T15:21:10.925 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.925 DEBUG:boto:Data: 2023-10-11T15:21:10.926 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.926 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.926 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.927 DEBUG:boto:Params: {} 2023-10-11T15:21:10.927 DEBUG:boto:Token: None 2023-10-11T15:21:10.928 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q5jC6rpOzC3KiYo2CvZLWtUkrDQ= 2023-10-11T15:21:10.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q5jC6rpOzC3KiYo2CvZLWtUkrDQ=', 'Content-Length': '0'} 2023-10-11T15:21:10.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052ee1f0652b6a70a-006526bd66-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.931 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.932 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.932 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.933 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.933 DEBUG:boto:Method: HEAD 2023-10-11T15:21:10.934 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.934 DEBUG:boto:Data: 2023-10-11T15:21:10.934 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.935 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.936 DEBUG:boto:Params: {} 2023-10-11T15:21:10.936 DEBUG:boto:Token: None 2023-10-11T15:21:10.937 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q5jC6rpOzC3KiYo2CvZLWtUkrDQ= 2023-10-11T15:21:10.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q5jC6rpOzC3KiYo2CvZLWtUkrDQ=', 'Content-Length': '0'} 2023-10-11T15:21:10.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d932f2935adf086a-006526bd66-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:10.940 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.941 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.941 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.942 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.942 DEBUG:boto:Method: GET 2023-10-11T15:21:10.943 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.943 DEBUG:boto:Data: 2023-10-11T15:21:10.944 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:10.944 DEBUG:boto:Port: 8001 2023-10-11T15:21:10.945 DEBUG:boto:Params: {} 2023-10-11T15:21:10.945 DEBUG:boto:Token: None 2023-10-11T15:21:10.946 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c= 2023-10-11T15:21:10.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c=', 'Content-Length': '0'} 2023-10-11T15:21:10.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002241af9683d76b70-006526bd66-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:10.990 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:10.991 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:10.991 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.992 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.992 DEBUG:boto:Method: GET 2023-10-11T15:21:10.993 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.993 DEBUG:boto:Data: 2023-10-11T15:21:10.994 DEBUG:boto:Headers: {} 2023-10-11T15:21:10.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:10.994 DEBUG:boto:Port: 8000 2023-10-11T15:21:10.995 DEBUG:boto:Params: {} 2023-10-11T15:21:10.995 DEBUG:boto:Token: None 2023-10-11T15:21:10.996 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:10 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:10.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c= 2023-10-11T15:21:10.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uDEO3OXUt70tDcsS9ZlRDyChj/c=', 'Content-Length': '0'} 2023-10-11T15:21:11.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe3055183587f973-006526bd66-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.002 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.002 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.003 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.003 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.004 DEBUG:boto:Method: GET 2023-10-11T15:21:11.004 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.005 DEBUG:boto:Data: 2023-10-11T15:21:11.005 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.006 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.006 DEBUG:boto:Params: {} 2023-10-11T15:21:11.007 DEBUG:boto:Token: None 2023-10-11T15:21:11.007 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc5b46a2678b46a-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.011 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.012 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.013 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.013 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.013 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.014 DEBUG:boto:Method: GET 2023-10-11T15:21:11.014 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.015 DEBUG:boto:Data: 2023-10-11T15:21:11.015 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.016 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.016 DEBUG:boto:Params: {} 2023-10-11T15:21:11.017 DEBUG:boto:Token: None 2023-10-11T15:21:11.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000468bacb75dc2e0da-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.022 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.023 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:11.023 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:11.023 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.024 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:11.024 DEBUG:boto:Data: 2023-10-11T15:21:11.025 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.025 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.026 DEBUG:boto:Params: {} 2023-10-11T15:21:11.026 DEBUG:boto:Token: None 2023-10-11T15:21:11.027 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj11 2023-10-11T15:21:11.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AC9yp748661Scp1PUBS8iRFZ7Hg= 2023-10-11T15:21:11.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AC9yp748661Scp1PUBS8iRFZ7Hg=', 'Content-Length': '0'} 2023-10-11T15:21:11.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef35cb752cad7652-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.030 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:11.031 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:11.031 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.032 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:11.032 DEBUG:boto:Data: 2023-10-11T15:21:11.033 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.033 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.034 DEBUG:boto:Params: {} 2023-10-11T15:21:11.034 DEBUG:boto:Token: None 2023-10-11T15:21:11.035 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj11 2023-10-11T15:21:11.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AC9yp748661Scp1PUBS8iRFZ7Hg= 2023-10-11T15:21:11.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AC9yp748661Scp1PUBS8iRFZ7Hg=', 'Content-Length': '0'} 2023-10-11T15:21:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000810e34d0932445b0-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:11.039 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:11.039 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:11.040 DEBUG:boto:Method: GET 2023-10-11T15:21:11.040 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:11.040 DEBUG:boto:Data: 2023-10-11T15:21:11.041 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.041 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.042 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.042 DEBUG:boto:Params: {} 2023-10-11T15:21:11.043 DEBUG:boto:Token: None 2023-10-11T15:21:11.043 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj11 2023-10-11T15:21:11.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a77x1Uj7ZZbtbmRYmVeULOpDAw4= 2023-10-11T15:21:11.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a77x1Uj7ZZbtbmRYmVeULOpDAw4=', 'Content-Length': '0'} 2023-10-11T15:21:11.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be990e616ac4199-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.047 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:11.047 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:11.048 DEBUG:boto:Method: GET 2023-10-11T15:21:11.048 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:11.049 DEBUG:boto:Data: 2023-10-11T15:21:11.049 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.050 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.050 DEBUG:boto:Params: {} 2023-10-11T15:21:11.051 DEBUG:boto:Token: None 2023-10-11T15:21:11.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj11 2023-10-11T15:21:11.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a77x1Uj7ZZbtbmRYmVeULOpDAw4= 2023-10-11T15:21:11.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a77x1Uj7ZZbtbmRYmVeULOpDAw4=', 'Content-Length': '0'} 2023-10-11T15:21:11.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad207a6aaa0aa10e-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.055 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.055 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.056 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.056 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.057 DEBUG:boto:Method: GET 2023-10-11T15:21:11.057 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.057 DEBUG:boto:Data: 2023-10-11T15:21:11.058 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.059 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.059 DEBUG:boto:Params: {} 2023-10-11T15:21:11.060 DEBUG:boto:Token: None 2023-10-11T15:21:11.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd191b988264d954-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.110 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.111 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.112 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.112 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.112 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.113 DEBUG:boto:Method: GET 2023-10-11T15:21:11.113 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:11.114 DEBUG:boto:Data: 2023-10-11T15:21:11.114 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.115 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.115 DEBUG:boto:Params: {} 2023-10-11T15:21:11.116 DEBUG:boto:Token: None 2023-10-11T15:21:11.116 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046fc9cfe495caad4-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.121 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:11.122 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.122 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.123 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.123 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.124 DEBUG:boto:Method: GET 2023-10-11T15:21:11.124 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.125 DEBUG:boto:Data: 2023-10-11T15:21:11.125 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.125 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.126 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.126 DEBUG:boto:Params: {} 2023-10-11T15:21:11.127 DEBUG:boto:Token: None 2023-10-11T15:21:11.127 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c= 2023-10-11T15:21:11.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c=', 'Content-Length': '0'} 2023-10-11T15:21:11.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a4b5aad4956a7e-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.131 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.131 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.132 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.132 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.133 DEBUG:boto:Method: GET 2023-10-11T15:21:11.133 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.133 DEBUG:boto:Data: 2023-10-11T15:21:11.134 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.135 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.135 DEBUG:boto:Params: {} 2023-10-11T15:21:11.136 DEBUG:boto:Token: None 2023-10-11T15:21:11.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c= 2023-10-11T15:21:11.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c=', 'Content-Length': '0'} 2023-10-11T15:21:11.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032f99cec777661b-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.140 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.140 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.141 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.141 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.141 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.142 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.142 DEBUG:boto:Data: 2023-10-11T15:21:11.143 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.143 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.143 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.144 DEBUG:boto:Params: {} 2023-10-11T15:21:11.144 DEBUG:boto:Token: None 2023-10-11T15:21:11.145 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.145 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I6rCThxNWrQKwoZSuVVCcBh3d84= 2023-10-11T15:21:11.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I6rCThxNWrQKwoZSuVVCcBh3d84=', 'Content-Length': '0'} 2023-10-11T15:21:11.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a2a4111cf97b45-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.148 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.149 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.149 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.149 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.150 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.150 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.151 DEBUG:boto:Data: 2023-10-11T15:21:11.151 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.152 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.152 DEBUG:boto:Params: {} 2023-10-11T15:21:11.153 DEBUG:boto:Token: None 2023-10-11T15:21:11.153 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I6rCThxNWrQKwoZSuVVCcBh3d84= 2023-10-11T15:21:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I6rCThxNWrQKwoZSuVVCcBh3d84=', 'Content-Length': '0'} 2023-10-11T15:21:11.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094cbcda54d278682-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:11.157 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.158 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.158 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.158 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.159 DEBUG:boto:Method: GET 2023-10-11T15:21:11.159 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.160 DEBUG:boto:Data: 2023-10-11T15:21:11.160 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.161 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.161 DEBUG:boto:Params: {} 2023-10-11T15:21:11.162 DEBUG:boto:Token: None 2023-10-11T15:21:11.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c= 2023-10-11T15:21:11.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c=', 'Content-Length': '0'} 2023-10-11T15:21:11.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d781b455aac99067-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.166 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.166 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.167 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.167 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.168 DEBUG:boto:Method: GET 2023-10-11T15:21:11.168 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.169 DEBUG:boto:Data: 2023-10-11T15:21:11.169 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.170 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.170 DEBUG:boto:Params: {} 2023-10-11T15:21:11.171 DEBUG:boto:Token: None 2023-10-11T15:21:11.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:11.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c= 2023-10-11T15:21:11.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BMzErp0VmsuTmsa3Bf/Ta+beP/c=', 'Content-Length': '0'} 2023-10-11T15:21:11.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d7d0c7603ae5d2-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.222 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.223 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.223 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.224 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.224 DEBUG:boto:Method: GET 2023-10-11T15:21:11.225 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.225 DEBUG:boto:Data: 2023-10-11T15:21:11.226 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.226 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.227 DEBUG:boto:Params: {} 2023-10-11T15:21:11.227 DEBUG:boto:Token: None 2023-10-11T15:21:11.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b81771d73624df4-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.232 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.233 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.233 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.234 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.234 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.235 DEBUG:boto:Method: GET 2023-10-11T15:21:11.235 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.235 DEBUG:boto:Data: 2023-10-11T15:21:11.236 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.237 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.237 DEBUG:boto:Params: {} 2023-10-11T15:21:11.237 DEBUG:boto:Token: None 2023-10-11T15:21:11.238 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006481d873ff4d9cde-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.243 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.243 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.244 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.244 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.245 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:11.245 DEBUG:boto:Data: 2023-10-11T15:21:11.245 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.246 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.247 DEBUG:boto:Params: {} 2023-10-11T15:21:11.247 DEBUG:boto:Token: None 2023-10-11T15:21:11.247 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17 2023-10-11T15:21:11.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lzIPdPjQKd5lxFpiMDeTX6WzseU= 2023-10-11T15:21:11.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lzIPdPjQKd5lxFpiMDeTX6WzseU=', 'Content-Length': '0'} 2023-10-11T15:21:11.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb8a3877d105c13-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.251 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.251 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.252 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.252 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:11.252 DEBUG:boto:Data: 2023-10-11T15:21:11.253 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.254 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.254 DEBUG:boto:Params: {} 2023-10-11T15:21:11.254 DEBUG:boto:Token: None 2023-10-11T15:21:11.255 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17 2023-10-11T15:21:11.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lzIPdPjQKd5lxFpiMDeTX6WzseU= 2023-10-11T15:21:11.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lzIPdPjQKd5lxFpiMDeTX6WzseU=', 'Content-Length': '0'} 2023-10-11T15:21:11.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000df3933facc25ed0-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:11.259 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.259 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.260 DEBUG:boto:Method: GET 2023-10-11T15:21:11.260 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:11.261 DEBUG:boto:Data: 2023-10-11T15:21:11.261 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.262 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.262 DEBUG:boto:Params: {} 2023-10-11T15:21:11.263 DEBUG:boto:Token: None 2023-10-11T15:21:11.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17 2023-10-11T15:21:11.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kVuS2dvp2/LHyMUkAEzOBWNp9II= 2023-10-11T15:21:11.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kVuS2dvp2/LHyMUkAEzOBWNp9II=', 'Content-Length': '0'} 2023-10-11T15:21:11.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543bfaff8ac8a95a-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.310 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:11.311 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:11.312 DEBUG:boto:Method: GET 2023-10-11T15:21:11.312 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:11.312 DEBUG:boto:Data: 2023-10-11T15:21:11.313 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.314 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.314 DEBUG:boto:Params: {} 2023-10-11T15:21:11.315 DEBUG:boto:Token: None 2023-10-11T15:21:11.315 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj17 2023-10-11T15:21:11.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kVuS2dvp2/LHyMUkAEzOBWNp9II= 2023-10-11T15:21:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kVuS2dvp2/LHyMUkAEzOBWNp9II=', 'Content-Length': '0'} 2023-10-11T15:21:11.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c13bd2ab7841b11-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.319 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.320 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.320 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.321 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.321 DEBUG:boto:Method: GET 2023-10-11T15:21:11.321 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.322 DEBUG:boto:Data: 2023-10-11T15:21:11.322 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.323 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.324 DEBUG:boto:Params: {} 2023-10-11T15:21:11.324 DEBUG:boto:Token: None 2023-10-11T15:21:11.325 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e29f528c094f3b0f-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.329 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.330 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.330 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.331 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.331 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.332 DEBUG:boto:Method: GET 2023-10-11T15:21:11.332 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:11.333 DEBUG:boto:Data: 2023-10-11T15:21:11.333 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.334 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.335 DEBUG:boto:Params: {} 2023-10-11T15:21:11.335 DEBUG:boto:Token: None 2023-10-11T15:21:11.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001566e149d509272-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.386 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:11.388 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.388 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.389 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.389 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.390 DEBUG:boto:Method: GET 2023-10-11T15:21:11.390 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.391 DEBUG:boto:Data: 2023-10-11T15:21:11.391 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.392 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.393 DEBUG:boto:Params: {} 2023-10-11T15:21:11.393 DEBUG:boto:Token: None 2023-10-11T15:21:11.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM= 2023-10-11T15:21:11.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM=', 'Content-Length': '0'} 2023-10-11T15:21:11.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b675e05d0fe9d7e-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.398 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.398 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.399 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.399 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.400 DEBUG:boto:Method: GET 2023-10-11T15:21:11.400 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.401 DEBUG:boto:Data: 2023-10-11T15:21:11.401 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.402 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.402 DEBUG:boto:Params: {} 2023-10-11T15:21:11.403 DEBUG:boto:Token: None 2023-10-11T15:21:11.403 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM= 2023-10-11T15:21:11.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM=', 'Content-Length': '0'} 2023-10-11T15:21:11.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22c927c4fdbd808-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.410 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.410 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.411 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.411 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.412 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.412 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.412 DEBUG:boto:Data: 2023-10-11T15:21:11.413 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.414 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.415 DEBUG:boto:Params: {} 2023-10-11T15:21:11.415 DEBUG:boto:Token: None 2023-10-11T15:21:11.415 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.416 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5e8itQ0f1K7KF1iHkQpyNswRwEA= 2023-10-11T15:21:11.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5e8itQ0f1K7KF1iHkQpyNswRwEA=', 'Content-Length': '0'} 2023-10-11T15:21:11.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94b5c8da0293ce5-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.419 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.420 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.420 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.421 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.421 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.421 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.422 DEBUG:boto:Data: 2023-10-11T15:21:11.422 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.423 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.424 DEBUG:boto:Params: {} 2023-10-11T15:21:11.425 DEBUG:boto:Token: None 2023-10-11T15:21:11.426 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5e8itQ0f1K7KF1iHkQpyNswRwEA= 2023-10-11T15:21:11.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5e8itQ0f1K7KF1iHkQpyNswRwEA=', 'Content-Length': '0'} 2023-10-11T15:21:11.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c25d30dde2d476-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:11.431 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.431 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.432 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.432 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.432 DEBUG:boto:Method: GET 2023-10-11T15:21:11.433 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.433 DEBUG:boto:Data: 2023-10-11T15:21:11.434 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.435 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.435 DEBUG:boto:Params: {} 2023-10-11T15:21:11.436 DEBUG:boto:Token: None 2023-10-11T15:21:11.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM= 2023-10-11T15:21:11.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM=', 'Content-Length': '0'} 2023-10-11T15:21:11.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a9b57c8eed74ff-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.440 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.441 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.441 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.442 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.442 DEBUG:boto:Method: GET 2023-10-11T15:21:11.443 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.443 DEBUG:boto:Data: 2023-10-11T15:21:11.443 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.444 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.444 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.445 DEBUG:boto:Params: {} 2023-10-11T15:21:11.445 DEBUG:boto:Token: None 2023-10-11T15:21:11.446 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:11.446 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM= 2023-10-11T15:21:11.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9jc1mYmCgBF4zvhN5zysRhSNaM=', 'Content-Length': '0'} 2023-10-11T15:21:11.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009983cf3bd328045a-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.450 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.450 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.451 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.451 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.452 DEBUG:boto:Method: GET 2023-10-11T15:21:11.452 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.453 DEBUG:boto:Data: 2023-10-11T15:21:11.453 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.454 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.455 DEBUG:boto:Params: {} 2023-10-11T15:21:11.455 DEBUG:boto:Token: None 2023-10-11T15:21:11.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106d5c6d35e8d101-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.460 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.460 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.461 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.461 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.462 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.462 DEBUG:boto:Method: GET 2023-10-11T15:21:11.463 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.463 DEBUG:boto:Data: 2023-10-11T15:21:11.464 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.464 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.465 DEBUG:boto:Params: {} 2023-10-11T15:21:11.466 DEBUG:boto:Token: None 2023-10-11T15:21:11.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635caad324350827-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.472 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.473 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.473 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.473 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.474 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:11.474 DEBUG:boto:Data: 2023-10-11T15:21:11.475 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.475 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.476 DEBUG:boto:Params: {} 2023-10-11T15:21:11.476 DEBUG:boto:Token: None 2023-10-11T15:21:11.477 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19 2023-10-11T15:21:11.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K3wRrH2gZaddi+oMgfKh5Nu2y1M= 2023-10-11T15:21:11.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K3wRrH2gZaddi+oMgfKh5Nu2y1M=', 'Content-Length': '0'} 2023-10-11T15:21:11.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0e259d3959982d-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.480 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.481 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.481 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.481 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:11.482 DEBUG:boto:Data: 2023-10-11T15:21:11.482 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.483 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.483 DEBUG:boto:Params: {} 2023-10-11T15:21:11.484 DEBUG:boto:Token: None 2023-10-11T15:21:11.484 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19 2023-10-11T15:21:11.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K3wRrH2gZaddi+oMgfKh5Nu2y1M= 2023-10-11T15:21:11.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K3wRrH2gZaddi+oMgfKh5Nu2y1M=', 'Content-Length': '0'} 2023-10-11T15:21:11.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0b4e714e6c8697-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:11.488 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.489 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.489 DEBUG:boto:Method: GET 2023-10-11T15:21:11.489 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:11.490 DEBUG:boto:Data: 2023-10-11T15:21:11.490 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.491 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.492 DEBUG:boto:Params: {} 2023-10-11T15:21:11.492 DEBUG:boto:Token: None 2023-10-11T15:21:11.492 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19 2023-10-11T15:21:11.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oD4EjHacSm+J1gO+e3CuanDH4c8= 2023-10-11T15:21:11.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oD4EjHacSm+J1gO+e3CuanDH4c8=', 'Content-Length': '0'} 2023-10-11T15:21:11.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e8d596b09d6d87-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.538 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:11.539 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:11.539 DEBUG:boto:Method: GET 2023-10-11T15:21:11.540 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:11.541 DEBUG:boto:Data: 2023-10-11T15:21:11.541 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.541 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.542 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.542 DEBUG:boto:Params: {} 2023-10-11T15:21:11.543 DEBUG:boto:Token: None 2023-10-11T15:21:11.543 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj19 2023-10-11T15:21:11.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oD4EjHacSm+J1gO+e3CuanDH4c8= 2023-10-11T15:21:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oD4EjHacSm+J1gO+e3CuanDH4c8=', 'Content-Length': '0'} 2023-10-11T15:21:11.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0613f99ffe37e1-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.547 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.548 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.548 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.548 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.549 DEBUG:boto:Method: GET 2023-10-11T15:21:11.549 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.550 DEBUG:boto:Data: 2023-10-11T15:21:11.550 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.550 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.551 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.551 DEBUG:boto:Params: {} 2023-10-11T15:21:11.552 DEBUG:boto:Token: None 2023-10-11T15:21:11.552 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ae3f366cb8dcdf-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.556 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.557 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.558 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.558 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.558 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.559 DEBUG:boto:Method: GET 2023-10-11T15:21:11.559 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:11.559 DEBUG:boto:Data: 2023-10-11T15:21:11.560 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.560 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.561 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.561 DEBUG:boto:Params: {} 2023-10-11T15:21:11.561 DEBUG:boto:Token: None 2023-10-11T15:21:11.562 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.562 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565821e18ed18ed8-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.567 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:11.568 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.569 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.569 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.569 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.570 DEBUG:boto:Method: GET 2023-10-11T15:21:11.570 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.571 DEBUG:boto:Data: 2023-10-11T15:21:11.571 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.572 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.573 DEBUG:boto:Params: {} 2023-10-11T15:21:11.573 DEBUG:boto:Token: None 2023-10-11T15:21:11.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc= 2023-10-11T15:21:11.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc=', 'Content-Length': '0'} 2023-10-11T15:21:11.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee3264c8a438f20-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.577 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.578 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.578 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.579 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.579 DEBUG:boto:Method: GET 2023-10-11T15:21:11.579 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.580 DEBUG:boto:Data: 2023-10-11T15:21:11.580 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.581 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.582 DEBUG:boto:Params: {} 2023-10-11T15:21:11.582 DEBUG:boto:Token: None 2023-10-11T15:21:11.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc= 2023-10-11T15:21:11.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc=', 'Content-Length': '0'} 2023-10-11T15:21:11.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f663f681600d54cd-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.586 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.587 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.587 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.587 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.588 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.588 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.589 DEBUG:boto:Data: 2023-10-11T15:21:11.589 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.590 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.590 DEBUG:boto:Params: {} 2023-10-11T15:21:11.591 DEBUG:boto:Token: None 2023-10-11T15:21:11.591 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hEq/IYfUfqCOZ975lSa+CHO2HNk= 2023-10-11T15:21:11.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hEq/IYfUfqCOZ975lSa+CHO2HNk=', 'Content-Length': '0'} 2023-10-11T15:21:11.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b31e2e9038d9c5-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.594 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.595 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.595 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.596 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.596 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.596 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.597 DEBUG:boto:Data: 2023-10-11T15:21:11.597 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.598 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.598 DEBUG:boto:Params: {} 2023-10-11T15:21:11.599 DEBUG:boto:Token: None 2023-10-11T15:21:11.599 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hEq/IYfUfqCOZ975lSa+CHO2HNk= 2023-10-11T15:21:11.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hEq/IYfUfqCOZ975lSa+CHO2HNk=', 'Content-Length': '0'} 2023-10-11T15:21:11.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b28fdf63405375-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:11.603 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.604 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.604 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.604 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.605 DEBUG:boto:Method: GET 2023-10-11T15:21:11.605 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.606 DEBUG:boto:Data: 2023-10-11T15:21:11.606 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.606 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.607 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.607 DEBUG:boto:Params: {} 2023-10-11T15:21:11.608 DEBUG:boto:Token: None 2023-10-11T15:21:11.608 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc= 2023-10-11T15:21:11.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc=', 'Content-Length': '0'} 2023-10-11T15:21:11.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec37e758638eed84-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.614 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.614 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.615 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.615 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.615 DEBUG:boto:Method: GET 2023-10-11T15:21:11.616 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.616 DEBUG:boto:Data: 2023-10-11T15:21:11.617 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.617 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.618 DEBUG:boto:Params: {} 2023-10-11T15:21:11.618 DEBUG:boto:Token: None 2023-10-11T15:21:11.619 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:11.619 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc= 2023-10-11T15:21:11.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rq6XBsAX2/E0HymuKlwVWtPgEGc=', 'Content-Length': '0'} 2023-10-11T15:21:11.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0cd91f0cd1d4535-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.666 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.667 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.667 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.668 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.668 DEBUG:boto:Method: GET 2023-10-11T15:21:11.669 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.669 DEBUG:boto:Data: 2023-10-11T15:21:11.670 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.671 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.671 DEBUG:boto:Params: {} 2023-10-11T15:21:11.671 DEBUG:boto:Token: None 2023-10-11T15:21:11.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000522e0f725339697d-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.676 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.677 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.678 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.678 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.679 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.679 DEBUG:boto:Method: GET 2023-10-11T15:21:11.679 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.680 DEBUG:boto:Data: 2023-10-11T15:21:11.680 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.681 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.681 DEBUG:boto:Params: {} 2023-10-11T15:21:11.682 DEBUG:boto:Token: None 2023-10-11T15:21:11.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496d943a4c87d8ad-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.687 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.688 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.688 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.689 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.689 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:11.690 DEBUG:boto:Data: 2023-10-11T15:21:11.690 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.691 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.692 DEBUG:boto:Params: {} 2023-10-11T15:21:11.692 DEBUG:boto:Token: None 2023-10-11T15:21:11.692 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22 2023-10-11T15:21:11.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8QXRGfPqjzVZUBKMMea1sMi0/m0= 2023-10-11T15:21:11.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8QXRGfPqjzVZUBKMMea1sMi0/m0=', 'Content-Length': '0'} 2023-10-11T15:21:11.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fc68ec26a873ef-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.696 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.696 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.697 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.697 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:11.698 DEBUG:boto:Data: 2023-10-11T15:21:11.698 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.699 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.699 DEBUG:boto:Params: {} 2023-10-11T15:21:11.700 DEBUG:boto:Token: None 2023-10-11T15:21:11.700 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22 2023-10-11T15:21:11.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8QXRGfPqjzVZUBKMMea1sMi0/m0= 2023-10-11T15:21:11.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8QXRGfPqjzVZUBKMMea1sMi0/m0=', 'Content-Length': '0'} 2023-10-11T15:21:11.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b37f31b7fad5ad-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:11.704 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.705 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.705 DEBUG:boto:Method: GET 2023-10-11T15:21:11.705 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:11.706 DEBUG:boto:Data: 2023-10-11T15:21:11.706 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.707 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.708 DEBUG:boto:Params: {} 2023-10-11T15:21:11.708 DEBUG:boto:Token: None 2023-10-11T15:21:11.709 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22 2023-10-11T15:21:11.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/kvJleV5NiU90kDDQDJztJY0y8= 2023-10-11T15:21:11.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/kvJleV5NiU90kDDQDJztJY0y8=', 'Content-Length': '0'} 2023-10-11T15:21:11.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69588ba2c336bd3-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.754 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:11.755 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:11.755 DEBUG:boto:Method: GET 2023-10-11T15:21:11.756 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:11.756 DEBUG:boto:Data: 2023-10-11T15:21:11.757 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.758 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.758 DEBUG:boto:Params: {} 2023-10-11T15:21:11.759 DEBUG:boto:Token: None 2023-10-11T15:21:11.759 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj22 2023-10-11T15:21:11.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c/kvJleV5NiU90kDDQDJztJY0y8= 2023-10-11T15:21:11.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c/kvJleV5NiU90kDDQDJztJY0y8=', 'Content-Length': '0'} 2023-10-11T15:21:11.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f8c9a8351b6b8fe-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.763 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.763 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.764 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.764 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.765 DEBUG:boto:Method: GET 2023-10-11T15:21:11.765 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.765 DEBUG:boto:Data: 2023-10-11T15:21:11.766 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.767 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.767 DEBUG:boto:Params: {} 2023-10-11T15:21:11.768 DEBUG:boto:Token: None 2023-10-11T15:21:11.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f216d1f0280a49d4-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.773 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.773 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.774 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.774 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.775 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.775 DEBUG:boto:Method: GET 2023-10-11T15:21:11.776 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:11.776 DEBUG:boto:Data: 2023-10-11T15:21:11.777 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.778 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.778 DEBUG:boto:Params: {} 2023-10-11T15:21:11.779 DEBUG:boto:Token: None 2023-10-11T15:21:11.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6fbaa9ced1a50b-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.826 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:11.828 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.828 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.828 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.829 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.829 DEBUG:boto:Method: GET 2023-10-11T15:21:11.830 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.830 DEBUG:boto:Data: 2023-10-11T15:21:11.831 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.831 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.831 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.832 DEBUG:boto:Params: {} 2023-10-11T15:21:11.832 DEBUG:boto:Token: None 2023-10-11T15:21:11.833 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.833 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA= 2023-10-11T15:21:11.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA=', 'Content-Length': '0'} 2023-10-11T15:21:11.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda6248416b9cfa5-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.837 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.837 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.838 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.838 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.839 DEBUG:boto:Method: GET 2023-10-11T15:21:11.839 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.839 DEBUG:boto:Data: 2023-10-11T15:21:11.840 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.840 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.841 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.841 DEBUG:boto:Params: {} 2023-10-11T15:21:11.842 DEBUG:boto:Token: None 2023-10-11T15:21:11.842 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA= 2023-10-11T15:21:11.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA=', 'Content-Length': '0'} 2023-10-11T15:21:11.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005e707aeedb74ef-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.846 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.846 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.847 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.847 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.848 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.848 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.848 DEBUG:boto:Data: 2023-10-11T15:21:11.849 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.849 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.849 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.850 DEBUG:boto:Params: {} 2023-10-11T15:21:11.850 DEBUG:boto:Token: None 2023-10-11T15:21:11.851 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dw1iwQr/lX/TEemzsaKXRXSEG9I= 2023-10-11T15:21:11.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dw1iwQr/lX/TEemzsaKXRXSEG9I=', 'Content-Length': '0'} 2023-10-11T15:21:11.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000056640be62b0381b-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.854 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.855 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.855 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.855 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.856 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.856 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.857 DEBUG:boto:Data: 2023-10-11T15:21:11.857 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.858 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.858 DEBUG:boto:Params: {} 2023-10-11T15:21:11.859 DEBUG:boto:Token: None 2023-10-11T15:21:11.859 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dw1iwQr/lX/TEemzsaKXRXSEG9I= 2023-10-11T15:21:11.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dw1iwQr/lX/TEemzsaKXRXSEG9I=', 'Content-Length': '0'} 2023-10-11T15:21:11.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9efb66fe9bc973-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:11.863 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.863 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.864 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.864 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.864 DEBUG:boto:Method: GET 2023-10-11T15:21:11.865 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.865 DEBUG:boto:Data: 2023-10-11T15:21:11.866 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.866 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.867 DEBUG:boto:Params: {} 2023-10-11T15:21:11.867 DEBUG:boto:Token: None 2023-10-11T15:21:11.868 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA= 2023-10-11T15:21:11.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA=', 'Content-Length': '0'} 2023-10-11T15:21:11.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a6a03f902e9ffd-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.871 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.872 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.872 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.873 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.873 DEBUG:boto:Method: GET 2023-10-11T15:21:11.874 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.874 DEBUG:boto:Data: 2023-10-11T15:21:11.875 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.875 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.876 DEBUG:boto:Params: {} 2023-10-11T15:21:11.876 DEBUG:boto:Token: None 2023-10-11T15:21:11.877 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:11.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA= 2023-10-11T15:21:11.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K+hwainJU9RS/Jxh21yLarBNdbA=', 'Content-Length': '0'} 2023-10-11T15:21:11.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e2ab69053f00bf-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.881 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.881 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.881 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.882 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.882 DEBUG:boto:Method: GET 2023-10-11T15:21:11.883 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.883 DEBUG:boto:Data: 2023-10-11T15:21:11.884 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.885 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.885 DEBUG:boto:Params: {} 2023-10-11T15:21:11.885 DEBUG:boto:Token: None 2023-10-11T15:21:11.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d468e2fc127bfb35-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.890 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.891 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.891 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.892 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.892 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.892 DEBUG:boto:Method: GET 2023-10-11T15:21:11.893 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.893 DEBUG:boto:Data: 2023-10-11T15:21:11.894 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.895 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.895 DEBUG:boto:Params: {} 2023-10-11T15:21:11.895 DEBUG:boto:Token: None 2023-10-11T15:21:11.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87dff8ece35dd5c-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.901 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.901 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.902 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.902 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.903 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:11.903 DEBUG:boto:Data: 2023-10-11T15:21:11.904 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.904 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.905 DEBUG:boto:Params: {} 2023-10-11T15:21:11.905 DEBUG:boto:Token: None 2023-10-11T15:21:11.906 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33 2023-10-11T15:21:11.906 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OAxvifSREvUfvn9oKU6TpvfRZZ4= 2023-10-11T15:21:11.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OAxvifSREvUfvn9oKU6TpvfRZZ4=', 'Content-Length': '0'} 2023-10-11T15:21:11.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867c50bf7d43ab58-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.909 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.910 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.910 DEBUG:boto:Method: HEAD 2023-10-11T15:21:11.911 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:11.911 DEBUG:boto:Data: 2023-10-11T15:21:11.911 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.912 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.913 DEBUG:boto:Params: {} 2023-10-11T15:21:11.913 DEBUG:boto:Token: None 2023-10-11T15:21:11.914 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33 2023-10-11T15:21:11.914 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OAxvifSREvUfvn9oKU6TpvfRZZ4= 2023-10-11T15:21:11.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OAxvifSREvUfvn9oKU6TpvfRZZ4=', 'Content-Length': '0'} 2023-10-11T15:21:11.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9352d645fd38a8-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:11.917 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.918 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.918 DEBUG:boto:Method: GET 2023-10-11T15:21:11.919 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:11.919 DEBUG:boto:Data: 2023-10-11T15:21:11.920 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.920 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.920 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.921 DEBUG:boto:Params: {} 2023-10-11T15:21:11.921 DEBUG:boto:Token: None 2023-10-11T15:21:11.922 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33 2023-10-11T15:21:11.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPmvBrsniOZ/G8Oqusat4SFt3rM= 2023-10-11T15:21:11.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPmvBrsniOZ/G8Oqusat4SFt3rM=', 'Content-Length': '0'} 2023-10-11T15:21:11.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095eef8ffa0811db0-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.925 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:11.926 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:11.926 DEBUG:boto:Method: GET 2023-10-11T15:21:11.927 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:11.927 DEBUG:boto:Data: 2023-10-11T15:21:11.928 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.929 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.929 DEBUG:boto:Params: {} 2023-10-11T15:21:11.929 DEBUG:boto:Token: None 2023-10-11T15:21:11.930 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj33 2023-10-11T15:21:11.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPmvBrsniOZ/G8Oqusat4SFt3rM= 2023-10-11T15:21:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPmvBrsniOZ/G8Oqusat4SFt3rM=', 'Content-Length': '0'} 2023-10-11T15:21:11.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a12a78e5523198e-006526bd67-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.933 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.934 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.934 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.935 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.935 DEBUG:boto:Method: GET 2023-10-11T15:21:11.935 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.936 DEBUG:boto:Data: 2023-10-11T15:21:11.936 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.937 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.937 DEBUG:boto:Params: {} 2023-10-11T15:21:11.938 DEBUG:boto:Token: None 2023-10-11T15:21:11.938 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002897041f82c3fc89-006526bd67-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.943 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.944 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:11.944 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:11.945 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.945 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.945 DEBUG:boto:Method: GET 2023-10-11T15:21:11.946 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:11.946 DEBUG:boto:Data: 2023-10-11T15:21:11.947 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:11.947 DEBUG:boto:Port: 8000 2023-10-11T15:21:11.948 DEBUG:boto:Params: {} 2023-10-11T15:21:11.948 DEBUG:boto:Token: None 2023-10-11T15:21:11.949 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/ 2023-10-11T15:21:11.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE= 2023-10-11T15:21:11.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gp67xC0YRuiuGmXoGZw2n7wxkEE=', 'Content-Length': '0'} 2023-10-11T15:21:11.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000770f3eff37a9ca8-006526bd67-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:11.953 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:11.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:11.955 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:11.955 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:11.955 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:11.956 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:11.956 DEBUG:boto:Method: GET 2023-10-11T15:21:11.957 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:11.957 DEBUG:boto:Data: 2023-10-11T15:21:11.958 DEBUG:boto:Headers: {} 2023-10-11T15:21:11.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:11.958 DEBUG:boto:Port: 8001 2023-10-11T15:21:11.959 DEBUG:boto:Params: {} 2023-10-11T15:21:11.959 DEBUG:boto:Token: None 2023-10-11T15:21:11.960 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:11 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:11.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fMsNzO5s2c427RIpmY3+45obGuU= 2023-10-11T15:21:11.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fMsNzO5s2c427RIpmY3+45obGuU=', 'Content-Length': '0'} 2023-10-11T15:21:11.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d57ab79b84c63d-006526bd67-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.006 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.007 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.007 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.008 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.008 DEBUG:boto:Method: GET 2023-10-11T15:21:12.008 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.009 DEBUG:boto:Data: 2023-10-11T15:21:12.009 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.010 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.010 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.011 DEBUG:boto:Params: {} 2023-10-11T15:21:12.011 DEBUG:boto:Token: None 2023-10-11T15:21:12.012 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU= 2023-10-11T15:21:12.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU=', 'Content-Length': '0'} 2023-10-11T15:21:12.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3f0221264ed86b-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.015 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.016 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.016 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.017 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.017 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.017 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.018 DEBUG:boto:Data: 2023-10-11T15:21:12.018 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.019 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.020 DEBUG:boto:Params: {} 2023-10-11T15:21:12.020 DEBUG:boto:Token: None 2023-10-11T15:21:12.020 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7pHRokwQDLPAgCwDySrQ9rf2o0= 2023-10-11T15:21:12.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7pHRokwQDLPAgCwDySrQ9rf2o0=', 'Content-Length': '0'} 2023-10-11T15:21:12.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96cae1bb6867a13-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.026 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.026 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.027 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.027 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.028 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.028 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.028 DEBUG:boto:Data: 2023-10-11T15:21:12.029 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.030 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.030 DEBUG:boto:Params: {} 2023-10-11T15:21:12.030 DEBUG:boto:Token: None 2023-10-11T15:21:12.031 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d7pHRokwQDLPAgCwDySrQ9rf2o0= 2023-10-11T15:21:12.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d7pHRokwQDLPAgCwDySrQ9rf2o0=', 'Content-Length': '0'} 2023-10-11T15:21:12.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af62e6b6cea916fd-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:12.035 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.036 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.036 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.036 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.037 DEBUG:boto:Method: GET 2023-10-11T15:21:12.037 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.038 DEBUG:boto:Data: 2023-10-11T15:21:12.038 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.039 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.039 DEBUG:boto:Params: {} 2023-10-11T15:21:12.040 DEBUG:boto:Token: None 2023-10-11T15:21:12.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU= 2023-10-11T15:21:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU=', 'Content-Length': '0'} 2023-10-11T15:21:12.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1c9e4871469ff4-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.044 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.044 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.045 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.045 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.045 DEBUG:boto:Method: GET 2023-10-11T15:21:12.046 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.046 DEBUG:boto:Data: 2023-10-11T15:21:12.047 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.048 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.048 DEBUG:boto:Params: {} 2023-10-11T15:21:12.048 DEBUG:boto:Token: None 2023-10-11T15:21:12.049 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:12.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU= 2023-10-11T15:21:12.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uYImu6yM07d+E1xQ1TJKTO7yVgU=', 'Content-Length': '0'} 2023-10-11T15:21:12.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3845261da7c4c9-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.098 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.099 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.099 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.100 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.100 DEBUG:boto:Method: GET 2023-10-11T15:21:12.100 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.101 DEBUG:boto:Data: 2023-10-11T15:21:12.101 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.102 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.103 DEBUG:boto:Params: {} 2023-10-11T15:21:12.103 DEBUG:boto:Token: None 2023-10-11T15:21:12.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ad5e545363dc4d-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.107 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.108 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.108 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.109 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.109 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.110 DEBUG:boto:Method: GET 2023-10-11T15:21:12.110 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.111 DEBUG:boto:Data: 2023-10-11T15:21:12.111 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.112 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.112 DEBUG:boto:Params: {} 2023-10-11T15:21:12.113 DEBUG:boto:Token: None 2023-10-11T15:21:12.113 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5801d4fc60bd49c-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.118 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.119 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.119 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.120 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.120 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:12.120 DEBUG:boto:Data: 2023-10-11T15:21:12.121 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.121 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.122 DEBUG:boto:Params: {} 2023-10-11T15:21:12.122 DEBUG:boto:Token: None 2023-10-11T15:21:12.123 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34 2023-10-11T15:21:12.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:me6qbp2TzKetLS474m2bXlf0ioo= 2023-10-11T15:21:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:me6qbp2TzKetLS474m2bXlf0ioo=', 'Content-Length': '0'} 2023-10-11T15:21:12.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb853544fec71631-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.126 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.126 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.127 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.127 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:12.127 DEBUG:boto:Data: 2023-10-11T15:21:12.128 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.128 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.129 DEBUG:boto:Params: {} 2023-10-11T15:21:12.129 DEBUG:boto:Token: None 2023-10-11T15:21:12.130 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34 2023-10-11T15:21:12.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:me6qbp2TzKetLS474m2bXlf0ioo= 2023-10-11T15:21:12.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:me6qbp2TzKetLS474m2bXlf0ioo=', 'Content-Length': '0'} 2023-10-11T15:21:12.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229d200a68d107b0-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:12.134 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.134 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.134 DEBUG:boto:Method: GET 2023-10-11T15:21:12.135 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:12.135 DEBUG:boto:Data: 2023-10-11T15:21:12.136 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.136 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.137 DEBUG:boto:Params: {} 2023-10-11T15:21:12.137 DEBUG:boto:Token: None 2023-10-11T15:21:12.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34 2023-10-11T15:21:12.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k7cz+8qAFLpMsArGQEdwKeQUPNg= 2023-10-11T15:21:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k7cz+8qAFLpMsArGQEdwKeQUPNg=', 'Content-Length': '0'} 2023-10-11T15:21:12.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feff93eafe95daa0-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.182 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:12.183 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:12.183 DEBUG:boto:Method: GET 2023-10-11T15:21:12.184 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:12.184 DEBUG:boto:Data: 2023-10-11T15:21:12.184 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.185 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.186 DEBUG:boto:Params: {} 2023-10-11T15:21:12.186 DEBUG:boto:Token: None 2023-10-11T15:21:12.187 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj34 2023-10-11T15:21:12.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k7cz+8qAFLpMsArGQEdwKeQUPNg= 2023-10-11T15:21:12.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k7cz+8qAFLpMsArGQEdwKeQUPNg=', 'Content-Length': '0'} 2023-10-11T15:21:12.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008333b73e63ba040a-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.190 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.191 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.191 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.192 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.192 DEBUG:boto:Method: GET 2023-10-11T15:21:12.192 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.193 DEBUG:boto:Data: 2023-10-11T15:21:12.193 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.194 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.195 DEBUG:boto:Params: {} 2023-10-11T15:21:12.195 DEBUG:boto:Token: None 2023-10-11T15:21:12.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a98d0d64331689-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.200 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.200 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.201 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.201 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.202 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.202 DEBUG:boto:Method: GET 2023-10-11T15:21:12.203 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:12.203 DEBUG:boto:Data: 2023-10-11T15:21:12.203 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.204 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.205 DEBUG:boto:Params: {} 2023-10-11T15:21:12.205 DEBUG:boto:Token: None 2023-10-11T15:21:12.206 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f329d9573454655a-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.210 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:12.211 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.212 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.212 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.213 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.213 DEBUG:boto:Method: GET 2023-10-11T15:21:12.214 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.214 DEBUG:boto:Data: 2023-10-11T15:21:12.215 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.215 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.215 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.216 DEBUG:boto:Params: {} 2023-10-11T15:21:12.216 DEBUG:boto:Token: None 2023-10-11T15:21:12.217 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.217 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68= 2023-10-11T15:21:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68=', 'Content-Length': '0'} 2023-10-11T15:21:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056150988a5f6f9d9-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.221 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.221 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.222 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.222 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.222 DEBUG:boto:Method: GET 2023-10-11T15:21:12.223 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.223 DEBUG:boto:Data: 2023-10-11T15:21:12.224 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.224 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.225 DEBUG:boto:Params: {} 2023-10-11T15:21:12.225 DEBUG:boto:Token: None 2023-10-11T15:21:12.226 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68= 2023-10-11T15:21:12.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68=', 'Content-Length': '0'} 2023-10-11T15:21:12.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7b56e1651f3dbe-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.229 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.230 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.230 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.231 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.231 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.231 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.232 DEBUG:boto:Data: 2023-10-11T15:21:12.232 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.233 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.233 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.233 DEBUG:boto:Params: {} 2023-10-11T15:21:12.234 DEBUG:boto:Token: None 2023-10-11T15:21:12.234 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M6byGoEe6a8Z/64g/dRlhZaWwsw= 2023-10-11T15:21:12.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M6byGoEe6a8Z/64g/dRlhZaWwsw=', 'Content-Length': '0'} 2023-10-11T15:21:12.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025032fcc5461de3a-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.238 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.238 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.239 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.239 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.239 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.240 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.240 DEBUG:boto:Data: 2023-10-11T15:21:12.241 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.241 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.242 DEBUG:boto:Params: {} 2023-10-11T15:21:12.242 DEBUG:boto:Token: None 2023-10-11T15:21:12.243 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M6byGoEe6a8Z/64g/dRlhZaWwsw= 2023-10-11T15:21:12.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M6byGoEe6a8Z/64g/dRlhZaWwsw=', 'Content-Length': '0'} 2023-10-11T15:21:12.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e4a2081d014771-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:12.247 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.247 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.247 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.248 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.248 DEBUG:boto:Method: GET 2023-10-11T15:21:12.249 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.249 DEBUG:boto:Data: 2023-10-11T15:21:12.250 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.250 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.251 DEBUG:boto:Params: {} 2023-10-11T15:21:12.251 DEBUG:boto:Token: None 2023-10-11T15:21:12.252 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68= 2023-10-11T15:21:12.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68=', 'Content-Length': '0'} 2023-10-11T15:21:12.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5423a3608f1e0d8-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.256 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.256 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.257 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.257 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.257 DEBUG:boto:Method: GET 2023-10-11T15:21:12.258 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.258 DEBUG:boto:Data: 2023-10-11T15:21:12.259 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.259 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.260 DEBUG:boto:Params: {} 2023-10-11T15:21:12.260 DEBUG:boto:Token: None 2023-10-11T15:21:12.261 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:12.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68= 2023-10-11T15:21:12.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/UUN4GCCUoCSkfP1Iy6T5k9fy68=', 'Content-Length': '0'} 2023-10-11T15:21:12.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b800cc479a3b08-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.265 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.265 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.266 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.266 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.267 DEBUG:boto:Method: GET 2023-10-11T15:21:12.267 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.267 DEBUG:boto:Data: 2023-10-11T15:21:12.268 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.269 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.269 DEBUG:boto:Params: {} 2023-10-11T15:21:12.270 DEBUG:boto:Token: None 2023-10-11T15:21:12.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3c650c2f2c65de-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.314 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.315 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.316 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.316 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.317 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.317 DEBUG:boto:Method: GET 2023-10-11T15:21:12.317 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.318 DEBUG:boto:Data: 2023-10-11T15:21:12.318 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.319 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.320 DEBUG:boto:Params: {} 2023-10-11T15:21:12.320 DEBUG:boto:Token: None 2023-10-11T15:21:12.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.321 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5005be20139d57-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.325 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.326 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.326 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.327 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.327 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:12.327 DEBUG:boto:Data: 2023-10-11T15:21:12.328 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.329 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.329 DEBUG:boto:Params: {} 2023-10-11T15:21:12.330 DEBUG:boto:Token: None 2023-10-11T15:21:12.330 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38 2023-10-11T15:21:12.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IaIHF7xChBeJA8ibw6hNsnjaIIg= 2023-10-11T15:21:12.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IaIHF7xChBeJA8ibw6hNsnjaIIg=', 'Content-Length': '0'} 2023-10-11T15:21:12.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e097b3998cc52f61-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.334 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.334 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.335 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.335 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:12.335 DEBUG:boto:Data: 2023-10-11T15:21:12.336 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.337 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.337 DEBUG:boto:Params: {} 2023-10-11T15:21:12.337 DEBUG:boto:Token: None 2023-10-11T15:21:12.338 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38 2023-10-11T15:21:12.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IaIHF7xChBeJA8ibw6hNsnjaIIg= 2023-10-11T15:21:12.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IaIHF7xChBeJA8ibw6hNsnjaIIg=', 'Content-Length': '0'} 2023-10-11T15:21:12.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805da8e40881cd1e-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:12.342 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.342 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.343 DEBUG:boto:Method: GET 2023-10-11T15:21:12.343 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:12.344 DEBUG:boto:Data: 2023-10-11T15:21:12.344 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.345 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.345 DEBUG:boto:Params: {} 2023-10-11T15:21:12.346 DEBUG:boto:Token: None 2023-10-11T15:21:12.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38 2023-10-11T15:21:12.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RgYJ+zh46y/bIOViFylHsTJ+QOc= 2023-10-11T15:21:12.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RgYJ+zh46y/bIOViFylHsTJ+QOc=', 'Content-Length': '0'} 2023-10-11T15:21:12.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e3e01eb99005af-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.350 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:12.350 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:12.351 DEBUG:boto:Method: GET 2023-10-11T15:21:12.351 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:12.351 DEBUG:boto:Data: 2023-10-11T15:21:12.352 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.353 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.353 DEBUG:boto:Params: {} 2023-10-11T15:21:12.354 DEBUG:boto:Token: None 2023-10-11T15:21:12.354 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj38 2023-10-11T15:21:12.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RgYJ+zh46y/bIOViFylHsTJ+QOc= 2023-10-11T15:21:12.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RgYJ+zh46y/bIOViFylHsTJ+QOc=', 'Content-Length': '0'} 2023-10-11T15:21:12.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9a30209534438f-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.358 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.358 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.359 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.359 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.359 DEBUG:boto:Method: GET 2023-10-11T15:21:12.360 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.360 DEBUG:boto:Data: 2023-10-11T15:21:12.361 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.361 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.362 DEBUG:boto:Params: {} 2023-10-11T15:21:12.362 DEBUG:boto:Token: None 2023-10-11T15:21:12.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51f500da107777f-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.367 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.367 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.368 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.368 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.369 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.369 DEBUG:boto:Method: GET 2023-10-11T15:21:12.369 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:12.370 DEBUG:boto:Data: 2023-10-11T15:21:12.370 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.371 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.372 DEBUG:boto:Params: {} 2023-10-11T15:21:12.372 DEBUG:boto:Token: None 2023-10-11T15:21:12.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e4357f1d59c583-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.377 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:12.378 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.379 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.379 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.379 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.380 DEBUG:boto:Method: GET 2023-10-11T15:21:12.380 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.381 DEBUG:boto:Data: 2023-10-11T15:21:12.381 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.382 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.383 DEBUG:boto:Params: {} 2023-10-11T15:21:12.383 DEBUG:boto:Token: None 2023-10-11T15:21:12.384 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE= 2023-10-11T15:21:12.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE=', 'Content-Length': '0'} 2023-10-11T15:21:12.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc2463670189a90-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.388 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.388 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.388 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.389 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.389 DEBUG:boto:Method: GET 2023-10-11T15:21:12.390 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.390 DEBUG:boto:Data: 2023-10-11T15:21:12.391 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.391 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.392 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.392 DEBUG:boto:Params: {} 2023-10-11T15:21:12.392 DEBUG:boto:Token: None 2023-10-11T15:21:12.393 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.393 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE= 2023-10-11T15:21:12.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE=', 'Content-Length': '0'} 2023-10-11T15:21:12.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001717834d34fcf63d-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.397 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.397 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.398 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.398 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.399 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.399 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.400 DEBUG:boto:Data: 2023-10-11T15:21:12.400 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.400 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.401 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.401 DEBUG:boto:Params: {} 2023-10-11T15:21:12.402 DEBUG:boto:Token: None 2023-10-11T15:21:12.402 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hjmd/8cC5qYz2LEwlGl4JDIAL2E= 2023-10-11T15:21:12.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hjmd/8cC5qYz2LEwlGl4JDIAL2E=', 'Content-Length': '0'} 2023-10-11T15:21:12.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7776ecf263f4eb-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.406 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.406 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.407 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.407 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.407 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.408 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.408 DEBUG:boto:Data: 2023-10-11T15:21:12.409 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.409 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.410 DEBUG:boto:Params: {} 2023-10-11T15:21:12.410 DEBUG:boto:Token: None 2023-10-11T15:21:12.411 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hjmd/8cC5qYz2LEwlGl4JDIAL2E= 2023-10-11T15:21:12.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hjmd/8cC5qYz2LEwlGl4JDIAL2E=', 'Content-Length': '0'} 2023-10-11T15:21:12.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17f04982e13cec7-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:12.415 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.416 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.416 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.416 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.417 DEBUG:boto:Method: GET 2023-10-11T15:21:12.417 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.418 DEBUG:boto:Data: 2023-10-11T15:21:12.418 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.419 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.419 DEBUG:boto:Params: {} 2023-10-11T15:21:12.420 DEBUG:boto:Token: None 2023-10-11T15:21:12.420 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE= 2023-10-11T15:21:12.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE=', 'Content-Length': '0'} 2023-10-11T15:21:12.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008414385606ae9de1-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.466 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.467 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.468 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.468 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.468 DEBUG:boto:Method: GET 2023-10-11T15:21:12.469 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.469 DEBUG:boto:Data: 2023-10-11T15:21:12.470 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.470 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.471 DEBUG:boto:Params: {} 2023-10-11T15:21:12.471 DEBUG:boto:Token: None 2023-10-11T15:21:12.472 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:12.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE= 2023-10-11T15:21:12.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H+XH/ZPtrrITUp4xsG6FdTv9yoE=', 'Content-Length': '0'} 2023-10-11T15:21:12.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61082c493b428d5-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.476 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.476 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.476 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.477 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.477 DEBUG:boto:Method: GET 2023-10-11T15:21:12.478 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.478 DEBUG:boto:Data: 2023-10-11T15:21:12.479 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.479 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.480 DEBUG:boto:Params: {} 2023-10-11T15:21:12.480 DEBUG:boto:Token: None 2023-10-11T15:21:12.481 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee353035930f1670-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.485 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.485 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.486 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.486 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.487 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.487 DEBUG:boto:Method: GET 2023-10-11T15:21:12.488 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.488 DEBUG:boto:Data: 2023-10-11T15:21:12.488 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.489 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.490 DEBUG:boto:Params: {} 2023-10-11T15:21:12.490 DEBUG:boto:Token: None 2023-10-11T15:21:12.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061d512dad03eef40-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.495 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.495 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.496 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.496 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.497 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:12.497 DEBUG:boto:Data: 2023-10-11T15:21:12.497 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.498 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.499 DEBUG:boto:Params: {} 2023-10-11T15:21:12.499 DEBUG:boto:Token: None 2023-10-11T15:21:12.499 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44 2023-10-11T15:21:12.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ynllmj4aLCcLRdQ0wu1Esd3bcSY= 2023-10-11T15:21:12.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ynllmj4aLCcLRdQ0wu1Esd3bcSY=', 'Content-Length': '0'} 2023-10-11T15:21:12.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800094954c3580e8-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.503 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.503 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.504 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.504 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:12.505 DEBUG:boto:Data: 2023-10-11T15:21:12.505 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.506 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.506 DEBUG:boto:Params: {} 2023-10-11T15:21:12.507 DEBUG:boto:Token: None 2023-10-11T15:21:12.507 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44 2023-10-11T15:21:12.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ynllmj4aLCcLRdQ0wu1Esd3bcSY= 2023-10-11T15:21:12.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ynllmj4aLCcLRdQ0wu1Esd3bcSY=', 'Content-Length': '0'} 2023-10-11T15:21:12.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f7434ceae9ea9c1-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:12.511 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.511 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.512 DEBUG:boto:Method: GET 2023-10-11T15:21:12.512 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:12.513 DEBUG:boto:Data: 2023-10-11T15:21:12.513 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.514 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.514 DEBUG:boto:Params: {} 2023-10-11T15:21:12.515 DEBUG:boto:Token: None 2023-10-11T15:21:12.515 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44 2023-10-11T15:21:12.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m0yexV0N+EAFCPckVs2+lvHdw74= 2023-10-11T15:21:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m0yexV0N+EAFCPckVs2+lvHdw74=', 'Content-Length': '0'} 2023-10-11T15:21:12.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee2c83a1203d35fa-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.519 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:12.519 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:12.520 DEBUG:boto:Method: GET 2023-10-11T15:21:12.520 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:12.520 DEBUG:boto:Data: 2023-10-11T15:21:12.521 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.521 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.522 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.522 DEBUG:boto:Params: {} 2023-10-11T15:21:12.523 DEBUG:boto:Token: None 2023-10-11T15:21:12.523 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj44 2023-10-11T15:21:12.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m0yexV0N+EAFCPckVs2+lvHdw74= 2023-10-11T15:21:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m0yexV0N+EAFCPckVs2+lvHdw74=', 'Content-Length': '0'} 2023-10-11T15:21:12.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028a7c7d91bc47cb9-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.570 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.571 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.571 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.572 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.572 DEBUG:boto:Method: GET 2023-10-11T15:21:12.573 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.573 DEBUG:boto:Data: 2023-10-11T15:21:12.574 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.574 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.575 DEBUG:boto:Params: {} 2023-10-11T15:21:12.575 DEBUG:boto:Token: None 2023-10-11T15:21:12.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.576 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b95207268b6a899-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.580 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.581 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.581 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.582 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.582 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.582 DEBUG:boto:Method: GET 2023-10-11T15:21:12.583 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:12.583 DEBUG:boto:Data: 2023-10-11T15:21:12.584 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.584 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.585 DEBUG:boto:Params: {} 2023-10-11T15:21:12.585 DEBUG:boto:Token: None 2023-10-11T15:21:12.586 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afce8bd103431eef-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.591 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:12.592 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.592 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.593 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.593 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.593 DEBUG:boto:Method: GET 2023-10-11T15:21:12.594 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.594 DEBUG:boto:Data: 2023-10-11T15:21:12.595 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.596 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.596 DEBUG:boto:Params: {} 2023-10-11T15:21:12.597 DEBUG:boto:Token: None 2023-10-11T15:21:12.597 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI= 2023-10-11T15:21:12.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI=', 'Content-Length': '0'} 2023-10-11T15:21:12.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d31e28f97607cf-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.642 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.643 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.644 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.644 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.645 DEBUG:boto:Method: GET 2023-10-11T15:21:12.645 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.646 DEBUG:boto:Data: 2023-10-11T15:21:12.646 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.647 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.647 DEBUG:boto:Params: {} 2023-10-11T15:21:12.648 DEBUG:boto:Token: None 2023-10-11T15:21:12.649 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.649 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI= 2023-10-11T15:21:12.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI=', 'Content-Length': '0'} 2023-10-11T15:21:12.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d78a4b06140c69-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.652 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.653 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.653 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.654 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.654 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.655 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.655 DEBUG:boto:Data: 2023-10-11T15:21:12.655 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.656 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.657 DEBUG:boto:Params: {} 2023-10-11T15:21:12.657 DEBUG:boto:Token: None 2023-10-11T15:21:12.658 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:srxNHsDkNl2GKBIlJTc6/SPfOO8= 2023-10-11T15:21:12.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:srxNHsDkNl2GKBIlJTc6/SPfOO8=', 'Content-Length': '0'} 2023-10-11T15:21:12.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923f09a108288d17-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.661 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.662 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.662 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.663 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.663 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.663 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.664 DEBUG:boto:Data: 2023-10-11T15:21:12.664 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.665 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.665 DEBUG:boto:Params: {} 2023-10-11T15:21:12.666 DEBUG:boto:Token: None 2023-10-11T15:21:12.666 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.667 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:srxNHsDkNl2GKBIlJTc6/SPfOO8= 2023-10-11T15:21:12.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:srxNHsDkNl2GKBIlJTc6/SPfOO8=', 'Content-Length': '0'} 2023-10-11T15:21:12.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e2a653a2a2d37f-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:12.670 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.671 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.671 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.671 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.672 DEBUG:boto:Method: GET 2023-10-11T15:21:12.672 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.673 DEBUG:boto:Data: 2023-10-11T15:21:12.673 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.674 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.674 DEBUG:boto:Params: {} 2023-10-11T15:21:12.675 DEBUG:boto:Token: None 2023-10-11T15:21:12.675 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI= 2023-10-11T15:21:12.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI=', 'Content-Length': '0'} 2023-10-11T15:21:12.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003024ac58aecf253-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.679 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.680 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.680 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.680 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.681 DEBUG:boto:Method: GET 2023-10-11T15:21:12.681 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.682 DEBUG:boto:Data: 2023-10-11T15:21:12.682 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.683 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.683 DEBUG:boto:Params: {} 2023-10-11T15:21:12.684 DEBUG:boto:Token: None 2023-10-11T15:21:12.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:12.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI= 2023-10-11T15:21:12.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1oNdz3oNiYhgDys6BUMxS8rSiVI=', 'Content-Length': '0'} 2023-10-11T15:21:12.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add71f37552b2c0c-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.688 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.688 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.689 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.689 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.690 DEBUG:boto:Method: GET 2023-10-11T15:21:12.690 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.690 DEBUG:boto:Data: 2023-10-11T15:21:12.691 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.692 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.692 DEBUG:boto:Params: {} 2023-10-11T15:21:12.693 DEBUG:boto:Token: None 2023-10-11T15:21:12.693 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab7b02bf840303e-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.697 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.698 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.698 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.699 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.699 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.700 DEBUG:boto:Method: GET 2023-10-11T15:21:12.700 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.700 DEBUG:boto:Data: 2023-10-11T15:21:12.701 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.702 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.702 DEBUG:boto:Params: {} 2023-10-11T15:21:12.702 DEBUG:boto:Token: None 2023-10-11T15:21:12.703 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3313f837fc07219-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.707 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.708 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.708 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.709 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.709 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:12.709 DEBUG:boto:Data: 2023-10-11T15:21:12.710 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.710 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.711 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.711 DEBUG:boto:Params: {} 2023-10-11T15:21:12.712 DEBUG:boto:Token: None 2023-10-11T15:21:12.712 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51 2023-10-11T15:21:12.712 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ziq/LmIBx6XCD5EYCsYE7fqzuAQ= 2023-10-11T15:21:12.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ziq/LmIBx6XCD5EYCsYE7fqzuAQ=', 'Content-Length': '0'} 2023-10-11T15:21:12.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ffe71a653b624a-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.715 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.716 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.716 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.717 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:12.717 DEBUG:boto:Data: 2023-10-11T15:21:12.718 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.718 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.719 DEBUG:boto:Params: {} 2023-10-11T15:21:12.719 DEBUG:boto:Token: None 2023-10-11T15:21:12.720 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51 2023-10-11T15:21:12.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ziq/LmIBx6XCD5EYCsYE7fqzuAQ= 2023-10-11T15:21:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ziq/LmIBx6XCD5EYCsYE7fqzuAQ=', 'Content-Length': '0'} 2023-10-11T15:21:12.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceeb2534a4060368-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:12.724 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.724 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.724 DEBUG:boto:Method: GET 2023-10-11T15:21:12.725 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:12.725 DEBUG:boto:Data: 2023-10-11T15:21:12.726 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.726 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.727 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.727 DEBUG:boto:Params: {} 2023-10-11T15:21:12.728 DEBUG:boto:Token: None 2023-10-11T15:21:12.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51 2023-10-11T15:21:12.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ywwJnF8J8nykpk59tVUeTjLrnJ4= 2023-10-11T15:21:12.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ywwJnF8J8nykpk59tVUeTjLrnJ4=', 'Content-Length': '0'} 2023-10-11T15:21:12.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010aacc6bb7fa6c51-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.774 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:12.775 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:12.776 DEBUG:boto:Method: GET 2023-10-11T15:21:12.776 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:12.776 DEBUG:boto:Data: 2023-10-11T15:21:12.777 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.778 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.778 DEBUG:boto:Params: {} 2023-10-11T15:21:12.779 DEBUG:boto:Token: None 2023-10-11T15:21:12.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj51 2023-10-11T15:21:12.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ywwJnF8J8nykpk59tVUeTjLrnJ4= 2023-10-11T15:21:12.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ywwJnF8J8nykpk59tVUeTjLrnJ4=', 'Content-Length': '0'} 2023-10-11T15:21:12.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555aed7033ca922b-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.783 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.784 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.784 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.784 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.785 DEBUG:boto:Method: GET 2023-10-11T15:21:12.785 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.786 DEBUG:boto:Data: 2023-10-11T15:21:12.786 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.787 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.787 DEBUG:boto:Params: {} 2023-10-11T15:21:12.788 DEBUG:boto:Token: None 2023-10-11T15:21:12.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f01a061ebc56c0-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.793 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.793 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.794 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.794 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.795 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.795 DEBUG:boto:Method: GET 2023-10-11T15:21:12.795 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:12.796 DEBUG:boto:Data: 2023-10-11T15:21:12.796 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.797 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.797 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.798 DEBUG:boto:Params: {} 2023-10-11T15:21:12.798 DEBUG:boto:Token: None 2023-10-11T15:21:12.798 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae18eb9f93c97a5f-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.803 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:12.804 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.804 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.805 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.805 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.806 DEBUG:boto:Method: GET 2023-10-11T15:21:12.806 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.807 DEBUG:boto:Data: 2023-10-11T15:21:12.807 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.808 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.808 DEBUG:boto:Params: {} 2023-10-11T15:21:12.809 DEBUG:boto:Token: None 2023-10-11T15:21:12.809 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc= 2023-10-11T15:21:12.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc=', 'Content-Length': '0'} 2023-10-11T15:21:12.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957d02a7ae812171-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.813 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.813 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.814 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.814 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.815 DEBUG:boto:Method: GET 2023-10-11T15:21:12.815 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.815 DEBUG:boto:Data: 2023-10-11T15:21:12.816 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.817 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.817 DEBUG:boto:Params: {} 2023-10-11T15:21:12.818 DEBUG:boto:Token: None 2023-10-11T15:21:12.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc= 2023-10-11T15:21:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc=', 'Content-Length': '0'} 2023-10-11T15:21:12.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001180346f29e448af-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.822 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.822 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.822 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.823 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.823 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.824 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.824 DEBUG:boto:Data: 2023-10-11T15:21:12.825 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.825 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.826 DEBUG:boto:Params: {} 2023-10-11T15:21:12.826 DEBUG:boto:Token: None 2023-10-11T15:21:12.827 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ai1ENZG+KoErmVnVpYRAWO5AGJA= 2023-10-11T15:21:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ai1ENZG+KoErmVnVpYRAWO5AGJA=', 'Content-Length': '0'} 2023-10-11T15:21:12.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aaad2adff303ce3-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.830 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.831 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.831 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.832 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.832 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.832 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.833 DEBUG:boto:Data: 2023-10-11T15:21:12.833 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.834 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.834 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.834 DEBUG:boto:Params: {} 2023-10-11T15:21:12.835 DEBUG:boto:Token: None 2023-10-11T15:21:12.835 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ai1ENZG+KoErmVnVpYRAWO5AGJA= 2023-10-11T15:21:12.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ai1ENZG+KoErmVnVpYRAWO5AGJA=', 'Content-Length': '0'} 2023-10-11T15:21:12.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016817a2e42f375ad-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:12.840 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.840 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.840 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.841 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.841 DEBUG:boto:Method: GET 2023-10-11T15:21:12.842 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.842 DEBUG:boto:Data: 2023-10-11T15:21:12.843 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.846 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.846 DEBUG:boto:Params: {} 2023-10-11T15:21:12.847 DEBUG:boto:Token: None 2023-10-11T15:21:12.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc= 2023-10-11T15:21:12.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc=', 'Content-Length': '0'} 2023-10-11T15:21:12.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950428f2342a51fe-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.851 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.851 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.852 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.852 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.853 DEBUG:boto:Method: GET 2023-10-11T15:21:12.853 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.853 DEBUG:boto:Data: 2023-10-11T15:21:12.854 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.855 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.855 DEBUG:boto:Params: {} 2023-10-11T15:21:12.856 DEBUG:boto:Token: None 2023-10-11T15:21:12.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:12.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc= 2023-10-11T15:21:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73+7j8yTxZg9+MIY/PdjQQyHqwc=', 'Content-Length': '0'} 2023-10-11T15:21:12.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906c307a1c16d0f2-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.902 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.903 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.903 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.904 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.904 DEBUG:boto:Method: GET 2023-10-11T15:21:12.905 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.905 DEBUG:boto:Data: 2023-10-11T15:21:12.906 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.906 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.907 DEBUG:boto:Params: {} 2023-10-11T15:21:12.907 DEBUG:boto:Token: None 2023-10-11T15:21:12.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071eaa45e7ff19be8-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.912 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.912 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.913 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.913 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.914 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.914 DEBUG:boto:Method: GET 2023-10-11T15:21:12.915 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.915 DEBUG:boto:Data: 2023-10-11T15:21:12.915 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.916 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.917 DEBUG:boto:Params: {} 2023-10-11T15:21:12.917 DEBUG:boto:Token: None 2023-10-11T15:21:12.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e319800131e33e5e-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.922 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.923 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.923 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.924 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.924 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:12.924 DEBUG:boto:Data: 2023-10-11T15:21:12.925 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.926 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.926 DEBUG:boto:Params: {} 2023-10-11T15:21:12.926 DEBUG:boto:Token: None 2023-10-11T15:21:12.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55 2023-10-11T15:21:12.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XK4LjmhHy3QczGiy0gPAuwx2pd8= 2023-10-11T15:21:12.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XK4LjmhHy3QczGiy0gPAuwx2pd8=', 'Content-Length': '0'} 2023-10-11T15:21:12.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1852b17558ef31b-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.930 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.931 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.931 DEBUG:boto:Method: HEAD 2023-10-11T15:21:12.932 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:12.932 DEBUG:boto:Data: 2023-10-11T15:21:12.932 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.933 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.933 DEBUG:boto:Params: {} 2023-10-11T15:21:12.934 DEBUG:boto:Token: None 2023-10-11T15:21:12.934 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55 2023-10-11T15:21:12.935 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XK4LjmhHy3QczGiy0gPAuwx2pd8= 2023-10-11T15:21:12.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XK4LjmhHy3QczGiy0gPAuwx2pd8=', 'Content-Length': '0'} 2023-10-11T15:21:12.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006376df42149cfd58-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:12.938 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.939 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.939 DEBUG:boto:Method: GET 2023-10-11T15:21:12.940 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:12.940 DEBUG:boto:Data: 2023-10-11T15:21:12.940 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.941 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.942 DEBUG:boto:Params: {} 2023-10-11T15:21:12.942 DEBUG:boto:Token: None 2023-10-11T15:21:12.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55 2023-10-11T15:21:12.943 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:msrL+8AvLcP4PqOB5WWzoVUM5RM= 2023-10-11T15:21:12.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:msrL+8AvLcP4PqOB5WWzoVUM5RM=', 'Content-Length': '0'} 2023-10-11T15:21:12.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639a6d9dd0bcf66f-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.946 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:12.947 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:12.947 DEBUG:boto:Method: GET 2023-10-11T15:21:12.948 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:12.948 DEBUG:boto:Data: 2023-10-11T15:21:12.948 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.949 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.950 DEBUG:boto:Params: {} 2023-10-11T15:21:12.950 DEBUG:boto:Token: None 2023-10-11T15:21:12.951 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj55 2023-10-11T15:21:12.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:msrL+8AvLcP4PqOB5WWzoVUM5RM= 2023-10-11T15:21:12.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:msrL+8AvLcP4PqOB5WWzoVUM5RM=', 'Content-Length': '0'} 2023-10-11T15:21:12.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d0d2e9344a66797-006526bd68-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.954 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.955 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.955 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.955 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.956 DEBUG:boto:Method: GET 2023-10-11T15:21:12.956 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.957 DEBUG:boto:Data: 2023-10-11T15:21:12.957 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.958 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.958 DEBUG:boto:Params: {} 2023-10-11T15:21:12.959 DEBUG:boto:Token: None 2023-10-11T15:21:12.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd01f48d22f2e39-006526bd68-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.963 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.964 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:12.964 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:12.965 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.965 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.966 DEBUG:boto:Method: GET 2023-10-11T15:21:12.966 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:12.966 DEBUG:boto:Data: 2023-10-11T15:21:12.967 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:12.968 DEBUG:boto:Port: 8000 2023-10-11T15:21:12.968 DEBUG:boto:Params: {} 2023-10-11T15:21:12.969 DEBUG:boto:Token: None 2023-10-11T15:21:12.969 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/ 2023-10-11T15:21:12.970 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY= 2023-10-11T15:21:12.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gPgGkAcQUMxNw1ZyQhcDKcgcURY=', 'Content-Length': '0'} 2023-10-11T15:21:12.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbda83d4bea33a83-006526bd68-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:12.974 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:12.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:12.975 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:12.975 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:12.976 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:12.976 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:12.976 DEBUG:boto:Method: GET 2023-10-11T15:21:12.977 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:12.977 DEBUG:boto:Data: 2023-10-11T15:21:12.978 DEBUG:boto:Headers: {} 2023-10-11T15:21:12.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:12.979 DEBUG:boto:Port: 8001 2023-10-11T15:21:12.979 DEBUG:boto:Params: {} 2023-10-11T15:21:12.980 DEBUG:boto:Token: None 2023-10-11T15:21:12.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:12 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:12.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dAJrl/Q2hkgMLWzumBoS3rlGKzw= 2023-10-11T15:21:12.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dAJrl/Q2hkgMLWzumBoS3rlGKzw=', 'Content-Length': '0'} 2023-10-11T15:21:12.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d40fe06b6c86946-006526bd68-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.026 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.027 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.027 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.028 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.028 DEBUG:boto:Method: GET 2023-10-11T15:21:13.029 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.029 DEBUG:boto:Data: 2023-10-11T15:21:13.029 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.030 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.031 DEBUG:boto:Params: {} 2023-10-11T15:21:13.031 DEBUG:boto:Token: None 2023-10-11T15:21:13.032 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA= 2023-10-11T15:21:13.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA=', 'Content-Length': '0'} 2023-10-11T15:21:13.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b4e1d841db67f2-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.035 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.036 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.036 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.037 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.037 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.037 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.038 DEBUG:boto:Data: 2023-10-11T15:21:13.038 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.039 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.039 DEBUG:boto:Params: {} 2023-10-11T15:21:13.040 DEBUG:boto:Token: None 2023-10-11T15:21:13.040 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wSJ1UlRLbRRExKnLGSs7ilQQvk8= 2023-10-11T15:21:13.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wSJ1UlRLbRRExKnLGSs7ilQQvk8=', 'Content-Length': '0'} 2023-10-11T15:21:13.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8527c875d97a69-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.043 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.044 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.044 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.045 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.045 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.045 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.046 DEBUG:boto:Data: 2023-10-11T15:21:13.046 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.047 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.049 DEBUG:boto:Params: {} 2023-10-11T15:21:13.050 DEBUG:boto:Token: None 2023-10-11T15:21:13.050 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wSJ1UlRLbRRExKnLGSs7ilQQvk8= 2023-10-11T15:21:13.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wSJ1UlRLbRRExKnLGSs7ilQQvk8=', 'Content-Length': '0'} 2023-10-11T15:21:13.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff93cbc002003f70-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:13.054 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.055 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.055 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.056 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.056 DEBUG:boto:Method: GET 2023-10-11T15:21:13.056 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.057 DEBUG:boto:Data: 2023-10-11T15:21:13.057 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.058 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.059 DEBUG:boto:Params: {} 2023-10-11T15:21:13.059 DEBUG:boto:Token: None 2023-10-11T15:21:13.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA= 2023-10-11T15:21:13.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA=', 'Content-Length': '0'} 2023-10-11T15:21:13.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ed7f9c74f75339-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.063 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.063 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.064 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.064 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.065 DEBUG:boto:Method: GET 2023-10-11T15:21:13.065 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.065 DEBUG:boto:Data: 2023-10-11T15:21:13.066 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.067 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.067 DEBUG:boto:Params: {} 2023-10-11T15:21:13.068 DEBUG:boto:Token: None 2023-10-11T15:21:13.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:13.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA= 2023-10-11T15:21:13.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PG1+yizKslcH9lvjfB4TkPqcfXA=', 'Content-Length': '0'} 2023-10-11T15:21:13.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c47d8f95101867-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.072 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.072 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.073 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.073 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.074 DEBUG:boto:Method: GET 2023-10-11T15:21:13.074 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.074 DEBUG:boto:Data: 2023-10-11T15:21:13.075 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.076 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.076 DEBUG:boto:Params: {} 2023-10-11T15:21:13.076 DEBUG:boto:Token: None 2023-10-11T15:21:13.077 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.077 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a52dc91f2d29b694-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.126 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.127 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.128 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.128 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.128 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.129 DEBUG:boto:Method: GET 2023-10-11T15:21:13.129 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.130 DEBUG:boto:Data: 2023-10-11T15:21:13.130 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.131 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.131 DEBUG:boto:Params: {} 2023-10-11T15:21:13.133 DEBUG:boto:Token: None 2023-10-11T15:21:13.134 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d7221eaea72d1f-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.139 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.139 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.140 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.140 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.141 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:13.141 DEBUG:boto:Data: 2023-10-11T15:21:13.141 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.142 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.143 DEBUG:boto:Params: {} 2023-10-11T15:21:13.143 DEBUG:boto:Token: None 2023-10-11T15:21:13.144 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57 2023-10-11T15:21:13.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZabTb58fbIPUCcjE+DgHD5mUFyk= 2023-10-11T15:21:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZabTb58fbIPUCcjE+DgHD5mUFyk=', 'Content-Length': '0'} 2023-10-11T15:21:13.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bffbc87359cb25-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.147 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.147 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.148 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.148 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:13.149 DEBUG:boto:Data: 2023-10-11T15:21:13.149 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.150 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.151 DEBUG:boto:Params: {} 2023-10-11T15:21:13.151 DEBUG:boto:Token: None 2023-10-11T15:21:13.152 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57 2023-10-11T15:21:13.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZabTb58fbIPUCcjE+DgHD5mUFyk= 2023-10-11T15:21:13.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZabTb58fbIPUCcjE+DgHD5mUFyk=', 'Content-Length': '0'} 2023-10-11T15:21:13.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c95cd622b8aa94-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:13.155 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.156 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.156 DEBUG:boto:Method: GET 2023-10-11T15:21:13.157 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:13.157 DEBUG:boto:Data: 2023-10-11T15:21:13.157 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.158 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.159 DEBUG:boto:Params: {} 2023-10-11T15:21:13.159 DEBUG:boto:Token: None 2023-10-11T15:21:13.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57 2023-10-11T15:21:13.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1+E5DVVyAEZLCLSkvj+vmODbDbY= 2023-10-11T15:21:13.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1+E5DVVyAEZLCLSkvj+vmODbDbY=', 'Content-Length': '0'} 2023-10-11T15:21:13.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000420898557c71553-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.163 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:13.164 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:13.164 DEBUG:boto:Method: GET 2023-10-11T15:21:13.165 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:13.165 DEBUG:boto:Data: 2023-10-11T15:21:13.165 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.166 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.167 DEBUG:boto:Params: {} 2023-10-11T15:21:13.167 DEBUG:boto:Token: None 2023-10-11T15:21:13.168 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj57 2023-10-11T15:21:13.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1+E5DVVyAEZLCLSkvj+vmODbDbY= 2023-10-11T15:21:13.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1+E5DVVyAEZLCLSkvj+vmODbDbY=', 'Content-Length': '0'} 2023-10-11T15:21:13.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f47dabf3fb555d73-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.214 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.215 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.215 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.216 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.216 DEBUG:boto:Method: GET 2023-10-11T15:21:13.217 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.217 DEBUG:boto:Data: 2023-10-11T15:21:13.217 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.218 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.219 DEBUG:boto:Params: {} 2023-10-11T15:21:13.219 DEBUG:boto:Token: None 2023-10-11T15:21:13.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f824c3228a9fb32-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.224 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.225 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.225 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.225 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.226 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.226 DEBUG:boto:Method: GET 2023-10-11T15:21:13.227 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:13.227 DEBUG:boto:Data: 2023-10-11T15:21:13.228 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.228 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.229 DEBUG:boto:Params: {} 2023-10-11T15:21:13.229 DEBUG:boto:Token: None 2023-10-11T15:21:13.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086bf9137fab1f7f2-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.235 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:13.236 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.237 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.237 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.238 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.238 DEBUG:boto:Method: GET 2023-10-11T15:21:13.239 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.239 DEBUG:boto:Data: 2023-10-11T15:21:13.240 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.240 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.241 DEBUG:boto:Params: {} 2023-10-11T15:21:13.241 DEBUG:boto:Token: None 2023-10-11T15:21:13.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY= 2023-10-11T15:21:13.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY=', 'Content-Length': '0'} 2023-10-11T15:21:13.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b891f32809a2994-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.286 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.287 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.287 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.288 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.288 DEBUG:boto:Method: GET 2023-10-11T15:21:13.289 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.289 DEBUG:boto:Data: 2023-10-11T15:21:13.289 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.290 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.291 DEBUG:boto:Params: {} 2023-10-11T15:21:13.291 DEBUG:boto:Token: None 2023-10-11T15:21:13.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY= 2023-10-11T15:21:13.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY=', 'Content-Length': '0'} 2023-10-11T15:21:13.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cf190c31b119c2-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.295 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.296 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.296 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.296 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.297 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.297 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.298 DEBUG:boto:Data: 2023-10-11T15:21:13.298 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.299 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.299 DEBUG:boto:Params: {} 2023-10-11T15:21:13.299 DEBUG:boto:Token: None 2023-10-11T15:21:13.300 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G0sABHuAFc6lqExn1SGprt3FBY8= 2023-10-11T15:21:13.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G0sABHuAFc6lqExn1SGprt3FBY8=', 'Content-Length': '0'} 2023-10-11T15:21:13.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1f29531d297539-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.304 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.305 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.305 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.306 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.306 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.307 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.307 DEBUG:boto:Data: 2023-10-11T15:21:13.308 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.308 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.308 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.309 DEBUG:boto:Params: {} 2023-10-11T15:21:13.309 DEBUG:boto:Token: None 2023-10-11T15:21:13.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.310 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G0sABHuAFc6lqExn1SGprt3FBY8= 2023-10-11T15:21:13.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G0sABHuAFc6lqExn1SGprt3FBY8=', 'Content-Length': '0'} 2023-10-11T15:21:13.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056de9b0a05b04e97-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:13.313 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.314 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.314 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.315 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.315 DEBUG:boto:Method: GET 2023-10-11T15:21:13.316 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.316 DEBUG:boto:Data: 2023-10-11T15:21:13.317 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.317 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.318 DEBUG:boto:Params: {} 2023-10-11T15:21:13.319 DEBUG:boto:Token: None 2023-10-11T15:21:13.319 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY= 2023-10-11T15:21:13.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY=', 'Content-Length': '0'} 2023-10-11T15:21:13.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bedfe327bdf7f28a-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.323 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.323 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.323 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.324 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.324 DEBUG:boto:Method: GET 2023-10-11T15:21:13.325 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.325 DEBUG:boto:Data: 2023-10-11T15:21:13.325 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.326 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.327 DEBUG:boto:Params: {} 2023-10-11T15:21:13.327 DEBUG:boto:Token: None 2023-10-11T15:21:13.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:13.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY= 2023-10-11T15:21:13.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VzPiHO4hkjwXfkmAG4CoQZK4eNY=', 'Content-Length': '0'} 2023-10-11T15:21:13.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003285e735a005ed15-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.331 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.332 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.332 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.332 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.333 DEBUG:boto:Method: GET 2023-10-11T15:21:13.333 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.334 DEBUG:boto:Data: 2023-10-11T15:21:13.334 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.335 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.335 DEBUG:boto:Params: {} 2023-10-11T15:21:13.336 DEBUG:boto:Token: None 2023-10-11T15:21:13.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367426512d2f8336-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.340 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.341 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.342 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.342 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.342 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.343 DEBUG:boto:Method: GET 2023-10-11T15:21:13.343 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.344 DEBUG:boto:Data: 2023-10-11T15:21:13.344 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.345 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.345 DEBUG:boto:Params: {} 2023-10-11T15:21:13.346 DEBUG:boto:Token: None 2023-10-11T15:21:13.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474a44ee1cbf0684-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.394 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.395 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.396 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.396 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.397 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:13.397 DEBUG:boto:Data: 2023-10-11T15:21:13.398 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.398 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.399 DEBUG:boto:Params: {} 2023-10-11T15:21:13.399 DEBUG:boto:Token: None 2023-10-11T15:21:13.400 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68 2023-10-11T15:21:13.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBr4CjFqQVr8ZbZhDd0TVoFvJ/s= 2023-10-11T15:21:13.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBr4CjFqQVr8ZbZhDd0TVoFvJ/s=', 'Content-Length': '0'} 2023-10-11T15:21:13.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d352e976837c05f-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.404 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.404 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.404 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.405 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:13.405 DEBUG:boto:Data: 2023-10-11T15:21:13.406 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.406 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.407 DEBUG:boto:Params: {} 2023-10-11T15:21:13.407 DEBUG:boto:Token: None 2023-10-11T15:21:13.408 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68 2023-10-11T15:21:13.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBr4CjFqQVr8ZbZhDd0TVoFvJ/s= 2023-10-11T15:21:13.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBr4CjFqQVr8ZbZhDd0TVoFvJ/s=', 'Content-Length': '0'} 2023-10-11T15:21:13.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006da91b2d3bcbdc6-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:13.412 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.412 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.413 DEBUG:boto:Method: GET 2023-10-11T15:21:13.413 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:13.414 DEBUG:boto:Data: 2023-10-11T15:21:13.414 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.415 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.415 DEBUG:boto:Params: {} 2023-10-11T15:21:13.416 DEBUG:boto:Token: None 2023-10-11T15:21:13.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68 2023-10-11T15:21:13.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q2U6osWQOX6L5NEC+RJsMXJy9ew= 2023-10-11T15:21:13.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q2U6osWQOX6L5NEC+RJsMXJy9ew=', 'Content-Length': '0'} 2023-10-11T15:21:13.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a06ec5790f17423-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.420 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:13.421 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:13.421 DEBUG:boto:Method: GET 2023-10-11T15:21:13.421 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:13.422 DEBUG:boto:Data: 2023-10-11T15:21:13.422 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.423 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.424 DEBUG:boto:Params: {} 2023-10-11T15:21:13.424 DEBUG:boto:Token: None 2023-10-11T15:21:13.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj68 2023-10-11T15:21:13.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q2U6osWQOX6L5NEC+RJsMXJy9ew= 2023-10-11T15:21:13.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q2U6osWQOX6L5NEC+RJsMXJy9ew=', 'Content-Length': '0'} 2023-10-11T15:21:13.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be33f7ad92daf67-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.429 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.429 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.429 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.430 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.430 DEBUG:boto:Method: GET 2023-10-11T15:21:13.431 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.431 DEBUG:boto:Data: 2023-10-11T15:21:13.432 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.432 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.432 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.433 DEBUG:boto:Params: {} 2023-10-11T15:21:13.433 DEBUG:boto:Token: None 2023-10-11T15:21:13.434 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cb25aeff019753c-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.482 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.483 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.484 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.484 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.485 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.485 DEBUG:boto:Method: GET 2023-10-11T15:21:13.486 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:13.486 DEBUG:boto:Data: 2023-10-11T15:21:13.486 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.487 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.488 DEBUG:boto:Params: {} 2023-10-11T15:21:13.488 DEBUG:boto:Token: None 2023-10-11T15:21:13.489 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eca960f115bbd67-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.493 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:13.494 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.495 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.495 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.496 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.496 DEBUG:boto:Method: GET 2023-10-11T15:21:13.497 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.497 DEBUG:boto:Data: 2023-10-11T15:21:13.498 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.498 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.499 DEBUG:boto:Params: {} 2023-10-11T15:21:13.499 DEBUG:boto:Token: None 2023-10-11T15:21:13.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU= 2023-10-11T15:21:13.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU=', 'Content-Length': '0'} 2023-10-11T15:21:13.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ffd8509da86f34-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.504 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.504 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.505 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.505 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.505 DEBUG:boto:Method: GET 2023-10-11T15:21:13.506 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.506 DEBUG:boto:Data: 2023-10-11T15:21:13.507 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.508 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.508 DEBUG:boto:Params: {} 2023-10-11T15:21:13.508 DEBUG:boto:Token: None 2023-10-11T15:21:13.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU= 2023-10-11T15:21:13.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU=', 'Content-Length': '0'} 2023-10-11T15:21:13.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d9a5ccd2c2293e-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.513 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.513 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.514 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.514 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.515 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.515 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.515 DEBUG:boto:Data: 2023-10-11T15:21:13.516 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.517 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.517 DEBUG:boto:Params: {} 2023-10-11T15:21:13.517 DEBUG:boto:Token: None 2023-10-11T15:21:13.518 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QdLWM46hme9Gqa8Jcz5wlVSXPwU= 2023-10-11T15:21:13.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QdLWM46hme9Gqa8Jcz5wlVSXPwU=', 'Content-Length': '0'} 2023-10-11T15:21:13.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46cd25b371d1bbe-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.521 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.522 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.522 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.523 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.523 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.524 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.524 DEBUG:boto:Data: 2023-10-11T15:21:13.525 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.525 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.526 DEBUG:boto:Params: {} 2023-10-11T15:21:13.526 DEBUG:boto:Token: None 2023-10-11T15:21:13.527 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QdLWM46hme9Gqa8Jcz5wlVSXPwU= 2023-10-11T15:21:13.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QdLWM46hme9Gqa8Jcz5wlVSXPwU=', 'Content-Length': '0'} 2023-10-11T15:21:13.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69168678e77dc76-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:13.531 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.531 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.532 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.532 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.533 DEBUG:boto:Method: GET 2023-10-11T15:21:13.533 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.534 DEBUG:boto:Data: 2023-10-11T15:21:13.534 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.535 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.536 DEBUG:boto:Params: {} 2023-10-11T15:21:13.536 DEBUG:boto:Token: None 2023-10-11T15:21:13.537 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU= 2023-10-11T15:21:13.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU=', 'Content-Length': '0'} 2023-10-11T15:21:13.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e47c0817844650-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.540 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.541 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.541 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.542 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.542 DEBUG:boto:Method: GET 2023-10-11T15:21:13.543 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.543 DEBUG:boto:Data: 2023-10-11T15:21:13.543 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.544 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.545 DEBUG:boto:Params: {} 2023-10-11T15:21:13.545 DEBUG:boto:Token: None 2023-10-11T15:21:13.546 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:13.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU= 2023-10-11T15:21:13.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oLQH4gDbL/TqiytC7t/hwdLdDMU=', 'Content-Length': '0'} 2023-10-11T15:21:13.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412e69ec979fed51-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.550 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.550 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.551 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.551 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.551 DEBUG:boto:Method: GET 2023-10-11T15:21:13.552 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.552 DEBUG:boto:Data: 2023-10-11T15:21:13.553 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.554 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.554 DEBUG:boto:Params: {} 2023-10-11T15:21:13.554 DEBUG:boto:Token: None 2023-10-11T15:21:13.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000697cabefca06f62c-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.559 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.560 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.560 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.561 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.561 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.562 DEBUG:boto:Method: GET 2023-10-11T15:21:13.562 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.563 DEBUG:boto:Data: 2023-10-11T15:21:13.563 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.564 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.564 DEBUG:boto:Params: {} 2023-10-11T15:21:13.565 DEBUG:boto:Token: None 2023-10-11T15:21:13.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0702f2908c535d-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.570 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.571 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.571 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.572 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.572 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:13.572 DEBUG:boto:Data: 2023-10-11T15:21:13.573 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.573 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.574 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.574 DEBUG:boto:Params: {} 2023-10-11T15:21:13.574 DEBUG:boto:Token: None 2023-10-11T15:21:13.575 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76 2023-10-11T15:21:13.575 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zefacGvPLhhMEiThmao8VkPtmWI= 2023-10-11T15:21:13.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zefacGvPLhhMEiThmao8VkPtmWI=', 'Content-Length': '0'} 2023-10-11T15:21:13.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619a0f490c5c7275-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.578 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.579 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.579 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.580 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:13.580 DEBUG:boto:Data: 2023-10-11T15:21:13.580 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.581 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.581 DEBUG:boto:Params: {} 2023-10-11T15:21:13.582 DEBUG:boto:Token: None 2023-10-11T15:21:13.582 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76 2023-10-11T15:21:13.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zefacGvPLhhMEiThmao8VkPtmWI= 2023-10-11T15:21:13.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zefacGvPLhhMEiThmao8VkPtmWI=', 'Content-Length': '0'} 2023-10-11T15:21:13.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f64e47f3417ad35-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:13.586 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.587 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.587 DEBUG:boto:Method: GET 2023-10-11T15:21:13.588 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:13.588 DEBUG:boto:Data: 2023-10-11T15:21:13.588 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.589 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.590 DEBUG:boto:Params: {} 2023-10-11T15:21:13.590 DEBUG:boto:Token: None 2023-10-11T15:21:13.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76 2023-10-11T15:21:13.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DyKkT+o6cqVFN+m+oLw6/F17aZo= 2023-10-11T15:21:13.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DyKkT+o6cqVFN+m+oLw6/F17aZo=', 'Content-Length': '0'} 2023-10-11T15:21:13.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053f85f085e7100d-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.634 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:13.635 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:13.635 DEBUG:boto:Method: GET 2023-10-11T15:21:13.636 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:13.636 DEBUG:boto:Data: 2023-10-11T15:21:13.637 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.637 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.638 DEBUG:boto:Params: {} 2023-10-11T15:21:13.638 DEBUG:boto:Token: None 2023-10-11T15:21:13.639 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj76 2023-10-11T15:21:13.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DyKkT+o6cqVFN+m+oLw6/F17aZo= 2023-10-11T15:21:13.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DyKkT+o6cqVFN+m+oLw6/F17aZo=', 'Content-Length': '0'} 2023-10-11T15:21:13.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ca9f297c059d17-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.643 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.643 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.644 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.644 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.645 DEBUG:boto:Method: GET 2023-10-11T15:21:13.645 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.646 DEBUG:boto:Data: 2023-10-11T15:21:13.646 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.647 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.647 DEBUG:boto:Params: {} 2023-10-11T15:21:13.648 DEBUG:boto:Token: None 2023-10-11T15:21:13.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.649 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9b3c9d12e48019-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.652 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.653 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.654 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.654 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.655 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.655 DEBUG:boto:Method: GET 2023-10-11T15:21:13.656 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:13.656 DEBUG:boto:Data: 2023-10-11T15:21:13.657 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.657 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.657 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.658 DEBUG:boto:Params: {} 2023-10-11T15:21:13.658 DEBUG:boto:Token: None 2023-10-11T15:21:13.659 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.659 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094151f699d30746e-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.706 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:13.707 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.708 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.708 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.709 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.709 DEBUG:boto:Method: GET 2023-10-11T15:21:13.709 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.710 DEBUG:boto:Data: 2023-10-11T15:21:13.710 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.711 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.712 DEBUG:boto:Params: {} 2023-10-11T15:21:13.712 DEBUG:boto:Token: None 2023-10-11T15:21:13.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo= 2023-10-11T15:21:13.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo=', 'Content-Length': '0'} 2023-10-11T15:21:13.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573107f7cb4195b8-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.716 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.717 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.717 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.717 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.718 DEBUG:boto:Method: GET 2023-10-11T15:21:13.718 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.719 DEBUG:boto:Data: 2023-10-11T15:21:13.719 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.720 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.720 DEBUG:boto:Params: {} 2023-10-11T15:21:13.721 DEBUG:boto:Token: None 2023-10-11T15:21:13.721 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.722 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo= 2023-10-11T15:21:13.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo=', 'Content-Length': '0'} 2023-10-11T15:21:13.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564067c1e3b7a325-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.725 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.725 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.726 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.726 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.727 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.727 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.728 DEBUG:boto:Data: 2023-10-11T15:21:13.728 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.729 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.729 DEBUG:boto:Params: {} 2023-10-11T15:21:13.730 DEBUG:boto:Token: None 2023-10-11T15:21:13.730 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.731 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:whX7zAchKJACtAkC9MPoQzoUKyA= 2023-10-11T15:21:13.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:whX7zAchKJACtAkC9MPoQzoUKyA=', 'Content-Length': '0'} 2023-10-11T15:21:13.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431de46b3288135e-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.734 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.734 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.735 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.735 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.735 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.736 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.736 DEBUG:boto:Data: 2023-10-11T15:21:13.737 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.737 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.738 DEBUG:boto:Params: {} 2023-10-11T15:21:13.738 DEBUG:boto:Token: None 2023-10-11T15:21:13.739 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:whX7zAchKJACtAkC9MPoQzoUKyA= 2023-10-11T15:21:13.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:whX7zAchKJACtAkC9MPoQzoUKyA=', 'Content-Length': '0'} 2023-10-11T15:21:13.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602efa6ed7bc9a57-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:13.743 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.743 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.744 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.744 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.744 DEBUG:boto:Method: GET 2023-10-11T15:21:13.745 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.745 DEBUG:boto:Data: 2023-10-11T15:21:13.746 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.746 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.747 DEBUG:boto:Params: {} 2023-10-11T15:21:13.747 DEBUG:boto:Token: None 2023-10-11T15:21:13.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo= 2023-10-11T15:21:13.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo=', 'Content-Length': '0'} 2023-10-11T15:21:13.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7c73a04124fc26-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.751 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.752 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.752 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.752 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.753 DEBUG:boto:Method: GET 2023-10-11T15:21:13.753 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.754 DEBUG:boto:Data: 2023-10-11T15:21:13.754 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.755 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.755 DEBUG:boto:Params: {} 2023-10-11T15:21:13.756 DEBUG:boto:Token: None 2023-10-11T15:21:13.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:13.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo= 2023-10-11T15:21:13.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4kfiB3WSt/LX/yvgnl/QYp45lo=', 'Content-Length': '0'} 2023-10-11T15:21:13.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e7032aed65fed1c-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.760 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.760 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.761 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.761 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.762 DEBUG:boto:Method: GET 2023-10-11T15:21:13.762 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.762 DEBUG:boto:Data: 2023-10-11T15:21:13.763 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.764 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.764 DEBUG:boto:Params: {} 2023-10-11T15:21:13.765 DEBUG:boto:Token: None 2023-10-11T15:21:13.765 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b929994a4f7a3a27-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.769 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.770 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.770 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.771 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.771 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.771 DEBUG:boto:Method: GET 2023-10-11T15:21:13.772 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.772 DEBUG:boto:Data: 2023-10-11T15:21:13.773 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.773 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.774 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.774 DEBUG:boto:Params: {} 2023-10-11T15:21:13.775 DEBUG:boto:Token: None 2023-10-11T15:21:13.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629a0b945475c229-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.779 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.780 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.780 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.781 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.781 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:13.782 DEBUG:boto:Data: 2023-10-11T15:21:13.782 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.782 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.783 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.783 DEBUG:boto:Params: {} 2023-10-11T15:21:13.784 DEBUG:boto:Token: None 2023-10-11T15:21:13.784 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85 2023-10-11T15:21:13.785 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aBoAuHfSnVwq8TKt6O0eCXUnRw8= 2023-10-11T15:21:13.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aBoAuHfSnVwq8TKt6O0eCXUnRw8=', 'Content-Length': '0'} 2023-10-11T15:21:13.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c80ef13f7a405a-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.787 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.788 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.788 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.789 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:13.789 DEBUG:boto:Data: 2023-10-11T15:21:13.789 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.790 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.791 DEBUG:boto:Params: {} 2023-10-11T15:21:13.791 DEBUG:boto:Token: None 2023-10-11T15:21:13.791 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85 2023-10-11T15:21:13.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aBoAuHfSnVwq8TKt6O0eCXUnRw8= 2023-10-11T15:21:13.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aBoAuHfSnVwq8TKt6O0eCXUnRw8=', 'Content-Length': '0'} 2023-10-11T15:21:13.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a6554227b61918-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:13.795 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.796 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.796 DEBUG:boto:Method: GET 2023-10-11T15:21:13.796 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:13.797 DEBUG:boto:Data: 2023-10-11T15:21:13.797 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.798 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.799 DEBUG:boto:Params: {} 2023-10-11T15:21:13.799 DEBUG:boto:Token: None 2023-10-11T15:21:13.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85 2023-10-11T15:21:13.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qgfif0OwrZmoc+0UkvBsPFe2q/Q= 2023-10-11T15:21:13.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qgfif0OwrZmoc+0UkvBsPFe2q/Q=', 'Content-Length': '0'} 2023-10-11T15:21:13.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971e7a3d475f34d4-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.846 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:13.847 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:13.847 DEBUG:boto:Method: GET 2023-10-11T15:21:13.848 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:13.848 DEBUG:boto:Data: 2023-10-11T15:21:13.849 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.849 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.849 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.850 DEBUG:boto:Params: {} 2023-10-11T15:21:13.850 DEBUG:boto:Token: None 2023-10-11T15:21:13.851 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj85 2023-10-11T15:21:13.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qgfif0OwrZmoc+0UkvBsPFe2q/Q= 2023-10-11T15:21:13.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qgfif0OwrZmoc+0UkvBsPFe2q/Q=', 'Content-Length': '0'} 2023-10-11T15:21:13.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008820fa32b9928145-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.855 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.855 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.856 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.856 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.856 DEBUG:boto:Method: GET 2023-10-11T15:21:13.857 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.857 DEBUG:boto:Data: 2023-10-11T15:21:13.858 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.859 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.859 DEBUG:boto:Params: {} 2023-10-11T15:21:13.860 DEBUG:boto:Token: None 2023-10-11T15:21:13.860 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9f60e47ea73f2c-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.866 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.867 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.867 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.868 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.868 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.869 DEBUG:boto:Method: GET 2023-10-11T15:21:13.869 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:13.870 DEBUG:boto:Data: 2023-10-11T15:21:13.870 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.871 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.871 DEBUG:boto:Params: {} 2023-10-11T15:21:13.872 DEBUG:boto:Token: None 2023-10-11T15:21:13.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db8ad9198344257-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.877 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:13.878 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.878 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.879 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.879 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.880 DEBUG:boto:Method: GET 2023-10-11T15:21:13.880 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.881 DEBUG:boto:Data: 2023-10-11T15:21:13.881 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.882 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.882 DEBUG:boto:Params: {} 2023-10-11T15:21:13.883 DEBUG:boto:Token: None 2023-10-11T15:21:13.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM= 2023-10-11T15:21:13.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM=', 'Content-Length': '0'} 2023-10-11T15:21:13.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002662d8e62d910f27-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.887 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.888 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.888 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.889 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.889 DEBUG:boto:Method: GET 2023-10-11T15:21:13.890 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.890 DEBUG:boto:Data: 2023-10-11T15:21:13.890 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.891 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.892 DEBUG:boto:Params: {} 2023-10-11T15:21:13.892 DEBUG:boto:Token: None 2023-10-11T15:21:13.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM= 2023-10-11T15:21:13.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM=', 'Content-Length': '0'} 2023-10-11T15:21:13.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000bcb2c2c6057620-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.897 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.897 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.897 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.898 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.898 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.899 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.899 DEBUG:boto:Data: 2023-10-11T15:21:13.900 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.900 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.901 DEBUG:boto:Params: {} 2023-10-11T15:21:13.901 DEBUG:boto:Token: None 2023-10-11T15:21:13.902 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:++fsKIG8bevBasN/ntHERLVzch4= 2023-10-11T15:21:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:++fsKIG8bevBasN/ntHERLVzch4=', 'Content-Length': '0'} 2023-10-11T15:21:13.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b192151715429c-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.905 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.905 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.906 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.906 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.907 DEBUG:boto:Method: HEAD 2023-10-11T15:21:13.907 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.907 DEBUG:boto:Data: 2023-10-11T15:21:13.908 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.909 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.909 DEBUG:boto:Params: {} 2023-10-11T15:21:13.910 DEBUG:boto:Token: None 2023-10-11T15:21:13.910 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:++fsKIG8bevBasN/ntHERLVzch4= 2023-10-11T15:21:13.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:++fsKIG8bevBasN/ntHERLVzch4=', 'Content-Length': '0'} 2023-10-11T15:21:13.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009107e6fdf77f1fba-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:13.914 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.914 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.915 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.915 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.916 DEBUG:boto:Method: GET 2023-10-11T15:21:13.916 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.917 DEBUG:boto:Data: 2023-10-11T15:21:13.917 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.918 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.918 DEBUG:boto:Params: {} 2023-10-11T15:21:13.919 DEBUG:boto:Token: None 2023-10-11T15:21:13.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM= 2023-10-11T15:21:13.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM=', 'Content-Length': '0'} 2023-10-11T15:21:13.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006237978ed6a4fec-006526bd69-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.970 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:13.971 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:13.972 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.972 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.972 DEBUG:boto:Method: GET 2023-10-11T15:21:13.973 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.973 DEBUG:boto:Data: 2023-10-11T15:21:13.974 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.974 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.975 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.975 DEBUG:boto:Params: {} 2023-10-11T15:21:13.976 DEBUG:boto:Token: None 2023-10-11T15:21:13.976 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:13.977 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM= 2023-10-11T15:21:13.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VYl3SJAymOgqGwbS63wfvWDQwmM=', 'Content-Length': '0'} 2023-10-11T15:21:13.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef016b18d04694f-006526bd69-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.980 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.980 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.981 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.981 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.982 DEBUG:boto:Method: GET 2023-10-11T15:21:13.982 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.982 DEBUG:boto:Data: 2023-10-11T15:21:13.983 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:13.984 DEBUG:boto:Port: 8001 2023-10-11T15:21:13.984 DEBUG:boto:Params: {} 2023-10-11T15:21:13.985 DEBUG:boto:Token: None 2023-10-11T15:21:13.985 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.986 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:13.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027450cf12f95bc3d-006526bd69-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:13.990 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:13.990 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:13.991 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:13.991 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.992 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.992 DEBUG:boto:Method: GET 2023-10-11T15:21:13.992 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:13.993 DEBUG:boto:Data: 2023-10-11T15:21:13.993 DEBUG:boto:Headers: {} 2023-10-11T15:21:13.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:13.994 DEBUG:boto:Port: 8000 2023-10-11T15:21:13.995 DEBUG:boto:Params: {} 2023-10-11T15:21:13.995 DEBUG:boto:Token: None 2023-10-11T15:21:13.996 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:13 GMT /dghqvd-59/ 2023-10-11T15:21:13.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0= 2023-10-11T15:21:13.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PjA3ViPbTOQNSBKnr0c6kv8iLM0=', 'Content-Length': '0'} 2023-10-11T15:21:14.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e822564aac2419c-006526bd69-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.042 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:14.043 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:14.044 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:14.044 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.044 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:14.045 DEBUG:boto:Data: 2023-10-11T15:21:14.045 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.046 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:14.046 DEBUG:boto:Port: 8001 2023-10-11T15:21:14.047 DEBUG:boto:Params: {} 2023-10-11T15:21:14.047 DEBUG:boto:Token: None 2023-10-11T15:21:14.048 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj95 2023-10-11T15:21:14.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dMhPHk9TV95kJRvMyLuOXul9OI= 2023-10-11T15:21:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dMhPHk9TV95kJRvMyLuOXul9OI=', 'Content-Length': '0'} 2023-10-11T15:21:14.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4761cc17adf10e9-006526bd6a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.051 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:14.052 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:14.052 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.053 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:14.053 DEBUG:boto:Data: 2023-10-11T15:21:14.053 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.054 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.055 DEBUG:boto:Params: {} 2023-10-11T15:21:14.055 DEBUG:boto:Token: None 2023-10-11T15:21:14.056 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj95 2023-10-11T15:21:14.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dMhPHk9TV95kJRvMyLuOXul9OI= 2023-10-11T15:21:14.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dMhPHk9TV95kJRvMyLuOXul9OI=', 'Content-Length': '0'} 2023-10-11T15:21:14.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573a15fd700c73d8-006526bd6a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:14.060 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:14.060 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:14.061 DEBUG:boto:Method: GET 2023-10-11T15:21:14.061 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:14.061 DEBUG:boto:Data: 2023-10-11T15:21:14.062 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.062 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:14.063 DEBUG:boto:Port: 8001 2023-10-11T15:21:14.063 DEBUG:boto:Params: {} 2023-10-11T15:21:14.064 DEBUG:boto:Token: None 2023-10-11T15:21:14.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj95 2023-10-11T15:21:14.065 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JwQs65GRtZbHbWd3BKItgAJWXhM= 2023-10-11T15:21:14.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JwQs65GRtZbHbWd3BKItgAJWXhM=', 'Content-Length': '0'} 2023-10-11T15:21:14.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084549edb8dc2913a-006526bd6a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.070 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:14.070 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:14.071 DEBUG:boto:Method: GET 2023-10-11T15:21:14.071 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:14.072 DEBUG:boto:Data: 2023-10-11T15:21:14.072 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.073 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.073 DEBUG:boto:Params: {} 2023-10-11T15:21:14.074 DEBUG:boto:Token: None 2023-10-11T15:21:14.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj95 2023-10-11T15:21:14.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JwQs65GRtZbHbWd3BKItgAJWXhM= 2023-10-11T15:21:14.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JwQs65GRtZbHbWd3BKItgAJWXhM=', 'Content-Length': '0'} 2023-10-11T15:21:14.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3ad320430c0e18-006526bd6a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.078 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.078 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.079 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.079 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.080 DEBUG:boto:Method: GET 2023-10-11T15:21:14.080 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.080 DEBUG:boto:Data: 2023-10-11T15:21:14.081 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:14.082 DEBUG:boto:Port: 8001 2023-10-11T15:21:14.082 DEBUG:boto:Params: {} 2023-10-11T15:21:14.083 DEBUG:boto:Token: None 2023-10-11T15:21:14.083 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ob15oFMpawhL13rtXrjlHf31PPM= 2023-10-11T15:21:14.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ob15oFMpawhL13rtXrjlHf31PPM=', 'Content-Length': '0'} 2023-10-11T15:21:14.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d765f47f704fac-006526bd6a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.130 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:14.131 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.132 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.132 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.132 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.133 DEBUG:boto:Method: GET 2023-10-11T15:21:14.133 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:14.134 DEBUG:boto:Data: 2023-10-11T15:21:14.134 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.135 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.135 DEBUG:boto:Params: {} 2023-10-11T15:21:14.136 DEBUG:boto:Token: None 2023-10-11T15:21:14.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ob15oFMpawhL13rtXrjlHf31PPM= 2023-10-11T15:21:14.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ob15oFMpawhL13rtXrjlHf31PPM=', 'Content-Length': '0'} 2023-10-11T15:21:14.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628f047a2ca3669a-006526bd6a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.141 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:14.141 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:21:14.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:21:14.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:21:14.143 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:21:14.205 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:14.205 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:14.256 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:21:14.256 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:21:14.263 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 15 2023-10-11T15:21:14.309 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.309 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.310 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.310 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:14.310 DEBUG:boto:Data: 2023-10-11T15:21:14.311 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.312 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.312 DEBUG:boto:Params: {} 2023-10-11T15:21:14.313 DEBUG:boto:Token: None 2023-10-11T15:21:14.313 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20= 2023-10-11T15:21:14.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20=', 'Content-Length': '0'} 2023-10-11T15:21:14.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b0f8e45884e770f-006526bd6a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.318 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:14.318 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:14.319 DEBUG:boto:Method: PUT 2023-10-11T15:21:14.319 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:21:14.320 DEBUG:boto:Data: 2023-10-11T15:21:14.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:14.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.321 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.321 DEBUG:boto:Params: {} 2023-10-11T15:21:14.322 DEBUG:boto:Token: None 2023-10-11T15:21:14.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj23 2023-10-11T15:21:14.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P3TBhBg8OBV1kGUIuLzN51gpips= 2023-10-11T15:21:14.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P3TBhBg8OBV1kGUIuLzN51gpips='} 2023-10-11T15:21:14.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e97547c1ffa2c5ae-006526bd6a-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.329 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.329 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.330 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.330 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:14.331 DEBUG:boto:Data: 2023-10-11T15:21:14.331 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.332 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.332 DEBUG:boto:Params: {} 2023-10-11T15:21:14.333 DEBUG:boto:Token: None 2023-10-11T15:21:14.333 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20= 2023-10-11T15:21:14.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20=', 'Content-Length': '0'} 2023-10-11T15:21:14.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000813ac5af6e2801ae-006526bd6a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.338 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:14.338 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:14.339 DEBUG:boto:Method: PUT 2023-10-11T15:21:14.339 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:21:14.340 DEBUG:boto:Data: 2023-10-11T15:21:14.340 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:14.340 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.341 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.341 DEBUG:boto:Params: {} 2023-10-11T15:21:14.342 DEBUG:boto:Token: None 2023-10-11T15:21:14.342 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj46 2023-10-11T15:21:14.343 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:II0aIamQMaOqHWfeFbq/VBUPABY= 2023-10-11T15:21:14.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:II0aIamQMaOqHWfeFbq/VBUPABY='} 2023-10-11T15:21:14.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e59f8c7505060f3-006526bd6a-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.391 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.392 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.392 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.392 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:14.393 DEBUG:boto:Data: 2023-10-11T15:21:14.393 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.394 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.395 DEBUG:boto:Params: {} 2023-10-11T15:21:14.395 DEBUG:boto:Token: None 2023-10-11T15:21:14.396 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20= 2023-10-11T15:21:14.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20=', 'Content-Length': '0'} 2023-10-11T15:21:14.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053f108089aca08a7-006526bd6a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.400 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:14.400 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:14.401 DEBUG:boto:Method: PUT 2023-10-11T15:21:14.401 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:21:14.402 DEBUG:boto:Data: 2023-10-11T15:21:14.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:14.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.403 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.403 DEBUG:boto:Params: {} 2023-10-11T15:21:14.404 DEBUG:boto:Token: None 2023-10-11T15:21:14.404 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj69 2023-10-11T15:21:14.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o4zPt+cOOzQfaBaKheVjDF15ygs= 2023-10-11T15:21:14.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o4zPt+cOOzQfaBaKheVjDF15ygs='} 2023-10-11T15:21:14.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008df5731b22dc73f5-006526bd6a-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.410 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.411 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.411 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.411 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:14.412 DEBUG:boto:Data: 2023-10-11T15:21:14.412 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.413 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.413 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.413 DEBUG:boto:Params: {} 2023-10-11T15:21:14.414 DEBUG:boto:Token: None 2023-10-11T15:21:14.414 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20= 2023-10-11T15:21:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20=', 'Content-Length': '0'} 2023-10-11T15:21:14.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a44797401f1c3b6-006526bd6a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.418 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:14.419 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:14.419 DEBUG:boto:Method: PUT 2023-10-11T15:21:14.420 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:21:14.420 DEBUG:boto:Data: 2023-10-11T15:21:14.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:14.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.421 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.422 DEBUG:boto:Params: {} 2023-10-11T15:21:14.422 DEBUG:boto:Token: None 2023-10-11T15:21:14.423 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj92 2023-10-11T15:21:14.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rfUWAYPPoTXlj57Qo4XT5BaSIlA= 2023-10-11T15:21:14.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rfUWAYPPoTXlj57Qo4XT5BaSIlA='} 2023-10-11T15:21:14.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc91e28a291b2a26-006526bd6a-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.467 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:14.467 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:14.468 DEBUG:boto:Method: HEAD 2023-10-11T15:21:14.468 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:14.468 DEBUG:boto:Data: 2023-10-11T15:21:14.469 DEBUG:boto:Headers: {} 2023-10-11T15:21:14.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.469 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.470 DEBUG:boto:Params: {} 2023-10-11T15:21:14.470 DEBUG:boto:Token: None 2023-10-11T15:21:14.471 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/ 2023-10-11T15:21:14.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20= 2023-10-11T15:21:14.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:01IFnm7sHDl4MIwy6dghqvbyQ20=', 'Content-Length': '0'} 2023-10-11T15:21:14.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055075770d5df777a-006526bd6a-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.475 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:14.475 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:14.476 DEBUG:boto:Method: PUT 2023-10-11T15:21:14.476 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:21:14.476 DEBUG:boto:Data: 2023-10-11T15:21:14.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:14.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:14.477 DEBUG:boto:Port: 8000 2023-10-11T15:21:14.478 DEBUG:boto:Params: {} 2023-10-11T15:21:14.478 DEBUG:boto:Token: None 2023-10-11T15:21:14.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:14 GMT /dghqvd-59/obj115 2023-10-11T15:21:14.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r/ol9FO52C00Dhkv0UET5/7Ai/M= 2023-10-11T15:21:14.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r/ol9FO52C00Dhkv0UET5/7Ai/M='} 2023-10-11T15:21:14.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb258c2faf9a9790-006526bd6a-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:14.485 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:21:14.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:14.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:14.487 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:21:14.545 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:14.545 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:14.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:14.580+0000 7f92d9bac780 1 waiting to reach latest gen 3, on 2.. 2023-10-11T15:21:19.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:19.582+0000 7f92d9bac780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:19.584 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:21:19.584 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:24.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:24.584+0000 7f92d9bac780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:24.586 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:24.586 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:29.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:29.584+0000 7f92d9bac780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:29.587 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:29.588 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:34.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:34.585+0000 7f92d9bac780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:34.588 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:34.589 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:39.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:39.587+0000 7f92d9bac780 1 bucket sync caught up with source: 2023-10-11T15:21:39.590 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:39.590 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000002.76.6, , 00000000002.71.6, , , , , , 00000000001.77.6, , 00000000001.48.6, , , , ] 2023-10-11T15:21:39.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:39.587+0000 7f92d9bac780 0 bucket checkpoint complete 2023-10-11T15:21:39.606 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:21:39.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:39.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:39.608 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:21:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:39.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:39.705+0000 7fef3e686780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:39.708 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000004.171.6, 00000000004.288.6] 2023-10-11T15:21:39.708 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:21:44.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:44.706+0000 7fef3e686780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:44.709 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000004.171.6, 00000000004.288.6] 2023-10-11T15:21:44.709 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:21:49.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:49.707+0000 7fef3e686780 1 waiting for incremental sync to catch up: 2023-10-11T15:21:49.710 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000004.171.6, 00000000004.288.6] 2023-10-11T15:21:49.710 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:21:54.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:54.709+0000 7fef3e686780 1 bucket sync caught up with source: 2023-10-11T15:21:54.712 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:21:54.712 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000002.101.6, 00000000002.288.6, , , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:21:54.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:54.709+0000 7fef3e686780 0 bucket checkpoint complete 2023-10-11T15:21:54.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:21:54.730 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.730 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.731 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.731 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:54.731 DEBUG:boto:Data: 2023-10-11T15:21:54.732 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.733 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.733 DEBUG:boto:Params: {} 2023-10-11T15:21:54.734 DEBUG:boto:Token: None 2023-10-11T15:21:54.734 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/ 2023-10-11T15:21:54.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q/dyBBvKAvO66wD/mkVKzdim4NA= 2023-10-11T15:21:54.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q/dyBBvKAvO66wD/mkVKzdim4NA=', 'Content-Length': '0'} 2023-10-11T15:21:54.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b25eac52019ff9f-006526bd92-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.739 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.739 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.740 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.740 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:54.740 DEBUG:boto:Data: 2023-10-11T15:21:54.741 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.741 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.742 DEBUG:boto:Params: {} 2023-10-11T15:21:54.742 DEBUG:boto:Token: None 2023-10-11T15:21:54.743 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/ 2023-10-11T15:21:54.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q/dyBBvKAvO66wD/mkVKzdim4NA= 2023-10-11T15:21:54.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q/dyBBvKAvO66wD/mkVKzdim4NA=', 'Content-Length': '0'} 2023-10-11T15:21:54.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000434794e613e40817-006526bd92-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:21:54.748 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.748 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.749 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.749 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.750 DEBUG:boto:Method: GET 2023-10-11T15:21:54.750 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.751 DEBUG:boto:Data: 2023-10-11T15:21:54.751 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.752 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.752 DEBUG:boto:Params: {} 2023-10-11T15:21:54.753 DEBUG:boto:Token: None 2023-10-11T15:21:54.753 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/?versions 2023-10-11T15:21:54.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw= 2023-10-11T15:21:54.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw=', 'Content-Length': '0'} 2023-10-11T15:21:54.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca959cd61e843825-006526bd92-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.758 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:21:54.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:21:54.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:21:54.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:21:54.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:21:54.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:21:54.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:21:54.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:21:54.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:21:54.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:21:54.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:21:54.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:21:54.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:21:54.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:21:54.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:21:54.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:21:54.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:21:54.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:21:54.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:21:54.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:21:54.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:21:54.767 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.767 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.768 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.768 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.769 DEBUG:boto:Method: GET 2023-10-11T15:21:54.769 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.770 DEBUG:boto:Data: 2023-10-11T15:21:54.770 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.771 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.771 DEBUG:boto:Params: {} 2023-10-11T15:21:54.772 DEBUG:boto:Token: None 2023-10-11T15:21:54.772 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/?versions 2023-10-11T15:21:54.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw= 2023-10-11T15:21:54.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw=', 'Content-Length': '0'} 2023-10-11T15:21:54.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92a4ed58c95cb84-006526bd92-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.777 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:21:54.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:21:54.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:21:54.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:21:54.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:21:54.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:21:54.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:21:54.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:21:54.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:21:54.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:21:54.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:21:54.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:21:54.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:21:54.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:21:54.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:21:54.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:21:54.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:21:54.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:21:54.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:21:54.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:21:54.786 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.787 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.787 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.787 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.788 DEBUG:boto:Method: GET 2023-10-11T15:21:54.788 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.789 DEBUG:boto:Data: 2023-10-11T15:21:54.789 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.790 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.790 DEBUG:boto:Params: {} 2023-10-11T15:21:54.791 DEBUG:boto:Token: None 2023-10-11T15:21:54.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/?versions 2023-10-11T15:21:54.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw= 2023-10-11T15:21:54.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw=', 'Content-Length': '0'} 2023-10-11T15:21:54.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5174dc685a6606-006526bd92-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.795 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.797 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.797 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.798 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.798 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.798 DEBUG:boto:Method: GET 2023-10-11T15:21:54.799 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:21:54.799 DEBUG:boto:Data: 2023-10-11T15:21:54.800 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.801 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.801 DEBUG:boto:Params: {} 2023-10-11T15:21:54.801 DEBUG:boto:Token: None 2023-10-11T15:21:54.802 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/?versions 2023-10-11T15:21:54.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw= 2023-10-11T15:21:54.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pue04bxTITRF8/4TrYy19qwgyJw=', 'Content-Length': '0'} 2023-10-11T15:21:54.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c53b635a6870c2-006526bd92-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.807 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:54.808 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.809 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.809 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.810 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.810 DEBUG:boto:Method: GET 2023-10-11T15:21:54.811 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.811 DEBUG:boto:Data: 2023-10-11T15:21:54.811 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.812 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.812 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.813 DEBUG:boto:Params: {} 2023-10-11T15:21:54.813 DEBUG:boto:Token: None 2023-10-11T15:21:54.814 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0= 2023-10-11T15:21:54.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0=', 'Content-Length': '0'} 2023-10-11T15:21:54.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed16f07e45b5c3bf-006526bd92-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.817 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.818 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.818 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.819 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.819 DEBUG:boto:Method: GET 2023-10-11T15:21:54.820 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.820 DEBUG:boto:Data: 2023-10-11T15:21:54.820 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.821 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.821 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.822 DEBUG:boto:Params: {} 2023-10-11T15:21:54.822 DEBUG:boto:Token: None 2023-10-11T15:21:54.823 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0= 2023-10-11T15:21:54.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0=', 'Content-Length': '0'} 2023-10-11T15:21:54.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000395422cc202172b7-006526bd92-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.870 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.871 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.871 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.872 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.872 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.873 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.873 DEBUG:boto:Data: 2023-10-11T15:21:54.873 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.874 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.875 DEBUG:boto:Params: {} 2023-10-11T15:21:54.875 DEBUG:boto:Token: None 2023-10-11T15:21:54.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vl9dsYuKvrLI8CSUoQHcwtQAj24= 2023-10-11T15:21:54.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vl9dsYuKvrLI8CSUoQHcwtQAj24=', 'Content-Length': '0'} 2023-10-11T15:21:54.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d27dc7ed35d2674-006526bd92-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.879 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.880 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.880 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.880 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.881 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.881 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.882 DEBUG:boto:Data: 2023-10-11T15:21:54.882 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.883 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.883 DEBUG:boto:Params: {} 2023-10-11T15:21:54.884 DEBUG:boto:Token: None 2023-10-11T15:21:54.884 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vl9dsYuKvrLI8CSUoQHcwtQAj24= 2023-10-11T15:21:54.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vl9dsYuKvrLI8CSUoQHcwtQAj24=', 'Content-Length': '0'} 2023-10-11T15:21:54.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194daf691137caf4-006526bd92-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:54.888 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.889 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.889 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.889 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.890 DEBUG:boto:Method: GET 2023-10-11T15:21:54.890 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.891 DEBUG:boto:Data: 2023-10-11T15:21:54.891 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.892 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.892 DEBUG:boto:Params: {} 2023-10-11T15:21:54.893 DEBUG:boto:Token: None 2023-10-11T15:21:54.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0= 2023-10-11T15:21:54.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0=', 'Content-Length': '0'} 2023-10-11T15:21:54.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003052aa424505f396-006526bd92-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.897 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.897 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.898 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.898 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.899 DEBUG:boto:Method: GET 2023-10-11T15:21:54.899 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.899 DEBUG:boto:Data: 2023-10-11T15:21:54.900 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.901 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.901 DEBUG:boto:Params: {} 2023-10-11T15:21:54.901 DEBUG:boto:Token: None 2023-10-11T15:21:54.902 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:21:54.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0= 2023-10-11T15:21:54.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xTogMUnDitP5k4vHuI8iWQ/Cb0=', 'Content-Length': '0'} 2023-10-11T15:21:54.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003793eccd42f05835-006526bd92-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.905 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.906 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.906 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.907 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.907 DEBUG:boto:Method: GET 2023-10-11T15:21:54.907 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.908 DEBUG:boto:Data: 2023-10-11T15:21:54.908 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.909 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.910 DEBUG:boto:Params: {} 2023-10-11T15:21:54.910 DEBUG:boto:Token: None 2023-10-11T15:21:54.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/ 2023-10-11T15:21:54.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eKWdjKdpC3OnE6QmA930reJ9bG8= 2023-10-11T15:21:54.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eKWdjKdpC3OnE6QmA930reJ9bG8=', 'Content-Length': '0'} 2023-10-11T15:21:54.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5063ff1668a8bb7-006526bd92-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.914 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.915 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:54.916 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:54.916 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.916 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.917 DEBUG:boto:Method: GET 2023-10-11T15:21:54.917 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:54.918 DEBUG:boto:Data: 2023-10-11T15:21:54.918 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.919 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.919 DEBUG:boto:Params: {} 2023-10-11T15:21:54.920 DEBUG:boto:Token: None 2023-10-11T15:21:54.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/ 2023-10-11T15:21:54.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eKWdjKdpC3OnE6QmA930reJ9bG8= 2023-10-11T15:21:54.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eKWdjKdpC3OnE6QmA930reJ9bG8=', 'Content-Length': '0'} 2023-10-11T15:21:54.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d9c3617ded74124-006526bd92-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.925 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:54.926 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.927 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.927 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.927 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:54.928 DEBUG:boto:Data: 2023-10-11T15:21:54.928 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.929 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.929 DEBUG:boto:Params: {} 2023-10-11T15:21:54.930 DEBUG:boto:Token: None 2023-10-11T15:21:54.930 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11 2023-10-11T15:21:54.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a4qiu8CD91kEMVrMndoLd0Nn7vc= 2023-10-11T15:21:54.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a4qiu8CD91kEMVrMndoLd0Nn7vc=', 'Content-Length': '0'} 2023-10-11T15:21:54.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eac0da4c8b7b165-006526bd92-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.934 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.934 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.935 DEBUG:boto:Method: HEAD 2023-10-11T15:21:54.935 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:54.935 DEBUG:boto:Data: 2023-10-11T15:21:54.936 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.937 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.937 DEBUG:boto:Params: {} 2023-10-11T15:21:54.937 DEBUG:boto:Token: None 2023-10-11T15:21:54.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11 2023-10-11T15:21:54.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a4qiu8CD91kEMVrMndoLd0Nn7vc= 2023-10-11T15:21:54.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a4qiu8CD91kEMVrMndoLd0Nn7vc=', 'Content-Length': '0'} 2023-10-11T15:21:54.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d62d4231149679-006526bd92-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:21:54.942 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.942 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.943 DEBUG:boto:Method: GET 2023-10-11T15:21:54.943 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:54.943 DEBUG:boto:Data: 2023-10-11T15:21:54.944 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:54.945 DEBUG:boto:Port: 8001 2023-10-11T15:21:54.945 DEBUG:boto:Params: {} 2023-10-11T15:21:54.946 DEBUG:boto:Token: None 2023-10-11T15:21:54.946 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11 2023-10-11T15:21:54.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aLKT87FkP+m0l9DSKOZRahxSnAo= 2023-10-11T15:21:54.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aLKT87FkP+m0l9DSKOZRahxSnAo=', 'Content-Length': '0'} 2023-10-11T15:21:54.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ac7a949498d2f5-006526bd92-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:54.950 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:21:54.950 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:21:54.951 DEBUG:boto:Method: GET 2023-10-11T15:21:54.951 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:21:54.951 DEBUG:boto:Data: 2023-10-11T15:21:54.952 DEBUG:boto:Headers: {} 2023-10-11T15:21:54.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:54.953 DEBUG:boto:Port: 8000 2023-10-11T15:21:54.953 DEBUG:boto:Params: {} 2023-10-11T15:21:54.954 DEBUG:boto:Token: None 2023-10-11T15:21:54.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:54 GMT /dghqvd-59/obj11 2023-10-11T15:21:54.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aLKT87FkP+m0l9DSKOZRahxSnAo= 2023-10-11T15:21:54.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aLKT87FkP+m0l9DSKOZRahxSnAo=', 'Content-Length': '0'} 2023-10-11T15:21:54.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd1ae6680db2bed-006526bd92-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.002 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.003 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.003 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.004 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.004 DEBUG:boto:Method: GET 2023-10-11T15:21:55.005 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.005 DEBUG:boto:Data: 2023-10-11T15:21:55.005 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.006 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.007 DEBUG:boto:Params: {} 2023-10-11T15:21:55.007 DEBUG:boto:Token: None 2023-10-11T15:21:55.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368cab42404d43f9-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.012 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.012 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.013 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.013 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.014 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.014 DEBUG:boto:Method: GET 2023-10-11T15:21:55.015 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:21:55.015 DEBUG:boto:Data: 2023-10-11T15:21:55.015 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.016 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.017 DEBUG:boto:Params: {} 2023-10-11T15:21:55.017 DEBUG:boto:Token: None 2023-10-11T15:21:55.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ace8931d01e23b-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.023 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:21:55.024 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.024 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.024 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.025 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.025 DEBUG:boto:Method: GET 2023-10-11T15:21:55.026 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.026 DEBUG:boto:Data: 2023-10-11T15:21:55.027 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.027 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.028 DEBUG:boto:Params: {} 2023-10-11T15:21:55.028 DEBUG:boto:Token: None 2023-10-11T15:21:55.029 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg= 2023-10-11T15:21:55.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg=', 'Content-Length': '0'} 2023-10-11T15:21:55.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567d26868ae555b0-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.074 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.075 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.075 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.076 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.076 DEBUG:boto:Method: GET 2023-10-11T15:21:55.077 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.077 DEBUG:boto:Data: 2023-10-11T15:21:55.078 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.078 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.079 DEBUG:boto:Params: {} 2023-10-11T15:21:55.079 DEBUG:boto:Token: None 2023-10-11T15:21:55.080 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg= 2023-10-11T15:21:55.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg=', 'Content-Length': '0'} 2023-10-11T15:21:55.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c80e57bb2b22f53-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.083 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.084 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.084 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.085 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.085 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.086 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.086 DEBUG:boto:Data: 2023-10-11T15:21:55.086 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.087 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.087 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.088 DEBUG:boto:Params: {} 2023-10-11T15:21:55.088 DEBUG:boto:Token: None 2023-10-11T15:21:55.088 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:orVzqGuEWyBUvEqeHUJq5Si9PtA= 2023-10-11T15:21:55.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:orVzqGuEWyBUvEqeHUJq5Si9PtA=', 'Content-Length': '0'} 2023-10-11T15:21:55.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84f86a3391d268-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.092 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.092 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.093 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.093 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.094 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.094 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.094 DEBUG:boto:Data: 2023-10-11T15:21:55.095 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.096 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.096 DEBUG:boto:Params: {} 2023-10-11T15:21:55.096 DEBUG:boto:Token: None 2023-10-11T15:21:55.097 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.097 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:orVzqGuEWyBUvEqeHUJq5Si9PtA= 2023-10-11T15:21:55.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:orVzqGuEWyBUvEqeHUJq5Si9PtA=', 'Content-Length': '0'} 2023-10-11T15:21:55.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded03afdf6647c75-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:21:55.101 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.101 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.102 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.102 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.103 DEBUG:boto:Method: GET 2023-10-11T15:21:55.103 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.103 DEBUG:boto:Data: 2023-10-11T15:21:55.104 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.105 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.105 DEBUG:boto:Params: {} 2023-10-11T15:21:55.106 DEBUG:boto:Token: None 2023-10-11T15:21:55.106 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.107 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg= 2023-10-11T15:21:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg=', 'Content-Length': '0'} 2023-10-11T15:21:55.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a258c470da310dbb-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.110 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.110 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.111 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.111 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.111 DEBUG:boto:Method: GET 2023-10-11T15:21:55.112 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.112 DEBUG:boto:Data: 2023-10-11T15:21:55.113 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.113 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.113 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.114 DEBUG:boto:Params: {} 2023-10-11T15:21:55.114 DEBUG:boto:Token: None 2023-10-11T15:21:55.115 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:21:55.115 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg= 2023-10-11T15:21:55.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aDe65YWarC23wrI0UAqBc6lZUfg=', 'Content-Length': '0'} 2023-10-11T15:21:55.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239b797ada1b74ba-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.118 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.119 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.119 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.119 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.120 DEBUG:boto:Method: GET 2023-10-11T15:21:55.120 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.121 DEBUG:boto:Data: 2023-10-11T15:21:55.121 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.122 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.122 DEBUG:boto:Params: {} 2023-10-11T15:21:55.123 DEBUG:boto:Token: None 2023-10-11T15:21:55.123 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004695ca848defa6cc-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.127 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.128 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.128 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.128 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.129 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.129 DEBUG:boto:Method: GET 2023-10-11T15:21:55.130 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.130 DEBUG:boto:Data: 2023-10-11T15:21:55.131 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.131 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.132 DEBUG:boto:Params: {} 2023-10-11T15:21:55.132 DEBUG:boto:Token: None 2023-10-11T15:21:55.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a23224ca8e3638-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.138 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.139 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.139 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.140 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.140 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:21:55.141 DEBUG:boto:Data: 2023-10-11T15:21:55.141 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.142 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.142 DEBUG:boto:Params: {} 2023-10-11T15:21:55.143 DEBUG:boto:Token: None 2023-10-11T15:21:55.143 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115 2023-10-11T15:21:55.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yUDMe2GN2uzCv0c/QF31ZCvaR30= 2023-10-11T15:21:55.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yUDMe2GN2uzCv0c/QF31ZCvaR30=', 'Content-Length': '0'} 2023-10-11T15:21:55.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615ba53d8a0de862-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.147 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.147 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.148 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.148 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:21:55.149 DEBUG:boto:Data: 2023-10-11T15:21:55.149 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.150 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.150 DEBUG:boto:Params: {} 2023-10-11T15:21:55.151 DEBUG:boto:Token: None 2023-10-11T15:21:55.151 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115 2023-10-11T15:21:55.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yUDMe2GN2uzCv0c/QF31ZCvaR30= 2023-10-11T15:21:55.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yUDMe2GN2uzCv0c/QF31ZCvaR30=', 'Content-Length': '0'} 2023-10-11T15:21:55.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee34348811fc8dc8-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:21:55.155 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.155 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.156 DEBUG:boto:Method: GET 2023-10-11T15:21:55.156 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:21:55.157 DEBUG:boto:Data: 2023-10-11T15:21:55.157 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.158 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.158 DEBUG:boto:Params: {} 2023-10-11T15:21:55.159 DEBUG:boto:Token: None 2023-10-11T15:21:55.159 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115 2023-10-11T15:21:55.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qlwOMuHRXOKctnxGrv4uzdjwdSY= 2023-10-11T15:21:55.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qlwOMuHRXOKctnxGrv4uzdjwdSY=', 'Content-Length': '0'} 2023-10-11T15:21:55.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b6eb0d5929453a-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.210 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:21:55.211 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:21:55.211 DEBUG:boto:Method: GET 2023-10-11T15:21:55.212 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:21:55.212 DEBUG:boto:Data: 2023-10-11T15:21:55.213 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.213 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.213 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.214 DEBUG:boto:Params: {} 2023-10-11T15:21:55.214 DEBUG:boto:Token: None 2023-10-11T15:21:55.215 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj115 2023-10-11T15:21:55.215 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qlwOMuHRXOKctnxGrv4uzdjwdSY= 2023-10-11T15:21:55.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qlwOMuHRXOKctnxGrv4uzdjwdSY=', 'Content-Length': '0'} 2023-10-11T15:21:55.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e8c576705878cb-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.218 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.219 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.219 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.220 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.220 DEBUG:boto:Method: GET 2023-10-11T15:21:55.221 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.221 DEBUG:boto:Data: 2023-10-11T15:21:55.221 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.222 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.223 DEBUG:boto:Params: {} 2023-10-11T15:21:55.223 DEBUG:boto:Token: None 2023-10-11T15:21:55.224 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f795cbacb88a371a-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.227 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.228 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.229 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.229 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.229 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.230 DEBUG:boto:Method: GET 2023-10-11T15:21:55.230 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:21:55.231 DEBUG:boto:Data: 2023-10-11T15:21:55.231 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.232 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.233 DEBUG:boto:Params: {} 2023-10-11T15:21:55.233 DEBUG:boto:Token: None 2023-10-11T15:21:55.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653461ee8f430a7e-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.240 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:55.241 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.242 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.242 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.242 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.243 DEBUG:boto:Method: GET 2023-10-11T15:21:55.243 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.244 DEBUG:boto:Data: 2023-10-11T15:21:55.244 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.245 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.245 DEBUG:boto:Params: {} 2023-10-11T15:21:55.246 DEBUG:boto:Token: None 2023-10-11T15:21:55.246 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.247 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E= 2023-10-11T15:21:55.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E=', 'Content-Length': '0'} 2023-10-11T15:21:55.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4fb448b64b7f9a7-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.250 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.251 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.251 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.252 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.252 DEBUG:boto:Method: GET 2023-10-11T15:21:55.252 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.253 DEBUG:boto:Data: 2023-10-11T15:21:55.253 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.254 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.255 DEBUG:boto:Params: {} 2023-10-11T15:21:55.255 DEBUG:boto:Token: None 2023-10-11T15:21:55.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E= 2023-10-11T15:21:55.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E=', 'Content-Length': '0'} 2023-10-11T15:21:55.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a99e30d8fb260f-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.302 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.303 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.303 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.304 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.304 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.305 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.305 DEBUG:boto:Data: 2023-10-11T15:21:55.305 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.306 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.307 DEBUG:boto:Params: {} 2023-10-11T15:21:55.307 DEBUG:boto:Token: None 2023-10-11T15:21:55.308 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9N943BkzajGViHxeOy6C/tZxUwU= 2023-10-11T15:21:55.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9N943BkzajGViHxeOy6C/tZxUwU=', 'Content-Length': '0'} 2023-10-11T15:21:55.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025b5bd45e663f090-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.312 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.312 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.312 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.313 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.313 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.313 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.314 DEBUG:boto:Data: 2023-10-11T15:21:55.314 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.315 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.315 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.315 DEBUG:boto:Params: {} 2023-10-11T15:21:55.316 DEBUG:boto:Token: None 2023-10-11T15:21:55.316 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9N943BkzajGViHxeOy6C/tZxUwU= 2023-10-11T15:21:55.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9N943BkzajGViHxeOy6C/tZxUwU=', 'Content-Length': '0'} 2023-10-11T15:21:55.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf829875afc745bd-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:55.321 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.321 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.322 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.322 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.322 DEBUG:boto:Method: GET 2023-10-11T15:21:55.323 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.323 DEBUG:boto:Data: 2023-10-11T15:21:55.324 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.324 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.325 DEBUG:boto:Params: {} 2023-10-11T15:21:55.325 DEBUG:boto:Token: None 2023-10-11T15:21:55.326 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E= 2023-10-11T15:21:55.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E=', 'Content-Length': '0'} 2023-10-11T15:21:55.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013134f056504fd4e-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.370 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.371 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.371 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.372 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.372 DEBUG:boto:Method: GET 2023-10-11T15:21:55.372 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.373 DEBUG:boto:Data: 2023-10-11T15:21:55.373 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.374 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.374 DEBUG:boto:Params: {} 2023-10-11T15:21:55.375 DEBUG:boto:Token: None 2023-10-11T15:21:55.375 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:21:55.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E= 2023-10-11T15:21:55.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z3Pc1vnkr3QioT0sRZvhpuZ2/8E=', 'Content-Length': '0'} 2023-10-11T15:21:55.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e40b6613c08ab4e-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.379 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.379 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.379 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.380 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.380 DEBUG:boto:Method: GET 2023-10-11T15:21:55.381 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.381 DEBUG:boto:Data: 2023-10-11T15:21:55.382 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.382 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.383 DEBUG:boto:Params: {} 2023-10-11T15:21:55.383 DEBUG:boto:Token: None 2023-10-11T15:21:55.384 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b20f943973f2e7-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.388 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.388 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.389 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.389 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.390 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.390 DEBUG:boto:Method: GET 2023-10-11T15:21:55.391 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.391 DEBUG:boto:Data: 2023-10-11T15:21:55.391 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.392 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.392 DEBUG:boto:Params: {} 2023-10-11T15:21:55.393 DEBUG:boto:Token: None 2023-10-11T15:21:55.393 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a64292542042d30-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.399 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.399 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.400 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.400 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.400 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:55.401 DEBUG:boto:Data: 2023-10-11T15:21:55.401 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.402 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.402 DEBUG:boto:Params: {} 2023-10-11T15:21:55.403 DEBUG:boto:Token: None 2023-10-11T15:21:55.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17 2023-10-11T15:21:55.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GczEhUtrsbLQaQbAaqocecCjjP0= 2023-10-11T15:21:55.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GczEhUtrsbLQaQbAaqocecCjjP0=', 'Content-Length': '0'} 2023-10-11T15:21:55.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833031df1575d991-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.406 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.407 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.407 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.408 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:55.408 DEBUG:boto:Data: 2023-10-11T15:21:55.409 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.409 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.410 DEBUG:boto:Params: {} 2023-10-11T15:21:55.410 DEBUG:boto:Token: None 2023-10-11T15:21:55.411 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17 2023-10-11T15:21:55.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GczEhUtrsbLQaQbAaqocecCjjP0= 2023-10-11T15:21:55.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GczEhUtrsbLQaQbAaqocecCjjP0=', 'Content-Length': '0'} 2023-10-11T15:21:55.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02576159644eb2d-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:21:55.414 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.415 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.415 DEBUG:boto:Method: GET 2023-10-11T15:21:55.416 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:55.416 DEBUG:boto:Data: 2023-10-11T15:21:55.416 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.417 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.418 DEBUG:boto:Params: {} 2023-10-11T15:21:55.418 DEBUG:boto:Token: None 2023-10-11T15:21:55.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17 2023-10-11T15:21:55.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CPWNtwrRykkZoMIT7Ojwdg/Ix7Y= 2023-10-11T15:21:55.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CPWNtwrRykkZoMIT7Ojwdg/Ix7Y=', 'Content-Length': '0'} 2023-10-11T15:21:55.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005484e2796e4f00e9-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.422 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:21:55.423 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:21:55.423 DEBUG:boto:Method: GET 2023-10-11T15:21:55.424 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:21:55.424 DEBUG:boto:Data: 2023-10-11T15:21:55.424 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.425 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.426 DEBUG:boto:Params: {} 2023-10-11T15:21:55.426 DEBUG:boto:Token: None 2023-10-11T15:21:55.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj17 2023-10-11T15:21:55.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CPWNtwrRykkZoMIT7Ojwdg/Ix7Y= 2023-10-11T15:21:55.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CPWNtwrRykkZoMIT7Ojwdg/Ix7Y=', 'Content-Length': '0'} 2023-10-11T15:21:55.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c7749764539329-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.430 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.430 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.431 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.431 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.432 DEBUG:boto:Method: GET 2023-10-11T15:21:55.432 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.433 DEBUG:boto:Data: 2023-10-11T15:21:55.433 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.434 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.434 DEBUG:boto:Params: {} 2023-10-11T15:21:55.435 DEBUG:boto:Token: None 2023-10-11T15:21:55.435 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.436 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2ea19f4c2fb630-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.439 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.440 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.440 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.440 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.441 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.441 DEBUG:boto:Method: GET 2023-10-11T15:21:55.442 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:21:55.442 DEBUG:boto:Data: 2023-10-11T15:21:55.443 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.443 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.444 DEBUG:boto:Params: {} 2023-10-11T15:21:55.444 DEBUG:boto:Token: None 2023-10-11T15:21:55.445 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b4810a1c39248a-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.449 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:55.450 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.451 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.451 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.451 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.452 DEBUG:boto:Method: GET 2023-10-11T15:21:55.452 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.453 DEBUG:boto:Data: 2023-10-11T15:21:55.453 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.454 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.454 DEBUG:boto:Params: {} 2023-10-11T15:21:55.455 DEBUG:boto:Token: None 2023-10-11T15:21:55.455 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8= 2023-10-11T15:21:55.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8=', 'Content-Length': '0'} 2023-10-11T15:21:55.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d6fd8f59e5bad7d-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.506 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.507 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.508 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.508 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.509 DEBUG:boto:Method: GET 2023-10-11T15:21:55.509 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.509 DEBUG:boto:Data: 2023-10-11T15:21:55.510 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.511 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.511 DEBUG:boto:Params: {} 2023-10-11T15:21:55.512 DEBUG:boto:Token: None 2023-10-11T15:21:55.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8= 2023-10-11T15:21:55.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8=', 'Content-Length': '0'} 2023-10-11T15:21:55.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac153d889cb595b-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.516 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.516 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.517 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.517 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.518 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.518 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.519 DEBUG:boto:Data: 2023-10-11T15:21:55.519 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.520 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.521 DEBUG:boto:Params: {} 2023-10-11T15:21:55.521 DEBUG:boto:Token: None 2023-10-11T15:21:55.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/e5wK5rGD6bLpEhvF/JbGF8Ea9o= 2023-10-11T15:21:55.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/e5wK5rGD6bLpEhvF/JbGF8Ea9o=', 'Content-Length': '0'} 2023-10-11T15:21:55.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873eeab1ca4b0213-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.525 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.526 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.526 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.527 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.527 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.528 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.528 DEBUG:boto:Data: 2023-10-11T15:21:55.528 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.529 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.530 DEBUG:boto:Params: {} 2023-10-11T15:21:55.530 DEBUG:boto:Token: None 2023-10-11T15:21:55.531 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/e5wK5rGD6bLpEhvF/JbGF8Ea9o= 2023-10-11T15:21:55.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/e5wK5rGD6bLpEhvF/JbGF8Ea9o=', 'Content-Length': '0'} 2023-10-11T15:21:55.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3224e0cf613e64-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:55.535 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.536 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.536 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.537 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.537 DEBUG:boto:Method: GET 2023-10-11T15:21:55.537 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.538 DEBUG:boto:Data: 2023-10-11T15:21:55.538 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.539 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.540 DEBUG:boto:Params: {} 2023-10-11T15:21:55.540 DEBUG:boto:Token: None 2023-10-11T15:21:55.541 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.541 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8= 2023-10-11T15:21:55.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8=', 'Content-Length': '0'} 2023-10-11T15:21:55.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce4b4954285c789-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.544 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.545 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.545 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.546 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.546 DEBUG:boto:Method: GET 2023-10-11T15:21:55.547 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.547 DEBUG:boto:Data: 2023-10-11T15:21:55.547 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.548 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.549 DEBUG:boto:Params: {} 2023-10-11T15:21:55.549 DEBUG:boto:Token: None 2023-10-11T15:21:55.550 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:21:55.550 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8= 2023-10-11T15:21:55.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ntt8G43ylJKrPL0/keFZgsIhUG8=', 'Content-Length': '0'} 2023-10-11T15:21:55.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c1175d78c6e7ac-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.553 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.554 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.554 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.555 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.555 DEBUG:boto:Method: GET 2023-10-11T15:21:55.556 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.556 DEBUG:boto:Data: 2023-10-11T15:21:55.557 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.557 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.558 DEBUG:boto:Params: {} 2023-10-11T15:21:55.558 DEBUG:boto:Token: None 2023-10-11T15:21:55.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050bf21bc25844dc3-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.563 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.563 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.564 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.564 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.565 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.565 DEBUG:boto:Method: GET 2023-10-11T15:21:55.566 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.566 DEBUG:boto:Data: 2023-10-11T15:21:55.567 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.567 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.568 DEBUG:boto:Params: {} 2023-10-11T15:21:55.568 DEBUG:boto:Token: None 2023-10-11T15:21:55.569 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069ca78293007ba7-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.574 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.574 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.575 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.575 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.576 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:55.576 DEBUG:boto:Data: 2023-10-11T15:21:55.576 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.577 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.578 DEBUG:boto:Params: {} 2023-10-11T15:21:55.578 DEBUG:boto:Token: None 2023-10-11T15:21:55.578 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19 2023-10-11T15:21:55.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ligu0hOT1VK1ge5JixpJq+VUTqE= 2023-10-11T15:21:55.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ligu0hOT1VK1ge5JixpJq+VUTqE=', 'Content-Length': '0'} 2023-10-11T15:21:55.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e94d66a40c2933-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.582 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.582 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.583 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.583 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:55.584 DEBUG:boto:Data: 2023-10-11T15:21:55.584 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.585 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.586 DEBUG:boto:Params: {} 2023-10-11T15:21:55.586 DEBUG:boto:Token: None 2023-10-11T15:21:55.587 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19 2023-10-11T15:21:55.587 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ligu0hOT1VK1ge5JixpJq+VUTqE= 2023-10-11T15:21:55.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ligu0hOT1VK1ge5JixpJq+VUTqE=', 'Content-Length': '0'} 2023-10-11T15:21:55.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d573b4e1e92bd20-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:21:55.591 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.591 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.591 DEBUG:boto:Method: GET 2023-10-11T15:21:55.592 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:55.592 DEBUG:boto:Data: 2023-10-11T15:21:55.593 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.594 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.594 DEBUG:boto:Params: {} 2023-10-11T15:21:55.595 DEBUG:boto:Token: None 2023-10-11T15:21:55.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19 2023-10-11T15:21:55.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MpoaoHlah5xZXkoJw6+gG2Yw8p4= 2023-10-11T15:21:55.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MpoaoHlah5xZXkoJw6+gG2Yw8p4=', 'Content-Length': '0'} 2023-10-11T15:21:55.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c6dbba99ec9b67-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.599 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:21:55.599 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:21:55.600 DEBUG:boto:Method: GET 2023-10-11T15:21:55.600 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:21:55.601 DEBUG:boto:Data: 2023-10-11T15:21:55.601 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.602 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.602 DEBUG:boto:Params: {} 2023-10-11T15:21:55.603 DEBUG:boto:Token: None 2023-10-11T15:21:55.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj19 2023-10-11T15:21:55.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MpoaoHlah5xZXkoJw6+gG2Yw8p4= 2023-10-11T15:21:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MpoaoHlah5xZXkoJw6+gG2Yw8p4=', 'Content-Length': '0'} 2023-10-11T15:21:55.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca09dd7a733314a3-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.607 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.607 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.608 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.608 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.609 DEBUG:boto:Method: GET 2023-10-11T15:21:55.609 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.610 DEBUG:boto:Data: 2023-10-11T15:21:55.610 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.611 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.611 DEBUG:boto:Params: {} 2023-10-11T15:21:55.612 DEBUG:boto:Token: None 2023-10-11T15:21:55.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b3532b3f67b125-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.662 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.663 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.664 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.664 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.665 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.665 DEBUG:boto:Method: GET 2023-10-11T15:21:55.666 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:21:55.666 DEBUG:boto:Data: 2023-10-11T15:21:55.666 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.667 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.668 DEBUG:boto:Params: {} 2023-10-11T15:21:55.668 DEBUG:boto:Token: None 2023-10-11T15:21:55.669 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081452604c0739af0-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.674 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:55.675 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.675 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.676 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.676 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.676 DEBUG:boto:Method: GET 2023-10-11T15:21:55.677 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.677 DEBUG:boto:Data: 2023-10-11T15:21:55.678 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.679 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.679 DEBUG:boto:Params: {} 2023-10-11T15:21:55.679 DEBUG:boto:Token: None 2023-10-11T15:21:55.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI= 2023-10-11T15:21:55.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI=', 'Content-Length': '0'} 2023-10-11T15:21:55.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a157324643cbf9-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.684 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.684 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.685 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.685 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.686 DEBUG:boto:Method: GET 2023-10-11T15:21:55.686 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.687 DEBUG:boto:Data: 2023-10-11T15:21:55.687 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.688 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.688 DEBUG:boto:Params: {} 2023-10-11T15:21:55.689 DEBUG:boto:Token: None 2023-10-11T15:21:55.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI= 2023-10-11T15:21:55.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI=', 'Content-Length': '0'} 2023-10-11T15:21:55.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d50f3f5263de7a8-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.693 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.694 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.694 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.694 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.695 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.695 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.696 DEBUG:boto:Data: 2023-10-11T15:21:55.696 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.697 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.697 DEBUG:boto:Params: {} 2023-10-11T15:21:55.698 DEBUG:boto:Token: None 2023-10-11T15:21:55.698 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CpFPZoxavvt41dLFs6h9bdVE01o= 2023-10-11T15:21:55.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CpFPZoxavvt41dLFs6h9bdVE01o=', 'Content-Length': '0'} 2023-10-11T15:21:55.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4f41f551ab18cc-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.702 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.702 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.703 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.703 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.703 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.704 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.704 DEBUG:boto:Data: 2023-10-11T15:21:55.704 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.705 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.706 DEBUG:boto:Params: {} 2023-10-11T15:21:55.706 DEBUG:boto:Token: None 2023-10-11T15:21:55.706 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CpFPZoxavvt41dLFs6h9bdVE01o= 2023-10-11T15:21:55.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CpFPZoxavvt41dLFs6h9bdVE01o=', 'Content-Length': '0'} 2023-10-11T15:21:55.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058cf3a43b6ba0bf-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:55.710 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.711 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.711 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.712 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.712 DEBUG:boto:Method: GET 2023-10-11T15:21:55.713 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.713 DEBUG:boto:Data: 2023-10-11T15:21:55.713 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.714 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.715 DEBUG:boto:Params: {} 2023-10-11T15:21:55.715 DEBUG:boto:Token: None 2023-10-11T15:21:55.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI= 2023-10-11T15:21:55.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI=', 'Content-Length': '0'} 2023-10-11T15:21:55.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002546264b2be01c35-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.719 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.719 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.720 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.720 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.721 DEBUG:boto:Method: GET 2023-10-11T15:21:55.721 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.722 DEBUG:boto:Data: 2023-10-11T15:21:55.722 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.723 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.723 DEBUG:boto:Params: {} 2023-10-11T15:21:55.724 DEBUG:boto:Token: None 2023-10-11T15:21:55.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:21:55.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI= 2023-10-11T15:21:55.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9HDsOFxoRZEhOiufLlhDGG/YxVI=', 'Content-Length': '0'} 2023-10-11T15:21:55.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d47da377b751c0-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.728 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.728 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.729 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.729 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.729 DEBUG:boto:Method: GET 2023-10-11T15:21:55.730 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.730 DEBUG:boto:Data: 2023-10-11T15:21:55.731 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.732 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.732 DEBUG:boto:Params: {} 2023-10-11T15:21:55.733 DEBUG:boto:Token: None 2023-10-11T15:21:55.733 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051aaeaa8917d6892-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.778 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.779 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.780 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.780 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.780 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.781 DEBUG:boto:Method: GET 2023-10-11T15:21:55.781 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.781 DEBUG:boto:Data: 2023-10-11T15:21:55.782 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.782 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.783 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.783 DEBUG:boto:Params: {} 2023-10-11T15:21:55.784 DEBUG:boto:Token: None 2023-10-11T15:21:55.784 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b2feb48c15a91b-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.789 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.789 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.790 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.790 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.791 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:55.791 DEBUG:boto:Data: 2023-10-11T15:21:55.792 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.792 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.793 DEBUG:boto:Params: {} 2023-10-11T15:21:55.793 DEBUG:boto:Token: None 2023-10-11T15:21:55.793 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22 2023-10-11T15:21:55.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AoC346iCHd9BFKamDcky6OUSbRk= 2023-10-11T15:21:55.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AoC346iCHd9BFKamDcky6OUSbRk=', 'Content-Length': '0'} 2023-10-11T15:21:55.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096706e6e8504dd4b-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.797 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.797 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.798 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.798 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:55.799 DEBUG:boto:Data: 2023-10-11T15:21:55.799 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.800 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.800 DEBUG:boto:Params: {} 2023-10-11T15:21:55.801 DEBUG:boto:Token: None 2023-10-11T15:21:55.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22 2023-10-11T15:21:55.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AoC346iCHd9BFKamDcky6OUSbRk= 2023-10-11T15:21:55.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AoC346iCHd9BFKamDcky6OUSbRk=', 'Content-Length': '0'} 2023-10-11T15:21:55.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a2e0ded22a32ec-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:21:55.806 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.806 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.807 DEBUG:boto:Method: GET 2023-10-11T15:21:55.807 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:55.808 DEBUG:boto:Data: 2023-10-11T15:21:55.808 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.809 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.809 DEBUG:boto:Params: {} 2023-10-11T15:21:55.810 DEBUG:boto:Token: None 2023-10-11T15:21:55.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22 2023-10-11T15:21:55.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPXnVcor+CyY3wn7yNphsvYtSsE= 2023-10-11T15:21:55.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPXnVcor+CyY3wn7yNphsvYtSsE=', 'Content-Length': '0'} 2023-10-11T15:21:55.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b0fc76165d2a42-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.813 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:21:55.814 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:21:55.814 DEBUG:boto:Method: GET 2023-10-11T15:21:55.815 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:21:55.815 DEBUG:boto:Data: 2023-10-11T15:21:55.816 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.816 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.817 DEBUG:boto:Params: {} 2023-10-11T15:21:55.817 DEBUG:boto:Token: None 2023-10-11T15:21:55.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj22 2023-10-11T15:21:55.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IPXnVcor+CyY3wn7yNphsvYtSsE= 2023-10-11T15:21:55.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IPXnVcor+CyY3wn7yNphsvYtSsE=', 'Content-Length': '0'} 2023-10-11T15:21:55.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e54812962a9c9a-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.862 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.863 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.863 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.864 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.864 DEBUG:boto:Method: GET 2023-10-11T15:21:55.865 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.865 DEBUG:boto:Data: 2023-10-11T15:21:55.865 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.866 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.867 DEBUG:boto:Params: {} 2023-10-11T15:21:55.867 DEBUG:boto:Token: None 2023-10-11T15:21:55.868 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6012f13d771f1b9-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.872 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.872 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.873 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.873 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.874 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.874 DEBUG:boto:Method: GET 2023-10-11T15:21:55.875 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:21:55.875 DEBUG:boto:Data: 2023-10-11T15:21:55.876 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.876 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.876 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.877 DEBUG:boto:Params: {} 2023-10-11T15:21:55.877 DEBUG:boto:Token: None 2023-10-11T15:21:55.878 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.878 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328f431aecc0edf6-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.883 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:21:55.884 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.884 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.885 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.885 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.886 DEBUG:boto:Method: GET 2023-10-11T15:21:55.886 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.886 DEBUG:boto:Data: 2023-10-11T15:21:55.887 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.888 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.888 DEBUG:boto:Params: {} 2023-10-11T15:21:55.889 DEBUG:boto:Token: None 2023-10-11T15:21:55.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.890 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E= 2023-10-11T15:21:55.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E=', 'Content-Length': '0'} 2023-10-11T15:21:55.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cd672d512ee76d-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.893 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.894 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.894 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.895 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.895 DEBUG:boto:Method: GET 2023-10-11T15:21:55.895 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.896 DEBUG:boto:Data: 2023-10-11T15:21:55.896 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.897 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.898 DEBUG:boto:Params: {} 2023-10-11T15:21:55.898 DEBUG:boto:Token: None 2023-10-11T15:21:55.899 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.899 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E= 2023-10-11T15:21:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E=', 'Content-Length': '0'} 2023-10-11T15:21:55.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce44ae26c504eba-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.902 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.903 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.903 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.904 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.904 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.904 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.905 DEBUG:boto:Data: 2023-10-11T15:21:55.905 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.906 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.906 DEBUG:boto:Params: {} 2023-10-11T15:21:55.907 DEBUG:boto:Token: None 2023-10-11T15:21:55.907 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BHUWoENCeVEN954JQx0aDIW+2vI= 2023-10-11T15:21:55.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BHUWoENCeVEN954JQx0aDIW+2vI=', 'Content-Length': '0'} 2023-10-11T15:21:55.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000217211b68a0a27db-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.911 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.911 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.912 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.912 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.912 DEBUG:boto:Method: HEAD 2023-10-11T15:21:55.913 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.913 DEBUG:boto:Data: 2023-10-11T15:21:55.914 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.915 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.915 DEBUG:boto:Params: {} 2023-10-11T15:21:55.915 DEBUG:boto:Token: None 2023-10-11T15:21:55.916 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.916 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BHUWoENCeVEN954JQx0aDIW+2vI= 2023-10-11T15:21:55.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BHUWoENCeVEN954JQx0aDIW+2vI=', 'Content-Length': '0'} 2023-10-11T15:21:55.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8a6484bb733886-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:21:55.920 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.920 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.921 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.921 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.921 DEBUG:boto:Method: GET 2023-10-11T15:21:55.922 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.922 DEBUG:boto:Data: 2023-10-11T15:21:55.923 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.923 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.924 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.924 DEBUG:boto:Params: {} 2023-10-11T15:21:55.924 DEBUG:boto:Token: None 2023-10-11T15:21:55.925 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E= 2023-10-11T15:21:55.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E=', 'Content-Length': '0'} 2023-10-11T15:21:55.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e5b13e1e1bc448-006526bd93-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.970 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:55.971 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:55.971 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.972 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.972 DEBUG:boto:Method: GET 2023-10-11T15:21:55.973 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.973 DEBUG:boto:Data: 2023-10-11T15:21:55.974 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.974 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.974 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.975 DEBUG:boto:Params: {} 2023-10-11T15:21:55.975 DEBUG:boto:Token: None 2023-10-11T15:21:55.976 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:21:55.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E= 2023-10-11T15:21:55.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJUdEekFbRZInELrUp31h0zTa/E=', 'Content-Length': '0'} 2023-10-11T15:21:55.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000445b4d907b16a9fb-006526bd93-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.980 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.980 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.981 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.981 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.981 DEBUG:boto:Method: GET 2023-10-11T15:21:55.982 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.982 DEBUG:boto:Data: 2023-10-11T15:21:55.983 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:55.984 DEBUG:boto:Port: 8001 2023-10-11T15:21:55.984 DEBUG:boto:Params: {} 2023-10-11T15:21:55.984 DEBUG:boto:Token: None 2023-10-11T15:21:55.985 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.985 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:55.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b24bb9f04a0e60a-006526bd93-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:55.989 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:55.990 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:55.990 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:55.991 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.991 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.992 DEBUG:boto:Method: GET 2023-10-11T15:21:55.992 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:21:55.993 DEBUG:boto:Data: 2023-10-11T15:21:55.993 DEBUG:boto:Headers: {} 2023-10-11T15:21:55.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:55.994 DEBUG:boto:Port: 8000 2023-10-11T15:21:55.994 DEBUG:boto:Params: {} 2023-10-11T15:21:55.995 DEBUG:boto:Token: None 2023-10-11T15:21:55.995 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:55 GMT /dghqvd-59/ 2023-10-11T15:21:55.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM= 2023-10-11T15:21:55.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:55 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HyQqQ+A/Lp+CCnQQVZBh9yjx0pM=', 'Content-Length': '0'} 2023-10-11T15:21:56.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002242f9ca2d2f38d4-006526bd93-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:55 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.043 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.044 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:56.044 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:56.045 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.045 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:21:56.045 DEBUG:boto:Data: 2023-10-11T15:21:56.046 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.046 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.047 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.047 DEBUG:boto:Params: {} 2023-10-11T15:21:56.047 DEBUG:boto:Token: None 2023-10-11T15:21:56.048 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj23 2023-10-11T15:21:56.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMNq0Yzh/e0Rll2zQlehWxgYm+M= 2023-10-11T15:21:56.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMNq0Yzh/e0Rll2zQlehWxgYm+M=', 'Content-Length': '0'} 2023-10-11T15:21:56.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abf2ce56f3cb783-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.052 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:56.052 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:56.052 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.053 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:21:56.053 DEBUG:boto:Data: 2023-10-11T15:21:56.054 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.055 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.055 DEBUG:boto:Params: {} 2023-10-11T15:21:56.055 DEBUG:boto:Token: None 2023-10-11T15:21:56.056 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj23 2023-10-11T15:21:56.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMNq0Yzh/e0Rll2zQlehWxgYm+M= 2023-10-11T15:21:56.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMNq0Yzh/e0Rll2zQlehWxgYm+M=', 'Content-Length': '0'} 2023-10-11T15:21:56.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0e3ee4c64d6990-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:21:56.060 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:56.060 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:56.061 DEBUG:boto:Method: GET 2023-10-11T15:21:56.061 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:21:56.062 DEBUG:boto:Data: 2023-10-11T15:21:56.062 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.063 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.063 DEBUG:boto:Params: {} 2023-10-11T15:21:56.064 DEBUG:boto:Token: None 2023-10-11T15:21:56.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj23 2023-10-11T15:21:56.065 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r7f/VILILlyoTOjaZhwc6CXwUWI= 2023-10-11T15:21:56.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r7f/VILILlyoTOjaZhwc6CXwUWI=', 'Content-Length': '0'} 2023-10-11T15:21:56.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96e52d40e4a3f44-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.068 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:21:56.069 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:21:56.069 DEBUG:boto:Method: GET 2023-10-11T15:21:56.070 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:21:56.070 DEBUG:boto:Data: 2023-10-11T15:21:56.071 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.071 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.072 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.072 DEBUG:boto:Params: {} 2023-10-11T15:21:56.072 DEBUG:boto:Token: None 2023-10-11T15:21:56.073 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj23 2023-10-11T15:21:56.073 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r7f/VILILlyoTOjaZhwc6CXwUWI= 2023-10-11T15:21:56.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r7f/VILILlyoTOjaZhwc6CXwUWI=', 'Content-Length': '0'} 2023-10-11T15:21:56.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e7787b53917345-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.077 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.077 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.078 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.078 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.079 DEBUG:boto:Method: GET 2023-10-11T15:21:56.079 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.079 DEBUG:boto:Data: 2023-10-11T15:21:56.080 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.081 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.081 DEBUG:boto:Params: {} 2023-10-11T15:21:56.082 DEBUG:boto:Token: None 2023-10-11T15:21:56.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b494248e04db55-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.086 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.087 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.088 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.088 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.089 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.089 DEBUG:boto:Method: GET 2023-10-11T15:21:56.090 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:21:56.090 DEBUG:boto:Data: 2023-10-11T15:21:56.091 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.092 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.092 DEBUG:boto:Params: {} 2023-10-11T15:21:56.093 DEBUG:boto:Token: None 2023-10-11T15:21:56.093 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e59b4250d35aea-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.098 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:56.099 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.100 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.100 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.101 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.101 DEBUG:boto:Method: GET 2023-10-11T15:21:56.102 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.102 DEBUG:boto:Data: 2023-10-11T15:21:56.102 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.103 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.103 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.104 DEBUG:boto:Params: {} 2023-10-11T15:21:56.104 DEBUG:boto:Token: None 2023-10-11T15:21:56.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w= 2023-10-11T15:21:56.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ee5aa4c74c1aee-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.108 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.109 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.109 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.110 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.110 DEBUG:boto:Method: GET 2023-10-11T15:21:56.111 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.111 DEBUG:boto:Data: 2023-10-11T15:21:56.111 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.112 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.113 DEBUG:boto:Params: {} 2023-10-11T15:21:56.113 DEBUG:boto:Token: None 2023-10-11T15:21:56.113 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w= 2023-10-11T15:21:56.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1617ba20874558d-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.117 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.118 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.118 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.119 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.119 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.119 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.120 DEBUG:boto:Data: 2023-10-11T15:21:56.120 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.121 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.121 DEBUG:boto:Params: {} 2023-10-11T15:21:56.122 DEBUG:boto:Token: None 2023-10-11T15:21:56.122 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vaX5f4p0moEdw0Bx3PulAEhGvMY= 2023-10-11T15:21:56.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vaX5f4p0moEdw0Bx3PulAEhGvMY=', 'Content-Length': '0'} 2023-10-11T15:21:56.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eadbff6c77750b4-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.125 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.126 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.126 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.127 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.127 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.127 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.128 DEBUG:boto:Data: 2023-10-11T15:21:56.128 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.129 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.129 DEBUG:boto:Params: {} 2023-10-11T15:21:56.130 DEBUG:boto:Token: None 2023-10-11T15:21:56.130 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vaX5f4p0moEdw0Bx3PulAEhGvMY= 2023-10-11T15:21:56.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vaX5f4p0moEdw0Bx3PulAEhGvMY=', 'Content-Length': '0'} 2023-10-11T15:21:56.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf9f0012610323f-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:56.134 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.135 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.135 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.135 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.136 DEBUG:boto:Method: GET 2023-10-11T15:21:56.136 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.137 DEBUG:boto:Data: 2023-10-11T15:21:56.137 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.138 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.138 DEBUG:boto:Params: {} 2023-10-11T15:21:56.139 DEBUG:boto:Token: None 2023-10-11T15:21:56.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w= 2023-10-11T15:21:56.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f67d305953708e-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.143 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.143 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.144 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.144 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.144 DEBUG:boto:Method: GET 2023-10-11T15:21:56.145 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.145 DEBUG:boto:Data: 2023-10-11T15:21:56.146 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.147 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.147 DEBUG:boto:Params: {} 2023-10-11T15:21:56.148 DEBUG:boto:Token: None 2023-10-11T15:21:56.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:21:56.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w= 2023-10-11T15:21:56.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q/Yd1v4JfHymgAZD+K+UhQS65/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2ba0ecb5f5db12-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.152 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.152 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.152 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.153 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.153 DEBUG:boto:Method: GET 2023-10-11T15:21:56.154 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.154 DEBUG:boto:Data: 2023-10-11T15:21:56.155 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.155 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.156 DEBUG:boto:Params: {} 2023-10-11T15:21:56.156 DEBUG:boto:Token: None 2023-10-11T15:21:56.157 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f086a2c9ec3c353e-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.161 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.161 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.162 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.162 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.163 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.163 DEBUG:boto:Method: GET 2023-10-11T15:21:56.164 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.164 DEBUG:boto:Data: 2023-10-11T15:21:56.165 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.165 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.166 DEBUG:boto:Params: {} 2023-10-11T15:21:56.166 DEBUG:boto:Token: None 2023-10-11T15:21:56.167 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154ca01f5e16bf1a-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.214 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.215 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.215 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.216 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.216 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:56.217 DEBUG:boto:Data: 2023-10-11T15:21:56.217 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.217 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.218 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.218 DEBUG:boto:Params: {} 2023-10-11T15:21:56.219 DEBUG:boto:Token: None 2023-10-11T15:21:56.219 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33 2023-10-11T15:21:56.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UL6R0A4LagjrkhD90VQySHONi44= 2023-10-11T15:21:56.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UL6R0A4LagjrkhD90VQySHONi44=', 'Content-Length': '0'} 2023-10-11T15:21:56.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002962cb32848a41d8-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.223 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.223 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.224 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.224 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:56.225 DEBUG:boto:Data: 2023-10-11T15:21:56.225 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.225 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.226 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.226 DEBUG:boto:Params: {} 2023-10-11T15:21:56.227 DEBUG:boto:Token: None 2023-10-11T15:21:56.227 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33 2023-10-11T15:21:56.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UL6R0A4LagjrkhD90VQySHONi44= 2023-10-11T15:21:56.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UL6R0A4LagjrkhD90VQySHONi44=', 'Content-Length': '0'} 2023-10-11T15:21:56.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040934c1f0a8926cb-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:21:56.231 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.232 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.232 DEBUG:boto:Method: GET 2023-10-11T15:21:56.232 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:56.233 DEBUG:boto:Data: 2023-10-11T15:21:56.233 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.234 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.235 DEBUG:boto:Params: {} 2023-10-11T15:21:56.235 DEBUG:boto:Token: None 2023-10-11T15:21:56.236 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33 2023-10-11T15:21:56.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vqYl7JqeVfgic8tJ0Fi5BSZ7zO0= 2023-10-11T15:21:56.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vqYl7JqeVfgic8tJ0Fi5BSZ7zO0=', 'Content-Length': '0'} 2023-10-11T15:21:56.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b88aa1f848ffb8e-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.286 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:21:56.287 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:21:56.287 DEBUG:boto:Method: GET 2023-10-11T15:21:56.288 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:21:56.288 DEBUG:boto:Data: 2023-10-11T15:21:56.289 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.290 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.290 DEBUG:boto:Params: {} 2023-10-11T15:21:56.291 DEBUG:boto:Token: None 2023-10-11T15:21:56.291 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj33 2023-10-11T15:21:56.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vqYl7JqeVfgic8tJ0Fi5BSZ7zO0= 2023-10-11T15:21:56.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vqYl7JqeVfgic8tJ0Fi5BSZ7zO0=', 'Content-Length': '0'} 2023-10-11T15:21:56.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c593ac41adf5218-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.295 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.296 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.296 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.296 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.297 DEBUG:boto:Method: GET 2023-10-11T15:21:56.297 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.298 DEBUG:boto:Data: 2023-10-11T15:21:56.298 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.299 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.299 DEBUG:boto:Params: {} 2023-10-11T15:21:56.300 DEBUG:boto:Token: None 2023-10-11T15:21:56.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009242f51c8acad14e-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.305 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.306 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.306 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.307 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.307 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.308 DEBUG:boto:Method: GET 2023-10-11T15:21:56.308 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:21:56.309 DEBUG:boto:Data: 2023-10-11T15:21:56.309 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.310 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.310 DEBUG:boto:Params: {} 2023-10-11T15:21:56.311 DEBUG:boto:Token: None 2023-10-11T15:21:56.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f07687a62d5495-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.316 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:56.317 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.318 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.318 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.319 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.319 DEBUG:boto:Method: GET 2023-10-11T15:21:56.319 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.320 DEBUG:boto:Data: 2023-10-11T15:21:56.320 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.321 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.321 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.322 DEBUG:boto:Params: {} 2023-10-11T15:21:56.322 DEBUG:boto:Token: None 2023-10-11T15:21:56.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA= 2023-10-11T15:21:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA=', 'Content-Length': '0'} 2023-10-11T15:21:56.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002627c2a26017d342-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.326 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.326 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.327 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.327 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.328 DEBUG:boto:Method: GET 2023-10-11T15:21:56.328 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.329 DEBUG:boto:Data: 2023-10-11T15:21:56.329 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.329 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.330 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.330 DEBUG:boto:Params: {} 2023-10-11T15:21:56.331 DEBUG:boto:Token: None 2023-10-11T15:21:56.331 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA= 2023-10-11T15:21:56.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA=', 'Content-Length': '0'} 2023-10-11T15:21:56.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000827978fc60083e37-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.374 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.375 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.375 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.376 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.376 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.377 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.377 DEBUG:boto:Data: 2023-10-11T15:21:56.377 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.378 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.379 DEBUG:boto:Params: {} 2023-10-11T15:21:56.379 DEBUG:boto:Token: None 2023-10-11T15:21:56.380 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JJIYpmwW7FNZ92qxOKyuM0+zR1s= 2023-10-11T15:21:56.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JJIYpmwW7FNZ92qxOKyuM0+zR1s=', 'Content-Length': '0'} 2023-10-11T15:21:56.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134b0d69fb8f7414-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.383 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.384 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.384 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.384 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.385 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.385 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.386 DEBUG:boto:Data: 2023-10-11T15:21:56.386 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.387 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.387 DEBUG:boto:Params: {} 2023-10-11T15:21:56.388 DEBUG:boto:Token: None 2023-10-11T15:21:56.388 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JJIYpmwW7FNZ92qxOKyuM0+zR1s= 2023-10-11T15:21:56.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JJIYpmwW7FNZ92qxOKyuM0+zR1s=', 'Content-Length': '0'} 2023-10-11T15:21:56.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000999afe3036e2b49f-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:56.392 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.392 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.394 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.394 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.395 DEBUG:boto:Method: GET 2023-10-11T15:21:56.395 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.396 DEBUG:boto:Data: 2023-10-11T15:21:56.396 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.397 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.397 DEBUG:boto:Params: {} 2023-10-11T15:21:56.398 DEBUG:boto:Token: None 2023-10-11T15:21:56.399 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA= 2023-10-11T15:21:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA=', 'Content-Length': '0'} 2023-10-11T15:21:56.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4ddbe57c09c78f-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.403 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.403 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.404 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.404 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.404 DEBUG:boto:Method: GET 2023-10-11T15:21:56.405 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.405 DEBUG:boto:Data: 2023-10-11T15:21:56.406 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.406 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.407 DEBUG:boto:Params: {} 2023-10-11T15:21:56.407 DEBUG:boto:Token: None 2023-10-11T15:21:56.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:21:56.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA= 2023-10-11T15:21:56.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dA1UJQpKSF08qw3Wpucux9eyCQA=', 'Content-Length': '0'} 2023-10-11T15:21:56.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1197bcbdbf1644-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.411 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.412 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.412 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.413 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.413 DEBUG:boto:Method: GET 2023-10-11T15:21:56.413 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.414 DEBUG:boto:Data: 2023-10-11T15:21:56.414 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.415 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.415 DEBUG:boto:Params: {} 2023-10-11T15:21:56.416 DEBUG:boto:Token: None 2023-10-11T15:21:56.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000266f1be6f765f07e-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.420 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.421 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.421 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.421 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.422 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.422 DEBUG:boto:Method: GET 2023-10-11T15:21:56.423 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.423 DEBUG:boto:Data: 2023-10-11T15:21:56.423 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.424 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.425 DEBUG:boto:Params: {} 2023-10-11T15:21:56.425 DEBUG:boto:Token: None 2023-10-11T15:21:56.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b365faf30ae7a872-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.474 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.475 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.476 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.476 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.477 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:56.477 DEBUG:boto:Data: 2023-10-11T15:21:56.477 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.478 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.479 DEBUG:boto:Params: {} 2023-10-11T15:21:56.479 DEBUG:boto:Token: None 2023-10-11T15:21:56.480 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34 2023-10-11T15:21:56.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9eLaWI8qL/wd4kjf7fQr3p5oS5g= 2023-10-11T15:21:56.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9eLaWI8qL/wd4kjf7fQr3p5oS5g=', 'Content-Length': '0'} 2023-10-11T15:21:56.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92854d0491252f2-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.483 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.484 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.484 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.485 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:56.485 DEBUG:boto:Data: 2023-10-11T15:21:56.485 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.486 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.486 DEBUG:boto:Params: {} 2023-10-11T15:21:56.487 DEBUG:boto:Token: None 2023-10-11T15:21:56.487 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34 2023-10-11T15:21:56.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9eLaWI8qL/wd4kjf7fQr3p5oS5g= 2023-10-11T15:21:56.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9eLaWI8qL/wd4kjf7fQr3p5oS5g=', 'Content-Length': '0'} 2023-10-11T15:21:56.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010065fb0e4cf8e40-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:21:56.491 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.492 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.492 DEBUG:boto:Method: GET 2023-10-11T15:21:56.493 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:56.493 DEBUG:boto:Data: 2023-10-11T15:21:56.494 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.495 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.495 DEBUG:boto:Params: {} 2023-10-11T15:21:56.496 DEBUG:boto:Token: None 2023-10-11T15:21:56.496 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34 2023-10-11T15:21:56.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R2FQkQOz2eT9RimXknaj8asiICs= 2023-10-11T15:21:56.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R2FQkQOz2eT9RimXknaj8asiICs=', 'Content-Length': '0'} 2023-10-11T15:21:56.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4862b5b36facf40-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.546 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:21:56.547 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:21:56.547 DEBUG:boto:Method: GET 2023-10-11T15:21:56.548 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:21:56.548 DEBUG:boto:Data: 2023-10-11T15:21:56.549 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.549 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.549 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.550 DEBUG:boto:Params: {} 2023-10-11T15:21:56.550 DEBUG:boto:Token: None 2023-10-11T15:21:56.551 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj34 2023-10-11T15:21:56.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R2FQkQOz2eT9RimXknaj8asiICs= 2023-10-11T15:21:56.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R2FQkQOz2eT9RimXknaj8asiICs=', 'Content-Length': '0'} 2023-10-11T15:21:56.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436c256ed46ba931-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.554 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.555 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.555 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.556 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.556 DEBUG:boto:Method: GET 2023-10-11T15:21:56.557 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.557 DEBUG:boto:Data: 2023-10-11T15:21:56.557 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.558 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.559 DEBUG:boto:Params: {} 2023-10-11T15:21:56.559 DEBUG:boto:Token: None 2023-10-11T15:21:56.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a05ff6f2771de9a4-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.564 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.564 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.565 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.565 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.566 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.566 DEBUG:boto:Method: GET 2023-10-11T15:21:56.566 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:21:56.567 DEBUG:boto:Data: 2023-10-11T15:21:56.567 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.568 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.568 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.569 DEBUG:boto:Params: {} 2023-10-11T15:21:56.569 DEBUG:boto:Token: None 2023-10-11T15:21:56.570 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.570 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b8691c2317cb58c-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.574 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:56.575 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.576 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.576 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.577 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.577 DEBUG:boto:Method: GET 2023-10-11T15:21:56.578 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.578 DEBUG:boto:Data: 2023-10-11T15:21:56.578 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.579 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.580 DEBUG:boto:Params: {} 2023-10-11T15:21:56.580 DEBUG:boto:Token: None 2023-10-11T15:21:56.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g= 2023-10-11T15:21:56.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g=', 'Content-Length': '0'} 2023-10-11T15:21:56.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077159744f8a7cb56-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.584 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.585 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.585 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.585 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.586 DEBUG:boto:Method: GET 2023-10-11T15:21:56.586 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.587 DEBUG:boto:Data: 2023-10-11T15:21:56.587 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.587 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.588 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.588 DEBUG:boto:Params: {} 2023-10-11T15:21:56.589 DEBUG:boto:Token: None 2023-10-11T15:21:56.589 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g= 2023-10-11T15:21:56.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g=', 'Content-Length': '0'} 2023-10-11T15:21:56.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed2d00fe168808d6-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.592 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.593 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.593 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.594 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.594 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.594 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.595 DEBUG:boto:Data: 2023-10-11T15:21:56.595 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.596 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.596 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.596 DEBUG:boto:Params: {} 2023-10-11T15:21:56.597 DEBUG:boto:Token: None 2023-10-11T15:21:56.597 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bFpwIUQQ7SYTREWlynq+AR8d3pY= 2023-10-11T15:21:56.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bFpwIUQQ7SYTREWlynq+AR8d3pY=', 'Content-Length': '0'} 2023-10-11T15:21:56.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda9cb45161cd6dc-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.600 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.601 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.601 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.602 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.602 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.602 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.603 DEBUG:boto:Data: 2023-10-11T15:21:56.603 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.604 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.604 DEBUG:boto:Params: {} 2023-10-11T15:21:56.605 DEBUG:boto:Token: None 2023-10-11T15:21:56.605 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bFpwIUQQ7SYTREWlynq+AR8d3pY= 2023-10-11T15:21:56.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bFpwIUQQ7SYTREWlynq+AR8d3pY=', 'Content-Length': '0'} 2023-10-11T15:21:56.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003985fbf230a8a9e6-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:56.609 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.609 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.610 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.610 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.611 DEBUG:boto:Method: GET 2023-10-11T15:21:56.611 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.611 DEBUG:boto:Data: 2023-10-11T15:21:56.612 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.613 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.613 DEBUG:boto:Params: {} 2023-10-11T15:21:56.613 DEBUG:boto:Token: None 2023-10-11T15:21:56.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g= 2023-10-11T15:21:56.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g=', 'Content-Length': '0'} 2023-10-11T15:21:56.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98086d0e426b6e1-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.617 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.618 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.618 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.619 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.619 DEBUG:boto:Method: GET 2023-10-11T15:21:56.619 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.620 DEBUG:boto:Data: 2023-10-11T15:21:56.620 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.621 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.621 DEBUG:boto:Params: {} 2023-10-11T15:21:56.622 DEBUG:boto:Token: None 2023-10-11T15:21:56.622 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:21:56.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g= 2023-10-11T15:21:56.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9OjkDksoNUe7vqIBCUInzEHoX7g=', 'Content-Length': '0'} 2023-10-11T15:21:56.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db4d771c085b3387-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.626 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.627 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.627 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.628 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.628 DEBUG:boto:Method: GET 2023-10-11T15:21:56.628 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.629 DEBUG:boto:Data: 2023-10-11T15:21:56.629 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.630 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.630 DEBUG:boto:Params: {} 2023-10-11T15:21:56.631 DEBUG:boto:Token: None 2023-10-11T15:21:56.631 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c5cb05b81118b1-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.635 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.636 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.636 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.637 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.637 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.638 DEBUG:boto:Method: GET 2023-10-11T15:21:56.638 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.638 DEBUG:boto:Data: 2023-10-11T15:21:56.639 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.639 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.640 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.640 DEBUG:boto:Params: {} 2023-10-11T15:21:56.640 DEBUG:boto:Token: None 2023-10-11T15:21:56.641 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.641 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccc450a82608ab16-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.646 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.646 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.647 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.647 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.648 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:56.648 DEBUG:boto:Data: 2023-10-11T15:21:56.648 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.649 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.650 DEBUG:boto:Params: {} 2023-10-11T15:21:56.650 DEBUG:boto:Token: None 2023-10-11T15:21:56.650 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38 2023-10-11T15:21:56.651 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:id93+ycp0f5bT+m+YT8kk47vKQ4= 2023-10-11T15:21:56.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:id93+ycp0f5bT+m+YT8kk47vKQ4=', 'Content-Length': '0'} 2023-10-11T15:21:56.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000260695515c5046b6-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.654 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.654 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.655 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.655 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:56.655 DEBUG:boto:Data: 2023-10-11T15:21:56.656 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.656 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.657 DEBUG:boto:Params: {} 2023-10-11T15:21:56.657 DEBUG:boto:Token: None 2023-10-11T15:21:56.658 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38 2023-10-11T15:21:56.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:id93+ycp0f5bT+m+YT8kk47vKQ4= 2023-10-11T15:21:56.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:id93+ycp0f5bT+m+YT8kk47vKQ4=', 'Content-Length': '0'} 2023-10-11T15:21:56.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd21527ff90f92b-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:21:56.661 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.661 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.662 DEBUG:boto:Method: GET 2023-10-11T15:21:56.662 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:56.663 DEBUG:boto:Data: 2023-10-11T15:21:56.663 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.664 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.664 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.664 DEBUG:boto:Params: {} 2023-10-11T15:21:56.665 DEBUG:boto:Token: None 2023-10-11T15:21:56.665 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38 2023-10-11T15:21:56.666 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LB91xWXDDPArWfTtQdwa7MJX40s= 2023-10-11T15:21:56.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LB91xWXDDPArWfTtQdwa7MJX40s=', 'Content-Length': '0'} 2023-10-11T15:21:56.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd911fc22f36d76c-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.710 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:21:56.711 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:21:56.711 DEBUG:boto:Method: GET 2023-10-11T15:21:56.713 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:21:56.714 DEBUG:boto:Data: 2023-10-11T15:21:56.714 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.715 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.715 DEBUG:boto:Params: {} 2023-10-11T15:21:56.716 DEBUG:boto:Token: None 2023-10-11T15:21:56.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj38 2023-10-11T15:21:56.717 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LB91xWXDDPArWfTtQdwa7MJX40s= 2023-10-11T15:21:56.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LB91xWXDDPArWfTtQdwa7MJX40s=', 'Content-Length': '0'} 2023-10-11T15:21:56.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d05f82cd1b91304-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.720 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.721 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.721 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.721 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.722 DEBUG:boto:Method: GET 2023-10-11T15:21:56.722 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.723 DEBUG:boto:Data: 2023-10-11T15:21:56.723 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.724 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.724 DEBUG:boto:Params: {} 2023-10-11T15:21:56.725 DEBUG:boto:Token: None 2023-10-11T15:21:56.725 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ed5eac9813104f-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.729 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.730 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.731 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.731 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.732 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.732 DEBUG:boto:Method: GET 2023-10-11T15:21:56.732 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:21:56.733 DEBUG:boto:Data: 2023-10-11T15:21:56.733 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.734 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.735 DEBUG:boto:Params: {} 2023-10-11T15:21:56.735 DEBUG:boto:Token: None 2023-10-11T15:21:56.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027c4042ecccdad06-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.740 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:56.741 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.742 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.742 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.743 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.743 DEBUG:boto:Method: GET 2023-10-11T15:21:56.744 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.744 DEBUG:boto:Data: 2023-10-11T15:21:56.744 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.745 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.745 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.746 DEBUG:boto:Params: {} 2023-10-11T15:21:56.746 DEBUG:boto:Token: None 2023-10-11T15:21:56.747 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow= 2023-10-11T15:21:56.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow=', 'Content-Length': '0'} 2023-10-11T15:21:56.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e320968b54786049-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.751 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.751 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.751 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.752 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.752 DEBUG:boto:Method: GET 2023-10-11T15:21:56.753 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.753 DEBUG:boto:Data: 2023-10-11T15:21:56.754 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.754 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.755 DEBUG:boto:Params: {} 2023-10-11T15:21:56.755 DEBUG:boto:Token: None 2023-10-11T15:21:56.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow= 2023-10-11T15:21:56.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow=', 'Content-Length': '0'} 2023-10-11T15:21:56.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bc523706833706-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.806 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.807 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.807 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.808 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.808 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.809 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.809 DEBUG:boto:Data: 2023-10-11T15:21:56.809 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.810 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.811 DEBUG:boto:Params: {} 2023-10-11T15:21:56.811 DEBUG:boto:Token: None 2023-10-11T15:21:56.812 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mXX6CFz9xqOO5T4nYLNbTRKZu/w= 2023-10-11T15:21:56.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mXX6CFz9xqOO5T4nYLNbTRKZu/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9da849a6b408a3-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.815 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.816 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.816 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.816 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.817 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.817 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.818 DEBUG:boto:Data: 2023-10-11T15:21:56.818 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.819 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.819 DEBUG:boto:Params: {} 2023-10-11T15:21:56.820 DEBUG:boto:Token: None 2023-10-11T15:21:56.820 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mXX6CFz9xqOO5T4nYLNbTRKZu/w= 2023-10-11T15:21:56.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mXX6CFz9xqOO5T4nYLNbTRKZu/w=', 'Content-Length': '0'} 2023-10-11T15:21:56.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe9f9bb5c6d4fbec-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:56.824 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.824 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.825 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.825 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.826 DEBUG:boto:Method: GET 2023-10-11T15:21:56.826 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.827 DEBUG:boto:Data: 2023-10-11T15:21:56.827 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.828 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.828 DEBUG:boto:Params: {} 2023-10-11T15:21:56.828 DEBUG:boto:Token: None 2023-10-11T15:21:56.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow= 2023-10-11T15:21:56.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow=', 'Content-Length': '0'} 2023-10-11T15:21:56.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009948a03e7900b656-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.833 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.833 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.834 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.834 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.834 DEBUG:boto:Method: GET 2023-10-11T15:21:56.835 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.835 DEBUG:boto:Data: 2023-10-11T15:21:56.836 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.837 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.837 DEBUG:boto:Params: {} 2023-10-11T15:21:56.837 DEBUG:boto:Token: None 2023-10-11T15:21:56.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:21:56.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow= 2023-10-11T15:21:56.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bsqKsBcb7K6Da8QFrysL1aQZ+Ow=', 'Content-Length': '0'} 2023-10-11T15:21:56.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a233eefb76674d7-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.842 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.842 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.843 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.843 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.844 DEBUG:boto:Method: GET 2023-10-11T15:21:56.844 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.845 DEBUG:boto:Data: 2023-10-11T15:21:56.845 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.846 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.846 DEBUG:boto:Params: {} 2023-10-11T15:21:56.847 DEBUG:boto:Token: None 2023-10-11T15:21:56.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5c5ef0b348d7800-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.851 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.852 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.853 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.853 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.853 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.854 DEBUG:boto:Method: GET 2023-10-11T15:21:56.854 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.855 DEBUG:boto:Data: 2023-10-11T15:21:56.855 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.856 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.856 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.856 DEBUG:boto:Params: {} 2023-10-11T15:21:56.857 DEBUG:boto:Token: None 2023-10-11T15:21:56.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca22be5a357673d7-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.862 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.863 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.863 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.864 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.864 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:56.865 DEBUG:boto:Data: 2023-10-11T15:21:56.865 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.866 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.866 DEBUG:boto:Params: {} 2023-10-11T15:21:56.867 DEBUG:boto:Token: None 2023-10-11T15:21:56.867 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44 2023-10-11T15:21:56.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1O1QPyiN//BGskHRRSdjZhxcLI= 2023-10-11T15:21:56.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1O1QPyiN//BGskHRRSdjZhxcLI=', 'Content-Length': '0'} 2023-10-11T15:21:56.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e18c1205dfbd56a-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.871 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.871 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.872 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.872 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:56.872 DEBUG:boto:Data: 2023-10-11T15:21:56.873 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.874 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.874 DEBUG:boto:Params: {} 2023-10-11T15:21:56.875 DEBUG:boto:Token: None 2023-10-11T15:21:56.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44 2023-10-11T15:21:56.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1O1QPyiN//BGskHRRSdjZhxcLI= 2023-10-11T15:21:56.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1O1QPyiN//BGskHRRSdjZhxcLI=', 'Content-Length': '0'} 2023-10-11T15:21:56.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008658fe075fb9585e-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:21:56.879 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.879 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.880 DEBUG:boto:Method: GET 2023-10-11T15:21:56.880 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:56.881 DEBUG:boto:Data: 2023-10-11T15:21:56.881 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.882 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.882 DEBUG:boto:Params: {} 2023-10-11T15:21:56.883 DEBUG:boto:Token: None 2023-10-11T15:21:56.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44 2023-10-11T15:21:56.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h4vggn2bJM5u+eNCql6ruP981ks= 2023-10-11T15:21:56.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h4vggn2bJM5u+eNCql6ruP981ks=', 'Content-Length': '0'} 2023-10-11T15:21:56.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000febdb8e4b87b4ac-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.934 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:21:56.935 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:21:56.935 DEBUG:boto:Method: GET 2023-10-11T15:21:56.936 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:21:56.936 DEBUG:boto:Data: 2023-10-11T15:21:56.937 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.937 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.938 DEBUG:boto:Params: {} 2023-10-11T15:21:56.938 DEBUG:boto:Token: None 2023-10-11T15:21:56.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj44 2023-10-11T15:21:56.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h4vggn2bJM5u+eNCql6ruP981ks= 2023-10-11T15:21:56.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h4vggn2bJM5u+eNCql6ruP981ks=', 'Content-Length': '0'} 2023-10-11T15:21:56.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080675cb70254c126-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.942 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.943 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.943 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.944 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.944 DEBUG:boto:Method: GET 2023-10-11T15:21:56.945 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.945 DEBUG:boto:Data: 2023-10-11T15:21:56.945 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.946 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.947 DEBUG:boto:Params: {} 2023-10-11T15:21:56.947 DEBUG:boto:Token: None 2023-10-11T15:21:56.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed433752fac13c7f-006526bd94-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.951 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.952 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:56.952 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:56.953 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.953 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.954 DEBUG:boto:Method: GET 2023-10-11T15:21:56.954 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:21:56.955 DEBUG:boto:Data: 2023-10-11T15:21:56.955 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.956 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.956 DEBUG:boto:Params: {} 2023-10-11T15:21:56.957 DEBUG:boto:Token: None 2023-10-11T15:21:56.957 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/ 2023-10-11T15:21:56.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE= 2023-10-11T15:21:56.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrUJjSRD6vJ7SBTQhQoLkyllvsE=', 'Content-Length': '0'} 2023-10-11T15:21:56.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778e7c3b78c41e93-006526bd94-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.962 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:56.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:21:56.963 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:56.964 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:56.964 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.964 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.965 DEBUG:boto:Method: GET 2023-10-11T15:21:56.965 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.966 DEBUG:boto:Data: 2023-10-11T15:21:56.966 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.967 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.967 DEBUG:boto:Params: {} 2023-10-11T15:21:56.968 DEBUG:boto:Token: None 2023-10-11T15:21:56.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Po7BgNXMIfC/DYvpemP07MKu9GE= 2023-10-11T15:21:56.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Po7BgNXMIfC/DYvpemP07MKu9GE=', 'Content-Length': '0'} 2023-10-11T15:21:56.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ceb035e7ebff9b-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.972 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:56.972 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:56.973 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.973 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.974 DEBUG:boto:Method: GET 2023-10-11T15:21:56.974 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.975 DEBUG:boto:Data: 2023-10-11T15:21:56.975 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.976 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.976 DEBUG:boto:Params: {} 2023-10-11T15:21:56.977 DEBUG:boto:Token: None 2023-10-11T15:21:56.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.978 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Po7BgNXMIfC/DYvpemP07MKu9GE= 2023-10-11T15:21:56.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Po7BgNXMIfC/DYvpemP07MKu9GE=', 'Content-Length': '0'} 2023-10-11T15:21:56.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720acc19fd920abc-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.981 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:56.982 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:56.982 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.982 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.983 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.983 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.984 DEBUG:boto:Data: 2023-10-11T15:21:56.984 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:56.985 DEBUG:boto:Port: 8001 2023-10-11T15:21:56.985 DEBUG:boto:Params: {} 2023-10-11T15:21:56.986 DEBUG:boto:Token: None 2023-10-11T15:21:56.986 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0s0fLpeCcMSTpFwvASCml3oCLl0= 2023-10-11T15:21:56.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0s0fLpeCcMSTpFwvASCml3oCLl0=', 'Content-Length': '0'} 2023-10-11T15:21:56.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa176bc9a6f00067-006526bd94-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.990 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:56.991 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:56.991 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.991 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.992 DEBUG:boto:Method: HEAD 2023-10-11T15:21:56.992 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.992 DEBUG:boto:Data: 2023-10-11T15:21:56.993 DEBUG:boto:Headers: {} 2023-10-11T15:21:56.993 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:56.994 DEBUG:boto:Port: 8000 2023-10-11T15:21:56.994 DEBUG:boto:Params: {} 2023-10-11T15:21:56.995 DEBUG:boto:Token: None 2023-10-11T15:21:56.995 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:56 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:56.996 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0s0fLpeCcMSTpFwvASCml3oCLl0= 2023-10-11T15:21:56.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0s0fLpeCcMSTpFwvASCml3oCLl0=', 'Content-Length': '0'} 2023-10-11T15:21:56.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d834063f1f67eb-006526bd94-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:56.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:21:56.999 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:56.999 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:56.999 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.000 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.000 DEBUG:boto:Method: GET 2023-10-11T15:21:57.001 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.001 DEBUG:boto:Data: 2023-10-11T15:21:57.001 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.002 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.003 DEBUG:boto:Params: {} 2023-10-11T15:21:57.003 DEBUG:boto:Token: None 2023-10-11T15:21:57.003 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1UcJBTgA2yuLZVr2d2KZTSOtlTY= 2023-10-11T15:21:57.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1UcJBTgA2yuLZVr2d2KZTSOtlTY=', 'Content-Length': '0'} 2023-10-11T15:21:57.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0146870b78d7b7-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.050 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:57.051 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:57.051 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.051 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.052 DEBUG:boto:Method: GET 2023-10-11T15:21:57.052 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.053 DEBUG:boto:Data: 2023-10-11T15:21:57.053 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.053 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.054 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.054 DEBUG:boto:Params: {} 2023-10-11T15:21:57.055 DEBUG:boto:Token: None 2023-10-11T15:21:57.055 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:21:57.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1UcJBTgA2yuLZVr2d2KZTSOtlTY= 2023-10-11T15:21:57.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1UcJBTgA2yuLZVr2d2KZTSOtlTY=', 'Content-Length': '0'} 2023-10-11T15:21:57.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f125b3abffea61c-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.059 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.059 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.059 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.060 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.060 DEBUG:boto:Method: GET 2023-10-11T15:21:57.061 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.061 DEBUG:boto:Data: 2023-10-11T15:21:57.061 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.062 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.062 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.063 DEBUG:boto:Params: {} 2023-10-11T15:21:57.063 DEBUG:boto:Token: None 2023-10-11T15:21:57.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000038fa7538a286b2-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.067 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.068 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.068 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.069 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.069 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.070 DEBUG:boto:Method: GET 2023-10-11T15:21:57.070 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.070 DEBUG:boto:Data: 2023-10-11T15:21:57.071 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.071 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.072 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.072 DEBUG:boto:Params: {} 2023-10-11T15:21:57.072 DEBUG:boto:Token: None 2023-10-11T15:21:57.073 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.073 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c432b2c6a83952f-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.077 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.078 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:57.079 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:57.079 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.079 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:21:57.080 DEBUG:boto:Data: 2023-10-11T15:21:57.080 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.081 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.081 DEBUG:boto:Params: {} 2023-10-11T15:21:57.082 DEBUG:boto:Token: None 2023-10-11T15:21:57.082 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46 2023-10-11T15:21:57.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uD9shIP1rzrl9jdDNf2bvRQ/EuM= 2023-10-11T15:21:57.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uD9shIP1rzrl9jdDNf2bvRQ/EuM=', 'Content-Length': '0'} 2023-10-11T15:21:57.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036af9fc3ad86360c-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.086 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:57.086 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:57.087 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.087 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:21:57.087 DEBUG:boto:Data: 2023-10-11T15:21:57.088 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.088 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.088 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.089 DEBUG:boto:Params: {} 2023-10-11T15:21:57.089 DEBUG:boto:Token: None 2023-10-11T15:21:57.090 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46 2023-10-11T15:21:57.090 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uD9shIP1rzrl9jdDNf2bvRQ/EuM= 2023-10-11T15:21:57.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uD9shIP1rzrl9jdDNf2bvRQ/EuM=', 'Content-Length': '0'} 2023-10-11T15:21:57.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5bacbeb7e5b2bc2-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:21:57.094 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:57.094 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:57.094 DEBUG:boto:Method: GET 2023-10-11T15:21:57.095 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:21:57.095 DEBUG:boto:Data: 2023-10-11T15:21:57.096 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.096 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.097 DEBUG:boto:Params: {} 2023-10-11T15:21:57.097 DEBUG:boto:Token: None 2023-10-11T15:21:57.098 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46 2023-10-11T15:21:57.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VWzRGtM83FRcBdJCqFzTsCfPMWM= 2023-10-11T15:21:57.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VWzRGtM83FRcBdJCqFzTsCfPMWM=', 'Content-Length': '0'} 2023-10-11T15:21:57.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015cd582ae179d571-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.101 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:21:57.102 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:21:57.102 DEBUG:boto:Method: GET 2023-10-11T15:21:57.103 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:21:57.103 DEBUG:boto:Data: 2023-10-11T15:21:57.103 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.104 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.104 DEBUG:boto:Params: {} 2023-10-11T15:21:57.105 DEBUG:boto:Token: None 2023-10-11T15:21:57.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj46 2023-10-11T15:21:57.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VWzRGtM83FRcBdJCqFzTsCfPMWM= 2023-10-11T15:21:57.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VWzRGtM83FRcBdJCqFzTsCfPMWM=', 'Content-Length': '0'} 2023-10-11T15:21:57.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14688cb1666d2d4-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.109 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.110 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.110 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.110 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.111 DEBUG:boto:Method: GET 2023-10-11T15:21:57.111 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.112 DEBUG:boto:Data: 2023-10-11T15:21:57.112 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.113 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.113 DEBUG:boto:Params: {} 2023-10-11T15:21:57.113 DEBUG:boto:Token: None 2023-10-11T15:21:57.114 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029d44e223b5aaa07-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.118 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.118 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.119 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.119 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.120 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.120 DEBUG:boto:Method: GET 2023-10-11T15:21:57.120 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:21:57.121 DEBUG:boto:Data: 2023-10-11T15:21:57.121 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.122 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.122 DEBUG:boto:Params: {} 2023-10-11T15:21:57.123 DEBUG:boto:Token: None 2023-10-11T15:21:57.123 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d9e507412f045cd-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.128 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:57.129 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.130 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.130 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.130 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.131 DEBUG:boto:Method: GET 2023-10-11T15:21:57.131 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.132 DEBUG:boto:Data: 2023-10-11T15:21:57.133 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.134 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.134 DEBUG:boto:Params: {} 2023-10-11T15:21:57.135 DEBUG:boto:Token: None 2023-10-11T15:21:57.135 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms= 2023-10-11T15:21:57.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms=', 'Content-Length': '0'} 2023-10-11T15:21:57.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003296cadb4965ad23-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.139 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.139 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.139 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.140 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.140 DEBUG:boto:Method: GET 2023-10-11T15:21:57.140 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.141 DEBUG:boto:Data: 2023-10-11T15:21:57.141 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.142 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.143 DEBUG:boto:Params: {} 2023-10-11T15:21:57.143 DEBUG:boto:Token: None 2023-10-11T15:21:57.144 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms= 2023-10-11T15:21:57.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms=', 'Content-Length': '0'} 2023-10-11T15:21:57.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e0c07ecb7b686f-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.194 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.195 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.195 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.196 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.196 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.197 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.197 DEBUG:boto:Data: 2023-10-11T15:21:57.197 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.198 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.199 DEBUG:boto:Params: {} 2023-10-11T15:21:57.199 DEBUG:boto:Token: None 2023-10-11T15:21:57.200 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.200 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aidVQChmcryHS2mOYdOPpStywJI= 2023-10-11T15:21:57.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aidVQChmcryHS2mOYdOPpStywJI=', 'Content-Length': '0'} 2023-10-11T15:21:57.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d2c1db881fa392-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.203 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.204 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.204 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.204 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.205 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.205 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.205 DEBUG:boto:Data: 2023-10-11T15:21:57.206 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.206 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.207 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.207 DEBUG:boto:Params: {} 2023-10-11T15:21:57.207 DEBUG:boto:Token: None 2023-10-11T15:21:57.208 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aidVQChmcryHS2mOYdOPpStywJI= 2023-10-11T15:21:57.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aidVQChmcryHS2mOYdOPpStywJI=', 'Content-Length': '0'} 2023-10-11T15:21:57.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49a866a07768334-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:57.212 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.212 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.213 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.213 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.213 DEBUG:boto:Method: GET 2023-10-11T15:21:57.214 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.214 DEBUG:boto:Data: 2023-10-11T15:21:57.215 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.215 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.216 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.216 DEBUG:boto:Params: {} 2023-10-11T15:21:57.217 DEBUG:boto:Token: None 2023-10-11T15:21:57.217 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms= 2023-10-11T15:21:57.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms=', 'Content-Length': '0'} 2023-10-11T15:21:57.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099cba0eaa8372e3a-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.266 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.267 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.268 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.268 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.269 DEBUG:boto:Method: GET 2023-10-11T15:21:57.269 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.270 DEBUG:boto:Data: 2023-10-11T15:21:57.270 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.271 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.271 DEBUG:boto:Params: {} 2023-10-11T15:21:57.272 DEBUG:boto:Token: None 2023-10-11T15:21:57.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:21:57.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms= 2023-10-11T15:21:57.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zlrLH1fNftk8NWgtByZ/rft2kms=', 'Content-Length': '0'} 2023-10-11T15:21:57.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b9084458b77832a-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.278 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.279 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.279 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.279 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.280 DEBUG:boto:Method: GET 2023-10-11T15:21:57.280 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.281 DEBUG:boto:Data: 2023-10-11T15:21:57.281 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.282 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.282 DEBUG:boto:Params: {} 2023-10-11T15:21:57.283 DEBUG:boto:Token: None 2023-10-11T15:21:57.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a67481a9bc09ef6c-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.288 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.288 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.289 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.289 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.290 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.290 DEBUG:boto:Method: GET 2023-10-11T15:21:57.291 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.291 DEBUG:boto:Data: 2023-10-11T15:21:57.291 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.292 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.293 DEBUG:boto:Params: {} 2023-10-11T15:21:57.293 DEBUG:boto:Token: None 2023-10-11T15:21:57.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61762731a271ccd-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.299 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.300 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.300 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.300 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.301 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:57.301 DEBUG:boto:Data: 2023-10-11T15:21:57.302 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.303 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.303 DEBUG:boto:Params: {} 2023-10-11T15:21:57.303 DEBUG:boto:Token: None 2023-10-11T15:21:57.304 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51 2023-10-11T15:21:57.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJNTHO0zCeOtgH8SZci7g77FheM= 2023-10-11T15:21:57.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJNTHO0zCeOtgH8SZci7g77FheM=', 'Content-Length': '0'} 2023-10-11T15:21:57.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97b0c1d34c7b689-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.307 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.308 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.308 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.309 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:57.309 DEBUG:boto:Data: 2023-10-11T15:21:57.310 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.311 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.311 DEBUG:boto:Params: {} 2023-10-11T15:21:57.312 DEBUG:boto:Token: None 2023-10-11T15:21:57.312 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51 2023-10-11T15:21:57.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lJNTHO0zCeOtgH8SZci7g77FheM= 2023-10-11T15:21:57.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJNTHO0zCeOtgH8SZci7g77FheM=', 'Content-Length': '0'} 2023-10-11T15:21:57.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66e80d0e7df8b71-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:21:57.317 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.317 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.318 DEBUG:boto:Method: GET 2023-10-11T15:21:57.318 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:57.319 DEBUG:boto:Data: 2023-10-11T15:21:57.319 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.320 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.321 DEBUG:boto:Params: {} 2023-10-11T15:21:57.321 DEBUG:boto:Token: None 2023-10-11T15:21:57.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51 2023-10-11T15:21:57.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b9tqUYjgDk5ARhNbjAypEYq5tc4= 2023-10-11T15:21:57.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b9tqUYjgDk5ARhNbjAypEYq5tc4=', 'Content-Length': '0'} 2023-10-11T15:21:57.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009872b11ec1cc77e0-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.370 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:21:57.371 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:21:57.371 DEBUG:boto:Method: GET 2023-10-11T15:21:57.372 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:21:57.372 DEBUG:boto:Data: 2023-10-11T15:21:57.373 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.373 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.374 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.374 DEBUG:boto:Params: {} 2023-10-11T15:21:57.375 DEBUG:boto:Token: None 2023-10-11T15:21:57.375 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj51 2023-10-11T15:21:57.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b9tqUYjgDk5ARhNbjAypEYq5tc4= 2023-10-11T15:21:57.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b9tqUYjgDk5ARhNbjAypEYq5tc4=', 'Content-Length': '0'} 2023-10-11T15:21:57.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370912a68d2b5360-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.379 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.379 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.380 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.380 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.381 DEBUG:boto:Method: GET 2023-10-11T15:21:57.381 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.381 DEBUG:boto:Data: 2023-10-11T15:21:57.382 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.383 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.383 DEBUG:boto:Params: {} 2023-10-11T15:21:57.384 DEBUG:boto:Token: None 2023-10-11T15:21:57.384 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000e96e03b6c5da18-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.388 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.389 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.389 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.390 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.390 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.391 DEBUG:boto:Method: GET 2023-10-11T15:21:57.391 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:21:57.392 DEBUG:boto:Data: 2023-10-11T15:21:57.392 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.393 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.393 DEBUG:boto:Params: {} 2023-10-11T15:21:57.394 DEBUG:boto:Token: None 2023-10-11T15:21:57.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000685007bb9b18e1f0-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.442 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:57.443 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.444 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.444 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.444 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.445 DEBUG:boto:Method: GET 2023-10-11T15:21:57.445 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.446 DEBUG:boto:Data: 2023-10-11T15:21:57.446 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.447 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.447 DEBUG:boto:Params: {} 2023-10-11T15:21:57.448 DEBUG:boto:Token: None 2023-10-11T15:21:57.448 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto= 2023-10-11T15:21:57.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto=', 'Content-Length': '0'} 2023-10-11T15:21:57.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd5767c0355ef2ac-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.452 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.452 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.453 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.453 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.454 DEBUG:boto:Method: GET 2023-10-11T15:21:57.454 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.454 DEBUG:boto:Data: 2023-10-11T15:21:57.455 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.456 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.456 DEBUG:boto:Params: {} 2023-10-11T15:21:57.457 DEBUG:boto:Token: None 2023-10-11T15:21:57.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto= 2023-10-11T15:21:57.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto=', 'Content-Length': '0'} 2023-10-11T15:21:57.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59f26e230ea9095-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.461 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.461 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.462 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.462 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.462 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.463 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.463 DEBUG:boto:Data: 2023-10-11T15:21:57.464 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.464 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.465 DEBUG:boto:Params: {} 2023-10-11T15:21:57.465 DEBUG:boto:Token: None 2023-10-11T15:21:57.466 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.466 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDSsLfwZ+1+vWzRntXkBJ4ngP0c= 2023-10-11T15:21:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDSsLfwZ+1+vWzRntXkBJ4ngP0c=', 'Content-Length': '0'} 2023-10-11T15:21:57.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b896106a273c13-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.469 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.470 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.470 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.471 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.471 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.471 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.472 DEBUG:boto:Data: 2023-10-11T15:21:57.472 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.473 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.473 DEBUG:boto:Params: {} 2023-10-11T15:21:57.474 DEBUG:boto:Token: None 2023-10-11T15:21:57.474 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.475 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WDSsLfwZ+1+vWzRntXkBJ4ngP0c= 2023-10-11T15:21:57.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WDSsLfwZ+1+vWzRntXkBJ4ngP0c=', 'Content-Length': '0'} 2023-10-11T15:21:57.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700b9b3bd3cc4e84-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:57.478 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.479 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.479 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.480 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.480 DEBUG:boto:Method: GET 2023-10-11T15:21:57.480 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.481 DEBUG:boto:Data: 2023-10-11T15:21:57.481 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.482 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.483 DEBUG:boto:Params: {} 2023-10-11T15:21:57.483 DEBUG:boto:Token: None 2023-10-11T15:21:57.484 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto= 2023-10-11T15:21:57.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto=', 'Content-Length': '0'} 2023-10-11T15:21:57.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a34bc672fa8dff4-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.487 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.487 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.488 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.488 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.489 DEBUG:boto:Method: GET 2023-10-11T15:21:57.489 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.490 DEBUG:boto:Data: 2023-10-11T15:21:57.490 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.491 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.492 DEBUG:boto:Params: {} 2023-10-11T15:21:57.492 DEBUG:boto:Token: None 2023-10-11T15:21:57.493 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:21:57.493 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto= 2023-10-11T15:21:57.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z7RNt0qfeBfhUqgj/1hg9pS2tto=', 'Content-Length': '0'} 2023-10-11T15:21:57.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a236876ccc68b7-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.496 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.497 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.497 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.498 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.498 DEBUG:boto:Method: GET 2023-10-11T15:21:57.498 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.499 DEBUG:boto:Data: 2023-10-11T15:21:57.499 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.500 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.501 DEBUG:boto:Params: {} 2023-10-11T15:21:57.501 DEBUG:boto:Token: None 2023-10-11T15:21:57.502 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931d8d6825e97a81-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.550 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.551 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.552 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.552 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.553 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.553 DEBUG:boto:Method: GET 2023-10-11T15:21:57.554 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.555 DEBUG:boto:Data: 2023-10-11T15:21:57.555 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.556 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.556 DEBUG:boto:Params: {} 2023-10-11T15:21:57.557 DEBUG:boto:Token: None 2023-10-11T15:21:57.557 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e28e1a83b4a8af08-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.563 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.564 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.564 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.565 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.565 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:57.565 DEBUG:boto:Data: 2023-10-11T15:21:57.566 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.567 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.567 DEBUG:boto:Params: {} 2023-10-11T15:21:57.567 DEBUG:boto:Token: None 2023-10-11T15:21:57.568 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55 2023-10-11T15:21:57.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JsS1//yG4Lwnhri7217hM3OL4M0= 2023-10-11T15:21:57.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsS1//yG4Lwnhri7217hM3OL4M0=', 'Content-Length': '0'} 2023-10-11T15:21:57.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3b845973eebe03-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.572 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.572 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.573 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.574 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:57.574 DEBUG:boto:Data: 2023-10-11T15:21:57.574 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.575 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.576 DEBUG:boto:Params: {} 2023-10-11T15:21:57.576 DEBUG:boto:Token: None 2023-10-11T15:21:57.577 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55 2023-10-11T15:21:57.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JsS1//yG4Lwnhri7217hM3OL4M0= 2023-10-11T15:21:57.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsS1//yG4Lwnhri7217hM3OL4M0=', 'Content-Length': '0'} 2023-10-11T15:21:57.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c238cbb934ca9ec-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:21:57.581 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.581 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.582 DEBUG:boto:Method: GET 2023-10-11T15:21:57.582 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:57.582 DEBUG:boto:Data: 2023-10-11T15:21:57.583 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.584 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.584 DEBUG:boto:Params: {} 2023-10-11T15:21:57.585 DEBUG:boto:Token: None 2023-10-11T15:21:57.585 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55 2023-10-11T15:21:57.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qNhyDxyFpb/fWVHBCmnBRUBpyXs= 2023-10-11T15:21:57.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qNhyDxyFpb/fWVHBCmnBRUBpyXs=', 'Content-Length': '0'} 2023-10-11T15:21:57.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65ad42890d5cba0-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.589 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:21:57.589 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:21:57.590 DEBUG:boto:Method: GET 2023-10-11T15:21:57.590 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:21:57.591 DEBUG:boto:Data: 2023-10-11T15:21:57.591 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.592 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.593 DEBUG:boto:Params: {} 2023-10-11T15:21:57.593 DEBUG:boto:Token: None 2023-10-11T15:21:57.594 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj55 2023-10-11T15:21:57.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qNhyDxyFpb/fWVHBCmnBRUBpyXs= 2023-10-11T15:21:57.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qNhyDxyFpb/fWVHBCmnBRUBpyXs=', 'Content-Length': '0'} 2023-10-11T15:21:57.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408cb654a9fb3abb-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.597 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.598 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.598 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.599 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.599 DEBUG:boto:Method: GET 2023-10-11T15:21:57.600 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.600 DEBUG:boto:Data: 2023-10-11T15:21:57.600 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.602 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.602 DEBUG:boto:Params: {} 2023-10-11T15:21:57.603 DEBUG:boto:Token: None 2023-10-11T15:21:57.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007565d6f58c718420-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.607 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.608 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.608 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.609 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.609 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.610 DEBUG:boto:Method: GET 2023-10-11T15:21:57.610 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:21:57.610 DEBUG:boto:Data: 2023-10-11T15:21:57.611 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.612 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.612 DEBUG:boto:Params: {} 2023-10-11T15:21:57.613 DEBUG:boto:Token: None 2023-10-11T15:21:57.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014436dd0dcb007d9-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.618 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:57.619 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.619 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.620 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.620 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.621 DEBUG:boto:Method: GET 2023-10-11T15:21:57.621 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.621 DEBUG:boto:Data: 2023-10-11T15:21:57.622 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.623 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.623 DEBUG:boto:Params: {} 2023-10-11T15:21:57.624 DEBUG:boto:Token: None 2023-10-11T15:21:57.624 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.625 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4= 2023-10-11T15:21:57.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4=', 'Content-Length': '0'} 2023-10-11T15:21:57.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec54783c09f539e4-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.674 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.675 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.675 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.676 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.676 DEBUG:boto:Method: GET 2023-10-11T15:21:57.677 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.677 DEBUG:boto:Data: 2023-10-11T15:21:57.678 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.679 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.679 DEBUG:boto:Params: {} 2023-10-11T15:21:57.679 DEBUG:boto:Token: None 2023-10-11T15:21:57.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4= 2023-10-11T15:21:57.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4=', 'Content-Length': '0'} 2023-10-11T15:21:57.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da8f060c5d8e82d-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.684 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.685 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.685 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.685 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.686 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.686 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.687 DEBUG:boto:Data: 2023-10-11T15:21:57.687 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.688 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.688 DEBUG:boto:Params: {} 2023-10-11T15:21:57.689 DEBUG:boto:Token: None 2023-10-11T15:21:57.689 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0NmiLwl3jXvOefuqIWzH67hBk1Q= 2023-10-11T15:21:57.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0NmiLwl3jXvOefuqIWzH67hBk1Q=', 'Content-Length': '0'} 2023-10-11T15:21:57.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa40fc4697e5145b-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.693 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.693 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.694 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.694 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.694 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.695 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.695 DEBUG:boto:Data: 2023-10-11T15:21:57.696 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.696 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.697 DEBUG:boto:Params: {} 2023-10-11T15:21:57.697 DEBUG:boto:Token: None 2023-10-11T15:21:57.698 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0NmiLwl3jXvOefuqIWzH67hBk1Q= 2023-10-11T15:21:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0NmiLwl3jXvOefuqIWzH67hBk1Q=', 'Content-Length': '0'} 2023-10-11T15:21:57.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff813eb2fe7a704-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:57.702 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.702 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.703 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.703 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.703 DEBUG:boto:Method: GET 2023-10-11T15:21:57.704 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.704 DEBUG:boto:Data: 2023-10-11T15:21:57.705 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.705 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.706 DEBUG:boto:Params: {} 2023-10-11T15:21:57.706 DEBUG:boto:Token: None 2023-10-11T15:21:57.707 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4= 2023-10-11T15:21:57.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4=', 'Content-Length': '0'} 2023-10-11T15:21:57.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e6d850376d24d0-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.711 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.711 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.712 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.712 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.712 DEBUG:boto:Method: GET 2023-10-11T15:21:57.713 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.713 DEBUG:boto:Data: 2023-10-11T15:21:57.714 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.715 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.715 DEBUG:boto:Params: {} 2023-10-11T15:21:57.716 DEBUG:boto:Token: None 2023-10-11T15:21:57.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:21:57.717 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4= 2023-10-11T15:21:57.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POM8OL+z3/RpwZke4K6zMNbVCf4=', 'Content-Length': '0'} 2023-10-11T15:21:57.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfd47defd005008-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.720 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.720 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.721 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.721 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.722 DEBUG:boto:Method: GET 2023-10-11T15:21:57.722 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.723 DEBUG:boto:Data: 2023-10-11T15:21:57.723 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.724 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.724 DEBUG:boto:Params: {} 2023-10-11T15:21:57.725 DEBUG:boto:Token: None 2023-10-11T15:21:57.725 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351189d9d09d20bf-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.730 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.730 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.731 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.731 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.732 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.732 DEBUG:boto:Method: GET 2023-10-11T15:21:57.733 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.733 DEBUG:boto:Data: 2023-10-11T15:21:57.733 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.734 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.735 DEBUG:boto:Params: {} 2023-10-11T15:21:57.735 DEBUG:boto:Token: None 2023-10-11T15:21:57.736 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089db972e37d8963e-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.786 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.787 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.788 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.788 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.789 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:57.789 DEBUG:boto:Data: 2023-10-11T15:21:57.789 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.790 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.791 DEBUG:boto:Params: {} 2023-10-11T15:21:57.791 DEBUG:boto:Token: None 2023-10-11T15:21:57.792 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57 2023-10-11T15:21:57.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AKya9jX+HR6LjOGbIUjF+RnPPp0= 2023-10-11T15:21:57.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AKya9jX+HR6LjOGbIUjF+RnPPp0=', 'Content-Length': '0'} 2023-10-11T15:21:57.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006046334fc091424-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.795 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.796 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.796 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.796 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:57.797 DEBUG:boto:Data: 2023-10-11T15:21:57.797 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.798 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.799 DEBUG:boto:Params: {} 2023-10-11T15:21:57.799 DEBUG:boto:Token: None 2023-10-11T15:21:57.799 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57 2023-10-11T15:21:57.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AKya9jX+HR6LjOGbIUjF+RnPPp0= 2023-10-11T15:21:57.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AKya9jX+HR6LjOGbIUjF+RnPPp0=', 'Content-Length': '0'} 2023-10-11T15:21:57.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffdab7b3e13e0e9b-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:21:57.803 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.804 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.804 DEBUG:boto:Method: GET 2023-10-11T15:21:57.805 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:57.805 DEBUG:boto:Data: 2023-10-11T15:21:57.806 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.806 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.807 DEBUG:boto:Params: {} 2023-10-11T15:21:57.807 DEBUG:boto:Token: None 2023-10-11T15:21:57.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57 2023-10-11T15:21:57.808 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:84Ypa8oFdO/mFJq24DrKTeYI/BI= 2023-10-11T15:21:57.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:84Ypa8oFdO/mFJq24DrKTeYI/BI=', 'Content-Length': '0'} 2023-10-11T15:21:57.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b029522910ff7cbc-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.811 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:21:57.812 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:21:57.812 DEBUG:boto:Method: GET 2023-10-11T15:21:57.813 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:21:57.813 DEBUG:boto:Data: 2023-10-11T15:21:57.813 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.814 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.815 DEBUG:boto:Params: {} 2023-10-11T15:21:57.815 DEBUG:boto:Token: None 2023-10-11T15:21:57.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj57 2023-10-11T15:21:57.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:84Ypa8oFdO/mFJq24DrKTeYI/BI= 2023-10-11T15:21:57.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:84Ypa8oFdO/mFJq24DrKTeYI/BI=', 'Content-Length': '0'} 2023-10-11T15:21:57.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefdfb730d1a4f85-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.819 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.820 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.820 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.821 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.821 DEBUG:boto:Method: GET 2023-10-11T15:21:57.821 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.822 DEBUG:boto:Data: 2023-10-11T15:21:57.822 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.823 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.824 DEBUG:boto:Params: {} 2023-10-11T15:21:57.824 DEBUG:boto:Token: None 2023-10-11T15:21:57.825 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840683aa7f44a1c0-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.870 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.871 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.872 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.872 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.872 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.873 DEBUG:boto:Method: GET 2023-10-11T15:21:57.873 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:21:57.874 DEBUG:boto:Data: 2023-10-11T15:21:57.874 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.875 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.875 DEBUG:boto:Params: {} 2023-10-11T15:21:57.876 DEBUG:boto:Token: None 2023-10-11T15:21:57.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d10bc2d37295c1-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.881 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:57.882 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.883 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.883 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.883 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.884 DEBUG:boto:Method: GET 2023-10-11T15:21:57.884 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.885 DEBUG:boto:Data: 2023-10-11T15:21:57.885 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.886 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.886 DEBUG:boto:Params: {} 2023-10-11T15:21:57.887 DEBUG:boto:Token: None 2023-10-11T15:21:57.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY= 2023-10-11T15:21:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY=', 'Content-Length': '0'} 2023-10-11T15:21:57.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f839ebc5646a98d-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.891 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.891 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.892 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.892 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.892 DEBUG:boto:Method: GET 2023-10-11T15:21:57.893 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.893 DEBUG:boto:Data: 2023-10-11T15:21:57.894 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.895 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.895 DEBUG:boto:Params: {} 2023-10-11T15:21:57.895 DEBUG:boto:Token: None 2023-10-11T15:21:57.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY= 2023-10-11T15:21:57.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY=', 'Content-Length': '0'} 2023-10-11T15:21:57.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfbb12754688dc08-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.946 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.947 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.947 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.948 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.948 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.949 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.949 DEBUG:boto:Data: 2023-10-11T15:21:57.949 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.950 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.950 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.950 DEBUG:boto:Params: {} 2023-10-11T15:21:57.951 DEBUG:boto:Token: None 2023-10-11T15:21:57.951 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.952 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBBkFPMv2YhrmBilBdNMKmYAzWQ= 2023-10-11T15:21:57.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBBkFPMv2YhrmBilBdNMKmYAzWQ=', 'Content-Length': '0'} 2023-10-11T15:21:57.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b8a645b276e992-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.955 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.955 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.956 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.956 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.956 DEBUG:boto:Method: HEAD 2023-10-11T15:21:57.957 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.957 DEBUG:boto:Data: 2023-10-11T15:21:57.958 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.959 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.959 DEBUG:boto:Params: {} 2023-10-11T15:21:57.959 DEBUG:boto:Token: None 2023-10-11T15:21:57.960 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hBBkFPMv2YhrmBilBdNMKmYAzWQ= 2023-10-11T15:21:57.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hBBkFPMv2YhrmBilBdNMKmYAzWQ=', 'Content-Length': '0'} 2023-10-11T15:21:57.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026913bf777af7fa-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:57.964 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.964 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.964 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.965 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.965 DEBUG:boto:Method: GET 2023-10-11T15:21:57.966 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.966 DEBUG:boto:Data: 2023-10-11T15:21:57.967 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.968 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.968 DEBUG:boto:Params: {} 2023-10-11T15:21:57.968 DEBUG:boto:Token: None 2023-10-11T15:21:57.969 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY= 2023-10-11T15:21:57.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY=', 'Content-Length': '0'} 2023-10-11T15:21:57.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c7175ed6857932-006526bd95-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.972 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:57.973 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:57.973 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.974 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.974 DEBUG:boto:Method: GET 2023-10-11T15:21:57.974 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.975 DEBUG:boto:Data: 2023-10-11T15:21:57.975 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.976 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.976 DEBUG:boto:Params: {} 2023-10-11T15:21:57.977 DEBUG:boto:Token: None 2023-10-11T15:21:57.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:21:57.978 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY= 2023-10-11T15:21:57.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cnHfcH415741R4feDEJpC35f0WY=', 'Content-Length': '0'} 2023-10-11T15:21:57.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076acc20db53f0405-006526bd95-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.981 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.982 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.982 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.983 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.983 DEBUG:boto:Method: GET 2023-10-11T15:21:57.984 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.984 DEBUG:boto:Data: 2023-10-11T15:21:57.985 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:57.985 DEBUG:boto:Port: 8001 2023-10-11T15:21:57.986 DEBUG:boto:Params: {} 2023-10-11T15:21:57.986 DEBUG:boto:Token: None 2023-10-11T15:21:57.987 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:57.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bbc966005b2744-006526bd95-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:57.991 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:57.991 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:57.992 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:57.992 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.993 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.993 DEBUG:boto:Method: GET 2023-10-11T15:21:57.994 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:57.994 DEBUG:boto:Data: 2023-10-11T15:21:57.995 DEBUG:boto:Headers: {} 2023-10-11T15:21:57.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:57.995 DEBUG:boto:Port: 8000 2023-10-11T15:21:57.996 DEBUG:boto:Params: {} 2023-10-11T15:21:57.996 DEBUG:boto:Token: None 2023-10-11T15:21:57.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:57 GMT /dghqvd-59/ 2023-10-11T15:21:57.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw= 2023-10-11T15:21:57.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:57 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TmWaI8l61QB6is/DYd9H8LgQeHw=', 'Content-Length': '0'} 2023-10-11T15:21:58.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5574b51f6d0e24-006526bd95-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:57 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.002 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.002 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:58.003 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:58.003 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.004 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:58.004 DEBUG:boto:Data: 2023-10-11T15:21:58.004 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.005 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.006 DEBUG:boto:Params: {} 2023-10-11T15:21:58.006 DEBUG:boto:Token: None 2023-10-11T15:21:58.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj68 2023-10-11T15:21:58.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lgQpe1Is16vs2eQeMj6xD6afocQ= 2023-10-11T15:21:58.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lgQpe1Is16vs2eQeMj6xD6afocQ=', 'Content-Length': '0'} 2023-10-11T15:21:58.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f8dac9d5e494283-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.010 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:58.010 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:58.011 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.011 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:58.012 DEBUG:boto:Data: 2023-10-11T15:21:58.012 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.013 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.013 DEBUG:boto:Params: {} 2023-10-11T15:21:58.014 DEBUG:boto:Token: None 2023-10-11T15:21:58.014 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj68 2023-10-11T15:21:58.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lgQpe1Is16vs2eQeMj6xD6afocQ= 2023-10-11T15:21:58.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lgQpe1Is16vs2eQeMj6xD6afocQ=', 'Content-Length': '0'} 2023-10-11T15:21:58.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa70bf28f69ebf4-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:21:58.018 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:58.019 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:58.019 DEBUG:boto:Method: GET 2023-10-11T15:21:58.019 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:58.020 DEBUG:boto:Data: 2023-10-11T15:21:58.020 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.021 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.022 DEBUG:boto:Params: {} 2023-10-11T15:21:58.022 DEBUG:boto:Token: None 2023-10-11T15:21:58.023 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj68 2023-10-11T15:21:58.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IsaNDdr4tuj9TD5BcxdI82exEVc= 2023-10-11T15:21:58.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IsaNDdr4tuj9TD5BcxdI82exEVc=', 'Content-Length': '0'} 2023-10-11T15:21:58.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee10085e84ee8d24-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.066 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:21:58.067 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:21:58.067 DEBUG:boto:Method: GET 2023-10-11T15:21:58.068 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:21:58.068 DEBUG:boto:Data: 2023-10-11T15:21:58.069 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.070 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.070 DEBUG:boto:Params: {} 2023-10-11T15:21:58.071 DEBUG:boto:Token: None 2023-10-11T15:21:58.071 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj68 2023-10-11T15:21:58.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IsaNDdr4tuj9TD5BcxdI82exEVc= 2023-10-11T15:21:58.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IsaNDdr4tuj9TD5BcxdI82exEVc=', 'Content-Length': '0'} 2023-10-11T15:21:58.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1251dc8b2fbfef1-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.075 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.075 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.076 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.076 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.077 DEBUG:boto:Method: GET 2023-10-11T15:21:58.077 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.078 DEBUG:boto:Data: 2023-10-11T15:21:58.078 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.079 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.079 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.079 DEBUG:boto:Params: {} 2023-10-11T15:21:58.080 DEBUG:boto:Token: None 2023-10-11T15:21:58.080 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.081 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c27513261eb00af-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.084 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.085 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.086 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.086 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.086 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.087 DEBUG:boto:Method: GET 2023-10-11T15:21:58.087 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:21:58.088 DEBUG:boto:Data: 2023-10-11T15:21:58.088 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.089 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.090 DEBUG:boto:Params: {} 2023-10-11T15:21:58.090 DEBUG:boto:Token: None 2023-10-11T15:21:58.090 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048356d943685b971-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.138 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:21:58.140 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.140 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.253 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.254 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.254 DEBUG:boto:Method: GET 2023-10-11T15:21:58.255 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.255 DEBUG:boto:Data: 2023-10-11T15:21:58.256 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.256 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.257 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.257 DEBUG:boto:Params: {} 2023-10-11T15:21:58.258 DEBUG:boto:Token: None 2023-10-11T15:21:58.258 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4= 2023-10-11T15:21:58.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4=', 'Content-Length': '0'} 2023-10-11T15:21:58.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6b4fe468e72f030-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.262 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.263 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.263 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.264 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.264 DEBUG:boto:Method: GET 2023-10-11T15:21:58.265 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.265 DEBUG:boto:Data: 2023-10-11T15:21:58.265 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.266 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.267 DEBUG:boto:Params: {} 2023-10-11T15:21:58.267 DEBUG:boto:Token: None 2023-10-11T15:21:58.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4= 2023-10-11T15:21:58.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4=', 'Content-Length': '0'} 2023-10-11T15:21:58.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061549f6ef00d4d84-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.271 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.272 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.272 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.273 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.273 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.273 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.274 DEBUG:boto:Data: 2023-10-11T15:21:58.274 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.275 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.275 DEBUG:boto:Params: {} 2023-10-11T15:21:58.276 DEBUG:boto:Token: None 2023-10-11T15:21:58.276 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFg379rgfkwuo97gvRoD8atB/0A= 2023-10-11T15:21:58.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFg379rgfkwuo97gvRoD8atB/0A=', 'Content-Length': '0'} 2023-10-11T15:21:58.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb60deb8bc2ea04-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.280 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.280 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.281 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.281 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.281 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.282 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.282 DEBUG:boto:Data: 2023-10-11T15:21:58.283 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.283 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.283 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.284 DEBUG:boto:Params: {} 2023-10-11T15:21:58.284 DEBUG:boto:Token: None 2023-10-11T15:21:58.285 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFg379rgfkwuo97gvRoD8atB/0A= 2023-10-11T15:21:58.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFg379rgfkwuo97gvRoD8atB/0A=', 'Content-Length': '0'} 2023-10-11T15:21:58.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000063553f595612d7b-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:21:58.289 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.289 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.289 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.290 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.290 DEBUG:boto:Method: GET 2023-10-11T15:21:58.291 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.291 DEBUG:boto:Data: 2023-10-11T15:21:58.292 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.292 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.293 DEBUG:boto:Params: {} 2023-10-11T15:21:58.293 DEBUG:boto:Token: None 2023-10-11T15:21:58.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4= 2023-10-11T15:21:58.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4=', 'Content-Length': '0'} 2023-10-11T15:21:58.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01157fc450d6cdf-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.297 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.298 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.298 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.298 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.299 DEBUG:boto:Method: GET 2023-10-11T15:21:58.299 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.300 DEBUG:boto:Data: 2023-10-11T15:21:58.300 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.301 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.301 DEBUG:boto:Params: {} 2023-10-11T15:21:58.302 DEBUG:boto:Token: None 2023-10-11T15:21:58.302 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:21:58.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4= 2023-10-11T15:21:58.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1f3z6GokI7IHW68xv7Rt2od4ss4=', 'Content-Length': '0'} 2023-10-11T15:21:58.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000348b5d728121bd-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.306 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.306 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.307 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.307 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.308 DEBUG:boto:Method: GET 2023-10-11T15:21:58.308 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.309 DEBUG:boto:Data: 2023-10-11T15:21:58.309 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.310 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.310 DEBUG:boto:Params: {} 2023-10-11T15:21:58.311 DEBUG:boto:Token: None 2023-10-11T15:21:58.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde71c8448696f7e-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.315 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.316 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.316 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.317 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.317 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.318 DEBUG:boto:Method: GET 2023-10-11T15:21:58.318 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.319 DEBUG:boto:Data: 2023-10-11T15:21:58.319 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.320 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.320 DEBUG:boto:Params: {} 2023-10-11T15:21:58.321 DEBUG:boto:Token: None 2023-10-11T15:21:58.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f80955327acff1-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.326 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.327 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.327 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.328 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.328 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:21:58.329 DEBUG:boto:Data: 2023-10-11T15:21:58.329 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.329 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.330 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.330 DEBUG:boto:Params: {} 2023-10-11T15:21:58.331 DEBUG:boto:Token: None 2023-10-11T15:21:58.331 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69 2023-10-11T15:21:58.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:10ql/QgBjjQAdEqrCfm4xn1bfu8= 2023-10-11T15:21:58.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:10ql/QgBjjQAdEqrCfm4xn1bfu8=', 'Content-Length': '0'} 2023-10-11T15:21:58.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9b982e03da3859-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.335 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.335 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.335 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.336 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:21:58.336 DEBUG:boto:Data: 2023-10-11T15:21:58.337 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.337 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.338 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.338 DEBUG:boto:Params: {} 2023-10-11T15:21:58.338 DEBUG:boto:Token: None 2023-10-11T15:21:58.339 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69 2023-10-11T15:21:58.339 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:10ql/QgBjjQAdEqrCfm4xn1bfu8= 2023-10-11T15:21:58.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:10ql/QgBjjQAdEqrCfm4xn1bfu8=', 'Content-Length': '0'} 2023-10-11T15:21:58.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003efcd4b295edd8a6-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:21:58.343 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.343 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.344 DEBUG:boto:Method: GET 2023-10-11T15:21:58.344 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:21:58.345 DEBUG:boto:Data: 2023-10-11T15:21:58.345 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.346 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.346 DEBUG:boto:Params: {} 2023-10-11T15:21:58.347 DEBUG:boto:Token: None 2023-10-11T15:21:58.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69 2023-10-11T15:21:58.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1GZ3NgoJAfbzSqRVV9QwQnVTaTM= 2023-10-11T15:21:58.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1GZ3NgoJAfbzSqRVV9QwQnVTaTM=', 'Content-Length': '0'} 2023-10-11T15:21:58.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b027578c8d2ef5fd-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.394 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:21:58.395 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:21:58.395 DEBUG:boto:Method: GET 2023-10-11T15:21:58.396 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:21:58.396 DEBUG:boto:Data: 2023-10-11T15:21:58.397 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.398 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.398 DEBUG:boto:Params: {} 2023-10-11T15:21:58.399 DEBUG:boto:Token: None 2023-10-11T15:21:58.399 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj69 2023-10-11T15:21:58.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1GZ3NgoJAfbzSqRVV9QwQnVTaTM= 2023-10-11T15:21:58.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1GZ3NgoJAfbzSqRVV9QwQnVTaTM=', 'Content-Length': '0'} 2023-10-11T15:21:58.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007461278f5b559d99-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.403 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.404 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.404 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.405 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.405 DEBUG:boto:Method: GET 2023-10-11T15:21:58.406 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.406 DEBUG:boto:Data: 2023-10-11T15:21:58.407 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.407 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.408 DEBUG:boto:Params: {} 2023-10-11T15:21:58.408 DEBUG:boto:Token: None 2023-10-11T15:21:58.409 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086ae640cab9da6cd-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.413 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.414 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.414 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.415 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.415 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.416 DEBUG:boto:Method: GET 2023-10-11T15:21:58.416 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:21:58.416 DEBUG:boto:Data: 2023-10-11T15:21:58.417 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.418 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.418 DEBUG:boto:Params: {} 2023-10-11T15:21:58.419 DEBUG:boto:Token: None 2023-10-11T15:21:58.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62174bfc46dce1a-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.424 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:58.425 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.426 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.426 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.426 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.427 DEBUG:boto:Method: GET 2023-10-11T15:21:58.427 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.428 DEBUG:boto:Data: 2023-10-11T15:21:58.428 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.429 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.430 DEBUG:boto:Params: {} 2023-10-11T15:21:58.430 DEBUG:boto:Token: None 2023-10-11T15:21:58.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.431 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y= 2023-10-11T15:21:58.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y=', 'Content-Length': '0'} 2023-10-11T15:21:58.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a92aa4a0b894729a-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.434 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.435 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.435 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.436 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.436 DEBUG:boto:Method: GET 2023-10-11T15:21:58.436 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.437 DEBUG:boto:Data: 2023-10-11T15:21:58.437 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.438 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.439 DEBUG:boto:Params: {} 2023-10-11T15:21:58.439 DEBUG:boto:Token: None 2023-10-11T15:21:58.440 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y= 2023-10-11T15:21:58.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y=', 'Content-Length': '0'} 2023-10-11T15:21:58.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f63a2a82f2e5c3-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.486 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.487 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.487 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.488 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.488 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.489 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.489 DEBUG:boto:Data: 2023-10-11T15:21:58.490 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.491 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.491 DEBUG:boto:Params: {} 2023-10-11T15:21:58.491 DEBUG:boto:Token: None 2023-10-11T15:21:58.492 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I2ENI4uS2+yUF+4UrfDsMe09gP8= 2023-10-11T15:21:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I2ENI4uS2+yUF+4UrfDsMe09gP8=', 'Content-Length': '0'} 2023-10-11T15:21:58.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c8c3235be38e703-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.496 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.496 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.496 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.497 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.497 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.498 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.498 DEBUG:boto:Data: 2023-10-11T15:21:58.499 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.499 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.500 DEBUG:boto:Params: {} 2023-10-11T15:21:58.500 DEBUG:boto:Token: None 2023-10-11T15:21:58.501 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I2ENI4uS2+yUF+4UrfDsMe09gP8= 2023-10-11T15:21:58.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I2ENI4uS2+yUF+4UrfDsMe09gP8=', 'Content-Length': '0'} 2023-10-11T15:21:58.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273de4cf83fafe07-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:58.505 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.505 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.506 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.506 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.506 DEBUG:boto:Method: GET 2023-10-11T15:21:58.507 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.507 DEBUG:boto:Data: 2023-10-11T15:21:58.508 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.509 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.509 DEBUG:boto:Params: {} 2023-10-11T15:21:58.510 DEBUG:boto:Token: None 2023-10-11T15:21:58.510 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y= 2023-10-11T15:21:58.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y=', 'Content-Length': '0'} 2023-10-11T15:21:58.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daf9679cec44a24-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.514 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.514 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.515 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.515 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.515 DEBUG:boto:Method: GET 2023-10-11T15:21:58.516 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.516 DEBUG:boto:Data: 2023-10-11T15:21:58.517 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.518 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.518 DEBUG:boto:Params: {} 2023-10-11T15:21:58.518 DEBUG:boto:Token: None 2023-10-11T15:21:58.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:21:58.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y= 2023-10-11T15:21:58.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p4tUU+sxgD/zBJbOZhvIxHevw2Y=', 'Content-Length': '0'} 2023-10-11T15:21:58.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb68fbd6c54387d-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.523 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.523 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.524 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.524 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.524 DEBUG:boto:Method: GET 2023-10-11T15:21:58.525 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.525 DEBUG:boto:Data: 2023-10-11T15:21:58.526 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.527 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.527 DEBUG:boto:Params: {} 2023-10-11T15:21:58.528 DEBUG:boto:Token: None 2023-10-11T15:21:58.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edc0f4c6770c1ae3-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.532 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.533 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.534 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.534 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.534 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.535 DEBUG:boto:Method: GET 2023-10-11T15:21:58.535 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.536 DEBUG:boto:Data: 2023-10-11T15:21:58.536 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.537 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.538 DEBUG:boto:Params: {} 2023-10-11T15:21:58.538 DEBUG:boto:Token: None 2023-10-11T15:21:58.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024422ecd591516be-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.543 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.544 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.544 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.545 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.545 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:58.546 DEBUG:boto:Data: 2023-10-11T15:21:58.546 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.547 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.547 DEBUG:boto:Params: {} 2023-10-11T15:21:58.548 DEBUG:boto:Token: None 2023-10-11T15:21:58.548 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76 2023-10-11T15:21:58.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xZMRM/V3+/AX0o3L7waS5DlAQE8= 2023-10-11T15:21:58.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xZMRM/V3+/AX0o3L7waS5DlAQE8=', 'Content-Length': '0'} 2023-10-11T15:21:58.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d377468f2c5699-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.552 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.552 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.553 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.553 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:58.554 DEBUG:boto:Data: 2023-10-11T15:21:58.554 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.555 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.555 DEBUG:boto:Params: {} 2023-10-11T15:21:58.556 DEBUG:boto:Token: None 2023-10-11T15:21:58.556 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76 2023-10-11T15:21:58.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xZMRM/V3+/AX0o3L7waS5DlAQE8= 2023-10-11T15:21:58.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xZMRM/V3+/AX0o3L7waS5DlAQE8=', 'Content-Length': '0'} 2023-10-11T15:21:58.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b74fb420c652df4-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:21:58.560 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.560 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.561 DEBUG:boto:Method: GET 2023-10-11T15:21:58.561 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:58.562 DEBUG:boto:Data: 2023-10-11T15:21:58.562 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.563 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.564 DEBUG:boto:Params: {} 2023-10-11T15:21:58.565 DEBUG:boto:Token: None 2023-10-11T15:21:58.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76 2023-10-11T15:21:58.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/ZDFVIsyxUvdP7h19ir513B1ocM= 2023-10-11T15:21:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/ZDFVIsyxUvdP7h19ir513B1ocM=', 'Content-Length': '0'} 2023-10-11T15:21:58.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bd5a229bdf0ba9-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.610 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:21:58.611 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:21:58.611 DEBUG:boto:Method: GET 2023-10-11T15:21:58.612 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:21:58.612 DEBUG:boto:Data: 2023-10-11T15:21:58.613 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.614 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.614 DEBUG:boto:Params: {} 2023-10-11T15:21:58.615 DEBUG:boto:Token: None 2023-10-11T15:21:58.615 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj76 2023-10-11T15:21:58.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/ZDFVIsyxUvdP7h19ir513B1ocM= 2023-10-11T15:21:58.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/ZDFVIsyxUvdP7h19ir513B1ocM=', 'Content-Length': '0'} 2023-10-11T15:21:58.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0cdb33f1e3273e0-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.619 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.620 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.620 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.621 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.621 DEBUG:boto:Method: GET 2023-10-11T15:21:58.621 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.622 DEBUG:boto:Data: 2023-10-11T15:21:58.622 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.623 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.623 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.624 DEBUG:boto:Params: {} 2023-10-11T15:21:58.624 DEBUG:boto:Token: None 2023-10-11T15:21:58.625 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.625 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000614db516e4261f2-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.629 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.629 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.630 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.630 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.631 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.631 DEBUG:boto:Method: GET 2023-10-11T15:21:58.631 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:21:58.632 DEBUG:boto:Data: 2023-10-11T15:21:58.632 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.633 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.633 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.633 DEBUG:boto:Params: {} 2023-10-11T15:21:58.634 DEBUG:boto:Token: None 2023-10-11T15:21:58.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de40818640f0b8d7-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.639 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:58.640 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.641 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.641 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.641 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.642 DEBUG:boto:Method: GET 2023-10-11T15:21:58.642 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.643 DEBUG:boto:Data: 2023-10-11T15:21:58.643 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.644 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.644 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.644 DEBUG:boto:Params: {} 2023-10-11T15:21:58.645 DEBUG:boto:Token: None 2023-10-11T15:21:58.645 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.646 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc= 2023-10-11T15:21:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc=', 'Content-Length': '0'} 2023-10-11T15:21:58.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa469a727995e27-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.649 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.649 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.650 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.650 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.651 DEBUG:boto:Method: GET 2023-10-11T15:21:58.651 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.652 DEBUG:boto:Data: 2023-10-11T15:21:58.652 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.653 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.653 DEBUG:boto:Params: {} 2023-10-11T15:21:58.654 DEBUG:boto:Token: None 2023-10-11T15:21:58.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc= 2023-10-11T15:21:58.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc=', 'Content-Length': '0'} 2023-10-11T15:21:58.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ab35a5fff179cc9-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.658 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.659 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.659 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.659 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.660 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.660 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.661 DEBUG:boto:Data: 2023-10-11T15:21:58.661 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.662 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.662 DEBUG:boto:Params: {} 2023-10-11T15:21:58.663 DEBUG:boto:Token: None 2023-10-11T15:21:58.663 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N4iz+1t5TZ63Xpqvcs0CH7t3/50= 2023-10-11T15:21:58.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N4iz+1t5TZ63Xpqvcs0CH7t3/50=', 'Content-Length': '0'} 2023-10-11T15:21:58.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567866701b3a4baa-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.667 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.667 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.668 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.668 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.668 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.669 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.669 DEBUG:boto:Data: 2023-10-11T15:21:58.670 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.670 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.671 DEBUG:boto:Params: {} 2023-10-11T15:21:58.671 DEBUG:boto:Token: None 2023-10-11T15:21:58.672 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N4iz+1t5TZ63Xpqvcs0CH7t3/50= 2023-10-11T15:21:58.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N4iz+1t5TZ63Xpqvcs0CH7t3/50=', 'Content-Length': '0'} 2023-10-11T15:21:58.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa4f41cce9a2bf6-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:58.676 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.676 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.676 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.677 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.677 DEBUG:boto:Method: GET 2023-10-11T15:21:58.678 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.678 DEBUG:boto:Data: 2023-10-11T15:21:58.679 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.679 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.680 DEBUG:boto:Params: {} 2023-10-11T15:21:58.680 DEBUG:boto:Token: None 2023-10-11T15:21:58.681 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc= 2023-10-11T15:21:58.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc=', 'Content-Length': '0'} 2023-10-11T15:21:58.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dac7a0b164f2b8b5-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.685 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.685 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.685 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.686 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.686 DEBUG:boto:Method: GET 2023-10-11T15:21:58.687 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.687 DEBUG:boto:Data: 2023-10-11T15:21:58.688 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.689 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.689 DEBUG:boto:Params: {} 2023-10-11T15:21:58.689 DEBUG:boto:Token: None 2023-10-11T15:21:58.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:21:58.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc= 2023-10-11T15:21:58.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XCPzmcqO8nSLJGtI+Cu1lIJ8Moc=', 'Content-Length': '0'} 2023-10-11T15:21:58.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e660a0be0a4627b-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.734 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.735 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.735 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.736 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.736 DEBUG:boto:Method: GET 2023-10-11T15:21:58.736 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.737 DEBUG:boto:Data: 2023-10-11T15:21:58.737 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.738 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.738 DEBUG:boto:Params: {} 2023-10-11T15:21:58.739 DEBUG:boto:Token: None 2023-10-11T15:21:58.739 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008652c9adb12469f9-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.743 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.744 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.744 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.745 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.745 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.746 DEBUG:boto:Method: GET 2023-10-11T15:21:58.746 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.747 DEBUG:boto:Data: 2023-10-11T15:21:58.747 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.748 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.748 DEBUG:boto:Params: {} 2023-10-11T15:21:58.749 DEBUG:boto:Token: None 2023-10-11T15:21:58.749 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.750 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5555d2a995c74f9-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.754 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.755 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.755 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.755 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.756 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:58.756 DEBUG:boto:Data: 2023-10-11T15:21:58.757 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.757 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.758 DEBUG:boto:Params: {} 2023-10-11T15:21:58.758 DEBUG:boto:Token: None 2023-10-11T15:21:58.759 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85 2023-10-11T15:21:58.759 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LAk2GILuZcGshPjcyPmxZXkFXCI= 2023-10-11T15:21:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LAk2GILuZcGshPjcyPmxZXkFXCI=', 'Content-Length': '0'} 2023-10-11T15:21:58.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017a7e11c3eb901a7-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.762 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.763 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.763 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.764 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:58.764 DEBUG:boto:Data: 2023-10-11T15:21:58.764 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.765 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.766 DEBUG:boto:Params: {} 2023-10-11T15:21:58.766 DEBUG:boto:Token: None 2023-10-11T15:21:58.767 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85 2023-10-11T15:21:58.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LAk2GILuZcGshPjcyPmxZXkFXCI= 2023-10-11T15:21:58.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LAk2GILuZcGshPjcyPmxZXkFXCI=', 'Content-Length': '0'} 2023-10-11T15:21:58.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a77df7d7e05362b-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:21:58.770 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.771 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.771 DEBUG:boto:Method: GET 2023-10-11T15:21:58.773 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:58.774 DEBUG:boto:Data: 2023-10-11T15:21:58.774 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.775 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.776 DEBUG:boto:Params: {} 2023-10-11T15:21:58.776 DEBUG:boto:Token: None 2023-10-11T15:21:58.777 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85 2023-10-11T15:21:58.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8wzEmGJ2jASr+nmt3w9bTrkL5E= 2023-10-11T15:21:58.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8wzEmGJ2jASr+nmt3w9bTrkL5E=', 'Content-Length': '0'} 2023-10-11T15:21:58.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f534e8cd7470418e-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.781 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:21:58.781 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:21:58.782 DEBUG:boto:Method: GET 2023-10-11T15:21:58.782 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:21:58.783 DEBUG:boto:Data: 2023-10-11T15:21:58.783 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.783 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.784 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.784 DEBUG:boto:Params: {} 2023-10-11T15:21:58.785 DEBUG:boto:Token: None 2023-10-11T15:21:58.785 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj85 2023-10-11T15:21:58.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8wzEmGJ2jASr+nmt3w9bTrkL5E= 2023-10-11T15:21:58.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8wzEmGJ2jASr+nmt3w9bTrkL5E=', 'Content-Length': '0'} 2023-10-11T15:21:58.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075179e45eed35295-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.789 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.790 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.790 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.791 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.791 DEBUG:boto:Method: GET 2023-10-11T15:21:58.791 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.792 DEBUG:boto:Data: 2023-10-11T15:21:58.792 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.793 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.794 DEBUG:boto:Params: {} 2023-10-11T15:21:58.794 DEBUG:boto:Token: None 2023-10-11T15:21:58.795 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004615ac905992e01b-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.798 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.799 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.800 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.800 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.801 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.801 DEBUG:boto:Method: GET 2023-10-11T15:21:58.801 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:21:58.802 DEBUG:boto:Data: 2023-10-11T15:21:58.802 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.803 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.804 DEBUG:boto:Params: {} 2023-10-11T15:21:58.804 DEBUG:boto:Token: None 2023-10-11T15:21:58.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.805 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e7c165daf7279a-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.810 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:21:58.811 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.811 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.812 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.812 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.813 DEBUG:boto:Method: GET 2023-10-11T15:21:58.813 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.814 DEBUG:boto:Data: 2023-10-11T15:21:58.814 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.815 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.815 DEBUG:boto:Params: {} 2023-10-11T15:21:58.816 DEBUG:boto:Token: None 2023-10-11T15:21:58.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY= 2023-10-11T15:21:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY=', 'Content-Length': '0'} 2023-10-11T15:21:58.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cac1f761ddb7d49-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.820 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.821 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.821 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.822 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.822 DEBUG:boto:Method: GET 2023-10-11T15:21:58.823 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.823 DEBUG:boto:Data: 2023-10-11T15:21:58.824 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.824 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.824 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.825 DEBUG:boto:Params: {} 2023-10-11T15:21:58.825 DEBUG:boto:Token: None 2023-10-11T15:21:58.826 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY= 2023-10-11T15:21:58.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY=', 'Content-Length': '0'} 2023-10-11T15:21:58.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068374948ea7d3d3e-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.829 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.830 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.830 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.831 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.831 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.832 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.832 DEBUG:boto:Data: 2023-10-11T15:21:58.832 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.833 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.834 DEBUG:boto:Params: {} 2023-10-11T15:21:58.834 DEBUG:boto:Token: None 2023-10-11T15:21:58.835 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d+OEtA1MaeqRwJQ5wNV1T+dYLvY= 2023-10-11T15:21:58.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d+OEtA1MaeqRwJQ5wNV1T+dYLvY=', 'Content-Length': '0'} 2023-10-11T15:21:58.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3fdd0eec94f403-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.838 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.839 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.839 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.840 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.840 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.840 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.841 DEBUG:boto:Data: 2023-10-11T15:21:58.841 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.842 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.843 DEBUG:boto:Params: {} 2023-10-11T15:21:58.843 DEBUG:boto:Token: None 2023-10-11T15:21:58.844 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d+OEtA1MaeqRwJQ5wNV1T+dYLvY= 2023-10-11T15:21:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d+OEtA1MaeqRwJQ5wNV1T+dYLvY=', 'Content-Length': '0'} 2023-10-11T15:21:58.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdd211c5fde7d2d-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:21:58.848 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.848 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.849 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.849 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.849 DEBUG:boto:Method: GET 2023-10-11T15:21:58.850 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.850 DEBUG:boto:Data: 2023-10-11T15:21:58.851 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.852 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.852 DEBUG:boto:Params: {} 2023-10-11T15:21:58.853 DEBUG:boto:Token: None 2023-10-11T15:21:58.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY= 2023-10-11T15:21:58.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY=', 'Content-Length': '0'} 2023-10-11T15:21:58.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd5edea449685f5-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.857 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.857 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.858 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.858 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.859 DEBUG:boto:Method: GET 2023-10-11T15:21:58.859 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.860 DEBUG:boto:Data: 2023-10-11T15:21:58.860 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.861 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.861 DEBUG:boto:Params: {} 2023-10-11T15:21:58.862 DEBUG:boto:Token: None 2023-10-11T15:21:58.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:21:58.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY= 2023-10-11T15:21:58.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3d/ATtlxppWa47L7Rvx4gtKfrAY=', 'Content-Length': '0'} 2023-10-11T15:21:58.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b11b2c2e197d2c8-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.866 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.867 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.867 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.868 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.868 DEBUG:boto:Method: GET 2023-10-11T15:21:58.869 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.869 DEBUG:boto:Data: 2023-10-11T15:21:58.870 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.871 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.871 DEBUG:boto:Params: {} 2023-10-11T15:21:58.872 DEBUG:boto:Token: None 2023-10-11T15:21:58.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3aee96dd2063dd4-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.876 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.877 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.877 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.878 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.878 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.879 DEBUG:boto:Method: GET 2023-10-11T15:21:58.879 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.880 DEBUG:boto:Data: 2023-10-11T15:21:58.880 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.881 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.881 DEBUG:boto:Params: {} 2023-10-11T15:21:58.882 DEBUG:boto:Token: None 2023-10-11T15:21:58.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082888a751a543641-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.888 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.888 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.889 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.889 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.890 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:21:58.890 DEBUG:boto:Data: 2023-10-11T15:21:58.891 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.891 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.892 DEBUG:boto:Params: {} 2023-10-11T15:21:58.892 DEBUG:boto:Token: None 2023-10-11T15:21:58.893 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92 2023-10-11T15:21:58.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1KeCBJDOtCWCKvBtIRe7Sqk6jCw= 2023-10-11T15:21:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1KeCBJDOtCWCKvBtIRe7Sqk6jCw=', 'Content-Length': '0'} 2023-10-11T15:21:58.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003153fcb43c981f92-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.896 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.897 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.897 DEBUG:boto:Method: HEAD 2023-10-11T15:21:58.898 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:21:58.898 DEBUG:boto:Data: 2023-10-11T15:21:58.899 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.899 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.900 DEBUG:boto:Params: {} 2023-10-11T15:21:58.900 DEBUG:boto:Token: None 2023-10-11T15:21:58.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92 2023-10-11T15:21:58.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1KeCBJDOtCWCKvBtIRe7Sqk6jCw= 2023-10-11T15:21:58.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1KeCBJDOtCWCKvBtIRe7Sqk6jCw=', 'Content-Length': '0'} 2023-10-11T15:21:58.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004277b359f76230ca-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:21:58.905 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.905 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.906 DEBUG:boto:Method: GET 2023-10-11T15:21:58.906 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:21:58.907 DEBUG:boto:Data: 2023-10-11T15:21:58.907 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.908 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.909 DEBUG:boto:Params: {} 2023-10-11T15:21:58.909 DEBUG:boto:Token: None 2023-10-11T15:21:58.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92 2023-10-11T15:21:58.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sncvHg5B/e2LCA9faKgg72dHV1A= 2023-10-11T15:21:58.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sncvHg5B/e2LCA9faKgg72dHV1A=', 'Content-Length': '0'} 2023-10-11T15:21:58.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6f4ddab70661ce-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.954 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:21:58.955 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:21:58.956 DEBUG:boto:Method: GET 2023-10-11T15:21:58.956 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:21:58.956 DEBUG:boto:Data: 2023-10-11T15:21:58.957 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.958 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.958 DEBUG:boto:Params: {} 2023-10-11T15:21:58.959 DEBUG:boto:Token: None 2023-10-11T15:21:58.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj92 2023-10-11T15:21:58.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sncvHg5B/e2LCA9faKgg72dHV1A= 2023-10-11T15:21:58.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sncvHg5B/e2LCA9faKgg72dHV1A=', 'Content-Length': '0'} 2023-10-11T15:21:58.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618856f175e820f2-006526bd96-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.963 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.963 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.964 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.964 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.965 DEBUG:boto:Method: GET 2023-10-11T15:21:58.965 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.966 DEBUG:boto:Data: 2023-10-11T15:21:58.966 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.967 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.967 DEBUG:boto:Params: {} 2023-10-11T15:21:58.968 DEBUG:boto:Token: None 2023-10-11T15:21:58.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6aadb947bc3e31b-006526bd96-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.972 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.973 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:58.974 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:58.974 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.975 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.975 DEBUG:boto:Method: GET 2023-10-11T15:21:58.977 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:21:58.977 DEBUG:boto:Data: 2023-10-11T15:21:58.978 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.978 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.979 DEBUG:boto:Params: {} 2023-10-11T15:21:58.979 DEBUG:boto:Token: None 2023-10-11T15:21:58.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/ 2023-10-11T15:21:58.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA= 2023-10-11T15:21:58.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xYvpFfV0VbmQqgyxdCWFpUdZzVA=', 'Content-Length': '0'} 2023-10-11T15:21:58.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003692b128d70fec20-006526bd96-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.985 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:58.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:58.986 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:58.986 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:58.987 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.987 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.988 DEBUG:boto:Method: GET 2023-10-11T15:21:58.988 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.989 DEBUG:boto:Data: 2023-10-11T15:21:58.989 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:58.990 DEBUG:boto:Port: 8001 2023-10-11T15:21:58.990 DEBUG:boto:Params: {} 2023-10-11T15:21:58.991 DEBUG:boto:Token: None 2023-10-11T15:21:58.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:58 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b+PBJarJtJxjlIQ7AVTY4gqWdFE= 2023-10-11T15:21:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:58 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b+PBJarJtJxjlIQ7AVTY4gqWdFE=', 'Content-Length': '0'} 2023-10-11T15:21:58.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd761326b005900-006526bd96-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:58 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:58.995 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:58.996 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:58.996 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.996 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.997 DEBUG:boto:Method: GET 2023-10-11T15:21:58.997 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:58.998 DEBUG:boto:Data: 2023-10-11T15:21:58.998 DEBUG:boto:Headers: {} 2023-10-11T15:21:58.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:58.999 DEBUG:boto:Port: 8000 2023-10-11T15:21:58.999 DEBUG:boto:Params: {} 2023-10-11T15:21:59.000 DEBUG:boto:Token: None 2023-10-11T15:21:59.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk= 2023-10-11T15:21:59.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk=', 'Content-Length': '0'} 2023-10-11T15:21:59.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001658d7e56c293169-006526bd97-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.004 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.005 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.005 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.005 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.006 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.006 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.007 DEBUG:boto:Data: 2023-10-11T15:21:59.007 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.008 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.008 DEBUG:boto:Params: {} 2023-10-11T15:21:59.009 DEBUG:boto:Token: None 2023-10-11T15:21:59.009 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.010 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGfwl4O7SUQTig65YVG+l8VbzjI= 2023-10-11T15:21:59.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGfwl4O7SUQTig65YVG+l8VbzjI=', 'Content-Length': '0'} 2023-10-11T15:21:59.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dcaeb99533a03db-006526bd97-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.013 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.013 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.014 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.014 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.014 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.015 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.015 DEBUG:boto:Data: 2023-10-11T15:21:59.016 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.016 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.017 DEBUG:boto:Params: {} 2023-10-11T15:21:59.017 DEBUG:boto:Token: None 2023-10-11T15:21:59.018 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qGfwl4O7SUQTig65YVG+l8VbzjI= 2023-10-11T15:21:59.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qGfwl4O7SUQTig65YVG+l8VbzjI=', 'Content-Length': '0'} 2023-10-11T15:21:59.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2e2b41797bb819-006526bd97-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:59.022 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.022 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.022 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.023 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.023 DEBUG:boto:Method: GET 2023-10-11T15:21:59.024 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.024 DEBUG:boto:Data: 2023-10-11T15:21:59.024 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.025 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.026 DEBUG:boto:Params: {} 2023-10-11T15:21:59.026 DEBUG:boto:Token: None 2023-10-11T15:21:59.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk= 2023-10-11T15:21:59.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk=', 'Content-Length': '0'} 2023-10-11T15:21:59.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4889933b5e4fd77-006526bd97-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.030 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.031 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.031 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.031 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.032 DEBUG:boto:Method: GET 2023-10-11T15:21:59.032 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.033 DEBUG:boto:Data: 2023-10-11T15:21:59.033 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.034 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.034 DEBUG:boto:Params: {} 2023-10-11T15:21:59.035 DEBUG:boto:Token: None 2023-10-11T15:21:59.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:21:59.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk= 2023-10-11T15:21:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WxNsbbU4Qai+UdAszR9/R4/XqFk=', 'Content-Length': '0'} 2023-10-11T15:21:59.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f8961e02af15af-006526bd97-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.039 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.039 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.039 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.040 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.040 DEBUG:boto:Method: GET 2023-10-11T15:21:59.041 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.041 DEBUG:boto:Data: 2023-10-11T15:21:59.042 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.042 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.043 DEBUG:boto:Params: {} 2023-10-11T15:21:59.043 DEBUG:boto:Token: None 2023-10-11T15:21:59.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I= 2023-10-11T15:21:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I=', 'Content-Length': '0'} 2023-10-11T15:21:59.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000536e566a141e94d0-006526bd97-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.048 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:59.048 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.049 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.049 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.050 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.050 DEBUG:boto:Method: GET 2023-10-11T15:21:59.050 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.051 DEBUG:boto:Data: 2023-10-11T15:21:59.051 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.052 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.052 DEBUG:boto:Params: {} 2023-10-11T15:21:59.053 DEBUG:boto:Token: None 2023-10-11T15:21:59.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I= 2023-10-11T15:21:59.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I=', 'Content-Length': '0'} 2023-10-11T15:21:59.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395ac46a86c00469-006526bd97-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.058 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:59.059 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.059 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.060 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.060 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:59.061 DEBUG:boto:Data: 2023-10-11T15:21:59.061 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.062 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.062 DEBUG:boto:Params: {} 2023-10-11T15:21:59.063 DEBUG:boto:Token: None 2023-10-11T15:21:59.063 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95 2023-10-11T15:21:59.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knmN8D5HgthfK5yJamNTZ/jwTko= 2023-10-11T15:21:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knmN8D5HgthfK5yJamNTZ/jwTko=', 'Content-Length': '0'} 2023-10-11T15:21:59.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000166da31c96291464-006526bd97-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.066 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.067 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.067 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.068 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:59.068 DEBUG:boto:Data: 2023-10-11T15:21:59.069 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.069 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.070 DEBUG:boto:Params: {} 2023-10-11T15:21:59.070 DEBUG:boto:Token: None 2023-10-11T15:21:59.071 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95 2023-10-11T15:21:59.071 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knmN8D5HgthfK5yJamNTZ/jwTko= 2023-10-11T15:21:59.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knmN8D5HgthfK5yJamNTZ/jwTko=', 'Content-Length': '0'} 2023-10-11T15:21:59.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1033f73746b86b5-006526bd97-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:21:59.074 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.075 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.075 DEBUG:boto:Method: GET 2023-10-11T15:21:59.076 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:59.076 DEBUG:boto:Data: 2023-10-11T15:21:59.076 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.077 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.077 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.078 DEBUG:boto:Params: {} 2023-10-11T15:21:59.078 DEBUG:boto:Token: None 2023-10-11T15:21:59.079 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95 2023-10-11T15:21:59.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzADiyIrceuaq+HiwRHaIWDiP8w= 2023-10-11T15:21:59.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzADiyIrceuaq+HiwRHaIWDiP8w=', 'Content-Length': '0'} 2023-10-11T15:21:59.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000740313241bb0b3a5-006526bd97-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.126 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:21:59.127 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:21:59.127 DEBUG:boto:Method: GET 2023-10-11T15:21:59.128 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:21:59.128 DEBUG:boto:Data: 2023-10-11T15:21:59.129 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.129 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.130 DEBUG:boto:Params: {} 2023-10-11T15:21:59.130 DEBUG:boto:Token: None 2023-10-11T15:21:59.131 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj95 2023-10-11T15:21:59.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzADiyIrceuaq+HiwRHaIWDiP8w= 2023-10-11T15:21:59.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzADiyIrceuaq+HiwRHaIWDiP8w=', 'Content-Length': '0'} 2023-10-11T15:21:59.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008685268d0539a148-006526bd97-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.134 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.135 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.135 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.136 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.136 DEBUG:boto:Method: GET 2023-10-11T15:21:59.136 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.137 DEBUG:boto:Data: 2023-10-11T15:21:59.137 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:21:59.138 DEBUG:boto:Port: 8001 2023-10-11T15:21:59.139 DEBUG:boto:Params: {} 2023-10-11T15:21:59.139 DEBUG:boto:Token: None 2023-10-11T15:21:59.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I= 2023-10-11T15:21:59.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I=', 'Content-Length': '0'} 2023-10-11T15:21:59.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023a2c69498ac0c46-006526bd97-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.143 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:59.144 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.144 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.145 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.145 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.146 DEBUG:boto:Method: GET 2023-10-11T15:21:59.146 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:21:59.147 DEBUG:boto:Data: 2023-10-11T15:21:59.147 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.148 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.148 DEBUG:boto:Params: {} 2023-10-11T15:21:59.149 DEBUG:boto:Token: None 2023-10-11T15:21:59.149 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I= 2023-10-11T15:21:59.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nO1C9Fi9GsL+4d8/elTZkZFg11I=', 'Content-Length': '0'} 2023-10-11T15:21:59.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9843fe98d2a8e9-006526bd97-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.154 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:21:59.154 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:21:59.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:21:59.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:21:59.156 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:21:59.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:59.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:59.258 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:21:59.258 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:21:59.266 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 20 2023-10-11T15:21:59.314 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.314 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.315 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.315 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:59.316 DEBUG:boto:Data: 2023-10-11T15:21:59.316 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.317 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.318 DEBUG:boto:Params: {} 2023-10-11T15:21:59.318 DEBUG:boto:Token: None 2023-10-11T15:21:59.318 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8= 2023-10-11T15:21:59.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8=', 'Content-Length': '0'} 2023-10-11T15:21:59.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fa2b8dfe9a68009-006526bd97-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.324 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:21:59.324 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:21:59.324 DEBUG:boto:Method: PUT 2023-10-11T15:21:59.325 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:21:59.325 DEBUG:boto:Data: 2023-10-11T15:21:59.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:59.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.326 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.327 DEBUG:boto:Params: {} 2023-10-11T15:21:59.327 DEBUG:boto:Token: None 2023-10-11T15:21:59.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj29 2023-10-11T15:21:59.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lYPHxw/gcyq7PviQchsIy0pSEHk= 2023-10-11T15:21:59.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lYPHxw/gcyq7PviQchsIy0pSEHk='} 2023-10-11T15:21:59.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004767371f750c5d71-006526bd97-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.371 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.372 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.372 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.373 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:59.373 DEBUG:boto:Data: 2023-10-11T15:21:59.374 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.375 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.375 DEBUG:boto:Params: {} 2023-10-11T15:21:59.376 DEBUG:boto:Token: None 2023-10-11T15:21:59.376 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8= 2023-10-11T15:21:59.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8=', 'Content-Length': '0'} 2023-10-11T15:21:59.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000566a991c872b36f9-006526bd97-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.381 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:21:59.382 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:21:59.382 DEBUG:boto:Method: PUT 2023-10-11T15:21:59.383 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:21:59.385 DEBUG:boto:Data: 2023-10-11T15:21:59.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:59.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.386 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.387 DEBUG:boto:Params: {} 2023-10-11T15:21:59.387 DEBUG:boto:Token: None 2023-10-11T15:21:59.388 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj58 2023-10-11T15:21:59.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vDQbUWGjvgpDsGOmWxZquxV9KQI= 2023-10-11T15:21:59.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vDQbUWGjvgpDsGOmWxZquxV9KQI='} 2023-10-11T15:21:59.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e7ec8e39172ccc7-006526bd97-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.394 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.395 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.395 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.396 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:59.396 DEBUG:boto:Data: 2023-10-11T15:21:59.397 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.397 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.398 DEBUG:boto:Params: {} 2023-10-11T15:21:59.398 DEBUG:boto:Token: None 2023-10-11T15:21:59.399 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8= 2023-10-11T15:21:59.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8=', 'Content-Length': '0'} 2023-10-11T15:21:59.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee4af2c13be6b559-006526bd97-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.404 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:21:59.404 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:21:59.404 DEBUG:boto:Method: PUT 2023-10-11T15:21:59.405 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:21:59.405 DEBUG:boto:Data: 2023-10-11T15:21:59.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:59.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.407 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.407 DEBUG:boto:Params: {} 2023-10-11T15:21:59.408 DEBUG:boto:Token: None 2023-10-11T15:21:59.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj87 2023-10-11T15:21:59.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MyEWpzbNHhOR2g4DZRzSfsrNCl8= 2023-10-11T15:21:59.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MyEWpzbNHhOR2g4DZRzSfsrNCl8='} 2023-10-11T15:21:59.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000642e99a44e190468-006526bd97-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.459 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.459 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.460 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.460 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:59.461 DEBUG:boto:Data: 2023-10-11T15:21:59.461 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.462 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.462 DEBUG:boto:Params: {} 2023-10-11T15:21:59.463 DEBUG:boto:Token: None 2023-10-11T15:21:59.463 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8= 2023-10-11T15:21:59.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8=', 'Content-Length': '0'} 2023-10-11T15:21:59.467 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036849e099f354676-006526bd97-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.468 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:21:59.468 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:21:59.468 DEBUG:boto:Method: PUT 2023-10-11T15:21:59.469 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:21:59.469 DEBUG:boto:Data: 2023-10-11T15:21:59.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:59.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.471 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.471 DEBUG:boto:Params: {} 2023-10-11T15:21:59.471 DEBUG:boto:Token: None 2023-10-11T15:21:59.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj116 2023-10-11T15:21:59.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RXVpnK0tPDvPBPgwbIGD3k23hAY= 2023-10-11T15:21:59.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RXVpnK0tPDvPBPgwbIGD3k23hAY='} 2023-10-11T15:21:59.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003049cd8bb9864d3-006526bd97-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.478 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:21:59.479 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:21:59.479 DEBUG:boto:Method: HEAD 2023-10-11T15:21:59.480 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:21:59.480 DEBUG:boto:Data: 2023-10-11T15:21:59.480 DEBUG:boto:Headers: {} 2023-10-11T15:21:59.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.481 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.481 DEBUG:boto:Params: {} 2023-10-11T15:21:59.482 DEBUG:boto:Token: None 2023-10-11T15:21:59.482 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/ 2023-10-11T15:21:59.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8= 2023-10-11T15:21:59.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZV3fNdA6gQyM7rAZ20CP+m7oxG8=', 'Content-Length': '0'} 2023-10-11T15:21:59.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095e8c0386fa6df24-006526bd97-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.487 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:21:59.488 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:21:59.488 DEBUG:boto:Method: PUT 2023-10-11T15:21:59.488 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:21:59.489 DEBUG:boto:Data: 2023-10-11T15:21:59.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:21:59.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:21:59.490 DEBUG:boto:Port: 8000 2023-10-11T15:21:59.490 DEBUG:boto:Params: {} 2023-10-11T15:21:59.491 DEBUG:boto:Token: None 2023-10-11T15:21:59.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:21:59 GMT /dghqvd-59/obj145 2023-10-11T15:21:59.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LLNxATBMsGFIrZnYjaOmwilPpoA= 2023-10-11T15:21:59.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:21:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LLNxATBMsGFIrZnYjaOmwilPpoA='} 2023-10-11T15:21:59.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052a7a7a22f3fb634-006526bd97-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:21:59.535 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:21:59.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:59.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:21:59.537 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:21:59.593 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:21:59.593 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:21:59.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:21:59.625+0000 7fcdbacb5780 1 waiting to reach latest gen 4, on 3.. 2023-10-11T15:22:04.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:04.628+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:04.630 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:22:04.631 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:09.632 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:09.631+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:09.633 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:22:09.633 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:14.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:14.633+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:14.635 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:22:14.635 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:19.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:19.635+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:19.637 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:22:19.637 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:24.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:24.637+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:24.639 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:24.639 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:29.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:29.639+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:29.641 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:29.641 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:34.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:34.641+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:34.647 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:34.647 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:39.645 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:39.643+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:39.645 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:39.645 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:44.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:44.645+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:44.648 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:44.648 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:49.649 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:49.647+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:49.649 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:49.650 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:54.651 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:54.650+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:54.651 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:54.652 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:59.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:22:59.651+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:22:59.654 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:22:59.654 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:04.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:04.653+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:04.656 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:04.656 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:09.658 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:09.655+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:09.658 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:09.659 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:14.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:14.657+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:14.660 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:14.660 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:19.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:19.659+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:19.662 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:19.662 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:24.664 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:24.661+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:24.665 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:24.665 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:29.666 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:29.663+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:29.666 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:29.667 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:34.668 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:34.666+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:34.669 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:34.669 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:39.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:39.668+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:39.671 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:39.671 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:44.672 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:44.671+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:44.673 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:44.673 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:49.674 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:49.673+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:49.675 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:49.675 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:54.676 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:54.675+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:54.677 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:54.677 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:59.679 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:23:59.677+0000 7fcdbacb5780 1 waiting for incremental sync to catch up: 2023-10-11T15:23:59.679 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, , , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:23:59.679 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:24:04.681 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:04.679+0000 7fcdbacb5780 1 bucket sync caught up with source: 2023-10-11T15:24:04.681 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:24:04.681 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [00000000002.82.6, 00000000001.72.6, , , , , 00000000001.40.6, , , , , , , , , , , 00000000002.97.6, 00000000001.53.6, , , , , , , , , , ] 2023-10-11T15:24:04.682 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:04.679+0000 7fcdbacb5780 0 bucket checkpoint complete 2023-10-11T15:24:04.698 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:24:04.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:24:04.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:24:04.700 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:24:04.762 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:24:04.763 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:24:04.800 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:04.798+0000 7fa36b893780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:04.800 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000003.107.6, 00000000004.296.6, 00000000001.150.6, , 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:24:04.801 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000003.107.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:24:09.801 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:09.800+0000 7fa36b893780 1 bucket sync caught up with source: 2023-10-11T15:24:09.802 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000003.107.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:24:09.802 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000002.219.6, 00000000002.234.6, 00000000001.287.6, 00000000003.107.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000002.232.6, 00000000005.175.6, 00000000004.288.6] 2023-10-11T15:24:09.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:09.800+0000 7fa36b893780 0 bucket checkpoint complete 2023-10-11T15:24:09.818 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:24:09.819 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.819 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.820 DEBUG:boto:Method: HEAD 2023-10-11T15:24:09.820 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:09.821 DEBUG:boto:Data: 2023-10-11T15:24:09.821 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.822 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.822 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.822 DEBUG:boto:Params: {} 2023-10-11T15:24:09.823 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:24:09.824 DEBUG:boto:Token: None 2023-10-11T15:24:09.824 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/ 2023-10-11T15:24:09.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qwj6N4Nezq1mcQqiImUzioCdnl0= 2023-10-11T15:24:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qwj6N4Nezq1mcQqiImUzioCdnl0=', 'Content-Length': '0'} 2023-10-11T15:24:09.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018c1f1a566b7e562-006526be19-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.830 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.830 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.831 DEBUG:boto:Method: HEAD 2023-10-11T15:24:09.831 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:09.832 DEBUG:boto:Data: 2023-10-11T15:24:09.832 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.833 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.833 DEBUG:boto:Params: {} 2023-10-11T15:24:09.834 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:24:09.834 DEBUG:boto:Token: None 2023-10-11T15:24:09.835 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/ 2023-10-11T15:24:09.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qwj6N4Nezq1mcQqiImUzioCdnl0= 2023-10-11T15:24:09.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qwj6N4Nezq1mcQqiImUzioCdnl0=', 'Content-Length': '0'} 2023-10-11T15:24:09.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3419a8a630f5730-006526be19-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.845 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:24:09.845 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.845 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.846 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.846 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.847 DEBUG:boto:Method: GET 2023-10-11T15:24:09.847 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.848 DEBUG:boto:Data: 2023-10-11T15:24:09.848 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.849 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.849 DEBUG:boto:Params: {} 2023-10-11T15:24:09.850 DEBUG:boto:Token: None 2023-10-11T15:24:09.850 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/?versions 2023-10-11T15:24:09.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI= 2023-10-11T15:24:09.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI=', 'Content-Length': '0'} 2023-10-11T15:24:09.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09d1dead3bc5738-006526be19-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.854 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:09.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:24:09.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:24:09.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:24:09.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:24:09.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:24:09.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:24:09.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:24:09.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:24:09.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:24:09.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:24:09.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:24:09.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:24:09.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:24:09.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:24:09.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:24:09.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:24:09.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:24:09.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:24:09.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:24:09.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:24:09.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:24:09.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:24:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:24:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:24:09.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:24:09.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:24:09.866 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.867 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.867 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.868 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.868 DEBUG:boto:Method: GET 2023-10-11T15:24:09.868 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.869 DEBUG:boto:Data: 2023-10-11T15:24:09.869 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.870 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.871 DEBUG:boto:Params: {} 2023-10-11T15:24:09.871 DEBUG:boto:Token: None 2023-10-11T15:24:09.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/?versions 2023-10-11T15:24:09.872 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI= 2023-10-11T15:24:09.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI=', 'Content-Length': '0'} 2023-10-11T15:24:09.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088127c31cae791ec-006526be19-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.877 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:09.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:24:09.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:24:09.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:24:09.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:24:09.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:24:09.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:24:09.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:24:09.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:24:09.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:24:09.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:24:09.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:24:09.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:24:09.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:24:09.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:24:09.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:24:09.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:24:09.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:24:09.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:24:09.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:24:09.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:24:09.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:24:09.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:24:09.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:24:09.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:24:09.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:24:09.888 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.889 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.889 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.889 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.890 DEBUG:boto:Method: GET 2023-10-11T15:24:09.890 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.891 DEBUG:boto:Data: 2023-10-11T15:24:09.891 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.892 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.892 DEBUG:boto:Params: {} 2023-10-11T15:24:09.893 DEBUG:boto:Token: None 2023-10-11T15:24:09.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/?versions 2023-10-11T15:24:09.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI= 2023-10-11T15:24:09.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI=', 'Content-Length': '0'} 2023-10-11T15:24:09.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f15c0d11df4fc6e9-006526be19-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.898 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:09.899 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:09.900 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:09.900 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.901 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.901 DEBUG:boto:Method: GET 2023-10-11T15:24:09.902 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:24:09.902 DEBUG:boto:Data: 2023-10-11T15:24:09.902 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.903 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.904 DEBUG:boto:Params: {} 2023-10-11T15:24:09.904 DEBUG:boto:Token: None 2023-10-11T15:24:09.905 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/?versions 2023-10-11T15:24:09.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI= 2023-10-11T15:24:09.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pn+dQcOtGKLsQx1Wbl9frV18CI=', 'Content-Length': '0'} 2023-10-11T15:24:09.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c8c2b6c7abf7efb-006526be19-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.910 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:09.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:24:09.912 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.913 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.913 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.914 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.914 DEBUG:boto:Method: GET 2023-10-11T15:24:09.914 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.915 DEBUG:boto:Data: 2023-10-11T15:24:09.915 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.916 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.917 DEBUG:boto:Params: {} 2023-10-11T15:24:09.917 DEBUG:boto:Token: None 2023-10-11T15:24:09.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s= 2023-10-11T15:24:09.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s=', 'Content-Length': '0'} 2023-10-11T15:24:09.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55fb541276e1324-006526be19-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.922 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.922 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.923 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.923 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.924 DEBUG:boto:Method: GET 2023-10-11T15:24:09.924 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.924 DEBUG:boto:Data: 2023-10-11T15:24:09.925 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.926 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.926 DEBUG:boto:Params: {} 2023-10-11T15:24:09.927 DEBUG:boto:Token: None 2023-10-11T15:24:09.927 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s= 2023-10-11T15:24:09.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s=', 'Content-Length': '0'} 2023-10-11T15:24:09.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a47d4d81ad47fd-006526be19-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.931 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.932 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.932 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.932 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.933 DEBUG:boto:Method: HEAD 2023-10-11T15:24:09.933 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.934 DEBUG:boto:Data: 2023-10-11T15:24:09.934 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.935 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.935 DEBUG:boto:Params: {} 2023-10-11T15:24:09.936 DEBUG:boto:Token: None 2023-10-11T15:24:09.936 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.937 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ypYP2K5h4M1c4iUnX36wa3ckyAc= 2023-10-11T15:24:09.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ypYP2K5h4M1c4iUnX36wa3ckyAc=', 'Content-Length': '0'} 2023-10-11T15:24:09.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000102c9e1281c97b16-006526be19-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.940 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.940 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.941 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.941 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.942 DEBUG:boto:Method: HEAD 2023-10-11T15:24:09.942 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.942 DEBUG:boto:Data: 2023-10-11T15:24:09.943 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.944 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.944 DEBUG:boto:Params: {} 2023-10-11T15:24:09.944 DEBUG:boto:Token: None 2023-10-11T15:24:09.945 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ypYP2K5h4M1c4iUnX36wa3ckyAc= 2023-10-11T15:24:09.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ypYP2K5h4M1c4iUnX36wa3ckyAc=', 'Content-Length': '0'} 2023-10-11T15:24:09.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e51445459755eac-006526be19-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:24:09.949 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.950 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.950 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.950 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.951 DEBUG:boto:Method: GET 2023-10-11T15:24:09.951 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.952 DEBUG:boto:Data: 2023-10-11T15:24:09.952 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:09.953 DEBUG:boto:Port: 8001 2023-10-11T15:24:09.953 DEBUG:boto:Params: {} 2023-10-11T15:24:09.954 DEBUG:boto:Token: None 2023-10-11T15:24:09.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s= 2023-10-11T15:24:09.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s=', 'Content-Length': '0'} 2023-10-11T15:24:09.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0139bd42ea7eee-006526be19-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:09.958 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:09.959 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:09.959 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.960 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.960 DEBUG:boto:Method: GET 2023-10-11T15:24:09.960 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.961 DEBUG:boto:Data: 2023-10-11T15:24:09.961 DEBUG:boto:Headers: {} 2023-10-11T15:24:09.962 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:09.962 DEBUG:boto:Port: 8000 2023-10-11T15:24:09.963 DEBUG:boto:Params: {} 2023-10-11T15:24:09.963 DEBUG:boto:Token: None 2023-10-11T15:24:09.964 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:09 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:24:09.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s= 2023-10-11T15:24:09.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmMu2rwuNrOxsIwW7u8X89SI/7s=', 'Content-Length': '0'} 2023-10-11T15:24:09.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff69a132e224fa3-006526be19-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.010 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.011 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.011 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.012 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.012 DEBUG:boto:Method: GET 2023-10-11T15:24:10.013 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.013 DEBUG:boto:Data: 2023-10-11T15:24:10.014 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.014 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.015 DEBUG:boto:Params: {} 2023-10-11T15:24:10.016 DEBUG:boto:Token: None 2023-10-11T15:24:10.016 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f4df5f61167b585-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.020 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.021 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.022 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.022 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.023 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.023 DEBUG:boto:Method: GET 2023-10-11T15:24:10.024 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.024 DEBUG:boto:Data: 2023-10-11T15:24:10.024 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.025 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.025 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.026 DEBUG:boto:Params: {} 2023-10-11T15:24:10.026 DEBUG:boto:Token: None 2023-10-11T15:24:10.027 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.027 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70240c96dd195d6-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.033 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.034 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:10.034 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:10.034 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.035 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:24:10.035 DEBUG:boto:Data: 2023-10-11T15:24:10.036 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.037 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.037 DEBUG:boto:Params: {} 2023-10-11T15:24:10.038 DEBUG:boto:Token: None 2023-10-11T15:24:10.041 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj11 2023-10-11T15:24:10.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iLCKsZHtoO3Ww4rWsIo1UiChrMk= 2023-10-11T15:24:10.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iLCKsZHtoO3Ww4rWsIo1UiChrMk=', 'Content-Length': '0'} 2023-10-11T15:24:10.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5121cad7bebc929-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.047 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:10.048 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:10.048 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.048 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:24:10.049 DEBUG:boto:Data: 2023-10-11T15:24:10.049 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.050 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.051 DEBUG:boto:Params: {} 2023-10-11T15:24:10.051 DEBUG:boto:Token: None 2023-10-11T15:24:10.052 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj11 2023-10-11T15:24:10.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iLCKsZHtoO3Ww4rWsIo1UiChrMk= 2023-10-11T15:24:10.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iLCKsZHtoO3Ww4rWsIo1UiChrMk=', 'Content-Length': '0'} 2023-10-11T15:24:10.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb2311f30a6b642-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:24:10.056 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:10.056 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:10.057 DEBUG:boto:Method: GET 2023-10-11T15:24:10.057 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:24:10.057 DEBUG:boto:Data: 2023-10-11T15:24:10.058 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.059 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.059 DEBUG:boto:Params: {} 2023-10-11T15:24:10.060 DEBUG:boto:Token: None 2023-10-11T15:24:10.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj11 2023-10-11T15:24:10.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8Nv/gbSavKUZBOGEoHrIL3slkE= 2023-10-11T15:24:10.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8Nv/gbSavKUZBOGEoHrIL3slkE=', 'Content-Length': '0'} 2023-10-11T15:24:10.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d8b5285d580ccb0-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.065 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:24:10.065 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:24:10.065 DEBUG:boto:Method: GET 2023-10-11T15:24:10.066 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:24:10.066 DEBUG:boto:Data: 2023-10-11T15:24:10.067 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.068 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.068 DEBUG:boto:Params: {} 2023-10-11T15:24:10.069 DEBUG:boto:Token: None 2023-10-11T15:24:10.069 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj11 2023-10-11T15:24:10.070 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8Nv/gbSavKUZBOGEoHrIL3slkE= 2023-10-11T15:24:10.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8Nv/gbSavKUZBOGEoHrIL3slkE=', 'Content-Length': '0'} 2023-10-11T15:24:10.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df13260bc80498c6-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.114 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.115 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.116 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.116 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.117 DEBUG:boto:Method: GET 2023-10-11T15:24:10.118 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.118 DEBUG:boto:Data: 2023-10-11T15:24:10.119 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.119 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.120 DEBUG:boto:Params: {} 2023-10-11T15:24:10.120 DEBUG:boto:Token: None 2023-10-11T15:24:10.121 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a3279a246d7681f-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.125 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.126 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.127 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.128 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.128 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.129 DEBUG:boto:Method: GET 2023-10-11T15:24:10.129 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:24:10.129 DEBUG:boto:Data: 2023-10-11T15:24:10.130 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.131 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.131 DEBUG:boto:Params: {} 2023-10-11T15:24:10.132 DEBUG:boto:Token: None 2023-10-11T15:24:10.132 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.133 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be5e7e59e1fb6479-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.137 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:24:10.139 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.139 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.140 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.140 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.140 DEBUG:boto:Method: GET 2023-10-11T15:24:10.141 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.141 DEBUG:boto:Data: 2023-10-11T15:24:10.142 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.143 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.143 DEBUG:boto:Params: {} 2023-10-11T15:24:10.143 DEBUG:boto:Token: None 2023-10-11T15:24:10.144 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU= 2023-10-11T15:24:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU=', 'Content-Length': '0'} 2023-10-11T15:24:10.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2fd63b5674bd3c-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.148 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.148 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.149 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.149 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.149 DEBUG:boto:Method: GET 2023-10-11T15:24:10.150 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.150 DEBUG:boto:Data: 2023-10-11T15:24:10.151 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.152 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.152 DEBUG:boto:Params: {} 2023-10-11T15:24:10.153 DEBUG:boto:Token: None 2023-10-11T15:24:10.153 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.154 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU= 2023-10-11T15:24:10.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU=', 'Content-Length': '0'} 2023-10-11T15:24:10.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe22599a89fcc7d-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.157 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.157 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.158 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.158 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.158 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.159 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.159 DEBUG:boto:Data: 2023-10-11T15:24:10.160 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.160 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.161 DEBUG:boto:Params: {} 2023-10-11T15:24:10.161 DEBUG:boto:Token: None 2023-10-11T15:24:10.162 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rA39OM9f7VzOYKaU3Ct7Jy//F6g= 2023-10-11T15:24:10.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rA39OM9f7VzOYKaU3Ct7Jy//F6g=', 'Content-Length': '0'} 2023-10-11T15:24:10.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e292ba0e93f0e3-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.165 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.166 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.166 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.167 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.167 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.168 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.168 DEBUG:boto:Data: 2023-10-11T15:24:10.169 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.169 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.170 DEBUG:boto:Params: {} 2023-10-11T15:24:10.170 DEBUG:boto:Token: None 2023-10-11T15:24:10.171 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rA39OM9f7VzOYKaU3Ct7Jy//F6g= 2023-10-11T15:24:10.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rA39OM9f7VzOYKaU3Ct7Jy//F6g=', 'Content-Length': '0'} 2023-10-11T15:24:10.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003800116038ed799-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:24:10.175 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.175 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.176 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.176 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.176 DEBUG:boto:Method: GET 2023-10-11T15:24:10.177 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.177 DEBUG:boto:Data: 2023-10-11T15:24:10.178 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.178 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.179 DEBUG:boto:Params: {} 2023-10-11T15:24:10.179 DEBUG:boto:Token: None 2023-10-11T15:24:10.180 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU= 2023-10-11T15:24:10.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU=', 'Content-Length': '0'} 2023-10-11T15:24:10.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa761094d0c588d-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.226 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.227 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.227 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.228 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.228 DEBUG:boto:Method: GET 2023-10-11T15:24:10.229 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.229 DEBUG:boto:Data: 2023-10-11T15:24:10.230 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.230 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.231 DEBUG:boto:Params: {} 2023-10-11T15:24:10.231 DEBUG:boto:Token: None 2023-10-11T15:24:10.232 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:24:10.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU= 2023-10-11T15:24:10.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6dJjt5aWzS+Y75avrtZ+ZROc7sU=', 'Content-Length': '0'} 2023-10-11T15:24:10.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83ce45f2a7a5066-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.236 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.236 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.237 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.237 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.237 DEBUG:boto:Method: GET 2023-10-11T15:24:10.238 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.238 DEBUG:boto:Data: 2023-10-11T15:24:10.239 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.239 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.239 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.240 DEBUG:boto:Params: {} 2023-10-11T15:24:10.240 DEBUG:boto:Token: None 2023-10-11T15:24:10.241 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.241 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009afdbf2d262ff62d-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.244 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.245 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.246 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.246 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.247 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.247 DEBUG:boto:Method: GET 2023-10-11T15:24:10.247 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.248 DEBUG:boto:Data: 2023-10-11T15:24:10.248 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.249 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.249 DEBUG:boto:Params: {} 2023-10-11T15:24:10.250 DEBUG:boto:Token: None 2023-10-11T15:24:10.250 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e971ba38ad2f860-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.302 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.303 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.304 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.304 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.305 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:24:10.305 DEBUG:boto:Data: 2023-10-11T15:24:10.305 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.306 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.307 DEBUG:boto:Params: {} 2023-10-11T15:24:10.307 DEBUG:boto:Token: None 2023-10-11T15:24:10.308 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115 2023-10-11T15:24:10.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rnFCj/+3VlvvpE+1V1m3viO91Rs= 2023-10-11T15:24:10.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rnFCj/+3VlvvpE+1V1m3viO91Rs=', 'Content-Length': '0'} 2023-10-11T15:24:10.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4db97e2865d866-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.311 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.312 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.312 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.312 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:24:10.313 DEBUG:boto:Data: 2023-10-11T15:24:10.313 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.314 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.315 DEBUG:boto:Params: {} 2023-10-11T15:24:10.315 DEBUG:boto:Token: None 2023-10-11T15:24:10.315 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115 2023-10-11T15:24:10.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rnFCj/+3VlvvpE+1V1m3viO91Rs= 2023-10-11T15:24:10.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rnFCj/+3VlvvpE+1V1m3viO91Rs=', 'Content-Length': '0'} 2023-10-11T15:24:10.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080031bb798173864-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:24:10.319 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.320 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.320 DEBUG:boto:Method: GET 2023-10-11T15:24:10.320 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:24:10.321 DEBUG:boto:Data: 2023-10-11T15:24:10.321 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.322 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.323 DEBUG:boto:Params: {} 2023-10-11T15:24:10.323 DEBUG:boto:Token: None 2023-10-11T15:24:10.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115 2023-10-11T15:24:10.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v3NYnyaBQaIcdlIUnH76+w0CcRs= 2023-10-11T15:24:10.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v3NYnyaBQaIcdlIUnH76+w0CcRs=', 'Content-Length': '0'} 2023-10-11T15:24:10.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928ab3fd563521a2-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.328 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:24:10.328 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:24:10.328 DEBUG:boto:Method: GET 2023-10-11T15:24:10.329 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:24:10.329 DEBUG:boto:Data: 2023-10-11T15:24:10.330 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.331 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.331 DEBUG:boto:Params: {} 2023-10-11T15:24:10.332 DEBUG:boto:Token: None 2023-10-11T15:24:10.332 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj115 2023-10-11T15:24:10.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v3NYnyaBQaIcdlIUnH76+w0CcRs= 2023-10-11T15:24:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v3NYnyaBQaIcdlIUnH76+w0CcRs=', 'Content-Length': '0'} 2023-10-11T15:24:10.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000086f327556f6bf1e-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.336 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.336 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.336 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.337 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.337 DEBUG:boto:Method: GET 2023-10-11T15:24:10.338 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.338 DEBUG:boto:Data: 2023-10-11T15:24:10.339 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.340 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.340 DEBUG:boto:Params: {} 2023-10-11T15:24:10.340 DEBUG:boto:Token: None 2023-10-11T15:24:10.341 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027d1dac5659cf9bb-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.390 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.391 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.392 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.392 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.392 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.393 DEBUG:boto:Method: GET 2023-10-11T15:24:10.393 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:24:10.394 DEBUG:boto:Data: 2023-10-11T15:24:10.394 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.395 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.396 DEBUG:boto:Params: {} 2023-10-11T15:24:10.396 DEBUG:boto:Token: None 2023-10-11T15:24:10.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.397 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd1950e31db9a203-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.402 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:24:10.403 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.403 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.404 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.404 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.405 DEBUG:boto:Method: GET 2023-10-11T15:24:10.405 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.406 DEBUG:boto:Data: 2023-10-11T15:24:10.406 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.407 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.407 DEBUG:boto:Params: {} 2023-10-11T15:24:10.408 DEBUG:boto:Token: None 2023-10-11T15:24:10.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ= 2023-10-11T15:24:10.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ=', 'Content-Length': '0'} 2023-10-11T15:24:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c1cb974f33631f5-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.412 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.413 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.413 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.414 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.414 DEBUG:boto:Method: GET 2023-10-11T15:24:10.414 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.415 DEBUG:boto:Data: 2023-10-11T15:24:10.415 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.416 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.417 DEBUG:boto:Params: {} 2023-10-11T15:24:10.417 DEBUG:boto:Token: None 2023-10-11T15:24:10.418 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ= 2023-10-11T15:24:10.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ=', 'Content-Length': '0'} 2023-10-11T15:24:10.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd664e7c99d3b8d-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.421 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.422 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.422 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.423 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.423 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.424 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.424 DEBUG:boto:Data: 2023-10-11T15:24:10.424 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.425 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.426 DEBUG:boto:Params: {} 2023-10-11T15:24:10.426 DEBUG:boto:Token: None 2023-10-11T15:24:10.427 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KZc+SknwNErSJHL7RSW2Al4bmsY= 2023-10-11T15:24:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KZc+SknwNErSJHL7RSW2Al4bmsY=', 'Content-Length': '0'} 2023-10-11T15:24:10.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ea4df5639f85b2-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.431 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.432 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.433 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.433 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.434 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.434 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.435 DEBUG:boto:Data: 2023-10-11T15:24:10.435 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.436 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.436 DEBUG:boto:Params: {} 2023-10-11T15:24:10.437 DEBUG:boto:Token: None 2023-10-11T15:24:10.437 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KZc+SknwNErSJHL7RSW2Al4bmsY= 2023-10-11T15:24:10.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KZc+SknwNErSJHL7RSW2Al4bmsY=', 'Content-Length': '0'} 2023-10-11T15:24:10.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb97a058d0e4b256-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:24:10.441 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.442 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.442 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.442 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.443 DEBUG:boto:Method: GET 2023-10-11T15:24:10.443 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.444 DEBUG:boto:Data: 2023-10-11T15:24:10.444 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.445 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.446 DEBUG:boto:Params: {} 2023-10-11T15:24:10.446 DEBUG:boto:Token: None 2023-10-11T15:24:10.447 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ= 2023-10-11T15:24:10.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ=', 'Content-Length': '0'} 2023-10-11T15:24:10.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a37581dc616dd27-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.454 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.455 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.455 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.456 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.456 DEBUG:boto:Method: GET 2023-10-11T15:24:10.456 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.457 DEBUG:boto:Data: 2023-10-11T15:24:10.457 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.458 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.459 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.459 DEBUG:boto:Params: {} 2023-10-11T15:24:10.460 DEBUG:boto:Token: None 2023-10-11T15:24:10.460 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:24:10.461 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ= 2023-10-11T15:24:10.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2eDFGlt7npAqvzT2RHy19rvS8zQ=', 'Content-Length': '0'} 2023-10-11T15:24:10.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008406a82311f8d615-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.464 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.464 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.465 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.465 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.465 DEBUG:boto:Method: GET 2023-10-11T15:24:10.466 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.466 DEBUG:boto:Data: 2023-10-11T15:24:10.467 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.467 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.468 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.468 DEBUG:boto:Params: {} 2023-10-11T15:24:10.469 DEBUG:boto:Token: None 2023-10-11T15:24:10.469 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d15c603ac9f5694b-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.474 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.475 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.475 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.476 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.476 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.476 DEBUG:boto:Method: GET 2023-10-11T15:24:10.477 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.477 DEBUG:boto:Data: 2023-10-11T15:24:10.478 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.479 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.479 DEBUG:boto:Params: {} 2023-10-11T15:24:10.479 DEBUG:boto:Token: None 2023-10-11T15:24:10.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3a88217c95d2f8-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.526 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.528 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.528 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.529 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.529 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:24:10.529 DEBUG:boto:Data: 2023-10-11T15:24:10.530 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.531 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.531 DEBUG:boto:Params: {} 2023-10-11T15:24:10.532 DEBUG:boto:Token: None 2023-10-11T15:24:10.532 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116 2023-10-11T15:24:10.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i53fqtRQTRAVhQ/JqtrqntohQ6M= 2023-10-11T15:24:10.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i53fqtRQTRAVhQ/JqtrqntohQ6M=', 'Content-Length': '0'} 2023-10-11T15:24:10.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065220dd028eb1e1a-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.536 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.536 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.537 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.537 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:24:10.538 DEBUG:boto:Data: 2023-10-11T15:24:10.538 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.540 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.541 DEBUG:boto:Params: {} 2023-10-11T15:24:10.541 DEBUG:boto:Token: None 2023-10-11T15:24:10.542 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116 2023-10-11T15:24:10.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i53fqtRQTRAVhQ/JqtrqntohQ6M= 2023-10-11T15:24:10.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i53fqtRQTRAVhQ/JqtrqntohQ6M=', 'Content-Length': '0'} 2023-10-11T15:24:10.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b912f7ca346ca30-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:24:10.547 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.548 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.548 DEBUG:boto:Method: GET 2023-10-11T15:24:10.549 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:24:10.549 DEBUG:boto:Data: 2023-10-11T15:24:10.549 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.550 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.550 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.551 DEBUG:boto:Params: {} 2023-10-11T15:24:10.551 DEBUG:boto:Token: None 2023-10-11T15:24:10.552 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116 2023-10-11T15:24:10.552 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0wGPs5LA+/3oRVxGFZtxVA0xQo= 2023-10-11T15:24:10.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0wGPs5LA+/3oRVxGFZtxVA0xQo=', 'Content-Length': '0'} 2023-10-11T15:24:10.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1352201e9c59d89-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.598 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:24:10.599 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:24:10.599 DEBUG:boto:Method: GET 2023-10-11T15:24:10.600 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:24:10.600 DEBUG:boto:Data: 2023-10-11T15:24:10.601 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.602 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.602 DEBUG:boto:Params: {} 2023-10-11T15:24:10.602 DEBUG:boto:Token: None 2023-10-11T15:24:10.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj116 2023-10-11T15:24:10.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0wGPs5LA+/3oRVxGFZtxVA0xQo= 2023-10-11T15:24:10.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0wGPs5LA+/3oRVxGFZtxVA0xQo=', 'Content-Length': '0'} 2023-10-11T15:24:10.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3888f1086b235bc-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.607 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.607 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.608 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.609 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.609 DEBUG:boto:Method: GET 2023-10-11T15:24:10.610 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.610 DEBUG:boto:Data: 2023-10-11T15:24:10.611 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.612 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.612 DEBUG:boto:Params: {} 2023-10-11T15:24:10.612 DEBUG:boto:Token: None 2023-10-11T15:24:10.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065a21ca05583d6bd-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.617 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.618 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.618 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.619 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.619 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.620 DEBUG:boto:Method: GET 2023-10-11T15:24:10.620 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:24:10.621 DEBUG:boto:Data: 2023-10-11T15:24:10.621 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.622 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.622 DEBUG:boto:Params: {} 2023-10-11T15:24:10.623 DEBUG:boto:Token: None 2023-10-11T15:24:10.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c63765bba52c05e-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.628 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:24:10.629 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.629 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.630 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.631 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.631 DEBUG:boto:Method: GET 2023-10-11T15:24:10.632 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.632 DEBUG:boto:Data: 2023-10-11T15:24:10.632 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.633 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.633 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.634 DEBUG:boto:Params: {} 2023-10-11T15:24:10.634 DEBUG:boto:Token: None 2023-10-11T15:24:10.635 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y= 2023-10-11T15:24:10.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y=', 'Content-Length': '0'} 2023-10-11T15:24:10.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3c9cef37d7c19f-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.638 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.639 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.639 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.640 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.640 DEBUG:boto:Method: GET 2023-10-11T15:24:10.641 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.641 DEBUG:boto:Data: 2023-10-11T15:24:10.642 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.643 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.643 DEBUG:boto:Params: {} 2023-10-11T15:24:10.643 DEBUG:boto:Token: None 2023-10-11T15:24:10.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y= 2023-10-11T15:24:10.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y=', 'Content-Length': '0'} 2023-10-11T15:24:10.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e88101ccb939eb3c-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.648 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.648 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.648 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.649 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.649 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.650 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.650 DEBUG:boto:Data: 2023-10-11T15:24:10.651 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.652 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.652 DEBUG:boto:Params: {} 2023-10-11T15:24:10.653 DEBUG:boto:Token: None 2023-10-11T15:24:10.653 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0X6cU6lFcweBV4EP7dGewc8OVc= 2023-10-11T15:24:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0X6cU6lFcweBV4EP7dGewc8OVc=', 'Content-Length': '0'} 2023-10-11T15:24:10.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd5f821e9d68166-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.658 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.658 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.659 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.659 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.659 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.660 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.660 DEBUG:boto:Data: 2023-10-11T15:24:10.661 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.662 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.662 DEBUG:boto:Params: {} 2023-10-11T15:24:10.663 DEBUG:boto:Token: None 2023-10-11T15:24:10.664 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0X6cU6lFcweBV4EP7dGewc8OVc= 2023-10-11T15:24:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0X6cU6lFcweBV4EP7dGewc8OVc=', 'Content-Length': '0'} 2023-10-11T15:24:10.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b03f71d56bbffa-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:24:10.669 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.670 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.671 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.671 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.672 DEBUG:boto:Method: GET 2023-10-11T15:24:10.673 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.673 DEBUG:boto:Data: 2023-10-11T15:24:10.674 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.675 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.676 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.676 DEBUG:boto:Params: {} 2023-10-11T15:24:10.677 DEBUG:boto:Token: None 2023-10-11T15:24:10.678 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y= 2023-10-11T15:24:10.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y=', 'Content-Length': '0'} 2023-10-11T15:24:10.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e90f736846895541-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.683 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.684 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.685 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.685 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.686 DEBUG:boto:Method: GET 2023-10-11T15:24:10.686 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.687 DEBUG:boto:Data: 2023-10-11T15:24:10.687 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.688 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.688 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.688 DEBUG:boto:Params: {} 2023-10-11T15:24:10.689 DEBUG:boto:Token: None 2023-10-11T15:24:10.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:24:10.690 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y= 2023-10-11T15:24:10.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QhWsIC2jmD40EOOUN+odeuAbH6Y=', 'Content-Length': '0'} 2023-10-11T15:24:10.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009913fd797a7eda2e-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.738 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.739 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.739 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.740 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.740 DEBUG:boto:Method: GET 2023-10-11T15:24:10.741 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.741 DEBUG:boto:Data: 2023-10-11T15:24:10.741 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.742 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.743 DEBUG:boto:Params: {} 2023-10-11T15:24:10.743 DEBUG:boto:Token: None 2023-10-11T15:24:10.744 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024705ccb07f75c5a-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.748 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.748 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.749 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.749 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.750 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.750 DEBUG:boto:Method: GET 2023-10-11T15:24:10.750 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.751 DEBUG:boto:Data: 2023-10-11T15:24:10.751 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.752 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.753 DEBUG:boto:Params: {} 2023-10-11T15:24:10.753 DEBUG:boto:Token: None 2023-10-11T15:24:10.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab96fbaddc7bc96-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.758 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.759 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.760 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.760 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.760 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:24:10.761 DEBUG:boto:Data: 2023-10-11T15:24:10.761 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.762 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.762 DEBUG:boto:Params: {} 2023-10-11T15:24:10.763 DEBUG:boto:Token: None 2023-10-11T15:24:10.763 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145 2023-10-11T15:24:10.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:02AduBy6c4HgfkZNGTHlRi3NFdY= 2023-10-11T15:24:10.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:02AduBy6c4HgfkZNGTHlRi3NFdY=', 'Content-Length': '0'} 2023-10-11T15:24:10.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4605cfc139ec6f-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.767 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.767 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.767 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.768 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:24:10.768 DEBUG:boto:Data: 2023-10-11T15:24:10.769 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.769 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.770 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.770 DEBUG:boto:Params: {} 2023-10-11T15:24:10.770 DEBUG:boto:Token: None 2023-10-11T15:24:10.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145 2023-10-11T15:24:10.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:02AduBy6c4HgfkZNGTHlRi3NFdY= 2023-10-11T15:24:10.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:02AduBy6c4HgfkZNGTHlRi3NFdY=', 'Content-Length': '0'} 2023-10-11T15:24:10.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aacef1ca458efec8-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:24:10.775 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.775 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.776 DEBUG:boto:Method: GET 2023-10-11T15:24:10.776 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:24:10.776 DEBUG:boto:Data: 2023-10-11T15:24:10.777 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.778 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.778 DEBUG:boto:Params: {} 2023-10-11T15:24:10.779 DEBUG:boto:Token: None 2023-10-11T15:24:10.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145 2023-10-11T15:24:10.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4OsKUeDCtS3VAPmk4+xPnoxzcT4= 2023-10-11T15:24:10.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4OsKUeDCtS3VAPmk4+xPnoxzcT4=', 'Content-Length': '0'} 2023-10-11T15:24:10.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c08bceb335d9bb-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.783 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:24:10.783 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:24:10.784 DEBUG:boto:Method: GET 2023-10-11T15:24:10.784 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:24:10.785 DEBUG:boto:Data: 2023-10-11T15:24:10.785 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.786 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.786 DEBUG:boto:Params: {} 2023-10-11T15:24:10.787 DEBUG:boto:Token: None 2023-10-11T15:24:10.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj145 2023-10-11T15:24:10.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4OsKUeDCtS3VAPmk4+xPnoxzcT4= 2023-10-11T15:24:10.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4OsKUeDCtS3VAPmk4+xPnoxzcT4=', 'Content-Length': '0'} 2023-10-11T15:24:10.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a61dd2c94790741-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.791 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.791 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.792 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.792 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.793 DEBUG:boto:Method: GET 2023-10-11T15:24:10.793 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.793 DEBUG:boto:Data: 2023-10-11T15:24:10.794 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.795 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.795 DEBUG:boto:Params: {} 2023-10-11T15:24:10.796 DEBUG:boto:Token: None 2023-10-11T15:24:10.796 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df13565560c82a85-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.846 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.847 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.848 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.848 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.849 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.849 DEBUG:boto:Method: GET 2023-10-11T15:24:10.849 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:24:10.850 DEBUG:boto:Data: 2023-10-11T15:24:10.850 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.851 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.852 DEBUG:boto:Params: {} 2023-10-11T15:24:10.852 DEBUG:boto:Token: None 2023-10-11T15:24:10.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfbad47ac6bfd288-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.858 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:24:10.859 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.859 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.860 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.860 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.861 DEBUG:boto:Method: GET 2023-10-11T15:24:10.861 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.862 DEBUG:boto:Data: 2023-10-11T15:24:10.862 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.863 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.865 DEBUG:boto:Params: {} 2023-10-11T15:24:10.866 DEBUG:boto:Token: None 2023-10-11T15:24:10.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk= 2023-10-11T15:24:10.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk=', 'Content-Length': '0'} 2023-10-11T15:24:10.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5c144b3b02fdb0-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.870 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.871 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.871 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.871 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.872 DEBUG:boto:Method: GET 2023-10-11T15:24:10.872 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.873 DEBUG:boto:Data: 2023-10-11T15:24:10.873 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.874 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.874 DEBUG:boto:Params: {} 2023-10-11T15:24:10.875 DEBUG:boto:Token: None 2023-10-11T15:24:10.875 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk= 2023-10-11T15:24:10.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk=', 'Content-Length': '0'} 2023-10-11T15:24:10.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf5ed2b8d75880e-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.879 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.880 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.880 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.880 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.881 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.881 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.882 DEBUG:boto:Data: 2023-10-11T15:24:10.882 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.883 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.883 DEBUG:boto:Params: {} 2023-10-11T15:24:10.884 DEBUG:boto:Token: None 2023-10-11T15:24:10.884 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UHRUnSboI2EzOZyxCzt3tGlctDE= 2023-10-11T15:24:10.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UHRUnSboI2EzOZyxCzt3tGlctDE=', 'Content-Length': '0'} 2023-10-11T15:24:10.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd55cb5541dca8ad-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.888 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.888 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.889 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.889 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.889 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.890 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.890 DEBUG:boto:Data: 2023-10-11T15:24:10.891 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.891 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.892 DEBUG:boto:Params: {} 2023-10-11T15:24:10.892 DEBUG:boto:Token: None 2023-10-11T15:24:10.893 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UHRUnSboI2EzOZyxCzt3tGlctDE= 2023-10-11T15:24:10.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UHRUnSboI2EzOZyxCzt3tGlctDE=', 'Content-Length': '0'} 2023-10-11T15:24:10.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a6fe2aa055e2ba-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:24:10.897 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.897 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.898 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.898 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.899 DEBUG:boto:Method: GET 2023-10-11T15:24:10.899 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.900 DEBUG:boto:Data: 2023-10-11T15:24:10.900 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.901 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.901 DEBUG:boto:Params: {} 2023-10-11T15:24:10.902 DEBUG:boto:Token: None 2023-10-11T15:24:10.903 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk= 2023-10-11T15:24:10.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk=', 'Content-Length': '0'} 2023-10-11T15:24:10.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077cf4b55357a4a5a-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.907 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.907 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.907 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.908 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.908 DEBUG:boto:Method: GET 2023-10-11T15:24:10.909 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.909 DEBUG:boto:Data: 2023-10-11T15:24:10.910 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.910 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.911 DEBUG:boto:Params: {} 2023-10-11T15:24:10.911 DEBUG:boto:Token: None 2023-10-11T15:24:10.912 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:24:10.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk= 2023-10-11T15:24:10.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tnnm7CblmjoSafZWi0L5DSwMPQk=', 'Content-Length': '0'} 2023-10-11T15:24:10.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f848ccf50603aa4-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.915 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.916 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.916 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.917 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.917 DEBUG:boto:Method: GET 2023-10-11T15:24:10.918 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.918 DEBUG:boto:Data: 2023-10-11T15:24:10.918 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.919 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.920 DEBUG:boto:Params: {} 2023-10-11T15:24:10.920 DEBUG:boto:Token: None 2023-10-11T15:24:10.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000176c89b3cfd743b5-006526be1a-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.925 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.926 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:10.926 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:10.927 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.927 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.927 DEBUG:boto:Method: GET 2023-10-11T15:24:10.928 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:24:10.928 DEBUG:boto:Data: 2023-10-11T15:24:10.929 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.930 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.930 DEBUG:boto:Params: {} 2023-10-11T15:24:10.931 DEBUG:boto:Token: None 2023-10-11T15:24:10.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/ 2023-10-11T15:24:10.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k= 2023-10-11T15:24:10.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Txw6fbhCG2NUlC2IGw5OlFrw7k=', 'Content-Length': '0'} 2023-10-11T15:24:10.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a96c4c276461d4c-006526be1a-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.978 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:10.979 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.979 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.980 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.980 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:24:10.981 DEBUG:boto:Data: 2023-10-11T15:24:10.981 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.982 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.982 DEBUG:boto:Params: {} 2023-10-11T15:24:10.983 DEBUG:boto:Token: None 2023-10-11T15:24:10.983 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17 2023-10-11T15:24:10.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4owefJArBwWFMPaSB/BBWZBGCEI= 2023-10-11T15:24:10.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4owefJArBwWFMPaSB/BBWZBGCEI=', 'Content-Length': '0'} 2023-10-11T15:24:10.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e82cc4700592b88-006526be1a-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.987 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.987 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.988 DEBUG:boto:Method: HEAD 2023-10-11T15:24:10.988 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:24:10.989 DEBUG:boto:Data: 2023-10-11T15:24:10.989 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:10.990 DEBUG:boto:Port: 8000 2023-10-11T15:24:10.990 DEBUG:boto:Params: {} 2023-10-11T15:24:10.991 DEBUG:boto:Token: None 2023-10-11T15:24:10.991 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17 2023-10-11T15:24:10.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4owefJArBwWFMPaSB/BBWZBGCEI= 2023-10-11T15:24:10.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4owefJArBwWFMPaSB/BBWZBGCEI=', 'Content-Length': '0'} 2023-10-11T15:24:10.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afe71a8f9b67dd99-006526be1a-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:10.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:24:10.995 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:10.995 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:10.996 DEBUG:boto:Method: GET 2023-10-11T15:24:10.996 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:24:10.997 DEBUG:boto:Data: 2023-10-11T15:24:10.997 DEBUG:boto:Headers: {} 2023-10-11T15:24:10.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:10.998 DEBUG:boto:Port: 8001 2023-10-11T15:24:10.998 DEBUG:boto:Params: {} 2023-10-11T15:24:10.999 DEBUG:boto:Token: None 2023-10-11T15:24:10.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:10 GMT /dghqvd-59/obj17 2023-10-11T15:24:11.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uIdNC7W23yImubwxii/y81enBg8= 2023-10-11T15:24:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uIdNC7W23yImubwxii/y81enBg8=', 'Content-Length': '0'} 2023-10-11T15:24:11.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ebe72c4d8066c0-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.046 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:24:11.047 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:24:11.048 DEBUG:boto:Method: GET 2023-10-11T15:24:11.048 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:24:11.048 DEBUG:boto:Data: 2023-10-11T15:24:11.049 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.050 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.050 DEBUG:boto:Params: {} 2023-10-11T15:24:11.051 DEBUG:boto:Token: None 2023-10-11T15:24:11.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj17 2023-10-11T15:24:11.052 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AU56ORrT9n4VzHowZ6uleiLVOiw= 2023-10-11T15:24:11.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AU56ORrT9n4VzHowZ6uleiLVOiw=', 'Content-Length': '0'} 2023-10-11T15:24:11.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006241a7c87fd7c67c-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.055 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.056 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.056 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.057 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.057 DEBUG:boto:Method: GET 2023-10-11T15:24:11.058 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.058 DEBUG:boto:Data: 2023-10-11T15:24:11.058 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.059 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.060 DEBUG:boto:Params: {} 2023-10-11T15:24:11.060 DEBUG:boto:Token: None 2023-10-11T15:24:11.061 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3357f62ada8fefb-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.065 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.065 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.066 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.067 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.067 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.068 DEBUG:boto:Method: GET 2023-10-11T15:24:11.068 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:24:11.068 DEBUG:boto:Data: 2023-10-11T15:24:11.069 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.070 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.070 DEBUG:boto:Params: {} 2023-10-11T15:24:11.071 DEBUG:boto:Token: None 2023-10-11T15:24:11.071 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee14f1088968cc4d-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.077 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:24:11.078 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.079 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.079 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.080 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.080 DEBUG:boto:Method: GET 2023-10-11T15:24:11.081 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.081 DEBUG:boto:Data: 2023-10-11T15:24:11.082 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.083 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.083 DEBUG:boto:Params: {} 2023-10-11T15:24:11.084 DEBUG:boto:Token: None 2023-10-11T15:24:11.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY= 2023-10-11T15:24:11.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY=', 'Content-Length': '0'} 2023-10-11T15:24:11.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002290900df78174cc-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.088 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.088 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.089 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.089 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.090 DEBUG:boto:Method: GET 2023-10-11T15:24:11.090 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.091 DEBUG:boto:Data: 2023-10-11T15:24:11.091 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.091 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.092 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.093 DEBUG:boto:Params: {} 2023-10-11T15:24:11.093 DEBUG:boto:Token: None 2023-10-11T15:24:11.094 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY= 2023-10-11T15:24:11.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY=', 'Content-Length': '0'} 2023-10-11T15:24:11.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164aec7148a6c5d3-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.142 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.143 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.145 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.145 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.146 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.147 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.148 DEBUG:boto:Data: 2023-10-11T15:24:11.148 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.150 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.151 DEBUG:boto:Params: {} 2023-10-11T15:24:11.151 DEBUG:boto:Token: None 2023-10-11T15:24:11.152 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.153 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hZh0iCblC0fz7+snc8YKVEvBm7o= 2023-10-11T15:24:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hZh0iCblC0fz7+snc8YKVEvBm7o=', 'Content-Length': '0'} 2023-10-11T15:24:11.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb5ba067ebce05e-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.157 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.158 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.158 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.158 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.159 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.159 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.160 DEBUG:boto:Data: 2023-10-11T15:24:11.160 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.161 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.161 DEBUG:boto:Params: {} 2023-10-11T15:24:11.162 DEBUG:boto:Token: None 2023-10-11T15:24:11.162 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hZh0iCblC0fz7+snc8YKVEvBm7o= 2023-10-11T15:24:11.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hZh0iCblC0fz7+snc8YKVEvBm7o=', 'Content-Length': '0'} 2023-10-11T15:24:11.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9316ad291f6079e-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:24:11.167 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.168 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.168 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.170 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.171 DEBUG:boto:Method: GET 2023-10-11T15:24:11.172 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.172 DEBUG:boto:Data: 2023-10-11T15:24:11.173 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.174 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.175 DEBUG:boto:Params: {} 2023-10-11T15:24:11.176 DEBUG:boto:Token: None 2023-10-11T15:24:11.176 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY= 2023-10-11T15:24:11.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY=', 'Content-Length': '0'} 2023-10-11T15:24:11.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce23512628bf4aea-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.181 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.181 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.182 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.183 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.184 DEBUG:boto:Method: GET 2023-10-11T15:24:11.184 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.185 DEBUG:boto:Data: 2023-10-11T15:24:11.185 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.186 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.186 DEBUG:boto:Params: {} 2023-10-11T15:24:11.187 DEBUG:boto:Token: None 2023-10-11T15:24:11.187 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:24:11.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY= 2023-10-11T15:24:11.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xwUJHnqDqOKVlC/H/EcxH66clKY=', 'Content-Length': '0'} 2023-10-11T15:24:11.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f42f2596b5e8e25-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.191 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.192 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.193 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.193 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.193 DEBUG:boto:Method: GET 2023-10-11T15:24:11.194 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.194 DEBUG:boto:Data: 2023-10-11T15:24:11.195 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.196 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.196 DEBUG:boto:Params: {} 2023-10-11T15:24:11.196 DEBUG:boto:Token: None 2023-10-11T15:24:11.197 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4e1354c27684aa-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.201 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.202 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.203 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.203 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.203 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.204 DEBUG:boto:Method: GET 2023-10-11T15:24:11.204 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.205 DEBUG:boto:Data: 2023-10-11T15:24:11.205 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.206 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.206 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.206 DEBUG:boto:Params: {} 2023-10-11T15:24:11.207 DEBUG:boto:Token: None 2023-10-11T15:24:11.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c8cdfce17412531-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.213 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.213 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.214 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.214 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.215 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:24:11.215 DEBUG:boto:Data: 2023-10-11T15:24:11.215 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.216 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.217 DEBUG:boto:Params: {} 2023-10-11T15:24:11.217 DEBUG:boto:Token: None 2023-10-11T15:24:11.218 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19 2023-10-11T15:24:11.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wsy5icKIksiFZIQngdFfNqh59Lc= 2023-10-11T15:24:11.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wsy5icKIksiFZIQngdFfNqh59Lc=', 'Content-Length': '0'} 2023-10-11T15:24:11.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b53ad7547f354302-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.221 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.222 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.222 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.223 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:24:11.223 DEBUG:boto:Data: 2023-10-11T15:24:11.223 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.224 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.225 DEBUG:boto:Params: {} 2023-10-11T15:24:11.225 DEBUG:boto:Token: None 2023-10-11T15:24:11.225 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19 2023-10-11T15:24:11.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wsy5icKIksiFZIQngdFfNqh59Lc= 2023-10-11T15:24:11.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wsy5icKIksiFZIQngdFfNqh59Lc=', 'Content-Length': '0'} 2023-10-11T15:24:11.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a05bdfd95eea14-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:24:11.229 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.230 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.230 DEBUG:boto:Method: GET 2023-10-11T15:24:11.231 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:24:11.231 DEBUG:boto:Data: 2023-10-11T15:24:11.232 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.232 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.233 DEBUG:boto:Params: {} 2023-10-11T15:24:11.233 DEBUG:boto:Token: None 2023-10-11T15:24:11.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19 2023-10-11T15:24:11.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BHcgRSSdr1laU03SAXs+zlaxiCY= 2023-10-11T15:24:11.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BHcgRSSdr1laU03SAXs+zlaxiCY=', 'Content-Length': '0'} 2023-10-11T15:24:11.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06e689b7f74919c-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.238 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:24:11.238 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:24:11.238 DEBUG:boto:Method: GET 2023-10-11T15:24:11.239 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:24:11.239 DEBUG:boto:Data: 2023-10-11T15:24:11.240 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.241 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.241 DEBUG:boto:Params: {} 2023-10-11T15:24:11.241 DEBUG:boto:Token: None 2023-10-11T15:24:11.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj19 2023-10-11T15:24:11.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BHcgRSSdr1laU03SAXs+zlaxiCY= 2023-10-11T15:24:11.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BHcgRSSdr1laU03SAXs+zlaxiCY=', 'Content-Length': '0'} 2023-10-11T15:24:11.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e576ac160e60bddd-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.288 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.291 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.294 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.294 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.295 DEBUG:boto:Method: GET 2023-10-11T15:24:11.295 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.296 DEBUG:boto:Data: 2023-10-11T15:24:11.296 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.297 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.298 DEBUG:boto:Params: {} 2023-10-11T15:24:11.298 DEBUG:boto:Token: None 2023-10-11T15:24:11.299 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dec4702cea4d946-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.353 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.355 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.355 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.355 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.356 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.356 DEBUG:boto:Method: GET 2023-10-11T15:24:11.357 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:24:11.357 DEBUG:boto:Data: 2023-10-11T15:24:11.358 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.359 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.359 DEBUG:boto:Params: {} 2023-10-11T15:24:11.360 DEBUG:boto:Token: None 2023-10-11T15:24:11.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060fed393efef4ede-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.366 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:24:11.367 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.368 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.368 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.369 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.369 DEBUG:boto:Method: GET 2023-10-11T15:24:11.370 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.370 DEBUG:boto:Data: 2023-10-11T15:24:11.371 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.372 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.372 DEBUG:boto:Params: {} 2023-10-11T15:24:11.373 DEBUG:boto:Token: None 2023-10-11T15:24:11.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A= 2023-10-11T15:24:11.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A=', 'Content-Length': '0'} 2023-10-11T15:24:11.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1818b92b9e1a6d2-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.378 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.378 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.379 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.379 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.380 DEBUG:boto:Method: GET 2023-10-11T15:24:11.380 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.381 DEBUG:boto:Data: 2023-10-11T15:24:11.381 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.382 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.383 DEBUG:boto:Params: {} 2023-10-11T15:24:11.383 DEBUG:boto:Token: None 2023-10-11T15:24:11.384 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A= 2023-10-11T15:24:11.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A=', 'Content-Length': '0'} 2023-10-11T15:24:11.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22a92d0a971d54c-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.388 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.388 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.389 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.389 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.390 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.390 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.391 DEBUG:boto:Data: 2023-10-11T15:24:11.391 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.392 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.393 DEBUG:boto:Params: {} 2023-10-11T15:24:11.393 DEBUG:boto:Token: None 2023-10-11T15:24:11.394 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i9sf7Tw3+fdjjE+i+iUSDZlbzIw= 2023-10-11T15:24:11.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i9sf7Tw3+fdjjE+i+iUSDZlbzIw=', 'Content-Length': '0'} 2023-10-11T15:24:11.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e8073f0cc769a12-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.398 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.398 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.399 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.399 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.400 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.400 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.400 DEBUG:boto:Data: 2023-10-11T15:24:11.401 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.402 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.402 DEBUG:boto:Params: {} 2023-10-11T15:24:11.403 DEBUG:boto:Token: None 2023-10-11T15:24:11.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i9sf7Tw3+fdjjE+i+iUSDZlbzIw= 2023-10-11T15:24:11.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i9sf7Tw3+fdjjE+i+iUSDZlbzIw=', 'Content-Length': '0'} 2023-10-11T15:24:11.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88c548aa7d76243-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:24:11.408 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.409 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.409 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.410 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.410 DEBUG:boto:Method: GET 2023-10-11T15:24:11.411 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.411 DEBUG:boto:Data: 2023-10-11T15:24:11.412 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.412 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.413 DEBUG:boto:Params: {} 2023-10-11T15:24:11.413 DEBUG:boto:Token: None 2023-10-11T15:24:11.414 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A= 2023-10-11T15:24:11.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A=', 'Content-Length': '0'} 2023-10-11T15:24:11.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001608fbb077646ac5-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.417 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.418 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.418 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.419 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.419 DEBUG:boto:Method: GET 2023-10-11T15:24:11.420 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.420 DEBUG:boto:Data: 2023-10-11T15:24:11.420 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.421 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.422 DEBUG:boto:Params: {} 2023-10-11T15:24:11.422 DEBUG:boto:Token: None 2023-10-11T15:24:11.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:24:11.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A= 2023-10-11T15:24:11.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X38AwY/KgdEYFdww84Mn96lV+8A=', 'Content-Length': '0'} 2023-10-11T15:24:11.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005992d0e185552828-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.475 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.476 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.476 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.476 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.477 DEBUG:boto:Method: GET 2023-10-11T15:24:11.477 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.478 DEBUG:boto:Data: 2023-10-11T15:24:11.478 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.479 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.479 DEBUG:boto:Params: {} 2023-10-11T15:24:11.482 DEBUG:boto:Token: None 2023-10-11T15:24:11.482 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000529d66e06d25f9bb-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.489 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.490 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.490 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.491 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.491 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.492 DEBUG:boto:Method: GET 2023-10-11T15:24:11.492 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.493 DEBUG:boto:Data: 2023-10-11T15:24:11.493 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.494 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.494 DEBUG:boto:Params: {} 2023-10-11T15:24:11.495 DEBUG:boto:Token: None 2023-10-11T15:24:11.495 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.496 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006031765d1ddfba7-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.501 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.502 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.502 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.503 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.503 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:24:11.503 DEBUG:boto:Data: 2023-10-11T15:24:11.504 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.505 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.505 DEBUG:boto:Params: {} 2023-10-11T15:24:11.506 DEBUG:boto:Token: None 2023-10-11T15:24:11.506 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22 2023-10-11T15:24:11.507 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4p8J2E4AGGWQpcB6rKIDPJwcp/E= 2023-10-11T15:24:11.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4p8J2E4AGGWQpcB6rKIDPJwcp/E=', 'Content-Length': '0'} 2023-10-11T15:24:11.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c02c0bde16f4c14-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.510 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.511 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.511 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.511 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:24:11.512 DEBUG:boto:Data: 2023-10-11T15:24:11.512 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.513 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.513 DEBUG:boto:Params: {} 2023-10-11T15:24:11.514 DEBUG:boto:Token: None 2023-10-11T15:24:11.514 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22 2023-10-11T15:24:11.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4p8J2E4AGGWQpcB6rKIDPJwcp/E= 2023-10-11T15:24:11.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4p8J2E4AGGWQpcB6rKIDPJwcp/E=', 'Content-Length': '0'} 2023-10-11T15:24:11.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe778d023be19a4-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:24:11.519 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.519 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.520 DEBUG:boto:Method: GET 2023-10-11T15:24:11.520 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:24:11.521 DEBUG:boto:Data: 2023-10-11T15:24:11.521 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.522 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.523 DEBUG:boto:Params: {} 2023-10-11T15:24:11.523 DEBUG:boto:Token: None 2023-10-11T15:24:11.524 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22 2023-10-11T15:24:11.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1sBQXAMKbXqOfc5idW7gX+T7yc8= 2023-10-11T15:24:11.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1sBQXAMKbXqOfc5idW7gX+T7yc8=', 'Content-Length': '0'} 2023-10-11T15:24:11.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cbe0d06ef9bae56-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.570 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:24:11.571 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:24:11.571 DEBUG:boto:Method: GET 2023-10-11T15:24:11.572 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:24:11.572 DEBUG:boto:Data: 2023-10-11T15:24:11.573 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.573 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.574 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.574 DEBUG:boto:Params: {} 2023-10-11T15:24:11.575 DEBUG:boto:Token: None 2023-10-11T15:24:11.575 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj22 2023-10-11T15:24:11.576 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1sBQXAMKbXqOfc5idW7gX+T7yc8= 2023-10-11T15:24:11.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1sBQXAMKbXqOfc5idW7gX+T7yc8=', 'Content-Length': '0'} 2023-10-11T15:24:11.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e22b802da34109c-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.580 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.580 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.580 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.581 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.581 DEBUG:boto:Method: GET 2023-10-11T15:24:11.582 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.582 DEBUG:boto:Data: 2023-10-11T15:24:11.583 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.584 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.584 DEBUG:boto:Params: {} 2023-10-11T15:24:11.584 DEBUG:boto:Token: None 2023-10-11T15:24:11.585 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f834ac76b396ecb-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.590 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.591 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.592 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.592 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.593 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.593 DEBUG:boto:Method: GET 2023-10-11T15:24:11.594 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:24:11.594 DEBUG:boto:Data: 2023-10-11T15:24:11.595 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.596 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.596 DEBUG:boto:Params: {} 2023-10-11T15:24:11.597 DEBUG:boto:Token: None 2023-10-11T15:24:11.597 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.598 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b4fa97e48c3ab7-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.642 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:24:11.644 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.644 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.645 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.645 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.645 DEBUG:boto:Method: GET 2023-10-11T15:24:11.646 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.647 DEBUG:boto:Data: 2023-10-11T15:24:11.647 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.648 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.648 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.648 DEBUG:boto:Params: {} 2023-10-11T15:24:11.649 DEBUG:boto:Token: None 2023-10-11T15:24:11.650 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.650 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM= 2023-10-11T15:24:11.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM=', 'Content-Length': '0'} 2023-10-11T15:24:11.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d5766b034d06044-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.654 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.654 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.655 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.655 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.656 DEBUG:boto:Method: GET 2023-10-11T15:24:11.656 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.657 DEBUG:boto:Data: 2023-10-11T15:24:11.657 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.658 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.659 DEBUG:boto:Params: {} 2023-10-11T15:24:11.659 DEBUG:boto:Token: None 2023-10-11T15:24:11.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM= 2023-10-11T15:24:11.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM=', 'Content-Length': '0'} 2023-10-11T15:24:11.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bccef9287d6eeaa-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.665 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.665 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.666 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.666 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.667 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.667 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.668 DEBUG:boto:Data: 2023-10-11T15:24:11.668 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.669 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.669 DEBUG:boto:Params: {} 2023-10-11T15:24:11.670 DEBUG:boto:Token: None 2023-10-11T15:24:11.670 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.671 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4/uBwGKMqaBecBDvWqhNehhMZUA= 2023-10-11T15:24:11.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4/uBwGKMqaBecBDvWqhNehhMZUA=', 'Content-Length': '0'} 2023-10-11T15:24:11.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b2f40122976438-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.674 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.675 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.675 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.676 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.676 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.676 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.677 DEBUG:boto:Data: 2023-10-11T15:24:11.677 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.678 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.679 DEBUG:boto:Params: {} 2023-10-11T15:24:11.679 DEBUG:boto:Token: None 2023-10-11T15:24:11.679 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.680 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4/uBwGKMqaBecBDvWqhNehhMZUA= 2023-10-11T15:24:11.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4/uBwGKMqaBecBDvWqhNehhMZUA=', 'Content-Length': '0'} 2023-10-11T15:24:11.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b45ba0d10699ec-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:24:11.684 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.684 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.684 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.685 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.685 DEBUG:boto:Method: GET 2023-10-11T15:24:11.686 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.686 DEBUG:boto:Data: 2023-10-11T15:24:11.687 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.687 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.689 DEBUG:boto:Params: {} 2023-10-11T15:24:11.690 DEBUG:boto:Token: None 2023-10-11T15:24:11.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM= 2023-10-11T15:24:11.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM=', 'Content-Length': '0'} 2023-10-11T15:24:11.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aca6afad343c7df-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.694 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.695 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.695 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.695 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.696 DEBUG:boto:Method: GET 2023-10-11T15:24:11.696 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.697 DEBUG:boto:Data: 2023-10-11T15:24:11.697 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.698 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.698 DEBUG:boto:Params: {} 2023-10-11T15:24:11.699 DEBUG:boto:Token: None 2023-10-11T15:24:11.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:24:11.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM= 2023-10-11T15:24:11.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eCOYBod/+4RbztxQmYyZiC/qHvM=', 'Content-Length': '0'} 2023-10-11T15:24:11.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d77869a2e706539-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.703 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.704 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.704 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.704 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.705 DEBUG:boto:Method: GET 2023-10-11T15:24:11.705 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.706 DEBUG:boto:Data: 2023-10-11T15:24:11.706 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.707 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.707 DEBUG:boto:Params: {} 2023-10-11T15:24:11.708 DEBUG:boto:Token: None 2023-10-11T15:24:11.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff6b36dcb214cb03-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.713 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.713 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.714 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.714 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.715 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.715 DEBUG:boto:Method: GET 2023-10-11T15:24:11.715 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.716 DEBUG:boto:Data: 2023-10-11T15:24:11.716 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.717 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.717 DEBUG:boto:Params: {} 2023-10-11T15:24:11.718 DEBUG:boto:Token: None 2023-10-11T15:24:11.718 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b495039e73f3c67-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.766 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.767 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.768 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.768 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.769 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:24:11.769 DEBUG:boto:Data: 2023-10-11T15:24:11.769 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.770 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.771 DEBUG:boto:Params: {} 2023-10-11T15:24:11.771 DEBUG:boto:Token: None 2023-10-11T15:24:11.772 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23 2023-10-11T15:24:11.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Ig+9Ue8BCW11qkOA33naGXvIVc= 2023-10-11T15:24:11.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Ig+9Ue8BCW11qkOA33naGXvIVc=', 'Content-Length': '0'} 2023-10-11T15:24:11.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003983aabc006e4ca2-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.775 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.776 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.776 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.777 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:24:11.777 DEBUG:boto:Data: 2023-10-11T15:24:11.777 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.778 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.779 DEBUG:boto:Params: {} 2023-10-11T15:24:11.779 DEBUG:boto:Token: None 2023-10-11T15:24:11.779 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23 2023-10-11T15:24:11.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Ig+9Ue8BCW11qkOA33naGXvIVc= 2023-10-11T15:24:11.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Ig+9Ue8BCW11qkOA33naGXvIVc=', 'Content-Length': '0'} 2023-10-11T15:24:11.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032bd454b852c61aa-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:24:11.784 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.784 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.785 DEBUG:boto:Method: GET 2023-10-11T15:24:11.785 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:24:11.786 DEBUG:boto:Data: 2023-10-11T15:24:11.786 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.787 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.787 DEBUG:boto:Params: {} 2023-10-11T15:24:11.788 DEBUG:boto:Token: None 2023-10-11T15:24:11.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23 2023-10-11T15:24:11.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8JCkreawFmapWmaMBIaGFLPqhrY= 2023-10-11T15:24:11.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8JCkreawFmapWmaMBIaGFLPqhrY=', 'Content-Length': '0'} 2023-10-11T15:24:11.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236df580826a6048-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.792 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:24:11.792 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:24:11.793 DEBUG:boto:Method: GET 2023-10-11T15:24:11.793 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:24:11.794 DEBUG:boto:Data: 2023-10-11T15:24:11.794 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.795 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.796 DEBUG:boto:Params: {} 2023-10-11T15:24:11.796 DEBUG:boto:Token: None 2023-10-11T15:24:11.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj23 2023-10-11T15:24:11.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8JCkreawFmapWmaMBIaGFLPqhrY= 2023-10-11T15:24:11.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8JCkreawFmapWmaMBIaGFLPqhrY=', 'Content-Length': '0'} 2023-10-11T15:24:11.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f690999d471c9a-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.801 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.801 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.802 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.802 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.802 DEBUG:boto:Method: GET 2023-10-11T15:24:11.803 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.803 DEBUG:boto:Data: 2023-10-11T15:24:11.804 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.804 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.805 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.805 DEBUG:boto:Params: {} 2023-10-11T15:24:11.805 DEBUG:boto:Token: None 2023-10-11T15:24:11.806 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b7cf61ba4103869-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.810 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.811 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.811 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.812 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.812 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.812 DEBUG:boto:Method: GET 2023-10-11T15:24:11.813 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:24:11.813 DEBUG:boto:Data: 2023-10-11T15:24:11.814 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.815 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.815 DEBUG:boto:Params: {} 2023-10-11T15:24:11.816 DEBUG:boto:Token: None 2023-10-11T15:24:11.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3720cb51b7b0029-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.821 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:24:11.822 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.823 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.823 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.824 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.824 DEBUG:boto:Method: GET 2023-10-11T15:24:11.825 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.825 DEBUG:boto:Data: 2023-10-11T15:24:11.825 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.826 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.827 DEBUG:boto:Params: {} 2023-10-11T15:24:11.827 DEBUG:boto:Token: None 2023-10-11T15:24:11.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM= 2023-10-11T15:24:11.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM=', 'Content-Length': '0'} 2023-10-11T15:24:11.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054228bf3ff9bad6d-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.832 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.832 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.833 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.833 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.833 DEBUG:boto:Method: GET 2023-10-11T15:24:11.834 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.834 DEBUG:boto:Data: 2023-10-11T15:24:11.835 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.836 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.836 DEBUG:boto:Params: {} 2023-10-11T15:24:11.836 DEBUG:boto:Token: None 2023-10-11T15:24:11.837 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.837 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM= 2023-10-11T15:24:11.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM=', 'Content-Length': '0'} 2023-10-11T15:24:11.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8c25528d9c4ed7-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.886 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.887 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.887 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.888 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.888 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.889 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.889 DEBUG:boto:Data: 2023-10-11T15:24:11.889 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.890 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.891 DEBUG:boto:Params: {} 2023-10-11T15:24:11.891 DEBUG:boto:Token: None 2023-10-11T15:24:11.892 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bv5+E+T354VApd6H3piPkpj2Vqw= 2023-10-11T15:24:11.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bv5+E+T354VApd6H3piPkpj2Vqw=', 'Content-Length': '0'} 2023-10-11T15:24:11.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e6ca5ae243590f-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.896 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.897 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.897 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.898 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.898 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.898 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.899 DEBUG:boto:Data: 2023-10-11T15:24:11.899 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.900 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.900 DEBUG:boto:Params: {} 2023-10-11T15:24:11.901 DEBUG:boto:Token: None 2023-10-11T15:24:11.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Bv5+E+T354VApd6H3piPkpj2Vqw= 2023-10-11T15:24:11.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Bv5+E+T354VApd6H3piPkpj2Vqw=', 'Content-Length': '0'} 2023-10-11T15:24:11.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd644e61b35618d-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:24:11.905 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.906 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.906 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.907 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.907 DEBUG:boto:Method: GET 2023-10-11T15:24:11.907 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.908 DEBUG:boto:Data: 2023-10-11T15:24:11.908 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.909 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.909 DEBUG:boto:Params: {} 2023-10-11T15:24:11.910 DEBUG:boto:Token: None 2023-10-11T15:24:11.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM= 2023-10-11T15:24:11.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM=', 'Content-Length': '0'} 2023-10-11T15:24:11.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030bd80060f4ac906-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.914 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.915 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.915 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.916 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.916 DEBUG:boto:Method: GET 2023-10-11T15:24:11.916 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.917 DEBUG:boto:Data: 2023-10-11T15:24:11.917 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.918 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.919 DEBUG:boto:Params: {} 2023-10-11T15:24:11.919 DEBUG:boto:Token: None 2023-10-11T15:24:11.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:24:11.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM= 2023-10-11T15:24:11.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qOsoRMSKZKcWrxH1TBeXbLdTDjM=', 'Content-Length': '0'} 2023-10-11T15:24:11.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017b11530f7088d1d-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.923 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.924 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.924 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.925 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.925 DEBUG:boto:Method: GET 2023-10-11T15:24:11.926 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.926 DEBUG:boto:Data: 2023-10-11T15:24:11.926 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.927 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.927 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.928 DEBUG:boto:Params: {} 2023-10-11T15:24:11.928 DEBUG:boto:Token: None 2023-10-11T15:24:11.929 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.929 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb8be942cf096b4-006526be1b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.933 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.933 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:11.934 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:11.934 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.935 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.935 DEBUG:boto:Method: GET 2023-10-11T15:24:11.936 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:24:11.936 DEBUG:boto:Data: 2023-10-11T15:24:11.936 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.937 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.938 DEBUG:boto:Params: {} 2023-10-11T15:24:11.938 DEBUG:boto:Token: None 2023-10-11T15:24:11.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/ 2023-10-11T15:24:11.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw= 2023-10-11T15:24:11.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjY1kt6ZhEbkm4jL9nxavKS/5jw=', 'Content-Length': '0'} 2023-10-11T15:24:11.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008884a6cad7b42d9b-006526be1b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.944 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:11.945 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.945 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.945 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.946 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:24:11.946 DEBUG:boto:Data: 2023-10-11T15:24:11.947 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.947 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.948 DEBUG:boto:Params: {} 2023-10-11T15:24:11.948 DEBUG:boto:Token: None 2023-10-11T15:24:11.949 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29 2023-10-11T15:24:11.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7iOyFRgBg7QNuuTMfYFndnQNQys= 2023-10-11T15:24:11.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7iOyFRgBg7QNuuTMfYFndnQNQys=', 'Content-Length': '0'} 2023-10-11T15:24:11.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ac7541cfdcbc09-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.952 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.953 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.953 DEBUG:boto:Method: HEAD 2023-10-11T15:24:11.954 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:24:11.954 DEBUG:boto:Data: 2023-10-11T15:24:11.954 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.955 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.956 DEBUG:boto:Params: {} 2023-10-11T15:24:11.956 DEBUG:boto:Token: None 2023-10-11T15:24:11.957 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29 2023-10-11T15:24:11.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7iOyFRgBg7QNuuTMfYFndnQNQys= 2023-10-11T15:24:11.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7iOyFRgBg7QNuuTMfYFndnQNQys=', 'Content-Length': '0'} 2023-10-11T15:24:11.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df9f790b3f23a51-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:24:11.961 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.961 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.962 DEBUG:boto:Method: GET 2023-10-11T15:24:11.962 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:24:11.963 DEBUG:boto:Data: 2023-10-11T15:24:11.963 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:11.964 DEBUG:boto:Port: 8001 2023-10-11T15:24:11.964 DEBUG:boto:Params: {} 2023-10-11T15:24:11.965 DEBUG:boto:Token: None 2023-10-11T15:24:11.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29 2023-10-11T15:24:11.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHbDnSMsobUDdperiTaRjkLu0EI= 2023-10-11T15:24:11.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHbDnSMsobUDdperiTaRjkLu0EI=', 'Content-Length': '0'} 2023-10-11T15:24:11.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326c47f77c212b0f-006526be1b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:11.969 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:24:11.969 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:24:11.970 DEBUG:boto:Method: GET 2023-10-11T15:24:11.970 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:24:11.970 DEBUG:boto:Data: 2023-10-11T15:24:11.971 DEBUG:boto:Headers: {} 2023-10-11T15:24:11.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:11.972 DEBUG:boto:Port: 8000 2023-10-11T15:24:11.972 DEBUG:boto:Params: {} 2023-10-11T15:24:11.973 DEBUG:boto:Token: None 2023-10-11T15:24:11.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:11 GMT /dghqvd-59/obj29 2023-10-11T15:24:11.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tHbDnSMsobUDdperiTaRjkLu0EI= 2023-10-11T15:24:11.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tHbDnSMsobUDdperiTaRjkLu0EI=', 'Content-Length': '0'} 2023-10-11T15:24:11.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746c63909e4978ee-006526be1b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.018 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.019 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.019 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.020 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.020 DEBUG:boto:Method: GET 2023-10-11T15:24:12.021 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.021 DEBUG:boto:Data: 2023-10-11T15:24:12.022 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.023 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.023 DEBUG:boto:Params: {} 2023-10-11T15:24:12.023 DEBUG:boto:Token: None 2023-10-11T15:24:12.024 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005235017ae315b979-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.028 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.029 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.029 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.030 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.030 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.030 DEBUG:boto:Method: GET 2023-10-11T15:24:12.031 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:24:12.031 DEBUG:boto:Data: 2023-10-11T15:24:12.032 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.033 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.033 DEBUG:boto:Params: {} 2023-10-11T15:24:12.033 DEBUG:boto:Token: None 2023-10-11T15:24:12.034 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035151785a322d6d5-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.039 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:24:12.040 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.040 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.041 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.041 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.042 DEBUG:boto:Method: GET 2023-10-11T15:24:12.042 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.042 DEBUG:boto:Data: 2023-10-11T15:24:12.043 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.044 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.044 DEBUG:boto:Params: {} 2023-10-11T15:24:12.045 DEBUG:boto:Token: None 2023-10-11T15:24:12.045 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw= 2023-10-11T15:24:12.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw=', 'Content-Length': '0'} 2023-10-11T15:24:12.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c3c99ed8c49555a-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.094 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.095 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.095 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.097 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.098 DEBUG:boto:Method: GET 2023-10-11T15:24:12.098 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.099 DEBUG:boto:Data: 2023-10-11T15:24:12.099 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.100 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.100 DEBUG:boto:Params: {} 2023-10-11T15:24:12.101 DEBUG:boto:Token: None 2023-10-11T15:24:12.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw= 2023-10-11T15:24:12.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw=', 'Content-Length': '0'} 2023-10-11T15:24:12.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d622b661a971374-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.105 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.106 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.106 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.106 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.107 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.107 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.108 DEBUG:boto:Data: 2023-10-11T15:24:12.108 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.109 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.109 DEBUG:boto:Params: {} 2023-10-11T15:24:12.110 DEBUG:boto:Token: None 2023-10-11T15:24:12.110 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EFHrwfksrlkm2AI5nNR6OQc6wmI= 2023-10-11T15:24:12.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EFHrwfksrlkm2AI5nNR6OQc6wmI=', 'Content-Length': '0'} 2023-10-11T15:24:12.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2912ef5e903400e-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.114 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.114 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.114 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.115 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.115 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.116 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.116 DEBUG:boto:Data: 2023-10-11T15:24:12.116 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.117 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.118 DEBUG:boto:Params: {} 2023-10-11T15:24:12.118 DEBUG:boto:Token: None 2023-10-11T15:24:12.119 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EFHrwfksrlkm2AI5nNR6OQc6wmI= 2023-10-11T15:24:12.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EFHrwfksrlkm2AI5nNR6OQc6wmI=', 'Content-Length': '0'} 2023-10-11T15:24:12.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b89603fc233756-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:24:12.123 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.123 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.124 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.124 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.124 DEBUG:boto:Method: GET 2023-10-11T15:24:12.125 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.125 DEBUG:boto:Data: 2023-10-11T15:24:12.126 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.127 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.127 DEBUG:boto:Params: {} 2023-10-11T15:24:12.127 DEBUG:boto:Token: None 2023-10-11T15:24:12.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw= 2023-10-11T15:24:12.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw=', 'Content-Length': '0'} 2023-10-11T15:24:12.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d0109f30a8e341-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.132 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.132 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.133 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.133 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.134 DEBUG:boto:Method: GET 2023-10-11T15:24:12.134 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.134 DEBUG:boto:Data: 2023-10-11T15:24:12.135 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.136 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.136 DEBUG:boto:Params: {} 2023-10-11T15:24:12.137 DEBUG:boto:Token: None 2023-10-11T15:24:12.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:24:12.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw= 2023-10-11T15:24:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Y8o7XoqiuTfR4IBiLZ/M7GT1zw=', 'Content-Length': '0'} 2023-10-11T15:24:12.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ac06ecb746a3ee-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.141 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.141 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.142 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.142 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.143 DEBUG:boto:Method: GET 2023-10-11T15:24:12.143 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.144 DEBUG:boto:Data: 2023-10-11T15:24:12.144 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.145 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.145 DEBUG:boto:Params: {} 2023-10-11T15:24:12.146 DEBUG:boto:Token: None 2023-10-11T15:24:12.146 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062618f2b5c781095-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.150 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.151 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.152 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.152 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.152 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.153 DEBUG:boto:Method: GET 2023-10-11T15:24:12.153 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.154 DEBUG:boto:Data: 2023-10-11T15:24:12.154 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.155 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.155 DEBUG:boto:Params: {} 2023-10-11T15:24:12.156 DEBUG:boto:Token: None 2023-10-11T15:24:12.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad674f35fd9c96f-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.162 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.162 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.163 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.163 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.164 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:24:12.164 DEBUG:boto:Data: 2023-10-11T15:24:12.164 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.165 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.166 DEBUG:boto:Params: {} 2023-10-11T15:24:12.166 DEBUG:boto:Token: None 2023-10-11T15:24:12.167 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33 2023-10-11T15:24:12.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cYWiezCERCDC0U9Yz0eILZuvyzI= 2023-10-11T15:24:12.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cYWiezCERCDC0U9Yz0eILZuvyzI=', 'Content-Length': '0'} 2023-10-11T15:24:12.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a882aac99450c7-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.170 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.171 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.171 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.172 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:24:12.172 DEBUG:boto:Data: 2023-10-11T15:24:12.172 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.173 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.174 DEBUG:boto:Params: {} 2023-10-11T15:24:12.174 DEBUG:boto:Token: None 2023-10-11T15:24:12.175 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33 2023-10-11T15:24:12.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cYWiezCERCDC0U9Yz0eILZuvyzI= 2023-10-11T15:24:12.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cYWiezCERCDC0U9Yz0eILZuvyzI=', 'Content-Length': '0'} 2023-10-11T15:24:12.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f6f2d397a4b367-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:24:12.179 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.179 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.180 DEBUG:boto:Method: GET 2023-10-11T15:24:12.180 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:24:12.181 DEBUG:boto:Data: 2023-10-11T15:24:12.181 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.182 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.182 DEBUG:boto:Params: {} 2023-10-11T15:24:12.183 DEBUG:boto:Token: None 2023-10-11T15:24:12.183 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33 2023-10-11T15:24:12.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6sOxJCpDlrrL7UlcXnWX4YXLQU= 2023-10-11T15:24:12.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6sOxJCpDlrrL7UlcXnWX4YXLQU=', 'Content-Length': '0'} 2023-10-11T15:24:12.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f44859fba55a4b-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.187 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:24:12.188 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:24:12.188 DEBUG:boto:Method: GET 2023-10-11T15:24:12.189 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:24:12.189 DEBUG:boto:Data: 2023-10-11T15:24:12.189 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.190 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.190 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.191 DEBUG:boto:Params: {} 2023-10-11T15:24:12.191 DEBUG:boto:Token: None 2023-10-11T15:24:12.192 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj33 2023-10-11T15:24:12.192 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6sOxJCpDlrrL7UlcXnWX4YXLQU= 2023-10-11T15:24:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6sOxJCpDlrrL7UlcXnWX4YXLQU=', 'Content-Length': '0'} 2023-10-11T15:24:12.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6b7ca58bded117-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.195 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.196 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.196 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.197 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.197 DEBUG:boto:Method: GET 2023-10-11T15:24:12.197 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.198 DEBUG:boto:Data: 2023-10-11T15:24:12.198 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.199 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.199 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.200 DEBUG:boto:Params: {} 2023-10-11T15:24:12.200 DEBUG:boto:Token: None 2023-10-11T15:24:12.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f27ff38d92c4fdc-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.246 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.247 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.248 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.248 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.249 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.249 DEBUG:boto:Method: GET 2023-10-11T15:24:12.250 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:24:12.250 DEBUG:boto:Data: 2023-10-11T15:24:12.251 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.252 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.253 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.253 DEBUG:boto:Params: {} 2023-10-11T15:24:12.254 DEBUG:boto:Token: None 2023-10-11T15:24:12.256 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a413fc27a7d1cf68-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.263 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:24:12.264 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.264 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.265 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.265 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.266 DEBUG:boto:Method: GET 2023-10-11T15:24:12.266 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.267 DEBUG:boto:Data: 2023-10-11T15:24:12.267 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.268 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.268 DEBUG:boto:Params: {} 2023-10-11T15:24:12.269 DEBUG:boto:Token: None 2023-10-11T15:24:12.269 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY= 2023-10-11T15:24:12.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY=', 'Content-Length': '0'} 2023-10-11T15:24:12.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002419fce3bbedf82-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.273 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.274 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.274 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.275 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.275 DEBUG:boto:Method: GET 2023-10-11T15:24:12.275 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.276 DEBUG:boto:Data: 2023-10-11T15:24:12.277 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.277 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.278 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.278 DEBUG:boto:Params: {} 2023-10-11T15:24:12.278 DEBUG:boto:Token: None 2023-10-11T15:24:12.279 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY= 2023-10-11T15:24:12.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY=', 'Content-Length': '0'} 2023-10-11T15:24:12.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b151f9883477f31-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.283 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.283 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.284 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.285 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.285 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.286 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.287 DEBUG:boto:Data: 2023-10-11T15:24:12.288 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.288 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.289 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.290 DEBUG:boto:Params: {} 2023-10-11T15:24:12.290 DEBUG:boto:Token: None 2023-10-11T15:24:12.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:arT74N1tsrN5wgbbiiiYDNNwkD8= 2023-10-11T15:24:12.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:arT74N1tsrN5wgbbiiiYDNNwkD8=', 'Content-Length': '0'} 2023-10-11T15:24:12.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b07ae8e1ec8b5f-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.294 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.295 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.295 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.296 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.296 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.296 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.297 DEBUG:boto:Data: 2023-10-11T15:24:12.297 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.298 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.299 DEBUG:boto:Params: {} 2023-10-11T15:24:12.302 DEBUG:boto:Token: None 2023-10-11T15:24:12.302 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:arT74N1tsrN5wgbbiiiYDNNwkD8= 2023-10-11T15:24:12.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:arT74N1tsrN5wgbbiiiYDNNwkD8=', 'Content-Length': '0'} 2023-10-11T15:24:12.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082d7e6d34d8903be-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:24:12.306 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.307 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.307 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.308 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.308 DEBUG:boto:Method: GET 2023-10-11T15:24:12.308 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.309 DEBUG:boto:Data: 2023-10-11T15:24:12.309 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.310 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.311 DEBUG:boto:Params: {} 2023-10-11T15:24:12.311 DEBUG:boto:Token: None 2023-10-11T15:24:12.312 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY= 2023-10-11T15:24:12.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY=', 'Content-Length': '0'} 2023-10-11T15:24:12.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3632ce15aa5c700-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.316 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.317 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.317 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.317 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.318 DEBUG:boto:Method: GET 2023-10-11T15:24:12.318 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.319 DEBUG:boto:Data: 2023-10-11T15:24:12.319 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.320 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.321 DEBUG:boto:Params: {} 2023-10-11T15:24:12.321 DEBUG:boto:Token: None 2023-10-11T15:24:12.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:24:12.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY= 2023-10-11T15:24:12.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kScKuhicYV89lus1UVrv0ds0XOY=', 'Content-Length': '0'} 2023-10-11T15:24:12.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb86f98c46b9d5a8-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.326 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.326 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.327 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.327 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.328 DEBUG:boto:Method: GET 2023-10-11T15:24:12.328 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.328 DEBUG:boto:Data: 2023-10-11T15:24:12.329 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.329 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.330 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.330 DEBUG:boto:Params: {} 2023-10-11T15:24:12.331 DEBUG:boto:Token: None 2023-10-11T15:24:12.331 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b14c0fe52c41d9f-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.335 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.336 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.337 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.338 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.338 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.339 DEBUG:boto:Method: GET 2023-10-11T15:24:12.340 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.341 DEBUG:boto:Data: 2023-10-11T15:24:12.341 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.343 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.343 DEBUG:boto:Params: {} 2023-10-11T15:24:12.344 DEBUG:boto:Token: None 2023-10-11T15:24:12.345 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5e7590fd7885e5-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.350 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.351 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.351 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.352 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.352 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:24:12.352 DEBUG:boto:Data: 2023-10-11T15:24:12.353 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.354 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.354 DEBUG:boto:Params: {} 2023-10-11T15:24:12.355 DEBUG:boto:Token: None 2023-10-11T15:24:12.355 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34 2023-10-11T15:24:12.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9x0Z5XpkoOr6cqkiATryrPzTZc= 2023-10-11T15:24:12.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9x0Z5XpkoOr6cqkiATryrPzTZc=', 'Content-Length': '0'} 2023-10-11T15:24:12.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a369c4085c095de-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.358 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.359 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.359 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.360 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:24:12.360 DEBUG:boto:Data: 2023-10-11T15:24:12.361 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.361 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.362 DEBUG:boto:Params: {} 2023-10-11T15:24:12.362 DEBUG:boto:Token: None 2023-10-11T15:24:12.363 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34 2023-10-11T15:24:12.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U9x0Z5XpkoOr6cqkiATryrPzTZc= 2023-10-11T15:24:12.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U9x0Z5XpkoOr6cqkiATryrPzTZc=', 'Content-Length': '0'} 2023-10-11T15:24:12.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1749d9b185f36ba-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:24:12.367 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.367 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.368 DEBUG:boto:Method: GET 2023-10-11T15:24:12.368 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:24:12.369 DEBUG:boto:Data: 2023-10-11T15:24:12.370 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.371 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.372 DEBUG:boto:Params: {} 2023-10-11T15:24:12.373 DEBUG:boto:Token: None 2023-10-11T15:24:12.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34 2023-10-11T15:24:12.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I4cAnGhRoqq+uL9W1rJ9hEfXBL4= 2023-10-11T15:24:12.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I4cAnGhRoqq+uL9W1rJ9hEfXBL4=', 'Content-Length': '0'} 2023-10-11T15:24:12.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c4509a880010e4-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.418 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:24:12.419 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:24:12.420 DEBUG:boto:Method: GET 2023-10-11T15:24:12.421 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:24:12.422 DEBUG:boto:Data: 2023-10-11T15:24:12.423 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.424 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.425 DEBUG:boto:Params: {} 2023-10-11T15:24:12.426 DEBUG:boto:Token: None 2023-10-11T15:24:12.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj34 2023-10-11T15:24:12.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I4cAnGhRoqq+uL9W1rJ9hEfXBL4= 2023-10-11T15:24:12.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I4cAnGhRoqq+uL9W1rJ9hEfXBL4=', 'Content-Length': '0'} 2023-10-11T15:24:12.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d800d79bde969525-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.433 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.433 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.434 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.435 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.436 DEBUG:boto:Method: GET 2023-10-11T15:24:12.436 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.437 DEBUG:boto:Data: 2023-10-11T15:24:12.438 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.439 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.440 DEBUG:boto:Params: {} 2023-10-11T15:24:12.440 DEBUG:boto:Token: None 2023-10-11T15:24:12.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a80eeefe38aa0ae-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.446 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.448 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.448 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.449 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.450 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.451 DEBUG:boto:Method: GET 2023-10-11T15:24:12.451 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:24:12.452 DEBUG:boto:Data: 2023-10-11T15:24:12.453 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.455 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.455 DEBUG:boto:Params: {} 2023-10-11T15:24:12.456 DEBUG:boto:Token: None 2023-10-11T15:24:12.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56e3a8339d0a9be-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.506 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:24:12.508 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.508 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.509 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.509 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.510 DEBUG:boto:Method: GET 2023-10-11T15:24:12.510 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.511 DEBUG:boto:Data: 2023-10-11T15:24:12.511 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.511 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.512 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.513 DEBUG:boto:Params: {} 2023-10-11T15:24:12.513 DEBUG:boto:Token: None 2023-10-11T15:24:12.514 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k= 2023-10-11T15:24:12.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k=', 'Content-Length': '0'} 2023-10-11T15:24:12.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af7021111769f67-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.519 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.520 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.520 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.521 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.521 DEBUG:boto:Method: GET 2023-10-11T15:24:12.521 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.522 DEBUG:boto:Data: 2023-10-11T15:24:12.522 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.523 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.523 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.524 DEBUG:boto:Params: {} 2023-10-11T15:24:12.524 DEBUG:boto:Token: None 2023-10-11T15:24:12.525 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k= 2023-10-11T15:24:12.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k=', 'Content-Length': '0'} 2023-10-11T15:24:12.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c9457a70bbc19d3-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.528 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.529 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.529 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.530 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.530 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.530 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.531 DEBUG:boto:Data: 2023-10-11T15:24:12.531 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.532 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.532 DEBUG:boto:Params: {} 2023-10-11T15:24:12.533 DEBUG:boto:Token: None 2023-10-11T15:24:12.533 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.534 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BdeMg7E5azfBVuU6soq9Ok9oiSc= 2023-10-11T15:24:12.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BdeMg7E5azfBVuU6soq9Ok9oiSc=', 'Content-Length': '0'} 2023-10-11T15:24:12.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ff8d492f2e98f7-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.537 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.538 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.538 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.538 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.539 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.539 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.540 DEBUG:boto:Data: 2023-10-11T15:24:12.540 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.541 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.542 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.543 DEBUG:boto:Params: {} 2023-10-11T15:24:12.543 DEBUG:boto:Token: None 2023-10-11T15:24:12.544 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BdeMg7E5azfBVuU6soq9Ok9oiSc= 2023-10-11T15:24:12.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BdeMg7E5azfBVuU6soq9Ok9oiSc=', 'Content-Length': '0'} 2023-10-11T15:24:12.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ade5a3c7c7332dba-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:24:12.550 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.551 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.552 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.553 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.553 DEBUG:boto:Method: GET 2023-10-11T15:24:12.554 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.555 DEBUG:boto:Data: 2023-10-11T15:24:12.556 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.557 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.558 DEBUG:boto:Params: {} 2023-10-11T15:24:12.559 DEBUG:boto:Token: None 2023-10-11T15:24:12.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k= 2023-10-11T15:24:12.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k=', 'Content-Length': '0'} 2023-10-11T15:24:12.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d642af550f0299d5-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.565 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.566 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.567 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.567 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.568 DEBUG:boto:Method: GET 2023-10-11T15:24:12.569 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.569 DEBUG:boto:Data: 2023-10-11T15:24:12.570 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.571 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.572 DEBUG:boto:Params: {} 2023-10-11T15:24:12.572 DEBUG:boto:Token: None 2023-10-11T15:24:12.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:24:12.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k= 2023-10-11T15:24:12.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CnjkDaqzI+ieX0erX7fXP/IiG/k=', 'Content-Length': '0'} 2023-10-11T15:24:12.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3cf99ec35c9497e-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.577 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.577 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.578 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.578 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.578 DEBUG:boto:Method: GET 2023-10-11T15:24:12.579 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.579 DEBUG:boto:Data: 2023-10-11T15:24:12.580 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.580 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.581 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.581 DEBUG:boto:Params: {} 2023-10-11T15:24:12.581 DEBUG:boto:Token: None 2023-10-11T15:24:12.582 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e22b3615fd2e9e7c-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.587 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.588 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.588 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.588 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.589 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.589 DEBUG:boto:Method: GET 2023-10-11T15:24:12.590 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.591 DEBUG:boto:Data: 2023-10-11T15:24:12.591 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.592 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.593 DEBUG:boto:Params: {} 2023-10-11T15:24:12.593 DEBUG:boto:Token: None 2023-10-11T15:24:12.594 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c00c77e0e4f5dddf-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.600 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.601 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.602 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.602 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.602 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:24:12.603 DEBUG:boto:Data: 2023-10-11T15:24:12.603 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.604 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.604 DEBUG:boto:Params: {} 2023-10-11T15:24:12.605 DEBUG:boto:Token: None 2023-10-11T15:24:12.606 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38 2023-10-11T15:24:12.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ls7FyZ2WT2eEPY0J52/ANfLe8ZQ= 2023-10-11T15:24:12.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ls7FyZ2WT2eEPY0J52/ANfLe8ZQ=', 'Content-Length': '0'} 2023-10-11T15:24:12.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e55205c5458b66-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.610 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.610 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.611 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.612 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:24:12.613 DEBUG:boto:Data: 2023-10-11T15:24:12.613 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.614 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.614 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.615 DEBUG:boto:Params: {} 2023-10-11T15:24:12.616 DEBUG:boto:Token: None 2023-10-11T15:24:12.617 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38 2023-10-11T15:24:12.618 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ls7FyZ2WT2eEPY0J52/ANfLe8ZQ= 2023-10-11T15:24:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ls7FyZ2WT2eEPY0J52/ANfLe8ZQ=', 'Content-Length': '0'} 2023-10-11T15:24:12.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf3f35ebe11cd16-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:24:12.623 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.624 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.625 DEBUG:boto:Method: GET 2023-10-11T15:24:12.625 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:24:12.626 DEBUG:boto:Data: 2023-10-11T15:24:12.626 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.627 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.628 DEBUG:boto:Params: {} 2023-10-11T15:24:12.628 DEBUG:boto:Token: None 2023-10-11T15:24:12.629 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38 2023-10-11T15:24:12.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KT857yJJSh01hnaZQda2xdh2DgI= 2023-10-11T15:24:12.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KT857yJJSh01hnaZQda2xdh2DgI=', 'Content-Length': '0'} 2023-10-11T15:24:12.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e3bc94895538f2-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.678 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:24:12.679 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:24:12.680 DEBUG:boto:Method: GET 2023-10-11T15:24:12.681 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:24:12.682 DEBUG:boto:Data: 2023-10-11T15:24:12.683 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.683 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.684 DEBUG:boto:Params: {} 2023-10-11T15:24:12.685 DEBUG:boto:Token: None 2023-10-11T15:24:12.686 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj38 2023-10-11T15:24:12.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KT857yJJSh01hnaZQda2xdh2DgI= 2023-10-11T15:24:12.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KT857yJJSh01hnaZQda2xdh2DgI=', 'Content-Length': '0'} 2023-10-11T15:24:12.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f54de8a5c6984e9-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.690 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.691 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.691 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.692 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.692 DEBUG:boto:Method: GET 2023-10-11T15:24:12.693 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.693 DEBUG:boto:Data: 2023-10-11T15:24:12.694 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.695 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.695 DEBUG:boto:Params: {} 2023-10-11T15:24:12.696 DEBUG:boto:Token: None 2023-10-11T15:24:12.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a432ce4f776bad-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.702 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.704 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.704 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.705 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.706 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.706 DEBUG:boto:Method: GET 2023-10-11T15:24:12.707 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:24:12.707 DEBUG:boto:Data: 2023-10-11T15:24:12.707 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.710 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.710 DEBUG:boto:Params: {} 2023-10-11T15:24:12.711 DEBUG:boto:Token: None 2023-10-11T15:24:12.711 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.712 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54e2cc8a986f6bd-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.716 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:24:12.717 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.718 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.718 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.719 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.719 DEBUG:boto:Method: GET 2023-10-11T15:24:12.720 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.720 DEBUG:boto:Data: 2023-10-11T15:24:12.721 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.721 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.722 DEBUG:boto:Params: {} 2023-10-11T15:24:12.722 DEBUG:boto:Token: None 2023-10-11T15:24:12.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E= 2023-10-11T15:24:12.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E=', 'Content-Length': '0'} 2023-10-11T15:24:12.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5f6f4fff8eea82-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.727 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.727 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.728 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.728 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.729 DEBUG:boto:Method: GET 2023-10-11T15:24:12.729 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.730 DEBUG:boto:Data: 2023-10-11T15:24:12.731 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.732 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.733 DEBUG:boto:Params: {} 2023-10-11T15:24:12.734 DEBUG:boto:Token: None 2023-10-11T15:24:12.735 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E= 2023-10-11T15:24:12.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E=', 'Content-Length': '0'} 2023-10-11T15:24:12.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e53b3bd992e9cde-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.786 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.787 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.787 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.788 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.788 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.789 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.789 DEBUG:boto:Data: 2023-10-11T15:24:12.789 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.790 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.791 DEBUG:boto:Params: {} 2023-10-11T15:24:12.791 DEBUG:boto:Token: None 2023-10-11T15:24:12.792 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zPk6huZSEoa15iPHVcsEewdV4n0= 2023-10-11T15:24:12.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zPk6huZSEoa15iPHVcsEewdV4n0=', 'Content-Length': '0'} 2023-10-11T15:24:12.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019a5ebf751b6380-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.795 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.796 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.796 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.797 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.797 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.798 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.798 DEBUG:boto:Data: 2023-10-11T15:24:12.799 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.799 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.800 DEBUG:boto:Params: {} 2023-10-11T15:24:12.800 DEBUG:boto:Token: None 2023-10-11T15:24:12.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zPk6huZSEoa15iPHVcsEewdV4n0= 2023-10-11T15:24:12.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zPk6huZSEoa15iPHVcsEewdV4n0=', 'Content-Length': '0'} 2023-10-11T15:24:12.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c515565be35f2397-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:24:12.805 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.806 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.806 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.806 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.807 DEBUG:boto:Method: GET 2023-10-11T15:24:12.807 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.808 DEBUG:boto:Data: 2023-10-11T15:24:12.808 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.809 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.809 DEBUG:boto:Params: {} 2023-10-11T15:24:12.810 DEBUG:boto:Token: None 2023-10-11T15:24:12.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E= 2023-10-11T15:24:12.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E=', 'Content-Length': '0'} 2023-10-11T15:24:12.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1c72a06ee204d9-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.814 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.814 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.815 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.816 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.816 DEBUG:boto:Method: GET 2023-10-11T15:24:12.817 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.817 DEBUG:boto:Data: 2023-10-11T15:24:12.818 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.818 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.818 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.819 DEBUG:boto:Params: {} 2023-10-11T15:24:12.819 DEBUG:boto:Token: None 2023-10-11T15:24:12.820 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:24:12.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E= 2023-10-11T15:24:12.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OnO/3SV3deNncG7NreG9TLcUa7E=', 'Content-Length': '0'} 2023-10-11T15:24:12.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fbb6ac793d3a533-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.824 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.824 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.824 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.825 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.825 DEBUG:boto:Method: GET 2023-10-11T15:24:12.826 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.826 DEBUG:boto:Data: 2023-10-11T15:24:12.826 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.827 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.828 DEBUG:boto:Params: {} 2023-10-11T15:24:12.828 DEBUG:boto:Token: None 2023-10-11T15:24:12.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004892f3d92d43d8c3-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.833 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.833 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.834 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.834 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.835 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.835 DEBUG:boto:Method: GET 2023-10-11T15:24:12.835 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.836 DEBUG:boto:Data: 2023-10-11T15:24:12.836 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.837 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.837 DEBUG:boto:Params: {} 2023-10-11T15:24:12.838 DEBUG:boto:Token: None 2023-10-11T15:24:12.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f12d5e136e44f5a8-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.843 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.844 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.844 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.844 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.845 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:24:12.845 DEBUG:boto:Data: 2023-10-11T15:24:12.846 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.847 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.847 DEBUG:boto:Params: {} 2023-10-11T15:24:12.848 DEBUG:boto:Token: None 2023-10-11T15:24:12.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44 2023-10-11T15:24:12.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B9g4enPbxpkEu+AfV+K1xBjCTNU= 2023-10-11T15:24:12.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B9g4enPbxpkEu+AfV+K1xBjCTNU=', 'Content-Length': '0'} 2023-10-11T15:24:12.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd46e43bfd81da2-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.852 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.852 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.852 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.853 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:24:12.853 DEBUG:boto:Data: 2023-10-11T15:24:12.854 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.854 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.855 DEBUG:boto:Params: {} 2023-10-11T15:24:12.855 DEBUG:boto:Token: None 2023-10-11T15:24:12.856 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44 2023-10-11T15:24:12.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B9g4enPbxpkEu+AfV+K1xBjCTNU= 2023-10-11T15:24:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B9g4enPbxpkEu+AfV+K1xBjCTNU=', 'Content-Length': '0'} 2023-10-11T15:24:12.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002359d64fb0a34909-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:24:12.859 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.860 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.860 DEBUG:boto:Method: GET 2023-10-11T15:24:12.861 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:24:12.861 DEBUG:boto:Data: 2023-10-11T15:24:12.861 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.862 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.863 DEBUG:boto:Params: {} 2023-10-11T15:24:12.863 DEBUG:boto:Token: None 2023-10-11T15:24:12.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44 2023-10-11T15:24:12.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GtaMiN5w2EkWhDNMx2lqhxH0Y6o= 2023-10-11T15:24:12.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GtaMiN5w2EkWhDNMx2lqhxH0Y6o=', 'Content-Length': '0'} 2023-10-11T15:24:12.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd1865f6253e7280-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.910 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:24:12.911 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:24:12.912 DEBUG:boto:Method: GET 2023-10-11T15:24:12.912 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:24:12.913 DEBUG:boto:Data: 2023-10-11T15:24:12.913 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.914 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.914 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.915 DEBUG:boto:Params: {} 2023-10-11T15:24:12.915 DEBUG:boto:Token: None 2023-10-11T15:24:12.916 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj44 2023-10-11T15:24:12.916 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GtaMiN5w2EkWhDNMx2lqhxH0Y6o= 2023-10-11T15:24:12.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GtaMiN5w2EkWhDNMx2lqhxH0Y6o=', 'Content-Length': '0'} 2023-10-11T15:24:12.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfcbe5074725dc97-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.920 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.920 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.921 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.921 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.922 DEBUG:boto:Method: GET 2023-10-11T15:24:12.922 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.923 DEBUG:boto:Data: 2023-10-11T15:24:12.923 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.923 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.924 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.924 DEBUG:boto:Params: {} 2023-10-11T15:24:12.924 DEBUG:boto:Token: None 2023-10-11T15:24:12.925 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003745cef7b31777ae-006526be1c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.930 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.931 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:12.932 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:12.933 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.933 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.934 DEBUG:boto:Method: GET 2023-10-11T15:24:12.935 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:24:12.935 DEBUG:boto:Data: 2023-10-11T15:24:12.936 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.937 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.938 DEBUG:boto:Params: {} 2023-10-11T15:24:12.939 DEBUG:boto:Token: None 2023-10-11T15:24:12.940 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/ 2023-10-11T15:24:12.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM= 2023-10-11T15:24:12.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0QhN08FFBqszrFsglUad4DIw/KM=', 'Content-Length': '0'} 2023-10-11T15:24:12.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e06667dd44e77a5-006526be1c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.947 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:12.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:24:12.948 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:12.949 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:12.949 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.950 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.950 DEBUG:boto:Method: GET 2023-10-11T15:24:12.951 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.951 DEBUG:boto:Data: 2023-10-11T15:24:12.952 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.954 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.954 DEBUG:boto:Params: {} 2023-10-11T15:24:12.955 DEBUG:boto:Token: None 2023-10-11T15:24:12.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w= 2023-10-11T15:24:12.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w=', 'Content-Length': '0'} 2023-10-11T15:24:12.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092609ae6bf335d76-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.960 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:12.961 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:12.961 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.961 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.962 DEBUG:boto:Method: GET 2023-10-11T15:24:12.962 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.963 DEBUG:boto:Data: 2023-10-11T15:24:12.963 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.964 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.965 DEBUG:boto:Params: {} 2023-10-11T15:24:12.965 DEBUG:boto:Token: None 2023-10-11T15:24:12.966 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w= 2023-10-11T15:24:12.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w=', 'Content-Length': '0'} 2023-10-11T15:24:12.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd423602d6f08d4-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.970 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:12.970 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:12.971 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.971 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.971 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.972 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.972 DEBUG:boto:Data: 2023-10-11T15:24:12.973 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.974 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.974 DEBUG:boto:Params: {} 2023-10-11T15:24:12.975 DEBUG:boto:Token: None 2023-10-11T15:24:12.975 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7PIGLJgcIuKP2EV0oPbCZmbKUYc= 2023-10-11T15:24:12.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7PIGLJgcIuKP2EV0oPbCZmbKUYc=', 'Content-Length': '0'} 2023-10-11T15:24:12.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7329995f93b842c-006526be1c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.979 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:12.979 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:12.979 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.980 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.980 DEBUG:boto:Method: HEAD 2023-10-11T15:24:12.981 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.981 DEBUG:boto:Data: 2023-10-11T15:24:12.981 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:12.982 DEBUG:boto:Port: 8000 2023-10-11T15:24:12.983 DEBUG:boto:Params: {} 2023-10-11T15:24:12.983 DEBUG:boto:Token: None 2023-10-11T15:24:12.984 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7PIGLJgcIuKP2EV0oPbCZmbKUYc= 2023-10-11T15:24:12.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7PIGLJgcIuKP2EV0oPbCZmbKUYc=', 'Content-Length': '0'} 2023-10-11T15:24:12.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc55071b33614bd-006526be1c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:12.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:24:12.988 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:12.989 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:12.989 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.990 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.991 DEBUG:boto:Method: GET 2023-10-11T15:24:12.992 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.993 DEBUG:boto:Data: 2023-10-11T15:24:12.993 DEBUG:boto:Headers: {} 2023-10-11T15:24:12.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:12.995 DEBUG:boto:Port: 8001 2023-10-11T15:24:12.996 DEBUG:boto:Params: {} 2023-10-11T15:24:12.997 DEBUG:boto:Token: None 2023-10-11T15:24:12.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:12 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:12.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w= 2023-10-11T15:24:12.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ebk00+uWaRt9VZBDOIneWpUW5+w=', 'Content-Length': '0'} 2023-10-11T15:24:13.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008414a1a13cae6e8f-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.003 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:13.003 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:13.004 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:13.005 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:24:13.006 DEBUG:boto:Method: GET 2023-10-11T15:24:13.007 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:13.007 DEBUG:boto:Data: 2023-10-11T15:24:13.008 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.010 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.010 DEBUG:boto:Params: {} 2023-10-11T15:24:13.011 DEBUG:boto:Token: None 2023-10-11T15:24:13.012 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:24:13.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MJE51FWcIJBKwz5aotJOmcItB4E= 2023-10-11T15:24:13.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MJE51FWcIJBKwz5aotJOmcItB4E=', 'Content-Length': '0'} 2023-10-11T15:24:13.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ef3f088269a76e4-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.018 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.019 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.019 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.020 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.020 DEBUG:boto:Method: GET 2023-10-11T15:24:13.021 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.022 DEBUG:boto:Data: 2023-10-11T15:24:13.022 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.023 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.024 DEBUG:boto:Params: {} 2023-10-11T15:24:13.024 DEBUG:boto:Token: None 2023-10-11T15:24:13.025 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb959bd86834bc2-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.029 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.030 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.030 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.031 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.032 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.033 DEBUG:boto:Method: GET 2023-10-11T15:24:13.033 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.034 DEBUG:boto:Data: 2023-10-11T15:24:13.035 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.037 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.037 DEBUG:boto:Params: {} 2023-10-11T15:24:13.038 DEBUG:boto:Token: None 2023-10-11T15:24:13.039 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a091e8ee94621255-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.046 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.047 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:13.048 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:13.048 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.049 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:24:13.050 DEBUG:boto:Data: 2023-10-11T15:24:13.051 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.052 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.052 DEBUG:boto:Params: {} 2023-10-11T15:24:13.053 DEBUG:boto:Token: None 2023-10-11T15:24:13.054 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj46 2023-10-11T15:24:13.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPr6LbTTEy0G8M6KPw62tx1d1RQ= 2023-10-11T15:24:13.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPr6LbTTEy0G8M6KPw62tx1d1RQ=', 'Content-Length': '0'} 2023-10-11T15:24:13.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e83e79261a113757-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.058 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:13.058 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:13.058 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.059 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:24:13.059 DEBUG:boto:Data: 2023-10-11T15:24:13.060 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.060 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.060 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.061 DEBUG:boto:Params: {} 2023-10-11T15:24:13.061 DEBUG:boto:Token: None 2023-10-11T15:24:13.062 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj46 2023-10-11T15:24:13.062 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPr6LbTTEy0G8M6KPw62tx1d1RQ= 2023-10-11T15:24:13.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPr6LbTTEy0G8M6KPw62tx1d1RQ=', 'Content-Length': '0'} 2023-10-11T15:24:13.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026aee92fafe36134-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:24:13.066 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:13.067 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:13.068 DEBUG:boto:Method: GET 2023-10-11T15:24:13.068 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:24:13.069 DEBUG:boto:Data: 2023-10-11T15:24:13.070 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.071 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.071 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.072 DEBUG:boto:Params: {} 2023-10-11T15:24:13.073 DEBUG:boto:Token: None 2023-10-11T15:24:13.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj46 2023-10-11T15:24:13.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LkQiAwzsUxdNKP4C/e1A97VnPpU= 2023-10-11T15:24:13.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LkQiAwzsUxdNKP4C/e1A97VnPpU=', 'Content-Length': '0'} 2023-10-11T15:24:13.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099fbed7e6d5ccb8e-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.078 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:24:13.079 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:24:13.079 DEBUG:boto:Method: GET 2023-10-11T15:24:13.080 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:24:13.080 DEBUG:boto:Data: 2023-10-11T15:24:13.081 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.081 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.082 DEBUG:boto:Params: {} 2023-10-11T15:24:13.082 DEBUG:boto:Token: None 2023-10-11T15:24:13.083 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj46 2023-10-11T15:24:13.083 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LkQiAwzsUxdNKP4C/e1A97VnPpU= 2023-10-11T15:24:13.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LkQiAwzsUxdNKP4C/e1A97VnPpU=', 'Content-Length': '0'} 2023-10-11T15:24:13.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd64d9387a521700-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.126 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.127 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.128 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.128 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.129 DEBUG:boto:Method: GET 2023-10-11T15:24:13.130 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.130 DEBUG:boto:Data: 2023-10-11T15:24:13.131 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.132 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.133 DEBUG:boto:Params: {} 2023-10-11T15:24:13.134 DEBUG:boto:Token: None 2023-10-11T15:24:13.135 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a56d2d72b09300-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.140 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.141 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.142 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.143 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.143 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.144 DEBUG:boto:Method: GET 2023-10-11T15:24:13.144 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:24:13.145 DEBUG:boto:Data: 2023-10-11T15:24:13.145 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.146 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.146 DEBUG:boto:Params: {} 2023-10-11T15:24:13.147 DEBUG:boto:Token: None 2023-10-11T15:24:13.147 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2540e308f91afa0-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.153 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:24:13.154 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.155 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.155 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.155 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.156 DEBUG:boto:Method: GET 2023-10-11T15:24:13.156 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.157 DEBUG:boto:Data: 2023-10-11T15:24:13.157 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.158 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.159 DEBUG:boto:Params: {} 2023-10-11T15:24:13.159 DEBUG:boto:Token: None 2023-10-11T15:24:13.159 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY= 2023-10-11T15:24:13.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY=', 'Content-Length': '0'} 2023-10-11T15:24:13.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575fba503e046cfd-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.163 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.164 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.164 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.164 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.165 DEBUG:boto:Method: GET 2023-10-11T15:24:13.166 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.166 DEBUG:boto:Data: 2023-10-11T15:24:13.167 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.167 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.167 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.168 DEBUG:boto:Params: {} 2023-10-11T15:24:13.168 DEBUG:boto:Token: None 2023-10-11T15:24:13.169 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY= 2023-10-11T15:24:13.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY=', 'Content-Length': '0'} 2023-10-11T15:24:13.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aef92d9e16764e3-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.172 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.173 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.173 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.174 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.174 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.175 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.175 DEBUG:boto:Data: 2023-10-11T15:24:13.175 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.176 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.177 DEBUG:boto:Params: {} 2023-10-11T15:24:13.177 DEBUG:boto:Token: None 2023-10-11T15:24:13.178 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.178 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pcr3LMc9IaRxEbZn1tDa8lrR6HM= 2023-10-11T15:24:13.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pcr3LMc9IaRxEbZn1tDa8lrR6HM=', 'Content-Length': '0'} 2023-10-11T15:24:13.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000538e059153e86cdc-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.181 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.181 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.182 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.182 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.182 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.183 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.183 DEBUG:boto:Data: 2023-10-11T15:24:13.184 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.184 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.185 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.185 DEBUG:boto:Params: {} 2023-10-11T15:24:13.185 DEBUG:boto:Token: None 2023-10-11T15:24:13.186 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.186 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pcr3LMc9IaRxEbZn1tDa8lrR6HM= 2023-10-11T15:24:13.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pcr3LMc9IaRxEbZn1tDa8lrR6HM=', 'Content-Length': '0'} 2023-10-11T15:24:13.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef5df7904719d48-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:24:13.190 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.190 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.191 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.191 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.192 DEBUG:boto:Method: GET 2023-10-11T15:24:13.192 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.192 DEBUG:boto:Data: 2023-10-11T15:24:13.193 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.194 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.194 DEBUG:boto:Params: {} 2023-10-11T15:24:13.195 DEBUG:boto:Token: None 2023-10-11T15:24:13.195 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY= 2023-10-11T15:24:13.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY=', 'Content-Length': '0'} 2023-10-11T15:24:13.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021691d9bf289c703-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.242 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.243 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.243 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.244 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.244 DEBUG:boto:Method: GET 2023-10-11T15:24:13.245 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.245 DEBUG:boto:Data: 2023-10-11T15:24:13.245 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.246 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.247 DEBUG:boto:Params: {} 2023-10-11T15:24:13.247 DEBUG:boto:Token: None 2023-10-11T15:24:13.248 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:24:13.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY= 2023-10-11T15:24:13.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l9sNyIMyKcLPo7PZ/An4IVpmjvY=', 'Content-Length': '0'} 2023-10-11T15:24:13.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b68572170fa7bd-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.251 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.252 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.252 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.253 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.253 DEBUG:boto:Method: GET 2023-10-11T15:24:13.254 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.254 DEBUG:boto:Data: 2023-10-11T15:24:13.255 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.255 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.256 DEBUG:boto:Params: {} 2023-10-11T15:24:13.256 DEBUG:boto:Token: None 2023-10-11T15:24:13.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042028ad9e1cab9ac-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.261 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.262 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.263 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.263 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.264 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.265 DEBUG:boto:Method: GET 2023-10-11T15:24:13.266 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.266 DEBUG:boto:Data: 2023-10-11T15:24:13.267 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.268 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.269 DEBUG:boto:Params: {} 2023-10-11T15:24:13.270 DEBUG:boto:Token: None 2023-10-11T15:24:13.271 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acab2b9980cce52c-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.318 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.322 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.322 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.323 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.323 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:24:13.323 DEBUG:boto:Data: 2023-10-11T15:24:13.324 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.325 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.325 DEBUG:boto:Params: {} 2023-10-11T15:24:13.325 DEBUG:boto:Token: None 2023-10-11T15:24:13.326 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51 2023-10-11T15:24:13.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SWLORVg34FiIlOODc9BUxpzoDJI= 2023-10-11T15:24:13.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SWLORVg34FiIlOODc9BUxpzoDJI=', 'Content-Length': '0'} 2023-10-11T15:24:13.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a04263f04c59722-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.330 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.330 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.331 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.331 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:24:13.332 DEBUG:boto:Data: 2023-10-11T15:24:13.332 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.333 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.334 DEBUG:boto:Params: {} 2023-10-11T15:24:13.334 DEBUG:boto:Token: None 2023-10-11T15:24:13.335 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51 2023-10-11T15:24:13.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SWLORVg34FiIlOODc9BUxpzoDJI= 2023-10-11T15:24:13.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SWLORVg34FiIlOODc9BUxpzoDJI=', 'Content-Length': '0'} 2023-10-11T15:24:13.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000679ceb633fd2ae6-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:24:13.339 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.339 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.340 DEBUG:boto:Method: GET 2023-10-11T15:24:13.341 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:24:13.342 DEBUG:boto:Data: 2023-10-11T15:24:13.342 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.344 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.345 DEBUG:boto:Params: {} 2023-10-11T15:24:13.345 DEBUG:boto:Token: None 2023-10-11T15:24:13.346 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51 2023-10-11T15:24:13.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oncm9zYT68snGVXfY/fLf7Vxu9w= 2023-10-11T15:24:13.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oncm9zYT68snGVXfY/fLf7Vxu9w=', 'Content-Length': '0'} 2023-10-11T15:24:13.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f833882c1d9c0d7-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.350 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:24:13.350 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:24:13.351 DEBUG:boto:Method: GET 2023-10-11T15:24:13.351 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:24:13.352 DEBUG:boto:Data: 2023-10-11T15:24:13.352 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.353 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.353 DEBUG:boto:Params: {} 2023-10-11T15:24:13.354 DEBUG:boto:Token: None 2023-10-11T15:24:13.354 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj51 2023-10-11T15:24:13.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oncm9zYT68snGVXfY/fLf7Vxu9w= 2023-10-11T15:24:13.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oncm9zYT68snGVXfY/fLf7Vxu9w=', 'Content-Length': '0'} 2023-10-11T15:24:13.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a63fb1272612e1fd-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.358 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.358 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.359 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.359 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.360 DEBUG:boto:Method: GET 2023-10-11T15:24:13.360 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.361 DEBUG:boto:Data: 2023-10-11T15:24:13.361 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.362 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.362 DEBUG:boto:Params: {} 2023-10-11T15:24:13.363 DEBUG:boto:Token: None 2023-10-11T15:24:13.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097273b27c49ef826-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.367 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.368 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.368 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.369 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.369 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.370 DEBUG:boto:Method: GET 2023-10-11T15:24:13.371 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:24:13.371 DEBUG:boto:Data: 2023-10-11T15:24:13.371 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.372 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.373 DEBUG:boto:Params: {} 2023-10-11T15:24:13.373 DEBUG:boto:Token: None 2023-10-11T15:24:13.374 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f0cc78fc3f6944-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.379 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:24:13.380 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.380 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.381 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.381 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.381 DEBUG:boto:Method: GET 2023-10-11T15:24:13.382 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.382 DEBUG:boto:Data: 2023-10-11T15:24:13.383 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.383 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.384 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.384 DEBUG:boto:Params: {} 2023-10-11T15:24:13.384 DEBUG:boto:Token: None 2023-10-11T15:24:13.385 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY= 2023-10-11T15:24:13.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY=', 'Content-Length': '0'} 2023-10-11T15:24:13.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e255839daf155e52-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.388 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.389 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.389 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.390 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.390 DEBUG:boto:Method: GET 2023-10-11T15:24:13.391 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.391 DEBUG:boto:Data: 2023-10-11T15:24:13.392 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.392 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.393 DEBUG:boto:Params: {} 2023-10-11T15:24:13.393 DEBUG:boto:Token: None 2023-10-11T15:24:13.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY= 2023-10-11T15:24:13.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY=', 'Content-Length': '0'} 2023-10-11T15:24:13.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509d1bbd329d14ea-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.398 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.398 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.398 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.399 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.399 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.400 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.400 DEBUG:boto:Data: 2023-10-11T15:24:13.400 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.401 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.402 DEBUG:boto:Params: {} 2023-10-11T15:24:13.402 DEBUG:boto:Token: None 2023-10-11T15:24:13.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFI4P0rJfrfkpJXeGp6qWcw1Ie8= 2023-10-11T15:24:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFI4P0rJfrfkpJXeGp6qWcw1Ie8=', 'Content-Length': '0'} 2023-10-11T15:24:13.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000503204cdc6ec2667-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.406 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.407 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.407 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.408 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.409 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.410 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.410 DEBUG:boto:Data: 2023-10-11T15:24:13.411 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.412 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.413 DEBUG:boto:Params: {} 2023-10-11T15:24:13.413 DEBUG:boto:Token: None 2023-10-11T15:24:13.414 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.415 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WFI4P0rJfrfkpJXeGp6qWcw1Ie8= 2023-10-11T15:24:13.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WFI4P0rJfrfkpJXeGp6qWcw1Ie8=', 'Content-Length': '0'} 2023-10-11T15:24:13.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0fffa42073e332-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:24:13.418 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.419 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.419 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.420 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.420 DEBUG:boto:Method: GET 2023-10-11T15:24:13.420 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.421 DEBUG:boto:Data: 2023-10-11T15:24:13.421 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.422 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.423 DEBUG:boto:Params: {} 2023-10-11T15:24:13.423 DEBUG:boto:Token: None 2023-10-11T15:24:13.424 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY= 2023-10-11T15:24:13.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY=', 'Content-Length': '0'} 2023-10-11T15:24:13.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c5d97f0180956e-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.428 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.428 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.429 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.429 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.429 DEBUG:boto:Method: GET 2023-10-11T15:24:13.430 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.430 DEBUG:boto:Data: 2023-10-11T15:24:13.431 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.431 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.432 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.432 DEBUG:boto:Params: {} 2023-10-11T15:24:13.432 DEBUG:boto:Token: None 2023-10-11T15:24:13.433 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:24:13.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY= 2023-10-11T15:24:13.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:p5lWLzfXQlSv67lRXZhtnGdd5PY=', 'Content-Length': '0'} 2023-10-11T15:24:13.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442bf6e8340f6395-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.437 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.437 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.438 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.438 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.438 DEBUG:boto:Method: GET 2023-10-11T15:24:13.439 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.439 DEBUG:boto:Data: 2023-10-11T15:24:13.440 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.441 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.441 DEBUG:boto:Params: {} 2023-10-11T15:24:13.442 DEBUG:boto:Token: None 2023-10-11T15:24:13.442 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075f2272002067079-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.494 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.495 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.496 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.496 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.496 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.497 DEBUG:boto:Method: GET 2023-10-11T15:24:13.497 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.498 DEBUG:boto:Data: 2023-10-11T15:24:13.498 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.499 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.500 DEBUG:boto:Params: {} 2023-10-11T15:24:13.500 DEBUG:boto:Token: None 2023-10-11T15:24:13.501 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d635e5bd8027cd7d-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.507 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.508 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.509 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.509 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.510 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:24:13.510 DEBUG:boto:Data: 2023-10-11T15:24:13.511 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.511 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.511 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.512 DEBUG:boto:Params: {} 2023-10-11T15:24:13.512 DEBUG:boto:Token: None 2023-10-11T15:24:13.513 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55 2023-10-11T15:24:13.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pXKtT93Ui7c3SdBcF4B/ZDzC2jA= 2023-10-11T15:24:13.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pXKtT93Ui7c3SdBcF4B/ZDzC2jA=', 'Content-Length': '0'} 2023-10-11T15:24:13.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88479a1208c9e3c-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.516 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.517 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.517 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.518 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:24:13.518 DEBUG:boto:Data: 2023-10-11T15:24:13.519 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.520 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.520 DEBUG:boto:Params: {} 2023-10-11T15:24:13.520 DEBUG:boto:Token: None 2023-10-11T15:24:13.521 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55 2023-10-11T15:24:13.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pXKtT93Ui7c3SdBcF4B/ZDzC2jA= 2023-10-11T15:24:13.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pXKtT93Ui7c3SdBcF4B/ZDzC2jA=', 'Content-Length': '0'} 2023-10-11T15:24:13.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d389a27f93f3a0d3-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:24:13.525 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.525 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.526 DEBUG:boto:Method: GET 2023-10-11T15:24:13.526 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:24:13.527 DEBUG:boto:Data: 2023-10-11T15:24:13.527 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.529 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.530 DEBUG:boto:Params: {} 2023-10-11T15:24:13.531 DEBUG:boto:Token: None 2023-10-11T15:24:13.531 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55 2023-10-11T15:24:13.532 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iBDZcJlx8YheScJlNLll/JIFptI= 2023-10-11T15:24:13.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iBDZcJlx8YheScJlNLll/JIFptI=', 'Content-Length': '0'} 2023-10-11T15:24:13.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eeb33b26b73031d-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.536 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:24:13.537 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:24:13.537 DEBUG:boto:Method: GET 2023-10-11T15:24:13.538 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:24:13.538 DEBUG:boto:Data: 2023-10-11T15:24:13.539 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.540 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.540 DEBUG:boto:Params: {} 2023-10-11T15:24:13.540 DEBUG:boto:Token: None 2023-10-11T15:24:13.541 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj55 2023-10-11T15:24:13.542 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iBDZcJlx8YheScJlNLll/JIFptI= 2023-10-11T15:24:13.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iBDZcJlx8YheScJlNLll/JIFptI=', 'Content-Length': '0'} 2023-10-11T15:24:13.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe6f11726248feb-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.590 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.591 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.591 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.592 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.592 DEBUG:boto:Method: GET 2023-10-11T15:24:13.593 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.593 DEBUG:boto:Data: 2023-10-11T15:24:13.594 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.595 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.595 DEBUG:boto:Params: {} 2023-10-11T15:24:13.596 DEBUG:boto:Token: None 2023-10-11T15:24:13.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9458eaea62196d1-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.600 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.601 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.601 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.602 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.602 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.603 DEBUG:boto:Method: GET 2023-10-11T15:24:13.603 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:24:13.603 DEBUG:boto:Data: 2023-10-11T15:24:13.604 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.605 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.605 DEBUG:boto:Params: {} 2023-10-11T15:24:13.606 DEBUG:boto:Token: None 2023-10-11T15:24:13.606 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046102061813c8f25-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.612 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:24:13.613 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.613 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.614 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.614 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.615 DEBUG:boto:Method: GET 2023-10-11T15:24:13.615 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.616 DEBUG:boto:Data: 2023-10-11T15:24:13.616 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.616 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.617 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.617 DEBUG:boto:Params: {} 2023-10-11T15:24:13.618 DEBUG:boto:Token: None 2023-10-11T15:24:13.618 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.619 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o= 2023-10-11T15:24:13.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o=', 'Content-Length': '0'} 2023-10-11T15:24:13.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002749f52733d47dde-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.622 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.623 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.623 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.624 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.624 DEBUG:boto:Method: GET 2023-10-11T15:24:13.625 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.625 DEBUG:boto:Data: 2023-10-11T15:24:13.626 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.626 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.626 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.627 DEBUG:boto:Params: {} 2023-10-11T15:24:13.627 DEBUG:boto:Token: None 2023-10-11T15:24:13.628 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.628 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o= 2023-10-11T15:24:13.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o=', 'Content-Length': '0'} 2023-10-11T15:24:13.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb987973c063815-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.632 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.632 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.633 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.633 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.633 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.634 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.634 DEBUG:boto:Data: 2023-10-11T15:24:13.635 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.635 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.635 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.636 DEBUG:boto:Params: {} 2023-10-11T15:24:13.636 DEBUG:boto:Token: None 2023-10-11T15:24:13.637 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.637 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tMZyL+WW2zNzLvHFYQexhTVaNfg= 2023-10-11T15:24:13.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tMZyL+WW2zNzLvHFYQexhTVaNfg=', 'Content-Length': '0'} 2023-10-11T15:24:13.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007316cd616fb200fa-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.640 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.641 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.641 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.642 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.642 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.643 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.643 DEBUG:boto:Data: 2023-10-11T15:24:13.643 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.644 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.644 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.644 DEBUG:boto:Params: {} 2023-10-11T15:24:13.645 DEBUG:boto:Token: None 2023-10-11T15:24:13.645 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.646 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tMZyL+WW2zNzLvHFYQexhTVaNfg= 2023-10-11T15:24:13.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tMZyL+WW2zNzLvHFYQexhTVaNfg=', 'Content-Length': '0'} 2023-10-11T15:24:13.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092a4853993bd5114-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:24:13.649 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.650 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.650 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.650 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.651 DEBUG:boto:Method: GET 2023-10-11T15:24:13.651 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.652 DEBUG:boto:Data: 2023-10-11T15:24:13.652 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.653 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.653 DEBUG:boto:Params: {} 2023-10-11T15:24:13.654 DEBUG:boto:Token: None 2023-10-11T15:24:13.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o= 2023-10-11T15:24:13.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o=', 'Content-Length': '0'} 2023-10-11T15:24:13.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8312c1aa7cbff0d-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.658 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.658 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.659 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.659 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.660 DEBUG:boto:Method: GET 2023-10-11T15:24:13.660 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.661 DEBUG:boto:Data: 2023-10-11T15:24:13.661 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.662 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.662 DEBUG:boto:Params: {} 2023-10-11T15:24:13.663 DEBUG:boto:Token: None 2023-10-11T15:24:13.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:24:13.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o= 2023-10-11T15:24:13.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lhz546ej5gVcfWiB7274eDYza9o=', 'Content-Length': '0'} 2023-10-11T15:24:13.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3c8d9905f99b35-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.710 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.711 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.711 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.712 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.712 DEBUG:boto:Method: GET 2023-10-11T15:24:13.713 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.713 DEBUG:boto:Data: 2023-10-11T15:24:13.713 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.714 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.715 DEBUG:boto:Params: {} 2023-10-11T15:24:13.715 DEBUG:boto:Token: None 2023-10-11T15:24:13.716 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052bccd32079541ea-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.720 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.720 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.721 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.721 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.722 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.722 DEBUG:boto:Method: GET 2023-10-11T15:24:13.723 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.723 DEBUG:boto:Data: 2023-10-11T15:24:13.723 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.724 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.725 DEBUG:boto:Params: {} 2023-10-11T15:24:13.725 DEBUG:boto:Token: None 2023-10-11T15:24:13.726 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038532dd4d87ffba4-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.731 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.732 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.732 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.732 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.733 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:24:13.733 DEBUG:boto:Data: 2023-10-11T15:24:13.734 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.734 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.734 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.735 DEBUG:boto:Params: {} 2023-10-11T15:24:13.735 DEBUG:boto:Token: None 2023-10-11T15:24:13.736 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57 2023-10-11T15:24:13.736 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HZaSND+C21/4SuCZyuH4S8YOKJo= 2023-10-11T15:24:13.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HZaSND+C21/4SuCZyuH4S8YOKJo=', 'Content-Length': '0'} 2023-10-11T15:24:13.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad654e80f67f24c-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.739 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.740 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.740 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.741 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:24:13.741 DEBUG:boto:Data: 2023-10-11T15:24:13.741 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.742 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.742 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.743 DEBUG:boto:Params: {} 2023-10-11T15:24:13.743 DEBUG:boto:Token: None 2023-10-11T15:24:13.744 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57 2023-10-11T15:24:13.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HZaSND+C21/4SuCZyuH4S8YOKJo= 2023-10-11T15:24:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HZaSND+C21/4SuCZyuH4S8YOKJo=', 'Content-Length': '0'} 2023-10-11T15:24:13.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b87f6f688ade4a4-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:24:13.748 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.748 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.749 DEBUG:boto:Method: GET 2023-10-11T15:24:13.749 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:24:13.749 DEBUG:boto:Data: 2023-10-11T15:24:13.750 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.751 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.751 DEBUG:boto:Params: {} 2023-10-11T15:24:13.752 DEBUG:boto:Token: None 2023-10-11T15:24:13.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57 2023-10-11T15:24:13.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:peOAzGdoEoVFJ0hzPlYT8LHkLr8= 2023-10-11T15:24:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:peOAzGdoEoVFJ0hzPlYT8LHkLr8=', 'Content-Length': '0'} 2023-10-11T15:24:13.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c4fe64db21aab2-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.802 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:24:13.803 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:24:13.803 DEBUG:boto:Method: GET 2023-10-11T15:24:13.804 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:24:13.804 DEBUG:boto:Data: 2023-10-11T15:24:13.805 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.805 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.806 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.806 DEBUG:boto:Params: {} 2023-10-11T15:24:13.807 DEBUG:boto:Token: None 2023-10-11T15:24:13.807 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj57 2023-10-11T15:24:13.808 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:peOAzGdoEoVFJ0hzPlYT8LHkLr8= 2023-10-11T15:24:13.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:peOAzGdoEoVFJ0hzPlYT8LHkLr8=', 'Content-Length': '0'} 2023-10-11T15:24:13.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ad44c7224a6fb4-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.811 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.812 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.813 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.813 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.814 DEBUG:boto:Method: GET 2023-10-11T15:24:13.815 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.815 DEBUG:boto:Data: 2023-10-11T15:24:13.816 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.817 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.817 DEBUG:boto:Params: {} 2023-10-11T15:24:13.818 DEBUG:boto:Token: None 2023-10-11T15:24:13.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c56b5519fd964281-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.823 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.824 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.825 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.826 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.827 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.827 DEBUG:boto:Method: GET 2023-10-11T15:24:13.828 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:24:13.829 DEBUG:boto:Data: 2023-10-11T15:24:13.830 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.831 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.832 DEBUG:boto:Params: {} 2023-10-11T15:24:13.833 DEBUG:boto:Token: None 2023-10-11T15:24:13.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094684bb544e626bc-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.882 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:24:13.884 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.884 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.884 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.885 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.885 DEBUG:boto:Method: GET 2023-10-11T15:24:13.886 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.886 DEBUG:boto:Data: 2023-10-11T15:24:13.887 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.887 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.888 DEBUG:boto:Params: {} 2023-10-11T15:24:13.888 DEBUG:boto:Token: None 2023-10-11T15:24:13.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4= 2023-10-11T15:24:13.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4=', 'Content-Length': '0'} 2023-10-11T15:24:13.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c5e62431fc5b17-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.893 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.894 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.895 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.895 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.896 DEBUG:boto:Method: GET 2023-10-11T15:24:13.897 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.897 DEBUG:boto:Data: 2023-10-11T15:24:13.898 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.899 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.899 DEBUG:boto:Params: {} 2023-10-11T15:24:13.900 DEBUG:boto:Token: None 2023-10-11T15:24:13.901 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4= 2023-10-11T15:24:13.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4=', 'Content-Length': '0'} 2023-10-11T15:24:13.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0e52722c547b04a-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.905 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.905 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.906 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.906 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.907 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.907 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.908 DEBUG:boto:Data: 2023-10-11T15:24:13.908 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.909 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.909 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.909 DEBUG:boto:Params: {} 2023-10-11T15:24:13.910 DEBUG:boto:Token: None 2023-10-11T15:24:13.910 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M2y3L+2sh4Qas6cf4hG2qpV12O0= 2023-10-11T15:24:13.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M2y3L+2sh4Qas6cf4hG2qpV12O0=', 'Content-Length': '0'} 2023-10-11T15:24:13.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000910a5a10391698f0-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.914 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.914 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.915 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.915 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.916 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.916 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.917 DEBUG:boto:Data: 2023-10-11T15:24:13.917 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.918 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.919 DEBUG:boto:Params: {} 2023-10-11T15:24:13.919 DEBUG:boto:Token: None 2023-10-11T15:24:13.920 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M2y3L+2sh4Qas6cf4hG2qpV12O0= 2023-10-11T15:24:13.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M2y3L+2sh4Qas6cf4hG2qpV12O0=', 'Content-Length': '0'} 2023-10-11T15:24:13.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158c8ab9259d1c3a-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:24:13.924 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.924 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.925 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.925 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.926 DEBUG:boto:Method: GET 2023-10-11T15:24:13.926 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.927 DEBUG:boto:Data: 2023-10-11T15:24:13.927 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.928 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.929 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.929 DEBUG:boto:Params: {} 2023-10-11T15:24:13.929 DEBUG:boto:Token: None 2023-10-11T15:24:13.930 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4= 2023-10-11T15:24:13.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4=', 'Content-Length': '0'} 2023-10-11T15:24:13.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036019464eb654ee3-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.935 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.936 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.936 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.937 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.937 DEBUG:boto:Method: GET 2023-10-11T15:24:13.938 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.938 DEBUG:boto:Data: 2023-10-11T15:24:13.939 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.940 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.940 DEBUG:boto:Params: {} 2023-10-11T15:24:13.941 DEBUG:boto:Token: None 2023-10-11T15:24:13.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:24:13.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4= 2023-10-11T15:24:13.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5VNUc8zMa8K0lXjUcNrwQIR5wZ4=', 'Content-Length': '0'} 2023-10-11T15:24:13.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64b88ebd8ea26fd-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.945 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.945 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.946 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.946 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.947 DEBUG:boto:Method: GET 2023-10-11T15:24:13.948 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.948 DEBUG:boto:Data: 2023-10-11T15:24:13.948 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.949 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.950 DEBUG:boto:Params: {} 2023-10-11T15:24:13.950 DEBUG:boto:Token: None 2023-10-11T15:24:13.951 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7929374f1d637e5-006526be1d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.955 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.956 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:13.956 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:13.957 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.958 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.958 DEBUG:boto:Method: GET 2023-10-11T15:24:13.958 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:24:13.959 DEBUG:boto:Data: 2023-10-11T15:24:13.959 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.960 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.961 DEBUG:boto:Params: {} 2023-10-11T15:24:13.961 DEBUG:boto:Token: None 2023-10-11T15:24:13.962 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/ 2023-10-11T15:24:13.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM= 2023-10-11T15:24:13.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kUXI5CqTo+dHt+cWl5kqdiNu3gM=', 'Content-Length': '0'} 2023-10-11T15:24:13.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d8c44b051cd37e-006526be1d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.968 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:13.968 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.969 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.969 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.970 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:24:13.970 DEBUG:boto:Data: 2023-10-11T15:24:13.971 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.972 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.972 DEBUG:boto:Params: {} 2023-10-11T15:24:13.973 DEBUG:boto:Token: None 2023-10-11T15:24:13.973 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58 2023-10-11T15:24:13.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KayFwS4oOt7uI0T4wuauIXfrsvM= 2023-10-11T15:24:13.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KayFwS4oOt7uI0T4wuauIXfrsvM=', 'Content-Length': '0'} 2023-10-11T15:24:13.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001421870ef9ce0da2-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.977 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.977 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.978 DEBUG:boto:Method: HEAD 2023-10-11T15:24:13.979 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:24:13.979 DEBUG:boto:Data: 2023-10-11T15:24:13.979 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:13.980 DEBUG:boto:Port: 8000 2023-10-11T15:24:13.981 DEBUG:boto:Params: {} 2023-10-11T15:24:13.981 DEBUG:boto:Token: None 2023-10-11T15:24:13.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58 2023-10-11T15:24:13.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KayFwS4oOt7uI0T4wuauIXfrsvM= 2023-10-11T15:24:13.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KayFwS4oOt7uI0T4wuauIXfrsvM=', 'Content-Length': '0'} 2023-10-11T15:24:13.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8672d47fa02af55-006526be1d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:13.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:24:13.986 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:13.986 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:13.987 DEBUG:boto:Method: GET 2023-10-11T15:24:13.987 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:24:13.987 DEBUG:boto:Data: 2023-10-11T15:24:13.988 DEBUG:boto:Headers: {} 2023-10-11T15:24:13.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:13.989 DEBUG:boto:Port: 8001 2023-10-11T15:24:13.990 DEBUG:boto:Params: {} 2023-10-11T15:24:13.990 DEBUG:boto:Token: None 2023-10-11T15:24:13.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:13 GMT /dghqvd-59/obj58 2023-10-11T15:24:13.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+YqZfO0gYXnpQD0/kdZR+n3QPAU= 2023-10-11T15:24:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+YqZfO0gYXnpQD0/kdZR+n3QPAU=', 'Content-Length': '0'} 2023-10-11T15:24:13.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65bf17b23297cfa-006526be1d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.034 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:24:14.035 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:24:14.035 DEBUG:boto:Method: GET 2023-10-11T15:24:14.036 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:24:14.036 DEBUG:boto:Data: 2023-10-11T15:24:14.037 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.037 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.038 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.038 DEBUG:boto:Params: {} 2023-10-11T15:24:14.039 DEBUG:boto:Token: None 2023-10-11T15:24:14.039 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj58 2023-10-11T15:24:14.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a8SdBckhGTB9tYfz+9JzKnzgoIo= 2023-10-11T15:24:14.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a8SdBckhGTB9tYfz+9JzKnzgoIo=', 'Content-Length': '0'} 2023-10-11T15:24:14.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ee22ca50a678ea-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.043 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.044 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.044 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.044 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.045 DEBUG:boto:Method: GET 2023-10-11T15:24:14.045 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.046 DEBUG:boto:Data: 2023-10-11T15:24:14.046 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.047 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.048 DEBUG:boto:Params: {} 2023-10-11T15:24:14.048 DEBUG:boto:Token: None 2023-10-11T15:24:14.049 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a362968d2d8c6089-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.053 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.054 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.054 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.055 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.055 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.055 DEBUG:boto:Method: GET 2023-10-11T15:24:14.056 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:24:14.056 DEBUG:boto:Data: 2023-10-11T15:24:14.057 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.058 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.058 DEBUG:boto:Params: {} 2023-10-11T15:24:14.059 DEBUG:boto:Token: None 2023-10-11T15:24:14.059 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e09602b3b0fb23-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.065 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:24:14.066 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.066 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.067 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.067 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.068 DEBUG:boto:Method: GET 2023-10-11T15:24:14.068 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.069 DEBUG:boto:Data: 2023-10-11T15:24:14.069 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.069 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.070 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.070 DEBUG:boto:Params: {} 2023-10-11T15:24:14.071 DEBUG:boto:Token: None 2023-10-11T15:24:14.071 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU= 2023-10-11T15:24:14.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU=', 'Content-Length': '0'} 2023-10-11T15:24:14.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ab1b6ff0820264-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.076 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.076 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.077 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.077 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.078 DEBUG:boto:Method: GET 2023-10-11T15:24:14.078 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.079 DEBUG:boto:Data: 2023-10-11T15:24:14.079 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.080 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.081 DEBUG:boto:Params: {} 2023-10-11T15:24:14.081 DEBUG:boto:Token: None 2023-10-11T15:24:14.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU= 2023-10-11T15:24:14.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU=', 'Content-Length': '0'} 2023-10-11T15:24:14.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8b602d1c82deac-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.086 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.086 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.087 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.087 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.088 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.088 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.089 DEBUG:boto:Data: 2023-10-11T15:24:14.089 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.090 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.090 DEBUG:boto:Params: {} 2023-10-11T15:24:14.091 DEBUG:boto:Token: None 2023-10-11T15:24:14.091 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fQqO4BX82URBu3bVwWAhirLeEgI= 2023-10-11T15:24:14.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQqO4BX82URBu3bVwWAhirLeEgI=', 'Content-Length': '0'} 2023-10-11T15:24:14.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4f9de22ad1ae76-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.095 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.096 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.096 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.097 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.097 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.098 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.098 DEBUG:boto:Data: 2023-10-11T15:24:14.098 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.099 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.100 DEBUG:boto:Params: {} 2023-10-11T15:24:14.100 DEBUG:boto:Token: None 2023-10-11T15:24:14.101 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fQqO4BX82URBu3bVwWAhirLeEgI= 2023-10-11T15:24:14.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQqO4BX82URBu3bVwWAhirLeEgI=', 'Content-Length': '0'} 2023-10-11T15:24:14.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891474d997b76b85-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:24:14.105 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.106 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.106 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.107 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.107 DEBUG:boto:Method: GET 2023-10-11T15:24:14.107 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.108 DEBUG:boto:Data: 2023-10-11T15:24:14.108 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.109 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.110 DEBUG:boto:Params: {} 2023-10-11T15:24:14.110 DEBUG:boto:Token: None 2023-10-11T15:24:14.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU= 2023-10-11T15:24:14.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU=', 'Content-Length': '0'} 2023-10-11T15:24:14.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa959e4dff457fb7-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.154 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.155 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.155 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.156 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.156 DEBUG:boto:Method: GET 2023-10-11T15:24:14.157 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.157 DEBUG:boto:Data: 2023-10-11T15:24:14.157 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.158 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.159 DEBUG:boto:Params: {} 2023-10-11T15:24:14.159 DEBUG:boto:Token: None 2023-10-11T15:24:14.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:24:14.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU= 2023-10-11T15:24:14.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yYJiNcFnrTWZfm3gCqPlQlWilNU=', 'Content-Length': '0'} 2023-10-11T15:24:14.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d707a32114601c-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.164 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.164 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.165 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.165 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.166 DEBUG:boto:Method: GET 2023-10-11T15:24:14.166 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.167 DEBUG:boto:Data: 2023-10-11T15:24:14.167 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.168 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.169 DEBUG:boto:Params: {} 2023-10-11T15:24:14.169 DEBUG:boto:Token: None 2023-10-11T15:24:14.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.170 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ce04cb500290d9-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.174 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.175 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.175 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.176 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.176 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.177 DEBUG:boto:Method: GET 2023-10-11T15:24:14.177 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.178 DEBUG:boto:Data: 2023-10-11T15:24:14.178 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.179 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.179 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.180 DEBUG:boto:Params: {} 2023-10-11T15:24:14.180 DEBUG:boto:Token: None 2023-10-11T15:24:14.181 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ce6ca0b143357c-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.226 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.227 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.227 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.228 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.228 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:24:14.229 DEBUG:boto:Data: 2023-10-11T15:24:14.229 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.230 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.231 DEBUG:boto:Params: {} 2023-10-11T15:24:14.231 DEBUG:boto:Token: None 2023-10-11T15:24:14.232 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68 2023-10-11T15:24:14.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mnHS9yonT5sCdo5+TPyAPpn27Lk= 2023-10-11T15:24:14.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mnHS9yonT5sCdo5+TPyAPpn27Lk=', 'Content-Length': '0'} 2023-10-11T15:24:14.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f29603f0e62ff3-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.235 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.236 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.236 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.237 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:24:14.237 DEBUG:boto:Data: 2023-10-11T15:24:14.237 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.238 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.238 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.239 DEBUG:boto:Params: {} 2023-10-11T15:24:14.239 DEBUG:boto:Token: None 2023-10-11T15:24:14.240 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68 2023-10-11T15:24:14.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mnHS9yonT5sCdo5+TPyAPpn27Lk= 2023-10-11T15:24:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mnHS9yonT5sCdo5+TPyAPpn27Lk=', 'Content-Length': '0'} 2023-10-11T15:24:14.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8634eb9be010c27-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:24:14.244 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.244 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.244 DEBUG:boto:Method: GET 2023-10-11T15:24:14.245 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:24:14.246 DEBUG:boto:Data: 2023-10-11T15:24:14.247 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.247 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.248 DEBUG:boto:Params: {} 2023-10-11T15:24:14.248 DEBUG:boto:Token: None 2023-10-11T15:24:14.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68 2023-10-11T15:24:14.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kv7CLsRzw/4xEhS7nKJ4rDsnL5E= 2023-10-11T15:24:14.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kv7CLsRzw/4xEhS7nKJ4rDsnL5E=', 'Content-Length': '0'} 2023-10-11T15:24:14.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa3158ef92cba06-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.252 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:24:14.253 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:24:14.253 DEBUG:boto:Method: GET 2023-10-11T15:24:14.254 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:24:14.254 DEBUG:boto:Data: 2023-10-11T15:24:14.255 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.255 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.256 DEBUG:boto:Params: {} 2023-10-11T15:24:14.256 DEBUG:boto:Token: None 2023-10-11T15:24:14.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj68 2023-10-11T15:24:14.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kv7CLsRzw/4xEhS7nKJ4rDsnL5E= 2023-10-11T15:24:14.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kv7CLsRzw/4xEhS7nKJ4rDsnL5E=', 'Content-Length': '0'} 2023-10-11T15:24:14.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3923be51cee342a-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.261 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.261 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.262 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.262 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.262 DEBUG:boto:Method: GET 2023-10-11T15:24:14.263 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.263 DEBUG:boto:Data: 2023-10-11T15:24:14.264 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.264 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.265 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.265 DEBUG:boto:Params: {} 2023-10-11T15:24:14.266 DEBUG:boto:Token: None 2023-10-11T15:24:14.266 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000481426f725c890-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.270 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.271 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.271 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.272 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.272 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.272 DEBUG:boto:Method: GET 2023-10-11T15:24:14.273 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:24:14.273 DEBUG:boto:Data: 2023-10-11T15:24:14.274 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.274 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.275 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.275 DEBUG:boto:Params: {} 2023-10-11T15:24:14.276 DEBUG:boto:Token: None 2023-10-11T15:24:14.276 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.276 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e751b7dbaeb8375b-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.281 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:24:14.283 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.283 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.283 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.284 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.284 DEBUG:boto:Method: GET 2023-10-11T15:24:14.285 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.285 DEBUG:boto:Data: 2023-10-11T15:24:14.286 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.286 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.287 DEBUG:boto:Params: {} 2023-10-11T15:24:14.287 DEBUG:boto:Token: None 2023-10-11T15:24:14.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE= 2023-10-11T15:24:14.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE=', 'Content-Length': '0'} 2023-10-11T15:24:14.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c27d3ef84f73042-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.292 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.292 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.293 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.293 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.294 DEBUG:boto:Method: GET 2023-10-11T15:24:14.294 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.295 DEBUG:boto:Data: 2023-10-11T15:24:14.295 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.296 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.296 DEBUG:boto:Params: {} 2023-10-11T15:24:14.297 DEBUG:boto:Token: None 2023-10-11T15:24:14.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE= 2023-10-11T15:24:14.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE=', 'Content-Length': '0'} 2023-10-11T15:24:14.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4cf3f08b12d4eca-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.342 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.343 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.343 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.344 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.344 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.345 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.345 DEBUG:boto:Data: 2023-10-11T15:24:14.345 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.346 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.347 DEBUG:boto:Params: {} 2023-10-11T15:24:14.347 DEBUG:boto:Token: None 2023-10-11T15:24:14.348 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CH8NPG1RUpgyoNr2f3yvB/o/u84= 2023-10-11T15:24:14.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CH8NPG1RUpgyoNr2f3yvB/o/u84=', 'Content-Length': '0'} 2023-10-11T15:24:14.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa2ba24984c31ac-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.351 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.352 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.352 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.353 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.353 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.353 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.354 DEBUG:boto:Data: 2023-10-11T15:24:14.354 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.355 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.355 DEBUG:boto:Params: {} 2023-10-11T15:24:14.356 DEBUG:boto:Token: None 2023-10-11T15:24:14.356 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CH8NPG1RUpgyoNr2f3yvB/o/u84= 2023-10-11T15:24:14.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CH8NPG1RUpgyoNr2f3yvB/o/u84=', 'Content-Length': '0'} 2023-10-11T15:24:14.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfb651dbb69f306-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:24:14.361 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.361 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.362 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.362 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.363 DEBUG:boto:Method: GET 2023-10-11T15:24:14.363 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.363 DEBUG:boto:Data: 2023-10-11T15:24:14.364 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.364 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.365 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.365 DEBUG:boto:Params: {} 2023-10-11T15:24:14.366 DEBUG:boto:Token: None 2023-10-11T15:24:14.366 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.367 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE= 2023-10-11T15:24:14.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE=', 'Content-Length': '0'} 2023-10-11T15:24:14.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d0d965f59fbe44-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.410 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.411 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.411 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.412 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.412 DEBUG:boto:Method: GET 2023-10-11T15:24:14.413 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.413 DEBUG:boto:Data: 2023-10-11T15:24:14.414 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.415 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.415 DEBUG:boto:Params: {} 2023-10-11T15:24:14.416 DEBUG:boto:Token: None 2023-10-11T15:24:14.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:24:14.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE= 2023-10-11T15:24:14.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kf8IF4QJpEL8K+E3UxUVBUE0PWE=', 'Content-Length': '0'} 2023-10-11T15:24:14.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e9f8ecbe04a583-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.420 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.420 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.421 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.421 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.422 DEBUG:boto:Method: GET 2023-10-11T15:24:14.422 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.423 DEBUG:boto:Data: 2023-10-11T15:24:14.423 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.424 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.424 DEBUG:boto:Params: {} 2023-10-11T15:24:14.425 DEBUG:boto:Token: None 2023-10-11T15:24:14.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d053ffd13449dca-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.430 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.430 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.431 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.431 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.432 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.432 DEBUG:boto:Method: GET 2023-10-11T15:24:14.433 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.433 DEBUG:boto:Data: 2023-10-11T15:24:14.434 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.435 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.435 DEBUG:boto:Params: {} 2023-10-11T15:24:14.435 DEBUG:boto:Token: None 2023-10-11T15:24:14.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ee4fd9bfc37063-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.441 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.442 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.443 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.443 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.443 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:24:14.444 DEBUG:boto:Data: 2023-10-11T15:24:14.444 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.445 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.445 DEBUG:boto:Params: {} 2023-10-11T15:24:14.446 DEBUG:boto:Token: None 2023-10-11T15:24:14.446 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69 2023-10-11T15:24:14.447 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Dqewbk4/VbkpX5KgK4WvUP3+pq8= 2023-10-11T15:24:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Dqewbk4/VbkpX5KgK4WvUP3+pq8=', 'Content-Length': '0'} 2023-10-11T15:24:14.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3ef948a3c73862-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.450 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.450 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.451 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.451 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:24:14.452 DEBUG:boto:Data: 2023-10-11T15:24:14.452 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.453 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.453 DEBUG:boto:Params: {} 2023-10-11T15:24:14.454 DEBUG:boto:Token: None 2023-10-11T15:24:14.454 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69 2023-10-11T15:24:14.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Dqewbk4/VbkpX5KgK4WvUP3+pq8= 2023-10-11T15:24:14.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Dqewbk4/VbkpX5KgK4WvUP3+pq8=', 'Content-Length': '0'} 2023-10-11T15:24:14.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824523e7895e2895-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:24:14.458 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.458 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.459 DEBUG:boto:Method: GET 2023-10-11T15:24:14.459 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:24:14.460 DEBUG:boto:Data: 2023-10-11T15:24:14.460 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.461 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.461 DEBUG:boto:Params: {} 2023-10-11T15:24:14.462 DEBUG:boto:Token: None 2023-10-11T15:24:14.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69 2023-10-11T15:24:14.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V8hE5TRVbYVY7xNpXpVOsQ252vI= 2023-10-11T15:24:14.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V8hE5TRVbYVY7xNpXpVOsQ252vI=', 'Content-Length': '0'} 2023-10-11T15:24:14.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e0f87f4632d690-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.466 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:24:14.467 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:24:14.467 DEBUG:boto:Method: GET 2023-10-11T15:24:14.468 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:24:14.468 DEBUG:boto:Data: 2023-10-11T15:24:14.468 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.469 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.470 DEBUG:boto:Params: {} 2023-10-11T15:24:14.470 DEBUG:boto:Token: None 2023-10-11T15:24:14.471 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj69 2023-10-11T15:24:14.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V8hE5TRVbYVY7xNpXpVOsQ252vI= 2023-10-11T15:24:14.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V8hE5TRVbYVY7xNpXpVOsQ252vI=', 'Content-Length': '0'} 2023-10-11T15:24:14.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1acf8b0ff45cd50-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.514 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.515 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.516 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.516 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.517 DEBUG:boto:Method: GET 2023-10-11T15:24:14.517 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.517 DEBUG:boto:Data: 2023-10-11T15:24:14.518 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.519 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.519 DEBUG:boto:Params: {} 2023-10-11T15:24:14.520 DEBUG:boto:Token: None 2023-10-11T15:24:14.520 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007870eb771923b92e-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.525 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.526 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.527 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.527 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.528 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.529 DEBUG:boto:Method: GET 2023-10-11T15:24:14.529 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:24:14.530 DEBUG:boto:Data: 2023-10-11T15:24:14.530 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.531 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.531 DEBUG:boto:Params: {} 2023-10-11T15:24:14.532 DEBUG:boto:Token: None 2023-10-11T15:24:14.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd52d3afdc47b462-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.539 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:24:14.540 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.541 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.542 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.543 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.544 DEBUG:boto:Method: GET 2023-10-11T15:24:14.544 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.545 DEBUG:boto:Data: 2023-10-11T15:24:14.546 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.547 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.550 DEBUG:boto:Params: {} 2023-10-11T15:24:14.551 DEBUG:boto:Token: None 2023-10-11T15:24:14.552 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ= 2023-10-11T15:24:14.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ=', 'Content-Length': '0'} 2023-10-11T15:24:14.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066793715a63b9e1-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.557 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.558 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.559 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.560 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.560 DEBUG:boto:Method: GET 2023-10-11T15:24:14.561 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.562 DEBUG:boto:Data: 2023-10-11T15:24:14.562 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.564 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.564 DEBUG:boto:Params: {} 2023-10-11T15:24:14.565 DEBUG:boto:Token: None 2023-10-11T15:24:14.566 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ= 2023-10-11T15:24:14.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ=', 'Content-Length': '0'} 2023-10-11T15:24:14.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1ce1b53dd98cd6-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.571 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.572 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.572 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.573 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.573 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.573 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.574 DEBUG:boto:Data: 2023-10-11T15:24:14.574 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.575 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.575 DEBUG:boto:Params: {} 2023-10-11T15:24:14.576 DEBUG:boto:Token: None 2023-10-11T15:24:14.576 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ukVFr9J8Ffri0w+rnjStsgGS7sY= 2023-10-11T15:24:14.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ukVFr9J8Ffri0w+rnjStsgGS7sY=', 'Content-Length': '0'} 2023-10-11T15:24:14.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc7047060ccffbec-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.580 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.581 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.581 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.582 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.582 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.582 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.583 DEBUG:boto:Data: 2023-10-11T15:24:14.583 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.584 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.584 DEBUG:boto:Params: {} 2023-10-11T15:24:14.585 DEBUG:boto:Token: None 2023-10-11T15:24:14.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ukVFr9J8Ffri0w+rnjStsgGS7sY= 2023-10-11T15:24:14.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ukVFr9J8Ffri0w+rnjStsgGS7sY=', 'Content-Length': '0'} 2023-10-11T15:24:14.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c8f045d509470a7-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:24:14.589 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.590 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.590 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.591 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.591 DEBUG:boto:Method: GET 2023-10-11T15:24:14.592 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.592 DEBUG:boto:Data: 2023-10-11T15:24:14.592 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.593 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.594 DEBUG:boto:Params: {} 2023-10-11T15:24:14.594 DEBUG:boto:Token: None 2023-10-11T15:24:14.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ= 2023-10-11T15:24:14.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ=', 'Content-Length': '0'} 2023-10-11T15:24:14.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ab4be169584c2a2-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.599 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.599 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.599 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.600 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.600 DEBUG:boto:Method: GET 2023-10-11T15:24:14.601 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.601 DEBUG:boto:Data: 2023-10-11T15:24:14.602 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.602 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.603 DEBUG:boto:Params: {} 2023-10-11T15:24:14.603 DEBUG:boto:Token: None 2023-10-11T15:24:14.604 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:24:14.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ= 2023-10-11T15:24:14.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OM6zoXtdYs+WiWq88fxxXqPdiLQ=', 'Content-Length': '0'} 2023-10-11T15:24:14.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea808e2c11ed5f64-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.608 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.608 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.609 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.609 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.609 DEBUG:boto:Method: GET 2023-10-11T15:24:14.610 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.610 DEBUG:boto:Data: 2023-10-11T15:24:14.611 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.612 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.612 DEBUG:boto:Params: {} 2023-10-11T15:24:14.612 DEBUG:boto:Token: None 2023-10-11T15:24:14.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99e00e1e222a280-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.617 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.618 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.618 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.619 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.619 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.620 DEBUG:boto:Method: GET 2023-10-11T15:24:14.620 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.620 DEBUG:boto:Data: 2023-10-11T15:24:14.621 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.622 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.622 DEBUG:boto:Params: {} 2023-10-11T15:24:14.623 DEBUG:boto:Token: None 2023-10-11T15:24:14.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e69707fff0cc850-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.670 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.671 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.672 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.672 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.673 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:24:14.673 DEBUG:boto:Data: 2023-10-11T15:24:14.674 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.674 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.675 DEBUG:boto:Params: {} 2023-10-11T15:24:14.675 DEBUG:boto:Token: None 2023-10-11T15:24:14.676 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76 2023-10-11T15:24:14.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ziuqoBsudGvnJjtfCMDbT10gXWU= 2023-10-11T15:24:14.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ziuqoBsudGvnJjtfCMDbT10gXWU=', 'Content-Length': '0'} 2023-10-11T15:24:14.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2c1e20182d7092-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.680 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.680 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.681 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.681 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:24:14.682 DEBUG:boto:Data: 2023-10-11T15:24:14.682 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.683 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.683 DEBUG:boto:Params: {} 2023-10-11T15:24:14.684 DEBUG:boto:Token: None 2023-10-11T15:24:14.684 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76 2023-10-11T15:24:14.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ziuqoBsudGvnJjtfCMDbT10gXWU= 2023-10-11T15:24:14.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ziuqoBsudGvnJjtfCMDbT10gXWU=', 'Content-Length': '0'} 2023-10-11T15:24:14.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8f3157f3a4a854-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:24:14.688 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.689 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.689 DEBUG:boto:Method: GET 2023-10-11T15:24:14.690 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:24:14.690 DEBUG:boto:Data: 2023-10-11T15:24:14.691 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.691 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.692 DEBUG:boto:Params: {} 2023-10-11T15:24:14.692 DEBUG:boto:Token: None 2023-10-11T15:24:14.693 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76 2023-10-11T15:24:14.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nrLLRVSNcTSevlO4C992RKZ0mpE= 2023-10-11T15:24:14.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nrLLRVSNcTSevlO4C992RKZ0mpE=', 'Content-Length': '0'} 2023-10-11T15:24:14.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d464c429f5859c7d-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.697 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:24:14.697 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:24:14.698 DEBUG:boto:Method: GET 2023-10-11T15:24:14.698 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:24:14.698 DEBUG:boto:Data: 2023-10-11T15:24:14.699 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.700 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.700 DEBUG:boto:Params: {} 2023-10-11T15:24:14.701 DEBUG:boto:Token: None 2023-10-11T15:24:14.701 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj76 2023-10-11T15:24:14.702 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nrLLRVSNcTSevlO4C992RKZ0mpE= 2023-10-11T15:24:14.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nrLLRVSNcTSevlO4C992RKZ0mpE=', 'Content-Length': '0'} 2023-10-11T15:24:14.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008524d7b157084046-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.705 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.705 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.706 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.706 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.707 DEBUG:boto:Method: GET 2023-10-11T15:24:14.707 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.707 DEBUG:boto:Data: 2023-10-11T15:24:14.708 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.709 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.709 DEBUG:boto:Params: {} 2023-10-11T15:24:14.710 DEBUG:boto:Token: None 2023-10-11T15:24:14.710 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000550fc761a997e487-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.714 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.715 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.715 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.716 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.716 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.717 DEBUG:boto:Method: GET 2023-10-11T15:24:14.717 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:24:14.718 DEBUG:boto:Data: 2023-10-11T15:24:14.718 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.719 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.719 DEBUG:boto:Params: {} 2023-10-11T15:24:14.720 DEBUG:boto:Token: None 2023-10-11T15:24:14.720 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037fb4fe5c8888734-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.726 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:24:14.727 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.728 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.728 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.728 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.729 DEBUG:boto:Method: GET 2023-10-11T15:24:14.729 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.730 DEBUG:boto:Data: 2023-10-11T15:24:14.730 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.731 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.731 DEBUG:boto:Params: {} 2023-10-11T15:24:14.732 DEBUG:boto:Token: None 2023-10-11T15:24:14.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI= 2023-10-11T15:24:14.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI=', 'Content-Length': '0'} 2023-10-11T15:24:14.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a44c3eec8244cae-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.736 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.737 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.737 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.738 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.738 DEBUG:boto:Method: GET 2023-10-11T15:24:14.738 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.739 DEBUG:boto:Data: 2023-10-11T15:24:14.739 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.740 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.741 DEBUG:boto:Params: {} 2023-10-11T15:24:14.741 DEBUG:boto:Token: None 2023-10-11T15:24:14.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI= 2023-10-11T15:24:14.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI=', 'Content-Length': '0'} 2023-10-11T15:24:14.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b77082af9a4a4a-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.790 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.791 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.791 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.792 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.792 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.793 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.793 DEBUG:boto:Data: 2023-10-11T15:24:14.793 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.794 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.795 DEBUG:boto:Params: {} 2023-10-11T15:24:14.795 DEBUG:boto:Token: None 2023-10-11T15:24:14.796 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sGDTrbgIyDQSFXrWqQj7LmYqrjs= 2023-10-11T15:24:14.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sGDTrbgIyDQSFXrWqQj7LmYqrjs=', 'Content-Length': '0'} 2023-10-11T15:24:14.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa88b621e0bbdef2-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.799 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.800 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.800 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.801 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.801 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.801 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.802 DEBUG:boto:Data: 2023-10-11T15:24:14.802 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.803 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.803 DEBUG:boto:Params: {} 2023-10-11T15:24:14.804 DEBUG:boto:Token: None 2023-10-11T15:24:14.804 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.805 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sGDTrbgIyDQSFXrWqQj7LmYqrjs= 2023-10-11T15:24:14.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sGDTrbgIyDQSFXrWqQj7LmYqrjs=', 'Content-Length': '0'} 2023-10-11T15:24:14.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db8839e42b1f084-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:24:14.808 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.809 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.809 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.810 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.810 DEBUG:boto:Method: GET 2023-10-11T15:24:14.811 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.811 DEBUG:boto:Data: 2023-10-11T15:24:14.811 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.812 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.812 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.813 DEBUG:boto:Params: {} 2023-10-11T15:24:14.813 DEBUG:boto:Token: None 2023-10-11T15:24:14.814 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI= 2023-10-11T15:24:14.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI=', 'Content-Length': '0'} 2023-10-11T15:24:14.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000469c60396e728c25-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.858 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.859 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.860 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.860 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.860 DEBUG:boto:Method: GET 2023-10-11T15:24:14.861 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.861 DEBUG:boto:Data: 2023-10-11T15:24:14.862 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.863 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.863 DEBUG:boto:Params: {} 2023-10-11T15:24:14.864 DEBUG:boto:Token: None 2023-10-11T15:24:14.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:24:14.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI= 2023-10-11T15:24:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BY9xDmY4zqBYFn3I/5AyF/lIMGI=', 'Content-Length': '0'} 2023-10-11T15:24:14.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae4c5bbd752c6c4-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.868 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.868 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.869 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.869 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.870 DEBUG:boto:Method: GET 2023-10-11T15:24:14.870 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.871 DEBUG:boto:Data: 2023-10-11T15:24:14.871 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.872 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.872 DEBUG:boto:Params: {} 2023-10-11T15:24:14.873 DEBUG:boto:Token: None 2023-10-11T15:24:14.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc8b42ae43884de-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.877 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.878 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.878 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.879 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.879 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.880 DEBUG:boto:Method: GET 2023-10-11T15:24:14.880 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.880 DEBUG:boto:Data: 2023-10-11T15:24:14.881 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.882 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.882 DEBUG:boto:Params: {} 2023-10-11T15:24:14.883 DEBUG:boto:Token: None 2023-10-11T15:24:14.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f588b547f1d1f2-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.889 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.889 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.890 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.890 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.890 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:24:14.891 DEBUG:boto:Data: 2023-10-11T15:24:14.891 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.892 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.892 DEBUG:boto:Params: {} 2023-10-11T15:24:14.893 DEBUG:boto:Token: None 2023-10-11T15:24:14.893 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85 2023-10-11T15:24:14.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/teqUQn7lTMh61pLHEei7jgSQ+g= 2023-10-11T15:24:14.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/teqUQn7lTMh61pLHEei7jgSQ+g=', 'Content-Length': '0'} 2023-10-11T15:24:14.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ef0852405e69a5-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.897 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.897 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.898 DEBUG:boto:Method: HEAD 2023-10-11T15:24:14.898 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:24:14.899 DEBUG:boto:Data: 2023-10-11T15:24:14.899 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.900 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.900 DEBUG:boto:Params: {} 2023-10-11T15:24:14.901 DEBUG:boto:Token: None 2023-10-11T15:24:14.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85 2023-10-11T15:24:14.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/teqUQn7lTMh61pLHEei7jgSQ+g= 2023-10-11T15:24:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/teqUQn7lTMh61pLHEei7jgSQ+g=', 'Content-Length': '0'} 2023-10-11T15:24:14.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd7b65036a75e16-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:24:14.905 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.906 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.906 DEBUG:boto:Method: GET 2023-10-11T15:24:14.906 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:24:14.907 DEBUG:boto:Data: 2023-10-11T15:24:14.907 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.908 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.909 DEBUG:boto:Params: {} 2023-10-11T15:24:14.909 DEBUG:boto:Token: None 2023-10-11T15:24:14.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85 2023-10-11T15:24:14.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y1Z/Y/81gNLFHQENQFvRfZAIjPc= 2023-10-11T15:24:14.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y1Z/Y/81gNLFHQENQFvRfZAIjPc=', 'Content-Length': '0'} 2023-10-11T15:24:14.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda09e22015beb86-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.913 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:24:14.914 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:24:14.914 DEBUG:boto:Method: GET 2023-10-11T15:24:14.915 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:24:14.915 DEBUG:boto:Data: 2023-10-11T15:24:14.915 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.916 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.917 DEBUG:boto:Params: {} 2023-10-11T15:24:14.917 DEBUG:boto:Token: None 2023-10-11T15:24:14.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj85 2023-10-11T15:24:14.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:y1Z/Y/81gNLFHQENQFvRfZAIjPc= 2023-10-11T15:24:14.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:y1Z/Y/81gNLFHQENQFvRfZAIjPc=', 'Content-Length': '0'} 2023-10-11T15:24:14.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f78fd28829a91f-006526be1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.966 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.967 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.968 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.968 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.969 DEBUG:boto:Method: GET 2023-10-11T15:24:14.970 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.971 DEBUG:boto:Data: 2023-10-11T15:24:14.971 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.972 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.972 DEBUG:boto:Params: {} 2023-10-11T15:24:14.973 DEBUG:boto:Token: None 2023-10-11T15:24:14.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9a0f6f586ed662-006526be1e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.977 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.978 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:14.979 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:14.979 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.980 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.980 DEBUG:boto:Method: GET 2023-10-11T15:24:14.981 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:24:14.981 DEBUG:boto:Data: 2023-10-11T15:24:14.981 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:14.982 DEBUG:boto:Port: 8000 2023-10-11T15:24:14.983 DEBUG:boto:Params: {} 2023-10-11T15:24:14.983 DEBUG:boto:Token: None 2023-10-11T15:24:14.984 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/ 2023-10-11T15:24:14.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI= 2023-10-11T15:24:14.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hUm9UNXY6jpVXmrI+7TUEpF7NHI=', 'Content-Length': '0'} 2023-10-11T15:24:14.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000430a179a8b002d85-006526be1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:14.990 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:14.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:24:14.991 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:14.991 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:14.992 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:14.992 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:14.992 DEBUG:boto:Method: GET 2023-10-11T15:24:14.993 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:14.993 DEBUG:boto:Data: 2023-10-11T15:24:14.994 DEBUG:boto:Headers: {} 2023-10-11T15:24:14.994 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:14.995 DEBUG:boto:Port: 8001 2023-10-11T15:24:14.995 DEBUG:boto:Params: {} 2023-10-11T15:24:14.996 DEBUG:boto:Token: None 2023-10-11T15:24:14.996 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:14 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:14.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mZMlhQzzj4Qjb+0LnzgUyLhMWDw= 2023-10-11T15:24:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mZMlhQzzj4Qjb+0LnzgUyLhMWDw=', 'Content-Length': '0'} 2023-10-11T15:24:15.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008ceec356eaff2a6-006526be1e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.000 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.001 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.001 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.002 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.002 DEBUG:boto:Method: GET 2023-10-11T15:24:15.002 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.003 DEBUG:boto:Data: 2023-10-11T15:24:15.003 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.004 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.004 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.005 DEBUG:boto:Params: {} 2023-10-11T15:24:15.005 DEBUG:boto:Token: None 2023-10-11T15:24:15.006 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.006 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34= 2023-10-11T15:24:15.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34=', 'Content-Length': '0'} 2023-10-11T15:24:15.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1070b5c06dcc37d-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.009 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.010 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.010 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.011 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.011 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.012 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.012 DEBUG:boto:Data: 2023-10-11T15:24:15.012 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.013 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.014 DEBUG:boto:Params: {} 2023-10-11T15:24:15.014 DEBUG:boto:Token: None 2023-10-11T15:24:15.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bfXHeDDF5jjCJPw64hqm/dn0S8Y= 2023-10-11T15:24:15.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bfXHeDDF5jjCJPw64hqm/dn0S8Y=', 'Content-Length': '0'} 2023-10-11T15:24:15.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbdc5cbd112a952a-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.019 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.019 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.019 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.020 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.020 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.021 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.021 DEBUG:boto:Data: 2023-10-11T15:24:15.021 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.022 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.023 DEBUG:boto:Params: {} 2023-10-11T15:24:15.023 DEBUG:boto:Token: None 2023-10-11T15:24:15.024 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bfXHeDDF5jjCJPw64hqm/dn0S8Y= 2023-10-11T15:24:15.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bfXHeDDF5jjCJPw64hqm/dn0S8Y=', 'Content-Length': '0'} 2023-10-11T15:24:15.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6292578363c478-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:24:15.028 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.028 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.029 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.029 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.030 DEBUG:boto:Method: GET 2023-10-11T15:24:15.030 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.031 DEBUG:boto:Data: 2023-10-11T15:24:15.031 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.032 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.032 DEBUG:boto:Params: {} 2023-10-11T15:24:15.033 DEBUG:boto:Token: None 2023-10-11T15:24:15.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34= 2023-10-11T15:24:15.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34=', 'Content-Length': '0'} 2023-10-11T15:24:15.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9fae657069be45-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.037 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.037 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.038 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.038 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.039 DEBUG:boto:Method: GET 2023-10-11T15:24:15.039 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.040 DEBUG:boto:Data: 2023-10-11T15:24:15.040 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.041 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.041 DEBUG:boto:Params: {} 2023-10-11T15:24:15.042 DEBUG:boto:Token: None 2023-10-11T15:24:15.042 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:24:15.043 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34= 2023-10-11T15:24:15.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81hCtODXgGSlzhvSXf9IdnN0e34=', 'Content-Length': '0'} 2023-10-11T15:24:15.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c65271678403bf7-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.086 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.087 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.088 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.088 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.088 DEBUG:boto:Method: GET 2023-10-11T15:24:15.089 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.089 DEBUG:boto:Data: 2023-10-11T15:24:15.090 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.090 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.091 DEBUG:boto:Params: {} 2023-10-11T15:24:15.092 DEBUG:boto:Token: None 2023-10-11T15:24:15.092 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000705be0c1fe93f992-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.097 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.097 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.098 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.098 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.099 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.099 DEBUG:boto:Method: GET 2023-10-11T15:24:15.099 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.100 DEBUG:boto:Data: 2023-10-11T15:24:15.100 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.101 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.101 DEBUG:boto:Params: {} 2023-10-11T15:24:15.102 DEBUG:boto:Token: None 2023-10-11T15:24:15.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c70208f891311df3-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.108 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.109 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.109 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.110 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.110 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:24:15.111 DEBUG:boto:Data: 2023-10-11T15:24:15.111 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.112 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.112 DEBUG:boto:Params: {} 2023-10-11T15:24:15.113 DEBUG:boto:Token: None 2023-10-11T15:24:15.113 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87 2023-10-11T15:24:15.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5JqO3MqlnR0aXnRL7c022nr0EqI= 2023-10-11T15:24:15.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5JqO3MqlnR0aXnRL7c022nr0EqI=', 'Content-Length': '0'} 2023-10-11T15:24:15.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89867d259328b7a-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.117 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.117 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.118 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.118 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:24:15.118 DEBUG:boto:Data: 2023-10-11T15:24:15.119 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.120 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.120 DEBUG:boto:Params: {} 2023-10-11T15:24:15.121 DEBUG:boto:Token: None 2023-10-11T15:24:15.121 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87 2023-10-11T15:24:15.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5JqO3MqlnR0aXnRL7c022nr0EqI= 2023-10-11T15:24:15.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5JqO3MqlnR0aXnRL7c022nr0EqI=', 'Content-Length': '0'} 2023-10-11T15:24:15.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724b90c8b4f030ef-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:24:15.125 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.125 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.126 DEBUG:boto:Method: GET 2023-10-11T15:24:15.126 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:24:15.126 DEBUG:boto:Data: 2023-10-11T15:24:15.127 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.128 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.128 DEBUG:boto:Params: {} 2023-10-11T15:24:15.129 DEBUG:boto:Token: None 2023-10-11T15:24:15.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87 2023-10-11T15:24:15.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t5KWyxuEPBhZMjJb8wS3uJ1316Y= 2023-10-11T15:24:15.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t5KWyxuEPBhZMjJb8wS3uJ1316Y=', 'Content-Length': '0'} 2023-10-11T15:24:15.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052bdd6e68fd96d98-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.133 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:24:15.133 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:24:15.134 DEBUG:boto:Method: GET 2023-10-11T15:24:15.134 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:24:15.135 DEBUG:boto:Data: 2023-10-11T15:24:15.135 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.136 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.136 DEBUG:boto:Params: {} 2023-10-11T15:24:15.137 DEBUG:boto:Token: None 2023-10-11T15:24:15.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj87 2023-10-11T15:24:15.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t5KWyxuEPBhZMjJb8wS3uJ1316Y= 2023-10-11T15:24:15.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t5KWyxuEPBhZMjJb8wS3uJ1316Y=', 'Content-Length': '0'} 2023-10-11T15:24:15.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7117f195f44174-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.141 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.142 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.142 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.143 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.143 DEBUG:boto:Method: GET 2023-10-11T15:24:15.144 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.144 DEBUG:boto:Data: 2023-10-11T15:24:15.145 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.146 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.146 DEBUG:boto:Params: {} 2023-10-11T15:24:15.147 DEBUG:boto:Token: None 2023-10-11T15:24:15.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126a65294485cf09-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.153 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.155 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.155 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.156 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.156 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.157 DEBUG:boto:Method: GET 2023-10-11T15:24:15.158 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:24:15.158 DEBUG:boto:Data: 2023-10-11T15:24:15.159 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.162 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.162 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.163 DEBUG:boto:Params: {} 2023-10-11T15:24:15.163 DEBUG:boto:Token: None 2023-10-11T15:24:15.164 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf423f162f74638-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.171 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:24:15.173 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.173 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.174 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.175 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.175 DEBUG:boto:Method: GET 2023-10-11T15:24:15.176 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.176 DEBUG:boto:Data: 2023-10-11T15:24:15.177 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.178 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.178 DEBUG:boto:Params: {} 2023-10-11T15:24:15.179 DEBUG:boto:Token: None 2023-10-11T15:24:15.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM= 2023-10-11T15:24:15.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM=', 'Content-Length': '0'} 2023-10-11T15:24:15.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17e958f15222b96-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.185 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.186 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.186 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.186 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.187 DEBUG:boto:Method: GET 2023-10-11T15:24:15.187 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.188 DEBUG:boto:Data: 2023-10-11T15:24:15.188 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.189 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.190 DEBUG:boto:Params: {} 2023-10-11T15:24:15.190 DEBUG:boto:Token: None 2023-10-11T15:24:15.191 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM= 2023-10-11T15:24:15.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM=', 'Content-Length': '0'} 2023-10-11T15:24:15.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a230f15bedc63e2-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.195 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.195 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.196 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.196 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.197 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.197 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.197 DEBUG:boto:Data: 2023-10-11T15:24:15.198 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.199 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.199 DEBUG:boto:Params: {} 2023-10-11T15:24:15.200 DEBUG:boto:Token: None 2023-10-11T15:24:15.200 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e33nhU+2TO6gAQM3s+9P8xKpgus= 2023-10-11T15:24:15.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e33nhU+2TO6gAQM3s+9P8xKpgus=', 'Content-Length': '0'} 2023-10-11T15:24:15.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73f5effe5f245bf-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.205 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.205 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.206 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.207 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.207 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.208 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.208 DEBUG:boto:Data: 2023-10-11T15:24:15.208 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.209 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.210 DEBUG:boto:Params: {} 2023-10-11T15:24:15.210 DEBUG:boto:Token: None 2023-10-11T15:24:15.211 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e33nhU+2TO6gAQM3s+9P8xKpgus= 2023-10-11T15:24:15.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e33nhU+2TO6gAQM3s+9P8xKpgus=', 'Content-Length': '0'} 2023-10-11T15:24:15.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6ff80182482f7b-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:24:15.215 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.215 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.216 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.216 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.217 DEBUG:boto:Method: GET 2023-10-11T15:24:15.217 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.218 DEBUG:boto:Data: 2023-10-11T15:24:15.218 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.219 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.219 DEBUG:boto:Params: {} 2023-10-11T15:24:15.220 DEBUG:boto:Token: None 2023-10-11T15:24:15.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM= 2023-10-11T15:24:15.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM=', 'Content-Length': '0'} 2023-10-11T15:24:15.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a0a3e900514ac9-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.224 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.225 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.225 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.225 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.226 DEBUG:boto:Method: GET 2023-10-11T15:24:15.226 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.227 DEBUG:boto:Data: 2023-10-11T15:24:15.227 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.228 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.229 DEBUG:boto:Params: {} 2023-10-11T15:24:15.229 DEBUG:boto:Token: None 2023-10-11T15:24:15.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:24:15.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM= 2023-10-11T15:24:15.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:w1CTbEdctkqDFE8Chvt7XAy6SzM=', 'Content-Length': '0'} 2023-10-11T15:24:15.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2e92c266925d6a-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.233 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.234 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.234 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.235 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.235 DEBUG:boto:Method: GET 2023-10-11T15:24:15.235 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.236 DEBUG:boto:Data: 2023-10-11T15:24:15.236 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.237 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.238 DEBUG:boto:Params: {} 2023-10-11T15:24:15.238 DEBUG:boto:Token: None 2023-10-11T15:24:15.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c400ebcd2472f3-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.243 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.244 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.244 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.244 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.245 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.245 DEBUG:boto:Method: GET 2023-10-11T15:24:15.246 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.246 DEBUG:boto:Data: 2023-10-11T15:24:15.247 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.247 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.248 DEBUG:boto:Params: {} 2023-10-11T15:24:15.248 DEBUG:boto:Token: None 2023-10-11T15:24:15.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2b29170682feea-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.254 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.255 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.255 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.256 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.256 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:24:15.256 DEBUG:boto:Data: 2023-10-11T15:24:15.257 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.258 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.258 DEBUG:boto:Params: {} 2023-10-11T15:24:15.259 DEBUG:boto:Token: None 2023-10-11T15:24:15.259 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92 2023-10-11T15:24:15.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPG1yW4mwAd4TYUR6CYnAIA2htM= 2023-10-11T15:24:15.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPG1yW4mwAd4TYUR6CYnAIA2htM=', 'Content-Length': '0'} 2023-10-11T15:24:15.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba38ede0f66478a-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.262 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.263 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.263 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.264 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:24:15.264 DEBUG:boto:Data: 2023-10-11T15:24:15.264 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.265 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.265 DEBUG:boto:Params: {} 2023-10-11T15:24:15.266 DEBUG:boto:Token: None 2023-10-11T15:24:15.266 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92 2023-10-11T15:24:15.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPG1yW4mwAd4TYUR6CYnAIA2htM= 2023-10-11T15:24:15.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPG1yW4mwAd4TYUR6CYnAIA2htM=', 'Content-Length': '0'} 2023-10-11T15:24:15.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d5a319da377566-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:24:15.270 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.271 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.271 DEBUG:boto:Method: GET 2023-10-11T15:24:15.272 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:24:15.272 DEBUG:boto:Data: 2023-10-11T15:24:15.273 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.273 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.274 DEBUG:boto:Params: {} 2023-10-11T15:24:15.274 DEBUG:boto:Token: None 2023-10-11T15:24:15.275 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92 2023-10-11T15:24:15.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Yox2voNM2/RHrlGUNj1oJ7GV7I= 2023-10-11T15:24:15.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Yox2voNM2/RHrlGUNj1oJ7GV7I=', 'Content-Length': '0'} 2023-10-11T15:24:15.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2746e874a3a5ff8-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.278 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:24:15.279 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:24:15.279 DEBUG:boto:Method: GET 2023-10-11T15:24:15.279 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:24:15.280 DEBUG:boto:Data: 2023-10-11T15:24:15.280 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.281 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.282 DEBUG:boto:Params: {} 2023-10-11T15:24:15.282 DEBUG:boto:Token: None 2023-10-11T15:24:15.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj92 2023-10-11T15:24:15.283 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Yox2voNM2/RHrlGUNj1oJ7GV7I= 2023-10-11T15:24:15.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Yox2voNM2/RHrlGUNj1oJ7GV7I=', 'Content-Length': '0'} 2023-10-11T15:24:15.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adecafd8e4699f51-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.286 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.287 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.288 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.288 DEBUG:boto:Method: GET 2023-10-11T15:24:15.288 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.289 DEBUG:boto:Data: 2023-10-11T15:24:15.289 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.290 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.291 DEBUG:boto:Params: {} 2023-10-11T15:24:15.291 DEBUG:boto:Token: None 2023-10-11T15:24:15.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336f172336f492d1-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.295 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.296 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.297 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.297 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.297 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.298 DEBUG:boto:Method: GET 2023-10-11T15:24:15.298 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:24:15.299 DEBUG:boto:Data: 2023-10-11T15:24:15.299 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.300 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.300 DEBUG:boto:Params: {} 2023-10-11T15:24:15.301 DEBUG:boto:Token: None 2023-10-11T15:24:15.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e716b3006a2cbed1-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.306 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:24:15.308 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.308 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.308 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.309 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.309 DEBUG:boto:Method: GET 2023-10-11T15:24:15.310 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.310 DEBUG:boto:Data: 2023-10-11T15:24:15.311 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.312 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.312 DEBUG:boto:Params: {} 2023-10-11T15:24:15.312 DEBUG:boto:Token: None 2023-10-11T15:24:15.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4= 2023-10-11T15:24:15.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4=', 'Content-Length': '0'} 2023-10-11T15:24:15.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029822e379d48706a-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.358 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.359 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.359 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.360 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.360 DEBUG:boto:Method: GET 2023-10-11T15:24:15.361 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.361 DEBUG:boto:Data: 2023-10-11T15:24:15.362 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.362 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.363 DEBUG:boto:Params: {} 2023-10-11T15:24:15.363 DEBUG:boto:Token: None 2023-10-11T15:24:15.364 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4= 2023-10-11T15:24:15.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4=', 'Content-Length': '0'} 2023-10-11T15:24:15.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f62aebe21f20a3f7-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.370 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.371 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.371 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.371 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.372 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.372 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.373 DEBUG:boto:Data: 2023-10-11T15:24:15.373 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.373 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.374 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.374 DEBUG:boto:Params: {} 2023-10-11T15:24:15.375 DEBUG:boto:Token: None 2023-10-11T15:24:15.375 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur1jvy+8Mhp6KFt88Ez4k+mHQRk= 2023-10-11T15:24:15.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur1jvy+8Mhp6KFt88Ez4k+mHQRk=', 'Content-Length': '0'} 2023-10-11T15:24:15.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004189839dd01aedd8-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.379 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.379 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.380 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.380 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.381 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.381 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.381 DEBUG:boto:Data: 2023-10-11T15:24:15.382 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.383 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.383 DEBUG:boto:Params: {} 2023-10-11T15:24:15.383 DEBUG:boto:Token: None 2023-10-11T15:24:15.384 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ur1jvy+8Mhp6KFt88Ez4k+mHQRk= 2023-10-11T15:24:15.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ur1jvy+8Mhp6KFt88Ez4k+mHQRk=', 'Content-Length': '0'} 2023-10-11T15:24:15.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e3f32999564470-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:24:15.388 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.389 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.389 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.389 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.390 DEBUG:boto:Method: GET 2023-10-11T15:24:15.390 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.391 DEBUG:boto:Data: 2023-10-11T15:24:15.391 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.392 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.393 DEBUG:boto:Params: {} 2023-10-11T15:24:15.393 DEBUG:boto:Token: None 2023-10-11T15:24:15.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4= 2023-10-11T15:24:15.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4=', 'Content-Length': '0'} 2023-10-11T15:24:15.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f50b8b53bec481-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.397 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.398 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.398 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.399 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.399 DEBUG:boto:Method: GET 2023-10-11T15:24:15.400 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.400 DEBUG:boto:Data: 2023-10-11T15:24:15.400 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.401 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.402 DEBUG:boto:Params: {} 2023-10-11T15:24:15.402 DEBUG:boto:Token: None 2023-10-11T15:24:15.403 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:24:15.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4= 2023-10-11T15:24:15.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vW3YXGe7sn9XCUTO/VYnmCQfAc4=', 'Content-Length': '0'} 2023-10-11T15:24:15.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c5e9f987f41bd4-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.446 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.447 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.448 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.448 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.448 DEBUG:boto:Method: GET 2023-10-11T15:24:15.449 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.449 DEBUG:boto:Data: 2023-10-11T15:24:15.450 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.450 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.451 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.451 DEBUG:boto:Params: {} 2023-10-11T15:24:15.451 DEBUG:boto:Token: None 2023-10-11T15:24:15.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.452 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a4678422824641-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.456 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.457 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.458 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.458 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.458 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.459 DEBUG:boto:Method: GET 2023-10-11T15:24:15.459 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.460 DEBUG:boto:Data: 2023-10-11T15:24:15.460 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.461 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.461 DEBUG:boto:Params: {} 2023-10-11T15:24:15.462 DEBUG:boto:Token: None 2023-10-11T15:24:15.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c219eb255ee9c32-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.467 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.468 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.469 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.469 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.470 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:24:15.470 DEBUG:boto:Data: 2023-10-11T15:24:15.470 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.471 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.472 DEBUG:boto:Params: {} 2023-10-11T15:24:15.472 DEBUG:boto:Token: None 2023-10-11T15:24:15.473 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95 2023-10-11T15:24:15.473 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MV6G28048VAxgbh40hzki9amIQU= 2023-10-11T15:24:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MV6G28048VAxgbh40hzki9amIQU=', 'Content-Length': '0'} 2023-10-11T15:24:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021085236ce7a4c6e-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.476 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.477 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.477 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.477 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:24:15.478 DEBUG:boto:Data: 2023-10-11T15:24:15.478 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.479 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.479 DEBUG:boto:Params: {} 2023-10-11T15:24:15.480 DEBUG:boto:Token: None 2023-10-11T15:24:15.480 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95 2023-10-11T15:24:15.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MV6G28048VAxgbh40hzki9amIQU= 2023-10-11T15:24:15.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MV6G28048VAxgbh40hzki9amIQU=', 'Content-Length': '0'} 2023-10-11T15:24:15.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007491c97a4c3facb9-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:24:15.484 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.485 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.485 DEBUG:boto:Method: GET 2023-10-11T15:24:15.486 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:24:15.487 DEBUG:boto:Data: 2023-10-11T15:24:15.487 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.488 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.489 DEBUG:boto:Params: {} 2023-10-11T15:24:15.489 DEBUG:boto:Token: None 2023-10-11T15:24:15.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95 2023-10-11T15:24:15.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yXxtywsrkQ3zqZEbYTnPCECK8CA= 2023-10-11T15:24:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yXxtywsrkQ3zqZEbYTnPCECK8CA=', 'Content-Length': '0'} 2023-10-11T15:24:15.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e02476ef75212a-006526be1f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.534 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:24:15.535 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:24:15.536 DEBUG:boto:Method: GET 2023-10-11T15:24:15.536 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:24:15.537 DEBUG:boto:Data: 2023-10-11T15:24:15.537 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.538 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.538 DEBUG:boto:Params: {} 2023-10-11T15:24:15.539 DEBUG:boto:Token: None 2023-10-11T15:24:15.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj95 2023-10-11T15:24:15.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yXxtywsrkQ3zqZEbYTnPCECK8CA= 2023-10-11T15:24:15.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yXxtywsrkQ3zqZEbYTnPCECK8CA=', 'Content-Length': '0'} 2023-10-11T15:24:15.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd1cf2ed1aca873-006526be1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.543 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.544 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.544 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.545 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.545 DEBUG:boto:Method: GET 2023-10-11T15:24:15.546 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.546 DEBUG:boto:Data: 2023-10-11T15:24:15.547 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:24:15.548 DEBUG:boto:Port: 8001 2023-10-11T15:24:15.548 DEBUG:boto:Params: {} 2023-10-11T15:24:15.549 DEBUG:boto:Token: None 2023-10-11T15:24:15.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.550 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be750193eda40154-006526be1f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.553 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.554 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.555 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.555 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.556 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.556 DEBUG:boto:Method: GET 2023-10-11T15:24:15.556 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:24:15.557 DEBUG:boto:Data: 2023-10-11T15:24:15.557 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.558 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.558 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.559 DEBUG:boto:Params: {} 2023-10-11T15:24:15.559 DEBUG:boto:Token: None 2023-10-11T15:24:15.560 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ= 2023-10-11T15:24:15.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEkDUsPsPVCOUmlh+lPO12SNtVQ=', 'Content-Length': '0'} 2023-10-11T15:24:15.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b04cfae45d7d36-006526be1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.610 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:24:15.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:24:15.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:24:15.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:24:15.613 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:24:15.671 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:24:15.672 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:24:15.723 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:24:15.723 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:24:15.733 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 25 2023-10-11T15:24:15.789 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.790 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.790 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.791 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:15.791 DEBUG:boto:Data: 2023-10-11T15:24:15.791 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.792 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.792 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.793 DEBUG:boto:Params: {} 2023-10-11T15:24:15.793 DEBUG:boto:Token: None 2023-10-11T15:24:15.794 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas= 2023-10-11T15:24:15.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas=', 'Content-Length': '0'} 2023-10-11T15:24:15.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e053b399ad28fae-006526be1f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.799 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:24:15.799 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:24:15.800 DEBUG:boto:Method: PUT 2023-10-11T15:24:15.800 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:24:15.800 DEBUG:boto:Data: 2023-10-11T15:24:15.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:24:15.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.802 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.802 DEBUG:boto:Params: {} 2023-10-11T15:24:15.803 DEBUG:boto:Token: None 2023-10-11T15:24:15.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj31 2023-10-11T15:24:15.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1Zn+L8baXzwamHxuubvZP8J8dZU= 2023-10-11T15:24:15.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1Zn+L8baXzwamHxuubvZP8J8dZU='} 2023-10-11T15:24:15.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdfba717b1426500-006526be1f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.810 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.810 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.810 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.811 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:15.811 DEBUG:boto:Data: 2023-10-11T15:24:15.812 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.812 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.812 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.813 DEBUG:boto:Params: {} 2023-10-11T15:24:15.813 DEBUG:boto:Token: None 2023-10-11T15:24:15.814 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas= 2023-10-11T15:24:15.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas=', 'Content-Length': '0'} 2023-10-11T15:24:15.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc4c9585bea9eb22-006526be1f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.819 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:24:15.819 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:24:15.820 DEBUG:boto:Method: PUT 2023-10-11T15:24:15.820 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:24:15.820 DEBUG:boto:Data: 2023-10-11T15:24:15.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:24:15.821 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.822 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.822 DEBUG:boto:Params: {} 2023-10-11T15:24:15.822 DEBUG:boto:Token: None 2023-10-11T15:24:15.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj62 2023-10-11T15:24:15.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SSRDPyOGKrtNd0rpf6wKvtFBPtA= 2023-10-11T15:24:15.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SSRDPyOGKrtNd0rpf6wKvtFBPtA='} 2023-10-11T15:24:15.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a23af64111112562-006526be1f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.830 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.830 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.830 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.831 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:15.831 DEBUG:boto:Data: 2023-10-11T15:24:15.832 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.832 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.833 DEBUG:boto:Params: {} 2023-10-11T15:24:15.833 DEBUG:boto:Token: None 2023-10-11T15:24:15.834 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas= 2023-10-11T15:24:15.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas=', 'Content-Length': '0'} 2023-10-11T15:24:15.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e2fa57b7b745866-006526be1f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.839 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:24:15.839 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:24:15.839 DEBUG:boto:Method: PUT 2023-10-11T15:24:15.840 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:24:15.840 DEBUG:boto:Data: 2023-10-11T15:24:15.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:24:15.841 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.841 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.842 DEBUG:boto:Params: {} 2023-10-11T15:24:15.842 DEBUG:boto:Token: None 2023-10-11T15:24:15.843 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj93 2023-10-11T15:24:15.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ORKRU0oxxW4EFekWX6zDf6w9Q9g= 2023-10-11T15:24:15.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ORKRU0oxxW4EFekWX6zDf6w9Q9g='} 2023-10-11T15:24:15.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4f4f80509bcf6d6-006526be1f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.849 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.849 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.850 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.850 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:15.851 DEBUG:boto:Data: 2023-10-11T15:24:15.851 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.852 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.852 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.852 DEBUG:boto:Params: {} 2023-10-11T15:24:15.853 DEBUG:boto:Token: None 2023-10-11T15:24:15.853 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas= 2023-10-11T15:24:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas=', 'Content-Length': '0'} 2023-10-11T15:24:15.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000178ca302c61201ac-006526be1f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.858 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:24:15.858 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:24:15.859 DEBUG:boto:Method: PUT 2023-10-11T15:24:15.859 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:24:15.860 DEBUG:boto:Data: 2023-10-11T15:24:15.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:24:15.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.861 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.861 DEBUG:boto:Params: {} 2023-10-11T15:24:15.862 DEBUG:boto:Token: None 2023-10-11T15:24:15.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj124 2023-10-11T15:24:15.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4AnohkcrOXxeGfw4Xtr4SX8876c= 2023-10-11T15:24:15.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4AnohkcrOXxeGfw4Xtr4SX8876c='} 2023-10-11T15:24:15.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019d491423dc760de-006526be1f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.907 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:24:15.908 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:24:15.908 DEBUG:boto:Method: HEAD 2023-10-11T15:24:15.909 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:24:15.909 DEBUG:boto:Data: 2023-10-11T15:24:15.909 DEBUG:boto:Headers: {} 2023-10-11T15:24:15.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.910 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.911 DEBUG:boto:Params: {} 2023-10-11T15:24:15.911 DEBUG:boto:Token: None 2023-10-11T15:24:15.912 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/ 2023-10-11T15:24:15.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas= 2023-10-11T15:24:15.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dJKO1m7mlq0lnZC2tnpwrX3sdas=', 'Content-Length': '0'} 2023-10-11T15:24:15.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039c228558cc427c1-006526be1f-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.916 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:24:15.917 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:24:15.917 DEBUG:boto:Method: PUT 2023-10-11T15:24:15.918 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:24:15.918 DEBUG:boto:Data: 2023-10-11T15:24:15.918 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:24:15.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:24:15.919 DEBUG:boto:Port: 8000 2023-10-11T15:24:15.920 DEBUG:boto:Params: {} 2023-10-11T15:24:15.920 DEBUG:boto:Token: None 2023-10-11T15:24:15.921 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:24:15 GMT /dghqvd-59/obj155 2023-10-11T15:24:15.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aZgDeuGgFfhm3f13LargePGjBmI= 2023-10-11T15:24:15.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:24:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aZgDeuGgFfhm3f13LargePGjBmI='} 2023-10-11T15:24:15.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a6c68db7cfff4e2-006526be1f-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:24:15.927 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:24:15.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:24:15.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:24:15.929 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:24:15.985 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:24:15.986 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:24:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:16.019+0000 7f89af12d780 1 waiting to reach latest gen 5, on 4.. 2023-10-11T15:24:21.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:21.022+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:21.024 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-10-11T15:24:21.024 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:26.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:26.024+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:26.026 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:26.026 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:31.027 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:31.025+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:31.028 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:31.028 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:36.030 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:36.027+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:36.030 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:36.030 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:41.032 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:41.029+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:41.032 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:41.032 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:46.034 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:46.032+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:46.034 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:46.035 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:51.037 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:51.034+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:51.037 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:51.037 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:56.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:24:56.037+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:24:56.040 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000001.83.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:24:56.040 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:01.041 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:01.039+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:01.042 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:01.042 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:06.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:06.041+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:06.044 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:06.044 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:11.046 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:11.043+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:11.046 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:11.047 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:16.048 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:16.047+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:16.049 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:16.049 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:21.050 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:21.049+0000 7f89af12d780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:21.051 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:21.051 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:26.051+0000 7f89af12d780 1 bucket sync caught up with source: 2023-10-11T15:25:26.053 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:26.053 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , 00000000002.49.6, , 00000000001.68.6, , 00000000001.83.6, , , , , , , , , , 00000000001.87.6, , , , , , , , , , , , , , 00000000001.47.6] 2023-10-11T15:25:26.053 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:26.051+0000 7f89af12d780 0 bucket checkpoint complete 2023-10-11T15:25:26.070 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:25:26.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:25:26.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:25:26.072 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:25:26.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:25:26.136 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:25:26.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:26.172+0000 7ffaa1689780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:26.174 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000003.107.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000003.262.6, 00000000006.179.6, 00000000004.288.6] 2023-10-11T15:25:26.174 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000004.118.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000003.262.6, 00000000006.179.6, 00000000004.288.6] 2023-10-11T15:25:31.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:31.173+0000 7ffaa1689780 1 bucket sync caught up with source: 2023-10-11T15:25:31.175 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000004.118.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000003.262.6, 00000000006.179.6, 00000000004.288.6] 2023-10-11T15:25:31.176 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000004.118.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000003.262.6, 00000000006.179.6, 00000000004.288.6] 2023-10-11T15:25:31.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:31.173+0000 7ffaa1689780 0 bucket checkpoint complete 2023-10-11T15:25:31.192 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:25:31.192 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.193 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.193 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.194 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:31.194 DEBUG:boto:Data: 2023-10-11T15:25:31.195 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.196 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.196 DEBUG:boto:Params: {} 2023-10-11T15:25:31.197 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:25:31.197 DEBUG:boto:Token: None 2023-10-11T15:25:31.198 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.198 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ejeKCdqqq5BMoeKS+aRuJA+zS4E= 2023-10-11T15:25:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ejeKCdqqq5BMoeKS+aRuJA+zS4E=', 'Content-Length': '0'} 2023-10-11T15:25:31.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fdfd0121e2cb3ba-006526be6b-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.204 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.205 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.205 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.205 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:31.206 DEBUG:boto:Data: 2023-10-11T15:25:31.206 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.207 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.208 DEBUG:boto:Params: {} 2023-10-11T15:25:31.208 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:25:31.209 DEBUG:boto:Token: None 2023-10-11T15:25:31.209 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.210 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ejeKCdqqq5BMoeKS+aRuJA+zS4E= 2023-10-11T15:25:31.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ejeKCdqqq5BMoeKS+aRuJA+zS4E=', 'Content-Length': '0'} 2023-10-11T15:25:31.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c2def19a6a3d742-006526be6b-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.216 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:25:31.217 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.217 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.218 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.218 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.219 DEBUG:boto:Method: GET 2023-10-11T15:25:31.219 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.219 DEBUG:boto:Data: 2023-10-11T15:25:31.220 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.221 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.221 DEBUG:boto:Params: {} 2023-10-11T15:25:31.222 DEBUG:boto:Token: None 2023-10-11T15:25:31.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/?versions 2023-10-11T15:25:31.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E= 2023-10-11T15:25:31.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E=', 'Content-Length': '0'} 2023-10-11T15:25:31.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000468d6758c0c3780b-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.227 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:25:31.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:25:31.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:25:31.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:25:31.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:25:31.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:25:31.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:25:31.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:25:31.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:25:31.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:25:31.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:25:31.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:25:31.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:25:31.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:25:31.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:25:31.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:25:31.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:25:31.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:25:31.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:25:31.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:25:31.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:25:31.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:25:31.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:25:31.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:25:31.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:25:31.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:25:31.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:25:31.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:25:31.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:25:31.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:25:31.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:25:31.241 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.241 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.241 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.242 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.242 DEBUG:boto:Method: GET 2023-10-11T15:25:31.243 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.243 DEBUG:boto:Data: 2023-10-11T15:25:31.243 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.244 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.245 DEBUG:boto:Params: {} 2023-10-11T15:25:31.245 DEBUG:boto:Token: None 2023-10-11T15:25:31.246 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/?versions 2023-10-11T15:25:31.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E= 2023-10-11T15:25:31.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E=', 'Content-Length': '0'} 2023-10-11T15:25:31.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c3a31a8f03d3b0c-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.252 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:25:31.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:25:31.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:25:31.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:25:31.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:25:31.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:25:31.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:25:31.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:25:31.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:25:31.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:25:31.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:25:31.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:25:31.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:25:31.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:25:31.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:25:31.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:25:31.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:25:31.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:25:31.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:25:31.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:25:31.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:25:31.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:25:31.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:25:31.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:25:31.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:25:31.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:25:31.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:25:31.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:25:31.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:25:31.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:25:31.265 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.265 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.266 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.266 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.266 DEBUG:boto:Method: GET 2023-10-11T15:25:31.267 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.267 DEBUG:boto:Data: 2023-10-11T15:25:31.268 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.269 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.269 DEBUG:boto:Params: {} 2023-10-11T15:25:31.269 DEBUG:boto:Token: None 2023-10-11T15:25:31.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/?versions 2023-10-11T15:25:31.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E= 2023-10-11T15:25:31.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E=', 'Content-Length': '0'} 2023-10-11T15:25:31.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b486600671fddbbd-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.275 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.276 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.277 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.277 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.278 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.278 DEBUG:boto:Method: GET 2023-10-11T15:25:31.278 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:25:31.279 DEBUG:boto:Data: 2023-10-11T15:25:31.279 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.280 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.280 DEBUG:boto:Params: {} 2023-10-11T15:25:31.281 DEBUG:boto:Token: None 2023-10-11T15:25:31.281 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/?versions 2023-10-11T15:25:31.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E= 2023-10-11T15:25:31.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jO8bn6yEAxNIZvlmlbgCIzyoZ+E=', 'Content-Length': '0'} 2023-10-11T15:25:31.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ddfcfd48a134ef-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.287 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:25:31.289 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.289 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.290 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.290 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.291 DEBUG:boto:Method: GET 2023-10-11T15:25:31.291 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.292 DEBUG:boto:Data: 2023-10-11T15:25:31.292 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.293 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.293 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.293 DEBUG:boto:Params: {} 2023-10-11T15:25:31.294 DEBUG:boto:Token: None 2023-10-11T15:25:31.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.295 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg= 2023-10-11T15:25:31.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg=', 'Content-Length': '0'} 2023-10-11T15:25:31.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015bbbbabd94d9ba9-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.298 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.299 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.299 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.300 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.300 DEBUG:boto:Method: GET 2023-10-11T15:25:31.300 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.301 DEBUG:boto:Data: 2023-10-11T15:25:31.301 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.302 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.303 DEBUG:boto:Params: {} 2023-10-11T15:25:31.303 DEBUG:boto:Token: None 2023-10-11T15:25:31.303 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.304 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg= 2023-10-11T15:25:31.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg=', 'Content-Length': '0'} 2023-10-11T15:25:31.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe420e06cc71fe6-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.354 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.355 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.355 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.356 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.356 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.357 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.357 DEBUG:boto:Data: 2023-10-11T15:25:31.358 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.358 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.359 DEBUG:boto:Params: {} 2023-10-11T15:25:31.359 DEBUG:boto:Token: None 2023-10-11T15:25:31.360 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VCvnQo+Hq/YXtf60cuc1jR9A0O0= 2023-10-11T15:25:31.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VCvnQo+Hq/YXtf60cuc1jR9A0O0=', 'Content-Length': '0'} 2023-10-11T15:25:31.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039bb244cf3aca3c5-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.364 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.364 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.365 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.365 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.366 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.367 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.367 DEBUG:boto:Data: 2023-10-11T15:25:31.368 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.368 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.369 DEBUG:boto:Params: {} 2023-10-11T15:25:31.370 DEBUG:boto:Token: None 2023-10-11T15:25:31.370 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.371 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VCvnQo+Hq/YXtf60cuc1jR9A0O0= 2023-10-11T15:25:31.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VCvnQo+Hq/YXtf60cuc1jR9A0O0=', 'Content-Length': '0'} 2023-10-11T15:25:31.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001e0feda2d737c97-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:25:31.375 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.375 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.375 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.376 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.376 DEBUG:boto:Method: GET 2023-10-11T15:25:31.377 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.377 DEBUG:boto:Data: 2023-10-11T15:25:31.378 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.379 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.379 DEBUG:boto:Params: {} 2023-10-11T15:25:31.380 DEBUG:boto:Token: None 2023-10-11T15:25:31.380 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg= 2023-10-11T15:25:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg=', 'Content-Length': '0'} 2023-10-11T15:25:31.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09392c36ec5eddb-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.430 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.431 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.432 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.432 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.433 DEBUG:boto:Method: GET 2023-10-11T15:25:31.434 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.435 DEBUG:boto:Data: 2023-10-11T15:25:31.436 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.437 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.437 DEBUG:boto:Params: {} 2023-10-11T15:25:31.438 DEBUG:boto:Token: None 2023-10-11T15:25:31.438 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:25:31.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg= 2023-10-11T15:25:31.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G62u2XN7t2bwj9yl9h5RbSg3wWg=', 'Content-Length': '0'} 2023-10-11T15:25:31.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f19eb3cf0de6b0cf-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.443 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.443 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.444 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.444 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.444 DEBUG:boto:Method: GET 2023-10-11T15:25:31.445 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.445 DEBUG:boto:Data: 2023-10-11T15:25:31.446 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.447 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.447 DEBUG:boto:Params: {} 2023-10-11T15:25:31.447 DEBUG:boto:Token: None 2023-10-11T15:25:31.448 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be07fd6284fb7615-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.452 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.453 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.454 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.454 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.455 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.455 DEBUG:boto:Method: GET 2023-10-11T15:25:31.455 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.456 DEBUG:boto:Data: 2023-10-11T15:25:31.456 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.457 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.458 DEBUG:boto:Params: {} 2023-10-11T15:25:31.458 DEBUG:boto:Token: None 2023-10-11T15:25:31.459 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5c02e234142041-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.464 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.465 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.465 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.466 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.466 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:25:31.467 DEBUG:boto:Data: 2023-10-11T15:25:31.467 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.468 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.468 DEBUG:boto:Params: {} 2023-10-11T15:25:31.469 DEBUG:boto:Token: None 2023-10-11T15:25:31.469 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11 2023-10-11T15:25:31.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TiT7lQOJH7REDF6RyhSIISbDWaI= 2023-10-11T15:25:31.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TiT7lQOJH7REDF6RyhSIISbDWaI=', 'Content-Length': '0'} 2023-10-11T15:25:31.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd98d04900c5665c-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.473 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.473 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.474 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.474 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:25:31.475 DEBUG:boto:Data: 2023-10-11T15:25:31.475 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.475 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.476 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.476 DEBUG:boto:Params: {} 2023-10-11T15:25:31.477 DEBUG:boto:Token: None 2023-10-11T15:25:31.477 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11 2023-10-11T15:25:31.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TiT7lQOJH7REDF6RyhSIISbDWaI= 2023-10-11T15:25:31.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TiT7lQOJH7REDF6RyhSIISbDWaI=', 'Content-Length': '0'} 2023-10-11T15:25:31.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe39d1d2931976e6-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:25:31.481 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.482 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.482 DEBUG:boto:Method: GET 2023-10-11T15:25:31.482 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:25:31.483 DEBUG:boto:Data: 2023-10-11T15:25:31.483 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.484 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.484 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.485 DEBUG:boto:Params: {} 2023-10-11T15:25:31.485 DEBUG:boto:Token: None 2023-10-11T15:25:31.486 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11 2023-10-11T15:25:31.486 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7l/iKUz6dP16hGXVfZHqiBy29AE= 2023-10-11T15:25:31.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7l/iKUz6dP16hGXVfZHqiBy29AE=', 'Content-Length': '0'} 2023-10-11T15:25:31.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e138e6e5389da659-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.490 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:25:31.490 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:25:31.490 DEBUG:boto:Method: GET 2023-10-11T15:25:31.491 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:25:31.491 DEBUG:boto:Data: 2023-10-11T15:25:31.492 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.493 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.493 DEBUG:boto:Params: {} 2023-10-11T15:25:31.494 DEBUG:boto:Token: None 2023-10-11T15:25:31.494 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj11 2023-10-11T15:25:31.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7l/iKUz6dP16hGXVfZHqiBy29AE= 2023-10-11T15:25:31.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7l/iKUz6dP16hGXVfZHqiBy29AE=', 'Content-Length': '0'} 2023-10-11T15:25:31.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7415d604bb0dd17-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.498 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.499 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.499 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.500 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.500 DEBUG:boto:Method: GET 2023-10-11T15:25:31.501 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.501 DEBUG:boto:Data: 2023-10-11T15:25:31.501 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.502 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.502 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.503 DEBUG:boto:Params: {} 2023-10-11T15:25:31.503 DEBUG:boto:Token: None 2023-10-11T15:25:31.504 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6d9a1a378a8b68-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.508 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.509 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.509 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.510 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.510 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.510 DEBUG:boto:Method: GET 2023-10-11T15:25:31.511 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:25:31.511 DEBUG:boto:Data: 2023-10-11T15:25:31.512 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.512 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.512 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.513 DEBUG:boto:Params: {} 2023-10-11T15:25:31.513 DEBUG:boto:Token: None 2023-10-11T15:25:31.514 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.514 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e4ee9fd04941be-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.519 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:25:31.520 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.521 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.521 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.521 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.522 DEBUG:boto:Method: GET 2023-10-11T15:25:31.522 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.523 DEBUG:boto:Data: 2023-10-11T15:25:31.523 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.524 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.524 DEBUG:boto:Params: {} 2023-10-11T15:25:31.525 DEBUG:boto:Token: None 2023-10-11T15:25:31.525 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg= 2023-10-11T15:25:31.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg=', 'Content-Length': '0'} 2023-10-11T15:25:31.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33e7b8329467180-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.570 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.571 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.571 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.572 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.572 DEBUG:boto:Method: GET 2023-10-11T15:25:31.573 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.573 DEBUG:boto:Data: 2023-10-11T15:25:31.574 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.575 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.575 DEBUG:boto:Params: {} 2023-10-11T15:25:31.576 DEBUG:boto:Token: None 2023-10-11T15:25:31.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg= 2023-10-11T15:25:31.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg=', 'Content-Length': '0'} 2023-10-11T15:25:31.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc94356cda310721-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.580 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.581 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.581 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.581 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.582 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.582 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.583 DEBUG:boto:Data: 2023-10-11T15:25:31.583 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.584 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.585 DEBUG:boto:Params: {} 2023-10-11T15:25:31.585 DEBUG:boto:Token: None 2023-10-11T15:25:31.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYH5vEWrMCwtXoozfWkEfUFcfB4= 2023-10-11T15:25:31.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYH5vEWrMCwtXoozfWkEfUFcfB4=', 'Content-Length': '0'} 2023-10-11T15:25:31.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3cc0ae00297d61-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.589 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.589 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.590 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.590 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.591 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.591 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.592 DEBUG:boto:Data: 2023-10-11T15:25:31.592 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.593 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.593 DEBUG:boto:Params: {} 2023-10-11T15:25:31.594 DEBUG:boto:Token: None 2023-10-11T15:25:31.594 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYH5vEWrMCwtXoozfWkEfUFcfB4= 2023-10-11T15:25:31.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYH5vEWrMCwtXoozfWkEfUFcfB4=', 'Content-Length': '0'} 2023-10-11T15:25:31.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b7c08d06a719c3-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:25:31.598 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.598 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.599 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.599 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.600 DEBUG:boto:Method: GET 2023-10-11T15:25:31.600 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.600 DEBUG:boto:Data: 2023-10-11T15:25:31.601 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.602 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.602 DEBUG:boto:Params: {} 2023-10-11T15:25:31.603 DEBUG:boto:Token: None 2023-10-11T15:25:31.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg= 2023-10-11T15:25:31.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg=', 'Content-Length': '0'} 2023-10-11T15:25:31.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d593832b38c698a3-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.607 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.607 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.608 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.608 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.609 DEBUG:boto:Method: GET 2023-10-11T15:25:31.609 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.610 DEBUG:boto:Data: 2023-10-11T15:25:31.610 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.611 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.612 DEBUG:boto:Params: {} 2023-10-11T15:25:31.612 DEBUG:boto:Token: None 2023-10-11T15:25:31.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:25:31.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg= 2023-10-11T15:25:31.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QPi7CqND3m7ZfiVNytv6g1wsqjg=', 'Content-Length': '0'} 2023-10-11T15:25:31.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf900ea01736693-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.658 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.659 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.659 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.660 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.660 DEBUG:boto:Method: GET 2023-10-11T15:25:31.661 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.661 DEBUG:boto:Data: 2023-10-11T15:25:31.662 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.663 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.663 DEBUG:boto:Params: {} 2023-10-11T15:25:31.664 DEBUG:boto:Token: None 2023-10-11T15:25:31.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc0442f98783d6aa-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.669 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.669 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.670 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.670 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.671 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.671 DEBUG:boto:Method: GET 2023-10-11T15:25:31.672 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.673 DEBUG:boto:Data: 2023-10-11T15:25:31.673 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.674 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.674 DEBUG:boto:Params: {} 2023-10-11T15:25:31.675 DEBUG:boto:Token: None 2023-10-11T15:25:31.675 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000856f016fdfd4cd23-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.681 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.682 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.682 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.683 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.683 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:25:31.684 DEBUG:boto:Data: 2023-10-11T15:25:31.684 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.684 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.685 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.685 DEBUG:boto:Params: {} 2023-10-11T15:25:31.686 DEBUG:boto:Token: None 2023-10-11T15:25:31.686 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115 2023-10-11T15:25:31.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lkKMFwx7kTSvuTtmhejJQWcgwzg= 2023-10-11T15:25:31.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lkKMFwx7kTSvuTtmhejJQWcgwzg=', 'Content-Length': '0'} 2023-10-11T15:25:31.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3653c02a9221e5-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.690 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.690 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.691 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.692 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:25:31.693 DEBUG:boto:Data: 2023-10-11T15:25:31.693 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.695 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.696 DEBUG:boto:Params: {} 2023-10-11T15:25:31.696 DEBUG:boto:Token: None 2023-10-11T15:25:31.697 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115 2023-10-11T15:25:31.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lkKMFwx7kTSvuTtmhejJQWcgwzg= 2023-10-11T15:25:31.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lkKMFwx7kTSvuTtmhejJQWcgwzg=', 'Content-Length': '0'} 2023-10-11T15:25:31.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f958cd9684e342e7-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:25:31.702 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.702 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.703 DEBUG:boto:Method: GET 2023-10-11T15:25:31.703 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:25:31.704 DEBUG:boto:Data: 2023-10-11T15:25:31.705 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.706 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.707 DEBUG:boto:Params: {} 2023-10-11T15:25:31.708 DEBUG:boto:Token: None 2023-10-11T15:25:31.709 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115 2023-10-11T15:25:31.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2dUfTwmYoXk4Y3T+19FZTuwK0o4= 2023-10-11T15:25:31.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2dUfTwmYoXk4Y3T+19FZTuwK0o4=', 'Content-Length': '0'} 2023-10-11T15:25:31.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000027d6ba6e658192-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.714 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:25:31.714 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:25:31.715 DEBUG:boto:Method: GET 2023-10-11T15:25:31.715 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:25:31.716 DEBUG:boto:Data: 2023-10-11T15:25:31.716 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.716 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.717 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.717 DEBUG:boto:Params: {} 2023-10-11T15:25:31.718 DEBUG:boto:Token: None 2023-10-11T15:25:31.718 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj115 2023-10-11T15:25:31.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2dUfTwmYoXk4Y3T+19FZTuwK0o4= 2023-10-11T15:25:31.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2dUfTwmYoXk4Y3T+19FZTuwK0o4=', 'Content-Length': '0'} 2023-10-11T15:25:31.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaab7681136acd8-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.722 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.722 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.723 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.723 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.724 DEBUG:boto:Method: GET 2023-10-11T15:25:31.724 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.725 DEBUG:boto:Data: 2023-10-11T15:25:31.725 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.725 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.726 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.726 DEBUG:boto:Params: {} 2023-10-11T15:25:31.727 DEBUG:boto:Token: None 2023-10-11T15:25:31.727 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d27bb7c826eb01b-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.731 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.732 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.733 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.733 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.733 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.734 DEBUG:boto:Method: GET 2023-10-11T15:25:31.734 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:25:31.735 DEBUG:boto:Data: 2023-10-11T15:25:31.736 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.737 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.738 DEBUG:boto:Params: {} 2023-10-11T15:25:31.739 DEBUG:boto:Token: None 2023-10-11T15:25:31.740 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba7622aa5a0cca2-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.746 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:25:31.748 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.748 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.749 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.749 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.750 DEBUG:boto:Method: GET 2023-10-11T15:25:31.750 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.751 DEBUG:boto:Data: 2023-10-11T15:25:31.751 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.752 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.752 DEBUG:boto:Params: {} 2023-10-11T15:25:31.753 DEBUG:boto:Token: None 2023-10-11T15:25:31.753 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI= 2023-10-11T15:25:31.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI=', 'Content-Length': '0'} 2023-10-11T15:25:31.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bfb5b0f9677277-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.757 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.758 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.759 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.759 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.759 DEBUG:boto:Method: GET 2023-10-11T15:25:31.760 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.760 DEBUG:boto:Data: 2023-10-11T15:25:31.761 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.762 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.762 DEBUG:boto:Params: {} 2023-10-11T15:25:31.762 DEBUG:boto:Token: None 2023-10-11T15:25:31.763 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI= 2023-10-11T15:25:31.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI=', 'Content-Length': '0'} 2023-10-11T15:25:31.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b20164de2b18f6-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.767 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.767 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.768 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.768 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.769 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.769 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.769 DEBUG:boto:Data: 2023-10-11T15:25:31.770 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.771 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.771 DEBUG:boto:Params: {} 2023-10-11T15:25:31.772 DEBUG:boto:Token: None 2023-10-11T15:25:31.772 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uQ5Slt+YGmQyvpVp0KV1WeR+Q5E= 2023-10-11T15:25:31.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uQ5Slt+YGmQyvpVp0KV1WeR+Q5E=', 'Content-Length': '0'} 2023-10-11T15:25:31.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3c00dde7cdb00f-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.776 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.777 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.777 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.777 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.778 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.778 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.779 DEBUG:boto:Data: 2023-10-11T15:25:31.779 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.780 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.780 DEBUG:boto:Params: {} 2023-10-11T15:25:31.781 DEBUG:boto:Token: None 2023-10-11T15:25:31.781 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uQ5Slt+YGmQyvpVp0KV1WeR+Q5E= 2023-10-11T15:25:31.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uQ5Slt+YGmQyvpVp0KV1WeR+Q5E=', 'Content-Length': '0'} 2023-10-11T15:25:31.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000131ccfe344b5d784-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:25:31.786 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.786 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.786 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.787 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.787 DEBUG:boto:Method: GET 2023-10-11T15:25:31.788 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.788 DEBUG:boto:Data: 2023-10-11T15:25:31.789 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.789 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.790 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.790 DEBUG:boto:Params: {} 2023-10-11T15:25:31.791 DEBUG:boto:Token: None 2023-10-11T15:25:31.791 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI= 2023-10-11T15:25:31.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI=', 'Content-Length': '0'} 2023-10-11T15:25:31.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ed1f2f8d83adbd-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.795 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.796 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.796 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.796 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.797 DEBUG:boto:Method: GET 2023-10-11T15:25:31.797 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.798 DEBUG:boto:Data: 2023-10-11T15:25:31.798 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.799 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.800 DEBUG:boto:Params: {} 2023-10-11T15:25:31.800 DEBUG:boto:Token: None 2023-10-11T15:25:31.800 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:25:31.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI= 2023-10-11T15:25:31.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oNTWZN4ajzSpqfmwSqVaFeareLI=', 'Content-Length': '0'} 2023-10-11T15:25:31.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408bb2bfa4daf2a8-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.850 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.851 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.851 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.852 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.852 DEBUG:boto:Method: GET 2023-10-11T15:25:31.853 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.853 DEBUG:boto:Data: 2023-10-11T15:25:31.854 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.855 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.855 DEBUG:boto:Params: {} 2023-10-11T15:25:31.856 DEBUG:boto:Token: None 2023-10-11T15:25:31.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7649612dbf7e12-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.861 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.861 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.862 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.862 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.863 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.863 DEBUG:boto:Method: GET 2023-10-11T15:25:31.863 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.864 DEBUG:boto:Data: 2023-10-11T15:25:31.864 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.865 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.866 DEBUG:boto:Params: {} 2023-10-11T15:25:31.866 DEBUG:boto:Token: None 2023-10-11T15:25:31.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac1bfb4711aeb540-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.873 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.874 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.874 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.875 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.875 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:25:31.875 DEBUG:boto:Data: 2023-10-11T15:25:31.876 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.876 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.877 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.877 DEBUG:boto:Params: {} 2023-10-11T15:25:31.877 DEBUG:boto:Token: None 2023-10-11T15:25:31.878 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116 2023-10-11T15:25:31.878 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NARrPUrM3UKJ6EgUcPx9GB/92Bc= 2023-10-11T15:25:31.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NARrPUrM3UKJ6EgUcPx9GB/92Bc=', 'Content-Length': '0'} 2023-10-11T15:25:31.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134b4c771be554d6-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.881 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.882 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.882 DEBUG:boto:Method: HEAD 2023-10-11T15:25:31.882 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:25:31.883 DEBUG:boto:Data: 2023-10-11T15:25:31.883 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.884 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.884 DEBUG:boto:Params: {} 2023-10-11T15:25:31.885 DEBUG:boto:Token: None 2023-10-11T15:25:31.885 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116 2023-10-11T15:25:31.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NARrPUrM3UKJ6EgUcPx9GB/92Bc= 2023-10-11T15:25:31.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NARrPUrM3UKJ6EgUcPx9GB/92Bc=', 'Content-Length': '0'} 2023-10-11T15:25:31.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1fded1363ba45e6-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:25:31.889 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.890 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.890 DEBUG:boto:Method: GET 2023-10-11T15:25:31.890 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:25:31.891 DEBUG:boto:Data: 2023-10-11T15:25:31.891 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.892 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.893 DEBUG:boto:Params: {} 2023-10-11T15:25:31.893 DEBUG:boto:Token: None 2023-10-11T15:25:31.894 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116 2023-10-11T15:25:31.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yBLdvHX7tSrd8N8iHKhGXzxcr/M= 2023-10-11T15:25:31.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yBLdvHX7tSrd8N8iHKhGXzxcr/M=', 'Content-Length': '0'} 2023-10-11T15:25:31.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f541bccf08a24e8a-006526be6b-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.938 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:25:31.939 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:25:31.939 DEBUG:boto:Method: GET 2023-10-11T15:25:31.940 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:25:31.940 DEBUG:boto:Data: 2023-10-11T15:25:31.941 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.942 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.942 DEBUG:boto:Params: {} 2023-10-11T15:25:31.943 DEBUG:boto:Token: None 2023-10-11T15:25:31.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/obj116 2023-10-11T15:25:31.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yBLdvHX7tSrd8N8iHKhGXzxcr/M= 2023-10-11T15:25:31.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yBLdvHX7tSrd8N8iHKhGXzxcr/M=', 'Content-Length': '0'} 2023-10-11T15:25:31.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de2e0467c5ab13f-006526be6b-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.947 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.948 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.949 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.949 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.949 DEBUG:boto:Method: GET 2023-10-11T15:25:31.950 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.950 DEBUG:boto:Data: 2023-10-11T15:25:31.951 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:31.952 DEBUG:boto:Port: 8001 2023-10-11T15:25:31.952 DEBUG:boto:Params: {} 2023-10-11T15:25:31.953 DEBUG:boto:Token: None 2023-10-11T15:25:31.953 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9fce7898dd1b21-006526be6b-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:31.957 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:31.958 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:31.959 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:31.959 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.959 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.960 DEBUG:boto:Method: GET 2023-10-11T15:25:31.960 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:25:31.961 DEBUG:boto:Data: 2023-10-11T15:25:31.961 DEBUG:boto:Headers: {} 2023-10-11T15:25:31.962 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:31.962 DEBUG:boto:Port: 8000 2023-10-11T15:25:31.962 DEBUG:boto:Params: {} 2023-10-11T15:25:31.963 DEBUG:boto:Token: None 2023-10-11T15:25:31.963 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:31 GMT /dghqvd-59/ 2023-10-11T15:25:31.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU= 2023-10-11T15:25:31.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:31 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VutVh3uYoUVIhHQfFcWOno/i3UU=', 'Content-Length': '0'} 2023-10-11T15:25:31.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090b888884337b0f2-006526be6b-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:31 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.014 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:25:32.016 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.016 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.017 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.017 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.018 DEBUG:boto:Method: GET 2023-10-11T15:25:32.018 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.019 DEBUG:boto:Data: 2023-10-11T15:25:32.019 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.020 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.020 DEBUG:boto:Params: {} 2023-10-11T15:25:32.021 DEBUG:boto:Token: None 2023-10-11T15:25:32.021 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.022 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g= 2023-10-11T15:25:32.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g=', 'Content-Length': '0'} 2023-10-11T15:25:32.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007acab89d3b13bc39-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.025 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.026 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.026 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.027 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.027 DEBUG:boto:Method: GET 2023-10-11T15:25:32.028 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.028 DEBUG:boto:Data: 2023-10-11T15:25:32.029 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.029 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.030 DEBUG:boto:Params: {} 2023-10-11T15:25:32.030 DEBUG:boto:Token: None 2023-10-11T15:25:32.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g= 2023-10-11T15:25:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g=', 'Content-Length': '0'} 2023-10-11T15:25:32.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d0303db4e01436-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.035 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.035 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.035 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.036 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.036 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.037 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.037 DEBUG:boto:Data: 2023-10-11T15:25:32.037 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.038 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.039 DEBUG:boto:Params: {} 2023-10-11T15:25:32.039 DEBUG:boto:Token: None 2023-10-11T15:25:32.040 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:774gjV2Fr62vnFcQYTShfbi7BA8= 2023-10-11T15:25:32.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:774gjV2Fr62vnFcQYTShfbi7BA8=', 'Content-Length': '0'} 2023-10-11T15:25:32.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003660d4b044d48798-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.043 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.044 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.044 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.045 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.045 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.045 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.046 DEBUG:boto:Data: 2023-10-11T15:25:32.046 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.047 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.047 DEBUG:boto:Params: {} 2023-10-11T15:25:32.048 DEBUG:boto:Token: None 2023-10-11T15:25:32.048 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:774gjV2Fr62vnFcQYTShfbi7BA8= 2023-10-11T15:25:32.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:774gjV2Fr62vnFcQYTShfbi7BA8=', 'Content-Length': '0'} 2023-10-11T15:25:32.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b82a3d9cda6492-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:25:32.053 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.054 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.054 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.055 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.055 DEBUG:boto:Method: GET 2023-10-11T15:25:32.056 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.056 DEBUG:boto:Data: 2023-10-11T15:25:32.056 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.057 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.058 DEBUG:boto:Params: {} 2023-10-11T15:25:32.058 DEBUG:boto:Token: None 2023-10-11T15:25:32.059 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.059 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g= 2023-10-11T15:25:32.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g=', 'Content-Length': '0'} 2023-10-11T15:25:32.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb8a91601ea9439-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.063 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.063 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.064 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.064 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.064 DEBUG:boto:Method: GET 2023-10-11T15:25:32.065 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.065 DEBUG:boto:Data: 2023-10-11T15:25:32.066 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.067 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.067 DEBUG:boto:Params: {} 2023-10-11T15:25:32.068 DEBUG:boto:Token: None 2023-10-11T15:25:32.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:25:32.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g= 2023-10-11T15:25:32.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lHE2C/d+Zck4aaBi8YwpjlB1c6g=', 'Content-Length': '0'} 2023-10-11T15:25:32.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003deca68cffa30e92-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.072 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.072 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.073 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.073 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.074 DEBUG:boto:Method: GET 2023-10-11T15:25:32.074 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.074 DEBUG:boto:Data: 2023-10-11T15:25:32.075 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.076 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.076 DEBUG:boto:Params: {} 2023-10-11T15:25:32.077 DEBUG:boto:Token: None 2023-10-11T15:25:32.077 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000023239b0560a00c6-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.081 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.082 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.083 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.083 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.083 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.084 DEBUG:boto:Method: GET 2023-10-11T15:25:32.084 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.085 DEBUG:boto:Data: 2023-10-11T15:25:32.085 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.086 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.086 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.086 DEBUG:boto:Params: {} 2023-10-11T15:25:32.087 DEBUG:boto:Token: None 2023-10-11T15:25:32.087 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.088 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc331b488dbecab0-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.093 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.093 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.094 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.094 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.095 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:25:32.095 DEBUG:boto:Data: 2023-10-11T15:25:32.096 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.096 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.097 DEBUG:boto:Params: {} 2023-10-11T15:25:32.097 DEBUG:boto:Token: None 2023-10-11T15:25:32.098 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124 2023-10-11T15:25:32.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LMTQkH5U1gchjfm80ElyjQuLpD0= 2023-10-11T15:25:32.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LMTQkH5U1gchjfm80ElyjQuLpD0=', 'Content-Length': '0'} 2023-10-11T15:25:32.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2a77213f3fb472-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.101 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.102 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.102 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.103 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:25:32.103 DEBUG:boto:Data: 2023-10-11T15:25:32.103 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.104 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.105 DEBUG:boto:Params: {} 2023-10-11T15:25:32.105 DEBUG:boto:Token: None 2023-10-11T15:25:32.105 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124 2023-10-11T15:25:32.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LMTQkH5U1gchjfm80ElyjQuLpD0= 2023-10-11T15:25:32.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LMTQkH5U1gchjfm80ElyjQuLpD0=', 'Content-Length': '0'} 2023-10-11T15:25:32.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f0131af32fd012-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:25:32.109 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.110 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.110 DEBUG:boto:Method: GET 2023-10-11T15:25:32.111 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:25:32.111 DEBUG:boto:Data: 2023-10-11T15:25:32.111 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.112 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.113 DEBUG:boto:Params: {} 2023-10-11T15:25:32.113 DEBUG:boto:Token: None 2023-10-11T15:25:32.114 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124 2023-10-11T15:25:32.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RK7CmrtRdWkZ8YOunkyCm2h9l4w= 2023-10-11T15:25:32.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RK7CmrtRdWkZ8YOunkyCm2h9l4w=', 'Content-Length': '0'} 2023-10-11T15:25:32.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070cb8ad4c2e2dc83-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.117 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:25:32.118 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:25:32.118 DEBUG:boto:Method: GET 2023-10-11T15:25:32.119 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:25:32.119 DEBUG:boto:Data: 2023-10-11T15:25:32.120 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.121 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.121 DEBUG:boto:Params: {} 2023-10-11T15:25:32.121 DEBUG:boto:Token: None 2023-10-11T15:25:32.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj124 2023-10-11T15:25:32.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RK7CmrtRdWkZ8YOunkyCm2h9l4w= 2023-10-11T15:25:32.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RK7CmrtRdWkZ8YOunkyCm2h9l4w=', 'Content-Length': '0'} 2023-10-11T15:25:32.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066c6853acb912412-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.170 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.171 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.171 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.172 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.172 DEBUG:boto:Method: GET 2023-10-11T15:25:32.173 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.173 DEBUG:boto:Data: 2023-10-11T15:25:32.174 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.174 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.175 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.175 DEBUG:boto:Params: {} 2023-10-11T15:25:32.176 DEBUG:boto:Token: None 2023-10-11T15:25:32.176 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3e2e2fcf9db39c-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.180 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.181 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.182 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.182 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.182 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.183 DEBUG:boto:Method: GET 2023-10-11T15:25:32.183 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:25:32.184 DEBUG:boto:Data: 2023-10-11T15:25:32.184 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.185 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.185 DEBUG:boto:Params: {} 2023-10-11T15:25:32.186 DEBUG:boto:Token: None 2023-10-11T15:25:32.186 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000934d293b10697eb0-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.192 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:25:32.193 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.193 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.194 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.194 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.195 DEBUG:boto:Method: GET 2023-10-11T15:25:32.195 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.196 DEBUG:boto:Data: 2023-10-11T15:25:32.196 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.196 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.197 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.197 DEBUG:boto:Params: {} 2023-10-11T15:25:32.198 DEBUG:boto:Token: None 2023-10-11T15:25:32.198 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I= 2023-10-11T15:25:32.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I=', 'Content-Length': '0'} 2023-10-11T15:25:32.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c28e092aebc2bdd5-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.246 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.247 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.247 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.248 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.248 DEBUG:boto:Method: GET 2023-10-11T15:25:32.249 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.249 DEBUG:boto:Data: 2023-10-11T15:25:32.249 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.250 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.250 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.251 DEBUG:boto:Params: {} 2023-10-11T15:25:32.251 DEBUG:boto:Token: None 2023-10-11T15:25:32.253 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.254 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I= 2023-10-11T15:25:32.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I=', 'Content-Length': '0'} 2023-10-11T15:25:32.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e3fa55c9016972-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.258 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.258 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.259 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.260 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.260 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.261 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.261 DEBUG:boto:Data: 2023-10-11T15:25:32.262 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.263 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.263 DEBUG:boto:Params: {} 2023-10-11T15:25:32.264 DEBUG:boto:Token: None 2023-10-11T15:25:32.264 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GTwzb7dKL2iPdMbQyu61F93ai9o= 2023-10-11T15:25:32.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GTwzb7dKL2iPdMbQyu61F93ai9o=', 'Content-Length': '0'} 2023-10-11T15:25:32.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050bdcf1bf7da8b82-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.268 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.268 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.269 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.269 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.270 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.270 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.270 DEBUG:boto:Data: 2023-10-11T15:25:32.271 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.272 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.272 DEBUG:boto:Params: {} 2023-10-11T15:25:32.273 DEBUG:boto:Token: None 2023-10-11T15:25:32.273 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GTwzb7dKL2iPdMbQyu61F93ai9o= 2023-10-11T15:25:32.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GTwzb7dKL2iPdMbQyu61F93ai9o=', 'Content-Length': '0'} 2023-10-11T15:25:32.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35a96d83363293e-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:25:32.278 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.279 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.279 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.279 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.280 DEBUG:boto:Method: GET 2023-10-11T15:25:32.280 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.281 DEBUG:boto:Data: 2023-10-11T15:25:32.281 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.282 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.282 DEBUG:boto:Params: {} 2023-10-11T15:25:32.283 DEBUG:boto:Token: None 2023-10-11T15:25:32.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I= 2023-10-11T15:25:32.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I=', 'Content-Length': '0'} 2023-10-11T15:25:32.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a83bdf99487b38-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.287 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.287 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.288 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.288 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.289 DEBUG:boto:Method: GET 2023-10-11T15:25:32.289 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.290 DEBUG:boto:Data: 2023-10-11T15:25:32.290 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.291 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.291 DEBUG:boto:Params: {} 2023-10-11T15:25:32.292 DEBUG:boto:Token: None 2023-10-11T15:25:32.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:25:32.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I= 2023-10-11T15:25:32.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VVu5Z7ciaJkmMAF1dnaQILWQH0I=', 'Content-Length': '0'} 2023-10-11T15:25:32.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c34b01f7d912ff-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.296 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.296 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.297 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.297 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.298 DEBUG:boto:Method: GET 2023-10-11T15:25:32.298 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.299 DEBUG:boto:Data: 2023-10-11T15:25:32.299 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.300 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.300 DEBUG:boto:Params: {} 2023-10-11T15:25:32.301 DEBUG:boto:Token: None 2023-10-11T15:25:32.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c7b4aaf72dd1cbf-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.346 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.347 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.348 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.348 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.349 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.349 DEBUG:boto:Method: GET 2023-10-11T15:25:32.349 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.350 DEBUG:boto:Data: 2023-10-11T15:25:32.350 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.351 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.351 DEBUG:boto:Params: {} 2023-10-11T15:25:32.352 DEBUG:boto:Token: None 2023-10-11T15:25:32.352 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000898d00aa6a1bc96b-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.358 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.358 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.359 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.359 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.360 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:25:32.360 DEBUG:boto:Data: 2023-10-11T15:25:32.360 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.361 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.362 DEBUG:boto:Params: {} 2023-10-11T15:25:32.362 DEBUG:boto:Token: None 2023-10-11T15:25:32.362 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145 2023-10-11T15:25:32.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3HnmSh8LEr04OpIwTPP57kP1u4= 2023-10-11T15:25:32.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3HnmSh8LEr04OpIwTPP57kP1u4=', 'Content-Length': '0'} 2023-10-11T15:25:32.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b4a46c8c5ec997-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.366 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.367 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.367 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.367 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:25:32.368 DEBUG:boto:Data: 2023-10-11T15:25:32.368 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.369 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.369 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.369 DEBUG:boto:Params: {} 2023-10-11T15:25:32.370 DEBUG:boto:Token: None 2023-10-11T15:25:32.370 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145 2023-10-11T15:25:32.371 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N3HnmSh8LEr04OpIwTPP57kP1u4= 2023-10-11T15:25:32.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N3HnmSh8LEr04OpIwTPP57kP1u4=', 'Content-Length': '0'} 2023-10-11T15:25:32.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18750ccef78b37c-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:25:32.374 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.375 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.375 DEBUG:boto:Method: GET 2023-10-11T15:25:32.375 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:25:32.376 DEBUG:boto:Data: 2023-10-11T15:25:32.376 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.377 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.378 DEBUG:boto:Params: {} 2023-10-11T15:25:32.378 DEBUG:boto:Token: None 2023-10-11T15:25:32.379 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145 2023-10-11T15:25:32.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:usjhbnpuBducn5dl5Ftxi9EicGQ= 2023-10-11T15:25:32.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:usjhbnpuBducn5dl5Ftxi9EicGQ=', 'Content-Length': '0'} 2023-10-11T15:25:32.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35e5e9f03b56bb9-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.382 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:25:32.383 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:25:32.383 DEBUG:boto:Method: GET 2023-10-11T15:25:32.383 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:25:32.384 DEBUG:boto:Data: 2023-10-11T15:25:32.384 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.385 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.386 DEBUG:boto:Params: {} 2023-10-11T15:25:32.386 DEBUG:boto:Token: None 2023-10-11T15:25:32.387 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj145 2023-10-11T15:25:32.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:usjhbnpuBducn5dl5Ftxi9EicGQ= 2023-10-11T15:25:32.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:usjhbnpuBducn5dl5Ftxi9EicGQ=', 'Content-Length': '0'} 2023-10-11T15:25:32.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e141969fad44c9-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.390 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.391 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.391 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.392 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.392 DEBUG:boto:Method: GET 2023-10-11T15:25:32.393 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.393 DEBUG:boto:Data: 2023-10-11T15:25:32.393 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.394 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.395 DEBUG:boto:Params: {} 2023-10-11T15:25:32.395 DEBUG:boto:Token: None 2023-10-11T15:25:32.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541de5739d0fcac3-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.399 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.400 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.401 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.401 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.401 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.402 DEBUG:boto:Method: GET 2023-10-11T15:25:32.402 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:25:32.403 DEBUG:boto:Data: 2023-10-11T15:25:32.403 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.404 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.404 DEBUG:boto:Params: {} 2023-10-11T15:25:32.405 DEBUG:boto:Token: None 2023-10-11T15:25:32.405 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4bd02df971e8d65-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.411 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:25:32.412 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.412 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.413 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.413 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.414 DEBUG:boto:Method: GET 2023-10-11T15:25:32.414 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.415 DEBUG:boto:Data: 2023-10-11T15:25:32.415 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.416 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.416 DEBUG:boto:Params: {} 2023-10-11T15:25:32.417 DEBUG:boto:Token: None 2023-10-11T15:25:32.417 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI= 2023-10-11T15:25:32.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI=', 'Content-Length': '0'} 2023-10-11T15:25:32.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f79ad2034c3c4fa0-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.421 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.421 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.422 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.422 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.423 DEBUG:boto:Method: GET 2023-10-11T15:25:32.423 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.424 DEBUG:boto:Data: 2023-10-11T15:25:32.424 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.425 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.425 DEBUG:boto:Params: {} 2023-10-11T15:25:32.426 DEBUG:boto:Token: None 2023-10-11T15:25:32.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI= 2023-10-11T15:25:32.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI=', 'Content-Length': '0'} 2023-10-11T15:25:32.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cb07552d70d2b22-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.470 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.471 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.472 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.472 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.472 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.473 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.473 DEBUG:boto:Data: 2023-10-11T15:25:32.474 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.475 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.475 DEBUG:boto:Params: {} 2023-10-11T15:25:32.476 DEBUG:boto:Token: None 2023-10-11T15:25:32.476 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.477 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3og9sEgfxPJSiU2lOrLj0tCL3d0= 2023-10-11T15:25:32.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3og9sEgfxPJSiU2lOrLj0tCL3d0=', 'Content-Length': '0'} 2023-10-11T15:25:32.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbcf7bd7f65d28dd-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.481 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.481 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.481 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.482 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.482 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.483 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.483 DEBUG:boto:Data: 2023-10-11T15:25:32.484 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.484 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.484 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.485 DEBUG:boto:Params: {} 2023-10-11T15:25:32.486 DEBUG:boto:Token: None 2023-10-11T15:25:32.486 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.487 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3og9sEgfxPJSiU2lOrLj0tCL3d0= 2023-10-11T15:25:32.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3og9sEgfxPJSiU2lOrLj0tCL3d0=', 'Content-Length': '0'} 2023-10-11T15:25:32.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7803b5d2974caf1-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:25:32.491 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.491 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.491 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.492 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.492 DEBUG:boto:Method: GET 2023-10-11T15:25:32.493 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.493 DEBUG:boto:Data: 2023-10-11T15:25:32.494 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.495 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.495 DEBUG:boto:Params: {} 2023-10-11T15:25:32.496 DEBUG:boto:Token: None 2023-10-11T15:25:32.496 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI= 2023-10-11T15:25:32.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI=', 'Content-Length': '0'} 2023-10-11T15:25:32.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d823edbadd32cb3-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.501 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.502 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.502 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.503 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.503 DEBUG:boto:Method: GET 2023-10-11T15:25:32.504 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.504 DEBUG:boto:Data: 2023-10-11T15:25:32.505 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.505 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.506 DEBUG:boto:Params: {} 2023-10-11T15:25:32.507 DEBUG:boto:Token: None 2023-10-11T15:25:32.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:25:32.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI= 2023-10-11T15:25:32.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rR3NsmqXIPd06mKSbsxG7m4/luI=', 'Content-Length': '0'} 2023-10-11T15:25:32.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c919617e1e45bf3-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.511 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.512 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.513 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.513 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.514 DEBUG:boto:Method: GET 2023-10-11T15:25:32.514 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.515 DEBUG:boto:Data: 2023-10-11T15:25:32.515 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.516 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.516 DEBUG:boto:Params: {} 2023-10-11T15:25:32.517 DEBUG:boto:Token: None 2023-10-11T15:25:32.518 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008702c757f2d1fd77-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.522 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.523 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.523 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.524 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.524 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.525 DEBUG:boto:Method: GET 2023-10-11T15:25:32.525 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.525 DEBUG:boto:Data: 2023-10-11T15:25:32.526 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.527 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.527 DEBUG:boto:Params: {} 2023-10-11T15:25:32.528 DEBUG:boto:Token: None 2023-10-11T15:25:32.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c96b9e264b30f22-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.535 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.536 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.536 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.537 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.537 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:25:32.538 DEBUG:boto:Data: 2023-10-11T15:25:32.538 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.539 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.539 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.540 DEBUG:boto:Params: {} 2023-10-11T15:25:32.540 DEBUG:boto:Token: None 2023-10-11T15:25:32.541 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155 2023-10-11T15:25:32.541 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dEfDjWwz3WeXXNe1YvfRjaPGwX4= 2023-10-11T15:25:32.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dEfDjWwz3WeXXNe1YvfRjaPGwX4=', 'Content-Length': '0'} 2023-10-11T15:25:32.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86db52bb36f5135-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.544 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.545 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.545 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.545 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:25:32.546 DEBUG:boto:Data: 2023-10-11T15:25:32.546 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.547 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.548 DEBUG:boto:Params: {} 2023-10-11T15:25:32.548 DEBUG:boto:Token: None 2023-10-11T15:25:32.549 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155 2023-10-11T15:25:32.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dEfDjWwz3WeXXNe1YvfRjaPGwX4= 2023-10-11T15:25:32.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dEfDjWwz3WeXXNe1YvfRjaPGwX4=', 'Content-Length': '0'} 2023-10-11T15:25:32.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077fa18dca9060a31-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:25:32.553 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.553 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.554 DEBUG:boto:Method: GET 2023-10-11T15:25:32.554 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:25:32.555 DEBUG:boto:Data: 2023-10-11T15:25:32.555 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.556 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.556 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.557 DEBUG:boto:Params: {} 2023-10-11T15:25:32.558 DEBUG:boto:Token: None 2023-10-11T15:25:32.558 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155 2023-10-11T15:25:32.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a1hnRGV3rXi9kv0ue4kHGfYLSzo= 2023-10-11T15:25:32.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a1hnRGV3rXi9kv0ue4kHGfYLSzo=', 'Content-Length': '0'} 2023-10-11T15:25:32.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000764f013e207ad784-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.562 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:25:32.563 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:25:32.563 DEBUG:boto:Method: GET 2023-10-11T15:25:32.564 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:25:32.564 DEBUG:boto:Data: 2023-10-11T15:25:32.565 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.567 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.567 DEBUG:boto:Params: {} 2023-10-11T15:25:32.568 DEBUG:boto:Token: None 2023-10-11T15:25:32.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj155 2023-10-11T15:25:32.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a1hnRGV3rXi9kv0ue4kHGfYLSzo= 2023-10-11T15:25:32.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a1hnRGV3rXi9kv0ue4kHGfYLSzo=', 'Content-Length': '0'} 2023-10-11T15:25:32.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa41c9dce1693a7-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.573 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.573 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.573 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.574 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.574 DEBUG:boto:Method: GET 2023-10-11T15:25:32.575 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.575 DEBUG:boto:Data: 2023-10-11T15:25:32.576 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.576 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.577 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.578 DEBUG:boto:Params: {} 2023-10-11T15:25:32.579 DEBUG:boto:Token: None 2023-10-11T15:25:32.579 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd4c55ed680cc239-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.584 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.585 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.585 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.586 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.586 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.587 DEBUG:boto:Method: GET 2023-10-11T15:25:32.587 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:25:32.587 DEBUG:boto:Data: 2023-10-11T15:25:32.588 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.589 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.589 DEBUG:boto:Params: {} 2023-10-11T15:25:32.590 DEBUG:boto:Token: None 2023-10-11T15:25:32.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053454bdb07da5a50-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.638 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:25:32.640 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.640 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.641 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.641 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.642 DEBUG:boto:Method: GET 2023-10-11T15:25:32.643 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.644 DEBUG:boto:Data: 2023-10-11T15:25:32.644 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.646 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.647 DEBUG:boto:Params: {} 2023-10-11T15:25:32.647 DEBUG:boto:Token: None 2023-10-11T15:25:32.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U= 2023-10-11T15:25:32.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U=', 'Content-Length': '0'} 2023-10-11T15:25:32.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f7a15e492a1006-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.652 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.652 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.653 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.653 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.653 DEBUG:boto:Method: GET 2023-10-11T15:25:32.654 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.655 DEBUG:boto:Data: 2023-10-11T15:25:32.655 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.656 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.656 DEBUG:boto:Params: {} 2023-10-11T15:25:32.657 DEBUG:boto:Token: None 2023-10-11T15:25:32.657 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U= 2023-10-11T15:25:32.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U=', 'Content-Length': '0'} 2023-10-11T15:25:32.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939176fa97f0f6d1-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.662 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.663 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.663 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.664 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.664 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.665 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.665 DEBUG:boto:Data: 2023-10-11T15:25:32.666 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.667 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.667 DEBUG:boto:Params: {} 2023-10-11T15:25:32.668 DEBUG:boto:Token: None 2023-10-11T15:25:32.668 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Mcjc2RZCFUXsHotZzq4v+NV+t4= 2023-10-11T15:25:32.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Mcjc2RZCFUXsHotZzq4v+NV+t4=', 'Content-Length': '0'} 2023-10-11T15:25:32.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002326c08cfdbc771c-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.672 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.672 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.672 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.673 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.673 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.674 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.674 DEBUG:boto:Data: 2023-10-11T15:25:32.675 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.675 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.676 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.676 DEBUG:boto:Params: {} 2023-10-11T15:25:32.677 DEBUG:boto:Token: None 2023-10-11T15:25:32.677 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Mcjc2RZCFUXsHotZzq4v+NV+t4= 2023-10-11T15:25:32.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Mcjc2RZCFUXsHotZzq4v+NV+t4=', 'Content-Length': '0'} 2023-10-11T15:25:32.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f50dc0245f50058c-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:25:32.681 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.682 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.682 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.683 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.684 DEBUG:boto:Method: GET 2023-10-11T15:25:32.684 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.685 DEBUG:boto:Data: 2023-10-11T15:25:32.685 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.685 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.686 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.686 DEBUG:boto:Params: {} 2023-10-11T15:25:32.687 DEBUG:boto:Token: None 2023-10-11T15:25:32.687 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U= 2023-10-11T15:25:32.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U=', 'Content-Length': '0'} 2023-10-11T15:25:32.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000041d271b9b70a651-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.738 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.739 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.740 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.740 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.741 DEBUG:boto:Method: GET 2023-10-11T15:25:32.741 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.742 DEBUG:boto:Data: 2023-10-11T15:25:32.742 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.743 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.744 DEBUG:boto:Params: {} 2023-10-11T15:25:32.744 DEBUG:boto:Token: None 2023-10-11T15:25:32.746 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:25:32.747 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U= 2023-10-11T15:25:32.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8s+X/m8GmdfWwLbv5hpDF8f7Y0U=', 'Content-Length': '0'} 2023-10-11T15:25:32.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9bc081dc8842c51-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.750 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.752 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.752 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.753 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.753 DEBUG:boto:Method: GET 2023-10-11T15:25:32.754 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.754 DEBUG:boto:Data: 2023-10-11T15:25:32.755 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.756 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.756 DEBUG:boto:Params: {} 2023-10-11T15:25:32.757 DEBUG:boto:Token: None 2023-10-11T15:25:32.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6f577b1ff387ee7-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.761 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.762 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.763 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.763 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.764 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.764 DEBUG:boto:Method: GET 2023-10-11T15:25:32.765 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.765 DEBUG:boto:Data: 2023-10-11T15:25:32.766 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.767 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.767 DEBUG:boto:Params: {} 2023-10-11T15:25:32.768 DEBUG:boto:Token: None 2023-10-11T15:25:32.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.769 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000308b1e2538a45ac8-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.774 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.775 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.775 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.775 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.776 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:25:32.776 DEBUG:boto:Data: 2023-10-11T15:25:32.777 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.777 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.778 DEBUG:boto:Params: {} 2023-10-11T15:25:32.778 DEBUG:boto:Token: None 2023-10-11T15:25:32.779 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17 2023-10-11T15:25:32.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:52TQb4FNK8II8NRinLzQg4wDp+E= 2023-10-11T15:25:32.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:52TQb4FNK8II8NRinLzQg4wDp+E=', 'Content-Length': '0'} 2023-10-11T15:25:32.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000876e407e90e1b686-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.783 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.783 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.784 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.784 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:25:32.785 DEBUG:boto:Data: 2023-10-11T15:25:32.785 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.786 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.786 DEBUG:boto:Params: {} 2023-10-11T15:25:32.787 DEBUG:boto:Token: None 2023-10-11T15:25:32.787 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17 2023-10-11T15:25:32.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:52TQb4FNK8II8NRinLzQg4wDp+E= 2023-10-11T15:25:32.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:52TQb4FNK8II8NRinLzQg4wDp+E=', 'Content-Length': '0'} 2023-10-11T15:25:32.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b48f94b9aad135-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:25:32.791 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.792 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.792 DEBUG:boto:Method: GET 2023-10-11T15:25:32.793 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:25:32.793 DEBUG:boto:Data: 2023-10-11T15:25:32.794 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.795 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.795 DEBUG:boto:Params: {} 2023-10-11T15:25:32.796 DEBUG:boto:Token: None 2023-10-11T15:25:32.796 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17 2023-10-11T15:25:32.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:f56fdoBGTv3DrJGH9MLE57j8a5k= 2023-10-11T15:25:32.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:f56fdoBGTv3DrJGH9MLE57j8a5k=', 'Content-Length': '0'} 2023-10-11T15:25:32.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034160047d081a589-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.800 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:25:32.801 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:25:32.801 DEBUG:boto:Method: GET 2023-10-11T15:25:32.801 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:25:32.802 DEBUG:boto:Data: 2023-10-11T15:25:32.802 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.803 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.804 DEBUG:boto:Params: {} 2023-10-11T15:25:32.804 DEBUG:boto:Token: None 2023-10-11T15:25:32.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj17 2023-10-11T15:25:32.805 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:f56fdoBGTv3DrJGH9MLE57j8a5k= 2023-10-11T15:25:32.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:f56fdoBGTv3DrJGH9MLE57j8a5k=', 'Content-Length': '0'} 2023-10-11T15:25:32.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29774ae8080c080-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.850 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.851 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.852 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.852 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.853 DEBUG:boto:Method: GET 2023-10-11T15:25:32.853 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.854 DEBUG:boto:Data: 2023-10-11T15:25:32.854 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.855 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.855 DEBUG:boto:Params: {} 2023-10-11T15:25:32.856 DEBUG:boto:Token: None 2023-10-11T15:25:32.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa6fd1279c7fdbd-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.861 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.862 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.862 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.863 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.863 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.864 DEBUG:boto:Method: GET 2023-10-11T15:25:32.864 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:25:32.865 DEBUG:boto:Data: 2023-10-11T15:25:32.865 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.866 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.867 DEBUG:boto:Params: {} 2023-10-11T15:25:32.868 DEBUG:boto:Token: None 2023-10-11T15:25:32.868 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.869 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf0f5f56b878f0d-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.874 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:25:32.875 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.875 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.876 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.876 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.877 DEBUG:boto:Method: GET 2023-10-11T15:25:32.877 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.878 DEBUG:boto:Data: 2023-10-11T15:25:32.878 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.879 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.879 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.880 DEBUG:boto:Params: {} 2023-10-11T15:25:32.880 DEBUG:boto:Token: None 2023-10-11T15:25:32.881 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY= 2023-10-11T15:25:32.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY=', 'Content-Length': '0'} 2023-10-11T15:25:32.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092501ccf49e8e41a-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.885 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.885 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.886 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.886 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.887 DEBUG:boto:Method: GET 2023-10-11T15:25:32.887 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.888 DEBUG:boto:Data: 2023-10-11T15:25:32.888 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.889 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.890 DEBUG:boto:Params: {} 2023-10-11T15:25:32.890 DEBUG:boto:Token: None 2023-10-11T15:25:32.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY= 2023-10-11T15:25:32.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY=', 'Content-Length': '0'} 2023-10-11T15:25:32.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd79ed4829ae8698-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.895 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.896 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.896 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.896 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.897 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.897 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.898 DEBUG:boto:Data: 2023-10-11T15:25:32.898 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.899 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.899 DEBUG:boto:Params: {} 2023-10-11T15:25:32.900 DEBUG:boto:Token: None 2023-10-11T15:25:32.900 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cWqgXuOPayFV2O6VbRhW1b7nWxw= 2023-10-11T15:25:32.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cWqgXuOPayFV2O6VbRhW1b7nWxw=', 'Content-Length': '0'} 2023-10-11T15:25:32.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891835e88497e6df-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.904 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.904 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.905 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.905 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.906 DEBUG:boto:Method: HEAD 2023-10-11T15:25:32.906 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.907 DEBUG:boto:Data: 2023-10-11T15:25:32.907 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.908 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.908 DEBUG:boto:Params: {} 2023-10-11T15:25:32.909 DEBUG:boto:Token: None 2023-10-11T15:25:32.909 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cWqgXuOPayFV2O6VbRhW1b7nWxw= 2023-10-11T15:25:32.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cWqgXuOPayFV2O6VbRhW1b7nWxw=', 'Content-Length': '0'} 2023-10-11T15:25:32.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd69d3ba87f8693-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:25:32.913 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.914 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.915 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.915 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.915 DEBUG:boto:Method: GET 2023-10-11T15:25:32.916 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.916 DEBUG:boto:Data: 2023-10-11T15:25:32.917 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.918 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.918 DEBUG:boto:Params: {} 2023-10-11T15:25:32.919 DEBUG:boto:Token: None 2023-10-11T15:25:32.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY= 2023-10-11T15:25:32.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY=', 'Content-Length': '0'} 2023-10-11T15:25:32.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066aa75ffe7a9cd8d-006526be6c-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.923 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:32.924 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:32.924 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.925 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.925 DEBUG:boto:Method: GET 2023-10-11T15:25:32.926 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.926 DEBUG:boto:Data: 2023-10-11T15:25:32.927 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.927 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.928 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.928 DEBUG:boto:Params: {} 2023-10-11T15:25:32.929 DEBUG:boto:Token: None 2023-10-11T15:25:32.929 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:25:32.930 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY= 2023-10-11T15:25:32.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqqjSQD3TXlkMJ2iNuWUn149BRY=', 'Content-Length': '0'} 2023-10-11T15:25:32.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815ed9da6ee7d741-006526be6c-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.934 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.934 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.934 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.935 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.935 DEBUG:boto:Method: GET 2023-10-11T15:25:32.936 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.936 DEBUG:boto:Data: 2023-10-11T15:25:32.937 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:32.938 DEBUG:boto:Port: 8001 2023-10-11T15:25:32.938 DEBUG:boto:Params: {} 2023-10-11T15:25:32.939 DEBUG:boto:Token: None 2023-10-11T15:25:32.939 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b29e1df80b0d5617-006526be6c-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.986 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:32.987 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:32.988 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:32.988 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.989 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.989 DEBUG:boto:Method: GET 2023-10-11T15:25:32.990 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:25:32.990 DEBUG:boto:Data: 2023-10-11T15:25:32.991 DEBUG:boto:Headers: {} 2023-10-11T15:25:32.991 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:32.991 DEBUG:boto:Port: 8000 2023-10-11T15:25:32.992 DEBUG:boto:Params: {} 2023-10-11T15:25:32.992 DEBUG:boto:Token: None 2023-10-11T15:25:32.993 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:32 GMT /dghqvd-59/ 2023-10-11T15:25:32.994 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM= 2023-10-11T15:25:32.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pOCvx5flYDfk69sKQEIO1m8RTCM=', 'Content-Length': '0'} 2023-10-11T15:25:32.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff460ebd23cf1342-006526be6c-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:32.999 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.000 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:33.000 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:33.000 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.001 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:25:33.001 DEBUG:boto:Data: 2023-10-11T15:25:33.002 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.003 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.003 DEBUG:boto:Params: {} 2023-10-11T15:25:33.004 DEBUG:boto:Token: None 2023-10-11T15:25:33.004 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj19 2023-10-11T15:25:33.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E3tjkoIZRWC9+EMcLp+OXLRmF7Q= 2023-10-11T15:25:33.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E3tjkoIZRWC9+EMcLp+OXLRmF7Q=', 'Content-Length': '0'} 2023-10-11T15:25:33.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002788cb14ed771eeb-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.009 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:33.009 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:33.010 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.010 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:25:33.011 DEBUG:boto:Data: 2023-10-11T15:25:33.011 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.011 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.012 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.012 DEBUG:boto:Params: {} 2023-10-11T15:25:33.013 DEBUG:boto:Token: None 2023-10-11T15:25:33.013 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj19 2023-10-11T15:25:33.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E3tjkoIZRWC9+EMcLp+OXLRmF7Q= 2023-10-11T15:25:33.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E3tjkoIZRWC9+EMcLp+OXLRmF7Q=', 'Content-Length': '0'} 2023-10-11T15:25:33.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e312ebd453f841-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:25:33.018 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:33.018 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:33.019 DEBUG:boto:Method: GET 2023-10-11T15:25:33.019 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:25:33.019 DEBUG:boto:Data: 2023-10-11T15:25:33.020 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.021 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.021 DEBUG:boto:Params: {} 2023-10-11T15:25:33.021 DEBUG:boto:Token: None 2023-10-11T15:25:33.022 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj19 2023-10-11T15:25:33.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9GXrvVPOEFN04p8mIzOIqQv0ie4= 2023-10-11T15:25:33.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9GXrvVPOEFN04p8mIzOIqQv0ie4=', 'Content-Length': '0'} 2023-10-11T15:25:33.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ede7835e3e9646-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.026 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:25:33.027 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:25:33.027 DEBUG:boto:Method: GET 2023-10-11T15:25:33.028 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:25:33.028 DEBUG:boto:Data: 2023-10-11T15:25:33.028 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.029 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.030 DEBUG:boto:Params: {} 2023-10-11T15:25:33.030 DEBUG:boto:Token: None 2023-10-11T15:25:33.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj19 2023-10-11T15:25:33.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9GXrvVPOEFN04p8mIzOIqQv0ie4= 2023-10-11T15:25:33.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9GXrvVPOEFN04p8mIzOIqQv0ie4=', 'Content-Length': '0'} 2023-10-11T15:25:33.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000344ec22b6c22e79-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.074 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.075 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.076 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.076 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.077 DEBUG:boto:Method: GET 2023-10-11T15:25:33.077 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.078 DEBUG:boto:Data: 2023-10-11T15:25:33.078 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.079 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.079 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.080 DEBUG:boto:Params: {} 2023-10-11T15:25:33.080 DEBUG:boto:Token: None 2023-10-11T15:25:33.081 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.081 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c3b05f7bd3338b-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.085 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.086 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.086 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.087 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.087 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.087 DEBUG:boto:Method: GET 2023-10-11T15:25:33.088 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:25:33.088 DEBUG:boto:Data: 2023-10-11T15:25:33.089 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.089 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.090 DEBUG:boto:Params: {} 2023-10-11T15:25:33.090 DEBUG:boto:Token: None 2023-10-11T15:25:33.091 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3880dc08479a3a-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.096 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:25:33.097 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.098 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.098 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.099 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.099 DEBUG:boto:Method: GET 2023-10-11T15:25:33.100 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.100 DEBUG:boto:Data: 2023-10-11T15:25:33.100 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.101 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.102 DEBUG:boto:Params: {} 2023-10-11T15:25:33.102 DEBUG:boto:Token: None 2023-10-11T15:25:33.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk= 2023-10-11T15:25:33.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk=', 'Content-Length': '0'} 2023-10-11T15:25:33.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beabd058094a53da-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.107 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.107 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.108 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.108 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.108 DEBUG:boto:Method: GET 2023-10-11T15:25:33.109 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.109 DEBUG:boto:Data: 2023-10-11T15:25:33.110 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.111 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.111 DEBUG:boto:Params: {} 2023-10-11T15:25:33.112 DEBUG:boto:Token: None 2023-10-11T15:25:33.112 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk= 2023-10-11T15:25:33.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk=', 'Content-Length': '0'} 2023-10-11T15:25:33.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa033ec0ee10b95f-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.116 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.117 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.118 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.119 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.119 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.120 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.121 DEBUG:boto:Data: 2023-10-11T15:25:33.122 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.123 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.124 DEBUG:boto:Params: {} 2023-10-11T15:25:33.125 DEBUG:boto:Token: None 2023-10-11T15:25:33.125 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DfrVzpvdogY4KFLPpXFoz9fJ0uI= 2023-10-11T15:25:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DfrVzpvdogY4KFLPpXFoz9fJ0uI=', 'Content-Length': '0'} 2023-10-11T15:25:33.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca3ec78396415673-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.130 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.131 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.132 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.132 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.133 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.134 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.134 DEBUG:boto:Data: 2023-10-11T15:25:33.135 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.137 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.137 DEBUG:boto:Params: {} 2023-10-11T15:25:33.138 DEBUG:boto:Token: None 2023-10-11T15:25:33.139 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.139 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DfrVzpvdogY4KFLPpXFoz9fJ0uI= 2023-10-11T15:25:33.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DfrVzpvdogY4KFLPpXFoz9fJ0uI=', 'Content-Length': '0'} 2023-10-11T15:25:33.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d696c2c92d6462-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:25:33.143 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.144 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.144 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.145 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.145 DEBUG:boto:Method: GET 2023-10-11T15:25:33.146 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.146 DEBUG:boto:Data: 2023-10-11T15:25:33.147 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.148 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.148 DEBUG:boto:Params: {} 2023-10-11T15:25:33.149 DEBUG:boto:Token: None 2023-10-11T15:25:33.149 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk= 2023-10-11T15:25:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk=', 'Content-Length': '0'} 2023-10-11T15:25:33.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f723c04c3bc08b40-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.153 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.154 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.154 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.155 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.155 DEBUG:boto:Method: GET 2023-10-11T15:25:33.155 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.156 DEBUG:boto:Data: 2023-10-11T15:25:33.156 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.157 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.157 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.158 DEBUG:boto:Params: {} 2023-10-11T15:25:33.158 DEBUG:boto:Token: None 2023-10-11T15:25:33.159 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:25:33.159 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk= 2023-10-11T15:25:33.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1j6869zNjkd9vsJoiyUWZoCLtk=', 'Content-Length': '0'} 2023-10-11T15:25:33.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036dcad2b48680ca5-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.202 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.203 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.203 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.204 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.204 DEBUG:boto:Method: GET 2023-10-11T15:25:33.205 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.205 DEBUG:boto:Data: 2023-10-11T15:25:33.206 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.206 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.206 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.207 DEBUG:boto:Params: {} 2023-10-11T15:25:33.207 DEBUG:boto:Token: None 2023-10-11T15:25:33.208 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.208 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3539e73e04492c9-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.212 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.213 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.213 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.214 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.214 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.215 DEBUG:boto:Method: GET 2023-10-11T15:25:33.215 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.215 DEBUG:boto:Data: 2023-10-11T15:25:33.216 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.217 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.217 DEBUG:boto:Params: {} 2023-10-11T15:25:33.217 DEBUG:boto:Token: None 2023-10-11T15:25:33.218 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000660e4b4750a8c872-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.223 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.224 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.225 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.225 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.226 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:25:33.226 DEBUG:boto:Data: 2023-10-11T15:25:33.226 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.227 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.227 DEBUG:boto:Params: {} 2023-10-11T15:25:33.228 DEBUG:boto:Token: None 2023-10-11T15:25:33.228 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22 2023-10-11T15:25:33.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mknB8Sy5rNaO3JOg/ZTfOblm91I= 2023-10-11T15:25:33.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mknB8Sy5rNaO3JOg/ZTfOblm91I=', 'Content-Length': '0'} 2023-10-11T15:25:33.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000247d6f0cbc1627d6-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.232 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.232 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.233 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.233 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:25:33.233 DEBUG:boto:Data: 2023-10-11T15:25:33.234 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.235 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.235 DEBUG:boto:Params: {} 2023-10-11T15:25:33.235 DEBUG:boto:Token: None 2023-10-11T15:25:33.236 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22 2023-10-11T15:25:33.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mknB8Sy5rNaO3JOg/ZTfOblm91I= 2023-10-11T15:25:33.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mknB8Sy5rNaO3JOg/ZTfOblm91I=', 'Content-Length': '0'} 2023-10-11T15:25:33.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffa1965394dbe12-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:25:33.240 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.240 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.241 DEBUG:boto:Method: GET 2023-10-11T15:25:33.241 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:25:33.241 DEBUG:boto:Data: 2023-10-11T15:25:33.242 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.242 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.243 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.243 DEBUG:boto:Params: {} 2023-10-11T15:25:33.244 DEBUG:boto:Token: None 2023-10-11T15:25:33.244 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22 2023-10-11T15:25:33.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fbPm7qegSZxkClLrL6HztJB2Rqw= 2023-10-11T15:25:33.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fbPm7qegSZxkClLrL6HztJB2Rqw=', 'Content-Length': '0'} 2023-10-11T15:25:33.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efc82abfbdc021f1-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.290 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:25:33.291 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:25:33.291 DEBUG:boto:Method: GET 2023-10-11T15:25:33.292 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:25:33.292 DEBUG:boto:Data: 2023-10-11T15:25:33.293 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.293 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.294 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.294 DEBUG:boto:Params: {} 2023-10-11T15:25:33.295 DEBUG:boto:Token: None 2023-10-11T15:25:33.295 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj22 2023-10-11T15:25:33.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fbPm7qegSZxkClLrL6HztJB2Rqw= 2023-10-11T15:25:33.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fbPm7qegSZxkClLrL6HztJB2Rqw=', 'Content-Length': '0'} 2023-10-11T15:25:33.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d7e1f56909a22c-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.299 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.299 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.300 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.300 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.301 DEBUG:boto:Method: GET 2023-10-11T15:25:33.301 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.302 DEBUG:boto:Data: 2023-10-11T15:25:33.302 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.303 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.303 DEBUG:boto:Params: {} 2023-10-11T15:25:33.304 DEBUG:boto:Token: None 2023-10-11T15:25:33.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720e15a4ad272c48-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.308 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.309 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.310 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.310 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.310 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.311 DEBUG:boto:Method: GET 2023-10-11T15:25:33.311 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:25:33.312 DEBUG:boto:Data: 2023-10-11T15:25:33.312 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.313 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.313 DEBUG:boto:Params: {} 2023-10-11T15:25:33.314 DEBUG:boto:Token: None 2023-10-11T15:25:33.314 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.315 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e758b26b45b483-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.366 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:25:33.368 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.368 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.369 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.369 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.369 DEBUG:boto:Method: GET 2023-10-11T15:25:33.370 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.370 DEBUG:boto:Data: 2023-10-11T15:25:33.371 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.372 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.372 DEBUG:boto:Params: {} 2023-10-11T15:25:33.372 DEBUG:boto:Token: None 2023-10-11T15:25:33.373 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk= 2023-10-11T15:25:33.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk=', 'Content-Length': '0'} 2023-10-11T15:25:33.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6dd8d0e1b5fd0d-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.377 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.377 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.378 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.378 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.379 DEBUG:boto:Method: GET 2023-10-11T15:25:33.379 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.380 DEBUG:boto:Data: 2023-10-11T15:25:33.380 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.380 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.381 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.381 DEBUG:boto:Params: {} 2023-10-11T15:25:33.382 DEBUG:boto:Token: None 2023-10-11T15:25:33.382 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk= 2023-10-11T15:25:33.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk=', 'Content-Length': '0'} 2023-10-11T15:25:33.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e77f1588f2d642a-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.386 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.386 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.387 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.387 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.388 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.388 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.388 DEBUG:boto:Data: 2023-10-11T15:25:33.389 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.390 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.390 DEBUG:boto:Params: {} 2023-10-11T15:25:33.390 DEBUG:boto:Token: None 2023-10-11T15:25:33.391 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kq2a1QyZvUbl01XVc8QrE7Sl7P8= 2023-10-11T15:25:33.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kq2a1QyZvUbl01XVc8QrE7Sl7P8=', 'Content-Length': '0'} 2023-10-11T15:25:33.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000105b165256c0fdc6-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.394 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.395 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.395 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.396 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.396 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.397 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.397 DEBUG:boto:Data: 2023-10-11T15:25:33.397 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.398 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.399 DEBUG:boto:Params: {} 2023-10-11T15:25:33.399 DEBUG:boto:Token: None 2023-10-11T15:25:33.400 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kq2a1QyZvUbl01XVc8QrE7Sl7P8= 2023-10-11T15:25:33.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kq2a1QyZvUbl01XVc8QrE7Sl7P8=', 'Content-Length': '0'} 2023-10-11T15:25:33.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58f96c1d5aa5609-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:25:33.403 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.404 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.404 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.405 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.405 DEBUG:boto:Method: GET 2023-10-11T15:25:33.406 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.406 DEBUG:boto:Data: 2023-10-11T15:25:33.406 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.407 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.408 DEBUG:boto:Params: {} 2023-10-11T15:25:33.408 DEBUG:boto:Token: None 2023-10-11T15:25:33.409 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk= 2023-10-11T15:25:33.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk=', 'Content-Length': '0'} 2023-10-11T15:25:33.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003da324a94cec1804-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.412 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.413 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.413 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.414 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.414 DEBUG:boto:Method: GET 2023-10-11T15:25:33.415 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.415 DEBUG:boto:Data: 2023-10-11T15:25:33.415 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.416 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.417 DEBUG:boto:Params: {} 2023-10-11T15:25:33.417 DEBUG:boto:Token: None 2023-10-11T15:25:33.418 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:25:33.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk= 2023-10-11T15:25:33.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hb4PygW9psYa5sb7ZTGZIEqcrjk=', 'Content-Length': '0'} 2023-10-11T15:25:33.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002292c8ff1b493d78-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.422 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.422 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.422 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.423 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.423 DEBUG:boto:Method: GET 2023-10-11T15:25:33.424 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.424 DEBUG:boto:Data: 2023-10-11T15:25:33.425 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.425 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.426 DEBUG:boto:Params: {} 2023-10-11T15:25:33.426 DEBUG:boto:Token: None 2023-10-11T15:25:33.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006394f98884c2a958-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.431 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.432 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.432 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.433 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.433 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.434 DEBUG:boto:Method: GET 2023-10-11T15:25:33.434 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.435 DEBUG:boto:Data: 2023-10-11T15:25:33.435 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.436 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.436 DEBUG:boto:Params: {} 2023-10-11T15:25:33.437 DEBUG:boto:Token: None 2023-10-11T15:25:33.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcd1409ca890d9d8-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.443 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.443 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.444 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.444 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.445 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:25:33.445 DEBUG:boto:Data: 2023-10-11T15:25:33.445 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.446 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.447 DEBUG:boto:Params: {} 2023-10-11T15:25:33.447 DEBUG:boto:Token: None 2023-10-11T15:25:33.448 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23 2023-10-11T15:25:33.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ePc3i4iEwRS01OHzFH78rM8hQpk= 2023-10-11T15:25:33.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ePc3i4iEwRS01OHzFH78rM8hQpk=', 'Content-Length': '0'} 2023-10-11T15:25:33.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a320ecd300b72282-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.451 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.451 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.452 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.452 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:25:33.453 DEBUG:boto:Data: 2023-10-11T15:25:33.453 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.454 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.454 DEBUG:boto:Params: {} 2023-10-11T15:25:33.455 DEBUG:boto:Token: None 2023-10-11T15:25:33.455 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23 2023-10-11T15:25:33.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ePc3i4iEwRS01OHzFH78rM8hQpk= 2023-10-11T15:25:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ePc3i4iEwRS01OHzFH78rM8hQpk=', 'Content-Length': '0'} 2023-10-11T15:25:33.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b2ed5b5f2e69833-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:25:33.459 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.460 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.460 DEBUG:boto:Method: GET 2023-10-11T15:25:33.461 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:25:33.461 DEBUG:boto:Data: 2023-10-11T15:25:33.461 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.462 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.463 DEBUG:boto:Params: {} 2023-10-11T15:25:33.463 DEBUG:boto:Token: None 2023-10-11T15:25:33.464 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23 2023-10-11T15:25:33.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/YVq7GkUMZMegVbtFguK0khanA= 2023-10-11T15:25:33.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/YVq7GkUMZMegVbtFguK0khanA=', 'Content-Length': '0'} 2023-10-11T15:25:33.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f28da9229e3a36-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.510 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:25:33.511 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:25:33.511 DEBUG:boto:Method: GET 2023-10-11T15:25:33.512 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:25:33.512 DEBUG:boto:Data: 2023-10-11T15:25:33.513 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.514 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.514 DEBUG:boto:Params: {} 2023-10-11T15:25:33.515 DEBUG:boto:Token: None 2023-10-11T15:25:33.515 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj23 2023-10-11T15:25:33.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F/YVq7GkUMZMegVbtFguK0khanA= 2023-10-11T15:25:33.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F/YVq7GkUMZMegVbtFguK0khanA=', 'Content-Length': '0'} 2023-10-11T15:25:33.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dce24411a27737-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.519 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.519 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.520 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.520 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.521 DEBUG:boto:Method: GET 2023-10-11T15:25:33.521 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.521 DEBUG:boto:Data: 2023-10-11T15:25:33.522 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.523 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.523 DEBUG:boto:Params: {} 2023-10-11T15:25:33.524 DEBUG:boto:Token: None 2023-10-11T15:25:33.524 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d01da9855669f94-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.528 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.529 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.530 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.530 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.531 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.531 DEBUG:boto:Method: GET 2023-10-11T15:25:33.531 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:25:33.532 DEBUG:boto:Data: 2023-10-11T15:25:33.532 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.533 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.534 DEBUG:boto:Params: {} 2023-10-11T15:25:33.534 DEBUG:boto:Token: None 2023-10-11T15:25:33.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea6cca256f00837-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.586 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:25:33.588 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.588 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.589 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.589 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.590 DEBUG:boto:Method: GET 2023-10-11T15:25:33.590 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.591 DEBUG:boto:Data: 2023-10-11T15:25:33.591 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.592 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.592 DEBUG:boto:Params: {} 2023-10-11T15:25:33.593 DEBUG:boto:Token: None 2023-10-11T15:25:33.593 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g= 2023-10-11T15:25:33.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g=', 'Content-Length': '0'} 2023-10-11T15:25:33.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680904af30029a8a-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.597 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.598 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.598 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.598 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.599 DEBUG:boto:Method: GET 2023-10-11T15:25:33.599 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.600 DEBUG:boto:Data: 2023-10-11T15:25:33.600 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.601 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.602 DEBUG:boto:Params: {} 2023-10-11T15:25:33.602 DEBUG:boto:Token: None 2023-10-11T15:25:33.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g= 2023-10-11T15:25:33.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g=', 'Content-Length': '0'} 2023-10-11T15:25:33.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043092a1d91349062-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.610 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.611 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.611 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.611 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.612 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.612 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.612 DEBUG:boto:Data: 2023-10-11T15:25:33.613 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.614 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.614 DEBUG:boto:Params: {} 2023-10-11T15:25:33.615 DEBUG:boto:Token: None 2023-10-11T15:25:33.615 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:atkTBcLp3X2PayiKUdqdch2OiC0= 2023-10-11T15:25:33.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:atkTBcLp3X2PayiKUdqdch2OiC0=', 'Content-Length': '0'} 2023-10-11T15:25:33.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a1f67455eed2981-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.619 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.619 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.619 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.620 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.620 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.621 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.621 DEBUG:boto:Data: 2023-10-11T15:25:33.622 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.622 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.623 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.623 DEBUG:boto:Params: {} 2023-10-11T15:25:33.624 DEBUG:boto:Token: None 2023-10-11T15:25:33.624 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.625 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:atkTBcLp3X2PayiKUdqdch2OiC0= 2023-10-11T15:25:33.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:atkTBcLp3X2PayiKUdqdch2OiC0=', 'Content-Length': '0'} 2023-10-11T15:25:33.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2ecbffd7edc637-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:25:33.629 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.629 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.629 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.630 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.630 DEBUG:boto:Method: GET 2023-10-11T15:25:33.631 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.632 DEBUG:boto:Data: 2023-10-11T15:25:33.632 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.633 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.633 DEBUG:boto:Params: {} 2023-10-11T15:25:33.634 DEBUG:boto:Token: None 2023-10-11T15:25:33.634 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g= 2023-10-11T15:25:33.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g=', 'Content-Length': '0'} 2023-10-11T15:25:33.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006712ae71daac3346-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.638 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.639 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.639 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.639 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.640 DEBUG:boto:Method: GET 2023-10-11T15:25:33.640 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.641 DEBUG:boto:Data: 2023-10-11T15:25:33.641 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.642 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.643 DEBUG:boto:Params: {} 2023-10-11T15:25:33.643 DEBUG:boto:Token: None 2023-10-11T15:25:33.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:25:33.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g= 2023-10-11T15:25:33.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NVe5GlOrLKnGzPGFT43dhydhr+g=', 'Content-Length': '0'} 2023-10-11T15:25:33.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf639d054c32892-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.648 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.648 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.649 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.649 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.650 DEBUG:boto:Method: GET 2023-10-11T15:25:33.650 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.651 DEBUG:boto:Data: 2023-10-11T15:25:33.651 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.652 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.653 DEBUG:boto:Params: {} 2023-10-11T15:25:33.653 DEBUG:boto:Token: None 2023-10-11T15:25:33.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007408efcb9d4c3296-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.658 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.659 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.659 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.660 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.660 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.661 DEBUG:boto:Method: GET 2023-10-11T15:25:33.661 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.662 DEBUG:boto:Data: 2023-10-11T15:25:33.662 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.663 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.663 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.663 DEBUG:boto:Params: {} 2023-10-11T15:25:33.664 DEBUG:boto:Token: None 2023-10-11T15:25:33.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e98ab1c97b2c56c-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.670 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.671 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.671 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.671 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.672 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:25:33.672 DEBUG:boto:Data: 2023-10-11T15:25:33.673 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.673 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.674 DEBUG:boto:Params: {} 2023-10-11T15:25:33.674 DEBUG:boto:Token: None 2023-10-11T15:25:33.675 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29 2023-10-11T15:25:33.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:08WIBtai+uUzXnW+3WumB0C9R64= 2023-10-11T15:25:33.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:08WIBtai+uUzXnW+3WumB0C9R64=', 'Content-Length': '0'} 2023-10-11T15:25:33.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae2a5044ff5142c8-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.678 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.679 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.679 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.680 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:25:33.680 DEBUG:boto:Data: 2023-10-11T15:25:33.680 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.681 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.682 DEBUG:boto:Params: {} 2023-10-11T15:25:33.682 DEBUG:boto:Token: None 2023-10-11T15:25:33.683 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29 2023-10-11T15:25:33.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:08WIBtai+uUzXnW+3WumB0C9R64= 2023-10-11T15:25:33.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:08WIBtai+uUzXnW+3WumB0C9R64=', 'Content-Length': '0'} 2023-10-11T15:25:33.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349b3e2c25b135f9-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:25:33.687 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.688 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.688 DEBUG:boto:Method: GET 2023-10-11T15:25:33.689 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:25:33.689 DEBUG:boto:Data: 2023-10-11T15:25:33.689 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.690 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.691 DEBUG:boto:Params: {} 2023-10-11T15:25:33.691 DEBUG:boto:Token: None 2023-10-11T15:25:33.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29 2023-10-11T15:25:33.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XrsUexuY4kzLBYa5tF9i0Dvmh/w= 2023-10-11T15:25:33.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XrsUexuY4kzLBYa5tF9i0Dvmh/w=', 'Content-Length': '0'} 2023-10-11T15:25:33.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d9fe052803aaf6-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.695 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:25:33.696 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:25:33.696 DEBUG:boto:Method: GET 2023-10-11T15:25:33.697 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:25:33.697 DEBUG:boto:Data: 2023-10-11T15:25:33.698 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.699 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.699 DEBUG:boto:Params: {} 2023-10-11T15:25:33.699 DEBUG:boto:Token: None 2023-10-11T15:25:33.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj29 2023-10-11T15:25:33.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XrsUexuY4kzLBYa5tF9i0Dvmh/w= 2023-10-11T15:25:33.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XrsUexuY4kzLBYa5tF9i0Dvmh/w=', 'Content-Length': '0'} 2023-10-11T15:25:33.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90e41b0a9bf091d-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.750 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.751 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.751 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.752 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.752 DEBUG:boto:Method: GET 2023-10-11T15:25:33.753 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.753 DEBUG:boto:Data: 2023-10-11T15:25:33.754 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.754 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.755 DEBUG:boto:Params: {} 2023-10-11T15:25:33.755 DEBUG:boto:Token: None 2023-10-11T15:25:33.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3cb0dc9a0c3827a-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.760 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.761 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.761 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.762 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.762 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.763 DEBUG:boto:Method: GET 2023-10-11T15:25:33.763 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:25:33.764 DEBUG:boto:Data: 2023-10-11T15:25:33.764 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.765 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.765 DEBUG:boto:Params: {} 2023-10-11T15:25:33.766 DEBUG:boto:Token: None 2023-10-11T15:25:33.766 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34efa057ec87f53-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.772 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:25:33.773 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.773 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.774 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.774 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.775 DEBUG:boto:Method: GET 2023-10-11T15:25:33.775 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.776 DEBUG:boto:Data: 2023-10-11T15:25:33.776 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.777 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.777 DEBUG:boto:Params: {} 2023-10-11T15:25:33.778 DEBUG:boto:Token: None 2023-10-11T15:25:33.778 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0= 2023-10-11T15:25:33.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0=', 'Content-Length': '0'} 2023-10-11T15:25:33.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692b65579b68c5e9-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.783 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.783 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.784 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.784 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.785 DEBUG:boto:Method: GET 2023-10-11T15:25:33.785 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.785 DEBUG:boto:Data: 2023-10-11T15:25:33.786 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.787 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.787 DEBUG:boto:Params: {} 2023-10-11T15:25:33.788 DEBUG:boto:Token: None 2023-10-11T15:25:33.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0= 2023-10-11T15:25:33.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0=', 'Content-Length': '0'} 2023-10-11T15:25:33.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9a51ddd28091b9-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.793 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.794 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.794 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.795 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.796 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.796 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.797 DEBUG:boto:Data: 2023-10-11T15:25:33.797 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.798 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.799 DEBUG:boto:Params: {} 2023-10-11T15:25:33.799 DEBUG:boto:Token: None 2023-10-11T15:25:33.800 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vtb9SR/GwRfLu2g7EsUFp4KBFl4= 2023-10-11T15:25:33.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vtb9SR/GwRfLu2g7EsUFp4KBFl4=', 'Content-Length': '0'} 2023-10-11T15:25:33.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfcc4bd0fbf4ca4-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.804 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.804 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.804 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.805 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.805 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.806 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.806 DEBUG:boto:Data: 2023-10-11T15:25:33.807 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.808 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.808 DEBUG:boto:Params: {} 2023-10-11T15:25:33.809 DEBUG:boto:Token: None 2023-10-11T15:25:33.809 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vtb9SR/GwRfLu2g7EsUFp4KBFl4= 2023-10-11T15:25:33.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vtb9SR/GwRfLu2g7EsUFp4KBFl4=', 'Content-Length': '0'} 2023-10-11T15:25:33.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5eb0a13bc31a94-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:25:33.813 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.814 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.814 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.815 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.815 DEBUG:boto:Method: GET 2023-10-11T15:25:33.816 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.816 DEBUG:boto:Data: 2023-10-11T15:25:33.816 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.817 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.818 DEBUG:boto:Params: {} 2023-10-11T15:25:33.819 DEBUG:boto:Token: None 2023-10-11T15:25:33.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0= 2023-10-11T15:25:33.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0=', 'Content-Length': '0'} 2023-10-11T15:25:33.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb18a78232db0053-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.870 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.871 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.871 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.872 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.872 DEBUG:boto:Method: GET 2023-10-11T15:25:33.873 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.873 DEBUG:boto:Data: 2023-10-11T15:25:33.874 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.874 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.875 DEBUG:boto:Params: {} 2023-10-11T15:25:33.875 DEBUG:boto:Token: None 2023-10-11T15:25:33.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:25:33.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0= 2023-10-11T15:25:33.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lXMd9HFAWRch0VVcSD+WE40nJF0=', 'Content-Length': '0'} 2023-10-11T15:25:33.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029634d21c72d1dee-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.880 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.880 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.881 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.881 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.882 DEBUG:boto:Method: GET 2023-10-11T15:25:33.882 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.882 DEBUG:boto:Data: 2023-10-11T15:25:33.883 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.884 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.884 DEBUG:boto:Params: {} 2023-10-11T15:25:33.885 DEBUG:boto:Token: None 2023-10-11T15:25:33.885 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5fa80f4ea6a0162-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.889 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.890 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.891 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.891 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.892 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.892 DEBUG:boto:Method: GET 2023-10-11T15:25:33.892 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.893 DEBUG:boto:Data: 2023-10-11T15:25:33.893 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.894 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.895 DEBUG:boto:Params: {} 2023-10-11T15:25:33.895 DEBUG:boto:Token: None 2023-10-11T15:25:33.896 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d82125316fc73be-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.901 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.902 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.902 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.903 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.903 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:25:33.904 DEBUG:boto:Data: 2023-10-11T15:25:33.904 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.905 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.905 DEBUG:boto:Params: {} 2023-10-11T15:25:33.906 DEBUG:boto:Token: None 2023-10-11T15:25:33.906 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31 2023-10-11T15:25:33.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6fu01oa42zsqlGI51xlh0q7ubeY= 2023-10-11T15:25:33.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6fu01oa42zsqlGI51xlh0q7ubeY=', 'Content-Length': '0'} 2023-10-11T15:25:33.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ac2a16a94efa03-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.910 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.910 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.911 DEBUG:boto:Method: HEAD 2023-10-11T15:25:33.911 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:25:33.912 DEBUG:boto:Data: 2023-10-11T15:25:33.912 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.913 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.913 DEBUG:boto:Params: {} 2023-10-11T15:25:33.914 DEBUG:boto:Token: None 2023-10-11T15:25:33.914 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31 2023-10-11T15:25:33.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6fu01oa42zsqlGI51xlh0q7ubeY= 2023-10-11T15:25:33.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6fu01oa42zsqlGI51xlh0q7ubeY=', 'Content-Length': '0'} 2023-10-11T15:25:33.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3216e8c8374cb4-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:25:33.918 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.919 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.919 DEBUG:boto:Method: GET 2023-10-11T15:25:33.920 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:25:33.920 DEBUG:boto:Data: 2023-10-11T15:25:33.920 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.921 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.922 DEBUG:boto:Params: {} 2023-10-11T15:25:33.922 DEBUG:boto:Token: None 2023-10-11T15:25:33.923 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31 2023-10-11T15:25:33.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z9UE4KRQMD+NFysu2K5eVM2i2J0= 2023-10-11T15:25:33.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z9UE4KRQMD+NFysu2K5eVM2i2J0=', 'Content-Length': '0'} 2023-10-11T15:25:33.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51ddf1723cf0f35-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.926 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:25:33.926 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:25:33.927 DEBUG:boto:Method: GET 2023-10-11T15:25:33.927 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:25:33.928 DEBUG:boto:Data: 2023-10-11T15:25:33.928 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.929 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.930 DEBUG:boto:Params: {} 2023-10-11T15:25:33.930 DEBUG:boto:Token: None 2023-10-11T15:25:33.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj31 2023-10-11T15:25:33.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z9UE4KRQMD+NFysu2K5eVM2i2J0= 2023-10-11T15:25:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z9UE4KRQMD+NFysu2K5eVM2i2J0=', 'Content-Length': '0'} 2023-10-11T15:25:33.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008954e8df25b7c5de-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.934 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.935 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.935 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.936 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.936 DEBUG:boto:Method: GET 2023-10-11T15:25:33.937 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.937 DEBUG:boto:Data: 2023-10-11T15:25:33.938 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.938 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.939 DEBUG:boto:Params: {} 2023-10-11T15:25:33.939 DEBUG:boto:Token: None 2023-10-11T15:25:33.940 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab90bf3429b19e33-006526be6d-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.944 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.945 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:33.945 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:33.946 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.946 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.947 DEBUG:boto:Method: GET 2023-10-11T15:25:33.947 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:25:33.947 DEBUG:boto:Data: 2023-10-11T15:25:33.948 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.949 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.949 DEBUG:boto:Params: {} 2023-10-11T15:25:33.950 DEBUG:boto:Token: None 2023-10-11T15:25:33.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/ 2023-10-11T15:25:33.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU= 2023-10-11T15:25:33.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DIBjSykkSEUCZc9/hjAB4nDJfBU=', 'Content-Length': '0'} 2023-10-11T15:25:33.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077134a84cab75bd4-006526be6d-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.955 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:33.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:25:33.956 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:33.957 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:33.957 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.958 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.958 DEBUG:boto:Method: GET 2023-10-11T15:25:33.959 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.959 DEBUG:boto:Data: 2023-10-11T15:25:33.959 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:33.960 DEBUG:boto:Port: 8001 2023-10-11T15:25:33.961 DEBUG:boto:Params: {} 2023-10-11T15:25:33.961 DEBUG:boto:Token: None 2023-10-11T15:25:33.962 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.962 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L7nnzuj/IEtc3aCsV/fc247ZEac= 2023-10-11T15:25:33.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L7nnzuj/IEtc3aCsV/fc247ZEac=', 'Content-Length': '0'} 2023-10-11T15:25:33.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000519fde83e715387a-006526be6d-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:33.965 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:33.966 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:33.966 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.967 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.967 DEBUG:boto:Method: GET 2023-10-11T15:25:33.968 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.968 DEBUG:boto:Data: 2023-10-11T15:25:33.968 DEBUG:boto:Headers: {} 2023-10-11T15:25:33.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:33.969 DEBUG:boto:Port: 8000 2023-10-11T15:25:33.970 DEBUG:boto:Params: {} 2023-10-11T15:25:33.970 DEBUG:boto:Token: None 2023-10-11T15:25:33.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:33 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:33.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L7nnzuj/IEtc3aCsV/fc247ZEac= 2023-10-11T15:25:33.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L7nnzuj/IEtc3aCsV/fc247ZEac=', 'Content-Length': '0'} 2023-10-11T15:25:33.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512240307ac0b956-006526be6d-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.014 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.015 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.015 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.016 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.016 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.017 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.017 DEBUG:boto:Data: 2023-10-11T15:25:34.017 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.018 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.019 DEBUG:boto:Params: {} 2023-10-11T15:25:34.019 DEBUG:boto:Token: None 2023-10-11T15:25:34.020 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SILElcFKOtzTbZLq0N9ThTWis0U= 2023-10-11T15:25:34.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SILElcFKOtzTbZLq0N9ThTWis0U=', 'Content-Length': '0'} 2023-10-11T15:25:34.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbaf8789d31c3db8-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.024 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.024 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.024 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.025 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.025 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.026 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.026 DEBUG:boto:Data: 2023-10-11T15:25:34.026 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.027 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.028 DEBUG:boto:Params: {} 2023-10-11T15:25:34.028 DEBUG:boto:Token: None 2023-10-11T15:25:34.028 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SILElcFKOtzTbZLq0N9ThTWis0U= 2023-10-11T15:25:34.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SILElcFKOtzTbZLq0N9ThTWis0U=', 'Content-Length': '0'} 2023-10-11T15:25:34.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf96b4d98040805-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:25:34.032 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.033 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.033 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.034 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.034 DEBUG:boto:Method: GET 2023-10-11T15:25:34.035 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.035 DEBUG:boto:Data: 2023-10-11T15:25:34.035 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.036 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.037 DEBUG:boto:Params: {} 2023-10-11T15:25:34.037 DEBUG:boto:Token: None 2023-10-11T15:25:34.038 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.038 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rgqRaZkDlLPY4a60IQcLo9qyViI= 2023-10-11T15:25:34.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rgqRaZkDlLPY4a60IQcLo9qyViI=', 'Content-Length': '0'} 2023-10-11T15:25:34.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8bafaecf49c3c1-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.086 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.087 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.088 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.088 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.088 DEBUG:boto:Method: GET 2023-10-11T15:25:34.089 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.089 DEBUG:boto:Data: 2023-10-11T15:25:34.090 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.091 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.091 DEBUG:boto:Params: {} 2023-10-11T15:25:34.091 DEBUG:boto:Token: None 2023-10-11T15:25:34.092 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:25:34.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rgqRaZkDlLPY4a60IQcLo9qyViI= 2023-10-11T15:25:34.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rgqRaZkDlLPY4a60IQcLo9qyViI=', 'Content-Length': '0'} 2023-10-11T15:25:34.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028b6a3f5bab2b285-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.096 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.096 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.097 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.097 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.098 DEBUG:boto:Method: GET 2023-10-11T15:25:34.098 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.099 DEBUG:boto:Data: 2023-10-11T15:25:34.099 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.100 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.100 DEBUG:boto:Params: {} 2023-10-11T15:25:34.101 DEBUG:boto:Token: None 2023-10-11T15:25:34.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e82737192c915e6-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.105 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.106 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.106 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.107 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.107 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.108 DEBUG:boto:Method: GET 2023-10-11T15:25:34.108 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.109 DEBUG:boto:Data: 2023-10-11T15:25:34.109 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.110 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.110 DEBUG:boto:Params: {} 2023-10-11T15:25:34.111 DEBUG:boto:Token: None 2023-10-11T15:25:34.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2694d2ab516abb-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.117 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.117 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.118 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.118 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.118 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:25:34.119 DEBUG:boto:Data: 2023-10-11T15:25:34.119 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.120 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.120 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.120 DEBUG:boto:Params: {} 2023-10-11T15:25:34.121 DEBUG:boto:Token: None 2023-10-11T15:25:34.121 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33 2023-10-11T15:25:34.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m5Mc9Hkxwx2veE9s5PH3DiFnfeo= 2023-10-11T15:25:34.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m5Mc9Hkxwx2veE9s5PH3DiFnfeo=', 'Content-Length': '0'} 2023-10-11T15:25:34.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b0113e87ba4d43-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.125 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.125 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.126 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.126 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:25:34.127 DEBUG:boto:Data: 2023-10-11T15:25:34.127 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.128 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.128 DEBUG:boto:Params: {} 2023-10-11T15:25:34.129 DEBUG:boto:Token: None 2023-10-11T15:25:34.129 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33 2023-10-11T15:25:34.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m5Mc9Hkxwx2veE9s5PH3DiFnfeo= 2023-10-11T15:25:34.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m5Mc9Hkxwx2veE9s5PH3DiFnfeo=', 'Content-Length': '0'} 2023-10-11T15:25:34.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97f946de7d7fd42-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:25:34.133 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.133 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.134 DEBUG:boto:Method: GET 2023-10-11T15:25:34.134 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:25:34.135 DEBUG:boto:Data: 2023-10-11T15:25:34.135 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.136 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.136 DEBUG:boto:Params: {} 2023-10-11T15:25:34.137 DEBUG:boto:Token: None 2023-10-11T15:25:34.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33 2023-10-11T15:25:34.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weQUsWj9UXlPFWHhJH/wVnlRP9s= 2023-10-11T15:25:34.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weQUsWj9UXlPFWHhJH/wVnlRP9s=', 'Content-Length': '0'} 2023-10-11T15:25:34.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9aaccf21268da2-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.141 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:25:34.142 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:25:34.142 DEBUG:boto:Method: GET 2023-10-11T15:25:34.143 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:25:34.143 DEBUG:boto:Data: 2023-10-11T15:25:34.143 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.144 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.145 DEBUG:boto:Params: {} 2023-10-11T15:25:34.145 DEBUG:boto:Token: None 2023-10-11T15:25:34.146 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj33 2023-10-11T15:25:34.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:weQUsWj9UXlPFWHhJH/wVnlRP9s= 2023-10-11T15:25:34.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:weQUsWj9UXlPFWHhJH/wVnlRP9s=', 'Content-Length': '0'} 2023-10-11T15:25:34.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef392e176922aab-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.190 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.191 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.192 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.192 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.192 DEBUG:boto:Method: GET 2023-10-11T15:25:34.193 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.193 DEBUG:boto:Data: 2023-10-11T15:25:34.194 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.195 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.195 DEBUG:boto:Params: {} 2023-10-11T15:25:34.195 DEBUG:boto:Token: None 2023-10-11T15:25:34.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a0d7ac62dc59b8-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.200 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.201 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.201 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.202 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.202 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.203 DEBUG:boto:Method: GET 2023-10-11T15:25:34.203 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:25:34.203 DEBUG:boto:Data: 2023-10-11T15:25:34.204 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.205 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.205 DEBUG:boto:Params: {} 2023-10-11T15:25:34.206 DEBUG:boto:Token: None 2023-10-11T15:25:34.206 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086e5653f6120d71e-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.212 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:25:34.213 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.213 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.214 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.214 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.215 DEBUG:boto:Method: GET 2023-10-11T15:25:34.215 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.215 DEBUG:boto:Data: 2023-10-11T15:25:34.216 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.217 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.217 DEBUG:boto:Params: {} 2023-10-11T15:25:34.218 DEBUG:boto:Token: None 2023-10-11T15:25:34.218 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.219 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw= 2023-10-11T15:25:34.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw=', 'Content-Length': '0'} 2023-10-11T15:25:34.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6f75f8d799e623-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.222 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.223 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.223 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.223 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.224 DEBUG:boto:Method: GET 2023-10-11T15:25:34.224 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.225 DEBUG:boto:Data: 2023-10-11T15:25:34.225 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.225 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.226 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.226 DEBUG:boto:Params: {} 2023-10-11T15:25:34.227 DEBUG:boto:Token: None 2023-10-11T15:25:34.227 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw= 2023-10-11T15:25:34.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw=', 'Content-Length': '0'} 2023-10-11T15:25:34.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a488a5707c2b19fa-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.231 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.232 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.232 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.232 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.233 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.233 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.233 DEBUG:boto:Data: 2023-10-11T15:25:34.234 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.235 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.235 DEBUG:boto:Params: {} 2023-10-11T15:25:34.236 DEBUG:boto:Token: None 2023-10-11T15:25:34.236 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.236 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G9BgLJMvsukQMuHLi0Ag7oOIt5E= 2023-10-11T15:25:34.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G9BgLJMvsukQMuHLi0Ag7oOIt5E=', 'Content-Length': '0'} 2023-10-11T15:25:34.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27d8577b398757f-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.240 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.240 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.240 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.241 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.241 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.242 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.242 DEBUG:boto:Data: 2023-10-11T15:25:34.243 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.243 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.244 DEBUG:boto:Params: {} 2023-10-11T15:25:34.244 DEBUG:boto:Token: None 2023-10-11T15:25:34.245 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G9BgLJMvsukQMuHLi0Ag7oOIt5E= 2023-10-11T15:25:34.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G9BgLJMvsukQMuHLi0Ag7oOIt5E=', 'Content-Length': '0'} 2023-10-11T15:25:34.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ffe1751458c662-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:25:34.249 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.249 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.250 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.250 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.250 DEBUG:boto:Method: GET 2023-10-11T15:25:34.251 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.251 DEBUG:boto:Data: 2023-10-11T15:25:34.252 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.252 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.253 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.253 DEBUG:boto:Params: {} 2023-10-11T15:25:34.253 DEBUG:boto:Token: None 2023-10-11T15:25:34.254 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw= 2023-10-11T15:25:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw=', 'Content-Length': '0'} 2023-10-11T15:25:34.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0b2734ba4cf7f4-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.299 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.300 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.300 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.300 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.301 DEBUG:boto:Method: GET 2023-10-11T15:25:34.301 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.302 DEBUG:boto:Data: 2023-10-11T15:25:34.302 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.303 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.303 DEBUG:boto:Params: {} 2023-10-11T15:25:34.304 DEBUG:boto:Token: None 2023-10-11T15:25:34.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:25:34.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw= 2023-10-11T15:25:34.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IJsPVPkeiFgvg/mDbtSN+an1zDw=', 'Content-Length': '0'} 2023-10-11T15:25:34.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9511438aecc20d1-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.309 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.309 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.309 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.310 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.310 DEBUG:boto:Method: GET 2023-10-11T15:25:34.311 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.311 DEBUG:boto:Data: 2023-10-11T15:25:34.312 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.312 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.313 DEBUG:boto:Params: {} 2023-10-11T15:25:34.313 DEBUG:boto:Token: None 2023-10-11T15:25:34.314 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c2e29a4a4a99261-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.318 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.319 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.319 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.320 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.320 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.321 DEBUG:boto:Method: GET 2023-10-11T15:25:34.321 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.322 DEBUG:boto:Data: 2023-10-11T15:25:34.322 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.323 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.323 DEBUG:boto:Params: {} 2023-10-11T15:25:34.324 DEBUG:boto:Token: None 2023-10-11T15:25:34.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3d6fe70607f786-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.330 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.331 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.331 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.331 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.332 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:25:34.332 DEBUG:boto:Data: 2023-10-11T15:25:34.333 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.334 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.334 DEBUG:boto:Params: {} 2023-10-11T15:25:34.334 DEBUG:boto:Token: None 2023-10-11T15:25:34.335 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34 2023-10-11T15:25:34.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FcylRZbCIj327Spege+pNzXniKI= 2023-10-11T15:25:34.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FcylRZbCIj327Spege+pNzXniKI=', 'Content-Length': '0'} 2023-10-11T15:25:34.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2aad18a4ebeeec-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.338 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.339 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.339 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.340 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:25:34.340 DEBUG:boto:Data: 2023-10-11T15:25:34.341 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.341 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.342 DEBUG:boto:Params: {} 2023-10-11T15:25:34.342 DEBUG:boto:Token: None 2023-10-11T15:25:34.343 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34 2023-10-11T15:25:34.343 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FcylRZbCIj327Spege+pNzXniKI= 2023-10-11T15:25:34.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FcylRZbCIj327Spege+pNzXniKI=', 'Content-Length': '0'} 2023-10-11T15:25:34.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2220e350b2091f0-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:25:34.347 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.347 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.348 DEBUG:boto:Method: GET 2023-10-11T15:25:34.348 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:25:34.349 DEBUG:boto:Data: 2023-10-11T15:25:34.349 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.350 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.350 DEBUG:boto:Params: {} 2023-10-11T15:25:34.351 DEBUG:boto:Token: None 2023-10-11T15:25:34.351 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34 2023-10-11T15:25:34.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6l4lrNn1YPlCF2ro91gi1YVT2qg= 2023-10-11T15:25:34.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6l4lrNn1YPlCF2ro91gi1YVT2qg=', 'Content-Length': '0'} 2023-10-11T15:25:34.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60040e63d4b7e01-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.355 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:25:34.355 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:25:34.356 DEBUG:boto:Method: GET 2023-10-11T15:25:34.356 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:25:34.357 DEBUG:boto:Data: 2023-10-11T15:25:34.357 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.358 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.358 DEBUG:boto:Params: {} 2023-10-11T15:25:34.359 DEBUG:boto:Token: None 2023-10-11T15:25:34.359 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj34 2023-10-11T15:25:34.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6l4lrNn1YPlCF2ro91gi1YVT2qg= 2023-10-11T15:25:34.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6l4lrNn1YPlCF2ro91gi1YVT2qg=', 'Content-Length': '0'} 2023-10-11T15:25:34.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4d358cccb8b06de-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.363 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.363 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.364 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.364 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.365 DEBUG:boto:Method: GET 2023-10-11T15:25:34.365 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.366 DEBUG:boto:Data: 2023-10-11T15:25:34.366 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.367 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.367 DEBUG:boto:Params: {} 2023-10-11T15:25:34.368 DEBUG:boto:Token: None 2023-10-11T15:25:34.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aa6507daf45d689-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.373 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.373 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.374 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.374 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.375 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.375 DEBUG:boto:Method: GET 2023-10-11T15:25:34.376 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:25:34.376 DEBUG:boto:Data: 2023-10-11T15:25:34.376 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.377 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.378 DEBUG:boto:Params: {} 2023-10-11T15:25:34.378 DEBUG:boto:Token: None 2023-10-11T15:25:34.379 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000539d89d1c913b64-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.384 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:25:34.385 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.386 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.386 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.387 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.387 DEBUG:boto:Method: GET 2023-10-11T15:25:34.387 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.388 DEBUG:boto:Data: 2023-10-11T15:25:34.388 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.389 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.389 DEBUG:boto:Params: {} 2023-10-11T15:25:34.390 DEBUG:boto:Token: None 2023-10-11T15:25:34.390 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g= 2023-10-11T15:25:34.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g=', 'Content-Length': '0'} 2023-10-11T15:25:34.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d1e78c7e29d90eb-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.394 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.395 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.395 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.396 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.396 DEBUG:boto:Method: GET 2023-10-11T15:25:34.397 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.397 DEBUG:boto:Data: 2023-10-11T15:25:34.398 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.398 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.399 DEBUG:boto:Params: {} 2023-10-11T15:25:34.399 DEBUG:boto:Token: None 2023-10-11T15:25:34.400 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g= 2023-10-11T15:25:34.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g=', 'Content-Length': '0'} 2023-10-11T15:25:34.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8c0c2d28b6dd9a-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.404 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.404 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.405 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.405 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.405 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.406 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.406 DEBUG:boto:Data: 2023-10-11T15:25:34.407 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.407 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.408 DEBUG:boto:Params: {} 2023-10-11T15:25:34.408 DEBUG:boto:Token: None 2023-10-11T15:25:34.409 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:16zAnnZBayCPAVkyBzDdvsKebn0= 2023-10-11T15:25:34.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:16zAnnZBayCPAVkyBzDdvsKebn0=', 'Content-Length': '0'} 2023-10-11T15:25:34.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fa81363488446a-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.412 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.413 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.413 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.414 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.414 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.415 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.415 DEBUG:boto:Data: 2023-10-11T15:25:34.416 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.416 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.417 DEBUG:boto:Params: {} 2023-10-11T15:25:34.417 DEBUG:boto:Token: None 2023-10-11T15:25:34.418 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:16zAnnZBayCPAVkyBzDdvsKebn0= 2023-10-11T15:25:34.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:16zAnnZBayCPAVkyBzDdvsKebn0=', 'Content-Length': '0'} 2023-10-11T15:25:34.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160d0f3e8e93bbd5-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:25:34.422 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.423 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.424 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.425 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.425 DEBUG:boto:Method: GET 2023-10-11T15:25:34.426 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.427 DEBUG:boto:Data: 2023-10-11T15:25:34.428 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.428 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.428 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.429 DEBUG:boto:Params: {} 2023-10-11T15:25:34.429 DEBUG:boto:Token: None 2023-10-11T15:25:34.430 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g= 2023-10-11T15:25:34.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g=', 'Content-Length': '0'} 2023-10-11T15:25:34.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098492f63f86a46a7-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.434 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.434 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.435 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.435 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.436 DEBUG:boto:Method: GET 2023-10-11T15:25:34.436 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.437 DEBUG:boto:Data: 2023-10-11T15:25:34.437 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.438 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.439 DEBUG:boto:Params: {} 2023-10-11T15:25:34.439 DEBUG:boto:Token: None 2023-10-11T15:25:34.440 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:25:34.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g= 2023-10-11T15:25:34.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zTbrcw2F8fXc+MLJD5VVdByK85g=', 'Content-Length': '0'} 2023-10-11T15:25:34.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd5eea424f7a4da-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.486 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.488 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.488 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.489 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.489 DEBUG:boto:Method: GET 2023-10-11T15:25:34.490 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.491 DEBUG:boto:Data: 2023-10-11T15:25:34.491 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.492 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.493 DEBUG:boto:Params: {} 2023-10-11T15:25:34.493 DEBUG:boto:Token: None 2023-10-11T15:25:34.494 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.494 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e93045c71bbef6e-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.498 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.499 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.499 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.501 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.502 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.502 DEBUG:boto:Method: GET 2023-10-11T15:25:34.502 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.503 DEBUG:boto:Data: 2023-10-11T15:25:34.503 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.504 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.505 DEBUG:boto:Params: {} 2023-10-11T15:25:34.505 DEBUG:boto:Token: None 2023-10-11T15:25:34.506 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a014753abde983cc-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.511 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.512 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.513 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.513 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.514 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:25:34.514 DEBUG:boto:Data: 2023-10-11T15:25:34.515 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.517 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.517 DEBUG:boto:Params: {} 2023-10-11T15:25:34.518 DEBUG:boto:Token: None 2023-10-11T15:25:34.519 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38 2023-10-11T15:25:34.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RsPWeakpXHI9L+1DCMJFamm5PbQ= 2023-10-11T15:25:34.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RsPWeakpXHI9L+1DCMJFamm5PbQ=', 'Content-Length': '0'} 2023-10-11T15:25:34.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdab4893f9a98685-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.523 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.524 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.524 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.525 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:25:34.525 DEBUG:boto:Data: 2023-10-11T15:25:34.525 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.526 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.527 DEBUG:boto:Params: {} 2023-10-11T15:25:34.527 DEBUG:boto:Token: None 2023-10-11T15:25:34.528 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38 2023-10-11T15:25:34.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RsPWeakpXHI9L+1DCMJFamm5PbQ= 2023-10-11T15:25:34.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RsPWeakpXHI9L+1DCMJFamm5PbQ=', 'Content-Length': '0'} 2023-10-11T15:25:34.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785dcaf7c9fb5765-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:25:34.531 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.532 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.532 DEBUG:boto:Method: GET 2023-10-11T15:25:34.533 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:25:34.534 DEBUG:boto:Data: 2023-10-11T15:25:34.535 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.536 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.537 DEBUG:boto:Params: {} 2023-10-11T15:25:34.538 DEBUG:boto:Token: None 2023-10-11T15:25:34.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38 2023-10-11T15:25:34.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3MvRkmno/8LUWUxLwomRTun/w00= 2023-10-11T15:25:34.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3MvRkmno/8LUWUxLwomRTun/w00=', 'Content-Length': '0'} 2023-10-11T15:25:34.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000188796577920969a-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.590 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:25:34.591 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:25:34.591 DEBUG:boto:Method: GET 2023-10-11T15:25:34.592 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:25:34.592 DEBUG:boto:Data: 2023-10-11T15:25:34.593 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.594 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.594 DEBUG:boto:Params: {} 2023-10-11T15:25:34.595 DEBUG:boto:Token: None 2023-10-11T15:25:34.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj38 2023-10-11T15:25:34.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3MvRkmno/8LUWUxLwomRTun/w00= 2023-10-11T15:25:34.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3MvRkmno/8LUWUxLwomRTun/w00=', 'Content-Length': '0'} 2023-10-11T15:25:34.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2a723886b94394-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.599 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.600 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.600 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.601 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.602 DEBUG:boto:Method: GET 2023-10-11T15:25:34.602 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.603 DEBUG:boto:Data: 2023-10-11T15:25:34.604 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.605 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.606 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.606 DEBUG:boto:Params: {} 2023-10-11T15:25:34.607 DEBUG:boto:Token: None 2023-10-11T15:25:34.608 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d33bbc1aa331ad-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.614 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.615 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.615 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.616 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.616 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.617 DEBUG:boto:Method: GET 2023-10-11T15:25:34.617 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:25:34.617 DEBUG:boto:Data: 2023-10-11T15:25:34.618 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.619 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.619 DEBUG:boto:Params: {} 2023-10-11T15:25:34.620 DEBUG:boto:Token: None 2023-10-11T15:25:34.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000527b4b68ef3be28d-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.666 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:25:34.668 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.668 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.669 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.669 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.670 DEBUG:boto:Method: GET 2023-10-11T15:25:34.670 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.670 DEBUG:boto:Data: 2023-10-11T15:25:34.671 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.672 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.672 DEBUG:boto:Params: {} 2023-10-11T15:25:34.673 DEBUG:boto:Token: None 2023-10-11T15:25:34.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw= 2023-10-11T15:25:34.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw=', 'Content-Length': '0'} 2023-10-11T15:25:34.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd31280c17cd389-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.677 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.677 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.678 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.678 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.679 DEBUG:boto:Method: GET 2023-10-11T15:25:34.679 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.680 DEBUG:boto:Data: 2023-10-11T15:25:34.680 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.681 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.681 DEBUG:boto:Params: {} 2023-10-11T15:25:34.682 DEBUG:boto:Token: None 2023-10-11T15:25:34.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw= 2023-10-11T15:25:34.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw=', 'Content-Length': '0'} 2023-10-11T15:25:34.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008688645e2ecb72b5-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.686 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.687 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.687 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.687 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.688 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.688 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.689 DEBUG:boto:Data: 2023-10-11T15:25:34.689 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.690 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.690 DEBUG:boto:Params: {} 2023-10-11T15:25:34.691 DEBUG:boto:Token: None 2023-10-11T15:25:34.691 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTjKDyH5o8TuJvEWU3sexOu7mLw= 2023-10-11T15:25:34.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTjKDyH5o8TuJvEWU3sexOu7mLw=', 'Content-Length': '0'} 2023-10-11T15:25:34.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034d092ebff79f5b0-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.694 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.695 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.695 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.696 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.696 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.696 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.697 DEBUG:boto:Data: 2023-10-11T15:25:34.697 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.698 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.698 DEBUG:boto:Params: {} 2023-10-11T15:25:34.699 DEBUG:boto:Token: None 2023-10-11T15:25:34.699 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTjKDyH5o8TuJvEWU3sexOu7mLw= 2023-10-11T15:25:34.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTjKDyH5o8TuJvEWU3sexOu7mLw=', 'Content-Length': '0'} 2023-10-11T15:25:34.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69159e5a7087d89-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:25:34.703 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.704 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.704 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.705 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.705 DEBUG:boto:Method: GET 2023-10-11T15:25:34.706 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.706 DEBUG:boto:Data: 2023-10-11T15:25:34.706 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.707 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.709 DEBUG:boto:Params: {} 2023-10-11T15:25:34.710 DEBUG:boto:Token: None 2023-10-11T15:25:34.710 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw= 2023-10-11T15:25:34.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw=', 'Content-Length': '0'} 2023-10-11T15:25:34.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f39edf96b85ab2ad-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.714 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.714 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.715 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.715 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.715 DEBUG:boto:Method: GET 2023-10-11T15:25:34.716 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.716 DEBUG:boto:Data: 2023-10-11T15:25:34.717 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.718 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.718 DEBUG:boto:Params: {} 2023-10-11T15:25:34.719 DEBUG:boto:Token: None 2023-10-11T15:25:34.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:25:34.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw= 2023-10-11T15:25:34.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I9UtzeMf/NRRm+Ha7lM0ZlOTQFw=', 'Content-Length': '0'} 2023-10-11T15:25:34.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40d0b81b8aca60c-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.723 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.723 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.724 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.724 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.724 DEBUG:boto:Method: GET 2023-10-11T15:25:34.725 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.725 DEBUG:boto:Data: 2023-10-11T15:25:34.726 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.726 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.727 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.727 DEBUG:boto:Params: {} 2023-10-11T15:25:34.727 DEBUG:boto:Token: None 2023-10-11T15:25:34.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.728 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ae5def267911b9-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.732 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.733 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.733 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.734 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.734 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.734 DEBUG:boto:Method: GET 2023-10-11T15:25:34.735 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.735 DEBUG:boto:Data: 2023-10-11T15:25:34.736 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.736 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.737 DEBUG:boto:Params: {} 2023-10-11T15:25:34.737 DEBUG:boto:Token: None 2023-10-11T15:25:34.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.738 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008326297a7045059-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.790 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.791 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.792 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.792 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.793 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:25:34.793 DEBUG:boto:Data: 2023-10-11T15:25:34.793 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.794 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.795 DEBUG:boto:Params: {} 2023-10-11T15:25:34.795 DEBUG:boto:Token: None 2023-10-11T15:25:34.796 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44 2023-10-11T15:25:34.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8JdxOHaPIsC5bpRbyR3vfYOxtoo= 2023-10-11T15:25:34.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8JdxOHaPIsC5bpRbyR3vfYOxtoo=', 'Content-Length': '0'} 2023-10-11T15:25:34.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007060f07d9717784a-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.800 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.800 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.800 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.801 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:25:34.801 DEBUG:boto:Data: 2023-10-11T15:25:34.802 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.802 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.802 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.803 DEBUG:boto:Params: {} 2023-10-11T15:25:34.803 DEBUG:boto:Token: None 2023-10-11T15:25:34.804 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44 2023-10-11T15:25:34.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8JdxOHaPIsC5bpRbyR3vfYOxtoo= 2023-10-11T15:25:34.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8JdxOHaPIsC5bpRbyR3vfYOxtoo=', 'Content-Length': '0'} 2023-10-11T15:25:34.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c7fa470ee0eff9-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:25:34.808 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.809 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.809 DEBUG:boto:Method: GET 2023-10-11T15:25:34.810 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:25:34.810 DEBUG:boto:Data: 2023-10-11T15:25:34.810 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.811 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.811 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.812 DEBUG:boto:Params: {} 2023-10-11T15:25:34.812 DEBUG:boto:Token: None 2023-10-11T15:25:34.813 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44 2023-10-11T15:25:34.813 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zt5bb9L+4GmgmI0Ckv0bBgtBV8U= 2023-10-11T15:25:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zt5bb9L+4GmgmI0Ckv0bBgtBV8U=', 'Content-Length': '0'} 2023-10-11T15:25:34.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007978456b284bd86a-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.817 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:25:34.817 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:25:34.818 DEBUG:boto:Method: GET 2023-10-11T15:25:34.818 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:25:34.819 DEBUG:boto:Data: 2023-10-11T15:25:34.819 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.820 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.821 DEBUG:boto:Params: {} 2023-10-11T15:25:34.821 DEBUG:boto:Token: None 2023-10-11T15:25:34.822 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj44 2023-10-11T15:25:34.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zt5bb9L+4GmgmI0Ckv0bBgtBV8U= 2023-10-11T15:25:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zt5bb9L+4GmgmI0Ckv0bBgtBV8U=', 'Content-Length': '0'} 2023-10-11T15:25:34.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed65784d8c232fa-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.825 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.826 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.826 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.827 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.827 DEBUG:boto:Method: GET 2023-10-11T15:25:34.828 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.829 DEBUG:boto:Data: 2023-10-11T15:25:34.829 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.830 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.830 DEBUG:boto:Params: {} 2023-10-11T15:25:34.831 DEBUG:boto:Token: None 2023-10-11T15:25:34.831 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004568f61bc648843d-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.835 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.836 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.837 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.837 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.837 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.838 DEBUG:boto:Method: GET 2023-10-11T15:25:34.839 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:25:34.839 DEBUG:boto:Data: 2023-10-11T15:25:34.840 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.840 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.841 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.841 DEBUG:boto:Params: {} 2023-10-11T15:25:34.842 DEBUG:boto:Token: None 2023-10-11T15:25:34.842 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052509b3e0abef4f3-006526be6e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.847 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:25:34.849 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.849 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.850 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.850 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.851 DEBUG:boto:Method: GET 2023-10-11T15:25:34.851 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.851 DEBUG:boto:Data: 2023-10-11T15:25:34.852 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.852 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.853 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.853 DEBUG:boto:Params: {} 2023-10-11T15:25:34.854 DEBUG:boto:Token: None 2023-10-11T15:25:34.854 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8= 2023-10-11T15:25:34.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8=', 'Content-Length': '0'} 2023-10-11T15:25:34.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ea6af91482bd51-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.898 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.899 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.899 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.900 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.900 DEBUG:boto:Method: GET 2023-10-11T15:25:34.901 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.901 DEBUG:boto:Data: 2023-10-11T15:25:34.902 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.903 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.903 DEBUG:boto:Params: {} 2023-10-11T15:25:34.903 DEBUG:boto:Token: None 2023-10-11T15:25:34.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8= 2023-10-11T15:25:34.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8=', 'Content-Length': '0'} 2023-10-11T15:25:34.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec955fd2d3664aa-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.908 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.908 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.909 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.909 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.910 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.910 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.911 DEBUG:boto:Data: 2023-10-11T15:25:34.911 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.912 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.912 DEBUG:boto:Params: {} 2023-10-11T15:25:34.913 DEBUG:boto:Token: None 2023-10-11T15:25:34.914 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYEQKc81N7PLSwctvpy7VjTofgY= 2023-10-11T15:25:34.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYEQKc81N7PLSwctvpy7VjTofgY=', 'Content-Length': '0'} 2023-10-11T15:25:34.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784ac9590aa8825b-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.918 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.918 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.919 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.919 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.920 DEBUG:boto:Method: HEAD 2023-10-11T15:25:34.920 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.920 DEBUG:boto:Data: 2023-10-11T15:25:34.921 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.921 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.922 DEBUG:boto:Params: {} 2023-10-11T15:25:34.922 DEBUG:boto:Token: None 2023-10-11T15:25:34.923 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nYEQKc81N7PLSwctvpy7VjTofgY= 2023-10-11T15:25:34.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYEQKc81N7PLSwctvpy7VjTofgY=', 'Content-Length': '0'} 2023-10-11T15:25:34.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fcd38fe157ed277-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:25:34.927 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.927 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.928 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.928 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.929 DEBUG:boto:Method: GET 2023-10-11T15:25:34.929 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.929 DEBUG:boto:Data: 2023-10-11T15:25:34.930 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.931 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.931 DEBUG:boto:Params: {} 2023-10-11T15:25:34.932 DEBUG:boto:Token: None 2023-10-11T15:25:34.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8= 2023-10-11T15:25:34.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8=', 'Content-Length': '0'} 2023-10-11T15:25:34.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167197151d25c3bd-006526be6e-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.935 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:34.936 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:34.936 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.937 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.937 DEBUG:boto:Method: GET 2023-10-11T15:25:34.938 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.938 DEBUG:boto:Data: 2023-10-11T15:25:34.939 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:34.940 DEBUG:boto:Port: 8000 2023-10-11T15:25:34.940 DEBUG:boto:Params: {} 2023-10-11T15:25:34.940 DEBUG:boto:Token: None 2023-10-11T15:25:34.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:25:34.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8= 2023-10-11T15:25:34.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KkDOJOjcxnzA8g8HZfFViySzNy8=', 'Content-Length': '0'} 2023-10-11T15:25:34.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e8cfc90fa2aec7-006526be6e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.986 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.987 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.988 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:34.988 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:34.988 DEBUG:boto:Method: GET 2023-10-11T15:25:34.989 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:25:34.989 DEBUG:boto:Data: 2023-10-11T15:25:34.989 DEBUG:boto:Headers: {} 2023-10-11T15:25:34.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:34.990 DEBUG:boto:Port: 8001 2023-10-11T15:25:34.991 DEBUG:boto:Params: {} 2023-10-11T15:25:34.991 DEBUG:boto:Token: None 2023-10-11T15:25:34.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:34 GMT /dghqvd-59/ 2023-10-11T15:25:34.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0= 2023-10-11T15:25:34.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cXjMNU10SV3Qh2aKwnG/2fPX/j0=', 'Content-Length': '0'} 2023-10-11T15:25:34.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e2d0b5e65e802a-006526be6e-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:34.996 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:34.996 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:34.997 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:34.998 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:34.998 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:34.999 DEBUG:boto:Method: GET 2023-10-11T15:25:35.000 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.000 DEBUG:boto:Data: 2023-10-11T15:25:35.001 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.002 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.002 DEBUG:boto:Params: {} 2023-10-11T15:25:35.003 DEBUG:boto:Token: None 2023-10-11T15:25:35.003 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dd9f791aac523c-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.009 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.010 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:35.010 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:35.011 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.011 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:25:35.012 DEBUG:boto:Data: 2023-10-11T15:25:35.012 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.013 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.013 DEBUG:boto:Params: {} 2023-10-11T15:25:35.014 DEBUG:boto:Token: None 2023-10-11T15:25:35.014 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj46 2023-10-11T15:25:35.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Vsy7KZLs395rFNwckHYPJDRTXA= 2023-10-11T15:25:35.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Vsy7KZLs395rFNwckHYPJDRTXA=', 'Content-Length': '0'} 2023-10-11T15:25:35.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ba38ea40f4d681-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.018 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:35.018 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:35.019 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.019 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:25:35.020 DEBUG:boto:Data: 2023-10-11T15:25:35.020 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.021 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.021 DEBUG:boto:Params: {} 2023-10-11T15:25:35.022 DEBUG:boto:Token: None 2023-10-11T15:25:35.022 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj46 2023-10-11T15:25:35.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Vsy7KZLs395rFNwckHYPJDRTXA= 2023-10-11T15:25:35.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Vsy7KZLs395rFNwckHYPJDRTXA=', 'Content-Length': '0'} 2023-10-11T15:25:35.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca342ca8a5f2821a-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:25:35.026 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:35.026 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:35.027 DEBUG:boto:Method: GET 2023-10-11T15:25:35.027 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:25:35.028 DEBUG:boto:Data: 2023-10-11T15:25:35.028 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.029 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.029 DEBUG:boto:Params: {} 2023-10-11T15:25:35.030 DEBUG:boto:Token: None 2023-10-11T15:25:35.030 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj46 2023-10-11T15:25:35.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/rymSl9IwepZWpFnEzmk/kd8sCM= 2023-10-11T15:25:35.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/rymSl9IwepZWpFnEzmk/kd8sCM=', 'Content-Length': '0'} 2023-10-11T15:25:35.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008497ef7113745d36-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.034 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:25:35.034 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:25:35.035 DEBUG:boto:Method: GET 2023-10-11T15:25:35.035 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:25:35.036 DEBUG:boto:Data: 2023-10-11T15:25:35.036 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.037 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.037 DEBUG:boto:Params: {} 2023-10-11T15:25:35.038 DEBUG:boto:Token: None 2023-10-11T15:25:35.038 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj46 2023-10-11T15:25:35.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/rymSl9IwepZWpFnEzmk/kd8sCM= 2023-10-11T15:25:35.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/rymSl9IwepZWpFnEzmk/kd8sCM=', 'Content-Length': '0'} 2023-10-11T15:25:35.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085185cea70992f59-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.042 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.043 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.043 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.043 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.044 DEBUG:boto:Method: GET 2023-10-11T15:25:35.044 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.045 DEBUG:boto:Data: 2023-10-11T15:25:35.045 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.046 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.046 DEBUG:boto:Params: {} 2023-10-11T15:25:35.047 DEBUG:boto:Token: None 2023-10-11T15:25:35.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b299e1b05efb52e8-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.051 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.052 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.052 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.053 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.053 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.054 DEBUG:boto:Method: GET 2023-10-11T15:25:35.054 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:25:35.054 DEBUG:boto:Data: 2023-10-11T15:25:35.055 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.056 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.056 DEBUG:boto:Params: {} 2023-10-11T15:25:35.056 DEBUG:boto:Token: None 2023-10-11T15:25:35.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f74379a900cc5b8-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.062 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:25:35.063 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.064 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.064 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.065 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.065 DEBUG:boto:Method: GET 2023-10-11T15:25:35.065 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.066 DEBUG:boto:Data: 2023-10-11T15:25:35.066 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.067 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.067 DEBUG:boto:Params: {} 2023-10-11T15:25:35.068 DEBUG:boto:Token: None 2023-10-11T15:25:35.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I= 2023-10-11T15:25:35.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I=', 'Content-Length': '0'} 2023-10-11T15:25:35.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0a2e859cc21c39-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.118 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.119 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.119 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.120 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.120 DEBUG:boto:Method: GET 2023-10-11T15:25:35.121 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.121 DEBUG:boto:Data: 2023-10-11T15:25:35.121 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.122 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.123 DEBUG:boto:Params: {} 2023-10-11T15:25:35.124 DEBUG:boto:Token: None 2023-10-11T15:25:35.125 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.125 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I= 2023-10-11T15:25:35.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I=', 'Content-Length': '0'} 2023-10-11T15:25:35.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa1439838349a628-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.129 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.129 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.130 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.130 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.131 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.131 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.132 DEBUG:boto:Data: 2023-10-11T15:25:35.132 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.133 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.133 DEBUG:boto:Params: {} 2023-10-11T15:25:35.134 DEBUG:boto:Token: None 2023-10-11T15:25:35.134 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P4SseEs+O4aMiAao5CROb2otahU= 2023-10-11T15:25:35.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P4SseEs+O4aMiAao5CROb2otahU=', 'Content-Length': '0'} 2023-10-11T15:25:35.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cbd424c87a3a4ec-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.138 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.138 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.139 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.139 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.140 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.140 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.140 DEBUG:boto:Data: 2023-10-11T15:25:35.141 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.142 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.142 DEBUG:boto:Params: {} 2023-10-11T15:25:35.143 DEBUG:boto:Token: None 2023-10-11T15:25:35.143 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:P4SseEs+O4aMiAao5CROb2otahU= 2023-10-11T15:25:35.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:P4SseEs+O4aMiAao5CROb2otahU=', 'Content-Length': '0'} 2023-10-11T15:25:35.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f478f8609236d3e-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:25:35.147 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.148 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.148 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.149 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.149 DEBUG:boto:Method: GET 2023-10-11T15:25:35.149 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.150 DEBUG:boto:Data: 2023-10-11T15:25:35.150 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.151 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.151 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.152 DEBUG:boto:Params: {} 2023-10-11T15:25:35.152 DEBUG:boto:Token: None 2023-10-11T15:25:35.153 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.153 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I= 2023-10-11T15:25:35.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I=', 'Content-Length': '0'} 2023-10-11T15:25:35.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568c739963c2563e-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.156 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.157 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.157 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.158 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.158 DEBUG:boto:Method: GET 2023-10-11T15:25:35.158 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.159 DEBUG:boto:Data: 2023-10-11T15:25:35.159 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.160 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.160 DEBUG:boto:Params: {} 2023-10-11T15:25:35.161 DEBUG:boto:Token: None 2023-10-11T15:25:35.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:25:35.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I= 2023-10-11T15:25:35.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cqDKFLFXisDOpgSfBJ5uuwi+y7I=', 'Content-Length': '0'} 2023-10-11T15:25:35.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac1bf0f00c6967d-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.210 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.211 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.211 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.212 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.212 DEBUG:boto:Method: GET 2023-10-11T15:25:35.213 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.213 DEBUG:boto:Data: 2023-10-11T15:25:35.214 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.214 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.215 DEBUG:boto:Params: {} 2023-10-11T15:25:35.215 DEBUG:boto:Token: None 2023-10-11T15:25:35.216 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007199327fbb21d5c4-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.220 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.221 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.221 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.222 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.222 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.223 DEBUG:boto:Method: GET 2023-10-11T15:25:35.223 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.224 DEBUG:boto:Data: 2023-10-11T15:25:35.224 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.225 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.225 DEBUG:boto:Params: {} 2023-10-11T15:25:35.226 DEBUG:boto:Token: None 2023-10-11T15:25:35.226 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.227 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335d20a2809f4e74-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.232 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.233 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.233 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.234 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.234 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:25:35.235 DEBUG:boto:Data: 2023-10-11T15:25:35.235 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.236 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.236 DEBUG:boto:Params: {} 2023-10-11T15:25:35.237 DEBUG:boto:Token: None 2023-10-11T15:25:35.237 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51 2023-10-11T15:25:35.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NuoQ2g5mbIt9iLNUl2e+EKvyPIk= 2023-10-11T15:25:35.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NuoQ2g5mbIt9iLNUl2e+EKvyPIk=', 'Content-Length': '0'} 2023-10-11T15:25:35.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138a17ed5029c8d8-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.241 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.241 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.242 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.242 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:25:35.242 DEBUG:boto:Data: 2023-10-11T15:25:35.243 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.243 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.244 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.244 DEBUG:boto:Params: {} 2023-10-11T15:25:35.245 DEBUG:boto:Token: None 2023-10-11T15:25:35.245 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51 2023-10-11T15:25:35.245 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NuoQ2g5mbIt9iLNUl2e+EKvyPIk= 2023-10-11T15:25:35.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NuoQ2g5mbIt9iLNUl2e+EKvyPIk=', 'Content-Length': '0'} 2023-10-11T15:25:35.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb314fbcd81190d5-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:25:35.249 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.249 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.250 DEBUG:boto:Method: GET 2023-10-11T15:25:35.250 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:25:35.251 DEBUG:boto:Data: 2023-10-11T15:25:35.251 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.252 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.252 DEBUG:boto:Params: {} 2023-10-11T15:25:35.253 DEBUG:boto:Token: None 2023-10-11T15:25:35.253 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51 2023-10-11T15:25:35.254 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mv75bWUn8vYLhlMv/74UB7PEWNs= 2023-10-11T15:25:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mv75bWUn8vYLhlMv/74UB7PEWNs=', 'Content-Length': '0'} 2023-10-11T15:25:35.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efbdc461685c70bb-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.257 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:25:35.257 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:25:35.258 DEBUG:boto:Method: GET 2023-10-11T15:25:35.258 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:25:35.259 DEBUG:boto:Data: 2023-10-11T15:25:35.259 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.260 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.260 DEBUG:boto:Params: {} 2023-10-11T15:25:35.261 DEBUG:boto:Token: None 2023-10-11T15:25:35.261 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj51 2023-10-11T15:25:35.262 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mv75bWUn8vYLhlMv/74UB7PEWNs= 2023-10-11T15:25:35.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mv75bWUn8vYLhlMv/74UB7PEWNs=', 'Content-Length': '0'} 2023-10-11T15:25:35.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729fa4c45377ec1e-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.265 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.265 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.266 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.266 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.267 DEBUG:boto:Method: GET 2023-10-11T15:25:35.267 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.268 DEBUG:boto:Data: 2023-10-11T15:25:35.268 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.269 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.269 DEBUG:boto:Params: {} 2023-10-11T15:25:35.270 DEBUG:boto:Token: None 2023-10-11T15:25:35.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000af0b1a3547a50a2-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.274 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.275 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.275 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.276 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.276 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.277 DEBUG:boto:Method: GET 2023-10-11T15:25:35.277 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:25:35.278 DEBUG:boto:Data: 2023-10-11T15:25:35.278 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.279 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.279 DEBUG:boto:Params: {} 2023-10-11T15:25:35.280 DEBUG:boto:Token: None 2023-10-11T15:25:35.280 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000668247fdcc81392a-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.285 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:25:35.286 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.287 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.287 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.287 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.288 DEBUG:boto:Method: GET 2023-10-11T15:25:35.288 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.289 DEBUG:boto:Data: 2023-10-11T15:25:35.289 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.290 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.291 DEBUG:boto:Params: {} 2023-10-11T15:25:35.291 DEBUG:boto:Token: None 2023-10-11T15:25:35.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE= 2023-10-11T15:25:35.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE=', 'Content-Length': '0'} 2023-10-11T15:25:35.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f56b70f0ebf51c8-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.296 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.297 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.298 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.298 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.299 DEBUG:boto:Method: GET 2023-10-11T15:25:35.300 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.301 DEBUG:boto:Data: 2023-10-11T15:25:35.302 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.303 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.303 DEBUG:boto:Params: {} 2023-10-11T15:25:35.304 DEBUG:boto:Token: None 2023-10-11T15:25:35.305 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE= 2023-10-11T15:25:35.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE=', 'Content-Length': '0'} 2023-10-11T15:25:35.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f4bdfb77f6b4bf2-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.350 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.351 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.351 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.352 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.352 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.353 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.353 DEBUG:boto:Data: 2023-10-11T15:25:35.354 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.354 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.355 DEBUG:boto:Params: {} 2023-10-11T15:25:35.355 DEBUG:boto:Token: None 2023-10-11T15:25:35.356 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GxO0lpiDxUaxI8CzwmpHODVXvMk= 2023-10-11T15:25:35.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GxO0lpiDxUaxI8CzwmpHODVXvMk=', 'Content-Length': '0'} 2023-10-11T15:25:35.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b863989a4686a7-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.359 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.360 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.360 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.361 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.361 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.361 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.362 DEBUG:boto:Data: 2023-10-11T15:25:35.362 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.363 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.363 DEBUG:boto:Params: {} 2023-10-11T15:25:35.364 DEBUG:boto:Token: None 2023-10-11T15:25:35.364 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GxO0lpiDxUaxI8CzwmpHODVXvMk= 2023-10-11T15:25:35.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GxO0lpiDxUaxI8CzwmpHODVXvMk=', 'Content-Length': '0'} 2023-10-11T15:25:35.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e850b6a0f4ccedf8-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:25:35.368 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.369 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.369 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.370 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.371 DEBUG:boto:Method: GET 2023-10-11T15:25:35.371 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.372 DEBUG:boto:Data: 2023-10-11T15:25:35.372 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.373 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.373 DEBUG:boto:Params: {} 2023-10-11T15:25:35.374 DEBUG:boto:Token: None 2023-10-11T15:25:35.374 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.375 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE= 2023-10-11T15:25:35.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE=', 'Content-Length': '0'} 2023-10-11T15:25:35.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee57cacbd22ac995-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.418 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.419 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.420 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.420 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.420 DEBUG:boto:Method: GET 2023-10-11T15:25:35.421 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.421 DEBUG:boto:Data: 2023-10-11T15:25:35.422 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.422 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.423 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.423 DEBUG:boto:Params: {} 2023-10-11T15:25:35.424 DEBUG:boto:Token: None 2023-10-11T15:25:35.424 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:25:35.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE= 2023-10-11T15:25:35.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lBqGlrthiFyvYQleQAYz1KJgxdE=', 'Content-Length': '0'} 2023-10-11T15:25:35.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000642104e9d712a42-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.428 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.429 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.429 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.430 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.430 DEBUG:boto:Method: GET 2023-10-11T15:25:35.430 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.431 DEBUG:boto:Data: 2023-10-11T15:25:35.431 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.432 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.432 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.433 DEBUG:boto:Params: {} 2023-10-11T15:25:35.433 DEBUG:boto:Token: None 2023-10-11T15:25:35.434 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a91c5a4a65f39c-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.438 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.439 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.439 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.439 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.440 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.440 DEBUG:boto:Method: GET 2023-10-11T15:25:35.441 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.441 DEBUG:boto:Data: 2023-10-11T15:25:35.442 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.442 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.443 DEBUG:boto:Params: {} 2023-10-11T15:25:35.443 DEBUG:boto:Token: None 2023-10-11T15:25:35.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfcdfb637931fae7-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.449 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.450 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.450 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.451 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.451 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:25:35.451 DEBUG:boto:Data: 2023-10-11T15:25:35.452 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.453 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.453 DEBUG:boto:Params: {} 2023-10-11T15:25:35.454 DEBUG:boto:Token: None 2023-10-11T15:25:35.454 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55 2023-10-11T15:25:35.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K2+cjudVC3oBuSRIU+UDklAqAR8= 2023-10-11T15:25:35.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K2+cjudVC3oBuSRIU+UDklAqAR8=', 'Content-Length': '0'} 2023-10-11T15:25:35.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d1b5e7d8b443a57-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.458 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.458 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.459 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.459 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:25:35.460 DEBUG:boto:Data: 2023-10-11T15:25:35.460 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.461 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.461 DEBUG:boto:Params: {} 2023-10-11T15:25:35.462 DEBUG:boto:Token: None 2023-10-11T15:25:35.462 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55 2023-10-11T15:25:35.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:K2+cjudVC3oBuSRIU+UDklAqAR8= 2023-10-11T15:25:35.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:K2+cjudVC3oBuSRIU+UDklAqAR8=', 'Content-Length': '0'} 2023-10-11T15:25:35.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2036b0f14c9594d-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:25:35.467 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.468 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.469 DEBUG:boto:Method: GET 2023-10-11T15:25:35.470 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:25:35.470 DEBUG:boto:Data: 2023-10-11T15:25:35.471 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.473 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.473 DEBUG:boto:Params: {} 2023-10-11T15:25:35.474 DEBUG:boto:Token: None 2023-10-11T15:25:35.475 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55 2023-10-11T15:25:35.475 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73x59V9sqLckyeFg8iD/MrirfXg= 2023-10-11T15:25:35.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73x59V9sqLckyeFg8iD/MrirfXg=', 'Content-Length': '0'} 2023-10-11T15:25:35.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc6dc23aa252e45-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.478 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:25:35.479 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:25:35.479 DEBUG:boto:Method: GET 2023-10-11T15:25:35.480 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:25:35.480 DEBUG:boto:Data: 2023-10-11T15:25:35.481 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.482 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.482 DEBUG:boto:Params: {} 2023-10-11T15:25:35.483 DEBUG:boto:Token: None 2023-10-11T15:25:35.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj55 2023-10-11T15:25:35.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:73x59V9sqLckyeFg8iD/MrirfXg= 2023-10-11T15:25:35.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:73x59V9sqLckyeFg8iD/MrirfXg=', 'Content-Length': '0'} 2023-10-11T15:25:35.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087304637c7e508c0-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.526 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.527 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.527 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.528 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.529 DEBUG:boto:Method: GET 2023-10-11T15:25:35.529 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.530 DEBUG:boto:Data: 2023-10-11T15:25:35.530 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.531 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.532 DEBUG:boto:Params: {} 2023-10-11T15:25:35.532 DEBUG:boto:Token: None 2023-10-11T15:25:35.533 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000564c4e8dfe42be29-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.537 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.538 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.538 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.539 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.539 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.539 DEBUG:boto:Method: GET 2023-10-11T15:25:35.540 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:25:35.540 DEBUG:boto:Data: 2023-10-11T15:25:35.541 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.542 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.543 DEBUG:boto:Params: {} 2023-10-11T15:25:35.543 DEBUG:boto:Token: None 2023-10-11T15:25:35.543 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd57b5c05d5f7eac-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.549 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:25:35.551 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.551 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.552 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.552 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.553 DEBUG:boto:Method: GET 2023-10-11T15:25:35.553 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.554 DEBUG:boto:Data: 2023-10-11T15:25:35.554 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.556 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.556 DEBUG:boto:Params: {} 2023-10-11T15:25:35.557 DEBUG:boto:Token: None 2023-10-11T15:25:35.558 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.559 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk= 2023-10-11T15:25:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk=', 'Content-Length': '0'} 2023-10-11T15:25:35.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630f0d13560c2e57-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.563 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.564 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.565 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.566 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.567 DEBUG:boto:Method: GET 2023-10-11T15:25:35.567 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.568 DEBUG:boto:Data: 2023-10-11T15:25:35.569 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.571 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.571 DEBUG:boto:Params: {} 2023-10-11T15:25:35.572 DEBUG:boto:Token: None 2023-10-11T15:25:35.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk= 2023-10-11T15:25:35.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk=', 'Content-Length': '0'} 2023-10-11T15:25:35.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003103edd02dd4e3b2-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.578 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.579 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.579 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.580 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.581 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.581 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.582 DEBUG:boto:Data: 2023-10-11T15:25:35.582 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.583 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.584 DEBUG:boto:Params: {} 2023-10-11T15:25:35.584 DEBUG:boto:Token: None 2023-10-11T15:25:35.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:918+X4l66JZ6/WNYcFYDBkId4iU= 2023-10-11T15:25:35.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:918+X4l66JZ6/WNYcFYDBkId4iU=', 'Content-Length': '0'} 2023-10-11T15:25:35.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3d14bc5f47d806-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.588 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.589 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.589 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.590 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.590 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.590 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.591 DEBUG:boto:Data: 2023-10-11T15:25:35.591 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.592 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.592 DEBUG:boto:Params: {} 2023-10-11T15:25:35.593 DEBUG:boto:Token: None 2023-10-11T15:25:35.594 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:918+X4l66JZ6/WNYcFYDBkId4iU= 2023-10-11T15:25:35.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:918+X4l66JZ6/WNYcFYDBkId4iU=', 'Content-Length': '0'} 2023-10-11T15:25:35.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c23bb4a4b66cde-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:25:35.597 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.598 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.598 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.599 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.600 DEBUG:boto:Method: GET 2023-10-11T15:25:35.601 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.601 DEBUG:boto:Data: 2023-10-11T15:25:35.602 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.603 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.604 DEBUG:boto:Params: {} 2023-10-11T15:25:35.604 DEBUG:boto:Token: None 2023-10-11T15:25:35.605 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk= 2023-10-11T15:25:35.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk=', 'Content-Length': '0'} 2023-10-11T15:25:35.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870b30dac33e44b8-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.609 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.610 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.610 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.610 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.611 DEBUG:boto:Method: GET 2023-10-11T15:25:35.611 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.612 DEBUG:boto:Data: 2023-10-11T15:25:35.612 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.613 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.613 DEBUG:boto:Params: {} 2023-10-11T15:25:35.614 DEBUG:boto:Token: None 2023-10-11T15:25:35.615 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:25:35.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk= 2023-10-11T15:25:35.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EePGsscL+qkOTayH9+KL1hqXGsk=', 'Content-Length': '0'} 2023-10-11T15:25:35.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce96608cab08618-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.618 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.619 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.619 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.619 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.620 DEBUG:boto:Method: GET 2023-10-11T15:25:35.620 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.621 DEBUG:boto:Data: 2023-10-11T15:25:35.621 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.622 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.622 DEBUG:boto:Params: {} 2023-10-11T15:25:35.623 DEBUG:boto:Token: None 2023-10-11T15:25:35.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.624 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8425f7b3a14ff9-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.627 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.628 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.628 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.629 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.629 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.630 DEBUG:boto:Method: GET 2023-10-11T15:25:35.630 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.631 DEBUG:boto:Data: 2023-10-11T15:25:35.631 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.632 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.632 DEBUG:boto:Params: {} 2023-10-11T15:25:35.633 DEBUG:boto:Token: None 2023-10-11T15:25:35.633 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.634 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f2a16d4ebf6e40-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.641 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.642 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.643 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.643 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.643 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:25:35.644 DEBUG:boto:Data: 2023-10-11T15:25:35.644 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.645 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.645 DEBUG:boto:Params: {} 2023-10-11T15:25:35.646 DEBUG:boto:Token: None 2023-10-11T15:25:35.647 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57 2023-10-11T15:25:35.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/SVbRZG1jH7mESbbjdINx8pLR0U= 2023-10-11T15:25:35.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/SVbRZG1jH7mESbbjdINx8pLR0U=', 'Content-Length': '0'} 2023-10-11T15:25:35.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5b8444a5244f3b-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.650 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.651 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.651 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.652 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:25:35.652 DEBUG:boto:Data: 2023-10-11T15:25:35.652 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.653 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.653 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.654 DEBUG:boto:Params: {} 2023-10-11T15:25:35.654 DEBUG:boto:Token: None 2023-10-11T15:25:35.655 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57 2023-10-11T15:25:35.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/SVbRZG1jH7mESbbjdINx8pLR0U= 2023-10-11T15:25:35.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/SVbRZG1jH7mESbbjdINx8pLR0U=', 'Content-Length': '0'} 2023-10-11T15:25:35.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089e218858de9c010-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:25:35.659 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.659 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.660 DEBUG:boto:Method: GET 2023-10-11T15:25:35.660 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:25:35.660 DEBUG:boto:Data: 2023-10-11T15:25:35.661 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.661 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.662 DEBUG:boto:Params: {} 2023-10-11T15:25:35.662 DEBUG:boto:Token: None 2023-10-11T15:25:35.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57 2023-10-11T15:25:35.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HWK2ihQ4Hntf1SI4H2yky2fJ3kk= 2023-10-11T15:25:35.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HWK2ihQ4Hntf1SI4H2yky2fJ3kk=', 'Content-Length': '0'} 2023-10-11T15:25:35.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec0647477066105-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.706 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:25:35.707 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:25:35.708 DEBUG:boto:Method: GET 2023-10-11T15:25:35.708 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:25:35.708 DEBUG:boto:Data: 2023-10-11T15:25:35.709 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.709 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.710 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.710 DEBUG:boto:Params: {} 2023-10-11T15:25:35.711 DEBUG:boto:Token: None 2023-10-11T15:25:35.711 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj57 2023-10-11T15:25:35.712 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HWK2ihQ4Hntf1SI4H2yky2fJ3kk= 2023-10-11T15:25:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HWK2ihQ4Hntf1SI4H2yky2fJ3kk=', 'Content-Length': '0'} 2023-10-11T15:25:35.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec2afb124322334-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.715 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.716 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.716 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.717 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.717 DEBUG:boto:Method: GET 2023-10-11T15:25:35.718 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.718 DEBUG:boto:Data: 2023-10-11T15:25:35.718 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.719 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.720 DEBUG:boto:Params: {} 2023-10-11T15:25:35.720 DEBUG:boto:Token: None 2023-10-11T15:25:35.721 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f29ca7664ed16f-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.725 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.726 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.726 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.727 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.727 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.728 DEBUG:boto:Method: GET 2023-10-11T15:25:35.728 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:25:35.728 DEBUG:boto:Data: 2023-10-11T15:25:35.729 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.730 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.730 DEBUG:boto:Params: {} 2023-10-11T15:25:35.731 DEBUG:boto:Token: None 2023-10-11T15:25:35.731 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce830f56c2c893c-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.737 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:25:35.738 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.738 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.739 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.739 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.740 DEBUG:boto:Method: GET 2023-10-11T15:25:35.740 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.740 DEBUG:boto:Data: 2023-10-11T15:25:35.741 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.742 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.742 DEBUG:boto:Params: {} 2023-10-11T15:25:35.743 DEBUG:boto:Token: None 2023-10-11T15:25:35.743 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E= 2023-10-11T15:25:35.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E=', 'Content-Length': '0'} 2023-10-11T15:25:35.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c28ecbe5532514-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.747 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.747 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.748 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.748 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.749 DEBUG:boto:Method: GET 2023-10-11T15:25:35.749 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.750 DEBUG:boto:Data: 2023-10-11T15:25:35.750 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.751 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.751 DEBUG:boto:Params: {} 2023-10-11T15:25:35.752 DEBUG:boto:Token: None 2023-10-11T15:25:35.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E= 2023-10-11T15:25:35.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E=', 'Content-Length': '0'} 2023-10-11T15:25:35.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3413b4792336dd0-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.756 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.757 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.757 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.757 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.758 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.758 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.759 DEBUG:boto:Data: 2023-10-11T15:25:35.759 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.760 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.760 DEBUG:boto:Params: {} 2023-10-11T15:25:35.761 DEBUG:boto:Token: None 2023-10-11T15:25:35.761 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:euIb5OZJiZn6Tso7oRLLJ0UY2GE= 2023-10-11T15:25:35.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:euIb5OZJiZn6Tso7oRLLJ0UY2GE=', 'Content-Length': '0'} 2023-10-11T15:25:35.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1144cc19327f742-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.765 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.765 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.766 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.766 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.767 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.767 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.768 DEBUG:boto:Data: 2023-10-11T15:25:35.768 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.769 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.769 DEBUG:boto:Params: {} 2023-10-11T15:25:35.770 DEBUG:boto:Token: None 2023-10-11T15:25:35.770 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:euIb5OZJiZn6Tso7oRLLJ0UY2GE= 2023-10-11T15:25:35.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:euIb5OZJiZn6Tso7oRLLJ0UY2GE=', 'Content-Length': '0'} 2023-10-11T15:25:35.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68bd4ae52438e60-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:25:35.774 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.775 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.775 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.776 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.776 DEBUG:boto:Method: GET 2023-10-11T15:25:35.776 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.777 DEBUG:boto:Data: 2023-10-11T15:25:35.778 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.778 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.779 DEBUG:boto:Params: {} 2023-10-11T15:25:35.779 DEBUG:boto:Token: None 2023-10-11T15:25:35.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.781 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E= 2023-10-11T15:25:35.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E=', 'Content-Length': '0'} 2023-10-11T15:25:35.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aef36a1b21ab3d72-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.784 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.784 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.785 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.785 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.785 DEBUG:boto:Method: GET 2023-10-11T15:25:35.786 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.786 DEBUG:boto:Data: 2023-10-11T15:25:35.787 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.788 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.788 DEBUG:boto:Params: {} 2023-10-11T15:25:35.788 DEBUG:boto:Token: None 2023-10-11T15:25:35.789 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:25:35.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E= 2023-10-11T15:25:35.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gUPTyPyocB2ggWJp8IQR/BU6g3E=', 'Content-Length': '0'} 2023-10-11T15:25:35.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1c6a5d08a5ffaf-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.793 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.793 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.794 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.794 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.795 DEBUG:boto:Method: GET 2023-10-11T15:25:35.795 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.796 DEBUG:boto:Data: 2023-10-11T15:25:35.796 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.797 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.797 DEBUG:boto:Params: {} 2023-10-11T15:25:35.798 DEBUG:boto:Token: None 2023-10-11T15:25:35.798 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.799 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedf03c36b6785b2-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.842 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.843 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.844 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.844 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.845 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.845 DEBUG:boto:Method: GET 2023-10-11T15:25:35.846 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.846 DEBUG:boto:Data: 2023-10-11T15:25:35.846 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.847 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.848 DEBUG:boto:Params: {} 2023-10-11T15:25:35.848 DEBUG:boto:Token: None 2023-10-11T15:25:35.849 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e068e60f0c0fe5a8-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.854 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.855 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.855 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.856 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.856 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:25:35.857 DEBUG:boto:Data: 2023-10-11T15:25:35.857 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.858 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.858 DEBUG:boto:Params: {} 2023-10-11T15:25:35.859 DEBUG:boto:Token: None 2023-10-11T15:25:35.859 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58 2023-10-11T15:25:35.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o0iS6ppPz0s+7Rqh0g4qK9U/CT4= 2023-10-11T15:25:35.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o0iS6ppPz0s+7Rqh0g4qK9U/CT4=', 'Content-Length': '0'} 2023-10-11T15:25:35.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac7ab5a7a66ff706-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.863 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.864 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.864 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.864 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:25:35.865 DEBUG:boto:Data: 2023-10-11T15:25:35.865 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.866 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.867 DEBUG:boto:Params: {} 2023-10-11T15:25:35.867 DEBUG:boto:Token: None 2023-10-11T15:25:35.867 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58 2023-10-11T15:25:35.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o0iS6ppPz0s+7Rqh0g4qK9U/CT4= 2023-10-11T15:25:35.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o0iS6ppPz0s+7Rqh0g4qK9U/CT4=', 'Content-Length': '0'} 2023-10-11T15:25:35.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f58c0a95a527dd-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:25:35.872 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.872 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.873 DEBUG:boto:Method: GET 2023-10-11T15:25:35.873 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:25:35.873 DEBUG:boto:Data: 2023-10-11T15:25:35.874 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.875 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.875 DEBUG:boto:Params: {} 2023-10-11T15:25:35.876 DEBUG:boto:Token: None 2023-10-11T15:25:35.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58 2023-10-11T15:25:35.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lrQ6F3oBzGLisvoV4wwjvK5NODI= 2023-10-11T15:25:35.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lrQ6F3oBzGLisvoV4wwjvK5NODI=', 'Content-Length': '0'} 2023-10-11T15:25:35.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082cd60eb67f896a3-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.880 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:25:35.880 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:25:35.881 DEBUG:boto:Method: GET 2023-10-11T15:25:35.881 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:25:35.882 DEBUG:boto:Data: 2023-10-11T15:25:35.882 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.883 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.884 DEBUG:boto:Params: {} 2023-10-11T15:25:35.884 DEBUG:boto:Token: None 2023-10-11T15:25:35.885 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj58 2023-10-11T15:25:35.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lrQ6F3oBzGLisvoV4wwjvK5NODI= 2023-10-11T15:25:35.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lrQ6F3oBzGLisvoV4wwjvK5NODI=', 'Content-Length': '0'} 2023-10-11T15:25:35.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598ae21cf3db95d3-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.930 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.931 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.931 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.932 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.932 DEBUG:boto:Method: GET 2023-10-11T15:25:35.933 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.933 DEBUG:boto:Data: 2023-10-11T15:25:35.934 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.935 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.935 DEBUG:boto:Params: {} 2023-10-11T15:25:35.936 DEBUG:boto:Token: None 2023-10-11T15:25:35.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0c1530a056a0f8-006526be6f-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.942 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.943 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:35.945 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:35.945 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.946 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.946 DEBUG:boto:Method: GET 2023-10-11T15:25:35.946 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:25:35.947 DEBUG:boto:Data: 2023-10-11T15:25:35.947 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.948 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.949 DEBUG:boto:Params: {} 2023-10-11T15:25:35.949 DEBUG:boto:Token: None 2023-10-11T15:25:35.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/ 2023-10-11T15:25:35.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ= 2023-10-11T15:25:35.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:igDfU+vlf7P6msTNrtg9vW5stHQ=', 'Content-Length': '0'} 2023-10-11T15:25:35.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059f4c1ea3ffe3bb-006526be6f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.955 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:35.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:25:35.957 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:35.957 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:35.957 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.958 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.959 DEBUG:boto:Method: GET 2023-10-11T15:25:35.959 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.960 DEBUG:boto:Data: 2023-10-11T15:25:35.960 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.960 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.961 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.961 DEBUG:boto:Params: {} 2023-10-11T15:25:35.962 DEBUG:boto:Token: None 2023-10-11T15:25:35.962 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.963 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nNdQYjmEyBn7HRxBux5Mc30zaBw= 2023-10-11T15:25:35.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nNdQYjmEyBn7HRxBux5Mc30zaBw=', 'Content-Length': '0'} 2023-10-11T15:25:35.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2babf94ba076497-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.967 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:35.967 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:35.968 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.968 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.968 DEBUG:boto:Method: GET 2023-10-11T15:25:35.969 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.969 DEBUG:boto:Data: 2023-10-11T15:25:35.970 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.971 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.972 DEBUG:boto:Params: {} 2023-10-11T15:25:35.972 DEBUG:boto:Token: None 2023-10-11T15:25:35.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nNdQYjmEyBn7HRxBux5Mc30zaBw= 2023-10-11T15:25:35.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nNdQYjmEyBn7HRxBux5Mc30zaBw=', 'Content-Length': '0'} 2023-10-11T15:25:35.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af91c2df56dd7462-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.977 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:35.977 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:35.977 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.978 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.978 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.979 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.979 DEBUG:boto:Data: 2023-10-11T15:25:35.980 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.980 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.981 DEBUG:boto:Params: {} 2023-10-11T15:25:35.981 DEBUG:boto:Token: None 2023-10-11T15:25:35.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bt79aRmpvh/jJ309CunxTI5ktK8= 2023-10-11T15:25:35.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bt79aRmpvh/jJ309CunxTI5ktK8=', 'Content-Length': '0'} 2023-10-11T15:25:35.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003367cfd1fb5b40ba-006526be6f-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.985 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:35.986 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:35.986 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.987 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.987 DEBUG:boto:Method: HEAD 2023-10-11T15:25:35.987 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.988 DEBUG:boto:Data: 2023-10-11T15:25:35.988 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:35.989 DEBUG:boto:Port: 8000 2023-10-11T15:25:35.990 DEBUG:boto:Params: {} 2023-10-11T15:25:35.990 DEBUG:boto:Token: None 2023-10-11T15:25:35.991 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:35 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bt79aRmpvh/jJ309CunxTI5ktK8= 2023-10-11T15:25:35.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bt79aRmpvh/jJ309CunxTI5ktK8=', 'Content-Length': '0'} 2023-10-11T15:25:35.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bfed517ded01bf-006526be6f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:35.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:25:35.994 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:35.995 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:35.995 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.996 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.996 DEBUG:boto:Method: GET 2023-10-11T15:25:35.996 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:35.997 DEBUG:boto:Data: 2023-10-11T15:25:35.997 DEBUG:boto:Headers: {} 2023-10-11T15:25:35.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:35.998 DEBUG:boto:Port: 8001 2023-10-11T15:25:35.999 DEBUG:boto:Params: {} 2023-10-11T15:25:35.999 DEBUG:boto:Token: None 2023-10-11T15:25:36.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:36.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DtfKaAlMnCZk0exwcCc6lyZsoCY= 2023-10-11T15:25:36.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DtfKaAlMnCZk0exwcCc6lyZsoCY=', 'Content-Length': '0'} 2023-10-11T15:25:36.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6fb9af9650be5a0-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.003 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:36.004 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:36.004 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:36.005 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:25:36.005 DEBUG:boto:Method: GET 2023-10-11T15:25:36.005 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:36.006 DEBUG:boto:Data: 2023-10-11T15:25:36.006 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.007 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.008 DEBUG:boto:Params: {} 2023-10-11T15:25:36.008 DEBUG:boto:Token: None 2023-10-11T15:25:36.009 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:25:36.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DtfKaAlMnCZk0exwcCc6lyZsoCY= 2023-10-11T15:25:36.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DtfKaAlMnCZk0exwcCc6lyZsoCY=', 'Content-Length': '0'} 2023-10-11T15:25:36.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da3e834f934713d4-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.012 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.013 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.013 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.013 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.014 DEBUG:boto:Method: GET 2023-10-11T15:25:36.014 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.015 DEBUG:boto:Data: 2023-10-11T15:25:36.015 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.016 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.016 DEBUG:boto:Params: {} 2023-10-11T15:25:36.017 DEBUG:boto:Token: None 2023-10-11T15:25:36.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b50c259c0cc479fd-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.066 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.067 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.068 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.068 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.069 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.069 DEBUG:boto:Method: GET 2023-10-11T15:25:36.070 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.070 DEBUG:boto:Data: 2023-10-11T15:25:36.071 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.071 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.071 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.072 DEBUG:boto:Params: {} 2023-10-11T15:25:36.072 DEBUG:boto:Token: None 2023-10-11T15:25:36.073 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.073 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b5f28217b303a8-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.078 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.079 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:36.079 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:36.080 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.080 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:25:36.081 DEBUG:boto:Data: 2023-10-11T15:25:36.081 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.082 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.082 DEBUG:boto:Params: {} 2023-10-11T15:25:36.083 DEBUG:boto:Token: None 2023-10-11T15:25:36.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62 2023-10-11T15:25:36.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wjHjAG+K7jDSQiQvdxdETGB+f9I= 2023-10-11T15:25:36.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wjHjAG+K7jDSQiQvdxdETGB+f9I=', 'Content-Length': '0'} 2023-10-11T15:25:36.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078184e08a6f6ed08-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.087 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:36.087 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:36.088 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.088 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:25:36.089 DEBUG:boto:Data: 2023-10-11T15:25:36.089 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.090 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.090 DEBUG:boto:Params: {} 2023-10-11T15:25:36.091 DEBUG:boto:Token: None 2023-10-11T15:25:36.091 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62 2023-10-11T15:25:36.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wjHjAG+K7jDSQiQvdxdETGB+f9I= 2023-10-11T15:25:36.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wjHjAG+K7jDSQiQvdxdETGB+f9I=', 'Content-Length': '0'} 2023-10-11T15:25:36.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f44fd7ff7a9ffb17-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:25:36.095 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:36.096 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:36.096 DEBUG:boto:Method: GET 2023-10-11T15:25:36.097 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:25:36.097 DEBUG:boto:Data: 2023-10-11T15:25:36.098 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.098 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.099 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.099 DEBUG:boto:Params: {} 2023-10-11T15:25:36.099 DEBUG:boto:Token: None 2023-10-11T15:25:36.100 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62 2023-10-11T15:25:36.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gZILKTw8AfwyWeYzzVMFJBmssug= 2023-10-11T15:25:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gZILKTw8AfwyWeYzzVMFJBmssug=', 'Content-Length': '0'} 2023-10-11T15:25:36.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006af4b3d622c433cf-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.104 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:25:36.104 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:25:36.105 DEBUG:boto:Method: GET 2023-10-11T15:25:36.105 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:25:36.105 DEBUG:boto:Data: 2023-10-11T15:25:36.106 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.107 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.107 DEBUG:boto:Params: {} 2023-10-11T15:25:36.108 DEBUG:boto:Token: None 2023-10-11T15:25:36.108 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj62 2023-10-11T15:25:36.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gZILKTw8AfwyWeYzzVMFJBmssug= 2023-10-11T15:25:36.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gZILKTw8AfwyWeYzzVMFJBmssug=', 'Content-Length': '0'} 2023-10-11T15:25:36.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000451ba0251cf0872d-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.154 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.155 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.155 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.156 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.156 DEBUG:boto:Method: GET 2023-10-11T15:25:36.157 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.157 DEBUG:boto:Data: 2023-10-11T15:25:36.158 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.158 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.159 DEBUG:boto:Params: {} 2023-10-11T15:25:36.159 DEBUG:boto:Token: None 2023-10-11T15:25:36.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002a0e36216dd24ac-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.164 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.165 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.166 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.166 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.167 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.167 DEBUG:boto:Method: GET 2023-10-11T15:25:36.167 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:25:36.168 DEBUG:boto:Data: 2023-10-11T15:25:36.168 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.169 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.170 DEBUG:boto:Params: {} 2023-10-11T15:25:36.170 DEBUG:boto:Token: None 2023-10-11T15:25:36.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039af4cf8611027cc-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.176 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:25:36.177 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.178 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.178 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.179 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.179 DEBUG:boto:Method: GET 2023-10-11T15:25:36.180 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.180 DEBUG:boto:Data: 2023-10-11T15:25:36.180 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.181 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.182 DEBUG:boto:Params: {} 2023-10-11T15:25:36.182 DEBUG:boto:Token: None 2023-10-11T15:25:36.183 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.183 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA= 2023-10-11T15:25:36.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA=', 'Content-Length': '0'} 2023-10-11T15:25:36.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82121798db79344-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.230 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.231 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.231 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.232 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.232 DEBUG:boto:Method: GET 2023-10-11T15:25:36.233 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.233 DEBUG:boto:Data: 2023-10-11T15:25:36.234 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.235 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.235 DEBUG:boto:Params: {} 2023-10-11T15:25:36.236 DEBUG:boto:Token: None 2023-10-11T15:25:36.236 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA= 2023-10-11T15:25:36.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA=', 'Content-Length': '0'} 2023-10-11T15:25:36.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080105694610911ae-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.240 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.241 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.241 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.242 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.242 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.242 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.243 DEBUG:boto:Data: 2023-10-11T15:25:36.243 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.244 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.244 DEBUG:boto:Params: {} 2023-10-11T15:25:36.245 DEBUG:boto:Token: None 2023-10-11T15:25:36.245 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QtdlAyWbOzN1+2CiX+iDJ/7CmmE= 2023-10-11T15:25:36.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QtdlAyWbOzN1+2CiX+iDJ/7CmmE=', 'Content-Length': '0'} 2023-10-11T15:25:36.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a7061e66695ff5e-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.249 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.249 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.250 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.250 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.251 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.251 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.252 DEBUG:boto:Data: 2023-10-11T15:25:36.252 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.252 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.253 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.253 DEBUG:boto:Params: {} 2023-10-11T15:25:36.254 DEBUG:boto:Token: None 2023-10-11T15:25:36.254 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QtdlAyWbOzN1+2CiX+iDJ/7CmmE= 2023-10-11T15:25:36.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QtdlAyWbOzN1+2CiX+iDJ/7CmmE=', 'Content-Length': '0'} 2023-10-11T15:25:36.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9241ab2fe1c9f2-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:25:36.258 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.259 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.259 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.260 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.260 DEBUG:boto:Method: GET 2023-10-11T15:25:36.261 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.261 DEBUG:boto:Data: 2023-10-11T15:25:36.261 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.262 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.263 DEBUG:boto:Params: {} 2023-10-11T15:25:36.263 DEBUG:boto:Token: None 2023-10-11T15:25:36.264 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA= 2023-10-11T15:25:36.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA=', 'Content-Length': '0'} 2023-10-11T15:25:36.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a40d8b4a5a34dd7-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.268 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.268 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.268 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.269 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.269 DEBUG:boto:Method: GET 2023-10-11T15:25:36.270 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.270 DEBUG:boto:Data: 2023-10-11T15:25:36.271 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.272 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.272 DEBUG:boto:Params: {} 2023-10-11T15:25:36.273 DEBUG:boto:Token: None 2023-10-11T15:25:36.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:25:36.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA= 2023-10-11T15:25:36.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYYgWz/J8n41ljobgIPM8t8pSRA=', 'Content-Length': '0'} 2023-10-11T15:25:36.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4378d59e58d9a3-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.277 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.277 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.278 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.278 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.278 DEBUG:boto:Method: GET 2023-10-11T15:25:36.279 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.279 DEBUG:boto:Data: 2023-10-11T15:25:36.280 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.281 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.281 DEBUG:boto:Params: {} 2023-10-11T15:25:36.281 DEBUG:boto:Token: None 2023-10-11T15:25:36.282 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d02a7522e63a2b21-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.286 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.287 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.288 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.288 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.288 DEBUG:boto:Method: GET 2023-10-11T15:25:36.289 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.289 DEBUG:boto:Data: 2023-10-11T15:25:36.290 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.291 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.291 DEBUG:boto:Params: {} 2023-10-11T15:25:36.292 DEBUG:boto:Token: None 2023-10-11T15:25:36.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eae0a8dadb7813b-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.297 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.298 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.299 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.299 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.300 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:25:36.300 DEBUG:boto:Data: 2023-10-11T15:25:36.300 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.301 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.301 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.301 DEBUG:boto:Params: {} 2023-10-11T15:25:36.302 DEBUG:boto:Token: None 2023-10-11T15:25:36.302 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68 2023-10-11T15:25:36.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S2qyI9hs8gpXNyLQjH8h52YzZwg= 2023-10-11T15:25:36.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S2qyI9hs8gpXNyLQjH8h52YzZwg=', 'Content-Length': '0'} 2023-10-11T15:25:36.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff7efef5d75bbd4-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.306 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.306 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.307 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.307 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:25:36.308 DEBUG:boto:Data: 2023-10-11T15:25:36.308 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.309 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.309 DEBUG:boto:Params: {} 2023-10-11T15:25:36.310 DEBUG:boto:Token: None 2023-10-11T15:25:36.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68 2023-10-11T15:25:36.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S2qyI9hs8gpXNyLQjH8h52YzZwg= 2023-10-11T15:25:36.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S2qyI9hs8gpXNyLQjH8h52YzZwg=', 'Content-Length': '0'} 2023-10-11T15:25:36.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000742d8eace52fc61-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:25:36.315 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.315 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.316 DEBUG:boto:Method: GET 2023-10-11T15:25:36.316 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:25:36.317 DEBUG:boto:Data: 2023-10-11T15:25:36.317 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.318 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.318 DEBUG:boto:Params: {} 2023-10-11T15:25:36.319 DEBUG:boto:Token: None 2023-10-11T15:25:36.319 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68 2023-10-11T15:25:36.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jp5u+JwH2hzQEJc+X6jxfSseZow= 2023-10-11T15:25:36.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jp5u+JwH2hzQEJc+X6jxfSseZow=', 'Content-Length': '0'} 2023-10-11T15:25:36.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004380f0287e2b710b-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.323 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:25:36.323 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:25:36.324 DEBUG:boto:Method: GET 2023-10-11T15:25:36.324 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:25:36.325 DEBUG:boto:Data: 2023-10-11T15:25:36.325 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.326 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.326 DEBUG:boto:Params: {} 2023-10-11T15:25:36.327 DEBUG:boto:Token: None 2023-10-11T15:25:36.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj68 2023-10-11T15:25:36.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jp5u+JwH2hzQEJc+X6jxfSseZow= 2023-10-11T15:25:36.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jp5u+JwH2hzQEJc+X6jxfSseZow=', 'Content-Length': '0'} 2023-10-11T15:25:36.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb675897ee419de-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.331 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.331 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.332 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.332 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.333 DEBUG:boto:Method: GET 2023-10-11T15:25:36.333 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.333 DEBUG:boto:Data: 2023-10-11T15:25:36.334 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.335 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.335 DEBUG:boto:Params: {} 2023-10-11T15:25:36.336 DEBUG:boto:Token: None 2023-10-11T15:25:36.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49bcbddee5e4e9e-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.382 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.383 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.384 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.384 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.385 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.385 DEBUG:boto:Method: GET 2023-10-11T15:25:36.386 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:25:36.386 DEBUG:boto:Data: 2023-10-11T15:25:36.387 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.388 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.388 DEBUG:boto:Params: {} 2023-10-11T15:25:36.389 DEBUG:boto:Token: None 2023-10-11T15:25:36.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.390 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fba374b3e1fbac48-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.394 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:25:36.396 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.396 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.396 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.397 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.397 DEBUG:boto:Method: GET 2023-10-11T15:25:36.398 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.398 DEBUG:boto:Data: 2023-10-11T15:25:36.399 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.399 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.400 DEBUG:boto:Params: {} 2023-10-11T15:25:36.400 DEBUG:boto:Token: None 2023-10-11T15:25:36.401 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY= 2023-10-11T15:25:36.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY=', 'Content-Length': '0'} 2023-10-11T15:25:36.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000762d11f4b9395148-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.405 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.405 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.406 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.406 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.407 DEBUG:boto:Method: GET 2023-10-11T15:25:36.407 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.407 DEBUG:boto:Data: 2023-10-11T15:25:36.408 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.409 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.409 DEBUG:boto:Params: {} 2023-10-11T15:25:36.410 DEBUG:boto:Token: None 2023-10-11T15:25:36.410 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY= 2023-10-11T15:25:36.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY=', 'Content-Length': '0'} 2023-10-11T15:25:36.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ba4c105cc728ad-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.414 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.414 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.415 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.415 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.416 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.416 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.416 DEBUG:boto:Data: 2023-10-11T15:25:36.417 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.418 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.418 DEBUG:boto:Params: {} 2023-10-11T15:25:36.418 DEBUG:boto:Token: None 2023-10-11T15:25:36.419 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ef/pNx6KMw2Yls/yLJ357wSNcCM= 2023-10-11T15:25:36.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ef/pNx6KMw2Yls/yLJ357wSNcCM=', 'Content-Length': '0'} 2023-10-11T15:25:36.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e9a70b2608b88e-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.422 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.423 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.423 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.424 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.424 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.424 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.425 DEBUG:boto:Data: 2023-10-11T15:25:36.425 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.426 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.427 DEBUG:boto:Params: {} 2023-10-11T15:25:36.427 DEBUG:boto:Token: None 2023-10-11T15:25:36.427 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ef/pNx6KMw2Yls/yLJ357wSNcCM= 2023-10-11T15:25:36.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ef/pNx6KMw2Yls/yLJ357wSNcCM=', 'Content-Length': '0'} 2023-10-11T15:25:36.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6cc691e50e47b8b-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:25:36.431 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.432 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.432 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.433 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.433 DEBUG:boto:Method: GET 2023-10-11T15:25:36.434 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.434 DEBUG:boto:Data: 2023-10-11T15:25:36.435 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.436 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.436 DEBUG:boto:Params: {} 2023-10-11T15:25:36.437 DEBUG:boto:Token: None 2023-10-11T15:25:36.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY= 2023-10-11T15:25:36.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY=', 'Content-Length': '0'} 2023-10-11T15:25:36.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0aa4642132fcda9-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.441 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.441 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.441 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.442 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.442 DEBUG:boto:Method: GET 2023-10-11T15:25:36.443 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.443 DEBUG:boto:Data: 2023-10-11T15:25:36.444 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.444 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.444 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.445 DEBUG:boto:Params: {} 2023-10-11T15:25:36.445 DEBUG:boto:Token: None 2023-10-11T15:25:36.446 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:25:36.446 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY= 2023-10-11T15:25:36.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vipsw+m3pxiEqleNB2aO9Upb8JY=', 'Content-Length': '0'} 2023-10-11T15:25:36.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000674adb8929fe91a-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.494 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.495 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.495 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.496 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.496 DEBUG:boto:Method: GET 2023-10-11T15:25:36.497 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.497 DEBUG:boto:Data: 2023-10-11T15:25:36.498 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.499 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.499 DEBUG:boto:Params: {} 2023-10-11T15:25:36.500 DEBUG:boto:Token: None 2023-10-11T15:25:36.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30b26ad56077007-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.504 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.505 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.505 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.506 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.506 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.506 DEBUG:boto:Method: GET 2023-10-11T15:25:36.507 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.507 DEBUG:boto:Data: 2023-10-11T15:25:36.508 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.509 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.509 DEBUG:boto:Params: {} 2023-10-11T15:25:36.510 DEBUG:boto:Token: None 2023-10-11T15:25:36.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9a28782546d567-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.516 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.517 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.518 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.518 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.519 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:25:36.519 DEBUG:boto:Data: 2023-10-11T15:25:36.520 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.520 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.521 DEBUG:boto:Params: {} 2023-10-11T15:25:36.521 DEBUG:boto:Token: None 2023-10-11T15:25:36.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69 2023-10-11T15:25:36.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4v8lLufmBoGI+afbHgq0sTRhUVU= 2023-10-11T15:25:36.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4v8lLufmBoGI+afbHgq0sTRhUVU=', 'Content-Length': '0'} 2023-10-11T15:25:36.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033beb4ee8bb710d0-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.525 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.525 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.526 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.526 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:25:36.527 DEBUG:boto:Data: 2023-10-11T15:25:36.527 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.528 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.528 DEBUG:boto:Params: {} 2023-10-11T15:25:36.529 DEBUG:boto:Token: None 2023-10-11T15:25:36.529 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69 2023-10-11T15:25:36.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4v8lLufmBoGI+afbHgq0sTRhUVU= 2023-10-11T15:25:36.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4v8lLufmBoGI+afbHgq0sTRhUVU=', 'Content-Length': '0'} 2023-10-11T15:25:36.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf380fcb96895bb-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:25:36.533 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.534 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.534 DEBUG:boto:Method: GET 2023-10-11T15:25:36.535 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:25:36.535 DEBUG:boto:Data: 2023-10-11T15:25:36.535 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.536 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.537 DEBUG:boto:Params: {} 2023-10-11T15:25:36.537 DEBUG:boto:Token: None 2023-10-11T15:25:36.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69 2023-10-11T15:25:36.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BDSym0CK0y4rsUzNJtHHfMZ+1NE= 2023-10-11T15:25:36.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BDSym0CK0y4rsUzNJtHHfMZ+1NE=', 'Content-Length': '0'} 2023-10-11T15:25:36.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000672a442f731db64b-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.542 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:25:36.543 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:25:36.543 DEBUG:boto:Method: GET 2023-10-11T15:25:36.543 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:25:36.544 DEBUG:boto:Data: 2023-10-11T15:25:36.544 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.545 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.545 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.546 DEBUG:boto:Params: {} 2023-10-11T15:25:36.546 DEBUG:boto:Token: None 2023-10-11T15:25:36.547 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj69 2023-10-11T15:25:36.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BDSym0CK0y4rsUzNJtHHfMZ+1NE= 2023-10-11T15:25:36.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BDSym0CK0y4rsUzNJtHHfMZ+1NE=', 'Content-Length': '0'} 2023-10-11T15:25:36.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009453954d2591f2f1-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.551 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.551 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.551 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.552 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.552 DEBUG:boto:Method: GET 2023-10-11T15:25:36.553 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.553 DEBUG:boto:Data: 2023-10-11T15:25:36.553 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.554 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.555 DEBUG:boto:Params: {} 2023-10-11T15:25:36.555 DEBUG:boto:Token: None 2023-10-11T15:25:36.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6da97e29718a96-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.559 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.560 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.561 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.561 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.561 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.562 DEBUG:boto:Method: GET 2023-10-11T15:25:36.562 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:25:36.563 DEBUG:boto:Data: 2023-10-11T15:25:36.563 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.564 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.564 DEBUG:boto:Params: {} 2023-10-11T15:25:36.565 DEBUG:boto:Token: None 2023-10-11T15:25:36.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088a725ca88a9e3bf-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.571 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:25:36.572 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.572 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.572 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.573 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.573 DEBUG:boto:Method: GET 2023-10-11T15:25:36.574 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.574 DEBUG:boto:Data: 2023-10-11T15:25:36.575 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.576 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.576 DEBUG:boto:Params: {} 2023-10-11T15:25:36.577 DEBUG:boto:Token: None 2023-10-11T15:25:36.577 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.578 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4= 2023-10-11T15:25:36.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4=', 'Content-Length': '0'} 2023-10-11T15:25:36.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9551796fdaed81f-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.626 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.627 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.628 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.629 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.629 DEBUG:boto:Method: GET 2023-10-11T15:25:36.630 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.631 DEBUG:boto:Data: 2023-10-11T15:25:36.632 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.633 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.634 DEBUG:boto:Params: {} 2023-10-11T15:25:36.634 DEBUG:boto:Token: None 2023-10-11T15:25:36.635 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4= 2023-10-11T15:25:36.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4=', 'Content-Length': '0'} 2023-10-11T15:25:36.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5165f0e4b1eaca1-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.639 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.639 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.640 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.640 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.641 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.641 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.641 DEBUG:boto:Data: 2023-10-11T15:25:36.642 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.643 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.643 DEBUG:boto:Params: {} 2023-10-11T15:25:36.643 DEBUG:boto:Token: None 2023-10-11T15:25:36.644 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mV4/H2rHTcSARHwNnLZwwm6a2Ic= 2023-10-11T15:25:36.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mV4/H2rHTcSARHwNnLZwwm6a2Ic=', 'Content-Length': '0'} 2023-10-11T15:25:36.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c096a9cec488d62-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.648 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.649 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.649 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.650 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.650 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.650 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.651 DEBUG:boto:Data: 2023-10-11T15:25:36.651 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.652 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.652 DEBUG:boto:Params: {} 2023-10-11T15:25:36.653 DEBUG:boto:Token: None 2023-10-11T15:25:36.654 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mV4/H2rHTcSARHwNnLZwwm6a2Ic= 2023-10-11T15:25:36.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mV4/H2rHTcSARHwNnLZwwm6a2Ic=', 'Content-Length': '0'} 2023-10-11T15:25:36.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11ec246d050ee55-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:25:36.658 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.658 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.659 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.659 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.659 DEBUG:boto:Method: GET 2023-10-11T15:25:36.660 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.660 DEBUG:boto:Data: 2023-10-11T15:25:36.661 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.662 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.662 DEBUG:boto:Params: {} 2023-10-11T15:25:36.663 DEBUG:boto:Token: None 2023-10-11T15:25:36.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4= 2023-10-11T15:25:36.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4=', 'Content-Length': '0'} 2023-10-11T15:25:36.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000313599a5271e8e77-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.667 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.667 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.668 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.668 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.669 DEBUG:boto:Method: GET 2023-10-11T15:25:36.669 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.670 DEBUG:boto:Data: 2023-10-11T15:25:36.670 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.671 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.672 DEBUG:boto:Params: {} 2023-10-11T15:25:36.673 DEBUG:boto:Token: None 2023-10-11T15:25:36.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:25:36.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4= 2023-10-11T15:25:36.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SLNl8gSQpGyFHHA2c71L4JOkJX4=', 'Content-Length': '0'} 2023-10-11T15:25:36.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e373afe5f27745ed-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.718 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.719 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.719 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.720 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.720 DEBUG:boto:Method: GET 2023-10-11T15:25:36.721 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.721 DEBUG:boto:Data: 2023-10-11T15:25:36.722 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.723 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.723 DEBUG:boto:Params: {} 2023-10-11T15:25:36.724 DEBUG:boto:Token: None 2023-10-11T15:25:36.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce5f73287fbbf2b-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.728 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.729 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.730 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.730 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.731 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.731 DEBUG:boto:Method: GET 2023-10-11T15:25:36.732 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.732 DEBUG:boto:Data: 2023-10-11T15:25:36.732 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.733 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.733 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.734 DEBUG:boto:Params: {} 2023-10-11T15:25:36.734 DEBUG:boto:Token: None 2023-10-11T15:25:36.735 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b97df3d11230c2c-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.740 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.741 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.741 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.742 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.742 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:25:36.743 DEBUG:boto:Data: 2023-10-11T15:25:36.743 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.744 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.744 DEBUG:boto:Params: {} 2023-10-11T15:25:36.745 DEBUG:boto:Token: None 2023-10-11T15:25:36.745 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76 2023-10-11T15:25:36.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4Jr+Lx4JwMQH/Vuobb6TZWDCI/4= 2023-10-11T15:25:36.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Jr+Lx4JwMQH/Vuobb6TZWDCI/4=', 'Content-Length': '0'} 2023-10-11T15:25:36.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c67f4ae08437bbc-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.749 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.750 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.750 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.751 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:25:36.751 DEBUG:boto:Data: 2023-10-11T15:25:36.752 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.753 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.753 DEBUG:boto:Params: {} 2023-10-11T15:25:36.754 DEBUG:boto:Token: None 2023-10-11T15:25:36.755 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76 2023-10-11T15:25:36.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4Jr+Lx4JwMQH/Vuobb6TZWDCI/4= 2023-10-11T15:25:36.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Jr+Lx4JwMQH/Vuobb6TZWDCI/4=', 'Content-Length': '0'} 2023-10-11T15:25:36.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1842a29d1cb25f3-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:25:36.760 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.760 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.760 DEBUG:boto:Method: GET 2023-10-11T15:25:36.761 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:25:36.761 DEBUG:boto:Data: 2023-10-11T15:25:36.762 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.763 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.763 DEBUG:boto:Params: {} 2023-10-11T15:25:36.764 DEBUG:boto:Token: None 2023-10-11T15:25:36.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76 2023-10-11T15:25:36.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1MhR0ce0UoNFMgO+UyrC7J1LbU= 2023-10-11T15:25:36.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1MhR0ce0UoNFMgO+UyrC7J1LbU=', 'Content-Length': '0'} 2023-10-11T15:25:36.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007235ca620a1e63b8-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.768 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:25:36.768 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:25:36.769 DEBUG:boto:Method: GET 2023-10-11T15:25:36.769 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:25:36.769 DEBUG:boto:Data: 2023-10-11T15:25:36.770 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.771 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.771 DEBUG:boto:Params: {} 2023-10-11T15:25:36.772 DEBUG:boto:Token: None 2023-10-11T15:25:36.772 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj76 2023-10-11T15:25:36.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1MhR0ce0UoNFMgO+UyrC7J1LbU= 2023-10-11T15:25:36.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1MhR0ce0UoNFMgO+UyrC7J1LbU=', 'Content-Length': '0'} 2023-10-11T15:25:36.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500aef91bbc20d41-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.822 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.823 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.824 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.824 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.825 DEBUG:boto:Method: GET 2023-10-11T15:25:36.825 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.825 DEBUG:boto:Data: 2023-10-11T15:25:36.826 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.827 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.827 DEBUG:boto:Params: {} 2023-10-11T15:25:36.827 DEBUG:boto:Token: None 2023-10-11T15:25:36.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423fe13dd6baded3-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.834 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.835 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.835 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.836 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.836 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.836 DEBUG:boto:Method: GET 2023-10-11T15:25:36.837 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:25:36.837 DEBUG:boto:Data: 2023-10-11T15:25:36.838 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.839 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.840 DEBUG:boto:Params: {} 2023-10-11T15:25:36.841 DEBUG:boto:Token: None 2023-10-11T15:25:36.841 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.842 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee212785ccfd151-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.848 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:25:36.849 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.849 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.850 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.851 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.852 DEBUG:boto:Method: GET 2023-10-11T15:25:36.852 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.853 DEBUG:boto:Data: 2023-10-11T15:25:36.853 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.854 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.855 DEBUG:boto:Params: {} 2023-10-11T15:25:36.856 DEBUG:boto:Token: None 2023-10-11T15:25:36.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI= 2023-10-11T15:25:36.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI=', 'Content-Length': '0'} 2023-10-11T15:25:36.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95bfab965e6d590-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.862 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.863 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.863 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.864 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.865 DEBUG:boto:Method: GET 2023-10-11T15:25:36.865 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.866 DEBUG:boto:Data: 2023-10-11T15:25:36.867 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.868 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.869 DEBUG:boto:Params: {} 2023-10-11T15:25:36.870 DEBUG:boto:Token: None 2023-10-11T15:25:36.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.872 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI= 2023-10-11T15:25:36.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI=', 'Content-Length': '0'} 2023-10-11T15:25:36.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8ac9cc51657bae-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.876 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.877 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.877 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.878 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.879 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.880 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.880 DEBUG:boto:Data: 2023-10-11T15:25:36.881 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.883 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.883 DEBUG:boto:Params: {} 2023-10-11T15:25:36.884 DEBUG:boto:Token: None 2023-10-11T15:25:36.885 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tez/vzO7GX4wbqnk6ljMXRScj3o= 2023-10-11T15:25:36.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tez/vzO7GX4wbqnk6ljMXRScj3o=', 'Content-Length': '0'} 2023-10-11T15:25:36.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db7d306a820fbc0-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.890 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.891 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.891 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.892 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.893 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.894 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.894 DEBUG:boto:Data: 2023-10-11T15:25:36.895 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.895 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.896 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.897 DEBUG:boto:Params: {} 2023-10-11T15:25:36.897 DEBUG:boto:Token: None 2023-10-11T15:25:36.898 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.899 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tez/vzO7GX4wbqnk6ljMXRScj3o= 2023-10-11T15:25:36.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tez/vzO7GX4wbqnk6ljMXRScj3o=', 'Content-Length': '0'} 2023-10-11T15:25:36.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fec34e5b99091b-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:25:36.903 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.903 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.904 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.904 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.905 DEBUG:boto:Method: GET 2023-10-11T15:25:36.905 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.906 DEBUG:boto:Data: 2023-10-11T15:25:36.906 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.907 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.907 DEBUG:boto:Params: {} 2023-10-11T15:25:36.908 DEBUG:boto:Token: None 2023-10-11T15:25:36.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI= 2023-10-11T15:25:36.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI=', 'Content-Length': '0'} 2023-10-11T15:25:36.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039408415103dab17-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.912 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.912 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.912 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.913 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.913 DEBUG:boto:Method: GET 2023-10-11T15:25:36.914 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.914 DEBUG:boto:Data: 2023-10-11T15:25:36.915 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.915 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.916 DEBUG:boto:Params: {} 2023-10-11T15:25:36.916 DEBUG:boto:Token: None 2023-10-11T15:25:36.917 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:25:36.917 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI= 2023-10-11T15:25:36.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8pxT2jK8iA0SzKoFv1OhS8NW6YI=', 'Content-Length': '0'} 2023-10-11T15:25:36.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e2f930c720d09b-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.962 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.963 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.963 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.964 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.964 DEBUG:boto:Method: GET 2023-10-11T15:25:36.965 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.965 DEBUG:boto:Data: 2023-10-11T15:25:36.966 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.966 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.967 DEBUG:boto:Params: {} 2023-10-11T15:25:36.967 DEBUG:boto:Token: None 2023-10-11T15:25:36.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066f584ebe5fee212-006526be70-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.972 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.973 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:36.973 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:36.974 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.974 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.975 DEBUG:boto:Method: GET 2023-10-11T15:25:36.975 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:25:36.975 DEBUG:boto:Data: 2023-10-11T15:25:36.976 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.977 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.977 DEBUG:boto:Params: {} 2023-10-11T15:25:36.977 DEBUG:boto:Token: None 2023-10-11T15:25:36.978 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/ 2023-10-11T15:25:36.978 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c= 2023-10-11T15:25:36.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5cd8s1M8XCwpmIjx5E+SuVH3f0c=', 'Content-Length': '0'} 2023-10-11T15:25:36.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a644520d05cefcb-006526be70-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.984 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:36.984 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.985 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.985 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.986 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:25:36.986 DEBUG:boto:Data: 2023-10-11T15:25:36.986 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:36.987 DEBUG:boto:Port: 8001 2023-10-11T15:25:36.988 DEBUG:boto:Params: {} 2023-10-11T15:25:36.988 DEBUG:boto:Token: None 2023-10-11T15:25:36.989 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85 2023-10-11T15:25:36.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rr7AUaS/Z/l7x9eURy+VnAMyt+I= 2023-10-11T15:25:36.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rr7AUaS/Z/l7x9eURy+VnAMyt+I=', 'Content-Length': '0'} 2023-10-11T15:25:36.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a704246367f410-006526be70-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:36.992 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:36.993 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:36.993 DEBUG:boto:Method: HEAD 2023-10-11T15:25:36.993 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:25:36.994 DEBUG:boto:Data: 2023-10-11T15:25:36.994 DEBUG:boto:Headers: {} 2023-10-11T15:25:36.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:36.995 DEBUG:boto:Port: 8000 2023-10-11T15:25:36.996 DEBUG:boto:Params: {} 2023-10-11T15:25:36.996 DEBUG:boto:Token: None 2023-10-11T15:25:36.996 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:36 GMT /dghqvd-59/obj85 2023-10-11T15:25:36.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rr7AUaS/Z/l7x9eURy+VnAMyt+I= 2023-10-11T15:25:36.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rr7AUaS/Z/l7x9eURy+VnAMyt+I=', 'Content-Length': '0'} 2023-10-11T15:25:36.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c913566f062bc2-006526be70-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:25:37.001 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:37.001 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:37.001 DEBUG:boto:Method: GET 2023-10-11T15:25:37.002 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:25:37.002 DEBUG:boto:Data: 2023-10-11T15:25:37.003 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.003 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.004 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.004 DEBUG:boto:Params: {} 2023-10-11T15:25:37.004 DEBUG:boto:Token: None 2023-10-11T15:25:37.005 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj85 2023-10-11T15:25:37.005 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1KpTk9GigW6G/iTFou+AMQQcrbs= 2023-10-11T15:25:37.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1KpTk9GigW6G/iTFou+AMQQcrbs=', 'Content-Length': '0'} 2023-10-11T15:25:37.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7df9c6bd3bcae5-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.054 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:25:37.055 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:25:37.056 DEBUG:boto:Method: GET 2023-10-11T15:25:37.056 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:25:37.056 DEBUG:boto:Data: 2023-10-11T15:25:37.057 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.058 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.058 DEBUG:boto:Params: {} 2023-10-11T15:25:37.059 DEBUG:boto:Token: None 2023-10-11T15:25:37.059 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj85 2023-10-11T15:25:37.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1KpTk9GigW6G/iTFou+AMQQcrbs= 2023-10-11T15:25:37.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1KpTk9GigW6G/iTFou+AMQQcrbs=', 'Content-Length': '0'} 2023-10-11T15:25:37.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1aa2532cc716d8-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.063 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.063 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.064 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.064 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.065 DEBUG:boto:Method: GET 2023-10-11T15:25:37.065 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.065 DEBUG:boto:Data: 2023-10-11T15:25:37.066 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.067 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.067 DEBUG:boto:Params: {} 2023-10-11T15:25:37.068 DEBUG:boto:Token: None 2023-10-11T15:25:37.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ad0da76c02a78f7-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.072 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.073 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.073 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.074 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.074 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.075 DEBUG:boto:Method: GET 2023-10-11T15:25:37.075 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:25:37.076 DEBUG:boto:Data: 2023-10-11T15:25:37.076 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.077 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.077 DEBUG:boto:Params: {} 2023-10-11T15:25:37.078 DEBUG:boto:Token: None 2023-10-11T15:25:37.078 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007986712becc7ff9d-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.126 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:25:37.128 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.128 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.129 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.129 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.129 DEBUG:boto:Method: GET 2023-10-11T15:25:37.130 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.130 DEBUG:boto:Data: 2023-10-11T15:25:37.131 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.132 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.132 DEBUG:boto:Params: {} 2023-10-11T15:25:37.133 DEBUG:boto:Token: None 2023-10-11T15:25:37.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU= 2023-10-11T15:25:37.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU=', 'Content-Length': '0'} 2023-10-11T15:25:37.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce943bb191687a31-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.137 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.137 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.138 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.138 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.139 DEBUG:boto:Method: GET 2023-10-11T15:25:37.139 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.140 DEBUG:boto:Data: 2023-10-11T15:25:37.140 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.141 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.141 DEBUG:boto:Params: {} 2023-10-11T15:25:37.142 DEBUG:boto:Token: None 2023-10-11T15:25:37.142 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.143 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU= 2023-10-11T15:25:37.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU=', 'Content-Length': '0'} 2023-10-11T15:25:37.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6181625764e5f4f-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.146 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.146 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.147 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.147 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.147 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.148 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.148 DEBUG:boto:Data: 2023-10-11T15:25:37.149 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.149 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.150 DEBUG:boto:Params: {} 2023-10-11T15:25:37.150 DEBUG:boto:Token: None 2023-10-11T15:25:37.151 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.151 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQxPP+uK1rnQl2PcpheGzmzatcQ= 2023-10-11T15:25:37.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQxPP+uK1rnQl2PcpheGzmzatcQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a8b9929dbbc8a15-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.155 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.155 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.156 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.157 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.157 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.158 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.158 DEBUG:boto:Data: 2023-10-11T15:25:37.159 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.161 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.161 DEBUG:boto:Params: {} 2023-10-11T15:25:37.161 DEBUG:boto:Token: None 2023-10-11T15:25:37.162 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQxPP+uK1rnQl2PcpheGzmzatcQ= 2023-10-11T15:25:37.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQxPP+uK1rnQl2PcpheGzmzatcQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea682bd0108bdcb3-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:25:37.167 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.167 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.168 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.168 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.168 DEBUG:boto:Method: GET 2023-10-11T15:25:37.169 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.169 DEBUG:boto:Data: 2023-10-11T15:25:37.170 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.171 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.171 DEBUG:boto:Params: {} 2023-10-11T15:25:37.173 DEBUG:boto:Token: None 2023-10-11T15:25:37.174 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU= 2023-10-11T15:25:37.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU=', 'Content-Length': '0'} 2023-10-11T15:25:37.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9549d4b1ef689bd-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.178 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.178 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.179 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.179 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.179 DEBUG:boto:Method: GET 2023-10-11T15:25:37.180 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.180 DEBUG:boto:Data: 2023-10-11T15:25:37.181 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.181 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.182 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.182 DEBUG:boto:Params: {} 2023-10-11T15:25:37.183 DEBUG:boto:Token: None 2023-10-11T15:25:37.183 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:25:37.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU= 2023-10-11T15:25:37.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6cpL+JjKcZ2hTeiztyQHfb0ENIU=', 'Content-Length': '0'} 2023-10-11T15:25:37.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218791049065de7d-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.187 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.188 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.188 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.189 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.189 DEBUG:boto:Method: GET 2023-10-11T15:25:37.190 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.190 DEBUG:boto:Data: 2023-10-11T15:25:37.191 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.191 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.191 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.192 DEBUG:boto:Params: {} 2023-10-11T15:25:37.192 DEBUG:boto:Token: None 2023-10-11T15:25:37.193 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.194 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b8293ab8558672-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.197 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.198 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.199 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.199 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.200 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.200 DEBUG:boto:Method: GET 2023-10-11T15:25:37.200 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.201 DEBUG:boto:Data: 2023-10-11T15:25:37.201 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.202 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.203 DEBUG:boto:Params: {} 2023-10-11T15:25:37.203 DEBUG:boto:Token: None 2023-10-11T15:25:37.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.204 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c11ee839a3c7e3b3-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.209 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.210 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.210 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.210 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.211 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:25:37.211 DEBUG:boto:Data: 2023-10-11T15:25:37.212 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.212 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.212 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.213 DEBUG:boto:Params: {} 2023-10-11T15:25:37.213 DEBUG:boto:Token: None 2023-10-11T15:25:37.214 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87 2023-10-11T15:25:37.214 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nFbZ3b/giWZJG2qv9CxfncvrJK4= 2023-10-11T15:25:37.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nFbZ3b/giWZJG2qv9CxfncvrJK4=', 'Content-Length': '0'} 2023-10-11T15:25:37.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005579ec4a9929cb4e-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.217 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.218 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.218 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.219 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:25:37.219 DEBUG:boto:Data: 2023-10-11T15:25:37.220 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.220 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.221 DEBUG:boto:Params: {} 2023-10-11T15:25:37.221 DEBUG:boto:Token: None 2023-10-11T15:25:37.222 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87 2023-10-11T15:25:37.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nFbZ3b/giWZJG2qv9CxfncvrJK4= 2023-10-11T15:25:37.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nFbZ3b/giWZJG2qv9CxfncvrJK4=', 'Content-Length': '0'} 2023-10-11T15:25:37.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001813eed728fbab24-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:25:37.226 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.226 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.227 DEBUG:boto:Method: GET 2023-10-11T15:25:37.227 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:25:37.227 DEBUG:boto:Data: 2023-10-11T15:25:37.228 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.229 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.229 DEBUG:boto:Params: {} 2023-10-11T15:25:37.229 DEBUG:boto:Token: None 2023-10-11T15:25:37.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87 2023-10-11T15:25:37.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:428Zcr8t/r/kzvQ5P9vO9NPJ0BQ= 2023-10-11T15:25:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:428Zcr8t/r/kzvQ5P9vO9NPJ0BQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096b9ddb3ae2f982e-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.234 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:25:37.234 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:25:37.235 DEBUG:boto:Method: GET 2023-10-11T15:25:37.235 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:25:37.235 DEBUG:boto:Data: 2023-10-11T15:25:37.236 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.237 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.238 DEBUG:boto:Params: {} 2023-10-11T15:25:37.239 DEBUG:boto:Token: None 2023-10-11T15:25:37.240 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj87 2023-10-11T15:25:37.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:428Zcr8t/r/kzvQ5P9vO9NPJ0BQ= 2023-10-11T15:25:37.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:428Zcr8t/r/kzvQ5P9vO9NPJ0BQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6a4cdfd2113685-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.286 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.288 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.289 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.289 DEBUG:boto:Method: GET 2023-10-11T15:25:37.290 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.291 DEBUG:boto:Data: 2023-10-11T15:25:37.292 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.293 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.293 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.294 DEBUG:boto:Params: {} 2023-10-11T15:25:37.295 DEBUG:boto:Token: None 2023-10-11T15:25:37.296 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbc2b7256a07863-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.300 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.301 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.302 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.302 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.303 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.303 DEBUG:boto:Method: GET 2023-10-11T15:25:37.304 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:25:37.304 DEBUG:boto:Data: 2023-10-11T15:25:37.305 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.306 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.306 DEBUG:boto:Params: {} 2023-10-11T15:25:37.307 DEBUG:boto:Token: None 2023-10-11T15:25:37.307 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d8d1edb819706f9-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.313 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:25:37.315 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.316 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.317 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.317 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.318 DEBUG:boto:Method: GET 2023-10-11T15:25:37.318 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.318 DEBUG:boto:Data: 2023-10-11T15:25:37.319 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.320 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.321 DEBUG:boto:Params: {} 2023-10-11T15:25:37.322 DEBUG:boto:Token: None 2023-10-11T15:25:37.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc= 2023-10-11T15:25:37.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc=', 'Content-Length': '0'} 2023-10-11T15:25:37.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53725581a10a119-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.327 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.328 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.328 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.329 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.329 DEBUG:boto:Method: GET 2023-10-11T15:25:37.330 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.330 DEBUG:boto:Data: 2023-10-11T15:25:37.331 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.332 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.332 DEBUG:boto:Params: {} 2023-10-11T15:25:37.333 DEBUG:boto:Token: None 2023-10-11T15:25:37.333 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc= 2023-10-11T15:25:37.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc=', 'Content-Length': '0'} 2023-10-11T15:25:37.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003243d6aebe785271-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.337 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.338 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.338 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.339 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.339 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.340 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.340 DEBUG:boto:Data: 2023-10-11T15:25:37.341 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.342 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.342 DEBUG:boto:Params: {} 2023-10-11T15:25:37.343 DEBUG:boto:Token: None 2023-10-11T15:25:37.344 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:giedx0Y7ArN21NvTBhFBVUE4qtI= 2023-10-11T15:25:37.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:giedx0Y7ArN21NvTBhFBVUE4qtI=', 'Content-Length': '0'} 2023-10-11T15:25:37.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006946c1bd23996bdc-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.350 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.350 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.351 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.352 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.352 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.353 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.354 DEBUG:boto:Data: 2023-10-11T15:25:37.355 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.356 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.357 DEBUG:boto:Params: {} 2023-10-11T15:25:37.358 DEBUG:boto:Token: None 2023-10-11T15:25:37.359 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:giedx0Y7ArN21NvTBhFBVUE4qtI= 2023-10-11T15:25:37.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:giedx0Y7ArN21NvTBhFBVUE4qtI=', 'Content-Length': '0'} 2023-10-11T15:25:37.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b729757f29214d4-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:25:37.364 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.365 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.366 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.366 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.367 DEBUG:boto:Method: GET 2023-10-11T15:25:37.368 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.368 DEBUG:boto:Data: 2023-10-11T15:25:37.368 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.369 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.369 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.370 DEBUG:boto:Params: {} 2023-10-11T15:25:37.370 DEBUG:boto:Token: None 2023-10-11T15:25:37.371 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc= 2023-10-11T15:25:37.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc=', 'Content-Length': '0'} 2023-10-11T15:25:37.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bdff65989cf5d63-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.418 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.419 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.419 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.420 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.421 DEBUG:boto:Method: GET 2023-10-11T15:25:37.421 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.422 DEBUG:boto:Data: 2023-10-11T15:25:37.423 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.424 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.425 DEBUG:boto:Params: {} 2023-10-11T15:25:37.426 DEBUG:boto:Token: None 2023-10-11T15:25:37.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:25:37.427 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc= 2023-10-11T15:25:37.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PcoUdJZ2Xgl8+bxifN3uSwzatRc=', 'Content-Length': '0'} 2023-10-11T15:25:37.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000249eb76d4dbe755d-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.431 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.432 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.432 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.433 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.433 DEBUG:boto:Method: GET 2023-10-11T15:25:37.434 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.434 DEBUG:boto:Data: 2023-10-11T15:25:37.435 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.435 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.436 DEBUG:boto:Params: {} 2023-10-11T15:25:37.436 DEBUG:boto:Token: None 2023-10-11T15:25:37.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c6cd6b70981bd5-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.443 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.444 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.444 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.444 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.445 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.445 DEBUG:boto:Method: GET 2023-10-11T15:25:37.446 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.446 DEBUG:boto:Data: 2023-10-11T15:25:37.447 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.447 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.448 DEBUG:boto:Params: {} 2023-10-11T15:25:37.448 DEBUG:boto:Token: None 2023-10-11T15:25:37.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c87692a8a2ffc8-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.494 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.495 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.496 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.496 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.497 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:25:37.497 DEBUG:boto:Data: 2023-10-11T15:25:37.497 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.499 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.499 DEBUG:boto:Params: {} 2023-10-11T15:25:37.500 DEBUG:boto:Token: None 2023-10-11T15:25:37.500 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92 2023-10-11T15:25:37.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U+d21TJmllF8x3G2tPmiPBd8cd0= 2023-10-11T15:25:37.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U+d21TJmllF8x3G2tPmiPBd8cd0=', 'Content-Length': '0'} 2023-10-11T15:25:37.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caaf4583473f8141-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.505 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.505 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.506 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.506 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:25:37.507 DEBUG:boto:Data: 2023-10-11T15:25:37.507 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.508 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.508 DEBUG:boto:Params: {} 2023-10-11T15:25:37.509 DEBUG:boto:Token: None 2023-10-11T15:25:37.510 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92 2023-10-11T15:25:37.510 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U+d21TJmllF8x3G2tPmiPBd8cd0= 2023-10-11T15:25:37.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U+d21TJmllF8x3G2tPmiPBd8cd0=', 'Content-Length': '0'} 2023-10-11T15:25:37.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38183f929fc5ce2-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:25:37.514 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.514 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.514 DEBUG:boto:Method: GET 2023-10-11T15:25:37.515 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:25:37.515 DEBUG:boto:Data: 2023-10-11T15:25:37.516 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.516 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.517 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.517 DEBUG:boto:Params: {} 2023-10-11T15:25:37.518 DEBUG:boto:Token: None 2023-10-11T15:25:37.518 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92 2023-10-11T15:25:37.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MlwLBNLO5A5oUP6X+A96llNevx0= 2023-10-11T15:25:37.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MlwLBNLO5A5oUP6X+A96llNevx0=', 'Content-Length': '0'} 2023-10-11T15:25:37.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f95bb876040465e-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.522 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:25:37.523 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:25:37.523 DEBUG:boto:Method: GET 2023-10-11T15:25:37.523 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:25:37.524 DEBUG:boto:Data: 2023-10-11T15:25:37.524 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.525 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.526 DEBUG:boto:Params: {} 2023-10-11T15:25:37.526 DEBUG:boto:Token: None 2023-10-11T15:25:37.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj92 2023-10-11T15:25:37.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MlwLBNLO5A5oUP6X+A96llNevx0= 2023-10-11T15:25:37.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MlwLBNLO5A5oUP6X+A96llNevx0=', 'Content-Length': '0'} 2023-10-11T15:25:37.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37de4500e3c588c-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.531 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.531 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.531 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.532 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.532 DEBUG:boto:Method: GET 2023-10-11T15:25:37.533 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.533 DEBUG:boto:Data: 2023-10-11T15:25:37.534 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.534 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.535 DEBUG:boto:Params: {} 2023-10-11T15:25:37.535 DEBUG:boto:Token: None 2023-10-11T15:25:37.536 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060d64ac3a6355295-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.540 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.540 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.541 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.541 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.542 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.542 DEBUG:boto:Method: GET 2023-10-11T15:25:37.543 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:25:37.543 DEBUG:boto:Data: 2023-10-11T15:25:37.543 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.544 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.545 DEBUG:boto:Params: {} 2023-10-11T15:25:37.545 DEBUG:boto:Token: None 2023-10-11T15:25:37.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f48dbc2d384b11b-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.551 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:25:37.552 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.552 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.553 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.553 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.554 DEBUG:boto:Method: GET 2023-10-11T15:25:37.554 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.555 DEBUG:boto:Data: 2023-10-11T15:25:37.555 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.556 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.556 DEBUG:boto:Params: {} 2023-10-11T15:25:37.557 DEBUG:boto:Token: None 2023-10-11T15:25:37.557 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.558 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898= 2023-10-11T15:25:37.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898=', 'Content-Length': '0'} 2023-10-11T15:25:37.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d669abca0efbb2-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.561 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.561 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.562 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.562 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.563 DEBUG:boto:Method: GET 2023-10-11T15:25:37.563 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.563 DEBUG:boto:Data: 2023-10-11T15:25:37.565 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.566 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.567 DEBUG:boto:Params: {} 2023-10-11T15:25:37.567 DEBUG:boto:Token: None 2023-10-11T15:25:37.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898= 2023-10-11T15:25:37.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898=', 'Content-Length': '0'} 2023-10-11T15:25:37.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bed08e8be13da96-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.571 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.572 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.572 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.573 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.573 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.573 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.574 DEBUG:boto:Data: 2023-10-11T15:25:37.574 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.575 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.575 DEBUG:boto:Params: {} 2023-10-11T15:25:37.576 DEBUG:boto:Token: None 2023-10-11T15:25:37.576 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQ0zhcHBCp4KcWwXJw7B1OrCYM0= 2023-10-11T15:25:37.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQ0zhcHBCp4KcWwXJw7B1OrCYM0=', 'Content-Length': '0'} 2023-10-11T15:25:37.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000529697baef2f724-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.580 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.580 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.580 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.581 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.581 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.582 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.582 DEBUG:boto:Data: 2023-10-11T15:25:37.582 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.583 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.584 DEBUG:boto:Params: {} 2023-10-11T15:25:37.584 DEBUG:boto:Token: None 2023-10-11T15:25:37.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQ0zhcHBCp4KcWwXJw7B1OrCYM0= 2023-10-11T15:25:37.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQ0zhcHBCp4KcWwXJw7B1OrCYM0=', 'Content-Length': '0'} 2023-10-11T15:25:37.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41dd7f656275d4a-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:25:37.589 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.590 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.590 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.591 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.591 DEBUG:boto:Method: GET 2023-10-11T15:25:37.592 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.592 DEBUG:boto:Data: 2023-10-11T15:25:37.592 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.593 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.594 DEBUG:boto:Params: {} 2023-10-11T15:25:37.594 DEBUG:boto:Token: None 2023-10-11T15:25:37.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898= 2023-10-11T15:25:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898=', 'Content-Length': '0'} 2023-10-11T15:25:37.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eafb3fcc7e6ec72-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.638 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.639 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.639 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.640 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.640 DEBUG:boto:Method: GET 2023-10-11T15:25:37.641 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.641 DEBUG:boto:Data: 2023-10-11T15:25:37.642 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.643 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.643 DEBUG:boto:Params: {} 2023-10-11T15:25:37.644 DEBUG:boto:Token: None 2023-10-11T15:25:37.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:25:37.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898= 2023-10-11T15:25:37.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cBBaSz3huJFPSkltCkuBUIyT898=', 'Content-Length': '0'} 2023-10-11T15:25:37.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b831e5d4c828e5-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.648 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.648 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.649 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.649 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.649 DEBUG:boto:Method: GET 2023-10-11T15:25:37.650 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.650 DEBUG:boto:Data: 2023-10-11T15:25:37.651 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.652 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.652 DEBUG:boto:Params: {} 2023-10-11T15:25:37.653 DEBUG:boto:Token: None 2023-10-11T15:25:37.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd59c81c2d78ef2-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.657 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.658 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.658 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.659 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.659 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.660 DEBUG:boto:Method: GET 2023-10-11T15:25:37.660 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.660 DEBUG:boto:Data: 2023-10-11T15:25:37.661 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.662 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.662 DEBUG:boto:Params: {} 2023-10-11T15:25:37.663 DEBUG:boto:Token: None 2023-10-11T15:25:37.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e5dbf88c037c67-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.669 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.669 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.670 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.671 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.671 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:25:37.671 DEBUG:boto:Data: 2023-10-11T15:25:37.672 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.673 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.673 DEBUG:boto:Params: {} 2023-10-11T15:25:37.673 DEBUG:boto:Token: None 2023-10-11T15:25:37.674 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93 2023-10-11T15:25:37.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kd6wKelNbOlqQhKBWouYcK58TwI= 2023-10-11T15:25:37.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kd6wKelNbOlqQhKBWouYcK58TwI=', 'Content-Length': '0'} 2023-10-11T15:25:37.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084530a81be63a134-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.678 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.678 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.678 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.679 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:25:37.679 DEBUG:boto:Data: 2023-10-11T15:25:37.680 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.680 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.681 DEBUG:boto:Params: {} 2023-10-11T15:25:37.681 DEBUG:boto:Token: None 2023-10-11T15:25:37.682 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93 2023-10-11T15:25:37.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kd6wKelNbOlqQhKBWouYcK58TwI= 2023-10-11T15:25:37.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kd6wKelNbOlqQhKBWouYcK58TwI=', 'Content-Length': '0'} 2023-10-11T15:25:37.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029db7103644423be-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:25:37.687 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.687 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.688 DEBUG:boto:Method: GET 2023-10-11T15:25:37.689 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:25:37.689 DEBUG:boto:Data: 2023-10-11T15:25:37.690 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.691 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.691 DEBUG:boto:Params: {} 2023-10-11T15:25:37.692 DEBUG:boto:Token: None 2023-10-11T15:25:37.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93 2023-10-11T15:25:37.693 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXtXjt/4PteQNeuESjIVXhMTnYw= 2023-10-11T15:25:37.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXtXjt/4PteQNeuESjIVXhMTnYw=', 'Content-Length': '0'} 2023-10-11T15:25:37.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006214e5318b5441a8-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.696 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:25:37.697 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:25:37.697 DEBUG:boto:Method: GET 2023-10-11T15:25:37.697 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:25:37.698 DEBUG:boto:Data: 2023-10-11T15:25:37.698 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.699 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.700 DEBUG:boto:Params: {} 2023-10-11T15:25:37.700 DEBUG:boto:Token: None 2023-10-11T15:25:37.701 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj93 2023-10-11T15:25:37.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXtXjt/4PteQNeuESjIVXhMTnYw= 2023-10-11T15:25:37.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXtXjt/4PteQNeuESjIVXhMTnYw=', 'Content-Length': '0'} 2023-10-11T15:25:37.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d292a3f7e90e03-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.705 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.705 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.705 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.706 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.706 DEBUG:boto:Method: GET 2023-10-11T15:25:37.707 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.707 DEBUG:boto:Data: 2023-10-11T15:25:37.708 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.708 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.709 DEBUG:boto:Params: {} 2023-10-11T15:25:37.709 DEBUG:boto:Token: None 2023-10-11T15:25:37.710 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4a1803c3b7c96e-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.714 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.715 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.715 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.715 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.716 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.716 DEBUG:boto:Method: GET 2023-10-11T15:25:37.717 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:25:37.717 DEBUG:boto:Data: 2023-10-11T15:25:37.717 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.718 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.719 DEBUG:boto:Params: {} 2023-10-11T15:25:37.719 DEBUG:boto:Token: None 2023-10-11T15:25:37.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2cb187f9c6b67e-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.725 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:25:37.726 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.726 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.727 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.727 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.728 DEBUG:boto:Method: GET 2023-10-11T15:25:37.728 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.729 DEBUG:boto:Data: 2023-10-11T15:25:37.729 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.730 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.731 DEBUG:boto:Params: {} 2023-10-11T15:25:37.731 DEBUG:boto:Token: None 2023-10-11T15:25:37.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y= 2023-10-11T15:25:37.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y=', 'Content-Length': '0'} 2023-10-11T15:25:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a0f4dcf9d3eacfa-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.736 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.736 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.737 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.737 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.738 DEBUG:boto:Method: GET 2023-10-11T15:25:37.738 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.738 DEBUG:boto:Data: 2023-10-11T15:25:37.739 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.740 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.740 DEBUG:boto:Params: {} 2023-10-11T15:25:37.741 DEBUG:boto:Token: None 2023-10-11T15:25:37.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y= 2023-10-11T15:25:37.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y=', 'Content-Length': '0'} 2023-10-11T15:25:37.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7f41d5717ca970-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.786 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.787 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.787 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.788 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.788 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.789 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.789 DEBUG:boto:Data: 2023-10-11T15:25:37.790 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.790 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.791 DEBUG:boto:Params: {} 2023-10-11T15:25:37.791 DEBUG:boto:Token: None 2023-10-11T15:25:37.792 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Dx7jUm9HZb/vUA4uVbeEKZBJNI= 2023-10-11T15:25:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Dx7jUm9HZb/vUA4uVbeEKZBJNI=', 'Content-Length': '0'} 2023-10-11T15:25:37.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000001ba195abac7779-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.796 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.797 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.797 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.797 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.798 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.798 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.799 DEBUG:boto:Data: 2023-10-11T15:25:37.799 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.800 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.800 DEBUG:boto:Params: {} 2023-10-11T15:25:37.801 DEBUG:boto:Token: None 2023-10-11T15:25:37.801 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2Dx7jUm9HZb/vUA4uVbeEKZBJNI= 2023-10-11T15:25:37.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2Dx7jUm9HZb/vUA4uVbeEKZBJNI=', 'Content-Length': '0'} 2023-10-11T15:25:37.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005494a065010b698c-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:25:37.805 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.806 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.806 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.807 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.807 DEBUG:boto:Method: GET 2023-10-11T15:25:37.807 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.808 DEBUG:boto:Data: 2023-10-11T15:25:37.808 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.809 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.809 DEBUG:boto:Params: {} 2023-10-11T15:25:37.810 DEBUG:boto:Token: None 2023-10-11T15:25:37.810 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.811 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y= 2023-10-11T15:25:37.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y=', 'Content-Length': '0'} 2023-10-11T15:25:37.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5dccba56e0b45a9-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.814 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.814 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.815 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.815 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.816 DEBUG:boto:Method: GET 2023-10-11T15:25:37.816 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.817 DEBUG:boto:Data: 2023-10-11T15:25:37.817 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.818 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.818 DEBUG:boto:Params: {} 2023-10-11T15:25:37.819 DEBUG:boto:Token: None 2023-10-11T15:25:37.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:25:37.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y= 2023-10-11T15:25:37.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:faZSuYxTrrcceD8I3Cm3tZUmZ5Y=', 'Content-Length': '0'} 2023-10-11T15:25:37.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004151dbb24516e10c-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.823 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.823 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.823 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.824 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.824 DEBUG:boto:Method: GET 2023-10-11T15:25:37.825 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.825 DEBUG:boto:Data: 2023-10-11T15:25:37.826 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.826 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.827 DEBUG:boto:Params: {} 2023-10-11T15:25:37.827 DEBUG:boto:Token: None 2023-10-11T15:25:37.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.828 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000029abb336e857a0f-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.832 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.833 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.833 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.834 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.834 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.835 DEBUG:boto:Method: GET 2023-10-11T15:25:37.835 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.836 DEBUG:boto:Data: 2023-10-11T15:25:37.836 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.837 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.838 DEBUG:boto:Params: {} 2023-10-11T15:25:37.838 DEBUG:boto:Token: None 2023-10-11T15:25:37.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00b5798911e1387-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.844 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.844 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.845 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.845 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.846 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:25:37.846 DEBUG:boto:Data: 2023-10-11T15:25:37.846 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.847 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.847 DEBUG:boto:Params: {} 2023-10-11T15:25:37.848 DEBUG:boto:Token: None 2023-10-11T15:25:37.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95 2023-10-11T15:25:37.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b+LZ8jeCMTJ0kEGNEB2CHmHYbuQ= 2023-10-11T15:25:37.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b+LZ8jeCMTJ0kEGNEB2CHmHYbuQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071225b4cc8a8878c-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.852 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.852 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.853 DEBUG:boto:Method: HEAD 2023-10-11T15:25:37.853 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:25:37.854 DEBUG:boto:Data: 2023-10-11T15:25:37.854 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.855 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.855 DEBUG:boto:Params: {} 2023-10-11T15:25:37.856 DEBUG:boto:Token: None 2023-10-11T15:25:37.856 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95 2023-10-11T15:25:37.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b+LZ8jeCMTJ0kEGNEB2CHmHYbuQ= 2023-10-11T15:25:37.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b+LZ8jeCMTJ0kEGNEB2CHmHYbuQ=', 'Content-Length': '0'} 2023-10-11T15:25:37.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2b040a8380c6338-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:25:37.860 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.860 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.860 DEBUG:boto:Method: GET 2023-10-11T15:25:37.861 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:25:37.861 DEBUG:boto:Data: 2023-10-11T15:25:37.862 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.863 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.863 DEBUG:boto:Params: {} 2023-10-11T15:25:37.864 DEBUG:boto:Token: None 2023-10-11T15:25:37.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95 2023-10-11T15:25:37.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOnzTDR/eTeakG2Mn0Ln+G/VBN0= 2023-10-11T15:25:37.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOnzTDR/eTeakG2Mn0Ln+G/VBN0=', 'Content-Length': '0'} 2023-10-11T15:25:37.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e01f0164a6ca316-006526be71-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.914 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:25:37.915 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:25:37.915 DEBUG:boto:Method: GET 2023-10-11T15:25:37.916 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:25:37.916 DEBUG:boto:Data: 2023-10-11T15:25:37.917 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.918 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.918 DEBUG:boto:Params: {} 2023-10-11T15:25:37.919 DEBUG:boto:Token: None 2023-10-11T15:25:37.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/obj95 2023-10-11T15:25:37.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GOnzTDR/eTeakG2Mn0Ln+G/VBN0= 2023-10-11T15:25:37.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GOnzTDR/eTeakG2Mn0Ln+G/VBN0=', 'Content-Length': '0'} 2023-10-11T15:25:37.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e918bf950b34614d-006526be71-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.923 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.924 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.924 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.924 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.925 DEBUG:boto:Method: GET 2023-10-11T15:25:37.925 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.926 DEBUG:boto:Data: 2023-10-11T15:25:37.926 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.927 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:25:37.927 DEBUG:boto:Port: 8001 2023-10-11T15:25:37.927 DEBUG:boto:Params: {} 2023-10-11T15:25:37.928 DEBUG:boto:Token: None 2023-10-11T15:25:37.928 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.929 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731417d18c00d557-006526be71-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.933 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.933 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:37.934 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:37.934 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.935 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.935 DEBUG:boto:Method: GET 2023-10-11T15:25:37.936 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:25:37.936 DEBUG:boto:Data: 2023-10-11T15:25:37.936 DEBUG:boto:Headers: {} 2023-10-11T15:25:37.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:37.937 DEBUG:boto:Port: 8000 2023-10-11T15:25:37.938 DEBUG:boto:Params: {} 2023-10-11T15:25:37.938 DEBUG:boto:Token: None 2023-10-11T15:25:37.938 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:25:37 GMT /dghqvd-59/ 2023-10-11T15:25:37.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk= 2023-10-11T15:25:37.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vZ9ErD1VzwP1cF45IPGcmpwPZEk=', 'Content-Length': '0'} 2023-10-11T15:25:37.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5962844f42a34e-006526be71-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:25:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:37.990 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:25:37.993 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:25:37.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:25:37.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:25:37.994 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:25:38.054 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:25:38.055 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:25:38.108 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:25:38.109 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-59 2023-10-11T15:25:38.121 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 30 2023-10-11T15:25:38.183 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:38.183 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:38.184 DEBUG:boto:Method: HEAD 2023-10-11T15:25:38.184 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:38.185 DEBUG:boto:Data: 2023-10-11T15:25:38.185 DEBUG:boto:Headers: {} 2023-10-11T15:25:38.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.186 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.186 DEBUG:boto:Params: {} 2023-10-11T15:25:38.187 DEBUG:boto:Token: None 2023-10-11T15:25:38.187 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/ 2023-10-11T15:25:38.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA= 2023-10-11T15:25:38.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA=', 'Content-Length': '0'} 2023-10-11T15:25:38.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000308583ec591adcee-006526be72-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.193 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:25:38.193 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:25:38.194 DEBUG:boto:Method: PUT 2023-10-11T15:25:38.194 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:25:38.195 DEBUG:boto:Data: 2023-10-11T15:25:38.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:25:38.196 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.198 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.199 DEBUG:boto:Params: {} 2023-10-11T15:25:38.200 DEBUG:boto:Token: None 2023-10-11T15:25:38.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/obj37 2023-10-11T15:25:38.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BLTGkPmy01j47tvnjDpJrAzuqQ8= 2023-10-11T15:25:38.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLTGkPmy01j47tvnjDpJrAzuqQ8='} 2023-10-11T15:25:38.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001200c25517a9ffaf-006526be72-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.208 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:38.208 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:38.209 DEBUG:boto:Method: HEAD 2023-10-11T15:25:38.209 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:38.209 DEBUG:boto:Data: 2023-10-11T15:25:38.210 DEBUG:boto:Headers: {} 2023-10-11T15:25:38.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.211 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.211 DEBUG:boto:Params: {} 2023-10-11T15:25:38.211 DEBUG:boto:Token: None 2023-10-11T15:25:38.212 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/ 2023-10-11T15:25:38.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA= 2023-10-11T15:25:38.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA=', 'Content-Length': '0'} 2023-10-11T15:25:38.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088880610126e4836-006526be72-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.217 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:25:38.217 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:25:38.218 DEBUG:boto:Method: PUT 2023-10-11T15:25:38.218 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:25:38.219 DEBUG:boto:Data: 2023-10-11T15:25:38.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:25:38.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.220 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.220 DEBUG:boto:Params: {} 2023-10-11T15:25:38.221 DEBUG:boto:Token: None 2023-10-11T15:25:38.221 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/obj74 2023-10-11T15:25:38.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q/nQCxEvak5Y4cca4LFw3+vj1Ro= 2023-10-11T15:25:38.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q/nQCxEvak5Y4cca4LFw3+vj1Ro='} 2023-10-11T15:25:38.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c753de7e091c275e-006526be72-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.267 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:38.267 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:38.268 DEBUG:boto:Method: HEAD 2023-10-11T15:25:38.268 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:38.269 DEBUG:boto:Data: 2023-10-11T15:25:38.269 DEBUG:boto:Headers: {} 2023-10-11T15:25:38.270 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.270 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.270 DEBUG:boto:Params: {} 2023-10-11T15:25:38.271 DEBUG:boto:Token: None 2023-10-11T15:25:38.271 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/ 2023-10-11T15:25:38.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA= 2023-10-11T15:25:38.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA=', 'Content-Length': '0'} 2023-10-11T15:25:38.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001abb9ecfbc80b532-006526be72-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.276 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:25:38.276 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:25:38.277 DEBUG:boto:Method: PUT 2023-10-11T15:25:38.277 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:25:38.278 DEBUG:boto:Data: 2023-10-11T15:25:38.278 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:25:38.279 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.279 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.279 DEBUG:boto:Params: {} 2023-10-11T15:25:38.280 DEBUG:boto:Token: None 2023-10-11T15:25:38.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/obj111 2023-10-11T15:25:38.281 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYoEyllFx5kNDi0BPD/e/ATRhsA= 2023-10-11T15:25:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYoEyllFx5kNDi0BPD/e/ATRhsA='} 2023-10-11T15:25:38.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043d5c5cd0004606f-006526be72-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.287 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:38.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:38.288 DEBUG:boto:Method: HEAD 2023-10-11T15:25:38.288 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:38.289 DEBUG:boto:Data: 2023-10-11T15:25:38.289 DEBUG:boto:Headers: {} 2023-10-11T15:25:38.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.290 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.290 DEBUG:boto:Params: {} 2023-10-11T15:25:38.291 DEBUG:boto:Token: None 2023-10-11T15:25:38.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/ 2023-10-11T15:25:38.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA= 2023-10-11T15:25:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA=', 'Content-Length': '0'} 2023-10-11T15:25:38.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080ec73f0c3afe4cd-006526be72-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.296 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:25:38.296 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:25:38.297 DEBUG:boto:Method: PUT 2023-10-11T15:25:38.297 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:25:38.298 DEBUG:boto:Data: 2023-10-11T15:25:38.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:25:38.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.299 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.299 DEBUG:boto:Params: {} 2023-10-11T15:25:38.300 DEBUG:boto:Token: None 2023-10-11T15:25:38.300 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/obj148 2023-10-11T15:25:38.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uwCgGkAzgt29dYoqVyKIOemt0rI= 2023-10-11T15:25:38.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uwCgGkAzgt29dYoqVyKIOemt0rI='} 2023-10-11T15:25:38.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dc4e9243e3b2d5d-006526be72-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.347 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:25:38.348 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:25:38.348 DEBUG:boto:Method: HEAD 2023-10-11T15:25:38.348 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:25:38.349 DEBUG:boto:Data: 2023-10-11T15:25:38.349 DEBUG:boto:Headers: {} 2023-10-11T15:25:38.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.350 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.350 DEBUG:boto:Params: {} 2023-10-11T15:25:38.351 DEBUG:boto:Token: None 2023-10-11T15:25:38.351 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/ 2023-10-11T15:25:38.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA= 2023-10-11T15:25:38.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VgQqRweLE0qZoCuBuzqUa5112tA=', 'Content-Length': '0'} 2023-10-11T15:25:38.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd9cb0477d825ad9-006526be72-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.356 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:25:38.357 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:25:38.357 DEBUG:boto:Method: PUT 2023-10-11T15:25:38.357 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:25:38.358 DEBUG:boto:Data: 2023-10-11T15:25:38.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:25:38.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:25:38.359 DEBUG:boto:Port: 8000 2023-10-11T15:25:38.359 DEBUG:boto:Params: {} 2023-10-11T15:25:38.360 DEBUG:boto:Token: None 2023-10-11T15:25:38.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:25:38 GMT /dghqvd-59/obj185 2023-10-11T15:25:38.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NTO47bxCnR01FqelaSAjBbtFZQ4= 2023-10-11T15:25:38.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:25:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NTO47bxCnR01FqelaSAjBbtFZQ4='} 2023-10-11T15:25:38.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002948485f6bee77bf-006526be72-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:25:38.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:25:38.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:25:38.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:25:38.368 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:25:38.424 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:25:38.425 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:25:38.461 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:38.460+0000 7fa8e2229780 1 waiting to reach latest gen 6, on 5.. 2023-10-11T15:25:43.464 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:43.463+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:43.465 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:43.465 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:48.468 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:48.466+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:48.468 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:48.468 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:53.470 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:53.468+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:53.470 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:53.471 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:25:58.471+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:25:58.473 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:25:58.473 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:03.475 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:03.472+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:03.475 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:03.476 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:08.478 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:08.475+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:08.478 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:08.478 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:13.480 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:13.478+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:13.481 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:13.481 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:18.483 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:18.480+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:18.483 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:18.483 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:23.485 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:23.483+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:23.486 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:23.486 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:28.488 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:28.485+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:28.488 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:28.489 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:33.490 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:33.488+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:33.491 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:33.491 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:38.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:38.491+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:38.494 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:38.494 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:43.494+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:43.496 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:43.497 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:48.499 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:48.497+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:48.499 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:48.499 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:53.501 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:53.500+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:53.502 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:53.502 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:58.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:26:58.502+0000 7fa8e2229780 1 waiting for incremental sync to catch up: 2023-10-11T15:26:58.504 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , , , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:26:58.505 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:27:03.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:03.505+0000 7fa8e2229780 1 bucket sync caught up with source: 2023-10-11T15:27:03.507 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:27:03.507 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:27:03.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:03.505+0000 7fa8e2229780 0 bucket checkpoint complete 2023-10-11T15:27:03.524 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:27:03.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:03.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:03.526 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:27:03.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:03.589 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:03.627 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:03.625+0000 7f91441b9780 1 waiting for incremental sync to catch up: 2023-10-11T15:27:03.627 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000006.179.6, 00000000006.308.6] 2023-10-11T15:27:03.627 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000007.193.6, 00000000006.308.6] 2023-10-11T15:27:08.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:08.626+0000 7f91441b9780 1 waiting for incremental sync to catch up: 2023-10-11T15:27:08.628 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000006.179.6, 00000000006.308.6] 2023-10-11T15:27:08.629 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000007.193.6, 00000000006.308.6] 2023-10-11T15:27:13.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:13.628+0000 7f91441b9780 1 bucket sync caught up with source: 2023-10-11T15:27:13.629 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000007.193.6, 00000000006.308.6] 2023-10-11T15:27:13.630 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.225.6, 00000000002.234.6, 00000000002.302.6, 00000000005.120.6, 00000000004.296.6, 00000000001.150.6, 00000000001.132.6, 00000000004.264.6, 00000000007.193.6, 00000000006.308.6] 2023-10-11T15:27:13.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:13.628+0000 7f91441b9780 0 bucket checkpoint complete 2023-10-11T15:27:13.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:27:13.647 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.647 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.648 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.648 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:27:13.648 DEBUG:boto:Data: 2023-10-11T15:27:13.649 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.649 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.650 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.650 DEBUG:boto:Params: {} 2023-10-11T15:27:13.651 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:27:13.651 DEBUG:boto:Token: None 2023-10-11T15:27:13.652 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O0KOYWVnNwlAbq+lFNi/Rm3SnKY= 2023-10-11T15:27:13.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O0KOYWVnNwlAbq+lFNi/Rm3SnKY=', 'Content-Length': '0'} 2023-10-11T15:27:13.657 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba34c454d3bb0bc2-006526bed1-1b7d-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.658 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.658 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.659 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.659 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:27:13.659 DEBUG:boto:Data: 2023-10-11T15:27:13.660 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.660 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.661 DEBUG:boto:Params: {} 2023-10-11T15:27:13.661 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:27:13.662 DEBUG:boto:Token: None 2023-10-11T15:27:13.662 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O0KOYWVnNwlAbq+lFNi/Rm3SnKY= 2023-10-11T15:27:13.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O0KOYWVnNwlAbq+lFNi/Rm3SnKY=', 'Content-Length': '0'} 2023-10-11T15:27:13.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000569c21daee77c3a-006526bed1-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.669 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:27:13.669 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.670 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.670 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.671 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.671 DEBUG:boto:Method: GET 2023-10-11T15:27:13.671 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.673 DEBUG:boto:Data: 2023-10-11T15:27:13.674 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.674 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.675 DEBUG:boto:Params: {} 2023-10-11T15:27:13.675 DEBUG:boto:Token: None 2023-10-11T15:27:13.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/?versions 2023-10-11T15:27:13.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U= 2023-10-11T15:27:13.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U=', 'Content-Length': '0'} 2023-10-11T15:27:13.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb760f4c99f7a0bc-006526bed1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.681 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:27:13.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:27:13.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:27:13.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:27:13.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:27:13.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:27:13.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:27:13.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:27:13.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:27:13.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:27:13.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:27:13.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:27:13.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:27:13.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:27:13.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:27:13.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:27:13.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:27:13.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:27:13.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:27:13.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:27:13.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:27:13.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:27:13.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:27:13.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:27:13.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:27:13.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:27:13.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:27:13.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:27:13.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:27:13.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:27:13.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:27:13.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:27:13.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:27:13.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:27:13.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:27:13.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:27:13.696 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.696 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.697 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.697 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.698 DEBUG:boto:Method: GET 2023-10-11T15:27:13.698 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.699 DEBUG:boto:Data: 2023-10-11T15:27:13.699 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.700 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.700 DEBUG:boto:Params: {} 2023-10-11T15:27:13.700 DEBUG:boto:Token: None 2023-10-11T15:27:13.701 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/?versions 2023-10-11T15:27:13.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U= 2023-10-11T15:27:13.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U=', 'Content-Length': '0'} 2023-10-11T15:27:13.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084fbef795c6077ec-006526bed1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.707 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:27:13.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:27:13.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:27:13.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:27:13.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:27:13.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:27:13.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:27:13.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:27:13.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:27:13.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:27:13.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:27:13.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:27:13.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:27:13.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:27:13.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:27:13.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:27:13.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:27:13.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:27:13.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:27:13.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:27:13.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:27:13.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:27:13.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:27:13.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:27:13.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:27:13.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:27:13.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:27:13.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:27:13.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:27:13.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:27:13.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:27:13.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:27:13.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:27:13.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:27:13.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:27:13.721 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.722 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.722 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.723 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.723 DEBUG:boto:Method: GET 2023-10-11T15:27:13.723 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.724 DEBUG:boto:Data: 2023-10-11T15:27:13.724 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.725 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.725 DEBUG:boto:Params: {} 2023-10-11T15:27:13.726 DEBUG:boto:Token: None 2023-10-11T15:27:13.726 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/?versions 2023-10-11T15:27:13.727 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U= 2023-10-11T15:27:13.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U=', 'Content-Length': '0'} 2023-10-11T15:27:13.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b74546f66d596fee-006526bed1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.731 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.733 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.733 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.734 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.734 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.734 DEBUG:boto:Method: GET 2023-10-11T15:27:13.735 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:13.735 DEBUG:boto:Data: 2023-10-11T15:27:13.736 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.736 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.737 DEBUG:boto:Params: {} 2023-10-11T15:27:13.737 DEBUG:boto:Token: None 2023-10-11T15:27:13.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/?versions 2023-10-11T15:27:13.738 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U= 2023-10-11T15:27:13.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:maqx6nPfMq1tIfl72hW+ZUu1Q6U=', 'Content-Length': '0'} 2023-10-11T15:27:13.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7171715bd80bdf2-006526bed1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.744 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:13.747 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.747 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.747 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.748 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.748 DEBUG:boto:Method: GET 2023-10-11T15:27:13.749 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.749 DEBUG:boto:Data: 2023-10-11T15:27:13.749 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.750 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.751 DEBUG:boto:Params: {} 2023-10-11T15:27:13.751 DEBUG:boto:Token: None 2023-10-11T15:27:13.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk= 2023-10-11T15:27:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk=', 'Content-Length': '0'} 2023-10-11T15:27:13.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde7e6160fc8d5fd-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.756 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.756 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.756 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.757 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.757 DEBUG:boto:Method: GET 2023-10-11T15:27:13.758 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.758 DEBUG:boto:Data: 2023-10-11T15:27:13.759 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.759 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.760 DEBUG:boto:Params: {} 2023-10-11T15:27:13.760 DEBUG:boto:Token: None 2023-10-11T15:27:13.761 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk= 2023-10-11T15:27:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk=', 'Content-Length': '0'} 2023-10-11T15:27:13.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877c43604067f043-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.806 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.807 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.807 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.808 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.808 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.809 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.809 DEBUG:boto:Data: 2023-10-11T15:27:13.809 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.810 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.811 DEBUG:boto:Params: {} 2023-10-11T15:27:13.811 DEBUG:boto:Token: None 2023-10-11T15:27:13.812 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:duEauA04wnkl5+q9I5xAMfJ5Iog= 2023-10-11T15:27:13.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:duEauA04wnkl5+q9I5xAMfJ5Iog=', 'Content-Length': '0'} 2023-10-11T15:27:13.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e67963dcc00693-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.815 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.815 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.816 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.816 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.817 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.817 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.817 DEBUG:boto:Data: 2023-10-11T15:27:13.818 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.818 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.819 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.819 DEBUG:boto:Params: {} 2023-10-11T15:27:13.820 DEBUG:boto:Token: None 2023-10-11T15:27:13.820 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:duEauA04wnkl5+q9I5xAMfJ5Iog= 2023-10-11T15:27:13.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:duEauA04wnkl5+q9I5xAMfJ5Iog=', 'Content-Length': '0'} 2023-10-11T15:27:13.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e081e77ddf05db32-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:13.824 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.825 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.825 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.825 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.826 DEBUG:boto:Method: GET 2023-10-11T15:27:13.826 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.827 DEBUG:boto:Data: 2023-10-11T15:27:13.827 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.828 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.828 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.829 DEBUG:boto:Params: {} 2023-10-11T15:27:13.829 DEBUG:boto:Token: None 2023-10-11T15:27:13.830 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk= 2023-10-11T15:27:13.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk=', 'Content-Length': '0'} 2023-10-11T15:27:13.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b73428eda60867e8-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.833 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.834 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.834 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.835 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.835 DEBUG:boto:Method: GET 2023-10-11T15:27:13.835 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.836 DEBUG:boto:Data: 2023-10-11T15:27:13.836 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.837 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.837 DEBUG:boto:Params: {} 2023-10-11T15:27:13.838 DEBUG:boto:Token: None 2023-10-11T15:27:13.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:13.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk= 2023-10-11T15:27:13.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1UczleMrLW9d5QQ38/fwHlr5zk=', 'Content-Length': '0'} 2023-10-11T15:27:13.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6fcd869a7d84a10-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.842 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.842 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.843 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.843 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.844 DEBUG:boto:Method: GET 2023-10-11T15:27:13.844 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.844 DEBUG:boto:Data: 2023-10-11T15:27:13.845 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.846 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.846 DEBUG:boto:Params: {} 2023-10-11T15:27:13.846 DEBUG:boto:Token: None 2023-10-11T15:27:13.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk= 2023-10-11T15:27:13.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk=', 'Content-Length': '0'} 2023-10-11T15:27:13.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000430344498ae4a514-006526bed1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.851 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.852 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.852 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.853 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.853 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.854 DEBUG:boto:Method: GET 2023-10-11T15:27:13.854 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.854 DEBUG:boto:Data: 2023-10-11T15:27:13.855 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.856 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.856 DEBUG:boto:Params: {} 2023-10-11T15:27:13.857 DEBUG:boto:Token: None 2023-10-11T15:27:13.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk= 2023-10-11T15:27:13.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk=', 'Content-Length': '0'} 2023-10-11T15:27:13.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa425f31ade894a3-006526bed1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.863 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.864 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.864 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.865 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.865 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:13.865 DEBUG:boto:Data: 2023-10-11T15:27:13.866 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.867 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.867 DEBUG:boto:Params: {} 2023-10-11T15:27:13.867 DEBUG:boto:Token: None 2023-10-11T15:27:13.868 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11 2023-10-11T15:27:13.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JjSPdEajZDnUATeDZmqKke+YmCo= 2023-10-11T15:27:13.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JjSPdEajZDnUATeDZmqKke+YmCo=', 'Content-Length': '0'} 2023-10-11T15:27:13.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000937febef18b7e2ec-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.871 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.872 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.872 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.873 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:13.873 DEBUG:boto:Data: 2023-10-11T15:27:13.873 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.874 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.875 DEBUG:boto:Params: {} 2023-10-11T15:27:13.875 DEBUG:boto:Token: None 2023-10-11T15:27:13.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11 2023-10-11T15:27:13.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JjSPdEajZDnUATeDZmqKke+YmCo= 2023-10-11T15:27:13.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JjSPdEajZDnUATeDZmqKke+YmCo=', 'Content-Length': '0'} 2023-10-11T15:27:13.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a7f5a8a9438c4d-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:13.879 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.880 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.880 DEBUG:boto:Method: GET 2023-10-11T15:27:13.880 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:13.881 DEBUG:boto:Data: 2023-10-11T15:27:13.881 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.882 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.883 DEBUG:boto:Params: {} 2023-10-11T15:27:13.883 DEBUG:boto:Token: None 2023-10-11T15:27:13.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11 2023-10-11T15:27:13.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZdENKnZOKKexQWOU47kD6yPVNg= 2023-10-11T15:27:13.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZdENKnZOKKexQWOU47kD6yPVNg=', 'Content-Length': '0'} 2023-10-11T15:27:13.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008727f6e904e8f087-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.930 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:13.931 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:13.931 DEBUG:boto:Method: GET 2023-10-11T15:27:13.932 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:13.932 DEBUG:boto:Data: 2023-10-11T15:27:13.933 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.933 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.934 DEBUG:boto:Params: {} 2023-10-11T15:27:13.934 DEBUG:boto:Token: None 2023-10-11T15:27:13.935 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj11 2023-10-11T15:27:13.935 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7ZdENKnZOKKexQWOU47kD6yPVNg= 2023-10-11T15:27:13.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZdENKnZOKKexQWOU47kD6yPVNg=', 'Content-Length': '0'} 2023-10-11T15:27:13.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002492dc06b90b9630-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.939 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.939 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.940 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.940 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.940 DEBUG:boto:Method: GET 2023-10-11T15:27:13.941 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.941 DEBUG:boto:Data: 2023-10-11T15:27:13.942 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.943 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.943 DEBUG:boto:Params: {} 2023-10-11T15:27:13.944 DEBUG:boto:Token: None 2023-10-11T15:27:13.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk= 2023-10-11T15:27:13.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk=', 'Content-Length': '0'} 2023-10-11T15:27:13.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053b3a680a0233f18-006526bed1-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.948 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.949 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:13.949 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:13.950 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.950 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.950 DEBUG:boto:Method: GET 2023-10-11T15:27:13.951 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:13.951 DEBUG:boto:Data: 2023-10-11T15:27:13.951 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.952 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.952 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.953 DEBUG:boto:Params: {} 2023-10-11T15:27:13.953 DEBUG:boto:Token: None 2023-10-11T15:27:13.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/ 2023-10-11T15:27:13.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk= 2023-10-11T15:27:13.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QkeCn8x3ZwX8xGDMaFwuHs+DUxk=', 'Content-Length': '0'} 2023-10-11T15:27:13.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4a05521a1eb763c-006526bed1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.960 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:13.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:13.961 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:13.961 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:13.961 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.962 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.962 DEBUG:boto:Method: GET 2023-10-11T15:27:13.963 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.963 DEBUG:boto:Data: 2023-10-11T15:27:13.963 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.964 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.965 DEBUG:boto:Params: {} 2023-10-11T15:27:13.965 DEBUG:boto:Token: None 2023-10-11T15:27:13.966 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4= 2023-10-11T15:27:13.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4=', 'Content-Length': '0'} 2023-10-11T15:27:13.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c87e7efaaf9eed3-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.969 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:13.970 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:13.970 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.971 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.971 DEBUG:boto:Method: GET 2023-10-11T15:27:13.972 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.972 DEBUG:boto:Data: 2023-10-11T15:27:13.973 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.973 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.974 DEBUG:boto:Params: {} 2023-10-11T15:27:13.974 DEBUG:boto:Token: None 2023-10-11T15:27:13.975 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4= 2023-10-11T15:27:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4=', 'Content-Length': '0'} 2023-10-11T15:27:13.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062cd968b437852c4-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.978 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:13.979 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:13.979 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.979 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.980 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.980 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.981 DEBUG:boto:Data: 2023-10-11T15:27:13.981 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.982 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.982 DEBUG:boto:Params: {} 2023-10-11T15:27:13.983 DEBUG:boto:Token: None 2023-10-11T15:27:13.983 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dPwlFtrhZvuOq93rUqCxtisNFwU= 2023-10-11T15:27:13.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dPwlFtrhZvuOq93rUqCxtisNFwU=', 'Content-Length': '0'} 2023-10-11T15:27:13.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f466a92b2ecf00-006526bed1-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.986 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:13.987 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:13.987 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.987 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.988 DEBUG:boto:Method: HEAD 2023-10-11T15:27:13.988 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.989 DEBUG:boto:Data: 2023-10-11T15:27:13.989 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:13.990 DEBUG:boto:Port: 8000 2023-10-11T15:27:13.990 DEBUG:boto:Params: {} 2023-10-11T15:27:13.991 DEBUG:boto:Token: None 2023-10-11T15:27:13.991 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dPwlFtrhZvuOq93rUqCxtisNFwU= 2023-10-11T15:27:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dPwlFtrhZvuOq93rUqCxtisNFwU=', 'Content-Length': '0'} 2023-10-11T15:27:13.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269478f620d183fa-006526bed1-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:13.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:13.994 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:13.995 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:13.995 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.996 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.996 DEBUG:boto:Method: GET 2023-10-11T15:27:13.997 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:13.997 DEBUG:boto:Data: 2023-10-11T15:27:13.997 DEBUG:boto:Headers: {} 2023-10-11T15:27:13.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:13.998 DEBUG:boto:Port: 8001 2023-10-11T15:27:13.999 DEBUG:boto:Params: {} 2023-10-11T15:27:13.999 DEBUG:boto:Token: None 2023-10-11T15:27:14.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:13 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:14.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4= 2023-10-11T15:27:14.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3iviFlQlu4DyZx4u5F+zRMRyEr4=', 'Content-Length': '0'} 2023-10-11T15:27:14.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004270102e8cadc3af-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.003 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:14.003 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:14.004 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:14.004 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:14.005 DEBUG:boto:Method: GET 2023-10-11T15:27:14.005 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:14.005 DEBUG:boto:Data: 2023-10-11T15:27:14.006 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.007 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.007 DEBUG:boto:Params: {} 2023-10-11T15:27:14.008 DEBUG:boto:Token: None 2023-10-11T15:27:14.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:14.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Q1gNzti3Q6Z7JOMuqLHEYtYnXwQ= 2023-10-11T15:27:14.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q1gNzti3Q6Z7JOMuqLHEYtYnXwQ=', 'Content-Length': '0'} 2023-10-11T15:27:14.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371a893681e34912-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.012 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.012 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.012 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.013 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.013 DEBUG:boto:Method: GET 2023-10-11T15:27:14.014 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.014 DEBUG:boto:Data: 2023-10-11T15:27:14.015 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.015 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.016 DEBUG:boto:Params: {} 2023-10-11T15:27:14.016 DEBUG:boto:Token: None 2023-10-11T15:27:14.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000150b139ac03ea1e-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.062 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.063 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.063 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.064 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.064 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.064 DEBUG:boto:Method: GET 2023-10-11T15:27:14.065 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.065 DEBUG:boto:Data: 2023-10-11T15:27:14.066 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.066 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.067 DEBUG:boto:Params: {} 2023-10-11T15:27:14.067 DEBUG:boto:Token: None 2023-10-11T15:27:14.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002abd012192372751-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.073 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.074 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:14.074 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:14.075 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.075 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:14.075 DEBUG:boto:Data: 2023-10-11T15:27:14.076 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.077 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.077 DEBUG:boto:Params: {} 2023-10-11T15:27:14.077 DEBUG:boto:Token: None 2023-10-11T15:27:14.078 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj111 2023-10-11T15:27:14.078 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JSZ3zL9Ng1wHrvlfppe631PgiGc= 2023-10-11T15:27:14.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JSZ3zL9Ng1wHrvlfppe631PgiGc=', 'Content-Length': '0'} 2023-10-11T15:27:14.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957e64cde420f26e-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.081 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:14.082 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:14.082 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.083 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:14.083 DEBUG:boto:Data: 2023-10-11T15:27:14.083 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.084 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.085 DEBUG:boto:Params: {} 2023-10-11T15:27:14.085 DEBUG:boto:Token: None 2023-10-11T15:27:14.085 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj111 2023-10-11T15:27:14.086 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JSZ3zL9Ng1wHrvlfppe631PgiGc= 2023-10-11T15:27:14.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JSZ3zL9Ng1wHrvlfppe631PgiGc=', 'Content-Length': '0'} 2023-10-11T15:27:14.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4088d8b0aef85b-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:14.089 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:14.090 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:14.090 DEBUG:boto:Method: GET 2023-10-11T15:27:14.091 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:14.091 DEBUG:boto:Data: 2023-10-11T15:27:14.091 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.092 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.093 DEBUG:boto:Params: {} 2023-10-11T15:27:14.093 DEBUG:boto:Token: None 2023-10-11T15:27:14.094 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj111 2023-10-11T15:27:14.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y2eV1RcNEQuCepEJW2wp+2pRW9Y= 2023-10-11T15:27:14.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y2eV1RcNEQuCepEJW2wp+2pRW9Y=', 'Content-Length': '0'} 2023-10-11T15:27:14.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1012ff82620efd-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.097 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:14.098 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:14.098 DEBUG:boto:Method: GET 2023-10-11T15:27:14.098 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:14.099 DEBUG:boto:Data: 2023-10-11T15:27:14.099 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.100 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.100 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.100 DEBUG:boto:Params: {} 2023-10-11T15:27:14.101 DEBUG:boto:Token: None 2023-10-11T15:27:14.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj111 2023-10-11T15:27:14.102 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y2eV1RcNEQuCepEJW2wp+2pRW9Y= 2023-10-11T15:27:14.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y2eV1RcNEQuCepEJW2wp+2pRW9Y=', 'Content-Length': '0'} 2023-10-11T15:27:14.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb46e39ea92feaa-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.146 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.147 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.147 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.147 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.148 DEBUG:boto:Method: GET 2023-10-11T15:27:14.148 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.149 DEBUG:boto:Data: 2023-10-11T15:27:14.149 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.150 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.150 DEBUG:boto:Params: {} 2023-10-11T15:27:14.151 DEBUG:boto:Token: None 2023-10-11T15:27:14.151 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c576ca62de0636a-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.155 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.155 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.156 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.156 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.157 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.157 DEBUG:boto:Method: GET 2023-10-11T15:27:14.157 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:14.158 DEBUG:boto:Data: 2023-10-11T15:27:14.158 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.159 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.159 DEBUG:boto:Params: {} 2023-10-11T15:27:14.160 DEBUG:boto:Token: None 2023-10-11T15:27:14.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0f2755d3863484-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.166 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:14.167 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.167 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.168 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.168 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.169 DEBUG:boto:Method: GET 2023-10-11T15:27:14.169 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.169 DEBUG:boto:Data: 2023-10-11T15:27:14.170 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.170 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.171 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.171 DEBUG:boto:Params: {} 2023-10-11T15:27:14.172 DEBUG:boto:Token: None 2023-10-11T15:27:14.172 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4= 2023-10-11T15:27:14.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4=', 'Content-Length': '0'} 2023-10-11T15:27:14.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b306ef2e4c68b3c4-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.175 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.176 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.176 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.176 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.177 DEBUG:boto:Method: GET 2023-10-11T15:27:14.177 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.178 DEBUG:boto:Data: 2023-10-11T15:27:14.178 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.179 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.179 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.179 DEBUG:boto:Params: {} 2023-10-11T15:27:14.180 DEBUG:boto:Token: None 2023-10-11T15:27:14.180 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4= 2023-10-11T15:27:14.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4=', 'Content-Length': '0'} 2023-10-11T15:27:14.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e6f10c441e8c58-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.184 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.184 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.184 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.185 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.185 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.185 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.186 DEBUG:boto:Data: 2023-10-11T15:27:14.186 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.187 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.187 DEBUG:boto:Params: {} 2023-10-11T15:27:14.188 DEBUG:boto:Token: None 2023-10-11T15:27:14.188 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSgCjQ+XK/2mxqqOaTuKHRojoPM= 2023-10-11T15:27:14.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSgCjQ+XK/2mxqqOaTuKHRojoPM=', 'Content-Length': '0'} 2023-10-11T15:27:14.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84d441c3033c482-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.191 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.192 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.192 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.192 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.193 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.193 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.194 DEBUG:boto:Data: 2023-10-11T15:27:14.194 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.195 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.195 DEBUG:boto:Params: {} 2023-10-11T15:27:14.196 DEBUG:boto:Token: None 2023-10-11T15:27:14.196 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uSgCjQ+XK/2mxqqOaTuKHRojoPM= 2023-10-11T15:27:14.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uSgCjQ+XK/2mxqqOaTuKHRojoPM=', 'Content-Length': '0'} 2023-10-11T15:27:14.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007619f4df6bd0e87d-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:14.199 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.200 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.200 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.201 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.201 DEBUG:boto:Method: GET 2023-10-11T15:27:14.201 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.202 DEBUG:boto:Data: 2023-10-11T15:27:14.202 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.203 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.203 DEBUG:boto:Params: {} 2023-10-11T15:27:14.204 DEBUG:boto:Token: None 2023-10-11T15:27:14.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4= 2023-10-11T15:27:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4=', 'Content-Length': '0'} 2023-10-11T15:27:14.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026f8d68d19b8d707-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.250 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.251 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.251 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.252 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.252 DEBUG:boto:Method: GET 2023-10-11T15:27:14.252 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.253 DEBUG:boto:Data: 2023-10-11T15:27:14.253 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.254 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.254 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.254 DEBUG:boto:Params: {} 2023-10-11T15:27:14.255 DEBUG:boto:Token: None 2023-10-11T15:27:14.255 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:14.256 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4= 2023-10-11T15:27:14.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knuaB4romKKnhvgeZlS3BMMW5a4=', 'Content-Length': '0'} 2023-10-11T15:27:14.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d598e13ef97ff98-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.259 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.259 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.260 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.260 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.261 DEBUG:boto:Method: GET 2023-10-11T15:27:14.261 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.261 DEBUG:boto:Data: 2023-10-11T15:27:14.262 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.263 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.263 DEBUG:boto:Params: {} 2023-10-11T15:27:14.264 DEBUG:boto:Token: None 2023-10-11T15:27:14.264 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040905b718886cc21-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.268 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.268 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.269 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.269 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.269 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.270 DEBUG:boto:Method: GET 2023-10-11T15:27:14.270 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.271 DEBUG:boto:Data: 2023-10-11T15:27:14.271 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.272 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.272 DEBUG:boto:Params: {} 2023-10-11T15:27:14.273 DEBUG:boto:Token: None 2023-10-11T15:27:14.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007afec72ded93e3b3-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.279 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.279 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.280 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.280 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.280 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:14.281 DEBUG:boto:Data: 2023-10-11T15:27:14.281 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.282 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.282 DEBUG:boto:Params: {} 2023-10-11T15:27:14.283 DEBUG:boto:Token: None 2023-10-11T15:27:14.283 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115 2023-10-11T15:27:14.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:va5xNRKhk5la02K6llazObXkKu8= 2023-10-11T15:27:14.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:va5xNRKhk5la02K6llazObXkKu8=', 'Content-Length': '0'} 2023-10-11T15:27:14.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ccab4088cce0bf-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.288 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.289 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.289 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.289 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:14.290 DEBUG:boto:Data: 2023-10-11T15:27:14.290 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.291 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.291 DEBUG:boto:Params: {} 2023-10-11T15:27:14.292 DEBUG:boto:Token: None 2023-10-11T15:27:14.292 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115 2023-10-11T15:27:14.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:va5xNRKhk5la02K6llazObXkKu8= 2023-10-11T15:27:14.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:va5xNRKhk5la02K6llazObXkKu8=', 'Content-Length': '0'} 2023-10-11T15:27:14.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f57f871b3919fc2-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:14.296 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.296 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.297 DEBUG:boto:Method: GET 2023-10-11T15:27:14.297 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:14.297 DEBUG:boto:Data: 2023-10-11T15:27:14.298 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.299 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.299 DEBUG:boto:Params: {} 2023-10-11T15:27:14.299 DEBUG:boto:Token: None 2023-10-11T15:27:14.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115 2023-10-11T15:27:14.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G6XAnX2cE/GejfbmI3wYzm+64UU= 2023-10-11T15:27:14.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G6XAnX2cE/GejfbmI3wYzm+64UU=', 'Content-Length': '0'} 2023-10-11T15:27:14.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559358daa310dd08-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.303 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:14.304 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:14.304 DEBUG:boto:Method: GET 2023-10-11T15:27:14.305 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:14.305 DEBUG:boto:Data: 2023-10-11T15:27:14.305 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.306 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.307 DEBUG:boto:Params: {} 2023-10-11T15:27:14.307 DEBUG:boto:Token: None 2023-10-11T15:27:14.307 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj115 2023-10-11T15:27:14.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G6XAnX2cE/GejfbmI3wYzm+64UU= 2023-10-11T15:27:14.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G6XAnX2cE/GejfbmI3wYzm+64UU=', 'Content-Length': '0'} 2023-10-11T15:27:14.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa57251ef53b5df-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.354 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.355 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.356 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.356 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.356 DEBUG:boto:Method: GET 2023-10-11T15:27:14.357 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.357 DEBUG:boto:Data: 2023-10-11T15:27:14.358 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.359 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.359 DEBUG:boto:Params: {} 2023-10-11T15:27:14.360 DEBUG:boto:Token: None 2023-10-11T15:27:14.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000309dcc87cd476c7a-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.364 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.365 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.365 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.366 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.366 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.366 DEBUG:boto:Method: GET 2023-10-11T15:27:14.367 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:14.367 DEBUG:boto:Data: 2023-10-11T15:27:14.368 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.369 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.369 DEBUG:boto:Params: {} 2023-10-11T15:27:14.369 DEBUG:boto:Token: None 2023-10-11T15:27:14.370 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f105366bd65ec894-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.376 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:14.377 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.377 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.377 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.378 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.378 DEBUG:boto:Method: GET 2023-10-11T15:27:14.379 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.379 DEBUG:boto:Data: 2023-10-11T15:27:14.380 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.380 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.381 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.381 DEBUG:boto:Params: {} 2023-10-11T15:27:14.381 DEBUG:boto:Token: None 2023-10-11T15:27:14.382 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.382 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk= 2023-10-11T15:27:14.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk=', 'Content-Length': '0'} 2023-10-11T15:27:14.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50f9d4f5405ee64-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.426 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.427 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.428 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.428 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.428 DEBUG:boto:Method: GET 2023-10-11T15:27:14.429 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.429 DEBUG:boto:Data: 2023-10-11T15:27:14.430 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.431 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.431 DEBUG:boto:Params: {} 2023-10-11T15:27:14.431 DEBUG:boto:Token: None 2023-10-11T15:27:14.432 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk= 2023-10-11T15:27:14.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk=', 'Content-Length': '0'} 2023-10-11T15:27:14.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd962ac9e57dcef-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.436 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.436 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.436 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.437 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.437 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.438 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.438 DEBUG:boto:Data: 2023-10-11T15:27:14.439 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.439 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.440 DEBUG:boto:Params: {} 2023-10-11T15:27:14.440 DEBUG:boto:Token: None 2023-10-11T15:27:14.441 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vKN3LVCTJdj5VOFbp6KxqToAlk4= 2023-10-11T15:27:14.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vKN3LVCTJdj5VOFbp6KxqToAlk4=', 'Content-Length': '0'} 2023-10-11T15:27:14.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a956e1f868e4d4c-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.445 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.445 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.446 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.446 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.446 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.447 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.447 DEBUG:boto:Data: 2023-10-11T15:27:14.448 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.448 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.449 DEBUG:boto:Params: {} 2023-10-11T15:27:14.449 DEBUG:boto:Token: None 2023-10-11T15:27:14.450 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vKN3LVCTJdj5VOFbp6KxqToAlk4= 2023-10-11T15:27:14.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vKN3LVCTJdj5VOFbp6KxqToAlk4=', 'Content-Length': '0'} 2023-10-11T15:27:14.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45e936e68cba42d-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:14.453 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.454 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.454 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.455 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.455 DEBUG:boto:Method: GET 2023-10-11T15:27:14.456 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.456 DEBUG:boto:Data: 2023-10-11T15:27:14.457 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.457 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.457 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.458 DEBUG:boto:Params: {} 2023-10-11T15:27:14.458 DEBUG:boto:Token: None 2023-10-11T15:27:14.459 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.459 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk= 2023-10-11T15:27:14.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk=', 'Content-Length': '0'} 2023-10-11T15:27:14.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004268e2ed5e722a38-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.462 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.463 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.463 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.464 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.464 DEBUG:boto:Method: GET 2023-10-11T15:27:14.464 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.465 DEBUG:boto:Data: 2023-10-11T15:27:14.465 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.466 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.466 DEBUG:boto:Params: {} 2023-10-11T15:27:14.467 DEBUG:boto:Token: None 2023-10-11T15:27:14.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:14.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk= 2023-10-11T15:27:14.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nTgpNFMVapu3rCvKBolGYYOd/Hk=', 'Content-Length': '0'} 2023-10-11T15:27:14.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092730969b31416c1-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.470 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.471 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.471 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.472 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.472 DEBUG:boto:Method: GET 2023-10-11T15:27:14.472 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.473 DEBUG:boto:Data: 2023-10-11T15:27:14.473 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.473 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.474 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.474 DEBUG:boto:Params: {} 2023-10-11T15:27:14.475 DEBUG:boto:Token: None 2023-10-11T15:27:14.475 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003580039f78dc64c7-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.479 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.480 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.480 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.480 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.481 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.481 DEBUG:boto:Method: GET 2023-10-11T15:27:14.482 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.482 DEBUG:boto:Data: 2023-10-11T15:27:14.482 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.483 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.484 DEBUG:boto:Params: {} 2023-10-11T15:27:14.484 DEBUG:boto:Token: None 2023-10-11T15:27:14.484 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df144005d8a1090b-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.490 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.490 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.491 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.491 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.492 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:14.492 DEBUG:boto:Data: 2023-10-11T15:27:14.492 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.493 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.494 DEBUG:boto:Params: {} 2023-10-11T15:27:14.494 DEBUG:boto:Token: None 2023-10-11T15:27:14.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116 2023-10-11T15:27:14.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WvkVa3uJT/h+UbhWBhrh5hZSwc= 2023-10-11T15:27:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WvkVa3uJT/h+UbhWBhrh5hZSwc=', 'Content-Length': '0'} 2023-10-11T15:27:14.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fcedea0adba36d2-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.498 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.498 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.499 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.499 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:14.499 DEBUG:boto:Data: 2023-10-11T15:27:14.500 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.500 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.500 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.501 DEBUG:boto:Params: {} 2023-10-11T15:27:14.501 DEBUG:boto:Token: None 2023-10-11T15:27:14.502 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116 2023-10-11T15:27:14.502 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4WvkVa3uJT/h+UbhWBhrh5hZSwc= 2023-10-11T15:27:14.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4WvkVa3uJT/h+UbhWBhrh5hZSwc=', 'Content-Length': '0'} 2023-10-11T15:27:14.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000810057d8f70f7df1-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:14.505 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.506 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.506 DEBUG:boto:Method: GET 2023-10-11T15:27:14.507 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:14.507 DEBUG:boto:Data: 2023-10-11T15:27:14.507 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.508 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.509 DEBUG:boto:Params: {} 2023-10-11T15:27:14.509 DEBUG:boto:Token: None 2023-10-11T15:27:14.509 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116 2023-10-11T15:27:14.510 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AYJkI76MJ4878fVbLB+uXC0ckos= 2023-10-11T15:27:14.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AYJkI76MJ4878fVbLB+uXC0ckos=', 'Content-Length': '0'} 2023-10-11T15:27:14.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f643aa95d3263a-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.513 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:14.513 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:14.514 DEBUG:boto:Method: GET 2023-10-11T15:27:14.514 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:14.514 DEBUG:boto:Data: 2023-10-11T15:27:14.515 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.516 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.516 DEBUG:boto:Params: {} 2023-10-11T15:27:14.517 DEBUG:boto:Token: None 2023-10-11T15:27:14.517 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj116 2023-10-11T15:27:14.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AYJkI76MJ4878fVbLB+uXC0ckos= 2023-10-11T15:27:14.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AYJkI76MJ4878fVbLB+uXC0ckos=', 'Content-Length': '0'} 2023-10-11T15:27:14.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c7ac4096e09d76-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.567 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.567 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.567 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.568 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.568 DEBUG:boto:Method: GET 2023-10-11T15:27:14.569 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.569 DEBUG:boto:Data: 2023-10-11T15:27:14.569 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.570 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.571 DEBUG:boto:Params: {} 2023-10-11T15:27:14.571 DEBUG:boto:Token: None 2023-10-11T15:27:14.571 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebef37889c94ab19-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.575 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.576 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.576 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.577 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.577 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.578 DEBUG:boto:Method: GET 2023-10-11T15:27:14.578 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:14.578 DEBUG:boto:Data: 2023-10-11T15:27:14.579 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.580 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.580 DEBUG:boto:Params: {} 2023-10-11T15:27:14.580 DEBUG:boto:Token: None 2023-10-11T15:27:14.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b24eff71faa8da-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.586 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:14.587 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.587 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.588 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.588 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.588 DEBUG:boto:Method: GET 2023-10-11T15:27:14.589 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.589 DEBUG:boto:Data: 2023-10-11T15:27:14.590 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.591 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.591 DEBUG:boto:Params: {} 2023-10-11T15:27:14.591 DEBUG:boto:Token: None 2023-10-11T15:27:14.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I= 2023-10-11T15:27:14.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I=', 'Content-Length': '0'} 2023-10-11T15:27:14.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e0de894ee0a2c9b-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.642 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.643 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.644 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.644 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.644 DEBUG:boto:Method: GET 2023-10-11T15:27:14.645 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.645 DEBUG:boto:Data: 2023-10-11T15:27:14.646 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.646 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.647 DEBUG:boto:Params: {} 2023-10-11T15:27:14.647 DEBUG:boto:Token: None 2023-10-11T15:27:14.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I= 2023-10-11T15:27:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I=', 'Content-Length': '0'} 2023-10-11T15:27:14.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000941f0326af9913e-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.652 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.652 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.653 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.653 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.653 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.654 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.654 DEBUG:boto:Data: 2023-10-11T15:27:14.655 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.656 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.656 DEBUG:boto:Params: {} 2023-10-11T15:27:14.656 DEBUG:boto:Token: None 2023-10-11T15:27:14.657 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.657 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:juCahsjvOTb9my/ECYAlYYjeryM= 2023-10-11T15:27:14.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:juCahsjvOTb9my/ECYAlYYjeryM=', 'Content-Length': '0'} 2023-10-11T15:27:14.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007628285cb4450353-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.660 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.661 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.661 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.662 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.662 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.663 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.663 DEBUG:boto:Data: 2023-10-11T15:27:14.663 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.664 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.664 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.665 DEBUG:boto:Params: {} 2023-10-11T15:27:14.665 DEBUG:boto:Token: None 2023-10-11T15:27:14.666 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.666 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:juCahsjvOTb9my/ECYAlYYjeryM= 2023-10-11T15:27:14.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:juCahsjvOTb9my/ECYAlYYjeryM=', 'Content-Length': '0'} 2023-10-11T15:27:14.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d59471aeb33279f-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:14.669 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.670 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.670 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.671 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.671 DEBUG:boto:Method: GET 2023-10-11T15:27:14.671 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.672 DEBUG:boto:Data: 2023-10-11T15:27:14.672 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.673 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.674 DEBUG:boto:Params: {} 2023-10-11T15:27:14.674 DEBUG:boto:Token: None 2023-10-11T15:27:14.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I= 2023-10-11T15:27:14.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I=', 'Content-Length': '0'} 2023-10-11T15:27:14.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001414684e3f46a870-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.678 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.678 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.679 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.679 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.679 DEBUG:boto:Method: GET 2023-10-11T15:27:14.680 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.680 DEBUG:boto:Data: 2023-10-11T15:27:14.681 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.682 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.682 DEBUG:boto:Params: {} 2023-10-11T15:27:14.682 DEBUG:boto:Token: None 2023-10-11T15:27:14.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:14.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I= 2023-10-11T15:27:14.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uxHR+De6qlV/QLYEbpBxKUgXK1I=', 'Content-Length': '0'} 2023-10-11T15:27:14.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956f15ee8a57c743-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.686 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.687 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.687 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.688 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.688 DEBUG:boto:Method: GET 2023-10-11T15:27:14.688 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.689 DEBUG:boto:Data: 2023-10-11T15:27:14.689 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.690 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.690 DEBUG:boto:Params: {} 2023-10-11T15:27:14.691 DEBUG:boto:Token: None 2023-10-11T15:27:14.691 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec9bb7c6141ab30-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.695 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.696 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.696 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.697 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.697 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.698 DEBUG:boto:Method: GET 2023-10-11T15:27:14.698 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.701 DEBUG:boto:Data: 2023-10-11T15:27:14.701 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.702 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.702 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.703 DEBUG:boto:Params: {} 2023-10-11T15:27:14.703 DEBUG:boto:Token: None 2023-10-11T15:27:14.704 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.704 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a9ab2e49a43a49b-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.709 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.710 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.710 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.710 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.711 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:14.711 DEBUG:boto:Data: 2023-10-11T15:27:14.712 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.713 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.713 DEBUG:boto:Params: {} 2023-10-11T15:27:14.713 DEBUG:boto:Token: None 2023-10-11T15:27:14.714 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124 2023-10-11T15:27:14.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z5wfkbdc0sB3nqqDX7Qe/vyXTic= 2023-10-11T15:27:14.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z5wfkbdc0sB3nqqDX7Qe/vyXTic=', 'Content-Length': '0'} 2023-10-11T15:27:14.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033801388ed0578b9-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.717 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.718 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.718 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.719 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:14.719 DEBUG:boto:Data: 2023-10-11T15:27:14.719 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.720 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.721 DEBUG:boto:Params: {} 2023-10-11T15:27:14.721 DEBUG:boto:Token: None 2023-10-11T15:27:14.722 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124 2023-10-11T15:27:14.722 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z5wfkbdc0sB3nqqDX7Qe/vyXTic= 2023-10-11T15:27:14.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z5wfkbdc0sB3nqqDX7Qe/vyXTic=', 'Content-Length': '0'} 2023-10-11T15:27:14.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000364bdd29841ae57c-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:14.726 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.726 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.726 DEBUG:boto:Method: GET 2023-10-11T15:27:14.727 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:14.727 DEBUG:boto:Data: 2023-10-11T15:27:14.728 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.729 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.729 DEBUG:boto:Params: {} 2023-10-11T15:27:14.729 DEBUG:boto:Token: None 2023-10-11T15:27:14.730 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124 2023-10-11T15:27:14.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nkCaArZObVDDzhOQuJMZQdFJC2o= 2023-10-11T15:27:14.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nkCaArZObVDDzhOQuJMZQdFJC2o=', 'Content-Length': '0'} 2023-10-11T15:27:14.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044246d20566037bd-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.734 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:14.734 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:14.734 DEBUG:boto:Method: GET 2023-10-11T15:27:14.735 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:14.735 DEBUG:boto:Data: 2023-10-11T15:27:14.736 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.736 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.737 DEBUG:boto:Params: {} 2023-10-11T15:27:14.737 DEBUG:boto:Token: None 2023-10-11T15:27:14.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj124 2023-10-11T15:27:14.738 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nkCaArZObVDDzhOQuJMZQdFJC2o= 2023-10-11T15:27:14.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nkCaArZObVDDzhOQuJMZQdFJC2o=', 'Content-Length': '0'} 2023-10-11T15:27:14.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f756a5b6d5e5556-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.782 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.783 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.783 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.783 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.784 DEBUG:boto:Method: GET 2023-10-11T15:27:14.784 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.785 DEBUG:boto:Data: 2023-10-11T15:27:14.785 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.786 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.786 DEBUG:boto:Params: {} 2023-10-11T15:27:14.787 DEBUG:boto:Token: None 2023-10-11T15:27:14.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c650331e5d5e13a-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.791 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.792 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.792 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.792 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.793 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.793 DEBUG:boto:Method: GET 2023-10-11T15:27:14.794 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:14.794 DEBUG:boto:Data: 2023-10-11T15:27:14.794 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.795 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.796 DEBUG:boto:Params: {} 2023-10-11T15:27:14.796 DEBUG:boto:Token: None 2023-10-11T15:27:14.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e923d6bebb5cf4a-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.802 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:14.803 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.804 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.804 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.805 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.805 DEBUG:boto:Method: GET 2023-10-11T15:27:14.805 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.806 DEBUG:boto:Data: 2023-10-11T15:27:14.806 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.807 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.807 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.808 DEBUG:boto:Params: {} 2023-10-11T15:27:14.808 DEBUG:boto:Token: None 2023-10-11T15:27:14.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM= 2023-10-11T15:27:14.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM=', 'Content-Length': '0'} 2023-10-11T15:27:14.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f01a7142efadf6a-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.854 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.855 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.855 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.856 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.856 DEBUG:boto:Method: GET 2023-10-11T15:27:14.856 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.857 DEBUG:boto:Data: 2023-10-11T15:27:14.857 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.858 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.858 DEBUG:boto:Params: {} 2023-10-11T15:27:14.859 DEBUG:boto:Token: None 2023-10-11T15:27:14.859 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM= 2023-10-11T15:27:14.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM=', 'Content-Length': '0'} 2023-10-11T15:27:14.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c133cf3f5cded0-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.863 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.863 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.864 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.864 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.864 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.865 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.865 DEBUG:boto:Data: 2023-10-11T15:27:14.865 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.866 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.867 DEBUG:boto:Params: {} 2023-10-11T15:27:14.867 DEBUG:boto:Token: None 2023-10-11T15:27:14.867 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R5BUWI2HNrrQItLZz72D+QeCAJ0= 2023-10-11T15:27:14.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R5BUWI2HNrrQItLZz72D+QeCAJ0=', 'Content-Length': '0'} 2023-10-11T15:27:14.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9728e7dba0a75d5-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.871 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.871 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.872 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.872 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.872 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.873 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.873 DEBUG:boto:Data: 2023-10-11T15:27:14.873 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.874 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.875 DEBUG:boto:Params: {} 2023-10-11T15:27:14.875 DEBUG:boto:Token: None 2023-10-11T15:27:14.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:R5BUWI2HNrrQItLZz72D+QeCAJ0= 2023-10-11T15:27:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:R5BUWI2HNrrQItLZz72D+QeCAJ0=', 'Content-Length': '0'} 2023-10-11T15:27:14.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f48ad3c23a818ab2-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:14.879 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.880 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.880 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.880 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.881 DEBUG:boto:Method: GET 2023-10-11T15:27:14.881 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.882 DEBUG:boto:Data: 2023-10-11T15:27:14.882 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.883 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.883 DEBUG:boto:Params: {} 2023-10-11T15:27:14.884 DEBUG:boto:Token: None 2023-10-11T15:27:14.884 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM= 2023-10-11T15:27:14.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM=', 'Content-Length': '0'} 2023-10-11T15:27:14.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed744d21a71a35e2-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.888 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.888 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.889 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.889 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.890 DEBUG:boto:Method: GET 2023-10-11T15:27:14.890 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.890 DEBUG:boto:Data: 2023-10-11T15:27:14.891 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.892 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.892 DEBUG:boto:Params: {} 2023-10-11T15:27:14.892 DEBUG:boto:Token: None 2023-10-11T15:27:14.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:14.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM= 2023-10-11T15:27:14.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YKuxH4wFAppjPP3G+Lai1czESJM=', 'Content-Length': '0'} 2023-10-11T15:27:14.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f590aed805c725-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.942 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.943 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.943 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.944 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.944 DEBUG:boto:Method: GET 2023-10-11T15:27:14.944 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.945 DEBUG:boto:Data: 2023-10-11T15:27:14.945 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.946 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.946 DEBUG:boto:Params: {} 2023-10-11T15:27:14.947 DEBUG:boto:Token: None 2023-10-11T15:27:14.947 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d16c7b750c96380-006526bed2-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.951 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.952 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:14.952 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:14.952 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.953 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.953 DEBUG:boto:Method: GET 2023-10-11T15:27:14.954 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:14.954 DEBUG:boto:Data: 2023-10-11T15:27:14.955 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.955 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.956 DEBUG:boto:Params: {} 2023-10-11T15:27:14.956 DEBUG:boto:Token: None 2023-10-11T15:27:14.957 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/ 2023-10-11T15:27:14.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o= 2023-10-11T15:27:14.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/U8s0rmBRHsSshUuIbo2+NqOu7o=', 'Content-Length': '0'} 2023-10-11T15:27:14.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb97e6a4f4f1650-006526bed2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.962 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:14.963 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.963 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.964 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.964 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:14.965 DEBUG:boto:Data: 2023-10-11T15:27:14.965 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.966 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.966 DEBUG:boto:Params: {} 2023-10-11T15:27:14.967 DEBUG:boto:Token: None 2023-10-11T15:27:14.967 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145 2023-10-11T15:27:14.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HTfOa9Wr72wZo4rFecRGdZYWIhE= 2023-10-11T15:27:14.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HTfOa9Wr72wZo4rFecRGdZYWIhE=', 'Content-Length': '0'} 2023-10-11T15:27:14.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31625dbd9c9ee9a-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.971 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.971 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.972 DEBUG:boto:Method: HEAD 2023-10-11T15:27:14.972 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:14.972 DEBUG:boto:Data: 2023-10-11T15:27:14.973 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:14.974 DEBUG:boto:Port: 8000 2023-10-11T15:27:14.974 DEBUG:boto:Params: {} 2023-10-11T15:27:14.974 DEBUG:boto:Token: None 2023-10-11T15:27:14.975 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145 2023-10-11T15:27:14.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HTfOa9Wr72wZo4rFecRGdZYWIhE= 2023-10-11T15:27:14.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HTfOa9Wr72wZo4rFecRGdZYWIhE=', 'Content-Length': '0'} 2023-10-11T15:27:14.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1f7dcfd47e285f-006526bed2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:14.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:14.979 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:14.979 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:14.979 DEBUG:boto:Method: GET 2023-10-11T15:27:14.980 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:14.980 DEBUG:boto:Data: 2023-10-11T15:27:14.981 DEBUG:boto:Headers: {} 2023-10-11T15:27:14.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:14.982 DEBUG:boto:Port: 8001 2023-10-11T15:27:14.982 DEBUG:boto:Params: {} 2023-10-11T15:27:14.983 DEBUG:boto:Token: None 2023-10-11T15:27:14.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:14 GMT /dghqvd-59/obj145 2023-10-11T15:27:14.984 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IXYm6jM3bvBAflEf2YhJVmCjruA= 2023-10-11T15:27:14.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IXYm6jM3bvBAflEf2YhJVmCjruA=', 'Content-Length': '0'} 2023-10-11T15:27:14.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69787f1326056bb-006526bed2-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.026 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:15.027 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:15.027 DEBUG:boto:Method: GET 2023-10-11T15:27:15.028 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:15.028 DEBUG:boto:Data: 2023-10-11T15:27:15.029 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.029 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.030 DEBUG:boto:Params: {} 2023-10-11T15:27:15.030 DEBUG:boto:Token: None 2023-10-11T15:27:15.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj145 2023-10-11T15:27:15.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8tTBLiddrNK6DlnPzArcrkd7PTA= 2023-10-11T15:27:15.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8tTBLiddrNK6DlnPzArcrkd7PTA=', 'Content-Length': '0'} 2023-10-11T15:27:15.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000726505c4c3293787-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.035 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.035 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.035 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.036 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.036 DEBUG:boto:Method: GET 2023-10-11T15:27:15.037 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.037 DEBUG:boto:Data: 2023-10-11T15:27:15.038 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.038 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.039 DEBUG:boto:Params: {} 2023-10-11T15:27:15.039 DEBUG:boto:Token: None 2023-10-11T15:27:15.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.040 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006451d61ee03b2f75-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.044 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.045 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.045 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.045 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.046 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.046 DEBUG:boto:Method: GET 2023-10-11T15:27:15.047 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:15.047 DEBUG:boto:Data: 2023-10-11T15:27:15.047 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.048 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.049 DEBUG:boto:Params: {} 2023-10-11T15:27:15.049 DEBUG:boto:Token: None 2023-10-11T15:27:15.049 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.050 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057cede9c3db7cf67-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.055 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:15.056 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.056 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.057 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.057 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.058 DEBUG:boto:Method: GET 2023-10-11T15:27:15.058 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.059 DEBUG:boto:Data: 2023-10-11T15:27:15.059 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.059 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.060 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.060 DEBUG:boto:Params: {} 2023-10-11T15:27:15.061 DEBUG:boto:Token: None 2023-10-11T15:27:15.061 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.062 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk= 2023-10-11T15:27:15.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk=', 'Content-Length': '0'} 2023-10-11T15:27:15.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005943f9051614ec9a-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.065 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.065 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.066 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.066 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.066 DEBUG:boto:Method: GET 2023-10-11T15:27:15.067 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.067 DEBUG:boto:Data: 2023-10-11T15:27:15.068 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.068 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.068 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.069 DEBUG:boto:Params: {} 2023-10-11T15:27:15.069 DEBUG:boto:Token: None 2023-10-11T15:27:15.070 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.070 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk= 2023-10-11T15:27:15.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk=', 'Content-Length': '0'} 2023-10-11T15:27:15.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ac79b8a42597e8-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.073 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.074 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.074 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.075 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.075 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.075 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.076 DEBUG:boto:Data: 2023-10-11T15:27:15.076 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.077 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.077 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.077 DEBUG:boto:Params: {} 2023-10-11T15:27:15.078 DEBUG:boto:Token: None 2023-10-11T15:27:15.078 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OZME0ephjl2XudWRUSywZjdSNxY= 2023-10-11T15:27:15.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OZME0ephjl2XudWRUSywZjdSNxY=', 'Content-Length': '0'} 2023-10-11T15:27:15.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000152a62724ea163f5-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.082 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.082 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.083 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.083 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.083 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.084 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.084 DEBUG:boto:Data: 2023-10-11T15:27:15.084 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.085 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.085 DEBUG:boto:Params: {} 2023-10-11T15:27:15.086 DEBUG:boto:Token: None 2023-10-11T15:27:15.086 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OZME0ephjl2XudWRUSywZjdSNxY= 2023-10-11T15:27:15.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OZME0ephjl2XudWRUSywZjdSNxY=', 'Content-Length': '0'} 2023-10-11T15:27:15.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000766fb78567361ed1-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:15.090 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.090 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.091 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.091 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.091 DEBUG:boto:Method: GET 2023-10-11T15:27:15.092 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.092 DEBUG:boto:Data: 2023-10-11T15:27:15.093 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.093 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.094 DEBUG:boto:Params: {} 2023-10-11T15:27:15.094 DEBUG:boto:Token: None 2023-10-11T15:27:15.095 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk= 2023-10-11T15:27:15.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk=', 'Content-Length': '0'} 2023-10-11T15:27:15.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000077762f340253c8c-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.098 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.099 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.099 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.099 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.100 DEBUG:boto:Method: GET 2023-10-11T15:27:15.100 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.100 DEBUG:boto:Data: 2023-10-11T15:27:15.101 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.102 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.102 DEBUG:boto:Params: {} 2023-10-11T15:27:15.103 DEBUG:boto:Token: None 2023-10-11T15:27:15.103 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:15.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk= 2023-10-11T15:27:15.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AnD2DV4c1aFlVEZ96Ca0kiTjOQk=', 'Content-Length': '0'} 2023-10-11T15:27:15.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000995bc8664c16c9fa-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.106 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.107 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.107 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.108 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.108 DEBUG:boto:Method: GET 2023-10-11T15:27:15.108 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.109 DEBUG:boto:Data: 2023-10-11T15:27:15.109 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.110 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.110 DEBUG:boto:Params: {} 2023-10-11T15:27:15.111 DEBUG:boto:Token: None 2023-10-11T15:27:15.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032327e7f59548624-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.115 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.116 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.116 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.117 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.117 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.117 DEBUG:boto:Method: GET 2023-10-11T15:27:15.118 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.118 DEBUG:boto:Data: 2023-10-11T15:27:15.119 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.119 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.120 DEBUG:boto:Params: {} 2023-10-11T15:27:15.120 DEBUG:boto:Token: None 2023-10-11T15:27:15.121 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087426e0017802248-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.126 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.126 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.127 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.127 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.128 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:15.128 DEBUG:boto:Data: 2023-10-11T15:27:15.128 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.129 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.129 DEBUG:boto:Params: {} 2023-10-11T15:27:15.130 DEBUG:boto:Token: None 2023-10-11T15:27:15.130 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148 2023-10-11T15:27:15.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mVnZ02jb3sbpiSdXFJtwwPkeLfk= 2023-10-11T15:27:15.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mVnZ02jb3sbpiSdXFJtwwPkeLfk=', 'Content-Length': '0'} 2023-10-11T15:27:15.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155a3cc5592ef924-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.134 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.134 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.134 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.135 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:15.135 DEBUG:boto:Data: 2023-10-11T15:27:15.135 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.136 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.137 DEBUG:boto:Params: {} 2023-10-11T15:27:15.137 DEBUG:boto:Token: None 2023-10-11T15:27:15.137 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148 2023-10-11T15:27:15.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mVnZ02jb3sbpiSdXFJtwwPkeLfk= 2023-10-11T15:27:15.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mVnZ02jb3sbpiSdXFJtwwPkeLfk=', 'Content-Length': '0'} 2023-10-11T15:27:15.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d956bac635d2ba-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:15.141 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.142 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.142 DEBUG:boto:Method: GET 2023-10-11T15:27:15.142 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:15.143 DEBUG:boto:Data: 2023-10-11T15:27:15.143 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.144 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.144 DEBUG:boto:Params: {} 2023-10-11T15:27:15.145 DEBUG:boto:Token: None 2023-10-11T15:27:15.145 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148 2023-10-11T15:27:15.146 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F74aFZMvHslK9unXVUanVJxeXIk= 2023-10-11T15:27:15.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F74aFZMvHslK9unXVUanVJxeXIk=', 'Content-Length': '0'} 2023-10-11T15:27:15.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ddc49e6db80ff3d-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.190 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:15.191 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:15.191 DEBUG:boto:Method: GET 2023-10-11T15:27:15.192 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:15.192 DEBUG:boto:Data: 2023-10-11T15:27:15.192 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.193 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.193 DEBUG:boto:Params: {} 2023-10-11T15:27:15.194 DEBUG:boto:Token: None 2023-10-11T15:27:15.194 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj148 2023-10-11T15:27:15.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F74aFZMvHslK9unXVUanVJxeXIk= 2023-10-11T15:27:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F74aFZMvHslK9unXVUanVJxeXIk=', 'Content-Length': '0'} 2023-10-11T15:27:15.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0743ac3dbb31b6-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.198 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.198 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.199 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.199 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.199 DEBUG:boto:Method: GET 2023-10-11T15:27:15.200 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.200 DEBUG:boto:Data: 2023-10-11T15:27:15.201 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.202 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.202 DEBUG:boto:Params: {} 2023-10-11T15:27:15.202 DEBUG:boto:Token: None 2023-10-11T15:27:15.203 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000218d21d9eb1ebb14-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.207 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.207 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.208 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.208 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.208 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.209 DEBUG:boto:Method: GET 2023-10-11T15:27:15.209 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:15.210 DEBUG:boto:Data: 2023-10-11T15:27:15.210 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.211 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.211 DEBUG:boto:Params: {} 2023-10-11T15:27:15.212 DEBUG:boto:Token: None 2023-10-11T15:27:15.212 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df1d88de9bbb099-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.217 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:15.219 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.219 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.219 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.220 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.220 DEBUG:boto:Method: GET 2023-10-11T15:27:15.221 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.221 DEBUG:boto:Data: 2023-10-11T15:27:15.221 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.222 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.223 DEBUG:boto:Params: {} 2023-10-11T15:27:15.223 DEBUG:boto:Token: None 2023-10-11T15:27:15.224 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw= 2023-10-11T15:27:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw=', 'Content-Length': '0'} 2023-10-11T15:27:15.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003cb514c235fa982-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.227 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.228 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.228 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.229 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.229 DEBUG:boto:Method: GET 2023-10-11T15:27:15.230 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.230 DEBUG:boto:Data: 2023-10-11T15:27:15.230 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.231 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.232 DEBUG:boto:Params: {} 2023-10-11T15:27:15.232 DEBUG:boto:Token: None 2023-10-11T15:27:15.233 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw= 2023-10-11T15:27:15.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw=', 'Content-Length': '0'} 2023-10-11T15:27:15.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e65f6c496c09530-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.282 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.283 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.283 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.284 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.284 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.284 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.285 DEBUG:boto:Data: 2023-10-11T15:27:15.285 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.286 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.286 DEBUG:boto:Params: {} 2023-10-11T15:27:15.287 DEBUG:boto:Token: None 2023-10-11T15:27:15.287 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nRXhIDvTuSj31pRPL9+NGUJs7tw= 2023-10-11T15:27:15.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nRXhIDvTuSj31pRPL9+NGUJs7tw=', 'Content-Length': '0'} 2023-10-11T15:27:15.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d2a112854f4957-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.291 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.291 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.291 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.292 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.292 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.293 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.293 DEBUG:boto:Data: 2023-10-11T15:27:15.293 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.294 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.295 DEBUG:boto:Params: {} 2023-10-11T15:27:15.295 DEBUG:boto:Token: None 2023-10-11T15:27:15.295 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nRXhIDvTuSj31pRPL9+NGUJs7tw= 2023-10-11T15:27:15.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nRXhIDvTuSj31pRPL9+NGUJs7tw=', 'Content-Length': '0'} 2023-10-11T15:27:15.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401fb33fc1e49739-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:15.299 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.300 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.300 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.301 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.301 DEBUG:boto:Method: GET 2023-10-11T15:27:15.302 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.302 DEBUG:boto:Data: 2023-10-11T15:27:15.302 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.303 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.304 DEBUG:boto:Params: {} 2023-10-11T15:27:15.304 DEBUG:boto:Token: None 2023-10-11T15:27:15.305 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw= 2023-10-11T15:27:15.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw=', 'Content-Length': '0'} 2023-10-11T15:27:15.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367feb113644a2c2-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.308 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.309 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.309 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.309 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.310 DEBUG:boto:Method: GET 2023-10-11T15:27:15.310 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.311 DEBUG:boto:Data: 2023-10-11T15:27:15.311 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.312 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.312 DEBUG:boto:Params: {} 2023-10-11T15:27:15.313 DEBUG:boto:Token: None 2023-10-11T15:27:15.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:15.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw= 2023-10-11T15:27:15.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NR3Kmy2L7vuZ6/tPB4K0qyJHuQw=', 'Content-Length': '0'} 2023-10-11T15:27:15.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334c24ddcae0fa16-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.318 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.318 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.319 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.319 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.320 DEBUG:boto:Method: GET 2023-10-11T15:27:15.320 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.320 DEBUG:boto:Data: 2023-10-11T15:27:15.321 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.321 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.322 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.322 DEBUG:boto:Params: {} 2023-10-11T15:27:15.323 DEBUG:boto:Token: None 2023-10-11T15:27:15.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086801d4c7008f31e-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.327 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.328 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.328 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.329 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.329 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.330 DEBUG:boto:Method: GET 2023-10-11T15:27:15.330 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.330 DEBUG:boto:Data: 2023-10-11T15:27:15.331 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.332 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.332 DEBUG:boto:Params: {} 2023-10-11T15:27:15.333 DEBUG:boto:Token: None 2023-10-11T15:27:15.333 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.334 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2357eecc36c696f-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.339 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.340 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.340 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.341 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.341 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:15.342 DEBUG:boto:Data: 2023-10-11T15:27:15.342 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.343 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.343 DEBUG:boto:Params: {} 2023-10-11T15:27:15.344 DEBUG:boto:Token: None 2023-10-11T15:27:15.344 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155 2023-10-11T15:27:15.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jIQ2MrlP1kh9jkQ4WrOkYRmr2Yw= 2023-10-11T15:27:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jIQ2MrlP1kh9jkQ4WrOkYRmr2Yw=', 'Content-Length': '0'} 2023-10-11T15:27:15.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010bd312f26db9767-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.348 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.348 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.348 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.349 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:15.349 DEBUG:boto:Data: 2023-10-11T15:27:15.350 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.350 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.351 DEBUG:boto:Params: {} 2023-10-11T15:27:15.351 DEBUG:boto:Token: None 2023-10-11T15:27:15.352 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155 2023-10-11T15:27:15.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jIQ2MrlP1kh9jkQ4WrOkYRmr2Yw= 2023-10-11T15:27:15.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jIQ2MrlP1kh9jkQ4WrOkYRmr2Yw=', 'Content-Length': '0'} 2023-10-11T15:27:15.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c702b2e9e90ee4-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:15.356 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.356 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.356 DEBUG:boto:Method: GET 2023-10-11T15:27:15.357 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:15.357 DEBUG:boto:Data: 2023-10-11T15:27:15.358 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.359 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.359 DEBUG:boto:Params: {} 2023-10-11T15:27:15.359 DEBUG:boto:Token: None 2023-10-11T15:27:15.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155 2023-10-11T15:27:15.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:34sojQssmxT7XqG3Dc5fYtwkjVc= 2023-10-11T15:27:15.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:34sojQssmxT7XqG3Dc5fYtwkjVc=', 'Content-Length': '0'} 2023-10-11T15:27:15.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdba553fd1a9530-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.406 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:15.407 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:15.407 DEBUG:boto:Method: GET 2023-10-11T15:27:15.408 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:15.408 DEBUG:boto:Data: 2023-10-11T15:27:15.408 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.409 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.410 DEBUG:boto:Params: {} 2023-10-11T15:27:15.410 DEBUG:boto:Token: None 2023-10-11T15:27:15.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj155 2023-10-11T15:27:15.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:34sojQssmxT7XqG3Dc5fYtwkjVc= 2023-10-11T15:27:15.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:34sojQssmxT7XqG3Dc5fYtwkjVc=', 'Content-Length': '0'} 2023-10-11T15:27:15.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ccc628c29b0878-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.414 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.415 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.415 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.415 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.416 DEBUG:boto:Method: GET 2023-10-11T15:27:15.416 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.417 DEBUG:boto:Data: 2023-10-11T15:27:15.417 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.418 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.418 DEBUG:boto:Params: {} 2023-10-11T15:27:15.419 DEBUG:boto:Token: None 2023-10-11T15:27:15.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e95069999782b5-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.423 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.423 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.424 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.424 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.425 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.425 DEBUG:boto:Method: GET 2023-10-11T15:27:15.425 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:15.426 DEBUG:boto:Data: 2023-10-11T15:27:15.426 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.427 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.428 DEBUG:boto:Params: {} 2023-10-11T15:27:15.428 DEBUG:boto:Token: None 2023-10-11T15:27:15.429 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae27855410a9461b-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.435 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:15.436 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.436 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.437 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.437 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.437 DEBUG:boto:Method: GET 2023-10-11T15:27:15.438 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.438 DEBUG:boto:Data: 2023-10-11T15:27:15.439 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.440 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.440 DEBUG:boto:Params: {} 2023-10-11T15:27:15.440 DEBUG:boto:Token: None 2023-10-11T15:27:15.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs= 2023-10-11T15:27:15.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs=', 'Content-Length': '0'} 2023-10-11T15:27:15.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028bf51b3ae1d257a-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.444 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.445 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.445 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.446 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.446 DEBUG:boto:Method: GET 2023-10-11T15:27:15.446 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.447 DEBUG:boto:Data: 2023-10-11T15:27:15.447 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.448 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.448 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.448 DEBUG:boto:Params: {} 2023-10-11T15:27:15.449 DEBUG:boto:Token: None 2023-10-11T15:27:15.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.450 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs= 2023-10-11T15:27:15.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs=', 'Content-Length': '0'} 2023-10-11T15:27:15.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6edb8ffc665e6f3-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.452 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.453 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.453 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.454 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.454 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.454 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.455 DEBUG:boto:Data: 2023-10-11T15:27:15.455 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.456 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.456 DEBUG:boto:Params: {} 2023-10-11T15:27:15.457 DEBUG:boto:Token: None 2023-10-11T15:27:15.457 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BwRqvtX0/E7kCs9A33Mn6IItFnE= 2023-10-11T15:27:15.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BwRqvtX0/E7kCs9A33Mn6IItFnE=', 'Content-Length': '0'} 2023-10-11T15:27:15.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b34388e162b3a3-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.460 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.461 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.461 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.461 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.462 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.462 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.462 DEBUG:boto:Data: 2023-10-11T15:27:15.463 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.464 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.464 DEBUG:boto:Params: {} 2023-10-11T15:27:15.464 DEBUG:boto:Token: None 2023-10-11T15:27:15.465 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BwRqvtX0/E7kCs9A33Mn6IItFnE= 2023-10-11T15:27:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BwRqvtX0/E7kCs9A33Mn6IItFnE=', 'Content-Length': '0'} 2023-10-11T15:27:15.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000312b493f5d03395-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:15.468 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.469 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.469 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.470 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.470 DEBUG:boto:Method: GET 2023-10-11T15:27:15.471 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.471 DEBUG:boto:Data: 2023-10-11T15:27:15.471 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.472 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.473 DEBUG:boto:Params: {} 2023-10-11T15:27:15.473 DEBUG:boto:Token: None 2023-10-11T15:27:15.474 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs= 2023-10-11T15:27:15.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs=', 'Content-Length': '0'} 2023-10-11T15:27:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059eae6eab21263c2-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.477 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.477 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.478 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.478 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.479 DEBUG:boto:Method: GET 2023-10-11T15:27:15.479 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.479 DEBUG:boto:Data: 2023-10-11T15:27:15.480 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.481 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.481 DEBUG:boto:Params: {} 2023-10-11T15:27:15.481 DEBUG:boto:Token: None 2023-10-11T15:27:15.482 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:15.482 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs= 2023-10-11T15:27:15.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JecRC+I5yTCk9z+fEtwQ94Y8gzs=', 'Content-Length': '0'} 2023-10-11T15:27:15.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ca0cbcf2eccaf7-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.485 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.486 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.486 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.486 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.487 DEBUG:boto:Method: GET 2023-10-11T15:27:15.487 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.488 DEBUG:boto:Data: 2023-10-11T15:27:15.488 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.489 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.489 DEBUG:boto:Params: {} 2023-10-11T15:27:15.489 DEBUG:boto:Token: None 2023-10-11T15:27:15.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.490 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061427ff724b60e67-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.494 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.494 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.495 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.495 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.496 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.496 DEBUG:boto:Method: GET 2023-10-11T15:27:15.496 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.497 DEBUG:boto:Data: 2023-10-11T15:27:15.497 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.498 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.498 DEBUG:boto:Params: {} 2023-10-11T15:27:15.499 DEBUG:boto:Token: None 2023-10-11T15:27:15.499 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f23276d9d56bb395-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.504 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.505 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.505 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.506 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.506 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:15.506 DEBUG:boto:Data: 2023-10-11T15:27:15.507 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.507 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.508 DEBUG:boto:Params: {} 2023-10-11T15:27:15.508 DEBUG:boto:Token: None 2023-10-11T15:27:15.509 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17 2023-10-11T15:27:15.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iw8UOih+waPevpd41Z4ghQP3srQ= 2023-10-11T15:27:15.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iw8UOih+waPevpd41Z4ghQP3srQ=', 'Content-Length': '0'} 2023-10-11T15:27:15.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de11a27b37419e79-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.512 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.512 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.513 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.513 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:15.513 DEBUG:boto:Data: 2023-10-11T15:27:15.514 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.515 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.515 DEBUG:boto:Params: {} 2023-10-11T15:27:15.515 DEBUG:boto:Token: None 2023-10-11T15:27:15.516 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17 2023-10-11T15:27:15.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iw8UOih+waPevpd41Z4ghQP3srQ= 2023-10-11T15:27:15.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iw8UOih+waPevpd41Z4ghQP3srQ=', 'Content-Length': '0'} 2023-10-11T15:27:15.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122771276e4e4030-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:15.522 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.523 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.523 DEBUG:boto:Method: GET 2023-10-11T15:27:15.524 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:15.524 DEBUG:boto:Data: 2023-10-11T15:27:15.524 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.525 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.525 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.526 DEBUG:boto:Params: {} 2023-10-11T15:27:15.526 DEBUG:boto:Token: None 2023-10-11T15:27:15.527 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17 2023-10-11T15:27:15.527 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ysBw/guf0p8pAvCLCB832LYgWkQ= 2023-10-11T15:27:15.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ysBw/guf0p8pAvCLCB832LYgWkQ=', 'Content-Length': '0'} 2023-10-11T15:27:15.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043da2884bfcccffe-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.530 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:15.530 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:15.531 DEBUG:boto:Method: GET 2023-10-11T15:27:15.531 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:15.531 DEBUG:boto:Data: 2023-10-11T15:27:15.532 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.533 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.533 DEBUG:boto:Params: {} 2023-10-11T15:27:15.533 DEBUG:boto:Token: None 2023-10-11T15:27:15.534 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj17 2023-10-11T15:27:15.534 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ysBw/guf0p8pAvCLCB832LYgWkQ= 2023-10-11T15:27:15.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ysBw/guf0p8pAvCLCB832LYgWkQ=', 'Content-Length': '0'} 2023-10-11T15:27:15.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd4a1f422596e13-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.578 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.643 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.643 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.644 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.644 DEBUG:boto:Method: GET 2023-10-11T15:27:15.645 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.645 DEBUG:boto:Data: 2023-10-11T15:27:15.645 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.646 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.647 DEBUG:boto:Params: {} 2023-10-11T15:27:15.647 DEBUG:boto:Token: None 2023-10-11T15:27:15.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8288909b9db291a-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.652 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.653 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.653 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.654 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.654 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.654 DEBUG:boto:Method: GET 2023-10-11T15:27:15.655 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:15.655 DEBUG:boto:Data: 2023-10-11T15:27:15.656 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.657 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.657 DEBUG:boto:Params: {} 2023-10-11T15:27:15.658 DEBUG:boto:Token: None 2023-10-11T15:27:15.658 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.659 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa33c293cf8bcd8f-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.664 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:15.665 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.665 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.665 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.666 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.666 DEBUG:boto:Method: GET 2023-10-11T15:27:15.667 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.667 DEBUG:boto:Data: 2023-10-11T15:27:15.668 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.668 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.669 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.669 DEBUG:boto:Params: {} 2023-10-11T15:27:15.669 DEBUG:boto:Token: None 2023-10-11T15:27:15.670 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.670 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc= 2023-10-11T15:27:15.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc=', 'Content-Length': '0'} 2023-10-11T15:27:15.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5243d7147ed4c3e-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.718 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.719 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.719 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.720 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.720 DEBUG:boto:Method: GET 2023-10-11T15:27:15.720 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.721 DEBUG:boto:Data: 2023-10-11T15:27:15.721 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.722 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.722 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.723 DEBUG:boto:Params: {} 2023-10-11T15:27:15.723 DEBUG:boto:Token: None 2023-10-11T15:27:15.724 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc= 2023-10-11T15:27:15.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc=', 'Content-Length': '0'} 2023-10-11T15:27:15.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef559245f8cd6d3-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.729 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.730 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.730 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.730 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.731 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.731 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.732 DEBUG:boto:Data: 2023-10-11T15:27:15.732 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.733 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.733 DEBUG:boto:Params: {} 2023-10-11T15:27:15.734 DEBUG:boto:Token: None 2023-10-11T15:27:15.734 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.735 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bZg2c1jMkKUN5H2lTUMESSzJKE8= 2023-10-11T15:27:15.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bZg2c1jMkKUN5H2lTUMESSzJKE8=', 'Content-Length': '0'} 2023-10-11T15:27:15.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c750b8bdebfc25f-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.738 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.738 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.739 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.739 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.739 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.740 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.740 DEBUG:boto:Data: 2023-10-11T15:27:15.740 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.741 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.742 DEBUG:boto:Params: {} 2023-10-11T15:27:15.742 DEBUG:boto:Token: None 2023-10-11T15:27:15.743 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bZg2c1jMkKUN5H2lTUMESSzJKE8= 2023-10-11T15:27:15.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bZg2c1jMkKUN5H2lTUMESSzJKE8=', 'Content-Length': '0'} 2023-10-11T15:27:15.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec287e42fc90d01-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:15.746 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.747 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.747 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.748 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.748 DEBUG:boto:Method: GET 2023-10-11T15:27:15.748 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.749 DEBUG:boto:Data: 2023-10-11T15:27:15.749 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.750 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.750 DEBUG:boto:Params: {} 2023-10-11T15:27:15.751 DEBUG:boto:Token: None 2023-10-11T15:27:15.751 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc= 2023-10-11T15:27:15.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc=', 'Content-Length': '0'} 2023-10-11T15:27:15.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000609b7af6fcdf5a16-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.755 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.755 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.756 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.756 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.757 DEBUG:boto:Method: GET 2023-10-11T15:27:15.757 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.757 DEBUG:boto:Data: 2023-10-11T15:27:15.758 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.759 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.759 DEBUG:boto:Params: {} 2023-10-11T15:27:15.759 DEBUG:boto:Token: None 2023-10-11T15:27:15.760 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:15.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc= 2023-10-11T15:27:15.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RuzAJq/T+7+u9vg9pW8SHbZymqc=', 'Content-Length': '0'} 2023-10-11T15:27:15.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa96961d7e9123c-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.763 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.764 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.764 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.764 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.765 DEBUG:boto:Method: GET 2023-10-11T15:27:15.765 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.765 DEBUG:boto:Data: 2023-10-11T15:27:15.766 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.767 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.767 DEBUG:boto:Params: {} 2023-10-11T15:27:15.767 DEBUG:boto:Token: None 2023-10-11T15:27:15.768 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069c3b37cb76c4c37-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.772 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.773 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.773 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.773 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.774 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.774 DEBUG:boto:Method: GET 2023-10-11T15:27:15.775 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.775 DEBUG:boto:Data: 2023-10-11T15:27:15.775 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.776 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.777 DEBUG:boto:Params: {} 2023-10-11T15:27:15.777 DEBUG:boto:Token: None 2023-10-11T15:27:15.778 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.778 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb74eadda3fe9db5-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.783 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.784 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.784 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.785 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.785 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:15.786 DEBUG:boto:Data: 2023-10-11T15:27:15.786 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.787 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.787 DEBUG:boto:Params: {} 2023-10-11T15:27:15.787 DEBUG:boto:Token: None 2023-10-11T15:27:15.788 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185 2023-10-11T15:27:15.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5qTaEJLnbUooPRXRfysOx3Hcz6I= 2023-10-11T15:27:15.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5qTaEJLnbUooPRXRfysOx3Hcz6I=', 'Content-Length': '0'} 2023-10-11T15:27:15.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008380c525e81d7fc4-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.791 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.792 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.792 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.792 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:15.793 DEBUG:boto:Data: 2023-10-11T15:27:15.793 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.794 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.794 DEBUG:boto:Params: {} 2023-10-11T15:27:15.795 DEBUG:boto:Token: None 2023-10-11T15:27:15.795 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185 2023-10-11T15:27:15.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5qTaEJLnbUooPRXRfysOx3Hcz6I= 2023-10-11T15:27:15.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5qTaEJLnbUooPRXRfysOx3Hcz6I=', 'Content-Length': '0'} 2023-10-11T15:27:15.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe8abb3291e8cb5-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:15.799 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.799 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.800 DEBUG:boto:Method: GET 2023-10-11T15:27:15.800 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:15.800 DEBUG:boto:Data: 2023-10-11T15:27:15.801 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.802 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.802 DEBUG:boto:Params: {} 2023-10-11T15:27:15.803 DEBUG:boto:Token: None 2023-10-11T15:27:15.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185 2023-10-11T15:27:15.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kIkmsEhnCvfHNXBiwdHKOAh9/dA= 2023-10-11T15:27:15.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kIkmsEhnCvfHNXBiwdHKOAh9/dA=', 'Content-Length': '0'} 2023-10-11T15:27:15.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005672e3c8c7c7125f-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.850 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:15.851 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:15.851 DEBUG:boto:Method: GET 2023-10-11T15:27:15.852 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:15.852 DEBUG:boto:Data: 2023-10-11T15:27:15.853 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.853 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.854 DEBUG:boto:Params: {} 2023-10-11T15:27:15.854 DEBUG:boto:Token: None 2023-10-11T15:27:15.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj185 2023-10-11T15:27:15.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kIkmsEhnCvfHNXBiwdHKOAh9/dA= 2023-10-11T15:27:15.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kIkmsEhnCvfHNXBiwdHKOAh9/dA=', 'Content-Length': '0'} 2023-10-11T15:27:15.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af431f8a028d7cda-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.858 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.859 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.859 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.860 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.860 DEBUG:boto:Method: GET 2023-10-11T15:27:15.860 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.861 DEBUG:boto:Data: 2023-10-11T15:27:15.861 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.862 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.862 DEBUG:boto:Params: {} 2023-10-11T15:27:15.863 DEBUG:boto:Token: None 2023-10-11T15:27:15.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ca7775c5a94bc8-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.867 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.868 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.868 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.869 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.869 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.869 DEBUG:boto:Method: GET 2023-10-11T15:27:15.870 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:15.870 DEBUG:boto:Data: 2023-10-11T15:27:15.871 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.871 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.872 DEBUG:boto:Params: {} 2023-10-11T15:27:15.872 DEBUG:boto:Token: None 2023-10-11T15:27:15.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd87ae5db778a63-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.878 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:15.879 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.880 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.880 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.880 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.881 DEBUG:boto:Method: GET 2023-10-11T15:27:15.881 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.882 DEBUG:boto:Data: 2023-10-11T15:27:15.882 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.883 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.883 DEBUG:boto:Params: {} 2023-10-11T15:27:15.884 DEBUG:boto:Token: None 2023-10-11T15:27:15.884 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY= 2023-10-11T15:27:15.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY=', 'Content-Length': '0'} 2023-10-11T15:27:15.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6e91a5cc817e94-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.888 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.888 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.889 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.889 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.889 DEBUG:boto:Method: GET 2023-10-11T15:27:15.890 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.890 DEBUG:boto:Data: 2023-10-11T15:27:15.891 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.891 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.892 DEBUG:boto:Params: {} 2023-10-11T15:27:15.892 DEBUG:boto:Token: None 2023-10-11T15:27:15.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY= 2023-10-11T15:27:15.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY=', 'Content-Length': '0'} 2023-10-11T15:27:15.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008faac4b66a089f29-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.896 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.897 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.897 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.897 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.898 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.898 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.899 DEBUG:boto:Data: 2023-10-11T15:27:15.899 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.900 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.900 DEBUG:boto:Params: {} 2023-10-11T15:27:15.901 DEBUG:boto:Token: None 2023-10-11T15:27:15.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRH5/p+IY6HHqCpU0J3gDKl1uE0= 2023-10-11T15:27:15.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRH5/p+IY6HHqCpU0J3gDKl1uE0=', 'Content-Length': '0'} 2023-10-11T15:27:15.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039023fbe2145a657-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.904 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.905 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.905 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.906 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.906 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.906 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.907 DEBUG:boto:Data: 2023-10-11T15:27:15.907 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.908 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.908 DEBUG:boto:Params: {} 2023-10-11T15:27:15.909 DEBUG:boto:Token: None 2023-10-11T15:27:15.909 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRH5/p+IY6HHqCpU0J3gDKl1uE0= 2023-10-11T15:27:15.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRH5/p+IY6HHqCpU0J3gDKl1uE0=', 'Content-Length': '0'} 2023-10-11T15:27:15.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041426e3e5b9ee174-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:15.913 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.913 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.914 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.914 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.914 DEBUG:boto:Method: GET 2023-10-11T15:27:15.915 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.915 DEBUG:boto:Data: 2023-10-11T15:27:15.916 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.917 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.917 DEBUG:boto:Params: {} 2023-10-11T15:27:15.917 DEBUG:boto:Token: None 2023-10-11T15:27:15.918 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY= 2023-10-11T15:27:15.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY=', 'Content-Length': '0'} 2023-10-11T15:27:15.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020500f684f075080-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.921 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.921 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.922 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.922 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.923 DEBUG:boto:Method: GET 2023-10-11T15:27:15.923 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.923 DEBUG:boto:Data: 2023-10-11T15:27:15.924 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.924 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.925 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.925 DEBUG:boto:Params: {} 2023-10-11T15:27:15.926 DEBUG:boto:Token: None 2023-10-11T15:27:15.926 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:15.926 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY= 2023-10-11T15:27:15.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xoqGvV3DyBFHl6WKDXKW/LeEsoY=', 'Content-Length': '0'} 2023-10-11T15:27:15.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e753d55ac48d8e1-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.929 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.930 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.930 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.931 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.931 DEBUG:boto:Method: GET 2023-10-11T15:27:15.934 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.934 DEBUG:boto:Data: 2023-10-11T15:27:15.935 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.935 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.935 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.936 DEBUG:boto:Params: {} 2023-10-11T15:27:15.937 DEBUG:boto:Token: None 2023-10-11T15:27:15.937 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d208811d14489f-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.942 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.942 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.943 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.943 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.943 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.944 DEBUG:boto:Method: GET 2023-10-11T15:27:15.944 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.944 DEBUG:boto:Data: 2023-10-11T15:27:15.945 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.945 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.946 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.946 DEBUG:boto:Params: {} 2023-10-11T15:27:15.947 DEBUG:boto:Token: None 2023-10-11T15:27:15.947 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.947 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65f0dc17b875a66-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.952 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.953 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.953 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.954 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.954 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:15.955 DEBUG:boto:Data: 2023-10-11T15:27:15.955 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.956 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.956 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.957 DEBUG:boto:Params: {} 2023-10-11T15:27:15.957 DEBUG:boto:Token: None 2023-10-11T15:27:15.957 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19 2023-10-11T15:27:15.958 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EgI2lHEfEY7mw+LOvNpCVRJCD5o= 2023-10-11T15:27:15.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EgI2lHEfEY7mw+LOvNpCVRJCD5o=', 'Content-Length': '0'} 2023-10-11T15:27:15.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9b41c80d8d1906-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.961 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.962 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.962 DEBUG:boto:Method: HEAD 2023-10-11T15:27:15.962 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:15.963 DEBUG:boto:Data: 2023-10-11T15:27:15.963 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.964 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.964 DEBUG:boto:Params: {} 2023-10-11T15:27:15.965 DEBUG:boto:Token: None 2023-10-11T15:27:15.965 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19 2023-10-11T15:27:15.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EgI2lHEfEY7mw+LOvNpCVRJCD5o= 2023-10-11T15:27:15.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EgI2lHEfEY7mw+LOvNpCVRJCD5o=', 'Content-Length': '0'} 2023-10-11T15:27:15.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a516de22ec4ad5a5-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:15.969 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.969 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.969 DEBUG:boto:Method: GET 2023-10-11T15:27:15.970 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:15.970 DEBUG:boto:Data: 2023-10-11T15:27:15.971 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.971 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.971 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.972 DEBUG:boto:Params: {} 2023-10-11T15:27:15.972 DEBUG:boto:Token: None 2023-10-11T15:27:15.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19 2023-10-11T15:27:15.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ErKgwm+UBDDxLrVPJi4zsNUkEoE= 2023-10-11T15:27:15.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ErKgwm+UBDDxLrVPJi4zsNUkEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c920252d165424cc-006526bed3-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.976 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:15.976 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:15.977 DEBUG:boto:Method: GET 2023-10-11T15:27:15.977 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:15.978 DEBUG:boto:Data: 2023-10-11T15:27:15.978 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.979 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.979 DEBUG:boto:Params: {} 2023-10-11T15:27:15.980 DEBUG:boto:Token: None 2023-10-11T15:27:15.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/obj19 2023-10-11T15:27:15.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ErKgwm+UBDDxLrVPJi4zsNUkEoE= 2023-10-11T15:27:15.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ErKgwm+UBDDxLrVPJi4zsNUkEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010324291e6984677-006526bed3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.983 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.984 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.984 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.984 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.985 DEBUG:boto:Method: GET 2023-10-11T15:27:15.985 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.986 DEBUG:boto:Data: 2023-10-11T15:27:15.986 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:15.987 DEBUG:boto:Port: 8001 2023-10-11T15:27:15.987 DEBUG:boto:Params: {} 2023-10-11T15:27:15.988 DEBUG:boto:Token: None 2023-10-11T15:27:15.988 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:15.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a7a49e49b2ba32-006526bed3-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:15.991 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:15.992 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:15.992 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:15.993 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.993 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.994 DEBUG:boto:Method: GET 2023-10-11T15:27:15.994 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:15.994 DEBUG:boto:Data: 2023-10-11T15:27:15.995 DEBUG:boto:Headers: {} 2023-10-11T15:27:15.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:15.995 DEBUG:boto:Port: 8000 2023-10-11T15:27:15.996 DEBUG:boto:Params: {} 2023-10-11T15:27:15.996 DEBUG:boto:Token: None 2023-10-11T15:27:15.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:15 GMT /dghqvd-59/ 2023-10-11T15:27:15.997 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE= 2023-10-11T15:27:15.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hkdB6KDbeHl5+yXSEGAADYPNEoE=', 'Content-Length': '0'} 2023-10-11T15:27:16.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a31da68352d08a3-006526bed3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.003 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:16.004 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.004 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.004 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.005 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.005 DEBUG:boto:Method: GET 2023-10-11T15:27:16.005 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.006 DEBUG:boto:Data: 2023-10-11T15:27:16.006 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.007 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.007 DEBUG:boto:Params: {} 2023-10-11T15:27:16.008 DEBUG:boto:Token: None 2023-10-11T15:27:16.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8= 2023-10-11T15:27:16.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8=', 'Content-Length': '0'} 2023-10-11T15:27:16.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5db9d4cf2fca6fc-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.054 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.055 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.055 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.056 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.056 DEBUG:boto:Method: GET 2023-10-11T15:27:16.056 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.057 DEBUG:boto:Data: 2023-10-11T15:27:16.057 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.058 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.059 DEBUG:boto:Params: {} 2023-10-11T15:27:16.059 DEBUG:boto:Token: None 2023-10-11T15:27:16.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8= 2023-10-11T15:27:16.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8=', 'Content-Length': '0'} 2023-10-11T15:27:16.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d09300a786cc0d-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.063 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.064 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.064 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.064 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.065 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.065 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.065 DEBUG:boto:Data: 2023-10-11T15:27:16.066 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.067 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.067 DEBUG:boto:Params: {} 2023-10-11T15:27:16.067 DEBUG:boto:Token: None 2023-10-11T15:27:16.068 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uVk50ZlTq3LeiG60BNxIXQdExX4= 2023-10-11T15:27:16.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uVk50ZlTq3LeiG60BNxIXQdExX4=', 'Content-Length': '0'} 2023-10-11T15:27:16.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed5f429c5ca31c5b-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.071 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.071 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.072 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.072 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.073 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.073 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.073 DEBUG:boto:Data: 2023-10-11T15:27:16.074 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.075 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.075 DEBUG:boto:Params: {} 2023-10-11T15:27:16.076 DEBUG:boto:Token: None 2023-10-11T15:27:16.076 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uVk50ZlTq3LeiG60BNxIXQdExX4= 2023-10-11T15:27:16.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uVk50ZlTq3LeiG60BNxIXQdExX4=', 'Content-Length': '0'} 2023-10-11T15:27:16.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12d35e2f9a7d19b-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:16.080 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.080 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.080 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.081 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.081 DEBUG:boto:Method: GET 2023-10-11T15:27:16.082 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.082 DEBUG:boto:Data: 2023-10-11T15:27:16.083 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.083 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.084 DEBUG:boto:Params: {} 2023-10-11T15:27:16.084 DEBUG:boto:Token: None 2023-10-11T15:27:16.085 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8= 2023-10-11T15:27:16.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8=', 'Content-Length': '0'} 2023-10-11T15:27:16.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a00e7237a8a5715-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.088 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.089 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.089 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.089 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.090 DEBUG:boto:Method: GET 2023-10-11T15:27:16.090 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.091 DEBUG:boto:Data: 2023-10-11T15:27:16.091 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.092 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.092 DEBUG:boto:Params: {} 2023-10-11T15:27:16.093 DEBUG:boto:Token: None 2023-10-11T15:27:16.093 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:16.094 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8= 2023-10-11T15:27:16.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zodoHKJzhLWI7Dy7OS4XgVtMuu8=', 'Content-Length': '0'} 2023-10-11T15:27:16.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe2ece1b6d23cd0-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.138 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.139 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.139 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.142 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.142 DEBUG:boto:Method: GET 2023-10-11T15:27:16.143 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.143 DEBUG:boto:Data: 2023-10-11T15:27:16.144 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.145 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.145 DEBUG:boto:Params: {} 2023-10-11T15:27:16.146 DEBUG:boto:Token: None 2023-10-11T15:27:16.146 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e5a06677752629-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.150 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.151 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.152 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.152 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.152 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.153 DEBUG:boto:Method: GET 2023-10-11T15:27:16.153 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.154 DEBUG:boto:Data: 2023-10-11T15:27:16.154 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.155 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.155 DEBUG:boto:Params: {} 2023-10-11T15:27:16.156 DEBUG:boto:Token: None 2023-10-11T15:27:16.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000039414084f0e27a1-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.162 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.162 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.163 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.163 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.164 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:16.164 DEBUG:boto:Data: 2023-10-11T15:27:16.164 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.165 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.165 DEBUG:boto:Params: {} 2023-10-11T15:27:16.166 DEBUG:boto:Token: None 2023-10-11T15:27:16.166 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22 2023-10-11T15:27:16.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wX36PhA8Oy+WjEamVZlqw7pvmvQ= 2023-10-11T15:27:16.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wX36PhA8Oy+WjEamVZlqw7pvmvQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5b6b53ee5952a8-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.169 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.170 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.170 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.171 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:16.171 DEBUG:boto:Data: 2023-10-11T15:27:16.171 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.172 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.172 DEBUG:boto:Params: {} 2023-10-11T15:27:16.173 DEBUG:boto:Token: None 2023-10-11T15:27:16.173 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22 2023-10-11T15:27:16.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wX36PhA8Oy+WjEamVZlqw7pvmvQ= 2023-10-11T15:27:16.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wX36PhA8Oy+WjEamVZlqw7pvmvQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b328bd61a3c697-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:16.177 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.178 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.178 DEBUG:boto:Method: GET 2023-10-11T15:27:16.179 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:16.179 DEBUG:boto:Data: 2023-10-11T15:27:16.179 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.180 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.180 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.181 DEBUG:boto:Params: {} 2023-10-11T15:27:16.181 DEBUG:boto:Token: None 2023-10-11T15:27:16.182 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22 2023-10-11T15:27:16.182 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V5dugLfi9213HhsjOHDb0gVo/8M= 2023-10-11T15:27:16.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V5dugLfi9213HhsjOHDb0gVo/8M=', 'Content-Length': '0'} 2023-10-11T15:27:16.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3beaf8e88f77040-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.185 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:16.186 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:16.186 DEBUG:boto:Method: GET 2023-10-11T15:27:16.187 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:16.187 DEBUG:boto:Data: 2023-10-11T15:27:16.188 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.188 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.189 DEBUG:boto:Params: {} 2023-10-11T15:27:16.189 DEBUG:boto:Token: None 2023-10-11T15:27:16.190 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj22 2023-10-11T15:27:16.190 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V5dugLfi9213HhsjOHDb0gVo/8M= 2023-10-11T15:27:16.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V5dugLfi9213HhsjOHDb0gVo/8M=', 'Content-Length': '0'} 2023-10-11T15:27:16.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da61f1c6bf0c2904-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.194 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.194 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.195 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.195 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.196 DEBUG:boto:Method: GET 2023-10-11T15:27:16.196 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.196 DEBUG:boto:Data: 2023-10-11T15:27:16.197 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.197 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.198 DEBUG:boto:Params: {} 2023-10-11T15:27:16.198 DEBUG:boto:Token: None 2023-10-11T15:27:16.199 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002569d109768ef2eb-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.202 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.203 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.204 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.204 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.204 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.205 DEBUG:boto:Method: GET 2023-10-11T15:27:16.205 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:16.206 DEBUG:boto:Data: 2023-10-11T15:27:16.206 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.207 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.207 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.207 DEBUG:boto:Params: {} 2023-10-11T15:27:16.208 DEBUG:boto:Token: None 2023-10-11T15:27:16.208 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.209 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068994d6b2eefd576-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.214 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:16.215 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.215 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.216 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.216 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.217 DEBUG:boto:Method: GET 2023-10-11T15:27:16.217 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.217 DEBUG:boto:Data: 2023-10-11T15:27:16.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.219 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.219 DEBUG:boto:Params: {} 2023-10-11T15:27:16.220 DEBUG:boto:Token: None 2023-10-11T15:27:16.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0= 2023-10-11T15:27:16.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0=', 'Content-Length': '0'} 2023-10-11T15:27:16.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc96277a99c4c14-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.224 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.224 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.225 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.225 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.225 DEBUG:boto:Method: GET 2023-10-11T15:27:16.226 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.226 DEBUG:boto:Data: 2023-10-11T15:27:16.227 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.228 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.228 DEBUG:boto:Params: {} 2023-10-11T15:27:16.229 DEBUG:boto:Token: None 2023-10-11T15:27:16.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0= 2023-10-11T15:27:16.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0=', 'Content-Length': '0'} 2023-10-11T15:27:16.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1df202cd7f9bbc5-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.278 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.279 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.279 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.279 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.280 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.280 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.281 DEBUG:boto:Data: 2023-10-11T15:27:16.281 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.282 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.282 DEBUG:boto:Params: {} 2023-10-11T15:27:16.283 DEBUG:boto:Token: None 2023-10-11T15:27:16.283 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xxB6Q1LPoIm0eRG+Dtt5qbqVBIw= 2023-10-11T15:27:16.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xxB6Q1LPoIm0eRG+Dtt5qbqVBIw=', 'Content-Length': '0'} 2023-10-11T15:27:16.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec4dbf421782093-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.287 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.287 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.287 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.288 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.288 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.289 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.289 DEBUG:boto:Data: 2023-10-11T15:27:16.289 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.290 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.291 DEBUG:boto:Params: {} 2023-10-11T15:27:16.291 DEBUG:boto:Token: None 2023-10-11T15:27:16.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xxB6Q1LPoIm0eRG+Dtt5qbqVBIw= 2023-10-11T15:27:16.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xxB6Q1LPoIm0eRG+Dtt5qbqVBIw=', 'Content-Length': '0'} 2023-10-11T15:27:16.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be871c12367551f-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:16.295 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.295 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.296 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.296 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.297 DEBUG:boto:Method: GET 2023-10-11T15:27:16.297 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.298 DEBUG:boto:Data: 2023-10-11T15:27:16.298 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.299 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.299 DEBUG:boto:Params: {} 2023-10-11T15:27:16.300 DEBUG:boto:Token: None 2023-10-11T15:27:16.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0= 2023-10-11T15:27:16.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0=', 'Content-Length': '0'} 2023-10-11T15:27:16.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098234cbf8a7c9573-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.304 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.304 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.304 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.305 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.305 DEBUG:boto:Method: GET 2023-10-11T15:27:16.306 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.306 DEBUG:boto:Data: 2023-10-11T15:27:16.306 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.307 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.308 DEBUG:boto:Params: {} 2023-10-11T15:27:16.308 DEBUG:boto:Token: None 2023-10-11T15:27:16.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:16.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0= 2023-10-11T15:27:16.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AkB5RRSvB3N7ONfY0gfJ0tCvaL0=', 'Content-Length': '0'} 2023-10-11T15:27:16.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000781c167c97a1f8ee-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.312 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.313 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.313 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.314 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.314 DEBUG:boto:Method: GET 2023-10-11T15:27:16.314 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.315 DEBUG:boto:Data: 2023-10-11T15:27:16.315 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.316 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.316 DEBUG:boto:Params: {} 2023-10-11T15:27:16.317 DEBUG:boto:Token: None 2023-10-11T15:27:16.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.318 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165b1e6516e4d0c6-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.362 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.363 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.363 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.364 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.364 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.365 DEBUG:boto:Method: GET 2023-10-11T15:27:16.365 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.365 DEBUG:boto:Data: 2023-10-11T15:27:16.366 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.367 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.367 DEBUG:boto:Params: {} 2023-10-11T15:27:16.367 DEBUG:boto:Token: None 2023-10-11T15:27:16.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6c4c49a0596447-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.373 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.374 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.374 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.374 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.375 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:16.375 DEBUG:boto:Data: 2023-10-11T15:27:16.376 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.376 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.377 DEBUG:boto:Params: {} 2023-10-11T15:27:16.377 DEBUG:boto:Token: None 2023-10-11T15:27:16.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23 2023-10-11T15:27:16.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zpjVOOE+zxjt55uzu6UYnSy9bko= 2023-10-11T15:27:16.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zpjVOOE+zxjt55uzu6UYnSy9bko=', 'Content-Length': '0'} 2023-10-11T15:27:16.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aa3340396cc26aa-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.381 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.381 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.381 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.382 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:16.382 DEBUG:boto:Data: 2023-10-11T15:27:16.383 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.383 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.383 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.384 DEBUG:boto:Params: {} 2023-10-11T15:27:16.384 DEBUG:boto:Token: None 2023-10-11T15:27:16.385 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23 2023-10-11T15:27:16.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zpjVOOE+zxjt55uzu6UYnSy9bko= 2023-10-11T15:27:16.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zpjVOOE+zxjt55uzu6UYnSy9bko=', 'Content-Length': '0'} 2023-10-11T15:27:16.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1084d472bf5126c-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:16.388 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.389 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.389 DEBUG:boto:Method: GET 2023-10-11T15:27:16.389 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:16.390 DEBUG:boto:Data: 2023-10-11T15:27:16.390 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.391 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.391 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.391 DEBUG:boto:Params: {} 2023-10-11T15:27:16.392 DEBUG:boto:Token: None 2023-10-11T15:27:16.392 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23 2023-10-11T15:27:16.393 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uBSANyO+I2n5B4GagQuWXdEaKls= 2023-10-11T15:27:16.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uBSANyO+I2n5B4GagQuWXdEaKls=', 'Content-Length': '0'} 2023-10-11T15:27:16.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095eb821b59aa13c5-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.395 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:16.396 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:16.396 DEBUG:boto:Method: GET 2023-10-11T15:27:16.397 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:16.397 DEBUG:boto:Data: 2023-10-11T15:27:16.397 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.398 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.399 DEBUG:boto:Params: {} 2023-10-11T15:27:16.399 DEBUG:boto:Token: None 2023-10-11T15:27:16.400 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj23 2023-10-11T15:27:16.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uBSANyO+I2n5B4GagQuWXdEaKls= 2023-10-11T15:27:16.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uBSANyO+I2n5B4GagQuWXdEaKls=', 'Content-Length': '0'} 2023-10-11T15:27:16.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a26dd8492988aed-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.403 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.404 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.404 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.405 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.405 DEBUG:boto:Method: GET 2023-10-11T15:27:16.405 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.406 DEBUG:boto:Data: 2023-10-11T15:27:16.406 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.407 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.407 DEBUG:boto:Params: {} 2023-10-11T15:27:16.408 DEBUG:boto:Token: None 2023-10-11T15:27:16.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000989beec6e845429a-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.412 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.413 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.413 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.414 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.414 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.415 DEBUG:boto:Method: GET 2023-10-11T15:27:16.415 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:16.416 DEBUG:boto:Data: 2023-10-11T15:27:16.416 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.417 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.417 DEBUG:boto:Params: {} 2023-10-11T15:27:16.418 DEBUG:boto:Token: None 2023-10-11T15:27:16.418 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c015ad1b9ddc71f7-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.466 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:16.468 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.468 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.468 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.469 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.469 DEBUG:boto:Method: GET 2023-10-11T15:27:16.470 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.470 DEBUG:boto:Data: 2023-10-11T15:27:16.471 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.472 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.472 DEBUG:boto:Params: {} 2023-10-11T15:27:16.473 DEBUG:boto:Token: None 2023-10-11T15:27:16.473 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E= 2023-10-11T15:27:16.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E=', 'Content-Length': '0'} 2023-10-11T15:27:16.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017ece23ba23e7389-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.477 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.478 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.478 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.479 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.479 DEBUG:boto:Method: GET 2023-10-11T15:27:16.479 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.480 DEBUG:boto:Data: 2023-10-11T15:27:16.480 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.481 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.481 DEBUG:boto:Params: {} 2023-10-11T15:27:16.482 DEBUG:boto:Token: None 2023-10-11T15:27:16.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E= 2023-10-11T15:27:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E=', 'Content-Length': '0'} 2023-10-11T15:27:16.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0c7f6044cb4e1db-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.487 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.488 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.489 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.489 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.490 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.491 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.492 DEBUG:boto:Data: 2023-10-11T15:27:16.492 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.493 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.493 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.493 DEBUG:boto:Params: {} 2023-10-11T15:27:16.494 DEBUG:boto:Token: None 2023-10-11T15:27:16.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ha+U7MWj1VBva5S6dbV5Q3skqlc= 2023-10-11T15:27:16.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ha+U7MWj1VBva5S6dbV5Q3skqlc=', 'Content-Length': '0'} 2023-10-11T15:27:16.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e879f3f39b1e7d2-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.498 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.498 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.499 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.499 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.499 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.500 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.500 DEBUG:boto:Data: 2023-10-11T15:27:16.501 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.502 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.502 DEBUG:boto:Params: {} 2023-10-11T15:27:16.503 DEBUG:boto:Token: None 2023-10-11T15:27:16.503 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.504 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ha+U7MWj1VBva5S6dbV5Q3skqlc= 2023-10-11T15:27:16.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ha+U7MWj1VBva5S6dbV5Q3skqlc=', 'Content-Length': '0'} 2023-10-11T15:27:16.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7e60f5dc2846d0-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:16.507 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.507 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.508 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.508 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.509 DEBUG:boto:Method: GET 2023-10-11T15:27:16.509 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.510 DEBUG:boto:Data: 2023-10-11T15:27:16.510 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.511 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.511 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.511 DEBUG:boto:Params: {} 2023-10-11T15:27:16.512 DEBUG:boto:Token: None 2023-10-11T15:27:16.513 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.513 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E= 2023-10-11T15:27:16.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E=', 'Content-Length': '0'} 2023-10-11T15:27:16.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3c60d3ed00daf8-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.517 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.517 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.518 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.518 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.518 DEBUG:boto:Method: GET 2023-10-11T15:27:16.519 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.519 DEBUG:boto:Data: 2023-10-11T15:27:16.520 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.521 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.521 DEBUG:boto:Params: {} 2023-10-11T15:27:16.521 DEBUG:boto:Token: None 2023-10-11T15:27:16.522 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:16.523 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E= 2023-10-11T15:27:16.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTefWnkwCr23aQ830qrWKC5RQ6E=', 'Content-Length': '0'} 2023-10-11T15:27:16.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53fd314ad79dc3d-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.525 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.526 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.526 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.527 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.527 DEBUG:boto:Method: GET 2023-10-11T15:27:16.528 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.528 DEBUG:boto:Data: 2023-10-11T15:27:16.528 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.529 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.530 DEBUG:boto:Params: {} 2023-10-11T15:27:16.530 DEBUG:boto:Token: None 2023-10-11T15:27:16.531 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003029deb6591278f4-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.535 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.536 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.536 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.537 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.538 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.538 DEBUG:boto:Method: GET 2023-10-11T15:27:16.539 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.540 DEBUG:boto:Data: 2023-10-11T15:27:16.541 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.541 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.541 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.542 DEBUG:boto:Params: {} 2023-10-11T15:27:16.542 DEBUG:boto:Token: None 2023-10-11T15:27:16.543 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.543 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a2adbb35ed8184-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.548 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.549 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.549 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.550 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.550 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:16.551 DEBUG:boto:Data: 2023-10-11T15:27:16.551 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.552 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.552 DEBUG:boto:Params: {} 2023-10-11T15:27:16.553 DEBUG:boto:Token: None 2023-10-11T15:27:16.553 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29 2023-10-11T15:27:16.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9gtPRMI+gQ2XokD3jQIK/sfnwOg= 2023-10-11T15:27:16.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9gtPRMI+gQ2XokD3jQIK/sfnwOg=', 'Content-Length': '0'} 2023-10-11T15:27:16.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d664eaa3a13e4f-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.556 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.557 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.557 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.558 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:16.558 DEBUG:boto:Data: 2023-10-11T15:27:16.558 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.559 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.560 DEBUG:boto:Params: {} 2023-10-11T15:27:16.560 DEBUG:boto:Token: None 2023-10-11T15:27:16.560 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29 2023-10-11T15:27:16.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9gtPRMI+gQ2XokD3jQIK/sfnwOg= 2023-10-11T15:27:16.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9gtPRMI+gQ2XokD3jQIK/sfnwOg=', 'Content-Length': '0'} 2023-10-11T15:27:16.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a246161ac5aaf63-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:16.564 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.564 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.565 DEBUG:boto:Method: GET 2023-10-11T15:27:16.565 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:16.565 DEBUG:boto:Data: 2023-10-11T15:27:16.566 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.567 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.567 DEBUG:boto:Params: {} 2023-10-11T15:27:16.568 DEBUG:boto:Token: None 2023-10-11T15:27:16.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29 2023-10-11T15:27:16.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jW0NVd/BKO2Z7GKPb/NMhe7Sy8c= 2023-10-11T15:27:16.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jW0NVd/BKO2Z7GKPb/NMhe7Sy8c=', 'Content-Length': '0'} 2023-10-11T15:27:16.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39338da1ea9b895-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.572 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:16.572 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:16.573 DEBUG:boto:Method: GET 2023-10-11T15:27:16.573 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:16.573 DEBUG:boto:Data: 2023-10-11T15:27:16.574 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.575 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.575 DEBUG:boto:Params: {} 2023-10-11T15:27:16.576 DEBUG:boto:Token: None 2023-10-11T15:27:16.576 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj29 2023-10-11T15:27:16.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jW0NVd/BKO2Z7GKPb/NMhe7Sy8c= 2023-10-11T15:27:16.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jW0NVd/BKO2Z7GKPb/NMhe7Sy8c=', 'Content-Length': '0'} 2023-10-11T15:27:16.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faaf6b1f3a89e8e1-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.626 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.627 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.627 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.628 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.628 DEBUG:boto:Method: GET 2023-10-11T15:27:16.629 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.629 DEBUG:boto:Data: 2023-10-11T15:27:16.630 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.630 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.631 DEBUG:boto:Params: {} 2023-10-11T15:27:16.631 DEBUG:boto:Token: None 2023-10-11T15:27:16.632 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073ea4b3666386065-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.636 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.636 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.637 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.637 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.637 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.638 DEBUG:boto:Method: GET 2023-10-11T15:27:16.638 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:16.639 DEBUG:boto:Data: 2023-10-11T15:27:16.639 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.640 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.640 DEBUG:boto:Params: {} 2023-10-11T15:27:16.641 DEBUG:boto:Token: None 2023-10-11T15:27:16.641 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.642 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000515e007b9f6cb041-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.647 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:16.648 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.648 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.648 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.649 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.650 DEBUG:boto:Method: GET 2023-10-11T15:27:16.650 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.650 DEBUG:boto:Data: 2023-10-11T15:27:16.651 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.652 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.652 DEBUG:boto:Params: {} 2023-10-11T15:27:16.653 DEBUG:boto:Token: None 2023-10-11T15:27:16.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.653 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ= 2023-10-11T15:27:16.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ed9a394c9a0734-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.698 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.699 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.699 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.700 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.700 DEBUG:boto:Method: GET 2023-10-11T15:27:16.701 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.701 DEBUG:boto:Data: 2023-10-11T15:27:16.701 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.702 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.702 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.703 DEBUG:boto:Params: {} 2023-10-11T15:27:16.703 DEBUG:boto:Token: None 2023-10-11T15:27:16.704 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.704 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ= 2023-10-11T15:27:16.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f471ec0274b74d18-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.707 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.707 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.708 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.708 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.708 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.709 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.709 DEBUG:boto:Data: 2023-10-11T15:27:16.710 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.710 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.710 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.711 DEBUG:boto:Params: {} 2023-10-11T15:27:16.711 DEBUG:boto:Token: None 2023-10-11T15:27:16.712 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.712 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u1DJeVcyox6Pl5slKXCyEerW6mI= 2023-10-11T15:27:16.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u1DJeVcyox6Pl5slKXCyEerW6mI=', 'Content-Length': '0'} 2023-10-11T15:27:16.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e204a3c3b7bd34-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.715 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.715 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.716 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.716 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.716 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.717 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.717 DEBUG:boto:Data: 2023-10-11T15:27:16.718 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.719 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.719 DEBUG:boto:Params: {} 2023-10-11T15:27:16.719 DEBUG:boto:Token: None 2023-10-11T15:27:16.720 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u1DJeVcyox6Pl5slKXCyEerW6mI= 2023-10-11T15:27:16.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u1DJeVcyox6Pl5slKXCyEerW6mI=', 'Content-Length': '0'} 2023-10-11T15:27:16.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030fc4025bdbeb287-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:16.724 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.724 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.724 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.725 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.725 DEBUG:boto:Method: GET 2023-10-11T15:27:16.726 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.726 DEBUG:boto:Data: 2023-10-11T15:27:16.726 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.727 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.728 DEBUG:boto:Params: {} 2023-10-11T15:27:16.728 DEBUG:boto:Token: None 2023-10-11T15:27:16.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ= 2023-10-11T15:27:16.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047608b5a0938a5d3-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.732 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.733 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.733 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.733 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.734 DEBUG:boto:Method: GET 2023-10-11T15:27:16.734 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.735 DEBUG:boto:Data: 2023-10-11T15:27:16.735 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.735 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.736 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.736 DEBUG:boto:Params: {} 2023-10-11T15:27:16.737 DEBUG:boto:Token: None 2023-10-11T15:27:16.737 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:16.738 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ= 2023-10-11T15:27:16.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W34mxCmV0OZhDuvHMP/EBlcxVIQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcb72f2a4a2f7ff-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.782 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.783 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.783 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.784 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.784 DEBUG:boto:Method: GET 2023-10-11T15:27:16.784 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.785 DEBUG:boto:Data: 2023-10-11T15:27:16.785 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.786 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.787 DEBUG:boto:Params: {} 2023-10-11T15:27:16.787 DEBUG:boto:Token: None 2023-10-11T15:27:16.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ddfa5a37469e23c-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.791 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.792 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.792 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.793 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.793 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.793 DEBUG:boto:Method: GET 2023-10-11T15:27:16.794 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.794 DEBUG:boto:Data: 2023-10-11T15:27:16.795 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.796 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.796 DEBUG:boto:Params: {} 2023-10-11T15:27:16.796 DEBUG:boto:Token: None 2023-10-11T15:27:16.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3ac5b91c2e3473-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.802 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.803 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.803 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.804 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.804 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:16.805 DEBUG:boto:Data: 2023-10-11T15:27:16.805 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.805 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.806 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.806 DEBUG:boto:Params: {} 2023-10-11T15:27:16.807 DEBUG:boto:Token: None 2023-10-11T15:27:16.807 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31 2023-10-11T15:27:16.807 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oz2hEQhNyrrzHDMswoYYkqhRr+Q= 2023-10-11T15:27:16.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oz2hEQhNyrrzHDMswoYYkqhRr+Q=', 'Content-Length': '0'} 2023-10-11T15:27:16.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bf0da4b73774ce-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.810 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.811 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.811 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.811 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:16.812 DEBUG:boto:Data: 2023-10-11T15:27:16.812 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.812 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.813 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.813 DEBUG:boto:Params: {} 2023-10-11T15:27:16.814 DEBUG:boto:Token: None 2023-10-11T15:27:16.814 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31 2023-10-11T15:27:16.815 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Oz2hEQhNyrrzHDMswoYYkqhRr+Q= 2023-10-11T15:27:16.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Oz2hEQhNyrrzHDMswoYYkqhRr+Q=', 'Content-Length': '0'} 2023-10-11T15:27:16.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7facb7304ca529-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:16.818 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.818 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.819 DEBUG:boto:Method: GET 2023-10-11T15:27:16.819 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:16.819 DEBUG:boto:Data: 2023-10-11T15:27:16.820 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.821 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.821 DEBUG:boto:Params: {} 2023-10-11T15:27:16.822 DEBUG:boto:Token: None 2023-10-11T15:27:16.822 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31 2023-10-11T15:27:16.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4gwhITNZxdMXtY75EHpiPZN04E= 2023-10-11T15:27:16.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4gwhITNZxdMXtY75EHpiPZN04E=', 'Content-Length': '0'} 2023-10-11T15:27:16.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aefa34be9d5753f-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.866 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:16.867 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:16.867 DEBUG:boto:Method: GET 2023-10-11T15:27:16.868 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:16.868 DEBUG:boto:Data: 2023-10-11T15:27:16.868 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.869 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.870 DEBUG:boto:Params: {} 2023-10-11T15:27:16.870 DEBUG:boto:Token: None 2023-10-11T15:27:16.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj31 2023-10-11T15:27:16.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i4gwhITNZxdMXtY75EHpiPZN04E= 2023-10-11T15:27:16.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i4gwhITNZxdMXtY75EHpiPZN04E=', 'Content-Length': '0'} 2023-10-11T15:27:16.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027278f590fcaa62f-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.874 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.874 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.875 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.875 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.876 DEBUG:boto:Method: GET 2023-10-11T15:27:16.876 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.877 DEBUG:boto:Data: 2023-10-11T15:27:16.877 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.878 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.878 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.878 DEBUG:boto:Params: {} 2023-10-11T15:27:16.879 DEBUG:boto:Token: None 2023-10-11T15:27:16.879 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.880 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab3c00473388e4f-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.883 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.883 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.884 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.884 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.885 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.885 DEBUG:boto:Method: GET 2023-10-11T15:27:16.886 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:16.886 DEBUG:boto:Data: 2023-10-11T15:27:16.887 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.888 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.888 DEBUG:boto:Params: {} 2023-10-11T15:27:16.888 DEBUG:boto:Token: None 2023-10-11T15:27:16.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064f23cf7a339f5d4-006526bed4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.894 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:16.895 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.896 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.896 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.897 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.897 DEBUG:boto:Method: GET 2023-10-11T15:27:16.898 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.898 DEBUG:boto:Data: 2023-10-11T15:27:16.899 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.899 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.900 DEBUG:boto:Params: {} 2023-10-11T15:27:16.900 DEBUG:boto:Token: None 2023-10-11T15:27:16.901 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ= 2023-10-11T15:27:16.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e286c4bbf1a48b30-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.904 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.905 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.905 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.906 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.906 DEBUG:boto:Method: GET 2023-10-11T15:27:16.906 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.907 DEBUG:boto:Data: 2023-10-11T15:27:16.907 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.908 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.909 DEBUG:boto:Params: {} 2023-10-11T15:27:16.909 DEBUG:boto:Token: None 2023-10-11T15:27:16.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ= 2023-10-11T15:27:16.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24d8467152cb509-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.913 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.913 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.914 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.914 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.915 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.915 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.916 DEBUG:boto:Data: 2023-10-11T15:27:16.916 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.917 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.917 DEBUG:boto:Params: {} 2023-10-11T15:27:16.918 DEBUG:boto:Token: None 2023-10-11T15:27:16.918 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.919 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7EItwiN6eH8O5fBEWHJyyY+O9eE= 2023-10-11T15:27:16.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7EItwiN6eH8O5fBEWHJyyY+O9eE=', 'Content-Length': '0'} 2023-10-11T15:27:16.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337fd65acda53f6d-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.922 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.922 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.923 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.923 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.923 DEBUG:boto:Method: HEAD 2023-10-11T15:27:16.924 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.924 DEBUG:boto:Data: 2023-10-11T15:27:16.925 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.925 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.926 DEBUG:boto:Params: {} 2023-10-11T15:27:16.926 DEBUG:boto:Token: None 2023-10-11T15:27:16.927 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.927 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7EItwiN6eH8O5fBEWHJyyY+O9eE= 2023-10-11T15:27:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7EItwiN6eH8O5fBEWHJyyY+O9eE=', 'Content-Length': '0'} 2023-10-11T15:27:16.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a14e9431d69025b-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:16.930 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.931 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.931 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.931 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.932 DEBUG:boto:Method: GET 2023-10-11T15:27:16.932 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.933 DEBUG:boto:Data: 2023-10-11T15:27:16.933 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.933 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.934 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.934 DEBUG:boto:Params: {} 2023-10-11T15:27:16.935 DEBUG:boto:Token: None 2023-10-11T15:27:16.935 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ= 2023-10-11T15:27:16.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caed215c80a15ae0-006526bed4-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.939 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:16.939 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:16.940 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.940 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.941 DEBUG:boto:Method: GET 2023-10-11T15:27:16.941 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.941 DEBUG:boto:Data: 2023-10-11T15:27:16.942 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:16.943 DEBUG:boto:Port: 8000 2023-10-11T15:27:16.943 DEBUG:boto:Params: {} 2023-10-11T15:27:16.943 DEBUG:boto:Token: None 2023-10-11T15:27:16.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:16.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ= 2023-10-11T15:27:16.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxi4YD7ff8MkEqiKdtteNXWNWNQ=', 'Content-Length': '0'} 2023-10-11T15:27:16.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8d2ce0a8c73089-006526bed4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.947 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.948 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:16.948 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:16.949 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:16.949 DEBUG:boto:Method: GET 2023-10-11T15:27:16.950 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:16.950 DEBUG:boto:Data: 2023-10-11T15:27:16.951 DEBUG:boto:Headers: {} 2023-10-11T15:27:16.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:16.952 DEBUG:boto:Port: 8001 2023-10-11T15:27:16.952 DEBUG:boto:Params: {} 2023-10-11T15:27:16.953 DEBUG:boto:Token: None 2023-10-11T15:27:16.953 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:16 GMT /dghqvd-59/ 2023-10-11T15:27:16.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0= 2023-10-11T15:27:16.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:16 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KsPu09tyC3uMXuM+Kwnywv9wlJ0=', 'Content-Length': '0'} 2023-10-11T15:27:16.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078291dee463575f0-006526bed4-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:16.998 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:16.999 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:16.999 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.000 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.000 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.001 DEBUG:boto:Method: GET 2023-10-11T15:27:17.001 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.001 DEBUG:boto:Data: 2023-10-11T15:27:17.002 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.003 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.003 DEBUG:boto:Params: {} 2023-10-11T15:27:17.004 DEBUG:boto:Token: None 2023-10-11T15:27:17.004 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcb374406093798-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.009 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.010 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:17.010 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:17.011 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.011 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:17.011 DEBUG:boto:Data: 2023-10-11T15:27:17.012 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.012 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.013 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.013 DEBUG:boto:Params: {} 2023-10-11T15:27:17.014 DEBUG:boto:Token: None 2023-10-11T15:27:17.014 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj33 2023-10-11T15:27:17.014 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0DMF8znO6VdlxzS9txcoXns/sDc= 2023-10-11T15:27:17.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0DMF8znO6VdlxzS9txcoXns/sDc=', 'Content-Length': '0'} 2023-10-11T15:27:17.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b9bb4c4a8857b1a-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.017 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:17.018 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:17.018 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.019 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:17.019 DEBUG:boto:Data: 2023-10-11T15:27:17.019 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.020 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.021 DEBUG:boto:Params: {} 2023-10-11T15:27:17.021 DEBUG:boto:Token: None 2023-10-11T15:27:17.021 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj33 2023-10-11T15:27:17.022 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0DMF8znO6VdlxzS9txcoXns/sDc= 2023-10-11T15:27:17.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0DMF8znO6VdlxzS9txcoXns/sDc=', 'Content-Length': '0'} 2023-10-11T15:27:17.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d22931efd4c50c-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:17.026 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:17.026 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:17.026 DEBUG:boto:Method: GET 2023-10-11T15:27:17.027 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:17.027 DEBUG:boto:Data: 2023-10-11T15:27:17.028 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.028 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.029 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.029 DEBUG:boto:Params: {} 2023-10-11T15:27:17.030 DEBUG:boto:Token: None 2023-10-11T15:27:17.030 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj33 2023-10-11T15:27:17.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ydViDBnI3EeZaDKxumKylv0pEo= 2023-10-11T15:27:17.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ydViDBnI3EeZaDKxumKylv0pEo=', 'Content-Length': '0'} 2023-10-11T15:27:17.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a265cf080a61a414-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.034 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:17.034 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:17.034 DEBUG:boto:Method: GET 2023-10-11T15:27:17.035 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:17.035 DEBUG:boto:Data: 2023-10-11T15:27:17.036 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.037 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.037 DEBUG:boto:Params: {} 2023-10-11T15:27:17.038 DEBUG:boto:Token: None 2023-10-11T15:27:17.038 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj33 2023-10-11T15:27:17.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ydViDBnI3EeZaDKxumKylv0pEo= 2023-10-11T15:27:17.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ydViDBnI3EeZaDKxumKylv0pEo=', 'Content-Length': '0'} 2023-10-11T15:27:17.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbacbaf1a39aada6-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.042 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.042 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.043 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.043 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.044 DEBUG:boto:Method: GET 2023-10-11T15:27:17.044 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.044 DEBUG:boto:Data: 2023-10-11T15:27:17.045 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.046 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.046 DEBUG:boto:Params: {} 2023-10-11T15:27:17.046 DEBUG:boto:Token: None 2023-10-11T15:27:17.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.047 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d1e50ef4e434a0-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.051 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.052 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.052 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.052 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.053 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.053 DEBUG:boto:Method: GET 2023-10-11T15:27:17.054 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:17.054 DEBUG:boto:Data: 2023-10-11T15:27:17.055 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.055 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.056 DEBUG:boto:Params: {} 2023-10-11T15:27:17.056 DEBUG:boto:Token: None 2023-10-11T15:27:17.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76ec910b6ec9b42-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.062 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:17.063 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.064 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.064 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.065 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.065 DEBUG:boto:Method: GET 2023-10-11T15:27:17.066 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.066 DEBUG:boto:Data: 2023-10-11T15:27:17.067 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.067 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.067 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.068 DEBUG:boto:Params: {} 2023-10-11T15:27:17.068 DEBUG:boto:Token: None 2023-10-11T15:27:17.069 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.069 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0= 2023-10-11T15:27:17.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0=', 'Content-Length': '0'} 2023-10-11T15:27:17.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006050e7163da4b965-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.073 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.073 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.073 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.074 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.074 DEBUG:boto:Method: GET 2023-10-11T15:27:17.075 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.075 DEBUG:boto:Data: 2023-10-11T15:27:17.076 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.076 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.077 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.077 DEBUG:boto:Params: {} 2023-10-11T15:27:17.078 DEBUG:boto:Token: None 2023-10-11T15:27:17.078 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.079 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0= 2023-10-11T15:27:17.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0=', 'Content-Length': '0'} 2023-10-11T15:27:17.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094eae1f8bd8a2145-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.081 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.082 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.082 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.083 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.083 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.084 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.084 DEBUG:boto:Data: 2023-10-11T15:27:17.084 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.085 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.086 DEBUG:boto:Params: {} 2023-10-11T15:27:17.086 DEBUG:boto:Token: None 2023-10-11T15:27:17.087 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B5WD55B4Y51+/BQxNfPKFcBUGKM= 2023-10-11T15:27:17.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B5WD55B4Y51+/BQxNfPKFcBUGKM=', 'Content-Length': '0'} 2023-10-11T15:27:17.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3953b22a67481e-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.090 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.091 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.091 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.091 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.092 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.092 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.093 DEBUG:boto:Data: 2023-10-11T15:27:17.093 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.094 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.094 DEBUG:boto:Params: {} 2023-10-11T15:27:17.095 DEBUG:boto:Token: None 2023-10-11T15:27:17.095 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B5WD55B4Y51+/BQxNfPKFcBUGKM= 2023-10-11T15:27:17.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B5WD55B4Y51+/BQxNfPKFcBUGKM=', 'Content-Length': '0'} 2023-10-11T15:27:17.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1869fa8f4a10942-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:17.099 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.100 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.100 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.101 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.101 DEBUG:boto:Method: GET 2023-10-11T15:27:17.101 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.102 DEBUG:boto:Data: 2023-10-11T15:27:17.102 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.103 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.103 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.104 DEBUG:boto:Params: {} 2023-10-11T15:27:17.104 DEBUG:boto:Token: None 2023-10-11T15:27:17.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0= 2023-10-11T15:27:17.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0=', 'Content-Length': '0'} 2023-10-11T15:27:17.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000243cd087984a4622-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.108 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.109 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.109 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.109 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.110 DEBUG:boto:Method: GET 2023-10-11T15:27:17.110 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.111 DEBUG:boto:Data: 2023-10-11T15:27:17.111 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.112 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.113 DEBUG:boto:Params: {} 2023-10-11T15:27:17.113 DEBUG:boto:Token: None 2023-10-11T15:27:17.114 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:17.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0= 2023-10-11T15:27:17.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GnlWr9W5H5qFAGl5sstO/pWDG/0=', 'Content-Length': '0'} 2023-10-11T15:27:17.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c231340b53a02c-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.117 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.118 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.118 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.119 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.119 DEBUG:boto:Method: GET 2023-10-11T15:27:17.120 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.120 DEBUG:boto:Data: 2023-10-11T15:27:17.121 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.122 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.122 DEBUG:boto:Params: {} 2023-10-11T15:27:17.123 DEBUG:boto:Token: None 2023-10-11T15:27:17.123 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c994485b25e6da8f-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.127 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.128 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.128 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.129 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.129 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.129 DEBUG:boto:Method: GET 2023-10-11T15:27:17.130 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.130 DEBUG:boto:Data: 2023-10-11T15:27:17.131 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.132 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.132 DEBUG:boto:Params: {} 2023-10-11T15:27:17.133 DEBUG:boto:Token: None 2023-10-11T15:27:17.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f8d544cf1003475-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.139 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.139 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.140 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.140 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.141 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:17.141 DEBUG:boto:Data: 2023-10-11T15:27:17.142 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.142 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.143 DEBUG:boto:Params: {} 2023-10-11T15:27:17.143 DEBUG:boto:Token: None 2023-10-11T15:27:17.144 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34 2023-10-11T15:27:17.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IMETXKZkidWgGDITyqXn0xA3Sf0= 2023-10-11T15:27:17.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IMETXKZkidWgGDITyqXn0xA3Sf0=', 'Content-Length': '0'} 2023-10-11T15:27:17.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083cf85d64b7b849a-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.147 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.148 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.148 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.149 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:17.149 DEBUG:boto:Data: 2023-10-11T15:27:17.149 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.150 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.151 DEBUG:boto:Params: {} 2023-10-11T15:27:17.151 DEBUG:boto:Token: None 2023-10-11T15:27:17.152 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34 2023-10-11T15:27:17.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IMETXKZkidWgGDITyqXn0xA3Sf0= 2023-10-11T15:27:17.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IMETXKZkidWgGDITyqXn0xA3Sf0=', 'Content-Length': '0'} 2023-10-11T15:27:17.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004850f14dd2e1b9ac-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:17.155 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.156 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.156 DEBUG:boto:Method: GET 2023-10-11T15:27:17.156 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:17.157 DEBUG:boto:Data: 2023-10-11T15:27:17.157 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.158 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.158 DEBUG:boto:Params: {} 2023-10-11T15:27:17.159 DEBUG:boto:Token: None 2023-10-11T15:27:17.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34 2023-10-11T15:27:17.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4Ppgayubshu0bjnlm8KNv5j2VLc= 2023-10-11T15:27:17.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Ppgayubshu0bjnlm8KNv5j2VLc=', 'Content-Length': '0'} 2023-10-11T15:27:17.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a47a0573fc38996-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.206 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:17.207 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:17.207 DEBUG:boto:Method: GET 2023-10-11T15:27:17.208 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:17.208 DEBUG:boto:Data: 2023-10-11T15:27:17.209 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.210 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.210 DEBUG:boto:Params: {} 2023-10-11T15:27:17.211 DEBUG:boto:Token: None 2023-10-11T15:27:17.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj34 2023-10-11T15:27:17.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4Ppgayubshu0bjnlm8KNv5j2VLc= 2023-10-11T15:27:17.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Ppgayubshu0bjnlm8KNv5j2VLc=', 'Content-Length': '0'} 2023-10-11T15:27:17.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb0a6fac082cac79-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.215 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.215 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.216 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.216 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.216 DEBUG:boto:Method: GET 2023-10-11T15:27:17.217 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.217 DEBUG:boto:Data: 2023-10-11T15:27:17.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.219 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.219 DEBUG:boto:Params: {} 2023-10-11T15:27:17.219 DEBUG:boto:Token: None 2023-10-11T15:27:17.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059eafc71bd1cb1c8-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.224 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.225 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.225 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.226 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.226 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.226 DEBUG:boto:Method: GET 2023-10-11T15:27:17.227 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:17.227 DEBUG:boto:Data: 2023-10-11T15:27:17.228 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.228 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.229 DEBUG:boto:Params: {} 2023-10-11T15:27:17.229 DEBUG:boto:Token: None 2023-10-11T15:27:17.230 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a0ea889287b8109-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.235 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:17.236 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.236 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.237 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.237 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.238 DEBUG:boto:Method: GET 2023-10-11T15:27:17.238 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.239 DEBUG:boto:Data: 2023-10-11T15:27:17.239 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.239 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.240 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.240 DEBUG:boto:Params: {} 2023-10-11T15:27:17.241 DEBUG:boto:Token: None 2023-10-11T15:27:17.241 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g= 2023-10-11T15:27:17.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g=', 'Content-Length': '0'} 2023-10-11T15:27:17.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a4c7335b94b12e-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.244 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.245 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.245 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.246 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.246 DEBUG:boto:Method: GET 2023-10-11T15:27:17.246 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.247 DEBUG:boto:Data: 2023-10-11T15:27:17.247 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.248 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.248 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.249 DEBUG:boto:Params: {} 2023-10-11T15:27:17.249 DEBUG:boto:Token: None 2023-10-11T15:27:17.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.250 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g= 2023-10-11T15:27:17.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g=', 'Content-Length': '0'} 2023-10-11T15:27:17.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2bfb85fa701d95-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.253 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.253 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.253 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.254 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.254 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.255 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.255 DEBUG:boto:Data: 2023-10-11T15:27:17.255 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.256 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.256 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.256 DEBUG:boto:Params: {} 2023-10-11T15:27:17.257 DEBUG:boto:Token: None 2023-10-11T15:27:17.257 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.258 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z0bg7wARZUdEf/TnA7aSbI15ZNk= 2023-10-11T15:27:17.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z0bg7wARZUdEf/TnA7aSbI15ZNk=', 'Content-Length': '0'} 2023-10-11T15:27:17.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2e41a686fa9ef0-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.261 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.261 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.262 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.262 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.262 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.263 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.263 DEBUG:boto:Data: 2023-10-11T15:27:17.264 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.264 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.265 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.265 DEBUG:boto:Params: {} 2023-10-11T15:27:17.265 DEBUG:boto:Token: None 2023-10-11T15:27:17.266 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.266 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z0bg7wARZUdEf/TnA7aSbI15ZNk= 2023-10-11T15:27:17.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z0bg7wARZUdEf/TnA7aSbI15ZNk=', 'Content-Length': '0'} 2023-10-11T15:27:17.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b9044444aa32739-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:17.269 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.270 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.270 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.271 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.271 DEBUG:boto:Method: GET 2023-10-11T15:27:17.272 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.272 DEBUG:boto:Data: 2023-10-11T15:27:17.273 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.273 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.274 DEBUG:boto:Params: {} 2023-10-11T15:27:17.274 DEBUG:boto:Token: None 2023-10-11T15:27:17.275 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g= 2023-10-11T15:27:17.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g=', 'Content-Length': '0'} 2023-10-11T15:27:17.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c5991c5e7ba4f2-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.278 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.279 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.279 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.280 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.280 DEBUG:boto:Method: GET 2023-10-11T15:27:17.280 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.281 DEBUG:boto:Data: 2023-10-11T15:27:17.281 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.282 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.282 DEBUG:boto:Params: {} 2023-10-11T15:27:17.283 DEBUG:boto:Token: None 2023-10-11T15:27:17.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:17.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g= 2023-10-11T15:27:17.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S3a1IJLvKjbBMWCjBpl5fVjv+4g=', 'Content-Length': '0'} 2023-10-11T15:27:17.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5fe484d9aa8b70-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.287 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.287 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.288 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.289 DEBUG:boto:Method: GET 2023-10-11T15:27:17.289 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.289 DEBUG:boto:Data: 2023-10-11T15:27:17.290 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.291 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.291 DEBUG:boto:Params: {} 2023-10-11T15:27:17.292 DEBUG:boto:Token: None 2023-10-11T15:27:17.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000830e006e0c46bbc4-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.296 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.296 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.297 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.297 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.298 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.298 DEBUG:boto:Method: GET 2023-10-11T15:27:17.299 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.299 DEBUG:boto:Data: 2023-10-11T15:27:17.299 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.300 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.300 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.301 DEBUG:boto:Params: {} 2023-10-11T15:27:17.301 DEBUG:boto:Token: None 2023-10-11T15:27:17.302 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000469434c73dda6583-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.307 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.308 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.308 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.309 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.309 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:17.309 DEBUG:boto:Data: 2023-10-11T15:27:17.310 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.310 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.311 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.311 DEBUG:boto:Params: {} 2023-10-11T15:27:17.311 DEBUG:boto:Token: None 2023-10-11T15:27:17.312 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37 2023-10-11T15:27:17.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QseFb8K/q2bxdWtxfzDESTEoxL0= 2023-10-11T15:27:17.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QseFb8K/q2bxdWtxfzDESTEoxL0=', 'Content-Length': '0'} 2023-10-11T15:27:17.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7a726c6ff554eb-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.316 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.316 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.317 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.317 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:17.317 DEBUG:boto:Data: 2023-10-11T15:27:17.318 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.319 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.319 DEBUG:boto:Params: {} 2023-10-11T15:27:17.319 DEBUG:boto:Token: None 2023-10-11T15:27:17.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37 2023-10-11T15:27:17.321 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QseFb8K/q2bxdWtxfzDESTEoxL0= 2023-10-11T15:27:17.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QseFb8K/q2bxdWtxfzDESTEoxL0=', 'Content-Length': '0'} 2023-10-11T15:27:17.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3fa2530e75a4d7f-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:17.324 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.324 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.325 DEBUG:boto:Method: GET 2023-10-11T15:27:17.325 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:17.326 DEBUG:boto:Data: 2023-10-11T15:27:17.326 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.327 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.327 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.328 DEBUG:boto:Params: {} 2023-10-11T15:27:17.328 DEBUG:boto:Token: None 2023-10-11T15:27:17.328 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37 2023-10-11T15:27:17.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r0bkImge5dmG1FEdJqPsl7jryNk= 2023-10-11T15:27:17.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r0bkImge5dmG1FEdJqPsl7jryNk=', 'Content-Length': '0'} 2023-10-11T15:27:17.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ff40f77a08ccd9-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.374 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:17.375 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:17.376 DEBUG:boto:Method: GET 2023-10-11T15:27:17.376 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:17.376 DEBUG:boto:Data: 2023-10-11T15:27:17.377 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.377 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.378 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.378 DEBUG:boto:Params: {} 2023-10-11T15:27:17.379 DEBUG:boto:Token: None 2023-10-11T15:27:17.379 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj37 2023-10-11T15:27:17.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r0bkImge5dmG1FEdJqPsl7jryNk= 2023-10-11T15:27:17.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r0bkImge5dmG1FEdJqPsl7jryNk=', 'Content-Length': '0'} 2023-10-11T15:27:17.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3d80ba80a7071a-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.383 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.384 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.384 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.385 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.385 DEBUG:boto:Method: GET 2023-10-11T15:27:17.385 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.386 DEBUG:boto:Data: 2023-10-11T15:27:17.386 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.387 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.388 DEBUG:boto:Params: {} 2023-10-11T15:27:17.388 DEBUG:boto:Token: None 2023-10-11T15:27:17.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8aa318dd8cc35a-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.393 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.394 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.394 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.395 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.395 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.395 DEBUG:boto:Method: GET 2023-10-11T15:27:17.396 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:17.396 DEBUG:boto:Data: 2023-10-11T15:27:17.397 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.397 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.398 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.398 DEBUG:boto:Params: {} 2023-10-11T15:27:17.399 DEBUG:boto:Token: None 2023-10-11T15:27:17.399 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.399 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553249c85c4c4b71-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.404 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:17.405 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.406 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.406 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.407 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.407 DEBUG:boto:Method: GET 2023-10-11T15:27:17.408 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.408 DEBUG:boto:Data: 2023-10-11T15:27:17.408 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.409 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.410 DEBUG:boto:Params: {} 2023-10-11T15:27:17.410 DEBUG:boto:Token: None 2023-10-11T15:27:17.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58= 2023-10-11T15:27:17.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58=', 'Content-Length': '0'} 2023-10-11T15:27:17.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99717f856f4976c-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.414 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.415 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.415 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.416 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.416 DEBUG:boto:Method: GET 2023-10-11T15:27:17.417 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.417 DEBUG:boto:Data: 2023-10-11T15:27:17.418 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.419 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.419 DEBUG:boto:Params: {} 2023-10-11T15:27:17.419 DEBUG:boto:Token: None 2023-10-11T15:27:17.420 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58= 2023-10-11T15:27:17.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58=', 'Content-Length': '0'} 2023-10-11T15:27:17.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aabf443fbb38f52f-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.423 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.424 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.424 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.425 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.425 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.426 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.426 DEBUG:boto:Data: 2023-10-11T15:27:17.427 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.428 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.428 DEBUG:boto:Params: {} 2023-10-11T15:27:17.428 DEBUG:boto:Token: None 2023-10-11T15:27:17.429 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XE/s9x8A9t4eKeDMpBhvtONg+3E= 2023-10-11T15:27:17.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XE/s9x8A9t4eKeDMpBhvtONg+3E=', 'Content-Length': '0'} 2023-10-11T15:27:17.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000809c395b9e84b669-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.432 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.433 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.433 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.434 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.434 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.435 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.435 DEBUG:boto:Data: 2023-10-11T15:27:17.436 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.437 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.437 DEBUG:boto:Params: {} 2023-10-11T15:27:17.437 DEBUG:boto:Token: None 2023-10-11T15:27:17.438 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XE/s9x8A9t4eKeDMpBhvtONg+3E= 2023-10-11T15:27:17.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XE/s9x8A9t4eKeDMpBhvtONg+3E=', 'Content-Length': '0'} 2023-10-11T15:27:17.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f217e9fe37b3e4d-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:17.442 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.442 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.442 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.443 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.443 DEBUG:boto:Method: GET 2023-10-11T15:27:17.444 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.444 DEBUG:boto:Data: 2023-10-11T15:27:17.445 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.445 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.446 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.446 DEBUG:boto:Params: {} 2023-10-11T15:27:17.447 DEBUG:boto:Token: None 2023-10-11T15:27:17.447 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58= 2023-10-11T15:27:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58=', 'Content-Length': '0'} 2023-10-11T15:27:17.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015004286b41ea737-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.451 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.452 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.452 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.453 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.453 DEBUG:boto:Method: GET 2023-10-11T15:27:17.454 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.454 DEBUG:boto:Data: 2023-10-11T15:27:17.454 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.455 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.456 DEBUG:boto:Params: {} 2023-10-11T15:27:17.456 DEBUG:boto:Token: None 2023-10-11T15:27:17.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:17.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58= 2023-10-11T15:27:17.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3mFbJC/EEhQxZLy5pwbvjsbUr58=', 'Content-Length': '0'} 2023-10-11T15:27:17.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028aa13f849625ab1-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.506 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.507 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.508 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.508 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.508 DEBUG:boto:Method: GET 2023-10-11T15:27:17.509 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.509 DEBUG:boto:Data: 2023-10-11T15:27:17.510 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.511 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.511 DEBUG:boto:Params: {} 2023-10-11T15:27:17.512 DEBUG:boto:Token: None 2023-10-11T15:27:17.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a1030827194340-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.516 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.517 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.517 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.517 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.518 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.518 DEBUG:boto:Method: GET 2023-10-11T15:27:17.519 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.519 DEBUG:boto:Data: 2023-10-11T15:27:17.520 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.520 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.521 DEBUG:boto:Params: {} 2023-10-11T15:27:17.521 DEBUG:boto:Token: None 2023-10-11T15:27:17.522 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097b9e60f7c78d627-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.527 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.528 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.528 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.529 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.529 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:17.530 DEBUG:boto:Data: 2023-10-11T15:27:17.530 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.531 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.531 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.531 DEBUG:boto:Params: {} 2023-10-11T15:27:17.532 DEBUG:boto:Token: None 2023-10-11T15:27:17.532 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38 2023-10-11T15:27:17.533 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MpI/pWGee4fR3JbWGg00lt5HKVU= 2023-10-11T15:27:17.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MpI/pWGee4fR3JbWGg00lt5HKVU=', 'Content-Length': '0'} 2023-10-11T15:27:17.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739e352ee0f42b6c-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.536 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.536 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.537 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.537 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:17.537 DEBUG:boto:Data: 2023-10-11T15:27:17.538 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.539 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.539 DEBUG:boto:Params: {} 2023-10-11T15:27:17.539 DEBUG:boto:Token: None 2023-10-11T15:27:17.540 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38 2023-10-11T15:27:17.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MpI/pWGee4fR3JbWGg00lt5HKVU= 2023-10-11T15:27:17.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MpI/pWGee4fR3JbWGg00lt5HKVU=', 'Content-Length': '0'} 2023-10-11T15:27:17.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac4d510f5eead9d-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:17.544 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.544 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.545 DEBUG:boto:Method: GET 2023-10-11T15:27:17.545 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:17.545 DEBUG:boto:Data: 2023-10-11T15:27:17.546 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.547 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.547 DEBUG:boto:Params: {} 2023-10-11T15:27:17.548 DEBUG:boto:Token: None 2023-10-11T15:27:17.548 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38 2023-10-11T15:27:17.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:elysiZoBirYHwnKgu6D3EAHnNa0= 2023-10-11T15:27:17.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:elysiZoBirYHwnKgu6D3EAHnNa0=', 'Content-Length': '0'} 2023-10-11T15:27:17.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2cc5e7cc3a520f-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.598 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:17.599 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:17.599 DEBUG:boto:Method: GET 2023-10-11T15:27:17.600 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:17.600 DEBUG:boto:Data: 2023-10-11T15:27:17.601 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.602 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.602 DEBUG:boto:Params: {} 2023-10-11T15:27:17.603 DEBUG:boto:Token: None 2023-10-11T15:27:17.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj38 2023-10-11T15:27:17.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:elysiZoBirYHwnKgu6D3EAHnNa0= 2023-10-11T15:27:17.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:elysiZoBirYHwnKgu6D3EAHnNa0=', 'Content-Length': '0'} 2023-10-11T15:27:17.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003807414d688445ab-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.607 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.607 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.608 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.608 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.609 DEBUG:boto:Method: GET 2023-10-11T15:27:17.609 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.610 DEBUG:boto:Data: 2023-10-11T15:27:17.610 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.611 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.612 DEBUG:boto:Params: {} 2023-10-11T15:27:17.612 DEBUG:boto:Token: None 2023-10-11T15:27:17.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009106415c1f4358e4-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.617 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.618 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.618 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.619 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.619 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.619 DEBUG:boto:Method: GET 2023-10-11T15:27:17.620 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:17.620 DEBUG:boto:Data: 2023-10-11T15:27:17.621 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.622 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.622 DEBUG:boto:Params: {} 2023-10-11T15:27:17.622 DEBUG:boto:Token: None 2023-10-11T15:27:17.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa758112870799e3-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.670 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:17.671 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.672 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.672 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.673 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.673 DEBUG:boto:Method: GET 2023-10-11T15:27:17.674 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.674 DEBUG:boto:Data: 2023-10-11T15:27:17.674 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.675 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.675 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.676 DEBUG:boto:Params: {} 2023-10-11T15:27:17.676 DEBUG:boto:Token: None 2023-10-11T15:27:17.677 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg= 2023-10-11T15:27:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg=', 'Content-Length': '0'} 2023-10-11T15:27:17.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000685c1fbc4ee013a1-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.681 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.681 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.681 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.682 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.682 DEBUG:boto:Method: GET 2023-10-11T15:27:17.683 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.683 DEBUG:boto:Data: 2023-10-11T15:27:17.684 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.684 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.685 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.685 DEBUG:boto:Params: {} 2023-10-11T15:27:17.686 DEBUG:boto:Token: None 2023-10-11T15:27:17.686 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg= 2023-10-11T15:27:17.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg=', 'Content-Length': '0'} 2023-10-11T15:27:17.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239965a7c95cc0ba-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.690 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.691 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.691 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.692 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.692 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.693 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.693 DEBUG:boto:Data: 2023-10-11T15:27:17.694 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.694 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.695 DEBUG:boto:Params: {} 2023-10-11T15:27:17.695 DEBUG:boto:Token: None 2023-10-11T15:27:17.696 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tn/RfNDS0zZp9llI8/HzjTmrl2Y= 2023-10-11T15:27:17.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tn/RfNDS0zZp9llI8/HzjTmrl2Y=', 'Content-Length': '0'} 2023-10-11T15:27:17.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee85d352edd655ca-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.699 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.700 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.700 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.700 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.701 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.701 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.702 DEBUG:boto:Data: 2023-10-11T15:27:17.702 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.703 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.704 DEBUG:boto:Params: {} 2023-10-11T15:27:17.704 DEBUG:boto:Token: None 2023-10-11T15:27:17.704 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.705 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tn/RfNDS0zZp9llI8/HzjTmrl2Y= 2023-10-11T15:27:17.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tn/RfNDS0zZp9llI8/HzjTmrl2Y=', 'Content-Length': '0'} 2023-10-11T15:27:17.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db739290936c0281-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:17.708 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.709 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.709 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.710 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.710 DEBUG:boto:Method: GET 2023-10-11T15:27:17.710 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.711 DEBUG:boto:Data: 2023-10-11T15:27:17.711 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.712 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.712 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.712 DEBUG:boto:Params: {} 2023-10-11T15:27:17.713 DEBUG:boto:Token: None 2023-10-11T15:27:17.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg= 2023-10-11T15:27:17.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg=', 'Content-Length': '0'} 2023-10-11T15:27:17.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f69f4e58d042da-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.717 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.718 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.718 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.718 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.719 DEBUG:boto:Method: GET 2023-10-11T15:27:17.719 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.720 DEBUG:boto:Data: 2023-10-11T15:27:17.720 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.721 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.721 DEBUG:boto:Params: {} 2023-10-11T15:27:17.722 DEBUG:boto:Token: None 2023-10-11T15:27:17.722 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:17.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg= 2023-10-11T15:27:17.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A5wnc499xgxT3ENdZI9xvH1UAXg=', 'Content-Length': '0'} 2023-10-11T15:27:17.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a2943e32dea3a9-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.726 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.727 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.727 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.728 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.728 DEBUG:boto:Method: GET 2023-10-11T15:27:17.728 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.729 DEBUG:boto:Data: 2023-10-11T15:27:17.729 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.730 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.731 DEBUG:boto:Params: {} 2023-10-11T15:27:17.731 DEBUG:boto:Token: None 2023-10-11T15:27:17.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adc683ca50d57779-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.735 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.736 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.737 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.737 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.737 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.738 DEBUG:boto:Method: GET 2023-10-11T15:27:17.738 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.739 DEBUG:boto:Data: 2023-10-11T15:27:17.739 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.740 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.740 DEBUG:boto:Params: {} 2023-10-11T15:27:17.741 DEBUG:boto:Token: None 2023-10-11T15:27:17.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055460ab3fdba9218-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.747 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.747 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.748 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.748 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.749 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:17.749 DEBUG:boto:Data: 2023-10-11T15:27:17.749 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.750 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.751 DEBUG:boto:Params: {} 2023-10-11T15:27:17.751 DEBUG:boto:Token: None 2023-10-11T15:27:17.751 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44 2023-10-11T15:27:17.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qf5c0fgtu8cTp5qM14yCGOnqqlM= 2023-10-11T15:27:17.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qf5c0fgtu8cTp5qM14yCGOnqqlM=', 'Content-Length': '0'} 2023-10-11T15:27:17.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003febfeadbd8b6d29-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.755 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.755 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.756 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.756 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:17.757 DEBUG:boto:Data: 2023-10-11T15:27:17.757 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.757 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.758 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.758 DEBUG:boto:Params: {} 2023-10-11T15:27:17.759 DEBUG:boto:Token: None 2023-10-11T15:27:17.759 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44 2023-10-11T15:27:17.759 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qf5c0fgtu8cTp5qM14yCGOnqqlM= 2023-10-11T15:27:17.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qf5c0fgtu8cTp5qM14yCGOnqqlM=', 'Content-Length': '0'} 2023-10-11T15:27:17.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a150acb669b5c143-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:17.763 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.763 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.764 DEBUG:boto:Method: GET 2023-10-11T15:27:17.764 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:17.765 DEBUG:boto:Data: 2023-10-11T15:27:17.765 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.766 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.766 DEBUG:boto:Params: {} 2023-10-11T15:27:17.767 DEBUG:boto:Token: None 2023-10-11T15:27:17.767 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44 2023-10-11T15:27:17.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:enkmgv4L8tfboqzE1CVwvKKub4s= 2023-10-11T15:27:17.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:enkmgv4L8tfboqzE1CVwvKKub4s=', 'Content-Length': '0'} 2023-10-11T15:27:17.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e3c4ef939d8f1b2-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.814 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:17.815 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:17.815 DEBUG:boto:Method: GET 2023-10-11T15:27:17.816 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:17.816 DEBUG:boto:Data: 2023-10-11T15:27:17.816 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.817 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.818 DEBUG:boto:Params: {} 2023-10-11T15:27:17.818 DEBUG:boto:Token: None 2023-10-11T15:27:17.819 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj44 2023-10-11T15:27:17.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:enkmgv4L8tfboqzE1CVwvKKub4s= 2023-10-11T15:27:17.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:enkmgv4L8tfboqzE1CVwvKKub4s=', 'Content-Length': '0'} 2023-10-11T15:27:17.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061347c8af4e88b7b-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.822 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.823 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.823 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.824 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.824 DEBUG:boto:Method: GET 2023-10-11T15:27:17.825 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.825 DEBUG:boto:Data: 2023-10-11T15:27:17.826 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.826 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.827 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.827 DEBUG:boto:Params: {} 2023-10-11T15:27:17.828 DEBUG:boto:Token: None 2023-10-11T15:27:17.828 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7e8418242cd39d-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.832 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.833 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.833 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.834 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.834 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.834 DEBUG:boto:Method: GET 2023-10-11T15:27:17.835 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:17.835 DEBUG:boto:Data: 2023-10-11T15:27:17.836 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.836 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.837 DEBUG:boto:Params: {} 2023-10-11T15:27:17.837 DEBUG:boto:Token: None 2023-10-11T15:27:17.838 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c6d2b37f4c8269-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.843 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:17.844 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.845 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.845 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.845 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.846 DEBUG:boto:Method: GET 2023-10-11T15:27:17.846 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.847 DEBUG:boto:Data: 2023-10-11T15:27:17.847 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.848 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.849 DEBUG:boto:Params: {} 2023-10-11T15:27:17.849 DEBUG:boto:Token: None 2023-10-11T15:27:17.850 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.850 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI= 2023-10-11T15:27:17.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI=', 'Content-Length': '0'} 2023-10-11T15:27:17.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33a756f9901bbf1-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.853 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.854 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.854 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.855 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.855 DEBUG:boto:Method: GET 2023-10-11T15:27:17.855 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.856 DEBUG:boto:Data: 2023-10-11T15:27:17.856 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.857 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.857 DEBUG:boto:Params: {} 2023-10-11T15:27:17.858 DEBUG:boto:Token: None 2023-10-11T15:27:17.858 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI= 2023-10-11T15:27:17.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI=', 'Content-Length': '0'} 2023-10-11T15:27:17.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000539274a41a08065-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.862 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.863 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.863 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.863 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.864 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.864 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.865 DEBUG:boto:Data: 2023-10-11T15:27:17.865 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.866 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.866 DEBUG:boto:Params: {} 2023-10-11T15:27:17.867 DEBUG:boto:Token: None 2023-10-11T15:27:17.867 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ghe1dveCCm09wivgqPMGCoVbt9w= 2023-10-11T15:27:17.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ghe1dveCCm09wivgqPMGCoVbt9w=', 'Content-Length': '0'} 2023-10-11T15:27:17.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095025544445117b0-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.871 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.871 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.871 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.872 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.872 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.873 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.873 DEBUG:boto:Data: 2023-10-11T15:27:17.874 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.874 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.875 DEBUG:boto:Params: {} 2023-10-11T15:27:17.875 DEBUG:boto:Token: None 2023-10-11T15:27:17.876 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ghe1dveCCm09wivgqPMGCoVbt9w= 2023-10-11T15:27:17.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ghe1dveCCm09wivgqPMGCoVbt9w=', 'Content-Length': '0'} 2023-10-11T15:27:17.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375fe29212a8f04c-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:17.880 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.880 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.880 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.881 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.881 DEBUG:boto:Method: GET 2023-10-11T15:27:17.882 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.882 DEBUG:boto:Data: 2023-10-11T15:27:17.883 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.883 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.884 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.884 DEBUG:boto:Params: {} 2023-10-11T15:27:17.885 DEBUG:boto:Token: None 2023-10-11T15:27:17.885 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI= 2023-10-11T15:27:17.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI=', 'Content-Length': '0'} 2023-10-11T15:27:17.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769807c393ed0ae6-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.889 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.890 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.890 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.891 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.891 DEBUG:boto:Method: GET 2023-10-11T15:27:17.892 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.892 DEBUG:boto:Data: 2023-10-11T15:27:17.893 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.893 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.894 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.894 DEBUG:boto:Params: {} 2023-10-11T15:27:17.895 DEBUG:boto:Token: None 2023-10-11T15:27:17.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:17.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI= 2023-10-11T15:27:17.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sLSpQYJBgPSjRJXqSJnhAtQJ/mI=', 'Content-Length': '0'} 2023-10-11T15:27:17.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085978c8a2abb9276-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.899 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.900 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.900 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.900 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.901 DEBUG:boto:Method: GET 2023-10-11T15:27:17.901 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.902 DEBUG:boto:Data: 2023-10-11T15:27:17.902 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.903 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.903 DEBUG:boto:Params: {} 2023-10-11T15:27:17.904 DEBUG:boto:Token: None 2023-10-11T15:27:17.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c96cbf48e1e47a-006526bed5-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.908 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.909 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:17.909 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:17.910 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.910 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.911 DEBUG:boto:Method: GET 2023-10-11T15:27:17.911 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:17.912 DEBUG:boto:Data: 2023-10-11T15:27:17.912 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.913 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.914 DEBUG:boto:Params: {} 2023-10-11T15:27:17.914 DEBUG:boto:Token: None 2023-10-11T15:27:17.915 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/ 2023-10-11T15:27:17.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o= 2023-10-11T15:27:17.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HrhuxIyEYUQFnLe1e9Al0M8NC8o=', 'Content-Length': '0'} 2023-10-11T15:27:17.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c2bbfe3c80aab5-006526bed5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.962 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:17.963 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.963 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.964 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.964 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:17.965 DEBUG:boto:Data: 2023-10-11T15:27:17.965 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.966 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.966 DEBUG:boto:Params: {} 2023-10-11T15:27:17.967 DEBUG:boto:Token: None 2023-10-11T15:27:17.967 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46 2023-10-11T15:27:17.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:USElRKtF/fAqKshgriVvz+440Nc= 2023-10-11T15:27:17.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:USElRKtF/fAqKshgriVvz+440Nc=', 'Content-Length': '0'} 2023-10-11T15:27:17.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005413470f2af0942b-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.971 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.971 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.972 DEBUG:boto:Method: HEAD 2023-10-11T15:27:17.972 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:17.972 DEBUG:boto:Data: 2023-10-11T15:27:17.973 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.973 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:17.974 DEBUG:boto:Port: 8000 2023-10-11T15:27:17.974 DEBUG:boto:Params: {} 2023-10-11T15:27:17.975 DEBUG:boto:Token: None 2023-10-11T15:27:17.975 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46 2023-10-11T15:27:17.976 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:USElRKtF/fAqKshgriVvz+440Nc= 2023-10-11T15:27:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:USElRKtF/fAqKshgriVvz+440Nc=', 'Content-Length': '0'} 2023-10-11T15:27:17.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cfba857a7f894b-006526bed5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:17.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:17.981 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:17.982 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:17.983 DEBUG:boto:Method: GET 2023-10-11T15:27:17.983 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:17.984 DEBUG:boto:Data: 2023-10-11T15:27:17.984 DEBUG:boto:Headers: {} 2023-10-11T15:27:17.984 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:17.985 DEBUG:boto:Port: 8001 2023-10-11T15:27:17.985 DEBUG:boto:Params: {} 2023-10-11T15:27:17.986 DEBUG:boto:Token: None 2023-10-11T15:27:17.987 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:17 GMT /dghqvd-59/obj46 2023-10-11T15:27:17.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vd3kmVDlgdvR0CWgNQHdtcHGhPY= 2023-10-11T15:27:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vd3kmVDlgdvR0CWgNQHdtcHGhPY=', 'Content-Length': '0'} 2023-10-11T15:27:17.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002949442c6d3b78-006526bed5-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.030 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:18.031 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:18.031 DEBUG:boto:Method: GET 2023-10-11T15:27:18.032 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:18.032 DEBUG:boto:Data: 2023-10-11T15:27:18.033 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.034 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.034 DEBUG:boto:Params: {} 2023-10-11T15:27:18.034 DEBUG:boto:Token: None 2023-10-11T15:27:18.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj46 2023-10-11T15:27:18.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yRQRWxXNXOouU8mkjcI7LpEdkXE= 2023-10-11T15:27:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yRQRWxXNXOouU8mkjcI7LpEdkXE=', 'Content-Length': '0'} 2023-10-11T15:27:18.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330f8d8937ecfdeb-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.038 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.039 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.039 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.040 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.040 DEBUG:boto:Method: GET 2023-10-11T15:27:18.040 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.041 DEBUG:boto:Data: 2023-10-11T15:27:18.041 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.042 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.042 DEBUG:boto:Params: {} 2023-10-11T15:27:18.043 DEBUG:boto:Token: None 2023-10-11T15:27:18.043 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a3f89e31a216fd-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.047 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.048 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.048 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.049 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.049 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.050 DEBUG:boto:Method: GET 2023-10-11T15:27:18.050 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:18.050 DEBUG:boto:Data: 2023-10-11T15:27:18.051 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.052 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.052 DEBUG:boto:Params: {} 2023-10-11T15:27:18.053 DEBUG:boto:Token: None 2023-10-11T15:27:18.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f005e2ce64c70387-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.059 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:18.060 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.060 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.061 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.061 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.061 DEBUG:boto:Method: GET 2023-10-11T15:27:18.062 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.062 DEBUG:boto:Data: 2023-10-11T15:27:18.063 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.064 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.064 DEBUG:boto:Params: {} 2023-10-11T15:27:18.065 DEBUG:boto:Token: None 2023-10-11T15:27:18.065 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM= 2023-10-11T15:27:18.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM=', 'Content-Length': '0'} 2023-10-11T15:27:18.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004207cc7f13e76409-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.069 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.069 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.070 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.070 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.071 DEBUG:boto:Method: GET 2023-10-11T15:27:18.071 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.072 DEBUG:boto:Data: 2023-10-11T15:27:18.072 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.072 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.073 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.073 DEBUG:boto:Params: {} 2023-10-11T15:27:18.074 DEBUG:boto:Token: None 2023-10-11T15:27:18.074 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.075 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM= 2023-10-11T15:27:18.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM=', 'Content-Length': '0'} 2023-10-11T15:27:18.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000071446afec08c7c9-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.078 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.079 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.079 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.079 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.080 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.080 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.081 DEBUG:boto:Data: 2023-10-11T15:27:18.081 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.082 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.082 DEBUG:boto:Params: {} 2023-10-11T15:27:18.083 DEBUG:boto:Token: None 2023-10-11T15:27:18.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KyccI/7MLGPnHfo/kr/9Tgl2o9Y= 2023-10-11T15:27:18.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KyccI/7MLGPnHfo/kr/9Tgl2o9Y=', 'Content-Length': '0'} 2023-10-11T15:27:18.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26c6e936fefe348-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.086 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.087 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.087 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.088 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.088 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.088 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.089 DEBUG:boto:Data: 2023-10-11T15:27:18.089 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.090 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.091 DEBUG:boto:Params: {} 2023-10-11T15:27:18.091 DEBUG:boto:Token: None 2023-10-11T15:27:18.092 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KyccI/7MLGPnHfo/kr/9Tgl2o9Y= 2023-10-11T15:27:18.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KyccI/7MLGPnHfo/kr/9Tgl2o9Y=', 'Content-Length': '0'} 2023-10-11T15:27:18.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e77a708690fd19f3-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:18.095 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.096 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.096 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.097 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.097 DEBUG:boto:Method: GET 2023-10-11T15:27:18.098 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.098 DEBUG:boto:Data: 2023-10-11T15:27:18.098 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.099 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.100 DEBUG:boto:Params: {} 2023-10-11T15:27:18.100 DEBUG:boto:Token: None 2023-10-11T15:27:18.101 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM= 2023-10-11T15:27:18.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM=', 'Content-Length': '0'} 2023-10-11T15:27:18.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a97f57ee28d0efd8-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.104 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.105 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.105 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.106 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.106 DEBUG:boto:Method: GET 2023-10-11T15:27:18.107 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.107 DEBUG:boto:Data: 2023-10-11T15:27:18.108 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.108 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.109 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.109 DEBUG:boto:Params: {} 2023-10-11T15:27:18.109 DEBUG:boto:Token: None 2023-10-11T15:27:18.110 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:18.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM= 2023-10-11T15:27:18.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UsKcVXsIFFctKmik5bai8+wMerM=', 'Content-Length': '0'} 2023-10-11T15:27:18.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4c35ea61a15542-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.154 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.155 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.155 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.156 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.156 DEBUG:boto:Method: GET 2023-10-11T15:27:18.157 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.157 DEBUG:boto:Data: 2023-10-11T15:27:18.158 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.158 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.159 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.159 DEBUG:boto:Params: {} 2023-10-11T15:27:18.160 DEBUG:boto:Token: None 2023-10-11T15:27:18.160 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.161 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008235bbaaa292cfe3-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.164 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.165 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.165 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.166 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.166 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.167 DEBUG:boto:Method: GET 2023-10-11T15:27:18.167 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.168 DEBUG:boto:Data: 2023-10-11T15:27:18.168 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.169 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.170 DEBUG:boto:Params: {} 2023-10-11T15:27:18.170 DEBUG:boto:Token: None 2023-10-11T15:27:18.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6bb2c849a21b7c-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.177 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.177 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.178 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.178 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.179 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:18.179 DEBUG:boto:Data: 2023-10-11T15:27:18.180 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.180 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.180 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.181 DEBUG:boto:Params: {} 2023-10-11T15:27:18.181 DEBUG:boto:Token: None 2023-10-11T15:27:18.182 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51 2023-10-11T15:27:18.182 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0VoT/Cm9akZl8stdkIC67w/2aTk= 2023-10-11T15:27:18.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0VoT/Cm9akZl8stdkIC67w/2aTk=', 'Content-Length': '0'} 2023-10-11T15:27:18.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c94f5579c9f7d3-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.188 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.188 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.189 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.189 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:18.190 DEBUG:boto:Data: 2023-10-11T15:27:18.190 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.190 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.191 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.191 DEBUG:boto:Params: {} 2023-10-11T15:27:18.192 DEBUG:boto:Token: None 2023-10-11T15:27:18.192 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51 2023-10-11T15:27:18.193 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0VoT/Cm9akZl8stdkIC67w/2aTk= 2023-10-11T15:27:18.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0VoT/Cm9akZl8stdkIC67w/2aTk=', 'Content-Length': '0'} 2023-10-11T15:27:18.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96ba6aa2f6b3ec3-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:18.196 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.197 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.197 DEBUG:boto:Method: GET 2023-10-11T15:27:18.198 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:18.198 DEBUG:boto:Data: 2023-10-11T15:27:18.199 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.199 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.199 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.200 DEBUG:boto:Params: {} 2023-10-11T15:27:18.200 DEBUG:boto:Token: None 2023-10-11T15:27:18.201 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51 2023-10-11T15:27:18.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J/EW+9DdgxYZw/XywXCdTIgMsWQ= 2023-10-11T15:27:18.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J/EW+9DdgxYZw/XywXCdTIgMsWQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f140e603f28f184-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.250 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:18.251 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:18.251 DEBUG:boto:Method: GET 2023-10-11T15:27:18.252 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:18.252 DEBUG:boto:Data: 2023-10-11T15:27:18.253 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.253 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.254 DEBUG:boto:Params: {} 2023-10-11T15:27:18.254 DEBUG:boto:Token: None 2023-10-11T15:27:18.255 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj51 2023-10-11T15:27:18.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J/EW+9DdgxYZw/XywXCdTIgMsWQ= 2023-10-11T15:27:18.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J/EW+9DdgxYZw/XywXCdTIgMsWQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6440bb101aac2e7-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.258 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.259 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.259 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.259 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.260 DEBUG:boto:Method: GET 2023-10-11T15:27:18.260 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.261 DEBUG:boto:Data: 2023-10-11T15:27:18.261 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.261 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.262 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.262 DEBUG:boto:Params: {} 2023-10-11T15:27:18.263 DEBUG:boto:Token: None 2023-10-11T15:27:18.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.264 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c37afca070d7c92-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.267 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.268 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.268 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.269 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.269 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.270 DEBUG:boto:Method: GET 2023-10-11T15:27:18.270 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:18.270 DEBUG:boto:Data: 2023-10-11T15:27:18.271 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.272 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.272 DEBUG:boto:Params: {} 2023-10-11T15:27:18.273 DEBUG:boto:Token: None 2023-10-11T15:27:18.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c76fd745617fe600-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.279 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:18.280 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.280 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.280 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.281 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.281 DEBUG:boto:Method: GET 2023-10-11T15:27:18.282 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.282 DEBUG:boto:Data: 2023-10-11T15:27:18.282 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.283 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.283 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.284 DEBUG:boto:Params: {} 2023-10-11T15:27:18.284 DEBUG:boto:Token: None 2023-10-11T15:27:18.285 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ= 2023-10-11T15:27:18.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000029c9a496438386d-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.288 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.288 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.289 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.289 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.290 DEBUG:boto:Method: GET 2023-10-11T15:27:18.290 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.290 DEBUG:boto:Data: 2023-10-11T15:27:18.291 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.292 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.292 DEBUG:boto:Params: {} 2023-10-11T15:27:18.293 DEBUG:boto:Token: None 2023-10-11T15:27:18.293 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ= 2023-10-11T15:27:18.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6104a1c76a4c9ad-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.296 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.297 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.297 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.298 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.298 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.299 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.299 DEBUG:boto:Data: 2023-10-11T15:27:18.299 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.300 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.300 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.301 DEBUG:boto:Params: {} 2023-10-11T15:27:18.301 DEBUG:boto:Token: None 2023-10-11T15:27:18.302 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81kPlW5wPMF6KRXXQJYI4H2d/j0= 2023-10-11T15:27:18.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81kPlW5wPMF6KRXXQJYI4H2d/j0=', 'Content-Length': '0'} 2023-10-11T15:27:18.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd9ec8c250bacfa-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.305 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.305 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.306 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.306 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.306 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.307 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.307 DEBUG:boto:Data: 2023-10-11T15:27:18.308 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.308 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.308 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.309 DEBUG:boto:Params: {} 2023-10-11T15:27:18.309 DEBUG:boto:Token: None 2023-10-11T15:27:18.310 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.310 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81kPlW5wPMF6KRXXQJYI4H2d/j0= 2023-10-11T15:27:18.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81kPlW5wPMF6KRXXQJYI4H2d/j0=', 'Content-Length': '0'} 2023-10-11T15:27:18.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375f0a31a3b1448c-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:18.313 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.314 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.314 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.314 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.315 DEBUG:boto:Method: GET 2023-10-11T15:27:18.315 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.316 DEBUG:boto:Data: 2023-10-11T15:27:18.316 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.317 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.317 DEBUG:boto:Params: {} 2023-10-11T15:27:18.318 DEBUG:boto:Token: None 2023-10-11T15:27:18.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ= 2023-10-11T15:27:18.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c837df207891a2-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.321 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.322 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.322 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.323 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.323 DEBUG:boto:Method: GET 2023-10-11T15:27:18.324 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.324 DEBUG:boto:Data: 2023-10-11T15:27:18.324 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.325 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.325 DEBUG:boto:Params: {} 2023-10-11T15:27:18.326 DEBUG:boto:Token: None 2023-10-11T15:27:18.326 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:18.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ= 2023-10-11T15:27:18.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXwOfrykFjvOLrCC92FhHstueaQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e333623ef848038-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.330 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.330 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.331 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.331 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.332 DEBUG:boto:Method: GET 2023-10-11T15:27:18.332 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.332 DEBUG:boto:Data: 2023-10-11T15:27:18.333 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.334 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.334 DEBUG:boto:Params: {} 2023-10-11T15:27:18.334 DEBUG:boto:Token: None 2023-10-11T15:27:18.335 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0d719e8d072ccff-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.339 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.339 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.340 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.340 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.341 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.341 DEBUG:boto:Method: GET 2023-10-11T15:27:18.341 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.342 DEBUG:boto:Data: 2023-10-11T15:27:18.342 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.343 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.343 DEBUG:boto:Params: {} 2023-10-11T15:27:18.344 DEBUG:boto:Token: None 2023-10-11T15:27:18.344 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3cc4eaae5796ae-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.349 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.350 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.351 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.351 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.351 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:18.352 DEBUG:boto:Data: 2023-10-11T15:27:18.352 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.352 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.353 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.353 DEBUG:boto:Params: {} 2023-10-11T15:27:18.354 DEBUG:boto:Token: None 2023-10-11T15:27:18.354 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55 2023-10-11T15:27:18.354 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ao31zTLO8vT7kVUpYPQ7H1YinS0= 2023-10-11T15:27:18.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ao31zTLO8vT7kVUpYPQ7H1YinS0=', 'Content-Length': '0'} 2023-10-11T15:27:18.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de0db6a33474c95-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.357 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.358 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.358 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.359 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:18.359 DEBUG:boto:Data: 2023-10-11T15:27:18.359 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.360 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.360 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.360 DEBUG:boto:Params: {} 2023-10-11T15:27:18.361 DEBUG:boto:Token: None 2023-10-11T15:27:18.361 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55 2023-10-11T15:27:18.362 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ao31zTLO8vT7kVUpYPQ7H1YinS0= 2023-10-11T15:27:18.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ao31zTLO8vT7kVUpYPQ7H1YinS0=', 'Content-Length': '0'} 2023-10-11T15:27:18.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516e275c67fd105c-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:18.366 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.366 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.367 DEBUG:boto:Method: GET 2023-10-11T15:27:18.367 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:18.368 DEBUG:boto:Data: 2023-10-11T15:27:18.368 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.369 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.369 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.369 DEBUG:boto:Params: {} 2023-10-11T15:27:18.370 DEBUG:boto:Token: None 2023-10-11T15:27:18.370 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55 2023-10-11T15:27:18.371 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63kjg5+w5KoF//uIn9bPTtqw06k= 2023-10-11T15:27:18.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63kjg5+w5KoF//uIn9bPTtqw06k=', 'Content-Length': '0'} 2023-10-11T15:27:18.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0af3aa59dad5cf1-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.418 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:18.419 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:18.419 DEBUG:boto:Method: GET 2023-10-11T15:27:18.420 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:18.420 DEBUG:boto:Data: 2023-10-11T15:27:18.420 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.421 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.422 DEBUG:boto:Params: {} 2023-10-11T15:27:18.422 DEBUG:boto:Token: None 2023-10-11T15:27:18.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj55 2023-10-11T15:27:18.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:63kjg5+w5KoF//uIn9bPTtqw06k= 2023-10-11T15:27:18.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:63kjg5+w5KoF//uIn9bPTtqw06k=', 'Content-Length': '0'} 2023-10-11T15:27:18.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a49effd6f308194-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.426 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.426 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.427 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.427 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.428 DEBUG:boto:Method: GET 2023-10-11T15:27:18.428 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.428 DEBUG:boto:Data: 2023-10-11T15:27:18.429 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.430 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.430 DEBUG:boto:Params: {} 2023-10-11T15:27:18.431 DEBUG:boto:Token: None 2023-10-11T15:27:18.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a623d0b9da2bf9b9-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.435 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.436 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.436 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.437 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.437 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.437 DEBUG:boto:Method: GET 2023-10-11T15:27:18.438 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:18.438 DEBUG:boto:Data: 2023-10-11T15:27:18.439 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.440 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.440 DEBUG:boto:Params: {} 2023-10-11T15:27:18.440 DEBUG:boto:Token: None 2023-10-11T15:27:18.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c97a31f2c4e4e6bf-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.446 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:18.448 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.448 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.448 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.449 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.449 DEBUG:boto:Method: GET 2023-10-11T15:27:18.450 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.450 DEBUG:boto:Data: 2023-10-11T15:27:18.450 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.451 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.452 DEBUG:boto:Params: {} 2023-10-11T15:27:18.452 DEBUG:boto:Token: None 2023-10-11T15:27:18.453 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U= 2023-10-11T15:27:18.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U=', 'Content-Length': '0'} 2023-10-11T15:27:18.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000598a8ab64708deb2-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.456 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.457 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.457 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.457 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.458 DEBUG:boto:Method: GET 2023-10-11T15:27:18.458 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.459 DEBUG:boto:Data: 2023-10-11T15:27:18.459 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.460 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.460 DEBUG:boto:Params: {} 2023-10-11T15:27:18.461 DEBUG:boto:Token: None 2023-10-11T15:27:18.461 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.462 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U= 2023-10-11T15:27:18.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U=', 'Content-Length': '0'} 2023-10-11T15:27:18.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c10717e0ed481ee0-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.465 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.465 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.466 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.466 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.466 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.467 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.467 DEBUG:boto:Data: 2023-10-11T15:27:18.468 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.468 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.469 DEBUG:boto:Params: {} 2023-10-11T15:27:18.469 DEBUG:boto:Token: None 2023-10-11T15:27:18.470 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.470 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DYR+fQT3KLOkaSlyw6h4pfELzbQ= 2023-10-11T15:27:18.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DYR+fQT3KLOkaSlyw6h4pfELzbQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bf3a64902bb83c-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.473 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.473 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.474 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.474 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.474 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.475 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.475 DEBUG:boto:Data: 2023-10-11T15:27:18.475 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.476 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.476 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.476 DEBUG:boto:Params: {} 2023-10-11T15:27:18.477 DEBUG:boto:Token: None 2023-10-11T15:27:18.477 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DYR+fQT3KLOkaSlyw6h4pfELzbQ= 2023-10-11T15:27:18.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DYR+fQT3KLOkaSlyw6h4pfELzbQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6f48f668816b693-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:18.481 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.481 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.482 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.482 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.482 DEBUG:boto:Method: GET 2023-10-11T15:27:18.483 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.483 DEBUG:boto:Data: 2023-10-11T15:27:18.484 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.484 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.484 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.485 DEBUG:boto:Params: {} 2023-10-11T15:27:18.485 DEBUG:boto:Token: None 2023-10-11T15:27:18.486 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.486 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U= 2023-10-11T15:27:18.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U=', 'Content-Length': '0'} 2023-10-11T15:27:18.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f251bbc3ded44a6-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.489 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.489 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.490 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.490 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.491 DEBUG:boto:Method: GET 2023-10-11T15:27:18.491 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.491 DEBUG:boto:Data: 2023-10-11T15:27:18.492 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.492 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.493 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.493 DEBUG:boto:Params: {} 2023-10-11T15:27:18.494 DEBUG:boto:Token: None 2023-10-11T15:27:18.494 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:18.494 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U= 2023-10-11T15:27:18.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wGHYvvuqckCCLydOEbWA7EFbb6U=', 'Content-Length': '0'} 2023-10-11T15:27:18.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c15464e9120bbd25-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.498 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.498 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.498 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.499 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.499 DEBUG:boto:Method: GET 2023-10-11T15:27:18.500 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.500 DEBUG:boto:Data: 2023-10-11T15:27:18.500 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.501 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.501 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.502 DEBUG:boto:Params: {} 2023-10-11T15:27:18.502 DEBUG:boto:Token: None 2023-10-11T15:27:18.502 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.503 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f451f2478eae6be9-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.506 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.507 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.507 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.508 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.508 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.509 DEBUG:boto:Method: GET 2023-10-11T15:27:18.509 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.509 DEBUG:boto:Data: 2023-10-11T15:27:18.510 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.511 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.511 DEBUG:boto:Params: {} 2023-10-11T15:27:18.511 DEBUG:boto:Token: None 2023-10-11T15:27:18.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d769db3f6fd745f-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.517 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.518 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.518 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.519 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.519 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:18.519 DEBUG:boto:Data: 2023-10-11T15:27:18.520 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.520 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.521 DEBUG:boto:Params: {} 2023-10-11T15:27:18.521 DEBUG:boto:Token: None 2023-10-11T15:27:18.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57 2023-10-11T15:27:18.522 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HvE1oTTmT08Ovb5fNNbDDNXnmJQ= 2023-10-11T15:27:18.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HvE1oTTmT08Ovb5fNNbDDNXnmJQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475f62927104c5a7-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.525 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.526 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.526 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.526 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:18.527 DEBUG:boto:Data: 2023-10-11T15:27:18.527 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.528 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.528 DEBUG:boto:Params: {} 2023-10-11T15:27:18.529 DEBUG:boto:Token: None 2023-10-11T15:27:18.529 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57 2023-10-11T15:27:18.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HvE1oTTmT08Ovb5fNNbDDNXnmJQ= 2023-10-11T15:27:18.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HvE1oTTmT08Ovb5fNNbDDNXnmJQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052cc6a11895ef915-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:18.533 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.534 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.534 DEBUG:boto:Method: GET 2023-10-11T15:27:18.534 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:18.535 DEBUG:boto:Data: 2023-10-11T15:27:18.535 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.536 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.536 DEBUG:boto:Params: {} 2023-10-11T15:27:18.537 DEBUG:boto:Token: None 2023-10-11T15:27:18.537 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57 2023-10-11T15:27:18.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YNqKlspjmd36wmzLhSx4KMkbX54= 2023-10-11T15:27:18.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YNqKlspjmd36wmzLhSx4KMkbX54=', 'Content-Length': '0'} 2023-10-11T15:27:18.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065675b4e339afecd-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.541 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:18.541 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:18.542 DEBUG:boto:Method: GET 2023-10-11T15:27:18.542 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:18.542 DEBUG:boto:Data: 2023-10-11T15:27:18.543 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.544 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.544 DEBUG:boto:Params: {} 2023-10-11T15:27:18.544 DEBUG:boto:Token: None 2023-10-11T15:27:18.545 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj57 2023-10-11T15:27:18.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YNqKlspjmd36wmzLhSx4KMkbX54= 2023-10-11T15:27:18.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YNqKlspjmd36wmzLhSx4KMkbX54=', 'Content-Length': '0'} 2023-10-11T15:27:18.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2b934f50fda565-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.548 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.549 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.549 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.550 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.550 DEBUG:boto:Method: GET 2023-10-11T15:27:18.550 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.551 DEBUG:boto:Data: 2023-10-11T15:27:18.551 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.552 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.552 DEBUG:boto:Params: {} 2023-10-11T15:27:18.553 DEBUG:boto:Token: None 2023-10-11T15:27:18.553 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b56740f046c4bd-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.557 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.558 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.558 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.558 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.559 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.559 DEBUG:boto:Method: GET 2023-10-11T15:27:18.560 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:18.560 DEBUG:boto:Data: 2023-10-11T15:27:18.560 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.561 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.561 DEBUG:boto:Params: {} 2023-10-11T15:27:18.562 DEBUG:boto:Token: None 2023-10-11T15:27:18.562 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.563 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b192a8c0e0c24e1f-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.568 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:18.569 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.569 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.570 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.570 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.571 DEBUG:boto:Method: GET 2023-10-11T15:27:18.571 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.571 DEBUG:boto:Data: 2023-10-11T15:27:18.572 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.573 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.573 DEBUG:boto:Params: {} 2023-10-11T15:27:18.573 DEBUG:boto:Token: None 2023-10-11T15:27:18.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc= 2023-10-11T15:27:18.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc=', 'Content-Length': '0'} 2023-10-11T15:27:18.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f0932c72daa884-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.577 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.578 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.578 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.578 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.579 DEBUG:boto:Method: GET 2023-10-11T15:27:18.579 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.580 DEBUG:boto:Data: 2023-10-11T15:27:18.580 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.580 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.581 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.581 DEBUG:boto:Params: {} 2023-10-11T15:27:18.582 DEBUG:boto:Token: None 2023-10-11T15:27:18.582 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.583 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc= 2023-10-11T15:27:18.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc=', 'Content-Length': '0'} 2023-10-11T15:27:18.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000980b3853b113abc8-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.586 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.586 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.587 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.587 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.588 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.588 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.588 DEBUG:boto:Data: 2023-10-11T15:27:18.589 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.589 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.590 DEBUG:boto:Params: {} 2023-10-11T15:27:18.590 DEBUG:boto:Token: None 2023-10-11T15:27:18.591 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rAKFCFihbVmnGcjwl1fc5cPefuQ= 2023-10-11T15:27:18.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rAKFCFihbVmnGcjwl1fc5cPefuQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d782a756a76951b2-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.595 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.596 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.596 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.596 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.597 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.597 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.598 DEBUG:boto:Data: 2023-10-11T15:27:18.598 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.599 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.599 DEBUG:boto:Params: {} 2023-10-11T15:27:18.600 DEBUG:boto:Token: None 2023-10-11T15:27:18.600 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rAKFCFihbVmnGcjwl1fc5cPefuQ= 2023-10-11T15:27:18.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rAKFCFihbVmnGcjwl1fc5cPefuQ=', 'Content-Length': '0'} 2023-10-11T15:27:18.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8adc8f2502a4e3-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:18.604 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.604 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.605 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.605 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.606 DEBUG:boto:Method: GET 2023-10-11T15:27:18.606 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.606 DEBUG:boto:Data: 2023-10-11T15:27:18.607 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.608 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.608 DEBUG:boto:Params: {} 2023-10-11T15:27:18.609 DEBUG:boto:Token: None 2023-10-11T15:27:18.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc= 2023-10-11T15:27:18.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc=', 'Content-Length': '0'} 2023-10-11T15:27:18.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262ac2fc6a243cef-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.654 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.655 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.655 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.656 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.656 DEBUG:boto:Method: GET 2023-10-11T15:27:18.657 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.657 DEBUG:boto:Data: 2023-10-11T15:27:18.657 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.658 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.659 DEBUG:boto:Params: {} 2023-10-11T15:27:18.659 DEBUG:boto:Token: None 2023-10-11T15:27:18.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:18.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc= 2023-10-11T15:27:18.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9cjOiuAnpjx4NaaKOyWLYsI6rtc=', 'Content-Length': '0'} 2023-10-11T15:27:18.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d4efb2ac35ba915-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.663 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.663 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.664 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.664 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.665 DEBUG:boto:Method: GET 2023-10-11T15:27:18.665 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.665 DEBUG:boto:Data: 2023-10-11T15:27:18.666 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.667 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.667 DEBUG:boto:Params: {} 2023-10-11T15:27:18.668 DEBUG:boto:Token: None 2023-10-11T15:27:18.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4be1121e1b8d4f2-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.672 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.673 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.673 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.674 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.674 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.675 DEBUG:boto:Method: GET 2023-10-11T15:27:18.675 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.675 DEBUG:boto:Data: 2023-10-11T15:27:18.676 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.677 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.677 DEBUG:boto:Params: {} 2023-10-11T15:27:18.678 DEBUG:boto:Token: None 2023-10-11T15:27:18.678 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000234ed01089413dc8-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.726 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.727 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.728 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.728 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.728 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:18.729 DEBUG:boto:Data: 2023-10-11T15:27:18.729 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.730 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.731 DEBUG:boto:Params: {} 2023-10-11T15:27:18.731 DEBUG:boto:Token: None 2023-10-11T15:27:18.732 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58 2023-10-11T15:27:18.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s1A2VevPrz5BK2JPXLsl+gEA5Ng= 2023-10-11T15:27:18.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s1A2VevPrz5BK2JPXLsl+gEA5Ng=', 'Content-Length': '0'} 2023-10-11T15:27:18.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd8dec02eb05b3e-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.736 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.736 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.736 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.737 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:18.737 DEBUG:boto:Data: 2023-10-11T15:27:18.738 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.738 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.739 DEBUG:boto:Params: {} 2023-10-11T15:27:18.739 DEBUG:boto:Token: None 2023-10-11T15:27:18.740 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58 2023-10-11T15:27:18.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s1A2VevPrz5BK2JPXLsl+gEA5Ng= 2023-10-11T15:27:18.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s1A2VevPrz5BK2JPXLsl+gEA5Ng=', 'Content-Length': '0'} 2023-10-11T15:27:18.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff7705bbc08b2c1-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:18.743 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.744 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.744 DEBUG:boto:Method: GET 2023-10-11T15:27:18.745 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:18.745 DEBUG:boto:Data: 2023-10-11T15:27:18.745 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.746 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.747 DEBUG:boto:Params: {} 2023-10-11T15:27:18.747 DEBUG:boto:Token: None 2023-10-11T15:27:18.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58 2023-10-11T15:27:18.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0vfyxVGISAZ+lxjCz4F6eA5TleM= 2023-10-11T15:27:18.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0vfyxVGISAZ+lxjCz4F6eA5TleM=', 'Content-Length': '0'} 2023-10-11T15:27:18.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab5b4e01418492a-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.751 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:18.752 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:18.752 DEBUG:boto:Method: GET 2023-10-11T15:27:18.753 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:18.753 DEBUG:boto:Data: 2023-10-11T15:27:18.753 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.754 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.755 DEBUG:boto:Params: {} 2023-10-11T15:27:18.755 DEBUG:boto:Token: None 2023-10-11T15:27:18.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj58 2023-10-11T15:27:18.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0vfyxVGISAZ+lxjCz4F6eA5TleM= 2023-10-11T15:27:18.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0vfyxVGISAZ+lxjCz4F6eA5TleM=', 'Content-Length': '0'} 2023-10-11T15:27:18.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd52d1bb9b46a3bb-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.759 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.760 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.760 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.761 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.761 DEBUG:boto:Method: GET 2023-10-11T15:27:18.761 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.762 DEBUG:boto:Data: 2023-10-11T15:27:18.762 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.763 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.763 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.764 DEBUG:boto:Params: {} 2023-10-11T15:27:18.764 DEBUG:boto:Token: None 2023-10-11T15:27:18.764 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053d7b6c21ee176ac-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.814 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.815 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.816 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.816 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.817 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.817 DEBUG:boto:Method: GET 2023-10-11T15:27:18.817 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:18.818 DEBUG:boto:Data: 2023-10-11T15:27:18.818 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.819 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.820 DEBUG:boto:Params: {} 2023-10-11T15:27:18.820 DEBUG:boto:Token: None 2023-10-11T15:27:18.821 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.821 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000124a3ff7c832ba19-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.826 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:18.828 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.828 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.828 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.829 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.829 DEBUG:boto:Method: GET 2023-10-11T15:27:18.830 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.830 DEBUG:boto:Data: 2023-10-11T15:27:18.831 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.831 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.832 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.832 DEBUG:boto:Params: {} 2023-10-11T15:27:18.833 DEBUG:boto:Token: None 2023-10-11T15:27:18.833 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0= 2023-10-11T15:27:18.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0=', 'Content-Length': '0'} 2023-10-11T15:27:18.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae4e01b04d84eb1-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.837 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.838 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.838 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.839 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.839 DEBUG:boto:Method: GET 2023-10-11T15:27:18.839 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.840 DEBUG:boto:Data: 2023-10-11T15:27:18.840 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.841 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.841 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.842 DEBUG:boto:Params: {} 2023-10-11T15:27:18.842 DEBUG:boto:Token: None 2023-10-11T15:27:18.843 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0= 2023-10-11T15:27:18.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0=', 'Content-Length': '0'} 2023-10-11T15:27:18.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092220e71ab3ea2be-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.847 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.847 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.848 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.848 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.848 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.849 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.849 DEBUG:boto:Data: 2023-10-11T15:27:18.850 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.850 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.851 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.851 DEBUG:boto:Params: {} 2023-10-11T15:27:18.851 DEBUG:boto:Token: None 2023-10-11T15:27:18.852 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.852 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kksyw50i4UORIz8KmXgavL0WMiM= 2023-10-11T15:27:18.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kksyw50i4UORIz8KmXgavL0WMiM=', 'Content-Length': '0'} 2023-10-11T15:27:18.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257ee304c4d9968e-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.855 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.856 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.856 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.857 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.857 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.857 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.858 DEBUG:boto:Data: 2023-10-11T15:27:18.858 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.859 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.859 DEBUG:boto:Params: {} 2023-10-11T15:27:18.860 DEBUG:boto:Token: None 2023-10-11T15:27:18.860 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.861 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kksyw50i4UORIz8KmXgavL0WMiM= 2023-10-11T15:27:18.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kksyw50i4UORIz8KmXgavL0WMiM=', 'Content-Length': '0'} 2023-10-11T15:27:18.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a96987a3f912d05-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:18.864 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.865 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.865 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.866 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.866 DEBUG:boto:Method: GET 2023-10-11T15:27:18.867 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.867 DEBUG:boto:Data: 2023-10-11T15:27:18.868 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.869 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.869 DEBUG:boto:Params: {} 2023-10-11T15:27:18.869 DEBUG:boto:Token: None 2023-10-11T15:27:18.870 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0= 2023-10-11T15:27:18.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0=', 'Content-Length': '0'} 2023-10-11T15:27:18.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000474837790f4443b0-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.874 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.875 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.875 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.875 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.876 DEBUG:boto:Method: GET 2023-10-11T15:27:18.876 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.877 DEBUG:boto:Data: 2023-10-11T15:27:18.877 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.878 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.879 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.879 DEBUG:boto:Params: {} 2023-10-11T15:27:18.880 DEBUG:boto:Token: None 2023-10-11T15:27:18.880 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:18.881 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0= 2023-10-11T15:27:18.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tS8l9UhEuZm92hKIKiGsy4pObF0=', 'Content-Length': '0'} 2023-10-11T15:27:18.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0ed916f56453bc-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.926 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.927 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.927 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.928 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.928 DEBUG:boto:Method: GET 2023-10-11T15:27:18.929 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.929 DEBUG:boto:Data: 2023-10-11T15:27:18.930 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.931 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.931 DEBUG:boto:Params: {} 2023-10-11T15:27:18.932 DEBUG:boto:Token: None 2023-10-11T15:27:18.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab1fe9eb4bef9ba5-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.937 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.938 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.938 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.939 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.939 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.940 DEBUG:boto:Method: GET 2023-10-11T15:27:18.940 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.941 DEBUG:boto:Data: 2023-10-11T15:27:18.941 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.942 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.942 DEBUG:boto:Params: {} 2023-10-11T15:27:18.943 DEBUG:boto:Token: None 2023-10-11T15:27:18.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d034044ccf6c01-006526bed6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.949 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:18.950 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.950 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.951 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.951 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:18.952 DEBUG:boto:Data: 2023-10-11T15:27:18.952 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.953 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.953 DEBUG:boto:Params: {} 2023-10-11T15:27:18.954 DEBUG:boto:Token: None 2023-10-11T15:27:18.954 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62 2023-10-11T15:27:18.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2CC88/K/dbGGFF/QF6j9v+m8d4Q= 2023-10-11T15:27:18.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2CC88/K/dbGGFF/QF6j9v+m8d4Q=', 'Content-Length': '0'} 2023-10-11T15:27:18.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7480e881a69958-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.958 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.959 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.959 DEBUG:boto:Method: HEAD 2023-10-11T15:27:18.959 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:18.960 DEBUG:boto:Data: 2023-10-11T15:27:18.960 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.961 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.961 DEBUG:boto:Params: {} 2023-10-11T15:27:18.962 DEBUG:boto:Token: None 2023-10-11T15:27:18.962 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62 2023-10-11T15:27:18.963 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2CC88/K/dbGGFF/QF6j9v+m8d4Q= 2023-10-11T15:27:18.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2CC88/K/dbGGFF/QF6j9v+m8d4Q=', 'Content-Length': '0'} 2023-10-11T15:27:18.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000156d23908abf90e5-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:18.967 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.967 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.968 DEBUG:boto:Method: GET 2023-10-11T15:27:18.968 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:18.969 DEBUG:boto:Data: 2023-10-11T15:27:18.969 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.970 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.970 DEBUG:boto:Params: {} 2023-10-11T15:27:18.971 DEBUG:boto:Token: None 2023-10-11T15:27:18.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62 2023-10-11T15:27:18.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DI0LWSPwrOstYK5qncE2xJMxHiE= 2023-10-11T15:27:18.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DI0LWSPwrOstYK5qncE2xJMxHiE=', 'Content-Length': '0'} 2023-10-11T15:27:18.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c34ccab8c0040d6-006526bed6-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.975 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:18.975 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:18.976 DEBUG:boto:Method: GET 2023-10-11T15:27:18.976 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:18.977 DEBUG:boto:Data: 2023-10-11T15:27:18.977 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:18.978 DEBUG:boto:Port: 8000 2023-10-11T15:27:18.979 DEBUG:boto:Params: {} 2023-10-11T15:27:18.979 DEBUG:boto:Token: None 2023-10-11T15:27:18.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/obj62 2023-10-11T15:27:18.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DI0LWSPwrOstYK5qncE2xJMxHiE= 2023-10-11T15:27:18.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DI0LWSPwrOstYK5qncE2xJMxHiE=', 'Content-Length': '0'} 2023-10-11T15:27:18.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe0f02cf1ecad537-006526bed6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:18.984 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:18.984 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:18.985 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.985 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.985 DEBUG:boto:Method: GET 2023-10-11T15:27:18.986 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:18.986 DEBUG:boto:Data: 2023-10-11T15:27:18.987 DEBUG:boto:Headers: {} 2023-10-11T15:27:18.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:18.988 DEBUG:boto:Port: 8001 2023-10-11T15:27:18.988 DEBUG:boto:Params: {} 2023-10-11T15:27:18.989 DEBUG:boto:Token: None 2023-10-11T15:27:18.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:18 GMT /dghqvd-59/ 2023-10-11T15:27:18.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc= 2023-10-11T15:27:18.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MgzRWFU2KCg9YuzQ3r/FtnIE8Vc=', 'Content-Length': '0'} 2023-10-11T15:27:18.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b6ea249ab083c2f-006526bed6-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.038 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.039 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.040 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.040 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:19.040 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:19.041 DEBUG:boto:Method: GET 2023-10-11T15:27:19.041 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:19.042 DEBUG:boto:Data: 2023-10-11T15:27:19.042 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.043 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.043 DEBUG:boto:Params: {} 2023-10-11T15:27:19.044 DEBUG:boto:Token: None 2023-10-11T15:27:19.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287b8395680a3689-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.051 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:19.052 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.052 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.053 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.053 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.053 DEBUG:boto:Method: GET 2023-10-11T15:27:19.054 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.054 DEBUG:boto:Data: 2023-10-11T15:27:19.055 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.056 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.056 DEBUG:boto:Params: {} 2023-10-11T15:27:19.057 DEBUG:boto:Token: None 2023-10-11T15:27:19.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI= 2023-10-11T15:27:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI=', 'Content-Length': '0'} 2023-10-11T15:27:19.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2bf1cd1d70143c6-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.061 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.061 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.062 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.062 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.063 DEBUG:boto:Method: GET 2023-10-11T15:27:19.063 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.064 DEBUG:boto:Data: 2023-10-11T15:27:19.064 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.064 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.065 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.065 DEBUG:boto:Params: {} 2023-10-11T15:27:19.066 DEBUG:boto:Token: None 2023-10-11T15:27:19.066 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI= 2023-10-11T15:27:19.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI=', 'Content-Length': '0'} 2023-10-11T15:27:19.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2aa23be158fc6d-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.110 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.111 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.111 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.112 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.112 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.113 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.113 DEBUG:boto:Data: 2023-10-11T15:27:19.113 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.114 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.115 DEBUG:boto:Params: {} 2023-10-11T15:27:19.115 DEBUG:boto:Token: None 2023-10-11T15:27:19.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.116 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5dCKSXetC6XHcGWhjTEBSFHLx4s= 2023-10-11T15:27:19.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5dCKSXetC6XHcGWhjTEBSFHLx4s=', 'Content-Length': '0'} 2023-10-11T15:27:19.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a23f79c361b67ac-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.119 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.120 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.120 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.121 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.121 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.121 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.122 DEBUG:boto:Data: 2023-10-11T15:27:19.122 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.123 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.123 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.123 DEBUG:boto:Params: {} 2023-10-11T15:27:19.124 DEBUG:boto:Token: None 2023-10-11T15:27:19.124 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.125 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5dCKSXetC6XHcGWhjTEBSFHLx4s= 2023-10-11T15:27:19.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5dCKSXetC6XHcGWhjTEBSFHLx4s=', 'Content-Length': '0'} 2023-10-11T15:27:19.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbbbfba7f6208451-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:19.128 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.129 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.129 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.130 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.130 DEBUG:boto:Method: GET 2023-10-11T15:27:19.131 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.131 DEBUG:boto:Data: 2023-10-11T15:27:19.131 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.132 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.133 DEBUG:boto:Params: {} 2023-10-11T15:27:19.133 DEBUG:boto:Token: None 2023-10-11T15:27:19.134 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI= 2023-10-11T15:27:19.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI=', 'Content-Length': '0'} 2023-10-11T15:27:19.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca51346e0950d1cf-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.138 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.138 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.139 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.139 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.139 DEBUG:boto:Method: GET 2023-10-11T15:27:19.140 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.140 DEBUG:boto:Data: 2023-10-11T15:27:19.141 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.141 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.142 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.142 DEBUG:boto:Params: {} 2023-10-11T15:27:19.142 DEBUG:boto:Token: None 2023-10-11T15:27:19.143 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:19.143 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI= 2023-10-11T15:27:19.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:isgwlZ0GBF0xVTbVuJVy1qrxecI=', 'Content-Length': '0'} 2023-10-11T15:27:19.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c2e839d32fe668-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.147 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.147 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.148 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.148 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.149 DEBUG:boto:Method: GET 2023-10-11T15:27:19.149 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.150 DEBUG:boto:Data: 2023-10-11T15:27:19.150 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.150 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.151 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.151 DEBUG:boto:Params: {} 2023-10-11T15:27:19.152 DEBUG:boto:Token: None 2023-10-11T15:27:19.152 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.153 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f01edd7e344748db-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.156 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.157 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.158 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.158 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.158 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.159 DEBUG:boto:Method: GET 2023-10-11T15:27:19.159 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.160 DEBUG:boto:Data: 2023-10-11T15:27:19.160 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.161 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.161 DEBUG:boto:Params: {} 2023-10-11T15:27:19.162 DEBUG:boto:Token: None 2023-10-11T15:27:19.162 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.163 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fb87f37b47922b-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.168 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.169 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.170 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.170 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.170 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:19.171 DEBUG:boto:Data: 2023-10-11T15:27:19.171 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.172 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.172 DEBUG:boto:Params: {} 2023-10-11T15:27:19.173 DEBUG:boto:Token: None 2023-10-11T15:27:19.173 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68 2023-10-11T15:27:19.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDa8XeJRKt+bCPi248winOT+xhw= 2023-10-11T15:27:19.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDa8XeJRKt+bCPi248winOT+xhw=', 'Content-Length': '0'} 2023-10-11T15:27:19.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f009492e233e598-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.177 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.177 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.178 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.178 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:19.179 DEBUG:boto:Data: 2023-10-11T15:27:19.179 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.180 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.180 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.180 DEBUG:boto:Params: {} 2023-10-11T15:27:19.181 DEBUG:boto:Token: None 2023-10-11T15:27:19.181 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68 2023-10-11T15:27:19.182 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eDa8XeJRKt+bCPi248winOT+xhw= 2023-10-11T15:27:19.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eDa8XeJRKt+bCPi248winOT+xhw=', 'Content-Length': '0'} 2023-10-11T15:27:19.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94d8ee9208f4533-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:19.185 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.186 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.186 DEBUG:boto:Method: GET 2023-10-11T15:27:19.187 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:19.187 DEBUG:boto:Data: 2023-10-11T15:27:19.187 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.189 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.189 DEBUG:boto:Params: {} 2023-10-11T15:27:19.190 DEBUG:boto:Token: None 2023-10-11T15:27:19.190 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68 2023-10-11T15:27:19.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AKoccX7X1s5NQDo8elp2zfcuIDg= 2023-10-11T15:27:19.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AKoccX7X1s5NQDo8elp2zfcuIDg=', 'Content-Length': '0'} 2023-10-11T15:27:19.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8ad6aea2001716-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.194 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:19.195 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:19.195 DEBUG:boto:Method: GET 2023-10-11T15:27:19.196 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:19.196 DEBUG:boto:Data: 2023-10-11T15:27:19.197 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.197 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.198 DEBUG:boto:Params: {} 2023-10-11T15:27:19.198 DEBUG:boto:Token: None 2023-10-11T15:27:19.199 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj68 2023-10-11T15:27:19.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AKoccX7X1s5NQDo8elp2zfcuIDg= 2023-10-11T15:27:19.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AKoccX7X1s5NQDo8elp2zfcuIDg=', 'Content-Length': '0'} 2023-10-11T15:27:19.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411676504f0a6c7b-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.242 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.243 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.244 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.244 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.245 DEBUG:boto:Method: GET 2023-10-11T15:27:19.245 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.245 DEBUG:boto:Data: 2023-10-11T15:27:19.246 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.247 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.247 DEBUG:boto:Params: {} 2023-10-11T15:27:19.248 DEBUG:boto:Token: None 2023-10-11T15:27:19.248 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0727be196b7a22-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.253 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.253 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.254 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.254 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.255 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.255 DEBUG:boto:Method: GET 2023-10-11T15:27:19.256 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:19.256 DEBUG:boto:Data: 2023-10-11T15:27:19.256 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.257 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.258 DEBUG:boto:Params: {} 2023-10-11T15:27:19.258 DEBUG:boto:Token: None 2023-10-11T15:27:19.259 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a69ad79766694b-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.264 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:19.266 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.266 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.267 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.267 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.268 DEBUG:boto:Method: GET 2023-10-11T15:27:19.268 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.269 DEBUG:boto:Data: 2023-10-11T15:27:19.269 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.269 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.270 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.270 DEBUG:boto:Params: {} 2023-10-11T15:27:19.271 DEBUG:boto:Token: None 2023-10-11T15:27:19.271 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc= 2023-10-11T15:27:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc=', 'Content-Length': '0'} 2023-10-11T15:27:19.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29807a26bde2772-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.276 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.276 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.277 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.277 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.278 DEBUG:boto:Method: GET 2023-10-11T15:27:19.278 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.278 DEBUG:boto:Data: 2023-10-11T15:27:19.279 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.279 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.280 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.280 DEBUG:boto:Params: {} 2023-10-11T15:27:19.281 DEBUG:boto:Token: None 2023-10-11T15:27:19.281 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc= 2023-10-11T15:27:19.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc=', 'Content-Length': '0'} 2023-10-11T15:27:19.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f20b4be7d28c9-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.286 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.286 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.287 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.287 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.287 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.288 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.288 DEBUG:boto:Data: 2023-10-11T15:27:19.289 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.289 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.290 DEBUG:boto:Params: {} 2023-10-11T15:27:19.290 DEBUG:boto:Token: None 2023-10-11T15:27:19.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WI/khmwPs1CEwZDzQfiFoYOJl9o= 2023-10-11T15:27:19.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WI/khmwPs1CEwZDzQfiFoYOJl9o=', 'Content-Length': '0'} 2023-10-11T15:27:19.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082ea6bb64ce3966f-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.294 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.295 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.295 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.295 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.296 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.296 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.297 DEBUG:boto:Data: 2023-10-11T15:27:19.297 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.298 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.298 DEBUG:boto:Params: {} 2023-10-11T15:27:19.299 DEBUG:boto:Token: None 2023-10-11T15:27:19.299 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WI/khmwPs1CEwZDzQfiFoYOJl9o= 2023-10-11T15:27:19.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WI/khmwPs1CEwZDzQfiFoYOJl9o=', 'Content-Length': '0'} 2023-10-11T15:27:19.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a95728748da692-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:19.304 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.304 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.304 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.305 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.305 DEBUG:boto:Method: GET 2023-10-11T15:27:19.306 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.306 DEBUG:boto:Data: 2023-10-11T15:27:19.307 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.308 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.308 DEBUG:boto:Params: {} 2023-10-11T15:27:19.308 DEBUG:boto:Token: None 2023-10-11T15:27:19.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc= 2023-10-11T15:27:19.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc=', 'Content-Length': '0'} 2023-10-11T15:27:19.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2b4ad689d1c3e6-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.354 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.355 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.355 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.356 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.356 DEBUG:boto:Method: GET 2023-10-11T15:27:19.357 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.357 DEBUG:boto:Data: 2023-10-11T15:27:19.358 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.359 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.359 DEBUG:boto:Params: {} 2023-10-11T15:27:19.360 DEBUG:boto:Token: None 2023-10-11T15:27:19.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:19.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc= 2023-10-11T15:27:19.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eIFTfzswgn+SB/k/4pKJdW3/dPc=', 'Content-Length': '0'} 2023-10-11T15:27:19.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004d97534add78f3-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.364 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.364 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.365 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.365 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.365 DEBUG:boto:Method: GET 2023-10-11T15:27:19.366 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.366 DEBUG:boto:Data: 2023-10-11T15:27:19.367 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.368 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.368 DEBUG:boto:Params: {} 2023-10-11T15:27:19.369 DEBUG:boto:Token: None 2023-10-11T15:27:19.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18f95b76d2142ed-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.373 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.374 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.375 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.375 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.375 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.376 DEBUG:boto:Method: GET 2023-10-11T15:27:19.376 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.377 DEBUG:boto:Data: 2023-10-11T15:27:19.377 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.378 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.378 DEBUG:boto:Params: {} 2023-10-11T15:27:19.379 DEBUG:boto:Token: None 2023-10-11T15:27:19.379 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9b331ac603a2b6-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.426 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.427 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.427 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.428 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.428 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:19.429 DEBUG:boto:Data: 2023-10-11T15:27:19.429 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.430 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.431 DEBUG:boto:Params: {} 2023-10-11T15:27:19.431 DEBUG:boto:Token: None 2023-10-11T15:27:19.432 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69 2023-10-11T15:27:19.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rgGelu1BqDR+1cRV5jwxBpdZtgI= 2023-10-11T15:27:19.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rgGelu1BqDR+1cRV5jwxBpdZtgI=', 'Content-Length': '0'} 2023-10-11T15:27:19.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c6ad3ede19238d-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.435 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.436 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.436 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.436 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:19.437 DEBUG:boto:Data: 2023-10-11T15:27:19.437 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.438 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.439 DEBUG:boto:Params: {} 2023-10-11T15:27:19.439 DEBUG:boto:Token: None 2023-10-11T15:27:19.440 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69 2023-10-11T15:27:19.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rgGelu1BqDR+1cRV5jwxBpdZtgI= 2023-10-11T15:27:19.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rgGelu1BqDR+1cRV5jwxBpdZtgI=', 'Content-Length': '0'} 2023-10-11T15:27:19.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000985faccb81167d29-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:19.444 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.444 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.445 DEBUG:boto:Method: GET 2023-10-11T15:27:19.445 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:19.446 DEBUG:boto:Data: 2023-10-11T15:27:19.446 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.447 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.448 DEBUG:boto:Params: {} 2023-10-11T15:27:19.448 DEBUG:boto:Token: None 2023-10-11T15:27:19.449 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69 2023-10-11T15:27:19.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s2pbRsw86XBFGN/fl3Uvs4GqBQ4= 2023-10-11T15:27:19.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s2pbRsw86XBFGN/fl3Uvs4GqBQ4=', 'Content-Length': '0'} 2023-10-11T15:27:19.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fee9392907ba65e-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.453 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:19.453 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:19.454 DEBUG:boto:Method: GET 2023-10-11T15:27:19.454 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:19.455 DEBUG:boto:Data: 2023-10-11T15:27:19.455 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.456 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.456 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.457 DEBUG:boto:Params: {} 2023-10-11T15:27:19.457 DEBUG:boto:Token: None 2023-10-11T15:27:19.458 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj69 2023-10-11T15:27:19.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s2pbRsw86XBFGN/fl3Uvs4GqBQ4= 2023-10-11T15:27:19.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s2pbRsw86XBFGN/fl3Uvs4GqBQ4=', 'Content-Length': '0'} 2023-10-11T15:27:19.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14b9614eda9eab6-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.462 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.462 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.463 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.463 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.464 DEBUG:boto:Method: GET 2023-10-11T15:27:19.464 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.465 DEBUG:boto:Data: 2023-10-11T15:27:19.465 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.466 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.467 DEBUG:boto:Params: {} 2023-10-11T15:27:19.467 DEBUG:boto:Token: None 2023-10-11T15:27:19.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000255a86481ee2526c-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.472 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.472 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.473 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.473 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.474 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.474 DEBUG:boto:Method: GET 2023-10-11T15:27:19.475 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:19.475 DEBUG:boto:Data: 2023-10-11T15:27:19.475 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.476 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.476 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.477 DEBUG:boto:Params: {} 2023-10-11T15:27:19.477 DEBUG:boto:Token: None 2023-10-11T15:27:19.478 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.478 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf12642944959563-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.483 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:19.484 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.485 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.485 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.486 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.486 DEBUG:boto:Method: GET 2023-10-11T15:27:19.487 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.487 DEBUG:boto:Data: 2023-10-11T15:27:19.488 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.488 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.489 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.489 DEBUG:boto:Params: {} 2023-10-11T15:27:19.490 DEBUG:boto:Token: None 2023-10-11T15:27:19.490 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE= 2023-10-11T15:27:19.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE=', 'Content-Length': '0'} 2023-10-11T15:27:19.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec70dcdc19ac8c4-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.538 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.539 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.539 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.540 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.540 DEBUG:boto:Method: GET 2023-10-11T15:27:19.540 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.541 DEBUG:boto:Data: 2023-10-11T15:27:19.541 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.542 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.543 DEBUG:boto:Params: {} 2023-10-11T15:27:19.543 DEBUG:boto:Token: None 2023-10-11T15:27:19.544 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE= 2023-10-11T15:27:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE=', 'Content-Length': '0'} 2023-10-11T15:27:19.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f48dfe3b5c5d71-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.547 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.548 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.548 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.549 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.549 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.550 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.550 DEBUG:boto:Data: 2023-10-11T15:27:19.551 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.551 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.551 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.552 DEBUG:boto:Params: {} 2023-10-11T15:27:19.552 DEBUG:boto:Token: None 2023-10-11T15:27:19.553 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.553 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3RkUlZv9KyIMUubeW57JDwmxdK8= 2023-10-11T15:27:19.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3RkUlZv9KyIMUubeW57JDwmxdK8=', 'Content-Length': '0'} 2023-10-11T15:27:19.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130d49914e7d0dc0-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.556 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.557 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.557 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.558 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.558 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.559 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.559 DEBUG:boto:Data: 2023-10-11T15:27:19.560 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.560 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.561 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.561 DEBUG:boto:Params: {} 2023-10-11T15:27:19.562 DEBUG:boto:Token: None 2023-10-11T15:27:19.562 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.563 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3RkUlZv9KyIMUubeW57JDwmxdK8= 2023-10-11T15:27:19.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3RkUlZv9KyIMUubeW57JDwmxdK8=', 'Content-Length': '0'} 2023-10-11T15:27:19.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de061f494fe271f-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:19.566 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.567 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.567 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.568 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.568 DEBUG:boto:Method: GET 2023-10-11T15:27:19.568 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.569 DEBUG:boto:Data: 2023-10-11T15:27:19.570 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.571 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.571 DEBUG:boto:Params: {} 2023-10-11T15:27:19.572 DEBUG:boto:Token: None 2023-10-11T15:27:19.572 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE= 2023-10-11T15:27:19.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE=', 'Content-Length': '0'} 2023-10-11T15:27:19.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ddcb02e6ff40f67-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.576 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.576 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.577 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.577 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.578 DEBUG:boto:Method: GET 2023-10-11T15:27:19.578 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.579 DEBUG:boto:Data: 2023-10-11T15:27:19.579 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.580 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.580 DEBUG:boto:Params: {} 2023-10-11T15:27:19.581 DEBUG:boto:Token: None 2023-10-11T15:27:19.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:19.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE= 2023-10-11T15:27:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HFGRdvJ8+nR+yfELtbpAiRGMuhE=', 'Content-Length': '0'} 2023-10-11T15:27:19.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9684889d5460f33-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.630 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.631 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.631 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.632 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.632 DEBUG:boto:Method: GET 2023-10-11T15:27:19.633 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.633 DEBUG:boto:Data: 2023-10-11T15:27:19.633 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.634 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.635 DEBUG:boto:Params: {} 2023-10-11T15:27:19.635 DEBUG:boto:Token: None 2023-10-11T15:27:19.636 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052fe30709044edbe-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.640 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.641 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.641 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.641 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.642 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.642 DEBUG:boto:Method: GET 2023-10-11T15:27:19.643 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.643 DEBUG:boto:Data: 2023-10-11T15:27:19.644 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.644 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.644 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.645 DEBUG:boto:Params: {} 2023-10-11T15:27:19.645 DEBUG:boto:Token: None 2023-10-11T15:27:19.646 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.646 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003772539607d2a787-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.652 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.652 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.653 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.653 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.654 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:19.654 DEBUG:boto:Data: 2023-10-11T15:27:19.654 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.655 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.656 DEBUG:boto:Params: {} 2023-10-11T15:27:19.656 DEBUG:boto:Token: None 2023-10-11T15:27:19.657 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74 2023-10-11T15:27:19.657 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Fpj7K50CjwpmzaUkZ4OevUCjt40= 2023-10-11T15:27:19.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Fpj7K50CjwpmzaUkZ4OevUCjt40=', 'Content-Length': '0'} 2023-10-11T15:27:19.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481573a7d0637573-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.660 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.661 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.661 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.662 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:19.662 DEBUG:boto:Data: 2023-10-11T15:27:19.662 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.663 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.663 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.664 DEBUG:boto:Params: {} 2023-10-11T15:27:19.664 DEBUG:boto:Token: None 2023-10-11T15:27:19.665 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74 2023-10-11T15:27:19.665 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Fpj7K50CjwpmzaUkZ4OevUCjt40= 2023-10-11T15:27:19.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Fpj7K50CjwpmzaUkZ4OevUCjt40=', 'Content-Length': '0'} 2023-10-11T15:27:19.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c320692ffd3e088-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:19.669 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.669 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.669 DEBUG:boto:Method: GET 2023-10-11T15:27:19.670 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:19.670 DEBUG:boto:Data: 2023-10-11T15:27:19.671 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.672 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.672 DEBUG:boto:Params: {} 2023-10-11T15:27:19.672 DEBUG:boto:Token: None 2023-10-11T15:27:19.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74 2023-10-11T15:27:19.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knwrv26niNziob5kj5oZArWcOak= 2023-10-11T15:27:19.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knwrv26niNziob5kj5oZArWcOak=', 'Content-Length': '0'} 2023-10-11T15:27:19.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab105a6ed10c65c-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.677 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:19.677 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:19.678 DEBUG:boto:Method: GET 2023-10-11T15:27:19.678 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:19.679 DEBUG:boto:Data: 2023-10-11T15:27:19.679 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.680 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.680 DEBUG:boto:Params: {} 2023-10-11T15:27:19.681 DEBUG:boto:Token: None 2023-10-11T15:27:19.681 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj74 2023-10-11T15:27:19.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:knwrv26niNziob5kj5oZArWcOak= 2023-10-11T15:27:19.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:knwrv26niNziob5kj5oZArWcOak=', 'Content-Length': '0'} 2023-10-11T15:27:19.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeebf3444db42682-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.685 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.686 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.686 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.686 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.687 DEBUG:boto:Method: GET 2023-10-11T15:27:19.687 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.688 DEBUG:boto:Data: 2023-10-11T15:27:19.688 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.689 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.689 DEBUG:boto:Params: {} 2023-10-11T15:27:19.690 DEBUG:boto:Token: None 2023-10-11T15:27:19.690 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004821f5efbaabf650-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.694 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.695 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.696 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.696 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.696 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.697 DEBUG:boto:Method: GET 2023-10-11T15:27:19.697 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:19.698 DEBUG:boto:Data: 2023-10-11T15:27:19.698 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.699 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.699 DEBUG:boto:Params: {} 2023-10-11T15:27:19.700 DEBUG:boto:Token: None 2023-10-11T15:27:19.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000149fddb444a9ca7b-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.706 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:19.708 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.708 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.709 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.709 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.710 DEBUG:boto:Method: GET 2023-10-11T15:27:19.710 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.711 DEBUG:boto:Data: 2023-10-11T15:27:19.711 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.712 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.712 DEBUG:boto:Params: {} 2023-10-11T15:27:19.713 DEBUG:boto:Token: None 2023-10-11T15:27:19.713 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.714 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s= 2023-10-11T15:27:19.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s=', 'Content-Length': '0'} 2023-10-11T15:27:19.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c142c24706bce80-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.717 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.718 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.718 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.719 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.719 DEBUG:boto:Method: GET 2023-10-11T15:27:19.720 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.720 DEBUG:boto:Data: 2023-10-11T15:27:19.720 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.721 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.722 DEBUG:boto:Params: {} 2023-10-11T15:27:19.722 DEBUG:boto:Token: None 2023-10-11T15:27:19.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s= 2023-10-11T15:27:19.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s=', 'Content-Length': '0'} 2023-10-11T15:27:19.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6d831e61c239c7-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.726 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.727 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.727 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.728 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.728 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.728 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.729 DEBUG:boto:Data: 2023-10-11T15:27:19.729 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.729 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.730 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.730 DEBUG:boto:Params: {} 2023-10-11T15:27:19.731 DEBUG:boto:Token: None 2023-10-11T15:27:19.731 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZXfKgTfAqY6vjw04GoDj3xJ6mXM= 2023-10-11T15:27:19.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZXfKgTfAqY6vjw04GoDj3xJ6mXM=', 'Content-Length': '0'} 2023-10-11T15:27:19.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2059a3e53b4011-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.735 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.735 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.736 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.736 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.736 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.737 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.737 DEBUG:boto:Data: 2023-10-11T15:27:19.738 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.739 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.739 DEBUG:boto:Params: {} 2023-10-11T15:27:19.739 DEBUG:boto:Token: None 2023-10-11T15:27:19.740 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZXfKgTfAqY6vjw04GoDj3xJ6mXM= 2023-10-11T15:27:19.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZXfKgTfAqY6vjw04GoDj3xJ6mXM=', 'Content-Length': '0'} 2023-10-11T15:27:19.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f354baa8b205f9ef-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:19.744 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.744 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.744 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.745 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.745 DEBUG:boto:Method: GET 2023-10-11T15:27:19.746 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.746 DEBUG:boto:Data: 2023-10-11T15:27:19.746 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.747 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.747 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.748 DEBUG:boto:Params: {} 2023-10-11T15:27:19.748 DEBUG:boto:Token: None 2023-10-11T15:27:19.749 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s= 2023-10-11T15:27:19.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s=', 'Content-Length': '0'} 2023-10-11T15:27:19.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c47766fde79525-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.794 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.795 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.795 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.796 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.796 DEBUG:boto:Method: GET 2023-10-11T15:27:19.796 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.797 DEBUG:boto:Data: 2023-10-11T15:27:19.797 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.798 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.798 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.798 DEBUG:boto:Params: {} 2023-10-11T15:27:19.799 DEBUG:boto:Token: None 2023-10-11T15:27:19.799 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:19.800 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s= 2023-10-11T15:27:19.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDQXC4k2B2B3EcdivhTM6cx+B2s=', 'Content-Length': '0'} 2023-10-11T15:27:19.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d73b5ce397da63d-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.803 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.804 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.804 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.804 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.805 DEBUG:boto:Method: GET 2023-10-11T15:27:19.805 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.806 DEBUG:boto:Data: 2023-10-11T15:27:19.806 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.807 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.807 DEBUG:boto:Params: {} 2023-10-11T15:27:19.808 DEBUG:boto:Token: None 2023-10-11T15:27:19.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057fc641e1631ab84-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.812 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.813 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.814 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.814 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.814 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.815 DEBUG:boto:Method: GET 2023-10-11T15:27:19.815 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.816 DEBUG:boto:Data: 2023-10-11T15:27:19.816 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.817 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.817 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.817 DEBUG:boto:Params: {} 2023-10-11T15:27:19.818 DEBUG:boto:Token: None 2023-10-11T15:27:19.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d21a8152cbda42fe-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.866 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.867 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.867 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.868 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.868 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:19.868 DEBUG:boto:Data: 2023-10-11T15:27:19.869 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.870 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.870 DEBUG:boto:Params: {} 2023-10-11T15:27:19.870 DEBUG:boto:Token: None 2023-10-11T15:27:19.871 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76 2023-10-11T15:27:19.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SSfIdDNzBnYha6JwAbKlsodEcbg= 2023-10-11T15:27:19.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SSfIdDNzBnYha6JwAbKlsodEcbg=', 'Content-Length': '0'} 2023-10-11T15:27:19.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4721b3f8cf12bba-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.875 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.875 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.875 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.876 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:19.876 DEBUG:boto:Data: 2023-10-11T15:27:19.876 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.877 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.878 DEBUG:boto:Params: {} 2023-10-11T15:27:19.878 DEBUG:boto:Token: None 2023-10-11T15:27:19.878 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76 2023-10-11T15:27:19.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SSfIdDNzBnYha6JwAbKlsodEcbg= 2023-10-11T15:27:19.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SSfIdDNzBnYha6JwAbKlsodEcbg=', 'Content-Length': '0'} 2023-10-11T15:27:19.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892a4b22caaeb7b5-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:19.882 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.883 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.883 DEBUG:boto:Method: GET 2023-10-11T15:27:19.883 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:19.884 DEBUG:boto:Data: 2023-10-11T15:27:19.884 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.885 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.885 DEBUG:boto:Params: {} 2023-10-11T15:27:19.886 DEBUG:boto:Token: None 2023-10-11T15:27:19.886 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76 2023-10-11T15:27:19.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JAYuhXZ02uiECtA62CG2q08MHFg= 2023-10-11T15:27:19.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JAYuhXZ02uiECtA62CG2q08MHFg=', 'Content-Length': '0'} 2023-10-11T15:27:19.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ff6e0eef28df38-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.890 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:19.890 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:19.891 DEBUG:boto:Method: GET 2023-10-11T15:27:19.891 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:19.891 DEBUG:boto:Data: 2023-10-11T15:27:19.892 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.892 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.893 DEBUG:boto:Params: {} 2023-10-11T15:27:19.893 DEBUG:boto:Token: None 2023-10-11T15:27:19.894 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj76 2023-10-11T15:27:19.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JAYuhXZ02uiECtA62CG2q08MHFg= 2023-10-11T15:27:19.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JAYuhXZ02uiECtA62CG2q08MHFg=', 'Content-Length': '0'} 2023-10-11T15:27:19.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72061e681790fc0-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.898 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.898 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.898 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.899 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.899 DEBUG:boto:Method: GET 2023-10-11T15:27:19.900 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.900 DEBUG:boto:Data: 2023-10-11T15:27:19.900 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.901 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.901 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.901 DEBUG:boto:Params: {} 2023-10-11T15:27:19.902 DEBUG:boto:Token: None 2023-10-11T15:27:19.902 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.903 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ad26ff5b01ffebe-006526bed7-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.906 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.907 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:19.907 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:19.908 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.908 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.909 DEBUG:boto:Method: GET 2023-10-11T15:27:19.909 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:19.909 DEBUG:boto:Data: 2023-10-11T15:27:19.910 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.911 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.911 DEBUG:boto:Params: {} 2023-10-11T15:27:19.911 DEBUG:boto:Token: None 2023-10-11T15:27:19.912 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/ 2023-10-11T15:27:19.912 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4= 2023-10-11T15:27:19.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vhHYolP2w6S6udUy59hxBTd/gM4=', 'Content-Length': '0'} 2023-10-11T15:27:19.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da16ed1ac5f8caff-006526bed7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.917 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:19.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:19.918 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.919 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.919 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.919 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.920 DEBUG:boto:Method: GET 2023-10-11T15:27:19.920 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.920 DEBUG:boto:Data: 2023-10-11T15:27:19.921 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.922 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.922 DEBUG:boto:Params: {} 2023-10-11T15:27:19.923 DEBUG:boto:Token: None 2023-10-11T15:27:19.923 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.924 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg= 2023-10-11T15:27:19.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg=', 'Content-Length': '0'} 2023-10-11T15:27:19.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040640cd14f7bf00e-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.927 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.927 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.928 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.928 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.929 DEBUG:boto:Method: GET 2023-10-11T15:27:19.929 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.929 DEBUG:boto:Data: 2023-10-11T15:27:19.930 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.931 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.931 DEBUG:boto:Params: {} 2023-10-11T15:27:19.932 DEBUG:boto:Token: None 2023-10-11T15:27:19.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg= 2023-10-11T15:27:19.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg=', 'Content-Length': '0'} 2023-10-11T15:27:19.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbc6cb0f01762db-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.936 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.936 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.937 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.937 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.937 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.938 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.938 DEBUG:boto:Data: 2023-10-11T15:27:19.939 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.940 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.940 DEBUG:boto:Params: {} 2023-10-11T15:27:19.940 DEBUG:boto:Token: None 2023-10-11T15:27:19.941 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fvBn3d8y5LIcVqqBZO8/ebm4ae0= 2023-10-11T15:27:19.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fvBn3d8y5LIcVqqBZO8/ebm4ae0=', 'Content-Length': '0'} 2023-10-11T15:27:19.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faddc5de0079861-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.944 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.945 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.945 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.946 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.946 DEBUG:boto:Method: HEAD 2023-10-11T15:27:19.947 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.947 DEBUG:boto:Data: 2023-10-11T15:27:19.947 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.948 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.949 DEBUG:boto:Params: {} 2023-10-11T15:27:19.949 DEBUG:boto:Token: None 2023-10-11T15:27:19.950 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fvBn3d8y5LIcVqqBZO8/ebm4ae0= 2023-10-11T15:27:19.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fvBn3d8y5LIcVqqBZO8/ebm4ae0=', 'Content-Length': '0'} 2023-10-11T15:27:19.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf9aaafc2837c647-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:19.954 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.954 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.955 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.955 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.956 DEBUG:boto:Method: GET 2023-10-11T15:27:19.956 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.956 DEBUG:boto:Data: 2023-10-11T15:27:19.957 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:19.958 DEBUG:boto:Port: 8001 2023-10-11T15:27:19.958 DEBUG:boto:Params: {} 2023-10-11T15:27:19.959 DEBUG:boto:Token: None 2023-10-11T15:27:19.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg= 2023-10-11T15:27:19.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg=', 'Content-Length': '0'} 2023-10-11T15:27:19.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f2251ee2fef3a3-006526bed7-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:19.963 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:19.963 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:19.964 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.964 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.964 DEBUG:boto:Method: GET 2023-10-11T15:27:19.965 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.965 DEBUG:boto:Data: 2023-10-11T15:27:19.966 DEBUG:boto:Headers: {} 2023-10-11T15:27:19.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:19.967 DEBUG:boto:Port: 8000 2023-10-11T15:27:19.967 DEBUG:boto:Params: {} 2023-10-11T15:27:19.968 DEBUG:boto:Token: None 2023-10-11T15:27:19.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:19 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:19.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg= 2023-10-11T15:27:19.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:doi/GlgGL5AeaNpoxzid0b7bEgg=', 'Content-Length': '0'} 2023-10-11T15:27:19.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b96256fcea4c92-006526bed7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.014 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.015 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.015 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.015 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.016 DEBUG:boto:Method: GET 2023-10-11T15:27:20.016 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.017 DEBUG:boto:Data: 2023-10-11T15:27:20.017 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.018 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.018 DEBUG:boto:Params: {} 2023-10-11T15:27:20.019 DEBUG:boto:Token: None 2023-10-11T15:27:20.019 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff0f2d8d7eb66b29-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.023 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.024 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.024 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.025 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.025 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.026 DEBUG:boto:Method: GET 2023-10-11T15:27:20.026 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.026 DEBUG:boto:Data: 2023-10-11T15:27:20.027 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.029 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.030 DEBUG:boto:Params: {} 2023-10-11T15:27:20.030 DEBUG:boto:Token: None 2023-10-11T15:27:20.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0665e0adfb9924f-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.036 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.037 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:20.037 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:20.038 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.038 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:20.039 DEBUG:boto:Data: 2023-10-11T15:27:20.039 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.040 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.040 DEBUG:boto:Params: {} 2023-10-11T15:27:20.041 DEBUG:boto:Token: None 2023-10-11T15:27:20.041 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj85 2023-10-11T15:27:20.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d36qL6qFVW3N+5rsGQpjTkOmoA4= 2023-10-11T15:27:20.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d36qL6qFVW3N+5rsGQpjTkOmoA4=', 'Content-Length': '0'} 2023-10-11T15:27:20.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd40f84073d39375-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.045 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:20.045 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:20.046 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.046 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:20.047 DEBUG:boto:Data: 2023-10-11T15:27:20.047 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.048 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.048 DEBUG:boto:Params: {} 2023-10-11T15:27:20.049 DEBUG:boto:Token: None 2023-10-11T15:27:20.049 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj85 2023-10-11T15:27:20.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d36qL6qFVW3N+5rsGQpjTkOmoA4= 2023-10-11T15:27:20.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d36qL6qFVW3N+5rsGQpjTkOmoA4=', 'Content-Length': '0'} 2023-10-11T15:27:20.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc184e954f4005d7-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:20.053 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:20.053 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:20.054 DEBUG:boto:Method: GET 2023-10-11T15:27:20.054 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:20.055 DEBUG:boto:Data: 2023-10-11T15:27:20.055 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.056 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.056 DEBUG:boto:Params: {} 2023-10-11T15:27:20.057 DEBUG:boto:Token: None 2023-10-11T15:27:20.057 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj85 2023-10-11T15:27:20.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8uB9vT1O5kNp8hzRGmdpdB8+LBo= 2023-10-11T15:27:20.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8uB9vT1O5kNp8hzRGmdpdB8+LBo=', 'Content-Length': '0'} 2023-10-11T15:27:20.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285a878bfd887f28-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.106 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:20.107 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:20.107 DEBUG:boto:Method: GET 2023-10-11T15:27:20.108 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:20.108 DEBUG:boto:Data: 2023-10-11T15:27:20.109 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.109 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.110 DEBUG:boto:Params: {} 2023-10-11T15:27:20.110 DEBUG:boto:Token: None 2023-10-11T15:27:20.111 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj85 2023-10-11T15:27:20.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8uB9vT1O5kNp8hzRGmdpdB8+LBo= 2023-10-11T15:27:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8uB9vT1O5kNp8hzRGmdpdB8+LBo=', 'Content-Length': '0'} 2023-10-11T15:27:20.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d62ef537f113804-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.114 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.115 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.115 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.116 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.116 DEBUG:boto:Method: GET 2023-10-11T15:27:20.116 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.117 DEBUG:boto:Data: 2023-10-11T15:27:20.117 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.118 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.119 DEBUG:boto:Params: {} 2023-10-11T15:27:20.119 DEBUG:boto:Token: None 2023-10-11T15:27:20.120 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7f1f5f15ccf2ab6-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.123 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.124 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.125 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.125 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.125 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.126 DEBUG:boto:Method: GET 2023-10-11T15:27:20.126 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:20.127 DEBUG:boto:Data: 2023-10-11T15:27:20.127 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.128 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.128 DEBUG:boto:Params: {} 2023-10-11T15:27:20.129 DEBUG:boto:Token: None 2023-10-11T15:27:20.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f12185b3db1f8b81-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.135 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:20.136 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.136 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.136 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.137 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.137 DEBUG:boto:Method: GET 2023-10-11T15:27:20.138 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.138 DEBUG:boto:Data: 2023-10-11T15:27:20.138 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.139 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.139 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.140 DEBUG:boto:Params: {} 2023-10-11T15:27:20.140 DEBUG:boto:Token: None 2023-10-11T15:27:20.141 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.141 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ= 2023-10-11T15:27:20.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ=', 'Content-Length': '0'} 2023-10-11T15:27:20.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a9769286a95e532-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.144 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.145 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.145 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.146 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.146 DEBUG:boto:Method: GET 2023-10-11T15:27:20.146 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.147 DEBUG:boto:Data: 2023-10-11T15:27:20.147 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.148 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.149 DEBUG:boto:Params: {} 2023-10-11T15:27:20.149 DEBUG:boto:Token: None 2023-10-11T15:27:20.150 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ= 2023-10-11T15:27:20.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ=', 'Content-Length': '0'} 2023-10-11T15:27:20.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c75d2fa850797f0-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.198 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.199 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.199 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.200 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.200 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.200 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.201 DEBUG:boto:Data: 2023-10-11T15:27:20.201 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.202 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.202 DEBUG:boto:Params: {} 2023-10-11T15:27:20.203 DEBUG:boto:Token: None 2023-10-11T15:27:20.203 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4aoAyV41dM1/pJ3XJneVZg/Nc3A= 2023-10-11T15:27:20.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4aoAyV41dM1/pJ3XJneVZg/Nc3A=', 'Content-Length': '0'} 2023-10-11T15:27:20.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ed87044ac320b4-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.206 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.206 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.207 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.207 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.208 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.208 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.208 DEBUG:boto:Data: 2023-10-11T15:27:20.209 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.210 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.210 DEBUG:boto:Params: {} 2023-10-11T15:27:20.210 DEBUG:boto:Token: None 2023-10-11T15:27:20.211 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4aoAyV41dM1/pJ3XJneVZg/Nc3A= 2023-10-11T15:27:20.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4aoAyV41dM1/pJ3XJneVZg/Nc3A=', 'Content-Length': '0'} 2023-10-11T15:27:20.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d6a625f5119aab4-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:20.215 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.215 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.216 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.216 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.216 DEBUG:boto:Method: GET 2023-10-11T15:27:20.217 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.217 DEBUG:boto:Data: 2023-10-11T15:27:20.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.219 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.219 DEBUG:boto:Params: {} 2023-10-11T15:27:20.219 DEBUG:boto:Token: None 2023-10-11T15:27:20.220 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ= 2023-10-11T15:27:20.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ=', 'Content-Length': '0'} 2023-10-11T15:27:20.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d442ba5b80aa1b29-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.224 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.224 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.224 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.225 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.225 DEBUG:boto:Method: GET 2023-10-11T15:27:20.226 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.226 DEBUG:boto:Data: 2023-10-11T15:27:20.226 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.227 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.228 DEBUG:boto:Params: {} 2023-10-11T15:27:20.228 DEBUG:boto:Token: None 2023-10-11T15:27:20.229 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:20.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ= 2023-10-11T15:27:20.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0fjgaJ+xgxy+fnmitYCKi3tF3AQ=', 'Content-Length': '0'} 2023-10-11T15:27:20.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c208c56c95faa4-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.233 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.234 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.234 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.235 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.235 DEBUG:boto:Method: GET 2023-10-11T15:27:20.236 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.236 DEBUG:boto:Data: 2023-10-11T15:27:20.236 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.237 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.238 DEBUG:boto:Params: {} 2023-10-11T15:27:20.238 DEBUG:boto:Token: None 2023-10-11T15:27:20.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f4d73540e67a3a0-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.282 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.283 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.283 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.284 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.284 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.285 DEBUG:boto:Method: GET 2023-10-11T15:27:20.285 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.285 DEBUG:boto:Data: 2023-10-11T15:27:20.286 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.287 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.287 DEBUG:boto:Params: {} 2023-10-11T15:27:20.288 DEBUG:boto:Token: None 2023-10-11T15:27:20.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b37a9af59ed3f64-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.294 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.295 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.295 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.295 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.296 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:20.296 DEBUG:boto:Data: 2023-10-11T15:27:20.296 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.297 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.298 DEBUG:boto:Params: {} 2023-10-11T15:27:20.298 DEBUG:boto:Token: None 2023-10-11T15:27:20.298 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87 2023-10-11T15:27:20.299 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oUx1g2eFgCaT6NXno5aBRyd+g1U= 2023-10-11T15:27:20.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oUx1g2eFgCaT6NXno5aBRyd+g1U=', 'Content-Length': '0'} 2023-10-11T15:27:20.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000661d874120c10cc6-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.302 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.302 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.303 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.303 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:20.303 DEBUG:boto:Data: 2023-10-11T15:27:20.304 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.304 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.305 DEBUG:boto:Params: {} 2023-10-11T15:27:20.305 DEBUG:boto:Token: None 2023-10-11T15:27:20.306 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87 2023-10-11T15:27:20.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oUx1g2eFgCaT6NXno5aBRyd+g1U= 2023-10-11T15:27:20.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oUx1g2eFgCaT6NXno5aBRyd+g1U=', 'Content-Length': '0'} 2023-10-11T15:27:20.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008275eea00f377a7-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:20.309 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.310 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.310 DEBUG:boto:Method: GET 2023-10-11T15:27:20.311 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:20.311 DEBUG:boto:Data: 2023-10-11T15:27:20.311 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.312 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.313 DEBUG:boto:Params: {} 2023-10-11T15:27:20.313 DEBUG:boto:Token: None 2023-10-11T15:27:20.314 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87 2023-10-11T15:27:20.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MWQor/uanUUYXZu+3dVAQIGCATk= 2023-10-11T15:27:20.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MWQor/uanUUYXZu+3dVAQIGCATk=', 'Content-Length': '0'} 2023-10-11T15:27:20.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8f55204e0203bb-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.317 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:20.318 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:20.318 DEBUG:boto:Method: GET 2023-10-11T15:27:20.318 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:20.319 DEBUG:boto:Data: 2023-10-11T15:27:20.319 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.320 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.320 DEBUG:boto:Params: {} 2023-10-11T15:27:20.321 DEBUG:boto:Token: None 2023-10-11T15:27:20.321 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj87 2023-10-11T15:27:20.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MWQor/uanUUYXZu+3dVAQIGCATk= 2023-10-11T15:27:20.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MWQor/uanUUYXZu+3dVAQIGCATk=', 'Content-Length': '0'} 2023-10-11T15:27:20.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37ef476b9cf093d-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.325 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.325 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.326 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.326 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.326 DEBUG:boto:Method: GET 2023-10-11T15:27:20.327 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.327 DEBUG:boto:Data: 2023-10-11T15:27:20.328 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.328 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.329 DEBUG:boto:Params: {} 2023-10-11T15:27:20.329 DEBUG:boto:Token: None 2023-10-11T15:27:20.330 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.330 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d8e2f114d2fef86-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.334 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.334 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.335 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.335 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.336 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.336 DEBUG:boto:Method: GET 2023-10-11T15:27:20.336 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:20.337 DEBUG:boto:Data: 2023-10-11T15:27:20.337 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.338 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.338 DEBUG:boto:Params: {} 2023-10-11T15:27:20.339 DEBUG:boto:Token: None 2023-10-11T15:27:20.339 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.340 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31fe74e1a988b78-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.345 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:20.347 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.347 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.347 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.348 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.348 DEBUG:boto:Method: GET 2023-10-11T15:27:20.348 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.349 DEBUG:boto:Data: 2023-10-11T15:27:20.349 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.350 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.351 DEBUG:boto:Params: {} 2023-10-11T15:27:20.351 DEBUG:boto:Token: None 2023-10-11T15:27:20.351 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0= 2023-10-11T15:27:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0=', 'Content-Length': '0'} 2023-10-11T15:27:20.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e80ba44b0ba0ca-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.355 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.356 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.356 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.356 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.357 DEBUG:boto:Method: GET 2023-10-11T15:27:20.357 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.358 DEBUG:boto:Data: 2023-10-11T15:27:20.358 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.359 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.359 DEBUG:boto:Params: {} 2023-10-11T15:27:20.360 DEBUG:boto:Token: None 2023-10-11T15:27:20.360 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0= 2023-10-11T15:27:20.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0=', 'Content-Length': '0'} 2023-10-11T15:27:20.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1bb301ca9be47d3-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.407 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.407 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.408 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.408 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.408 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.409 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.409 DEBUG:boto:Data: 2023-10-11T15:27:20.410 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.410 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.411 DEBUG:boto:Params: {} 2023-10-11T15:27:20.411 DEBUG:boto:Token: None 2023-10-11T15:27:20.412 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JRAsdIvZP0C5+qxRJMVMkqRKef0= 2023-10-11T15:27:20.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JRAsdIvZP0C5+qxRJMVMkqRKef0=', 'Content-Length': '0'} 2023-10-11T15:27:20.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f74693c29b3caf-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.415 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.416 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.416 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.417 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.417 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.417 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.418 DEBUG:boto:Data: 2023-10-11T15:27:20.418 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.419 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.419 DEBUG:boto:Params: {} 2023-10-11T15:27:20.420 DEBUG:boto:Token: None 2023-10-11T15:27:20.420 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JRAsdIvZP0C5+qxRJMVMkqRKef0= 2023-10-11T15:27:20.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JRAsdIvZP0C5+qxRJMVMkqRKef0=', 'Content-Length': '0'} 2023-10-11T15:27:20.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d634d120f5f75a-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:20.425 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.425 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.425 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.426 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.426 DEBUG:boto:Method: GET 2023-10-11T15:27:20.427 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.427 DEBUG:boto:Data: 2023-10-11T15:27:20.428 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.428 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.428 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.429 DEBUG:boto:Params: {} 2023-10-11T15:27:20.429 DEBUG:boto:Token: None 2023-10-11T15:27:20.430 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0= 2023-10-11T15:27:20.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0=', 'Content-Length': '0'} 2023-10-11T15:27:20.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337803c9caa22e7b-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.433 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.434 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.434 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.435 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.435 DEBUG:boto:Method: GET 2023-10-11T15:27:20.436 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.436 DEBUG:boto:Data: 2023-10-11T15:27:20.436 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.437 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.438 DEBUG:boto:Params: {} 2023-10-11T15:27:20.438 DEBUG:boto:Token: None 2023-10-11T15:27:20.439 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:20.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0= 2023-10-11T15:27:20.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wyk6wNmsmWOjlEkvN81JSTtHRy0=', 'Content-Length': '0'} 2023-10-11T15:27:20.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d2e920864671bc-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.442 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.443 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.443 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.444 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.444 DEBUG:boto:Method: GET 2023-10-11T15:27:20.444 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.445 DEBUG:boto:Data: 2023-10-11T15:27:20.445 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.446 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.446 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.446 DEBUG:boto:Params: {} 2023-10-11T15:27:20.447 DEBUG:boto:Token: None 2023-10-11T15:27:20.447 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.448 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025cd14b0dedef696-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.451 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.452 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.452 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.453 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.453 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.454 DEBUG:boto:Method: GET 2023-10-11T15:27:20.454 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.455 DEBUG:boto:Data: 2023-10-11T15:27:20.455 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.456 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.456 DEBUG:boto:Params: {} 2023-10-11T15:27:20.457 DEBUG:boto:Token: None 2023-10-11T15:27:20.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbf981a043780c05-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.463 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.464 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.464 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.464 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.465 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:20.465 DEBUG:boto:Data: 2023-10-11T15:27:20.466 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.466 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.467 DEBUG:boto:Params: {} 2023-10-11T15:27:20.467 DEBUG:boto:Token: None 2023-10-11T15:27:20.468 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92 2023-10-11T15:27:20.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SpbOobsn3+8Z6xeTEspGs0AgjLk= 2023-10-11T15:27:20.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SpbOobsn3+8Z6xeTEspGs0AgjLk=', 'Content-Length': '0'} 2023-10-11T15:27:20.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226666db583c72e2-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.471 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.472 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.472 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.473 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:20.473 DEBUG:boto:Data: 2023-10-11T15:27:20.473 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.474 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.474 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.475 DEBUG:boto:Params: {} 2023-10-11T15:27:20.475 DEBUG:boto:Token: None 2023-10-11T15:27:20.475 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92 2023-10-11T15:27:20.476 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SpbOobsn3+8Z6xeTEspGs0AgjLk= 2023-10-11T15:27:20.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SpbOobsn3+8Z6xeTEspGs0AgjLk=', 'Content-Length': '0'} 2023-10-11T15:27:20.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3172c3a0094b6d-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:20.479 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.480 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.480 DEBUG:boto:Method: GET 2023-10-11T15:27:20.480 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:20.481 DEBUG:boto:Data: 2023-10-11T15:27:20.481 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.482 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.482 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.482 DEBUG:boto:Params: {} 2023-10-11T15:27:20.483 DEBUG:boto:Token: None 2023-10-11T15:27:20.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92 2023-10-11T15:27:20.484 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4CLe3kbNfP5FmGiTKmOj4Z9g2SI= 2023-10-11T15:27:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4CLe3kbNfP5FmGiTKmOj4Z9g2SI=', 'Content-Length': '0'} 2023-10-11T15:27:20.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000533b9c0e9017e3a9-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.526 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:20.527 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:20.527 DEBUG:boto:Method: GET 2023-10-11T15:27:20.528 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:20.528 DEBUG:boto:Data: 2023-10-11T15:27:20.528 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.529 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.530 DEBUG:boto:Params: {} 2023-10-11T15:27:20.530 DEBUG:boto:Token: None 2023-10-11T15:27:20.530 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj92 2023-10-11T15:27:20.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4CLe3kbNfP5FmGiTKmOj4Z9g2SI= 2023-10-11T15:27:20.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4CLe3kbNfP5FmGiTKmOj4Z9g2SI=', 'Content-Length': '0'} 2023-10-11T15:27:20.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c08fdaa746853a9-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.534 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.535 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.535 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.535 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.536 DEBUG:boto:Method: GET 2023-10-11T15:27:20.536 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.537 DEBUG:boto:Data: 2023-10-11T15:27:20.537 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.538 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.538 DEBUG:boto:Params: {} 2023-10-11T15:27:20.539 DEBUG:boto:Token: None 2023-10-11T15:27:20.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c4c3cd51132944-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.543 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.544 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.544 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.545 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.545 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.546 DEBUG:boto:Method: GET 2023-10-11T15:27:20.546 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:20.547 DEBUG:boto:Data: 2023-10-11T15:27:20.547 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.548 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.548 DEBUG:boto:Params: {} 2023-10-11T15:27:20.549 DEBUG:boto:Token: None 2023-10-11T15:27:20.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e90ba41e525c8bd9-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.555 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:20.556 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.556 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.557 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.557 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.558 DEBUG:boto:Method: GET 2023-10-11T15:27:20.558 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.559 DEBUG:boto:Data: 2023-10-11T15:27:20.559 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.560 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.560 DEBUG:boto:Params: {} 2023-10-11T15:27:20.561 DEBUG:boto:Token: None 2023-10-11T15:27:20.561 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M= 2023-10-11T15:27:20.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M=', 'Content-Length': '0'} 2023-10-11T15:27:20.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007609d3a8a6282986-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.564 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.565 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.565 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.566 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.566 DEBUG:boto:Method: GET 2023-10-11T15:27:20.567 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.567 DEBUG:boto:Data: 2023-10-11T15:27:20.568 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.568 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.568 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.569 DEBUG:boto:Params: {} 2023-10-11T15:27:20.569 DEBUG:boto:Token: None 2023-10-11T15:27:20.570 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.570 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M= 2023-10-11T15:27:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M=', 'Content-Length': '0'} 2023-10-11T15:27:20.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002907fa4af165754-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.573 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.573 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.574 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.574 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.575 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.575 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.575 DEBUG:boto:Data: 2023-10-11T15:27:20.576 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.576 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.577 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.577 DEBUG:boto:Params: {} 2023-10-11T15:27:20.577 DEBUG:boto:Token: None 2023-10-11T15:27:20.578 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.578 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T7AICWhYmh/kABUIzAQ+Hs+W4pM= 2023-10-11T15:27:20.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T7AICWhYmh/kABUIzAQ+Hs+W4pM=', 'Content-Length': '0'} 2023-10-11T15:27:20.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6621bd0ccb20b4-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.581 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.582 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.582 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.583 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.583 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.583 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.584 DEBUG:boto:Data: 2023-10-11T15:27:20.584 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.585 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.585 DEBUG:boto:Params: {} 2023-10-11T15:27:20.586 DEBUG:boto:Token: None 2023-10-11T15:27:20.586 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T7AICWhYmh/kABUIzAQ+Hs+W4pM= 2023-10-11T15:27:20.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T7AICWhYmh/kABUIzAQ+Hs+W4pM=', 'Content-Length': '0'} 2023-10-11T15:27:20.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcf344891a7dcdf-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:20.590 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.590 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.591 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.591 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.591 DEBUG:boto:Method: GET 2023-10-11T15:27:20.592 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.592 DEBUG:boto:Data: 2023-10-11T15:27:20.593 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.593 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.594 DEBUG:boto:Params: {} 2023-10-11T15:27:20.594 DEBUG:boto:Token: None 2023-10-11T15:27:20.595 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M= 2023-10-11T15:27:20.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M=', 'Content-Length': '0'} 2023-10-11T15:27:20.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017c877e3f54cadf3-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.598 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.599 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.599 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.599 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.600 DEBUG:boto:Method: GET 2023-10-11T15:27:20.600 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.601 DEBUG:boto:Data: 2023-10-11T15:27:20.601 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.602 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.602 DEBUG:boto:Params: {} 2023-10-11T15:27:20.603 DEBUG:boto:Token: None 2023-10-11T15:27:20.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:20.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M= 2023-10-11T15:27:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8dqROfInIm1b77QJS7GGH3HN9M=', 'Content-Length': '0'} 2023-10-11T15:27:20.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a52cfc05c75324-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.607 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.607 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.607 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.608 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.608 DEBUG:boto:Method: GET 2023-10-11T15:27:20.609 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.609 DEBUG:boto:Data: 2023-10-11T15:27:20.609 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.610 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.611 DEBUG:boto:Params: {} 2023-10-11T15:27:20.611 DEBUG:boto:Token: None 2023-10-11T15:27:20.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a18df05c552f357-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.615 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.616 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.616 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.617 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.617 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.618 DEBUG:boto:Method: GET 2023-10-11T15:27:20.618 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.619 DEBUG:boto:Data: 2023-10-11T15:27:20.619 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.620 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.620 DEBUG:boto:Params: {} 2023-10-11T15:27:20.620 DEBUG:boto:Token: None 2023-10-11T15:27:20.621 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001494d8f09e0a1dc7-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.626 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.627 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.627 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.628 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.628 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:20.629 DEBUG:boto:Data: 2023-10-11T15:27:20.629 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.629 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.630 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.630 DEBUG:boto:Params: {} 2023-10-11T15:27:20.631 DEBUG:boto:Token: None 2023-10-11T15:27:20.631 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93 2023-10-11T15:27:20.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9EfrBfuzyJaUmJMfJAbXTEfX59E= 2023-10-11T15:27:20.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9EfrBfuzyJaUmJMfJAbXTEfX59E=', 'Content-Length': '0'} 2023-10-11T15:27:20.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f5c56a85109a806-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.635 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.635 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.635 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.636 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:20.636 DEBUG:boto:Data: 2023-10-11T15:27:20.637 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.638 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.638 DEBUG:boto:Params: {} 2023-10-11T15:27:20.641 DEBUG:boto:Token: None 2023-10-11T15:27:20.642 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93 2023-10-11T15:27:20.642 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9EfrBfuzyJaUmJMfJAbXTEfX59E= 2023-10-11T15:27:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9EfrBfuzyJaUmJMfJAbXTEfX59E=', 'Content-Length': '0'} 2023-10-11T15:27:20.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfde883f560eb2da-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:20.646 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.646 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.647 DEBUG:boto:Method: GET 2023-10-11T15:27:20.647 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:20.647 DEBUG:boto:Data: 2023-10-11T15:27:20.648 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.648 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.649 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.649 DEBUG:boto:Params: {} 2023-10-11T15:27:20.650 DEBUG:boto:Token: None 2023-10-11T15:27:20.650 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93 2023-10-11T15:27:20.651 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gArbPsUSwQciGFtnvsRWOXcHqbE= 2023-10-11T15:27:20.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gArbPsUSwQciGFtnvsRWOXcHqbE=', 'Content-Length': '0'} 2023-10-11T15:27:20.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d60db87ae5b5cd-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.694 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:20.695 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:20.695 DEBUG:boto:Method: GET 2023-10-11T15:27:20.696 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:20.696 DEBUG:boto:Data: 2023-10-11T15:27:20.697 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.698 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.698 DEBUG:boto:Params: {} 2023-10-11T15:27:20.699 DEBUG:boto:Token: None 2023-10-11T15:27:20.699 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj93 2023-10-11T15:27:20.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gArbPsUSwQciGFtnvsRWOXcHqbE= 2023-10-11T15:27:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gArbPsUSwQciGFtnvsRWOXcHqbE=', 'Content-Length': '0'} 2023-10-11T15:27:20.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b157cfc7343df4-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.703 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.703 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.704 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.704 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.705 DEBUG:boto:Method: GET 2023-10-11T15:27:20.705 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.705 DEBUG:boto:Data: 2023-10-11T15:27:20.706 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.707 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.707 DEBUG:boto:Params: {} 2023-10-11T15:27:20.708 DEBUG:boto:Token: None 2023-10-11T15:27:20.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359a6ff685f03b08-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.712 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.713 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.713 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.714 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.715 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.715 DEBUG:boto:Method: GET 2023-10-11T15:27:20.715 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:20.716 DEBUG:boto:Data: 2023-10-11T15:27:20.716 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.717 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.718 DEBUG:boto:Params: {} 2023-10-11T15:27:20.718 DEBUG:boto:Token: None 2023-10-11T15:27:20.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bc3c9f49708e6d-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.725 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:20.726 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.727 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.727 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.728 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.728 DEBUG:boto:Method: GET 2023-10-11T15:27:20.728 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.729 DEBUG:boto:Data: 2023-10-11T15:27:20.729 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.730 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.731 DEBUG:boto:Params: {} 2023-10-11T15:27:20.731 DEBUG:boto:Token: None 2023-10-11T15:27:20.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko= 2023-10-11T15:27:20.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko=', 'Content-Length': '0'} 2023-10-11T15:27:20.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc5624168563928-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.736 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.736 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.737 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.737 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.738 DEBUG:boto:Method: GET 2023-10-11T15:27:20.738 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.739 DEBUG:boto:Data: 2023-10-11T15:27:20.739 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.740 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.740 DEBUG:boto:Params: {} 2023-10-11T15:27:20.741 DEBUG:boto:Token: None 2023-10-11T15:27:20.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko= 2023-10-11T15:27:20.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko=', 'Content-Length': '0'} 2023-10-11T15:27:20.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000896320f570877fee-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.790 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.791 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.791 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.792 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.792 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.793 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.793 DEBUG:boto:Data: 2023-10-11T15:27:20.794 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.795 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.795 DEBUG:boto:Params: {} 2023-10-11T15:27:20.796 DEBUG:boto:Token: None 2023-10-11T15:27:20.796 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XEDnvQY5xKjS1roQhzjL38GVxlk= 2023-10-11T15:27:20.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XEDnvQY5xKjS1roQhzjL38GVxlk=', 'Content-Length': '0'} 2023-10-11T15:27:20.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ff819a9b30f227-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.800 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.800 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.801 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.801 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.802 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.802 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.803 DEBUG:boto:Data: 2023-10-11T15:27:20.803 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.804 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.804 DEBUG:boto:Params: {} 2023-10-11T15:27:20.805 DEBUG:boto:Token: None 2023-10-11T15:27:20.805 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XEDnvQY5xKjS1roQhzjL38GVxlk= 2023-10-11T15:27:20.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XEDnvQY5xKjS1roQhzjL38GVxlk=', 'Content-Length': '0'} 2023-10-11T15:27:20.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b020f44cc68cc71c-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:20.809 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.810 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.810 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.811 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.811 DEBUG:boto:Method: GET 2023-10-11T15:27:20.811 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.812 DEBUG:boto:Data: 2023-10-11T15:27:20.812 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.813 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.813 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.814 DEBUG:boto:Params: {} 2023-10-11T15:27:20.814 DEBUG:boto:Token: None 2023-10-11T15:27:20.815 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.815 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko= 2023-10-11T15:27:20.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko=', 'Content-Length': '0'} 2023-10-11T15:27:20.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022b0f2e0136f7ba1-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.862 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.863 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.863 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.864 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.864 DEBUG:boto:Method: GET 2023-10-11T15:27:20.865 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.865 DEBUG:boto:Data: 2023-10-11T15:27:20.865 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.866 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.867 DEBUG:boto:Params: {} 2023-10-11T15:27:20.867 DEBUG:boto:Token: None 2023-10-11T15:27:20.868 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:20.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko= 2023-10-11T15:27:20.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zL6EW1SMhoBh9ZsIvdQUTZVmTko=', 'Content-Length': '0'} 2023-10-11T15:27:20.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f328b40e1dee570-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.871 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.872 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.872 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.872 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.873 DEBUG:boto:Method: GET 2023-10-11T15:27:20.873 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.874 DEBUG:boto:Data: 2023-10-11T15:27:20.874 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.875 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.875 DEBUG:boto:Params: {} 2023-10-11T15:27:20.876 DEBUG:boto:Token: None 2023-10-11T15:27:20.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c997e9ce8291a076-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.880 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.881 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.882 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.883 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.883 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.884 DEBUG:boto:Method: GET 2023-10-11T15:27:20.884 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.884 DEBUG:boto:Data: 2023-10-11T15:27:20.885 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.886 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.886 DEBUG:boto:Params: {} 2023-10-11T15:27:20.887 DEBUG:boto:Token: None 2023-10-11T15:27:20.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c99592f7cf8de47-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.893 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.893 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.894 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.894 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.895 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:20.895 DEBUG:boto:Data: 2023-10-11T15:27:20.895 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.896 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.897 DEBUG:boto:Params: {} 2023-10-11T15:27:20.897 DEBUG:boto:Token: None 2023-10-11T15:27:20.897 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95 2023-10-11T15:27:20.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7G3JbPT/foFHg7bXwKSjdfwVsyY= 2023-10-11T15:27:20.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7G3JbPT/foFHg7bXwKSjdfwVsyY=', 'Content-Length': '0'} 2023-10-11T15:27:20.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a23e76dadf5a7a-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.901 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.901 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.902 DEBUG:boto:Method: HEAD 2023-10-11T15:27:20.902 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:20.903 DEBUG:boto:Data: 2023-10-11T15:27:20.903 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.904 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.904 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.904 DEBUG:boto:Params: {} 2023-10-11T15:27:20.905 DEBUG:boto:Token: None 2023-10-11T15:27:20.905 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95 2023-10-11T15:27:20.906 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7G3JbPT/foFHg7bXwKSjdfwVsyY= 2023-10-11T15:27:20.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7G3JbPT/foFHg7bXwKSjdfwVsyY=', 'Content-Length': '0'} 2023-10-11T15:27:20.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1d7f9644d25872-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:20.909 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.909 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.909 DEBUG:boto:Method: GET 2023-10-11T15:27:20.910 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:20.910 DEBUG:boto:Data: 2023-10-11T15:27:20.911 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.912 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.912 DEBUG:boto:Params: {} 2023-10-11T15:27:20.913 DEBUG:boto:Token: None 2023-10-11T15:27:20.913 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95 2023-10-11T15:27:20.914 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ukherfEpWu7Rd27RmtyXa0/J6XU= 2023-10-11T15:27:20.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ukherfEpWu7Rd27RmtyXa0/J6XU=', 'Content-Length': '0'} 2023-10-11T15:27:20.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009112287db9fa4d1f-006526bed8-1b7d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.917 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:20.917 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:20.917 DEBUG:boto:Method: GET 2023-10-11T15:27:20.918 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:20.918 DEBUG:boto:Data: 2023-10-11T15:27:20.919 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.920 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.920 DEBUG:boto:Params: {} 2023-10-11T15:27:20.921 DEBUG:boto:Token: None 2023-10-11T15:27:20.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/obj95 2023-10-11T15:27:20.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ukherfEpWu7Rd27RmtyXa0/J6XU= 2023-10-11T15:27:20.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ukherfEpWu7Rd27RmtyXa0/J6XU=', 'Content-Length': '0'} 2023-10-11T15:27:20.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb75c6d154cc70b7-006526bed8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.966 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.967 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.967 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.968 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.968 DEBUG:boto:Method: GET 2023-10-11T15:27:20.968 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.969 DEBUG:boto:Data: 2023-10-11T15:27:20.969 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:20.970 DEBUG:boto:Port: 8001 2023-10-11T15:27:20.970 DEBUG:boto:Params: {} 2023-10-11T15:27:20.971 DEBUG:boto:Token: None 2023-10-11T15:27:20.971 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad77d79e77fd2e70-006526bed8-1b7d-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.976 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.977 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:20.977 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:20.977 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.978 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.978 DEBUG:boto:Method: GET 2023-10-11T15:27:20.979 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:20.979 DEBUG:boto:Data: 2023-10-11T15:27:20.980 DEBUG:boto:Headers: {} 2023-10-11T15:27:20.980 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:20.981 DEBUG:boto:Port: 8000 2023-10-11T15:27:20.981 DEBUG:boto:Params: {} 2023-10-11T15:27:20.982 DEBUG:boto:Token: None 2023-10-11T15:27:20.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:20 GMT /dghqvd-59/ 2023-10-11T15:27:20.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU= 2023-10-11T15:27:20.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:20 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Be478a+A5Nf65q8LCGBeO0n1BIU=', 'Content-Length': '0'} 2023-10-11T15:27:20.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b62874545b3ba5f6-006526bed8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:20 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:20.989 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:20.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:27:20.990 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:27:20.991 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:27:20.992 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:27:20.990+0000 7f410f239700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 160567) UID: 0 2023-10-11T15:27:20.992 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:27:20.990+0000 7f41134a27c0 -1 shutting down 2023-10-11T15:27:23.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:27:27.089 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:27:27.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:27.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:27.091 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:27.150 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.150 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:27.184 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:27:27.185 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.185 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj11", 2023-10-11T15:27:27.186 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.186 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.186 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.187 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:27.187 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.187 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.188 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.188 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.189 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.189 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.189 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.917502Z", 2023-10-11T15:27:27.190 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.190 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.190 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.191 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.191 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.192 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.192 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.192 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.193 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.193 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_CkRPk0h0KQt0FxtlV2C_X_C8vqG-huA", 2023-10-11T15:27:27.193 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.194 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.194 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.199 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.200 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.200 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj111", 2023-10-11T15:27:27.200 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.201 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.201 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.201 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:27.202 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.202 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.202 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.203 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.203 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.203 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.204 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.282651Z", 2023-10-11T15:27:27.204 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.204 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.205 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.205 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.205 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.206 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.206 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.207 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_YsO9ueN3bi4PCKF8I4cy-9UTP652FUF", 2023-10-11T15:27:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.208 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj115", 2023-10-11T15:27:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.209 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.210 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.481238Z", 2023-10-11T15:27:27.211 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.212 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.213 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.213 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.213 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_IV3qTckjg6j7qH7V6YvRPX_fYNILCFm", 2023-10-11T15:27:27.213 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.213 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.214 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj116", 2023-10-11T15:27:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.215 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.216 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.474371Z", 2023-10-11T15:27:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.217 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.218 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_i6a5dUOswCe9An0bKUaDdtZkOa1AvFs", 2023-10-11T15:27:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.219 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj124", 2023-10-11T15:27:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.220 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.221 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.864790Z", 2023-10-11T15:27:27.222 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.223 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.224 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_BfnWM7D5tndBIpkALCbZJ79TL8Uqpy0", 2023-10-11T15:27:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.225 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj145", 2023-10-11T15:27:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.226 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.227 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.228 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.228 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.228 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.493223Z", 2023-10-11T15:27:27.228 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.229 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.229 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.229 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.229 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.229 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.230 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.230 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.230 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.230 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_zgxJhG3MovKAzMHjye1eoG-aEYGYNqe", 2023-10-11T15:27:27.231 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.231 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.231 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.231 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.232 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj148", 2023-10-11T15:27:27.232 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.232 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.232 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.232 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.233 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.233 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.233 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.233 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.233 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.234 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.234 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.302431Z", 2023-10-11T15:27:27.234 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.234 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.234 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.235 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.235 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.235 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.235 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.236 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.236 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.236 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_JqfNCEMSJfu7_HUFozUQtnmDCZZhBwP", 2023-10-11T15:27:27.236 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.236 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.237 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.237 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.237 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.237 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj155", 2023-10-11T15:27:27.238 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.238 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.238 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.238 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.238 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.239 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.239 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.239 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.239 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.239 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.240 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.923366Z", 2023-10-11T15:27:27.240 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.240 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.240 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.241 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.242 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.242 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Fd-iPFvSlbbxqAxftLwNk7rTzo8IvSB", 2023-10-11T15:27:27.242 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.242 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.243 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.243 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.243 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.243 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj17", 2023-10-11T15:27:27.243 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.244 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.244 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.244 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:27.244 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.244 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.245 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.245 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.245 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.245 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.245 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.599178Z", 2023-10-11T15:27:27.246 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.246 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.246 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.246 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.247 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_XuxGgID3ZRTJR6HtQLtx_rRNIBRyUY1", 2023-10-11T15:27:27.248 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.248 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.248 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.248 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.249 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj185", 2023-10-11T15:27:27.249 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.249 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.249 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.250 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:27:27.250 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.250 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.250 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.250 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.251 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.251 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.251 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.362612Z", 2023-10-11T15:27:27.251 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.251 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.252 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.253 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.253 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.253 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Y3sXR4NFyQ2dhb5UGrafp5tQTfowzdD", 2023-10-11T15:27:27.253 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.253 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.254 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.254 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.254 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.254 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj19", 2023-10-11T15:27:27.254 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.255 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.255 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.255 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:27.255 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.255 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.256 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.256 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.256 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.256 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.256 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.348518Z", 2023-10-11T15:27:27.257 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.257 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.257 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.257 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.257 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.258 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.258 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.258 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.258 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "__EJHdnsTxtTxcJsLcwqOHTcGefqNEan", 2023-10-11T15:27:27.259 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.259 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.259 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.259 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.260 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj22", 2023-10-11T15:27:27.260 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.260 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.260 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.260 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:27:27.261 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.261 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.261 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.261 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.261 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.262 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.262 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.938656Z", 2023-10-11T15:27:27.262 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.262 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.263 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.263 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.263 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.263 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.263 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.264 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.264 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.264 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_-qXaEPF1Q-irL3dRggGm_IKLdhk-QUP", 2023-10-11T15:27:27.264 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.264 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.265 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.265 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.265 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj23", 2023-10-11T15:27:27.266 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.266 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.266 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.266 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:27.267 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.267 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.267 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.267 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.267 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.268 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.268 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.324750Z", 2023-10-11T15:27:27.268 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.268 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.268 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.270 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_wmLUHS3mxwUzCTxfVU-OpyzsNfMtaZ2", 2023-10-11T15:27:27.270 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.270 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.270 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.271 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.271 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.271 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj29", 2023-10-11T15:27:27.271 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.271 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.272 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.272 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:27:27.272 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.272 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.272 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.330113Z", 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.273 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.274 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.274 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.274 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.274 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.274 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.275 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.275 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.275 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_o0m3pexMh8Z9hxzo1sFOYnOHpqgTmyb", 2023-10-11T15:27:27.275 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.275 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.276 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.276 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.276 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj31", 2023-10-11T15:27:27.276 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.277 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.277 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.277 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.277 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.278 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.278 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.278 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.278 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.279 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.805358Z", 2023-10-11T15:27:27.279 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.279 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.279 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.279 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.280 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.280 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.280 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.280 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.281 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_o6Pu69CHC4sY_rwyG6bDyx2b18E2ymF", 2023-10-11T15:27:27.281 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.281 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.281 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.281 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.282 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.282 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj33", 2023-10-11T15:27:27.282 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.282 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.283 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.283 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.283 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.283 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.284 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.284 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.284 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.284 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.957164Z", 2023-10-11T15:27:27.285 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.285 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.285 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.285 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.285 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.286 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.286 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.286 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.286 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_a28Hoqjjip_XURmvBJBe2cY9HSfgJYC", 2023-10-11T15:27:27.287 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.287 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.287 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.287 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.288 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj34", 2023-10-11T15:27:27.288 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.288 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.288 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.289 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.289 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.289 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.289 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.289 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.290 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.290 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.290 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.660795Z", 2023-10-11T15:27:27.290 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.290 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.291 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.291 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.291 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.291 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.291 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.292 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.292 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_qdZ5VX0iceAjG6tXz_gfBwSWunwlAuE", 2023-10-11T15:27:27.292 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.292 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.293 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.293 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.293 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.293 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj37", 2023-10-11T15:27:27.294 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.294 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.294 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.294 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:27.294 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.295 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.295 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.295 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.295 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.295 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.296 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.203398Z", 2023-10-11T15:27:27.296 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.296 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.296 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.296 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.297 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.297 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.297 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.297 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.298 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_VvCSwIE_WY6jsQDrko7x06LJ55Tuxf-", 2023-10-11T15:27:27.298 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.298 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.298 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.299 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.299 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.299 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj38", 2023-10-11T15:27:27.299 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.300 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.301 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.367798Z", 2023-10-11T15:27:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.302 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.303 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Malw7P7He7kQUcdRcNjTSdfvW-IUjb9", 2023-10-11T15:27:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj44", 2023-10-11T15:27:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.305 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.306 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.013924Z", 2023-10-11T15:27:27.307 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.308 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_m3u6ixowOd2IWlY8v7qzOp4Bs9Htm10", 2023-10-11T15:27:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.309 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.310 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj46", 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.311 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.312 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.312 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.312 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.312 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.313 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.344708Z", 2023-10-11T15:27:27.313 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.313 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.313 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.313 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.314 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.315 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_6DV65NtqdEzyVFU6CYtzdjqyxOQSxCk", 2023-10-11T15:27:27.315 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.315 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.315 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.315 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.316 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.316 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj51", 2023-10-11T15:27:27.316 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.316 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.316 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.317 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:27:27.317 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.317 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.317 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.317 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.318 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.318 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.318 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.682129Z", 2023-10-11T15:27:27.318 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.318 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.319 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.319 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.319 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.319 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.319 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.320 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.320 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.320 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_FIuZyenng4ffZg6f8X-C1RTEdlznxX9", 2023-10-11T15:27:27.320 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.321 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.321 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.321 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.321 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.321 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj55", 2023-10-11T15:27:27.322 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.322 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.322 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.322 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:27.322 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.323 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.323 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.323 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.323 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.324 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.324 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.032675Z", 2023-10-11T15:27:27.324 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.324 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.324 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.325 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.325 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.325 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.325 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.325 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.326 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.326 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_SyVviBv7Bz9FJUtPL-fphZ0QjctknPo", 2023-10-11T15:27:27.326 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.326 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.326 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.327 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.327 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.327 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj57", 2023-10-11T15:27:27.327 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.327 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.328 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.328 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.328 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.328 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.328 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.329 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.329 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.329 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.329 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.384920Z", 2023-10-11T15:27:27.329 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.330 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.330 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.330 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.330 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.331 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.331 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.331 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.331 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.331 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Lfvf0fVcFZvpWJSofyMURiyFZw9qjTo", 2023-10-11T15:27:27.332 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.332 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.332 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.332 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.332 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.333 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj58", 2023-10-11T15:27:27.333 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.333 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.333 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:27:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.334 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.335 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.335 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.335 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.390163Z", 2023-10-11T15:27:27.335 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.336 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.336 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.336 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.336 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.336 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.337 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.337 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.337 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.337 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_EmdT1DyUDGhCkI8I1A6gMAnM2WtX9hK", 2023-10-11T15:27:27.337 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.338 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.338 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.338 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.338 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.338 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj62", 2023-10-11T15:27:27.339 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.339 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.339 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.339 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:27.339 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.340 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.340 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.340 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.340 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.340 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.341 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.825386Z", 2023-10-11T15:27:27.341 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.341 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.341 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.341 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.342 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.342 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.342 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.342 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.342 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.343 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_2--tkCqlkfAPIBg4gibFQ0j8gwHIZA6", 2023-10-11T15:27:27.343 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.343 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.343 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.344 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.344 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.344 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj68", 2023-10-11T15:27:27.344 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.344 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.345 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.345 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:27:27.345 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.345 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.345 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.346 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.346 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.346 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.346 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.746100Z", 2023-10-11T15:27:27.346 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.347 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.347 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.347 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.347 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_5b0ZcX_6gNgZNJoO1mKGY7Ef13ec5yz", 2023-10-11T15:27:27.348 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.349 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.349 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.349 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.349 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.350 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj69", 2023-10-11T15:27:27.350 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.350 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.350 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.350 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:27:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.351 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.352 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:27.352 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.406312Z", 2023-10-11T15:27:27.352 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.352 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.353 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.353 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.353 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.353 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.353 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_zEIQwNCE_z5nvRN6A-aoZ0TmgDZqMf8", 2023-10-11T15:27:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.355 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.355 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.355 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.355 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj74", 2023-10-11T15:27:27.355 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.356 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.356 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.356 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:27.356 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.356 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.357 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.357 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.357 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.357 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.357 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.223624Z", 2023-10-11T15:27:27.358 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.358 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.358 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.358 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.358 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.359 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.359 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.359 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.359 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.359 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_XoNINh7yWN8C9KaAkKkpaTqQulo3ay-", 2023-10-11T15:27:27.360 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.360 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.360 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.360 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj76", 2023-10-11T15:27:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.361 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.362 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.363 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.363 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.363 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.446066Z", 2023-10-11T15:27:27.363 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.364 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.364 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.364 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.364 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.364 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.365 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.365 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.365 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.365 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_WJ7kntm_EDqh2fBCkLmaDsTzJ8uoISW", 2023-10-11T15:27:27.365 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.366 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.366 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.366 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.366 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj85", 2023-10-11T15:27:27.367 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.367 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.367 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.368 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.368 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.368 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.368 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.369 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.369 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.765362Z", 2023-10-11T15:27:27.369 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.369 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.369 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.370 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.370 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.370 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.370 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.370 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.371 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_rpA4YkkcyWbU1keZmhC5BKWizVG9BS6", 2023-10-11T15:27:27.371 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.371 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.372 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.372 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj87", 2023-10-11T15:27:27.372 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.373 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.373 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.373 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:27.373 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.373 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.374 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.375 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.375 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.376 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.376 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.410694Z", 2023-10-11T15:27:27.376 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.376 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.377 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.377 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.377 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.377 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.377 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.378 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.378 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_DcYMz9d5oOSROTUZbr0K3zYu_Fu39OT", 2023-10-11T15:27:27.378 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.379 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.379 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.379 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.379 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.379 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj92", 2023-10-11T15:27:27.380 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.380 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.380 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.380 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:27.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.381 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.381 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.381 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.381 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.382 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:27.382 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.424897Z", 2023-10-11T15:27:27.382 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.382 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.382 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.383 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.383 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.383 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.383 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.383 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.384 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.384 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_2GZIDV2vC1W2Z2xqluotnpHbTM7lSaB", 2023-10-11T15:27:27.384 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.384 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.385 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.385 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.385 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj93", 2023-10-11T15:27:27.385 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.386 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.386 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.386 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:27.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.387 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.387 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.387 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.387 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.387 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:27.388 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.844840Z", 2023-10-11T15:27:27.388 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.388 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.388 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.388 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.389 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.389 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.389 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.389 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.390 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.390 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_wthlPa8_PpbBcVQ1W5GtDa_Ey3Qsngz", 2023-10-11T15:27:27.390 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.390 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.390 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.391 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.391 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:27.391 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj95", 2023-10-11T15:27:27.391 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:27.392 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:27.392 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:27.392 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:27.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.392 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:27.393 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:27.393 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:27.393 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:27.393 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:27.393 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.464225Z", 2023-10-11T15:27:27.394 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:27.394 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:27.394 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:27.394 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:27.395 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:27.395 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:27.395 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:27.395 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:27.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:27.396 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_fzVj2xzXMHbPPNj9CakKkkoVeyaNJL_", 2023-10-11T15:27:27.396 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:27.396 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:27.396 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:27.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:27.397 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:27:27.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.403 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.458 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:27.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.543 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:27.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:27.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.676 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:27.727 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.728 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:27.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.810 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:27.861 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:27.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:27.944 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:27.995 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:27.996 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.079 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.130 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.214 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.267 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.267 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.357 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.409 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.493 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.544 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.544 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.627 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.764 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.815 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:28.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:28.898 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:28.950 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:28.950 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.034 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.085 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.086 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.168 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.220 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.220 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.304 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:27:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.356 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.439 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.491 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.579 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.712 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.765 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.848 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:29.899 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:29.899 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:29.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:29.984 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.035 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.036 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.119 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.170 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.171 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.257 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.311 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.395 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.448 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.448 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.533 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.585 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.585 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.669 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.720 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.721 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.806 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.858 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:30.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:30.945 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:30.996 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:30.996 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.081 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.133 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.218 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.270 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.270 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.355 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.407 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.408 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.492 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.543 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.628 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.764 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:31.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:31.900 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:31.953 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:31.954 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:32.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.038 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:32.091 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:32.091 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:32.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.177 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:32.227 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:32.228 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:32.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.283 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:32.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:32.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:32.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:27:32.381 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:27:32.434 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:32.434 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:32.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.500+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-10-11T15:27:32.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.500+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:32.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.506+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: init 2023-10-11T15:27:32.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.517+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:32.566 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.567 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-11T15:27:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:32.568 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:32.569 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj116[0] 2023-10-11T15:27:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj124[0] 2023-10-11T15:27:32.570 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj145[0] 2023-10-11T15:27:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:32.571 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj148[0] 2023-10-11T15:27:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-11T15:27:32.572 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:32.573 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj17[0] 2023-10-11T15:27:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.563+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:32.574 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:32.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:32.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:32.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:32.575 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-10-11T15:27:32.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:32.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:32.576 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:32.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:32.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:32.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:32.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-11T15:27:32.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.578 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:32.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:32.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:32.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:32.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:32.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:32.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:32.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:32.580 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:32.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-10-11T15:27:32.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:32.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:32.581 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:32.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:32.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:32.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:32.582 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:32.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:32.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:32.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:32.583 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:32.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:32.584 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.564+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:32.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.566+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.566+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:32.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.566+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:32.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.566+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:32.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:32.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:32.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj55[0] 2023-10-11T15:27:32.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:32.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:32.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.567+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj57[0] 2023-10-11T15:27:32.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.568+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.568+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:32.588 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.568+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:32.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.568+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:32.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.570+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.570+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:32.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.570+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:32.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.570+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj62[0] 2023-10-11T15:27:32.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:32.591 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: success 2023-10-11T15:27:32.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:32.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:32.592 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:32.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj68[0] 2023-10-11T15:27:32.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:32.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:32.593 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:32.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:32.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:32.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.586+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:32.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: success 2023-10-11T15:27:32.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: success 2023-10-11T15:27:32.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:32.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:32.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:32.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj76[0] 2023-10-11T15:27:32.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:32.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.587+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:32.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj85[0] 2023-10-11T15:27:32.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj87[0] 2023-10-11T15:27:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: success 2023-10-11T15:27:32.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:32.598 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.588+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: success 2023-10-11T15:27:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:32.599 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:32.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:32.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:32.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:32.600 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj93[0] 2023-10-11T15:27:32.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:32.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:32.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.590+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:32.601 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.591+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:32.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.591+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:32.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.591+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj11 2023-10-11T15:27:32.602 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.591+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:32.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.616+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:32.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.618+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: success 2023-10-11T15:27:32.620 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.619+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: success 2023-10-11T15:27:32.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.621+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:32.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.622+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: success 2023-10-11T15:27:32.624 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.623+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.634+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:32.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.634+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: success 2023-10-11T15:27:32.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.635+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:32.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.636+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: success 2023-10-11T15:27:32.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.637+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: success 2023-10-11T15:27:32.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.637+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: success 2023-10-11T15:27:32.639 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.637+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: success 2023-10-11T15:27:32.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.638+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:32.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.639+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:32.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.639+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: backing out with sync_status=-5 2023-10-11T15:27:32.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.639+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: ERROR: full sync failed. error: -5 2023-10-11T15:27:32.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.640+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-11T15:27:32.642 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.640+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:32.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.645+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: full 2023-10-11T15:27:32.657 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.656+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:32.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:32.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:32.704 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj111[0] 2023-10-11T15:27:32.705 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:32.706 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:32.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj116[0] 2023-10-11T15:27:32.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:32.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:32.707 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj124[0] 2023-10-11T15:27:32.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:32.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:32.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj145[0] 2023-10-11T15:27:32.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:32.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:32.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj148[0] 2023-10-11T15:27:32.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:32.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj155[0] 2023-10-11T15:27:32.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:32.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:32.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj17[0] 2023-10-11T15:27:32.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:32.710 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:32.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:32.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:32.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:32.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:32.711 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj22[0] 2023-10-11T15:27:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:32.712 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:32.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.701+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:32.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:32.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj29[0] 2023-10-11T15:27:32.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:32.713 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:32.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:32.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:32.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:32.714 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj34[0] 2023-10-11T15:27:32.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:32.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:32.717 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:32.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.702+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: success 2023-10-11T15:27:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:32.720 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.705+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj55[0] 2023-10-11T15:27:32.721 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: success 2023-10-11T15:27:32.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: success 2023-10-11T15:27:32.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:32.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:32.722 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj57[0] 2023-10-11T15:27:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:32.723 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj62[0] 2023-10-11T15:27:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:32.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-10-11T15:27:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:32.725 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: success 2023-10-11T15:27:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:32.726 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.707+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:32.727 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-10-11T15:27:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.728 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.708+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: success 2023-10-11T15:27:32.729 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:32.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:32.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj11 2023-10-11T15:27:32.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:32.730 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:32.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:32.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:32.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:32.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:32.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: success 2023-10-11T15:27:32.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: success 2023-10-11T15:27:32.733 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: success 2023-10-11T15:27:32.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj85[0] 2023-10-11T15:27:32.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:32.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-11T15:27:32.734 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:32.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:32.735 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:32.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:32.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:32.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:32.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj93[0] 2023-10-11T15:27:32.736 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:32.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:32.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.711+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:32.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.713+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:32.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.714+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.737 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.714+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.714+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:32.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.715+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: success 2023-10-11T15:27:32.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.715+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: success 2023-10-11T15:27:32.738 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.715+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: success 2023-10-11T15:27:32.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.715+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:32.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.715+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: success 2023-10-11T15:27:32.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.716+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: success 2023-10-11T15:27:32.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.716+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj23 2023-10-11T15:27:32.739 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.716+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: success 2023-10-11T15:27:32.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.718+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: success 2023-10-11T15:27:32.740 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.718+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj62 2023-10-11T15:27:32.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.719+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: backing out with sync_status=-5 2023-10-11T15:27:32.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.719+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: ERROR: full sync failed. error: -5 2023-10-11T15:27:32.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.720+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-11T15:27:32.741 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.720+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:32.768 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.767+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: full 2023-10-11T15:27:32.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.778+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:32.825 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:32.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:32.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:32.826 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj111[0] 2023-10-11T15:27:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:32.827 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:32.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:32.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:32.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:32.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-10-11T15:27:32.828 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:32.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:32.829 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:32.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj124[0] 2023-10-11T15:27:32.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:32.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:32.830 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj145[0] 2023-10-11T15:27:32.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:32.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:32.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj148[0] 2023-10-11T15:27:32.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:32.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-10-11T15:27:32.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.832 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:32.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:32.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj17[0] 2023-10-11T15:27:32.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:32.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:32.833 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:32.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.823+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:32.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:32.834 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:32.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:32.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:32.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj22[0] 2023-10-11T15:27:32.835 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:32.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:32.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:32.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:32.836 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:32.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj29[0] 2023-10-11T15:27:32.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:32.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:32.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:32.837 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:32.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:32.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:32.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:32.838 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:32.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj34[0] 2023-10-11T15:27:32.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:32.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:32.839 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:32.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:32.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:32.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:32.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:32.840 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:32.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:32.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:32.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:32.841 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.824+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:32.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:32.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:32.844 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:32.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: success 2023-10-11T15:27:32.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:32.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:32.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:32.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj55[0] 2023-10-11T15:27:32.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:32.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:32.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:32.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj57[0] 2023-10-11T15:27:32.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.847 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:32.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:32.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:32.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:32.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:32.848 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-10-11T15:27:32.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.828+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:32.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: success 2023-10-11T15:27:32.849 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:32.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: success 2023-10-11T15:27:32.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: success 2023-10-11T15:27:32.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:32.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:32.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:32.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:32.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj68[0] 2023-10-11T15:27:32.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:32.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:32.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:32.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:32.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:32.852 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:32.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:32.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:32.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj76[0] 2023-10-11T15:27:32.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:32.853 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:32.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-11T15:27:32.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:32.854 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:32.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-10-11T15:27:32.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.855 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.829+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: success 2023-10-11T15:27:32.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: success 2023-10-11T15:27:32.856 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:32.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:32.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: success 2023-10-11T15:27:32.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:32.857 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:32.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:32.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:32.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:32.858 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:32.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-10-11T15:27:32.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:32.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:32.859 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:32.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.830+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:32.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.831+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:32.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.831+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:32.860 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.832+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.832+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: success 2023-10-11T15:27:32.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: success 2023-10-11T15:27:32.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:32.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:32.861 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:32.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: success 2023-10-11T15:27:32.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:32.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:32.862 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.833+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: success 2023-10-11T15:27:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:32.863 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: success 2023-10-11T15:27:32.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.834+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj115 2023-10-11T15:27:32.864 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.835+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: backing out with sync_status=-5 2023-10-11T15:27:32.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.835+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: ERROR: full sync failed. error: -5 2023-10-11T15:27:32.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.836+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-11T15:27:32.865 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.836+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:32.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.881+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: full 2023-10-11T15:27:32.893 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.892+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:32.940 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:32.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:32.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:32.941 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:32.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-10-11T15:27:32.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:32.942 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:32.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:32.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:32.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:32.943 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:32.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj116[0] 2023-10-11T15:27:32.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:32.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:32.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-10-11T15:27:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:32.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj145[0] 2023-10-11T15:27:32.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:32.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:32.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-10-11T15:27:32.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:32.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:32.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:32.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj155[0] 2023-10-11T15:27:32.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:32.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:32.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-10-11T15:27:32.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:32.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:32.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:32.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:32.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:32.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:32.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.937+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:32.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:32.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:32.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj22[0] 2023-10-11T15:27:32.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:32.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:32.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:32.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:32.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:32.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj29[0] 2023-10-11T15:27:32.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:32.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:32.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:32.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:32.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:32.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:32.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:32.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:32.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj34[0] 2023-10-11T15:27:32.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:32.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:32.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:32.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:32.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:32.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:32.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:32.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:32.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.938+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:32.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.939+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:32.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.939+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:32.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.939+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:32.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: success 2023-10-11T15:27:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.941+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-10-11T15:27:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:32.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:32.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:32.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-10-11T15:27:32.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:32.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:32.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: success 2023-10-11T15:27:32.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: success 2023-10-11T15:27:32.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:32.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:32.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:32.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:32.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:32.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:32.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj62[0] 2023-10-11T15:27:32.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:32.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:32.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj68[0] 2023-10-11T15:27:32.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: success 2023-10-11T15:27:32.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:32.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:32.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:32.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:32.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:32.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:32.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.942+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:32.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:32.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:32.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:32.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:32.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj76[0] 2023-10-11T15:27:32.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:32.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:32.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj85[0] 2023-10-11T15:27:32.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:32.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:32.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj87[0] 2023-10-11T15:27:32.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:32.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:32.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:32.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:32.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:32.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj93[0] 2023-10-11T15:27:32.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:32.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:32.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:32.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:32.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.944+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:32.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.945+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:32.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.945+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: success 2023-10-11T15:27:32.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.946+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:32.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.946+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: success 2023-10-11T15:27:32.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.946+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: success 2023-10-11T15:27:32.974 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.946+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:32.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.946+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:32.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.947+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:32.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.947+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.947+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:32.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.947+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: success 2023-10-11T15:27:32.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.947+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:32.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.948+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: success 2023-10-11T15:27:32.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.948+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:32.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.948+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: success 2023-10-11T15:27:32.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.949+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:32.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.949+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: success 2023-10-11T15:27:32.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.949+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:32.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.950+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:32.977 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.950+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: success 2023-10-11T15:27:32.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.950+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj11 2023-10-11T15:27:32.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.951+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: backing out with sync_status=-5 2023-10-11T15:27:32.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.951+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: ERROR: full sync failed. error: -5 2023-10-11T15:27:32.978 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.952+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-11T15:27:32.979 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.952+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:32.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:32.997+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: full 2023-10-11T15:27:33.009 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.008+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:33.056 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:33.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:33.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:33.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:33.057 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:33.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj111[0] 2023-10-11T15:27:33.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:33.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:33.058 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:33.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:33.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:33.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj116[0] 2023-10-11T15:27:33.059 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:33.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:33.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj124[0] 2023-10-11T15:27:33.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:33.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:33.060 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-10-11T15:27:33.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:33.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:33.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:33.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:33.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj148[0] 2023-10-11T15:27:33.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:33.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.053+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:33.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj155[0] 2023-10-11T15:27:33.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:33.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:33.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj17[0] 2023-10-11T15:27:33.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:33.063 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:33.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:33.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:33.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:33.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:33.064 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj22[0] 2023-10-11T15:27:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-10-11T15:27:33.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:33.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:33.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:33.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:33.067 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj34[0] 2023-10-11T15:27:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:33.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:33.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:33.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:33.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:33.070 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:33.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:33.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:33.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:33.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:33.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.054+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:33.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.058+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:33.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.058+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:33.074 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:33.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: success 2023-10-11T15:27:33.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:33.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: success 2023-10-11T15:27:33.075 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:33.076 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:33.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:33.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:33.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj55[0] 2023-10-11T15:27:33.077 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:33.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:33.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj57[0] 2023-10-11T15:27:33.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:33.078 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj62[0] 2023-10-11T15:27:33.079 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj68[0] 2023-10-11T15:27:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: success 2023-10-11T15:27:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:33.080 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: success 2023-10-11T15:27:33.081 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj76[0] 2023-10-11T15:27:33.082 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-10-11T15:27:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-10-11T15:27:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 0 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: ERROR: failed to sync object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:33.083 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj87[0] 2023-10-11T15:27:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:33.084 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: success 2023-10-11T15:27:33.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj124 2023-10-11T15:27:33.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:33.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:33.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.059+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:33.085 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: success 2023-10-11T15:27:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:33.086 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj93[0] 2023-10-11T15:27:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:33.087 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.060+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.061+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: success 2023-10-11T15:27:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.061+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.061+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: success 2023-10-11T15:27:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.062+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:33.088 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.062+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: a sync operation returned error 2023-10-11T15:27:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.063+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: success 2023-10-11T15:27:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.063+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.063+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: success 2023-10-11T15:27:33.089 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.064+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: success 2023-10-11T15:27:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.064+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.064+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: success 2023-10-11T15:27:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.064+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: success 2023-10-11T15:27:33.090 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:33.091 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: success 2023-10-11T15:27:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: success 2023-10-11T15:27:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: backing out with sync_status=-5 2023-10-11T15:27:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.065+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: ERROR: full sync failed. error: -5 2023-10-11T15:27:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.067+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.092 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.067+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: waiting on bucket lease 2023-10-11T15:27:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.112+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: status after acquiring the lock is: full 2023-10-11T15:27:33.124 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.123+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bucket for full sync 2023-10-11T15:27:33.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj11 2023-10-11T15:27:33.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj11 op=0 op_state=1 2023-10-11T15:27:33.171 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj11[0] 2023-10-11T15:27:33.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj111 2023-10-11T15:27:33.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj111 op=0 op_state=1 2023-10-11T15:27:33.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj111[0] 2023-10-11T15:27:33.172 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj115 2023-10-11T15:27:33.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj115 op=0 op_state=1 2023-10-11T15:27:33.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj115[0] 2023-10-11T15:27:33.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj116 2023-10-11T15:27:33.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj116 op=0 op_state=1 2023-10-11T15:27:33.173 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj116[0] 2023-10-11T15:27:33.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj124 2023-10-11T15:27:33.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj124 op=0 op_state=1 2023-10-11T15:27:33.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj124[0] 2023-10-11T15:27:33.174 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj145 2023-10-11T15:27:33.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj145 op=0 op_state=1 2023-10-11T15:27:33.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj145[0] 2023-10-11T15:27:33.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj148 2023-10-11T15:27:33.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj148 op=0 op_state=1 2023-10-11T15:27:33.175 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj148[0] 2023-10-11T15:27:33.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj155 2023-10-11T15:27:33.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj155 op=0 op_state=1 2023-10-11T15:27:33.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj155[0] 2023-10-11T15:27:33.176 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj17 2023-10-11T15:27:33.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj17 op=0 op_state=1 2023-10-11T15:27:33.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj17[0] 2023-10-11T15:27:33.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj185 2023-10-11T15:27:33.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj185 op=0 op_state=1 2023-10-11T15:27:33.177 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj185[0] 2023-10-11T15:27:33.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj19 2023-10-11T15:27:33.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj19 op=0 op_state=1 2023-10-11T15:27:33.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj19[0] 2023-10-11T15:27:33.178 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj22 2023-10-11T15:27:33.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj22 op=0 op_state=1 2023-10-11T15:27:33.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj22[0] 2023-10-11T15:27:33.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj23 2023-10-11T15:27:33.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj23 op=0 op_state=1 2023-10-11T15:27:33.179 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj23[0] 2023-10-11T15:27:33.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj29 2023-10-11T15:27:33.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj29 op=0 op_state=1 2023-10-11T15:27:33.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.168+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj29[0] 2023-10-11T15:27:33.180 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj31 2023-10-11T15:27:33.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj31 op=0 op_state=1 2023-10-11T15:27:33.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj31[0] 2023-10-11T15:27:33.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj33 2023-10-11T15:27:33.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj33 op=0 op_state=1 2023-10-11T15:27:33.181 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj33[0] 2023-10-11T15:27:33.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj34 2023-10-11T15:27:33.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj34 op=0 op_state=1 2023-10-11T15:27:33.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj34[0] 2023-10-11T15:27:33.182 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj37 2023-10-11T15:27:33.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj37 op=0 op_state=1 2023-10-11T15:27:33.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj37[0] 2023-10-11T15:27:33.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj38 2023-10-11T15:27:33.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj38 op=0 op_state=1 2023-10-11T15:27:33.183 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj38[0] 2023-10-11T15:27:33.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj44 2023-10-11T15:27:33.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj44 op=0 op_state=1 2023-10-11T15:27:33.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj44[0] 2023-10-11T15:27:33.184 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj46 2023-10-11T15:27:33.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj46 op=0 op_state=1 2023-10-11T15:27:33.185 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.169+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj46[0] 2023-10-11T15:27:33.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj19]: success 2023-10-11T15:27:33.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj11]: success 2023-10-11T15:27:33.189 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj155]: success 2023-10-11T15:27:33.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj51 2023-10-11T15:27:33.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj51 op=0 op_state=1 2023-10-11T15:27:33.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj51[0] 2023-10-11T15:27:33.190 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj55 2023-10-11T15:27:33.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj55 op=0 op_state=1 2023-10-11T15:27:33.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj55[0] 2023-10-11T15:27:33.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj57 2023-10-11T15:27:33.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj57 op=0 op_state=1 2023-10-11T15:27:33.191 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.172+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj57[0] 2023-10-11T15:27:33.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.173+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj115]: success 2023-10-11T15:27:33.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.173+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj58 2023-10-11T15:27:33.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.173+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj58 op=0 op_state=1 2023-10-11T15:27:33.192 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.173+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj58[0] 2023-10-11T15:27:33.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj148]: success 2023-10-11T15:27:33.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj111]: success 2023-10-11T15:27:33.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj145]: success 2023-10-11T15:27:33.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj38]: success 2023-10-11T15:27:33.193 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj116]: success 2023-10-11T15:27:33.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj31]: success 2023-10-11T15:27:33.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj185]: success 2023-10-11T15:27:33.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj62 2023-10-11T15:27:33.194 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj62 op=0 op_state=1 2023-10-11T15:27:33.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj62[0] 2023-10-11T15:27:33.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj68 2023-10-11T15:27:33.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj68 op=0 op_state=1 2023-10-11T15:27:33.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj68[0] 2023-10-11T15:27:33.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj69 2023-10-11T15:27:33.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj69 op=0 op_state=1 2023-10-11T15:27:33.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj69[0] 2023-10-11T15:27:33.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj74 2023-10-11T15:27:33.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj74 op=0 op_state=1 2023-10-11T15:27:33.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj74[0] 2023-10-11T15:27:33.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj76 2023-10-11T15:27:33.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj76 op=0 op_state=1 2023-10-11T15:27:33.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj76[0] 2023-10-11T15:27:33.197 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj85 2023-10-11T15:27:33.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj85 op=0 op_state=1 2023-10-11T15:27:33.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj85[0] 2023-10-11T15:27:33.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj87 2023-10-11T15:27:33.198 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj87 op=0 op_state=1 2023-10-11T15:27:33.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj87[0] 2023-10-11T15:27:33.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj124]: success 2023-10-11T15:27:33.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj155 2023-10-11T15:27:33.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj46]: success 2023-10-11T15:27:33.199 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj17]: success 2023-10-11T15:27:33.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.174+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj29]: success 2023-10-11T15:27:33.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj92 2023-10-11T15:27:33.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj92 op=0 op_state=1 2023-10-11T15:27:33.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj92[0] 2023-10-11T15:27:33.200 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj34]: success 2023-10-11T15:27:33.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj93 2023-10-11T15:27:33.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj93 op=0 op_state=1 2023-10-11T15:27:33.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj93[0] 2023-10-11T15:27:33.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: [full sync] syncing object: dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0/obj95 2023-10-11T15:27:33.201 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b=bucket sync single entry (source_zone=daa316e9-869c-4074-a756-a8adfb4a3573) b= log_entry=obj95 op=0 op_state=1 2023-10-11T15:27:33.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: bucket sync: sync obj: daa316e9-869c-4074-a756-a8adfb4a3573/:dghqvd-59[daa316e9-869c-4074-a756-a8adfb4a3573.5751.24])/obj95[0] 2023-10-11T15:27:33.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj37]: success 2023-10-11T15:27:33.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.175+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj23]: success 2023-10-11T15:27:33.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.176+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj57]: success 2023-10-11T15:27:33.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.176+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj55]: success 2023-10-11T15:27:33.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj22]: success 2023-10-11T15:27:33.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj44]: success 2023-10-11T15:27:33.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj33]: success 2023-10-11T15:27:33.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj46 2023-10-11T15:27:33.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj76]: success 2023-10-11T15:27:33.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj69]: success 2023-10-11T15:27:33.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj68]: success 2023-10-11T15:27:33.204 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj51]: success 2023-10-11T15:27:33.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.179+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj58]: success 2023-10-11T15:27:33.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj93]: success 2023-10-11T15:27:33.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj62]: success 2023-10-11T15:27:33.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj87]: success 2023-10-11T15:27:33.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj85]: success 2023-10-11T15:27:33.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj92]: success 2023-10-11T15:27:33.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj74]: success 2023-10-11T15:27:33.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj93 2023-10-11T15:27:33.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:entry[obj95]: success 2023-10-11T15:27:33.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.180+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: updating marker oid=bucket.full-sync-status.daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24 marker=obj95 2023-10-11T15:27:33.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.181+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: set bucket state=incremental 2023-10-11T15:27:33.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.182+0000 7f259f35d780 5 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:full_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: bucket status objv={r=_qhHxhMLG_pnPtAQ4cBm0hbX:151,w=:0} 2023-10-11T15:27:33.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.183+0000 7f259f35d780 10 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-10-11T15:27:33.207 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.185+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.185+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.185+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.208 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: sync status for source bucket shard: 2 2023-10-11T15:27:33.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:0]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:1]: sync status for source bucket shard: 2 2023-10-11T15:27:33.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:2]: sync status for source bucket shard: 2 2023-10-11T15:27:33.209 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:1]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:1]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:2]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:2]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:4]: sync status for source bucket shard: 2 2023-10-11T15:27:33.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:4]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:4]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.210 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:3]: sync status for source bucket shard: 2 2023-10-11T15:27:33.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.186+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:3]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:3]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.211 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:8]: sync status for source bucket shard: 2 2023-10-11T15:27:33.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.215 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.187+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:8]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:8]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:11]: sync status for source bucket shard: 2 2023-10-11T15:27:33.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:11]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:11]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.216 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:7]: sync status for source bucket shard: 2 2023-10-11T15:27:33.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:6]: sync status for source bucket shard: 2 2023-10-11T15:27:33.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:7]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:7]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.188+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:6]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:6]: listing bilog for incremental sync; position=00000000002.100.6 2023-10-11T15:27:33.217 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.218 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:12]: sync status for source bucket shard: 2 2023-10-11T15:27:33.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:12]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:12]: listing bilog for incremental sync; position=00000000002.101.6 2023-10-11T15:27:33.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.191+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.222 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:13]: sync status for source bucket shard: 2 2023-10-11T15:27:33.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:14]: sync status for source bucket shard: 2 2023-10-11T15:27:33.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:15]: sync status for source bucket shard: 2 2023-10-11T15:27:33.223 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:17]: sync status for source bucket shard: 2 2023-10-11T15:27:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:19]: sync status for source bucket shard: 2 2023-10-11T15:27:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:13]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:13]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:14]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:14]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:15]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:15]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.224 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:17]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:17]: listing bilog for incremental sync; position=00000000002.85.6 2023-10-11T15:27:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.192+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:19]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:19]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:21]: sync status for source bucket shard: 2 2023-10-11T15:27:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:21]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:21]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.225 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:22]: sync status for source bucket shard: 2 2023-10-11T15:27:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:25]: sync status for source bucket shard: 2 2023-10-11T15:27:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:20]: sync status for source bucket shard: 2 2023-10-11T15:27:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:5]: sync status for source bucket shard: 2 2023-10-11T15:27:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:9]: sync status for source bucket shard: 2 2023-10-11T15:27:33.226 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:10]: sync status for source bucket shard: 2 2023-10-11T15:27:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:18]: sync status for source bucket shard: 2 2023-10-11T15:27:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:22]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:22]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:25]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:25]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:20]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:20]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:5]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:5]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:9]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:9]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:10]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:10]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.193+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:18]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:18]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-10-11T15:27:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:33]: sync status for source bucket shard: 2 2023-10-11T15:27:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:27]: sync status for source bucket shard: 2 2023-10-11T15:27:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:32]: sync status for source bucket shard: 2 2023-10-11T15:27:33.229 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:33]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:33]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:27]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:27]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:32]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:32]: listing bilog for incremental sync; position=00000000001.56.6 2023-10-11T15:27:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:16]: sync status for source bucket shard: 2 2023-10-11T15:27:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:16]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:16]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.230 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:24]: sync status for source bucket shard: 2 2023-10-11T15:27:33.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.194+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:24]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:24]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:26]: sync status for source bucket shard: 2 2023-10-11T15:27:33.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:30]: sync status for source bucket shard: 2 2023-10-11T15:27:33.231 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:26]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:26]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:30]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:30]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:29]: sync status for source bucket shard: 2 2023-10-11T15:27:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:31]: sync status for source bucket shard: 2 2023-10-11T15:27:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:23]: sync status for source bucket shard: 2 2023-10-11T15:27:33.232 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:28]: sync status for source bucket shard: 2 2023-10-11T15:27:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:35]: sync status for source bucket shard: 2 2023-10-11T15:27:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:34]: sync status for source bucket shard: 2 2023-10-11T15:27:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:29]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:29]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.233 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:31]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:31]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:23]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:23]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:28]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:28]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:35]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:35]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.195+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:34]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:34]: listing bilog for incremental sync; position= 2023-10-11T15:27:33.234 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.196+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:36]: sync status for source bucket shard: 2 2023-10-11T15:27:33.235 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:33.196+0000 7f259f35d780 20 RGW-SYNC:bucket[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24<-dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:36]:inc_sync[dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24:36]: listing bilog for incremental sync; position=00000000001.100.6 2023-10-11T15:27:33.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:33.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:33.261 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:33.353 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:33.354 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:33.387 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:27:33.388 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.388 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj11", 2023-10-11T15:27:33.388 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.388 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.389 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.389 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:27:33.389 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.389 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.390 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.390 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.390 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.390 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.390 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.917502Z", 2023-10-11T15:27:33.391 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.391 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.391 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.391 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.392 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.392 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.392 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.392 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.393 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14476178746186506756", 2023-10-11T15:27:33.393 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.393 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.393 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.393 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.394 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.394 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj111", 2023-10-11T15:27:33.394 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.394 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.394 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.395 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.395 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.395 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.395 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.396 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.396 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.396 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.282651Z", 2023-10-11T15:27:33.396 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.397 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.397 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.397 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.397 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.397 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.398 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.398 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.398 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.4888030454839795823", 2023-10-11T15:27:33.398 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.399 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.399 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.399 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.399 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.400 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj115", 2023-10-11T15:27:33.400 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.400 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.400 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.400 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.401 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.401 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.401 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.401 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.401 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.402 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.402 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.481238Z", 2023-10-11T15:27:33.402 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.402 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.402 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.403 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.403 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.403 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.403 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.403 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.404 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.16944103309488854928", 2023-10-11T15:27:33.404 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.404 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.405 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.406 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.406 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj116", 2023-10-11T15:27:33.407 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.407 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.407 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.407 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.408 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.408 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.408 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.408 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.409 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.409 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.474371Z", 2023-10-11T15:27:33.409 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.409 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.410 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.410 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.410 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.410 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.411 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.411 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.412 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.412 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.217471136694822099", 2023-10-11T15:27:33.412 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.413 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.413 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.413 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.413 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.414 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj124", 2023-10-11T15:27:33.414 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.414 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.414 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.414 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:27:33.415 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.415 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.415 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.415 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.416 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.416 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.416 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.864790Z", 2023-10-11T15:27:33.416 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.416 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.417 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.417 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.417 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.417 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.417 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.418 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.418 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1861272259701203166", 2023-10-11T15:27:33.418 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.418 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.419 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.419 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.419 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.419 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj145", 2023-10-11T15:27:33.420 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.420 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.420 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.420 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.421 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.421 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.421 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.421 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.422 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.422 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.493223Z", 2023-10-11T15:27:33.423 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.424 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.424 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.424 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.424 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.425 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.425 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.425 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.426 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5955913347240015412", 2023-10-11T15:27:33.426 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.426 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.426 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.427 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.427 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj148", 2023-10-11T15:27:33.427 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.427 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.428 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.428 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.428 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.429 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.429 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.429 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.429 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.430 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.302431Z", 2023-10-11T15:27:33.430 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.430 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.430 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.431 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.431 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.431 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.431 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.432 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.432 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10215546196655287645", 2023-10-11T15:27:33.433 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.433 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.433 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.434 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.434 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj155", 2023-10-11T15:27:33.434 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.434 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.434 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.435 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.435 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.435 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.436 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.436 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.436 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.437 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.437 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.923366Z", 2023-10-11T15:27:33.437 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.437 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.438 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.438 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.438 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.438 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.439 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.439 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.439 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5362776548195693794", 2023-10-11T15:27:33.439 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.440 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.440 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.440 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.440 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.441 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj17", 2023-10-11T15:27:33.441 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.441 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.441 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.441 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:27:33.442 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.442 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.442 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.443 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.443 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.443 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.443 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.599178Z", 2023-10-11T15:27:33.444 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.444 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.444 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.445 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.445 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.445 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.446 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.446 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.446 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.447 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.9787987605353649419", 2023-10-11T15:27:33.447 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.447 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.448 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.448 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.448 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.448 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj185", 2023-10-11T15:27:33.449 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.449 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.449 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.449 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:27:33.450 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.450 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.450 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.450 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.451 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.451 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.451 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.362612Z", 2023-10-11T15:27:33.451 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.452 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.452 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.452 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.452 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.452 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.453 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.453 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.453 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.453 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.2974707128548095935", 2023-10-11T15:27:33.454 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.454 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.454 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.455 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.455 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj19", 2023-10-11T15:27:33.456 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.456 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.457 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.457 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.457 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.458 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.458 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.458 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.458 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.458 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.459 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.348518Z", 2023-10-11T15:27:33.459 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.460 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.460 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.460 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.460 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.460 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.461 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.461 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.461 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.7837303318171134014", 2023-10-11T15:27:33.462 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.462 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.462 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.462 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.463 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.463 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj22", 2023-10-11T15:27:33.463 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.463 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.464 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.464 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.464 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.464 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.465 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.465 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.465 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.465 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.938656Z", 2023-10-11T15:27:33.465 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.466 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.466 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.466 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.466 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.467 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.467 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.468 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.468 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.468 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.6435643312402545131", 2023-10-11T15:27:33.468 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.468 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.469 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.469 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.469 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.469 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj23", 2023-10-11T15:27:33.469 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.470 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.470 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.470 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14 2023-10-11T15:27:33.470 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.471 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.471 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.471 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.471 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.471 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.472 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.324750Z", 2023-10-11T15:27:33.472 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.472 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.472 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.472 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.473 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.473 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.473 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.473 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.473 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.474 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.16822430881655473582", 2023-10-11T15:27:33.474 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.474 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.474 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.475 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.475 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.475 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj29", 2023-10-11T15:27:33.475 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.476 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.476 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.476 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:27:33.476 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.476 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.477 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.477 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.477 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.477 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.478 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.330113Z", 2023-10-11T15:27:33.478 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.478 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.478 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.478 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.479 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.479 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.479 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.479 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.479 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.480 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.5145141707541339505", 2023-10-11T15:27:33.480 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.480 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.480 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.480 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.481 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj31", 2023-10-11T15:27:33.481 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.481 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.481 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.482 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.482 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.482 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.482 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.483 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.483 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.483 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.805358Z", 2023-10-11T15:27:33.483 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.484 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.484 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.484 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.484 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.484 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.13689719212522759424", 2023-10-11T15:27:33.485 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.486 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.486 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.486 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.486 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj33", 2023-10-11T15:27:33.487 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.487 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.487 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.487 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.488 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.488 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.488 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.488 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.488 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.489 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.489 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.957164Z", 2023-10-11T15:27:33.489 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.489 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.489 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.490 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.490 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.490 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.490 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.490 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.491 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.491 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17826396092411312258", 2023-10-11T15:27:33.491 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.491 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.491 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.492 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj34", 2023-10-11T15:27:33.492 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.493 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.493 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.493 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.493 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.494 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.494 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.494 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.494 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.495 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.660795Z", 2023-10-11T15:27:33.495 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.495 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.495 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.495 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.496 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.496 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.496 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.496 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.497 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15795877277578395340", 2023-10-11T15:27:33.497 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.497 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.497 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.498 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.498 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj37", 2023-10-11T15:27:33.498 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.498 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.499 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.499 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 21 2023-10-11T15:27:33.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.499 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.499 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.500 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.500 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.500 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.500 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.203398Z", 2023-10-11T15:27:33.500 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.501 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.501 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.501 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.501 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.502 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.502 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.502 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.502 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1297250363407728559", 2023-10-11T15:27:33.503 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.503 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.503 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.503 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.504 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj38", 2023-10-11T15:27:33.504 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.504 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.504 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.504 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 20 2023-10-11T15:27:33.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.505 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.505 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.505 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.505 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.506 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.506 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.367798Z", 2023-10-11T15:27:33.506 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.506 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.506 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.507 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.507 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.507 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.507 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.508 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.508 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.508 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.17764269106167249878", 2023-10-11T15:27:33.508 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.508 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.509 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.509 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj44", 2023-10-11T15:27:33.509 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.510 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.510 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.510 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 17 2023-10-11T15:27:33.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.510 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.511 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.511 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.511 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.511 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.511 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.013924Z", 2023-10-11T15:27:33.512 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.512 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.512 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.512 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.512 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.513 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.513 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.513 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.513 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1424272109642346828", 2023-10-11T15:27:33.514 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.514 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.514 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.514 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.514 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.515 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj46", 2023-10-11T15:27:33.515 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.515 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.515 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.516 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 12 2023-10-11T15:27:33.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.516 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.516 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.516 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.517 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.517 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.517 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.344708Z", 2023-10-11T15:27:33.517 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.517 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.518 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.518 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.518 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.518 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.518 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.519 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.519 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.2187052628994711795", 2023-10-11T15:27:33.519 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.519 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.520 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.520 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.520 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj51", 2023-10-11T15:27:33.520 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.521 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.521 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 2 2023-10-11T15:27:33.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.522 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.522 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.522 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.522 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.522 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.523 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.682129Z", 2023-10-11T15:27:33.523 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.523 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.523 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.523 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.524 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.524 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.524 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.524 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.525 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8367720246898811753", 2023-10-11T15:27:33.525 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.525 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.525 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.526 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.526 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj55", 2023-10-11T15:27:33.526 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.526 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.527 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.527 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.527 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.527 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.528 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.528 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.528 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.032675Z", 2023-10-11T15:27:33.528 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.528 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.529 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.529 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.529 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.529 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.530 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.530 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.530 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11474120272324657781", 2023-10-11T15:27:33.530 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.531 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.531 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.531 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.531 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj57", 2023-10-11T15:27:33.532 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.532 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.532 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.532 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.533 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.533 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.533 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.533 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.533 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.534 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.384920Z", 2023-10-11T15:27:33.534 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.534 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.534 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.535 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.535 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.535 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.535 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.535 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.536 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.2875465366570165858", 2023-10-11T15:27:33.536 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.536 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.536 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.537 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.537 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.537 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj58", 2023-10-11T15:27:33.537 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.537 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.538 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.538 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 1 2023-10-11T15:27:33.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.538 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.538 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.539 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.539 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.539 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.539 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.390163Z", 2023-10-11T15:27:33.539 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.540 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.540 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.540 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.540 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.540 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.541 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.541 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.541 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.541 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1044493043319884999", 2023-10-11T15:27:33.542 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.542 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.542 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.542 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.542 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.543 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj62", 2023-10-11T15:27:33.543 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.543 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.543 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.544 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:27:33.544 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.544 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.544 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.544 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.545 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.545 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.545 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.825386Z", 2023-10-11T15:27:33.545 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.546 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.546 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.546 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.546 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.546 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.547 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.547 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.547 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.547 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11689926542113187170", 2023-10-11T15:27:33.547 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.548 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.548 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.549 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj68", 2023-10-11T15:27:33.549 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.549 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.549 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.549 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 19 2023-10-11T15:27:33.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.550 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.550 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.550 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.551 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.551 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.551 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.746100Z", 2023-10-11T15:27:33.551 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.551 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.552 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.552 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.552 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.552 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.552 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.553 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.553 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.3057446466243931406", 2023-10-11T15:27:33.553 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.553 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.554 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.554 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.554 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.554 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj69", 2023-10-11T15:27:33.555 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.555 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.555 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.555 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 18 2023-10-11T15:27:33.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.556 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.557 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.557 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.558 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.558 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.558 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.406312Z", 2023-10-11T15:27:33.558 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.559 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.559 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.559 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.559 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.559 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.560 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.560 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.560 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.560 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.10229208689028133877", 2023-10-11T15:27:33.560 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.561 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.561 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.561 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.561 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.561 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj74", 2023-10-11T15:27:33.562 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.562 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.562 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.562 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:27:33.563 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.563 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.563 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.563 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.564 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.564 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.564 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.223624Z", 2023-10-11T15:27:33.564 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.564 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.565 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.565 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.565 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.565 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.565 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.566 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.566 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.566 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.14363068269514729310", 2023-10-11T15:27:33.566 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.567 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.567 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.567 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.567 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.567 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj76", 2023-10-11T15:27:33.568 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.568 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.568 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.568 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.568 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.569 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.569 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.569 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.569 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.569 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.570 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.446066Z", 2023-10-11T15:27:33.570 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.570 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.570 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.571 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.571 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.571 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.571 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.571 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.572 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.572 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.1586399829621972610", 2023-10-11T15:27:33.572 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.572 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.572 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.573 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.573 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.573 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj85", 2023-10-11T15:27:33.573 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.574 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.575 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.576 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.577 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.765362Z", 2023-10-11T15:27:33.577 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.577 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.577 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.578 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.578 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.578 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.578 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.579 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.579 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.579 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.8029048904835253902", 2023-10-11T15:27:33.579 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.579 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.580 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.580 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.580 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.580 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj87", 2023-10-11T15:27:33.580 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.581 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.581 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.581 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.581 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.581 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.582 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.582 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.582 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.582 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.410694Z", 2023-10-11T15:27:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.583 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.584 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.584 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.584 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.584 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.585 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.7218876183685432424", 2023-10-11T15:27:33.585 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.585 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.585 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.585 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.586 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.586 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj92", 2023-10-11T15:27:33.586 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.586 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.586 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.587 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:27:33.587 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.587 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.587 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.588 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.588 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.588 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.588 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.424897Z", 2023-10-11T15:27:33.588 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.589 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.589 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.589 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.589 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.589 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.590 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.590 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.590 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.590 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.13587890633776507430", 2023-10-11T15:27:33.590 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.591 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.591 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.591 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.591 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.591 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj93", 2023-10-11T15:27:33.592 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.592 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.592 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.592 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15 2023-10-11T15:27:33.593 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.593 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.593 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.593 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.594 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.594 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.594 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.844840Z", 2023-10-11T15:27:33.594 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.594 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.595 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.595 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.595 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.595 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.595 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.596 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.596 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.596 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11886398017059485398", 2023-10-11T15:27:33.596 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.597 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.597 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.597 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.597 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.598 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj95", 2023-10-11T15:27:33.598 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.598 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.598 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 9, 2023-10-11T15:27:33.599 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.599 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.599 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.599 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.599 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.600 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.600 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.600 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.464225Z", 2023-10-11T15:27:33.600 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.601 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.601 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.601 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.601 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.601 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.602 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.602 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.602 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.602 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11475978418566738203", 2023-10-11T15:27:33.602 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.603 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.603 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.603 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:33.603 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:27:33.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:33.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:33.606 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:33.660 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:33.661 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:33.692 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:27:33.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.692 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj11", 2023-10-11T15:27:33.692 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.693 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.693 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.693 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:27:33.693 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.694 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.694 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.694 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.694 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.694 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.695 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.917502Z", 2023-10-11T15:27:33.695 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.695 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.695 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.695 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.696 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.696 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.696 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.696 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.697 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.697 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_SXsBU-yWLokR_jDu3M8h5-3Ee5cP5ca", 2023-10-11T15:27:33.697 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.697 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.697 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.698 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.698 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.698 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj111", 2023-10-11T15:27:33.698 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.699 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.699 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.699 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:33.699 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.699 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.700 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.700 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.700 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.700 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.700 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.282651Z", 2023-10-11T15:27:33.701 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.701 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.701 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.701 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.701 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.702 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.702 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.702 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.702 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.703 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_q2kz9uPjOA-BQVgZIcjtIhR3JZZDuUg", 2023-10-11T15:27:33.703 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.703 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.703 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.703 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.704 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.704 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj115", 2023-10-11T15:27:33.704 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.704 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.704 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.705 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.705 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.705 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.705 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.706 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.706 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.706 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.706 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.481238Z", 2023-10-11T15:27:33.706 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.707 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.707 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.707 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.707 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.707 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.708 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.708 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.708 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.708 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_nHRgaQWI_tDiRLt1OyP0pGWwlTaJZjy", 2023-10-11T15:27:33.708 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.709 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.709 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.709 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.709 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.710 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj116", 2023-10-11T15:27:33.710 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.710 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.710 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.710 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.711 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.711 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.711 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.711 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.711 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.712 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.712 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.474371Z", 2023-10-11T15:27:33.712 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.712 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.713 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.713 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.713 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.713 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.713 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.714 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.714 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.714 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_it5nbYIE-icVtXNUoMD9th-n5HTOgYL", 2023-10-11T15:27:33.714 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.714 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.715 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.715 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.715 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.715 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj124", 2023-10-11T15:27:33.716 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.716 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.716 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.716 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:33.716 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.717 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.717 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.717 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.717 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.717 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.718 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.864790Z", 2023-10-11T15:27:33.718 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.718 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.718 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.719 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.719 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.719 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.719 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.719 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.720 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.720 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_iCGmbW0Z7joOButPbzJySUxnsCvsEg3", 2023-10-11T15:27:33.720 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.720 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.720 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.721 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.721 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.721 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj145", 2023-10-11T15:27:33.721 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.721 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.722 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.722 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.722 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.722 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.722 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.723 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.723 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.723 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.723 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.493223Z", 2023-10-11T15:27:33.724 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.724 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.724 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.724 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.725 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.725 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.725 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.725 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.726 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.726 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_KLaMaM-OcnT7tEX_wBmtZtdRhDiVG8F", 2023-10-11T15:27:33.726 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.726 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.727 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.727 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.727 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj148", 2023-10-11T15:27:33.728 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.728 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.728 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.728 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:33.729 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.729 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.729 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.729 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.730 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.730 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.730 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.302431Z", 2023-10-11T15:27:33.730 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.730 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.731 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.731 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.731 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.731 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.732 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.732 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.732 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.732 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_9xUDDcgmzKvvPx0M-Na4wzed7wtm4kh", 2023-10-11T15:27:33.732 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.733 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.733 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.733 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.733 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj155", 2023-10-11T15:27:33.734 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.734 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.734 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.734 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.735 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.735 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.735 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.735 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.735 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.736 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.736 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.923366Z", 2023-10-11T15:27:33.736 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.736 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.736 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.737 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.737 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.737 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.737 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.737 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.738 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_ZJ-qoZlFAd38py_VzbbaMCncf6qT1Sb", 2023-10-11T15:27:33.738 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.738 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.739 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.739 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.739 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.739 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj17", 2023-10-11T15:27:33.739 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.740 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.740 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.740 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:33.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.741 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.741 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.741 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.741 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.741 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.742 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.599178Z", 2023-10-11T15:27:33.742 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.742 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.742 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.743 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.743 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.743 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.743 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.744 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.744 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_sdhAU6RnOAK0X9JklyOziICZ1wMhLfx", 2023-10-11T15:27:33.745 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.745 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.745 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.745 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.746 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.746 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj185", 2023-10-11T15:27:33.746 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.746 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.747 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.747 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:27:33.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.747 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.748 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.748 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.748 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.748 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.749 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.362612Z", 2023-10-11T15:27:33.749 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.749 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.749 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.749 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.750 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.750 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.750 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.750 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.750 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.751 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_I9NJsEigu08x-Bj-FgG2iVVX6XJwH8z", 2023-10-11T15:27:33.751 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.751 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.751 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.751 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.752 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.752 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj19", 2023-10-11T15:27:33.752 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.752 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.753 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.753 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.753 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.753 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.753 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.754 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.754 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.754 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.754 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.348518Z", 2023-10-11T15:27:33.755 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.755 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.755 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.755 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.755 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.756 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.756 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.756 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.756 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.756 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_I8s7Y--NI7jEy8pIZur3ZwMFrcOpFSP", 2023-10-11T15:27:33.757 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.757 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.757 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.757 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.757 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.758 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj22", 2023-10-11T15:27:33.758 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.758 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.758 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.759 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.759 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.759 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.759 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.759 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.761 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.761 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.761 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.938656Z", 2023-10-11T15:27:33.762 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.762 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.762 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.762 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.763 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.763 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.763 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.763 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.764 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_kAWOkPfw-woP1oo6xS8EeoOR0zgQUqu", 2023-10-11T15:27:33.764 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.764 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.764 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.765 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.765 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj23", 2023-10-11T15:27:33.765 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.765 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.766 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:33.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.766 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.767 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.767 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.767 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.767 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.767 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.324750Z", 2023-10-11T15:27:33.768 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.768 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.768 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.768 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.768 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.769 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.769 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.769 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.770 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_eph0SpRVZmLCaaejQYL1CJYZ4LGid6T", 2023-10-11T15:27:33.770 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.770 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.771 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.771 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj29", 2023-10-11T15:27:33.771 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.771 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.772 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.772 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.772 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.773 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.773 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.773 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.773 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.330113Z", 2023-10-11T15:27:33.773 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.774 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.774 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.774 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.774 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.775 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.775 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.775 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.775 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_sxxFUwAAYtdu-CYbWjkf_rhjUKVoHPf", 2023-10-11T15:27:33.776 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.776 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.776 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj31", 2023-10-11T15:27:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.777 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.778 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.778 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.778 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.778 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.779 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.779 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.779 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.805358Z", 2023-10-11T15:27:33.779 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.780 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.780 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.780 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.780 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.780 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.781 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.781 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.781 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.781 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_cWFodU3yhX2gFVnKTSbqAJK0qc86EpV", 2023-10-11T15:27:33.781 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.782 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.782 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.782 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.782 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.783 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj33", 2023-10-11T15:27:33.783 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.783 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.783 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.783 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.784 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.784 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.784 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.785 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.785 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.785 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:05.957164Z", 2023-10-11T15:27:33.785 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.785 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.786 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.786 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.786 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.786 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.786 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.787 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.787 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_OojJerfyiizeOlTQdwnRrM0VZPu6DUk", 2023-10-11T15:27:33.787 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.788 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.788 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.788 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.788 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.788 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj34", 2023-10-11T15:27:33.789 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.789 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.789 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.790 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.790 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.790 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.790 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.791 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.791 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.791 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.791 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.660795Z", 2023-10-11T15:27:33.791 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.792 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.792 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.792 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.792 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.793 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.793 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.793 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.793 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.793 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_s_f9SPnSfOJ6SREuRGvIalcH4NWqgj6", 2023-10-11T15:27:33.794 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.794 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.794 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.794 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.795 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.795 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj37", 2023-10-11T15:27:33.795 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.796 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.796 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.796 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 12 2023-10-11T15:27:33.796 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.796 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.797 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.797 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.797 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.797 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.798 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.203398Z", 2023-10-11T15:27:33.798 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.798 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.799 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.799 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.799 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.799 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.800 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.800 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.800 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_MV-lt70pXtP-Lwn98s9R5iZAgmM7zL6", 2023-10-11T15:27:33.801 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.801 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.801 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.801 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.801 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.802 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj38", 2023-10-11T15:27:33.802 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.802 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.802 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.802 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:27:33.803 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.803 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.803 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.803 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.367798Z", 2023-10-11T15:27:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.804 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.805 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_4v670yGPZGouT3JIBNwJXy32WrQNjfV", 2023-10-11T15:27:33.806 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.807 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj44", 2023-10-11T15:27:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 12 2023-10-11T15:27:33.808 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.809 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.013924Z", 2023-10-11T15:27:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.810 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.811 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_dnmnV0JVWtsqKs7mfISPQszAeoKk2kq", 2023-10-11T15:27:33.812 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj46", 2023-10-11T15:27:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.814 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 10 2023-10-11T15:27:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.815 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.344708Z", 2023-10-11T15:27:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.816 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.817 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.817 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.817 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.817 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.818 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.818 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_w5yNrqy2oEIoOuFteKiPKfOdVrSD9em", 2023-10-11T15:27:33.818 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.818 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.819 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.819 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.819 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj51", 2023-10-11T15:27:33.819 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.820 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.820 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.820 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 4 2023-10-11T15:27:33.820 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.820 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.821 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.821 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.821 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.821 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.821 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.682129Z", 2023-10-11T15:27:33.822 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.822 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.822 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.822 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.823 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.823 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.823 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.823 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.824 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_5xFs2fYMoyHQWrbcxfQY1y7_kVSubiw", 2023-10-11T15:27:33.824 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.824 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.824 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.825 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.825 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj55", 2023-10-11T15:27:33.825 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.825 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.826 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.826 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 6 2023-10-11T15:27:33.826 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.826 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.826 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.827 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.827 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.827 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.827 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:06.032675Z", 2023-10-11T15:27:33.827 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.828 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.828 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.828 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.828 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.828 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.829 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.829 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.829 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.829 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_K2zmf4430J4ym1NaN8uC6B-3JHLAvJR", 2023-10-11T15:27:33.830 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.830 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.830 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.830 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.830 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.831 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj57", 2023-10-11T15:27:33.831 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.831 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.831 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.831 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.832 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.832 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.832 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.832 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.832 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.833 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.833 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.384920Z", 2023-10-11T15:27:33.833 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.833 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.833 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.834 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.834 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.834 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.834 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.835 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.835 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.835 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_1B2AQ7ZOGu5lui0cKTlT0c1LiteWVwn", 2023-10-11T15:27:33.835 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.835 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.836 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.836 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.836 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.836 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj58", 2023-10-11T15:27:33.836 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.837 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.837 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.837 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 3 2023-10-11T15:27:33.837 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.837 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.838 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.838 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.838 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.838 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.839 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.390163Z", 2023-10-11T15:27:33.839 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.839 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.839 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.839 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.840 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.840 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.840 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.840 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.840 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.841 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_OU5diIe0m-zigfxJ1ZJ2U0xh5U6GKS3", 2023-10-11T15:27:33.841 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.841 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.841 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.842 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.842 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj62", 2023-10-11T15:27:33.842 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.842 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.842 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.843 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:33.843 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.843 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.843 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.843 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.844 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.844 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.844 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.825386Z", 2023-10-11T15:27:33.844 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.845 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.845 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.845 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.845 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.845 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.846 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.846 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.846 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.846 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_BWS44DyzGnj7Ql1IOnAfifqw4mNx6hr", 2023-10-11T15:27:33.846 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.847 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.847 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.847 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.847 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj68", 2023-10-11T15:27:33.848 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.848 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.848 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.848 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:27:33.848 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.849 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.850 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.746100Z", 2023-10-11T15:27:33.850 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.850 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.850 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.851 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.851 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.851 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.851 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.851 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.852 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.852 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_z_7sfd0RE44WiqAg4fVdqBD_3ptd2rL", 2023-10-11T15:27:33.852 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.852 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.852 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.853 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.853 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.853 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj69", 2023-10-11T15:27:33.853 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.854 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.854 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.854 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13 2023-10-11T15:27:33.855 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.855 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.855 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.856 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.856 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.856 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.857 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.406312Z", 2023-10-11T15:27:33.857 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.857 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.858 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.858 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.858 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.858 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.859 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.859 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.859 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.859 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_QQB2CIiHYZkNo9bW5TE2QZfFbRipc4v", 2023-10-11T15:27:33.859 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.860 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.860 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.860 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.860 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.860 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj74", 2023-10-11T15:27:33.861 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.861 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.861 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.861 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.862 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.862 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.862 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.862 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.862 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 6, 2023-10-11T15:27:33.863 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:25:38.223624Z", 2023-10-11T15:27:33.863 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.863 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.863 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.863 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.864 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.864 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.864 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.864 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.864 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.865 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_T1YRI-tPXFLVpL1CBmVyyTKeQrKG9p9", 2023-10-11T15:27:33.865 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.865 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.865 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.865 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.866 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj76", 2023-10-11T15:27:33.866 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.866 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.867 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.867 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.867 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.867 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.867 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.868 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.868 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.868 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.868 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.446066Z", 2023-10-11T15:27:33.868 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.869 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.869 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.869 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.869 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.869 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.870 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.870 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.870 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.870 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_egSIGtZWgBIqVpV6uDCLm1lusyNpTJj", 2023-10-11T15:27:33.870 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.871 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.871 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.871 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.871 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj85", 2023-10-11T15:27:33.872 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.872 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.872 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.872 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 5 2023-10-11T15:27:33.872 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.873 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.873 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.873 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.873 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.873 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.874 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:16:32.765362Z", 2023-10-11T15:27:33.874 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.874 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.874 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.874 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.875 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.875 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.875 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.875 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.876 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.876 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_k5bvHKIywKVxDCz7QycHNgpd7Z7UZmW", 2023-10-11T15:27:33.876 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.876 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.876 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.877 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.877 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.877 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj87", 2023-10-11T15:27:33.877 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.877 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.878 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.878 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 7 2023-10-11T15:27:33.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.878 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.878 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.880 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:59.410694Z", 2023-10-11T15:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.881 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.882 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.882 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.882 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.882 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.882 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_Idpidc3gKBJYTevg0R2hNDCKaXk_dzd", 2023-10-11T15:27:33.883 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.883 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.883 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.883 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.884 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.884 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj92", 2023-10-11T15:27:33.884 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.884 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.884 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.885 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 11 2023-10-11T15:27:33.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.885 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.885 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.885 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.886 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.886 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 5, 2023-10-11T15:27:33.886 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:21:14.424897Z", 2023-10-11T15:27:33.886 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.886 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.887 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.887 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.887 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.887 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.888 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.888 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.888 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_BJ5CQVUVEwziAMiAEuX9RgoMmSMTfbd", 2023-10-11T15:27:33.888 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj93", 2023-10-11T15:27:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 9 2023-10-11T15:27:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 3, 2023-10-11T15:27:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:24:15.844840Z", 2023-10-11T15:27:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_ZHp-4syV-zfm4_FGAPbSRXibzk-i_c4", 2023-10-11T15:27:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "obj95", 2023-10-11T15:27:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "instance": "", 2023-10-11T15:27:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "ver": { 2023-10-11T15:27:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pool": 13, 2023-10-11T15:27:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 8 2023-10-11T15:27:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "locator": "", 2023-10-11T15:27:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "exists": true, 2023-10-11T15:27:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "meta": { 2023-10-11T15:27:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "category": 1, 2023-10-11T15:27:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "size": 16, 2023-10-11T15:27:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "mtime": "2023-10-11T15:18:55.464225Z", 2023-10-11T15:27:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-10-11T15:27:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_class": "", 2023-10-11T15:27:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "owner": "rgw-multisite-test-user", 2023-10-11T15:27:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "owner_display_name": "Multisite Test User", 2023-10-11T15:27:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "content_type": "application/octet-stream", 2023-10-11T15:27:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "accounted_size": 3, 2023-10-11T15:27:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "user_data": "", 2023-10-11T15:27:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "appendable": false 2023-10-11T15:27:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "tag": "_o9lfXqS_BqB1ROHZQ3XkHtKXx3l2Kx6", 2023-10-11T15:27:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "flags": 0, 2023-10-11T15:27:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "pending_map": [], 2023-10-11T15:27:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "versioned_epoch": 0 2023-10-11T15:27:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:33.900 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:27:33.901 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:27:33.902 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:27:33.904 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:27:33.905 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:27:33.928 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:27:33.928 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:27:33.928 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:27:34.929 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:34.953 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:27:34.953 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:27:34.953 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:27:34.954 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:27:34.954 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-59 2023-10-11T15:27:34.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:34.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:34.956 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:27:35.053 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:35.054 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:35.096 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:35.093+0000 7fcb7e2af780 1 bucket sync caught up with source: 2023-10-11T15:27:35.096 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:27:35.097 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.100.6, , , , , , 00000000002.101.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.56.6, , , , 00000000001.100.6] 2023-10-11T15:27:35.097 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:35.093+0000 7fcb7e2af780 0 bucket checkpoint complete 2023-10-11T15:27:35.109 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-59 2023-10-11T15:27:35.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:35.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:27:35.111 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:27:35.167 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:35.168 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:35.202 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:35.200+0000 7fdb905c6780 1 bucket sync caught up with source: 2023-10-11T15:27:35.203 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000009.251.6, 00000000006.296.6, 00000000006.315.6, 00000000015.145.6, 00000000012.326.6, 00000000003.168.6, 00000000003.150.6, 00000000012.298.6, 00000000021.223.6, 00000000018.334.6] 2023-10-11T15:27:35.203 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000009.251.6, 00000000006.296.6, 00000000006.315.6, 00000000015.145.6, 00000000012.326.6, 00000000003.168.6, 00000000003.150.6, 00000000012.298.6, 00000000021.223.6, 00000000018.334.6] 2023-10-11T15:27:35.203 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:35.200+0000 7fdb905c6780 0 bucket checkpoint complete 2023-10-11T15:27:35.215 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:27:35.216 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:35.217 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:35.217 DEBUG:boto:Method: HEAD 2023-10-11T15:27:35.217 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:27:35.218 DEBUG:boto:Data: 2023-10-11T15:27:35.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:35.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:35.219 DEBUG:boto:Port: 8001 2023-10-11T15:27:35.220 DEBUG:boto:Params: {} 2023-10-11T15:27:35.220 DEBUG:boto:Token: None 2023-10-11T15:27:35.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:35 GMT /dghqvd-59/ 2023-10-11T15:27:35.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pGp76HVyA5zJKDPMLwme7ewEi9E= 2023-10-11T15:27:35.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pGp76HVyA5zJKDPMLwme7ewEi9E=', 'Content-Length': '0'} 2023-10-11T15:27:35.223 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-10-11T15:27:35.223 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:27:36.157 DEBUG:boto:Token: None 2023-10-11T15:27:36.157 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:35 GMT /dghqvd-59/ 2023-10-11T15:27:36.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pGp76HVyA5zJKDPMLwme7ewEi9E= 2023-10-11T15:27:36.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pGp76HVyA5zJKDPMLwme7ewEi9E=', 'Content-Length': '0'} 2023-10-11T15:27:36.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000504a157e9e6b4b04-006526bee8-1e24-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.165 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.166 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.166 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.167 DEBUG:boto:Path: /dghqvd-59/ 2023-10-11T15:27:36.167 DEBUG:boto:Data: 2023-10-11T15:27:36.168 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.168 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.169 DEBUG:boto:Params: {} 2023-10-11T15:27:36.169 DEBUG:boto:Token: None 2023-10-11T15:27:36.170 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.170 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CmBQ/KBVzb8yJat/X/wd352J53Q= 2023-10-11T15:27:36.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CmBQ/KBVzb8yJat/X/wd352J53Q=', 'Content-Length': '0'} 2023-10-11T15:27:36.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be6b9709196655c2-006526bee8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.175 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:27:36.176 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.176 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.177 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.177 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.177 DEBUG:boto:Method: GET 2023-10-11T15:27:36.178 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.178 DEBUG:boto:Data: 2023-10-11T15:27:36.179 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.179 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.180 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.180 DEBUG:boto:Params: {} 2023-10-11T15:27:36.180 DEBUG:boto:Token: None 2023-10-11T15:27:36.181 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/?versions 2023-10-11T15:27:36.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4= 2023-10-11T15:27:36.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4=', 'Content-Length': '0'} 2023-10-11T15:27:36.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018312517243019c2-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.185 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:27:36.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:27:36.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:27:36.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:27:36.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:27:36.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:27:36.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:27:36.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:27:36.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:27:36.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:27:36.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:27:36.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:27:36.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:27:36.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:27:36.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:27:36.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:27:36.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:27:36.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:27:36.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:27:36.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:27:36.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:27:36.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:27:36.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:27:36.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:27:36.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:27:36.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:27:36.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:27:36.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:27:36.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:27:36.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:27:36.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:27:36.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:27:36.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:27:36.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:27:36.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:27:36.200 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:27:36.200 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.201 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.201 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.202 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.202 DEBUG:boto:Method: GET 2023-10-11T15:27:36.203 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.203 DEBUG:boto:Data: 2023-10-11T15:27:36.203 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.204 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.205 DEBUG:boto:Params: {} 2023-10-11T15:27:36.205 DEBUG:boto:Token: None 2023-10-11T15:27:36.206 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/?versions 2023-10-11T15:27:36.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4= 2023-10-11T15:27:36.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4=', 'Content-Length': '0'} 2023-10-11T15:27:36.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005109b8f06673e0bd-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.212 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:27:36.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:27:36.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:27:36.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:27:36.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:27:36.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:27:36.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:27:36.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:27:36.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:27:36.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:27:36.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:27:36.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:27:36.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:27:36.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:27:36.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:27:36.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:27:36.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:27:36.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:27:36.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:27:36.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:27:36.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:27:36.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:27:36.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:27:36.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:27:36.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:27:36.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:27:36.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:27:36.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:27:36.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:27:36.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:27:36.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:27:36.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:27:36.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:27:36.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:27:36.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:27:36.229 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.229 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.230 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.230 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.230 DEBUG:boto:Method: GET 2023-10-11T15:27:36.231 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.231 DEBUG:boto:Data: 2023-10-11T15:27:36.232 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.232 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.233 DEBUG:boto:Params: {} 2023-10-11T15:27:36.233 DEBUG:boto:Token: None 2023-10-11T15:27:36.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/?versions 2023-10-11T15:27:36.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4= 2023-10-11T15:27:36.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4=', 'Content-Length': '0'} 2023-10-11T15:27:36.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b59c6c6d8dd266-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.238 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.240 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.241 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.241 DEBUG:boto:path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.242 DEBUG:boto:auth_path=/dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.242 DEBUG:boto:Method: GET 2023-10-11T15:27:36.242 DEBUG:boto:Path: /dghqvd-59/?versions&max-keys=999 2023-10-11T15:27:36.243 DEBUG:boto:Data: 2023-10-11T15:27:36.243 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.244 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.245 DEBUG:boto:Params: {} 2023-10-11T15:27:36.245 DEBUG:boto:Token: None 2023-10-11T15:27:36.245 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/?versions 2023-10-11T15:27:36.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4= 2023-10-11T15:27:36.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3e2VmP09jsWhi1b5rzPzpsGOMe4=', 'Content-Length': '0'} 2023-10-11T15:27:36.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9db9f489c023887-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.251 DEBUG:boto:b'dghqvd-59999falseobj11nulltrue2023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:36.254 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.254 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.254 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.255 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.255 DEBUG:boto:Method: GET 2023-10-11T15:27:36.256 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.256 DEBUG:boto:Data: 2023-10-11T15:27:36.256 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.257 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.258 DEBUG:boto:Params: {} 2023-10-11T15:27:36.258 DEBUG:boto:Token: None 2023-10-11T15:27:36.259 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY= 2023-10-11T15:27:36.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY=', 'Content-Length': '0'} 2023-10-11T15:27:36.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eee693d70c545a7-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.264 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.264 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.264 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.265 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.265 DEBUG:boto:Method: GET 2023-10-11T15:27:36.266 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.266 DEBUG:boto:Data: 2023-10-11T15:27:36.267 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.268 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.268 DEBUG:boto:Params: {} 2023-10-11T15:27:36.269 DEBUG:boto:Token: None 2023-10-11T15:27:36.269 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY= 2023-10-11T15:27:36.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY=', 'Content-Length': '0'} 2023-10-11T15:27:36.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2fca377c4b11198-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.314 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.315 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.315 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.316 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.316 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.317 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.317 DEBUG:boto:Data: 2023-10-11T15:27:36.317 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.318 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.319 DEBUG:boto:Params: {} 2023-10-11T15:27:36.319 DEBUG:boto:Token: None 2023-10-11T15:27:36.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XEDOZpxD3kPQQrUaeLryQqhmAms= 2023-10-11T15:27:36.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XEDOZpxD3kPQQrUaeLryQqhmAms=', 'Content-Length': '0'} 2023-10-11T15:27:36.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2aa54962e88effe-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.323 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.324 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.324 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.325 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.325 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.325 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.326 DEBUG:boto:Data: 2023-10-11T15:27:36.326 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.327 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.327 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.327 DEBUG:boto:Params: {} 2023-10-11T15:27:36.328 DEBUG:boto:Token: None 2023-10-11T15:27:36.328 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.329 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XEDOZpxD3kPQQrUaeLryQqhmAms= 2023-10-11T15:27:36.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XEDOZpxD3kPQQrUaeLryQqhmAms=', 'Content-Length': '0'} 2023-10-11T15:27:36.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6fd00d05ca1ebb-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:36.333 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.333 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.334 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.334 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.335 DEBUG:boto:Method: GET 2023-10-11T15:27:36.335 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.335 DEBUG:boto:Data: 2023-10-11T15:27:36.336 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.337 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.337 DEBUG:boto:Params: {} 2023-10-11T15:27:36.338 DEBUG:boto:Token: None 2023-10-11T15:27:36.338 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.339 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY= 2023-10-11T15:27:36.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY=', 'Content-Length': '0'} 2023-10-11T15:27:36.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5706cf45a441802-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.342 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.342 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.343 DEBUG:boto:path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.343 DEBUG:boto:auth_path=/dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.344 DEBUG:boto:Method: GET 2023-10-11T15:27:36.344 DEBUG:boto:Path: /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.345 DEBUG:boto:Data: 2023-10-11T15:27:36.345 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.346 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.346 DEBUG:boto:Params: {} 2023-10-11T15:27:36.347 DEBUG:boto:Token: None 2023-10-11T15:27:36.348 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11?versionId=null 2023-10-11T15:27:36.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY= 2023-10-11T15:27:36.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1BXuhy7X9aYpDmS0zB2ddt5tLUY=', 'Content-Length': '0'} 2023-10-11T15:27:36.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ae1fcdee9ba9947-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.351 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.352 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.352 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.353 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.353 DEBUG:boto:Method: GET 2023-10-11T15:27:36.354 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.354 DEBUG:boto:Data: 2023-10-11T15:27:36.355 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.355 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.356 DEBUG:boto:Params: {} 2023-10-11T15:27:36.356 DEBUG:boto:Token: None 2023-10-11T15:27:36.357 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000774fe85b6101bf93-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.361 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.362 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.362 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.363 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.363 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.364 DEBUG:boto:Method: GET 2023-10-11T15:27:36.364 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.364 DEBUG:boto:Data: 2023-10-11T15:27:36.365 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.365 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.366 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.366 DEBUG:boto:Params: {} 2023-10-11T15:27:36.367 DEBUG:boto:Token: None 2023-10-11T15:27:36.367 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017ec62e2e0ef18ce-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.373 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.373 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.374 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.374 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.375 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:36.375 DEBUG:boto:Data: 2023-10-11T15:27:36.376 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.376 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.377 DEBUG:boto:Params: {} 2023-10-11T15:27:36.377 DEBUG:boto:Token: None 2023-10-11T15:27:36.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11 2023-10-11T15:27:36.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:C/TkUOAjJK2rvPUOK/8NuxUHBvs= 2023-10-11T15:27:36.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C/TkUOAjJK2rvPUOK/8NuxUHBvs=', 'Content-Length': '0'} 2023-10-11T15:27:36.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca7659af1620ed8-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.382 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.382 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.382 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.383 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:36.383 DEBUG:boto:Data: 2023-10-11T15:27:36.384 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.384 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.385 DEBUG:boto:Params: {} 2023-10-11T15:27:36.385 DEBUG:boto:Token: None 2023-10-11T15:27:36.386 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11 2023-10-11T15:27:36.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:C/TkUOAjJK2rvPUOK/8NuxUHBvs= 2023-10-11T15:27:36.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C/TkUOAjJK2rvPUOK/8NuxUHBvs=', 'Content-Length': '0'} 2023-10-11T15:27:36.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d7f9b377387125-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:27:36.390 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.390 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.391 DEBUG:boto:Method: GET 2023-10-11T15:27:36.391 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:36.392 DEBUG:boto:Data: 2023-10-11T15:27:36.392 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.393 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.393 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.393 DEBUG:boto:Params: {} 2023-10-11T15:27:36.394 DEBUG:boto:Token: None 2023-10-11T15:27:36.394 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11 2023-10-11T15:27:36.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NyXZIKoYEBW6Cyuy2eeF4ZC8iYs= 2023-10-11T15:27:36.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NyXZIKoYEBW6Cyuy2eeF4ZC8iYs=', 'Content-Length': '0'} 2023-10-11T15:27:36.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac4d469d598f7ee-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.398 DEBUG:boto:path=/dghqvd-59/obj11 2023-10-11T15:27:36.399 DEBUG:boto:auth_path=/dghqvd-59/obj11 2023-10-11T15:27:36.399 DEBUG:boto:Method: GET 2023-10-11T15:27:36.400 DEBUG:boto:Path: /dghqvd-59/obj11 2023-10-11T15:27:36.400 DEBUG:boto:Data: 2023-10-11T15:27:36.401 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.401 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.402 DEBUG:boto:Params: {} 2023-10-11T15:27:36.402 DEBUG:boto:Token: None 2023-10-11T15:27:36.403 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj11 2023-10-11T15:27:36.403 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NyXZIKoYEBW6Cyuy2eeF4ZC8iYs= 2023-10-11T15:27:36.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NyXZIKoYEBW6Cyuy2eeF4ZC8iYs=', 'Content-Length': '0'} 2023-10-11T15:27:36.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002336f1ae4b8e8db6-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.450 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.451 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.451 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.452 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.452 DEBUG:boto:Method: GET 2023-10-11T15:27:36.453 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.453 DEBUG:boto:Data: 2023-10-11T15:27:36.454 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.455 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.455 DEBUG:boto:Params: {} 2023-10-11T15:27:36.455 DEBUG:boto:Token: None 2023-10-11T15:27:36.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e1d03688d2217e4-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.460 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.461 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.461 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.462 DEBUG:boto:path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.462 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.463 DEBUG:boto:Method: GET 2023-10-11T15:27:36.463 DEBUG:boto:Path: /dghqvd-59/?prefix=obj11 2023-10-11T15:27:36.464 DEBUG:boto:Data: 2023-10-11T15:27:36.464 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.464 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.465 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.465 DEBUG:boto:Params: {} 2023-10-11T15:27:36.466 DEBUG:boto:Token: None 2023-10-11T15:27:36.466 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5fb4840109f6089-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.472 DEBUG:boto:b'dghqvd-59obj111000falseobj112023-10-11T15:16:05.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:36.473 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.474 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.474 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.475 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.475 DEBUG:boto:Method: GET 2023-10-11T15:27:36.476 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.476 DEBUG:boto:Data: 2023-10-11T15:27:36.476 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.477 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.478 DEBUG:boto:Params: {} 2023-10-11T15:27:36.478 DEBUG:boto:Token: None 2023-10-11T15:27:36.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8= 2023-10-11T15:27:36.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8=', 'Content-Length': '0'} 2023-10-11T15:27:36.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15bad05ae0c9733-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.530 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.531 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.531 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.532 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.532 DEBUG:boto:Method: GET 2023-10-11T15:27:36.533 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.533 DEBUG:boto:Data: 2023-10-11T15:27:36.533 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.534 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.535 DEBUG:boto:Params: {} 2023-10-11T15:27:36.535 DEBUG:boto:Token: None 2023-10-11T15:27:36.536 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8= 2023-10-11T15:27:36.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8=', 'Content-Length': '0'} 2023-10-11T15:27:36.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f12655805482f2-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.540 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.541 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.541 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.541 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.542 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.542 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.543 DEBUG:boto:Data: 2023-10-11T15:27:36.543 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.544 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.544 DEBUG:boto:Params: {} 2023-10-11T15:27:36.545 DEBUG:boto:Token: None 2023-10-11T15:27:36.545 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ1DMAM52RXo+5GMo9GE5oW5krY= 2023-10-11T15:27:36.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ1DMAM52RXo+5GMo9GE5oW5krY=', 'Content-Length': '0'} 2023-10-11T15:27:36.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b2f15f3f3f692e-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.549 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.549 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.550 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.550 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.551 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.551 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.551 DEBUG:boto:Data: 2023-10-11T15:27:36.552 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.553 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.553 DEBUG:boto:Params: {} 2023-10-11T15:27:36.553 DEBUG:boto:Token: None 2023-10-11T15:27:36.554 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.554 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mJ1DMAM52RXo+5GMo9GE5oW5krY= 2023-10-11T15:27:36.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mJ1DMAM52RXo+5GMo9GE5oW5krY=', 'Content-Length': '0'} 2023-10-11T15:27:36.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959354f321114c35-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:36.558 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.558 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.559 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.559 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.559 DEBUG:boto:Method: GET 2023-10-11T15:27:36.560 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.560 DEBUG:boto:Data: 2023-10-11T15:27:36.561 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.562 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.562 DEBUG:boto:Params: {} 2023-10-11T15:27:36.563 DEBUG:boto:Token: None 2023-10-11T15:27:36.563 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.563 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8= 2023-10-11T15:27:36.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8=', 'Content-Length': '0'} 2023-10-11T15:27:36.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf349a66503a7a28-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.567 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.567 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.567 DEBUG:boto:path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.568 DEBUG:boto:auth_path=/dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.568 DEBUG:boto:Method: GET 2023-10-11T15:27:36.569 DEBUG:boto:Path: /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.569 DEBUG:boto:Data: 2023-10-11T15:27:36.570 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.570 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.571 DEBUG:boto:Params: {} 2023-10-11T15:27:36.571 DEBUG:boto:Token: None 2023-10-11T15:27:36.572 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111?versionId=null 2023-10-11T15:27:36.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8= 2023-10-11T15:27:36.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SvJW5XuAvk9bqwlev1+q+6UZvA8=', 'Content-Length': '0'} 2023-10-11T15:27:36.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ce7effb8af7715-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.575 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.576 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.576 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.577 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.577 DEBUG:boto:Method: GET 2023-10-11T15:27:36.578 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.578 DEBUG:boto:Data: 2023-10-11T15:27:36.579 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.579 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.580 DEBUG:boto:Params: {} 2023-10-11T15:27:36.580 DEBUG:boto:Token: None 2023-10-11T15:27:36.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055efcd628d6c24b7-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.585 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.586 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.586 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.587 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.587 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.587 DEBUG:boto:Method: GET 2023-10-11T15:27:36.588 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.588 DEBUG:boto:Data: 2023-10-11T15:27:36.589 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.590 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.590 DEBUG:boto:Params: {} 2023-10-11T15:27:36.591 DEBUG:boto:Token: None 2023-10-11T15:27:36.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0f027ab89bafcc1-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.642 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.643 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.644 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.644 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.645 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:36.645 DEBUG:boto:Data: 2023-10-11T15:27:36.645 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.646 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.647 DEBUG:boto:Params: {} 2023-10-11T15:27:36.647 DEBUG:boto:Token: None 2023-10-11T15:27:36.648 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111 2023-10-11T15:27:36.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rUVmDwuwuw5q5FCemhe9c+C2FqY= 2023-10-11T15:27:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rUVmDwuwuw5q5FCemhe9c+C2FqY=', 'Content-Length': '0'} 2023-10-11T15:27:36.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c168091a24c3eaa-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.651 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.652 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.652 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.653 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:36.653 DEBUG:boto:Data: 2023-10-11T15:27:36.653 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.654 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.654 DEBUG:boto:Params: {} 2023-10-11T15:27:36.655 DEBUG:boto:Token: None 2023-10-11T15:27:36.655 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111 2023-10-11T15:27:36.656 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rUVmDwuwuw5q5FCemhe9c+C2FqY= 2023-10-11T15:27:36.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rUVmDwuwuw5q5FCemhe9c+C2FqY=', 'Content-Length': '0'} 2023-10-11T15:27:36.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a7812eaf87c204-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:27:36.659 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.660 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.660 DEBUG:boto:Method: GET 2023-10-11T15:27:36.661 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:36.661 DEBUG:boto:Data: 2023-10-11T15:27:36.662 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.662 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.663 DEBUG:boto:Params: {} 2023-10-11T15:27:36.663 DEBUG:boto:Token: None 2023-10-11T15:27:36.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111 2023-10-11T15:27:36.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VbOHRnxIS+Tb1vcSuYkyJXvKSW4= 2023-10-11T15:27:36.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VbOHRnxIS+Tb1vcSuYkyJXvKSW4=', 'Content-Length': '0'} 2023-10-11T15:27:36.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066ecd900f3863df-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.668 DEBUG:boto:path=/dghqvd-59/obj111 2023-10-11T15:27:36.668 DEBUG:boto:auth_path=/dghqvd-59/obj111 2023-10-11T15:27:36.669 DEBUG:boto:Method: GET 2023-10-11T15:27:36.669 DEBUG:boto:Path: /dghqvd-59/obj111 2023-10-11T15:27:36.670 DEBUG:boto:Data: 2023-10-11T15:27:36.670 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.671 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.671 DEBUG:boto:Params: {} 2023-10-11T15:27:36.672 DEBUG:boto:Token: None 2023-10-11T15:27:36.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj111 2023-10-11T15:27:36.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VbOHRnxIS+Tb1vcSuYkyJXvKSW4= 2023-10-11T15:27:36.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VbOHRnxIS+Tb1vcSuYkyJXvKSW4=', 'Content-Length': '0'} 2023-10-11T15:27:36.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba44f18f1f8d0e3c-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.676 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.677 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.677 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.678 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.678 DEBUG:boto:Method: GET 2023-10-11T15:27:36.678 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.679 DEBUG:boto:Data: 2023-10-11T15:27:36.679 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.680 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.681 DEBUG:boto:Params: {} 2023-10-11T15:27:36.681 DEBUG:boto:Token: None 2023-10-11T15:27:36.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a616ed5bbe351a2-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.726 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.727 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.728 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.728 DEBUG:boto:path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.729 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.729 DEBUG:boto:Method: GET 2023-10-11T15:27:36.730 DEBUG:boto:Path: /dghqvd-59/?prefix=obj111 2023-10-11T15:27:36.730 DEBUG:boto:Data: 2023-10-11T15:27:36.730 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.731 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.731 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.732 DEBUG:boto:Params: {} 2023-10-11T15:27:36.732 DEBUG:boto:Token: None 2023-10-11T15:27:36.733 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6eeb8b0e9bed79-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.739 DEBUG:boto:b'dghqvd-59obj1111000falseobj1112023-10-11T15:25:38.282Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:36.740 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.740 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.741 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.741 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.742 DEBUG:boto:Method: GET 2023-10-11T15:27:36.742 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.742 DEBUG:boto:Data: 2023-10-11T15:27:36.743 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.744 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.744 DEBUG:boto:Params: {} 2023-10-11T15:27:36.745 DEBUG:boto:Token: None 2023-10-11T15:27:36.745 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E= 2023-10-11T15:27:36.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E=', 'Content-Length': '0'} 2023-10-11T15:27:36.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfac44b54ef832e2-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.749 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.750 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.750 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.751 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.751 DEBUG:boto:Method: GET 2023-10-11T15:27:36.752 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.752 DEBUG:boto:Data: 2023-10-11T15:27:36.752 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.753 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.753 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.754 DEBUG:boto:Params: {} 2023-10-11T15:27:36.754 DEBUG:boto:Token: None 2023-10-11T15:27:36.755 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.755 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E= 2023-10-11T15:27:36.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E=', 'Content-Length': '0'} 2023-10-11T15:27:36.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7144b13ae62396-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.806 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.807 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.807 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.808 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.808 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.809 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.809 DEBUG:boto:Data: 2023-10-11T15:27:36.810 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.810 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.811 DEBUG:boto:Params: {} 2023-10-11T15:27:36.811 DEBUG:boto:Token: None 2023-10-11T15:27:36.812 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8TX54j4sBPW55AuTbMyQ4UZQAc= 2023-10-11T15:27:36.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8TX54j4sBPW55AuTbMyQ4UZQAc=', 'Content-Length': '0'} 2023-10-11T15:27:36.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043be3d94ee821681-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.816 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.816 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.817 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.817 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.817 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.818 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.818 DEBUG:boto:Data: 2023-10-11T15:27:36.819 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.819 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.820 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.820 DEBUG:boto:Params: {} 2023-10-11T15:27:36.821 DEBUG:boto:Token: None 2023-10-11T15:27:36.821 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g8TX54j4sBPW55AuTbMyQ4UZQAc= 2023-10-11T15:27:36.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g8TX54j4sBPW55AuTbMyQ4UZQAc=', 'Content-Length': '0'} 2023-10-11T15:27:36.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad4314d9693f760-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:36.826 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.826 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.827 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.827 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.828 DEBUG:boto:Method: GET 2023-10-11T15:27:36.828 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.828 DEBUG:boto:Data: 2023-10-11T15:27:36.829 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.829 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.830 DEBUG:boto:Params: {} 2023-10-11T15:27:36.830 DEBUG:boto:Token: None 2023-10-11T15:27:36.831 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.831 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E= 2023-10-11T15:27:36.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E=', 'Content-Length': '0'} 2023-10-11T15:27:36.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b4705c73a92aa4-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.834 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.835 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.836 DEBUG:boto:path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.836 DEBUG:boto:auth_path=/dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.837 DEBUG:boto:Method: GET 2023-10-11T15:27:36.837 DEBUG:boto:Path: /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.838 DEBUG:boto:Data: 2023-10-11T15:27:36.838 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.839 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.839 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.840 DEBUG:boto:Params: {} 2023-10-11T15:27:36.840 DEBUG:boto:Token: None 2023-10-11T15:27:36.841 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115?versionId=null 2023-10-11T15:27:36.841 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E= 2023-10-11T15:27:36.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kHStqyNRHzTUVKs9bMqQAmegM+E=', 'Content-Length': '0'} 2023-10-11T15:27:36.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e992c9f7a1cbc9c3-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.844 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.845 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.846 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.846 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.847 DEBUG:boto:Method: GET 2023-10-11T15:27:36.847 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.848 DEBUG:boto:Data: 2023-10-11T15:27:36.848 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.848 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.849 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.849 DEBUG:boto:Params: {} 2023-10-11T15:27:36.850 DEBUG:boto:Token: None 2023-10-11T15:27:36.850 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.851 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000910ed61b05dc1523-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.855 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.855 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.856 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.857 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.857 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.858 DEBUG:boto:Method: GET 2023-10-11T15:27:36.858 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.859 DEBUG:boto:Data: 2023-10-11T15:27:36.859 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.859 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.860 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.860 DEBUG:boto:Params: {} 2023-10-11T15:27:36.861 DEBUG:boto:Token: None 2023-10-11T15:27:36.861 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.862 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032cb1f269fa86c06-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.867 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.868 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.869 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.869 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.869 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:36.870 DEBUG:boto:Data: 2023-10-11T15:27:36.870 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.871 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.872 DEBUG:boto:Params: {} 2023-10-11T15:27:36.872 DEBUG:boto:Token: None 2023-10-11T15:27:36.873 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115 2023-10-11T15:27:36.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XBxp08KOfwnZOUiHrasa1TYMcKY= 2023-10-11T15:27:36.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XBxp08KOfwnZOUiHrasa1TYMcKY=', 'Content-Length': '0'} 2023-10-11T15:27:36.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a2433fff549b18-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.876 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.877 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.877 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.878 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:36.879 DEBUG:boto:Data: 2023-10-11T15:27:36.879 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.879 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.880 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.880 DEBUG:boto:Params: {} 2023-10-11T15:27:36.881 DEBUG:boto:Token: None 2023-10-11T15:27:36.881 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115 2023-10-11T15:27:36.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XBxp08KOfwnZOUiHrasa1TYMcKY= 2023-10-11T15:27:36.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XBxp08KOfwnZOUiHrasa1TYMcKY=', 'Content-Length': '0'} 2023-10-11T15:27:36.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fbd7b3a62f97b8-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:27:36.885 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.886 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.886 DEBUG:boto:Method: GET 2023-10-11T15:27:36.886 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:36.887 DEBUG:boto:Data: 2023-10-11T15:27:36.888 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.888 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.889 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.889 DEBUG:boto:Params: {} 2023-10-11T15:27:36.890 DEBUG:boto:Token: None 2023-10-11T15:27:36.890 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115 2023-10-11T15:27:36.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:30Qw+76JtwicY69MqY6imqhCneg= 2023-10-11T15:27:36.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:30Qw+76JtwicY69MqY6imqhCneg=', 'Content-Length': '0'} 2023-10-11T15:27:36.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6680c6498a4fda-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.894 DEBUG:boto:path=/dghqvd-59/obj115 2023-10-11T15:27:36.895 DEBUG:boto:auth_path=/dghqvd-59/obj115 2023-10-11T15:27:36.895 DEBUG:boto:Method: GET 2023-10-11T15:27:36.896 DEBUG:boto:Path: /dghqvd-59/obj115 2023-10-11T15:27:36.896 DEBUG:boto:Data: 2023-10-11T15:27:36.897 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.897 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.898 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.898 DEBUG:boto:Params: {} 2023-10-11T15:27:36.899 DEBUG:boto:Token: None 2023-10-11T15:27:36.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj115 2023-10-11T15:27:36.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:30Qw+76JtwicY69MqY6imqhCneg= 2023-10-11T15:27:36.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:30Qw+76JtwicY69MqY6imqhCneg=', 'Content-Length': '0'} 2023-10-11T15:27:36.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a59d11273f3331d-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.904 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.905 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.905 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.905 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.906 DEBUG:boto:Method: GET 2023-10-11T15:27:36.906 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.907 DEBUG:boto:Data: 2023-10-11T15:27:36.907 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.908 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.909 DEBUG:boto:Params: {} 2023-10-11T15:27:36.909 DEBUG:boto:Token: None 2023-10-11T15:27:36.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbe30bda3eb5aeb2-006526bee8-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.915 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.915 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:36.916 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:36.916 DEBUG:boto:path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.917 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.917 DEBUG:boto:Method: GET 2023-10-11T15:27:36.918 DEBUG:boto:Path: /dghqvd-59/?prefix=obj115 2023-10-11T15:27:36.918 DEBUG:boto:Data: 2023-10-11T15:27:36.918 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.919 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.920 DEBUG:boto:Params: {} 2023-10-11T15:27:36.921 DEBUG:boto:Token: None 2023-10-11T15:27:36.921 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/ 2023-10-11T15:27:36.922 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go= 2023-10-11T15:27:36.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D5XnSTAbs/zuWocBNlKAAnw35go=', 'Content-Length': '0'} 2023-10-11T15:27:36.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010c446f4c90c34ab-006526bee8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.970 DEBUG:boto:b'dghqvd-59obj1151000falseobj1152023-10-11T15:21:14.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:36.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:36.972 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:36.973 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:36.974 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.975 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.975 DEBUG:boto:Method: GET 2023-10-11T15:27:36.976 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.976 DEBUG:boto:Data: 2023-10-11T15:27:36.977 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.978 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.979 DEBUG:boto:Params: {} 2023-10-11T15:27:36.979 DEBUG:boto:Token: None 2023-10-11T15:27:36.980 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:laO+abyk7apaAlT3FgJE8Ul2abY= 2023-10-11T15:27:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:laO+abyk7apaAlT3FgJE8Ul2abY=', 'Content-Length': '0'} 2023-10-11T15:27:36.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf39618b03afb2e-006526bee8-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.984 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:36.985 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:36.985 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.986 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.986 DEBUG:boto:Method: GET 2023-10-11T15:27:36.986 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.987 DEBUG:boto:Data: 2023-10-11T15:27:36.987 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:36.989 DEBUG:boto:Port: 8000 2023-10-11T15:27:36.989 DEBUG:boto:Params: {} 2023-10-11T15:27:36.990 DEBUG:boto:Token: None 2023-10-11T15:27:36.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:36 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:laO+abyk7apaAlT3FgJE8Ul2abY= 2023-10-11T15:27:36.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:laO+abyk7apaAlT3FgJE8Ul2abY=', 'Content-Length': '0'} 2023-10-11T15:27:36.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f0fe8cf5ec0173-006526bee8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:36.994 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:36.994 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:36.995 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.995 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.996 DEBUG:boto:Method: HEAD 2023-10-11T15:27:36.996 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:36.997 DEBUG:boto:Data: 2023-10-11T15:27:36.997 DEBUG:boto:Headers: {} 2023-10-11T15:27:36.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:36.998 DEBUG:boto:Port: 8001 2023-10-11T15:27:36.999 DEBUG:boto:Params: {} 2023-10-11T15:27:36.999 DEBUG:boto:Token: None 2023-10-11T15:27:37.000 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctjaUZ95dk3ukKrsbDBc4xQekoI= 2023-10-11T15:27:37.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctjaUZ95dk3ukKrsbDBc4xQekoI=', 'Content-Length': '0'} 2023-10-11T15:27:37.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d2870ecff7b413f-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.003 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.004 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.004 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.005 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.005 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.006 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.006 DEBUG:boto:Data: 2023-10-11T15:27:37.006 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.007 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.008 DEBUG:boto:Params: {} 2023-10-11T15:27:37.008 DEBUG:boto:Token: None 2023-10-11T15:27:37.009 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ctjaUZ95dk3ukKrsbDBc4xQekoI= 2023-10-11T15:27:37.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ctjaUZ95dk3ukKrsbDBc4xQekoI=', 'Content-Length': '0'} 2023-10-11T15:27:37.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765abfb4a6568951-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:37.013 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.013 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.014 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.014 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.015 DEBUG:boto:Method: GET 2023-10-11T15:27:37.015 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.015 DEBUG:boto:Data: 2023-10-11T15:27:37.016 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.017 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.017 DEBUG:boto:Params: {} 2023-10-11T15:27:37.018 DEBUG:boto:Token: None 2023-10-11T15:27:37.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ns1NLHFVtMAYftnCovjd9mJuIDU= 2023-10-11T15:27:37.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ns1NLHFVtMAYftnCovjd9mJuIDU=', 'Content-Length': '0'} 2023-10-11T15:27:37.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc6c68a0710c361-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.066 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.067 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.067 DEBUG:boto:path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.068 DEBUG:boto:auth_path=/dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.068 DEBUG:boto:Method: GET 2023-10-11T15:27:37.069 DEBUG:boto:Path: /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.069 DEBUG:boto:Data: 2023-10-11T15:27:37.069 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.070 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.071 DEBUG:boto:Params: {} 2023-10-11T15:27:37.071 DEBUG:boto:Token: None 2023-10-11T15:27:37.072 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116?versionId=null 2023-10-11T15:27:37.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ns1NLHFVtMAYftnCovjd9mJuIDU= 2023-10-11T15:27:37.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ns1NLHFVtMAYftnCovjd9mJuIDU=', 'Content-Length': '0'} 2023-10-11T15:27:37.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f112a78bbb249a3c-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.076 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.076 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.077 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.077 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.078 DEBUG:boto:Method: GET 2023-10-11T15:27:37.078 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.078 DEBUG:boto:Data: 2023-10-11T15:27:37.079 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.079 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.080 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.080 DEBUG:boto:Params: {} 2023-10-11T15:27:37.081 DEBUG:boto:Token: None 2023-10-11T15:27:37.081 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef66f8c4e52ee96-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.085 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.086 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.087 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.087 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.087 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.088 DEBUG:boto:Method: GET 2023-10-11T15:27:37.088 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.089 DEBUG:boto:Data: 2023-10-11T15:27:37.089 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.090 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.091 DEBUG:boto:Params: {} 2023-10-11T15:27:37.091 DEBUG:boto:Token: None 2023-10-11T15:27:37.091 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.092 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b309731196c217a-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.097 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.098 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.098 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.099 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.099 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:37.100 DEBUG:boto:Data: 2023-10-11T15:27:37.100 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.100 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.101 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.101 DEBUG:boto:Params: {} 2023-10-11T15:27:37.102 DEBUG:boto:Token: None 2023-10-11T15:27:37.102 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116 2023-10-11T15:27:37.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UjkxJxEUjNMBomlEwX7/d/EPrkg= 2023-10-11T15:27:37.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UjkxJxEUjNMBomlEwX7/d/EPrkg=', 'Content-Length': '0'} 2023-10-11T15:27:37.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d4283bdfea6bbc5-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.106 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.106 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.107 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.107 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:37.108 DEBUG:boto:Data: 2023-10-11T15:27:37.108 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.109 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.109 DEBUG:boto:Params: {} 2023-10-11T15:27:37.110 DEBUG:boto:Token: None 2023-10-11T15:27:37.110 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116 2023-10-11T15:27:37.111 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UjkxJxEUjNMBomlEwX7/d/EPrkg= 2023-10-11T15:27:37.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UjkxJxEUjNMBomlEwX7/d/EPrkg=', 'Content-Length': '0'} 2023-10-11T15:27:37.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000960162ca91f69371-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:27:37.115 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.115 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.116 DEBUG:boto:Method: GET 2023-10-11T15:27:37.116 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:37.117 DEBUG:boto:Data: 2023-10-11T15:27:37.117 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.118 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.118 DEBUG:boto:Params: {} 2023-10-11T15:27:37.119 DEBUG:boto:Token: None 2023-10-11T15:27:37.119 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116 2023-10-11T15:27:37.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D1DYkib7FgNfo4q37Fccjgn3xHE= 2023-10-11T15:27:37.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D1DYkib7FgNfo4q37Fccjgn3xHE=', 'Content-Length': '0'} 2023-10-11T15:27:37.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e797aef0906162b5-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.123 DEBUG:boto:path=/dghqvd-59/obj116 2023-10-11T15:27:37.124 DEBUG:boto:auth_path=/dghqvd-59/obj116 2023-10-11T15:27:37.124 DEBUG:boto:Method: GET 2023-10-11T15:27:37.125 DEBUG:boto:Path: /dghqvd-59/obj116 2023-10-11T15:27:37.125 DEBUG:boto:Data: 2023-10-11T15:27:37.126 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.127 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.127 DEBUG:boto:Params: {} 2023-10-11T15:27:37.128 DEBUG:boto:Token: None 2023-10-11T15:27:37.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj116 2023-10-11T15:27:37.129 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D1DYkib7FgNfo4q37Fccjgn3xHE= 2023-10-11T15:27:37.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D1DYkib7FgNfo4q37Fccjgn3xHE=', 'Content-Length': '0'} 2023-10-11T15:27:37.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849d7c2b6c1238ff-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.174 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.175 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.176 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.176 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.176 DEBUG:boto:Method: GET 2023-10-11T15:27:37.177 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.177 DEBUG:boto:Data: 2023-10-11T15:27:37.178 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.179 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.179 DEBUG:boto:Params: {} 2023-10-11T15:27:37.180 DEBUG:boto:Token: None 2023-10-11T15:27:37.180 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.181 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d57e65d7e4aa63ea-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.184 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.185 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.186 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.186 DEBUG:boto:path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.187 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.187 DEBUG:boto:Method: GET 2023-10-11T15:27:37.187 DEBUG:boto:Path: /dghqvd-59/?prefix=obj116 2023-10-11T15:27:37.188 DEBUG:boto:Data: 2023-10-11T15:27:37.188 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.189 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.190 DEBUG:boto:Params: {} 2023-10-11T15:27:37.190 DEBUG:boto:Token: None 2023-10-11T15:27:37.191 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea8a31a88370c70-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.196 DEBUG:boto:b'dghqvd-59obj1161000falseobj1162023-10-11T15:21:59.474Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:37.197 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.198 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.198 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.199 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.199 DEBUG:boto:Method: GET 2023-10-11T15:27:37.199 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.200 DEBUG:boto:Data: 2023-10-11T15:27:37.200 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.201 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.202 DEBUG:boto:Params: {} 2023-10-11T15:27:37.202 DEBUG:boto:Token: None 2023-10-11T15:27:37.203 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc= 2023-10-11T15:27:37.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc=', 'Content-Length': '0'} 2023-10-11T15:27:37.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb5be48eceb1a72-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.207 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.208 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.208 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.209 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.209 DEBUG:boto:Method: GET 2023-10-11T15:27:37.209 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.210 DEBUG:boto:Data: 2023-10-11T15:27:37.210 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.211 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.212 DEBUG:boto:Params: {} 2023-10-11T15:27:37.212 DEBUG:boto:Token: None 2023-10-11T15:27:37.213 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc= 2023-10-11T15:27:37.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc=', 'Content-Length': '0'} 2023-10-11T15:27:37.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f505e9a8dc81d0-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.217 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.217 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.217 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.218 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.218 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.219 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.219 DEBUG:boto:Data: 2023-10-11T15:27:37.220 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.220 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.221 DEBUG:boto:Params: {} 2023-10-11T15:27:37.221 DEBUG:boto:Token: None 2023-10-11T15:27:37.222 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gvGlY1Aun3PXR5PBp3JIrpQ9VHQ= 2023-10-11T15:27:37.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gvGlY1Aun3PXR5PBp3JIrpQ9VHQ=', 'Content-Length': '0'} 2023-10-11T15:27:37.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4da0f48bcde4155-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.225 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.225 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.226 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.226 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.227 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.227 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.227 DEBUG:boto:Data: 2023-10-11T15:27:37.228 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.229 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.229 DEBUG:boto:Params: {} 2023-10-11T15:27:37.229 DEBUG:boto:Token: None 2023-10-11T15:27:37.230 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gvGlY1Aun3PXR5PBp3JIrpQ9VHQ= 2023-10-11T15:27:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gvGlY1Aun3PXR5PBp3JIrpQ9VHQ=', 'Content-Length': '0'} 2023-10-11T15:27:37.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fafbc36de1cb9b60-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:37.234 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.234 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.235 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.235 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.235 DEBUG:boto:Method: GET 2023-10-11T15:27:37.236 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.236 DEBUG:boto:Data: 2023-10-11T15:27:37.237 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.238 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.238 DEBUG:boto:Params: {} 2023-10-11T15:27:37.239 DEBUG:boto:Token: None 2023-10-11T15:27:37.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc= 2023-10-11T15:27:37.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc=', 'Content-Length': '0'} 2023-10-11T15:27:37.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e4bebc99bbe8c4-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.286 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.287 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.287 DEBUG:boto:path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.288 DEBUG:boto:auth_path=/dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.288 DEBUG:boto:Method: GET 2023-10-11T15:27:37.289 DEBUG:boto:Path: /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.289 DEBUG:boto:Data: 2023-10-11T15:27:37.290 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.291 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.291 DEBUG:boto:Params: {} 2023-10-11T15:27:37.292 DEBUG:boto:Token: None 2023-10-11T15:27:37.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124?versionId=null 2023-10-11T15:27:37.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc= 2023-10-11T15:27:37.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+v8Oo9Bx/uo9JfMzkyckia7XpMc=', 'Content-Length': '0'} 2023-10-11T15:27:37.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300e1e531d99f194-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.296 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.296 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.297 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.297 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.297 DEBUG:boto:Method: GET 2023-10-11T15:27:37.298 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.298 DEBUG:boto:Data: 2023-10-11T15:27:37.299 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.299 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.300 DEBUG:boto:Params: {} 2023-10-11T15:27:37.300 DEBUG:boto:Token: None 2023-10-11T15:27:37.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000449017740da87074-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.305 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.306 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.306 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.307 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.307 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.308 DEBUG:boto:Method: GET 2023-10-11T15:27:37.308 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.308 DEBUG:boto:Data: 2023-10-11T15:27:37.309 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.310 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.310 DEBUG:boto:Params: {} 2023-10-11T15:27:37.311 DEBUG:boto:Token: None 2023-10-11T15:27:37.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.312 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805c650ee26df0e5-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.362 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.363 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.364 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.364 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.365 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:37.365 DEBUG:boto:Data: 2023-10-11T15:27:37.366 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.366 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.367 DEBUG:boto:Params: {} 2023-10-11T15:27:37.367 DEBUG:boto:Token: None 2023-10-11T15:27:37.368 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124 2023-10-11T15:27:37.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:25nN8Xlupudbq5Qs1GHvWki22xg= 2023-10-11T15:27:37.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:25nN8Xlupudbq5Qs1GHvWki22xg=', 'Content-Length': '0'} 2023-10-11T15:27:37.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830cfb7f86754352-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.372 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.372 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.372 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.373 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:37.373 DEBUG:boto:Data: 2023-10-11T15:27:37.374 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.374 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.375 DEBUG:boto:Params: {} 2023-10-11T15:27:37.375 DEBUG:boto:Token: None 2023-10-11T15:27:37.376 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124 2023-10-11T15:27:37.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:25nN8Xlupudbq5Qs1GHvWki22xg= 2023-10-11T15:27:37.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:25nN8Xlupudbq5Qs1GHvWki22xg=', 'Content-Length': '0'} 2023-10-11T15:27:37.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5fb7462e7679d2-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:27:37.380 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.380 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.381 DEBUG:boto:Method: GET 2023-10-11T15:27:37.381 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:37.382 DEBUG:boto:Data: 2023-10-11T15:27:37.382 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.383 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.383 DEBUG:boto:Params: {} 2023-10-11T15:27:37.384 DEBUG:boto:Token: None 2023-10-11T15:27:37.384 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124 2023-10-11T15:27:37.385 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uBL67/7eSxZbmIix2dDwV5nEqV8= 2023-10-11T15:27:37.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uBL67/7eSxZbmIix2dDwV5nEqV8=', 'Content-Length': '0'} 2023-10-11T15:27:37.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ea8653708b58ea-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.388 DEBUG:boto:path=/dghqvd-59/obj124 2023-10-11T15:27:37.388 DEBUG:boto:auth_path=/dghqvd-59/obj124 2023-10-11T15:27:37.389 DEBUG:boto:Method: GET 2023-10-11T15:27:37.389 DEBUG:boto:Path: /dghqvd-59/obj124 2023-10-11T15:27:37.390 DEBUG:boto:Data: 2023-10-11T15:27:37.390 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.391 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.391 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.391 DEBUG:boto:Params: {} 2023-10-11T15:27:37.392 DEBUG:boto:Token: None 2023-10-11T15:27:37.392 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj124 2023-10-11T15:27:37.393 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uBL67/7eSxZbmIix2dDwV5nEqV8= 2023-10-11T15:27:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uBL67/7eSxZbmIix2dDwV5nEqV8=', 'Content-Length': '0'} 2023-10-11T15:27:37.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a902136dea8da258-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.396 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.397 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.397 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.397 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.398 DEBUG:boto:Method: GET 2023-10-11T15:27:37.398 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.399 DEBUG:boto:Data: 2023-10-11T15:27:37.399 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.400 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.400 DEBUG:boto:Params: {} 2023-10-11T15:27:37.401 DEBUG:boto:Token: None 2023-10-11T15:27:37.401 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fef71db927a85bc9-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.450 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.452 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.452 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.453 DEBUG:boto:path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.453 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.454 DEBUG:boto:Method: GET 2023-10-11T15:27:37.454 DEBUG:boto:Path: /dghqvd-59/?prefix=obj124 2023-10-11T15:27:37.455 DEBUG:boto:Data: 2023-10-11T15:27:37.455 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.456 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.456 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.457 DEBUG:boto:Params: {} 2023-10-11T15:27:37.457 DEBUG:boto:Token: None 2023-10-11T15:27:37.458 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffe8e1b224d9954-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.463 DEBUG:boto:b'dghqvd-59obj1241000falseobj1242023-10-11T15:24:15.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:37.465 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.465 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.466 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.466 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.466 DEBUG:boto:Method: GET 2023-10-11T15:27:37.467 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.467 DEBUG:boto:Data: 2023-10-11T15:27:37.468 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.468 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.469 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.469 DEBUG:boto:Params: {} 2023-10-11T15:27:37.470 DEBUG:boto:Token: None 2023-10-11T15:27:37.470 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw= 2023-10-11T15:27:37.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw=', 'Content-Length': '0'} 2023-10-11T15:27:37.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071abfc355dc804ce-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.474 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.475 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.475 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.476 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.476 DEBUG:boto:Method: GET 2023-10-11T15:27:37.477 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.477 DEBUG:boto:Data: 2023-10-11T15:27:37.478 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.479 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.479 DEBUG:boto:Params: {} 2023-10-11T15:27:37.479 DEBUG:boto:Token: None 2023-10-11T15:27:37.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw= 2023-10-11T15:27:37.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw=', 'Content-Length': '0'} 2023-10-11T15:27:37.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041992dfcd8d0ca5e-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.530 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.531 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.531 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.532 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.532 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.533 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.533 DEBUG:boto:Data: 2023-10-11T15:27:37.534 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.535 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.535 DEBUG:boto:Params: {} 2023-10-11T15:27:37.535 DEBUG:boto:Token: None 2023-10-11T15:27:37.536 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.536 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/eR67CNM2eKmRUuLxpbeRP38jlc= 2023-10-11T15:27:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/eR67CNM2eKmRUuLxpbeRP38jlc=', 'Content-Length': '0'} 2023-10-11T15:27:37.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb5d5f01709f0ee7-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.540 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.540 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.541 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.541 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.541 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.542 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.542 DEBUG:boto:Data: 2023-10-11T15:27:37.543 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.543 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.543 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.544 DEBUG:boto:Params: {} 2023-10-11T15:27:37.544 DEBUG:boto:Token: None 2023-10-11T15:27:37.545 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/eR67CNM2eKmRUuLxpbeRP38jlc= 2023-10-11T15:27:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/eR67CNM2eKmRUuLxpbeRP38jlc=', 'Content-Length': '0'} 2023-10-11T15:27:37.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000907ef641bcb91330-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:37.549 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.549 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.550 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.550 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.551 DEBUG:boto:Method: GET 2023-10-11T15:27:37.551 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.552 DEBUG:boto:Data: 2023-10-11T15:27:37.552 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.552 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.553 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.553 DEBUG:boto:Params: {} 2023-10-11T15:27:37.554 DEBUG:boto:Token: None 2023-10-11T15:27:37.554 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw= 2023-10-11T15:27:37.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw=', 'Content-Length': '0'} 2023-10-11T15:27:37.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001174e2a2dbc33fac-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.558 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.559 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.559 DEBUG:boto:path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.560 DEBUG:boto:auth_path=/dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.560 DEBUG:boto:Method: GET 2023-10-11T15:27:37.561 DEBUG:boto:Path: /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.561 DEBUG:boto:Data: 2023-10-11T15:27:37.561 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.562 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.563 DEBUG:boto:Params: {} 2023-10-11T15:27:37.563 DEBUG:boto:Token: None 2023-10-11T15:27:37.564 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145?versionId=null 2023-10-11T15:27:37.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw= 2023-10-11T15:27:37.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i391SudnxBziiP+a8HBM79maxPw=', 'Content-Length': '0'} 2023-10-11T15:27:37.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007458bbf589d6b50c-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.568 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.568 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.569 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.569 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.570 DEBUG:boto:Method: GET 2023-10-11T15:27:37.570 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.570 DEBUG:boto:Data: 2023-10-11T15:27:37.571 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.572 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.572 DEBUG:boto:Params: {} 2023-10-11T15:27:37.573 DEBUG:boto:Token: None 2023-10-11T15:27:37.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5397e1b2e28fb78-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.577 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.578 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.579 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.579 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.579 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.580 DEBUG:boto:Method: GET 2023-10-11T15:27:37.580 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.581 DEBUG:boto:Data: 2023-10-11T15:27:37.581 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.582 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.582 DEBUG:boto:Params: {} 2023-10-11T15:27:37.583 DEBUG:boto:Token: None 2023-10-11T15:27:37.583 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600cc2dccb68d400-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.589 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.590 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.591 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.591 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.592 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:37.592 DEBUG:boto:Data: 2023-10-11T15:27:37.592 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.593 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.593 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.594 DEBUG:boto:Params: {} 2023-10-11T15:27:37.594 DEBUG:boto:Token: None 2023-10-11T15:27:37.595 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145 2023-10-11T15:27:37.595 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oYTsW+y20gkUumvPN337hRpDRF8= 2023-10-11T15:27:37.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oYTsW+y20gkUumvPN337hRpDRF8=', 'Content-Length': '0'} 2023-10-11T15:27:37.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fe6e6da40e1e337-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.598 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.599 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.599 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.600 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:37.600 DEBUG:boto:Data: 2023-10-11T15:27:37.600 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.601 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.601 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.602 DEBUG:boto:Params: {} 2023-10-11T15:27:37.602 DEBUG:boto:Token: None 2023-10-11T15:27:37.603 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145 2023-10-11T15:27:37.603 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oYTsW+y20gkUumvPN337hRpDRF8= 2023-10-11T15:27:37.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oYTsW+y20gkUumvPN337hRpDRF8=', 'Content-Length': '0'} 2023-10-11T15:27:37.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057dc6606adec73be-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:27:37.607 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.607 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.608 DEBUG:boto:Method: GET 2023-10-11T15:27:37.608 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:37.609 DEBUG:boto:Data: 2023-10-11T15:27:37.609 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.609 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.610 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.610 DEBUG:boto:Params: {} 2023-10-11T15:27:37.611 DEBUG:boto:Token: None 2023-10-11T15:27:37.611 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145 2023-10-11T15:27:37.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rJ0tjma0SeEcT/wQlB5DBiTXkVU= 2023-10-11T15:27:37.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rJ0tjma0SeEcT/wQlB5DBiTXkVU=', 'Content-Length': '0'} 2023-10-11T15:27:37.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50ff6ff0204a99f-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.615 DEBUG:boto:path=/dghqvd-59/obj145 2023-10-11T15:27:37.616 DEBUG:boto:auth_path=/dghqvd-59/obj145 2023-10-11T15:27:37.616 DEBUG:boto:Method: GET 2023-10-11T15:27:37.617 DEBUG:boto:Path: /dghqvd-59/obj145 2023-10-11T15:27:37.617 DEBUG:boto:Data: 2023-10-11T15:27:37.618 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.618 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.619 DEBUG:boto:Params: {} 2023-10-11T15:27:37.619 DEBUG:boto:Token: None 2023-10-11T15:27:37.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj145 2023-10-11T15:27:37.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rJ0tjma0SeEcT/wQlB5DBiTXkVU= 2023-10-11T15:27:37.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rJ0tjma0SeEcT/wQlB5DBiTXkVU=', 'Content-Length': '0'} 2023-10-11T15:27:37.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed3b3aa9e248cd4-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.666 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.667 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.667 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.668 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.668 DEBUG:boto:Method: GET 2023-10-11T15:27:37.669 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.669 DEBUG:boto:Data: 2023-10-11T15:27:37.670 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.671 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.671 DEBUG:boto:Params: {} 2023-10-11T15:27:37.671 DEBUG:boto:Token: None 2023-10-11T15:27:37.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002149a797500d5c16-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.676 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.677 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.677 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.678 DEBUG:boto:path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.678 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.679 DEBUG:boto:Method: GET 2023-10-11T15:27:37.679 DEBUG:boto:Path: /dghqvd-59/?prefix=obj145 2023-10-11T15:27:37.680 DEBUG:boto:Data: 2023-10-11T15:27:37.680 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.681 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.681 DEBUG:boto:Params: {} 2023-10-11T15:27:37.682 DEBUG:boto:Token: None 2023-10-11T15:27:37.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001847a3c0ad817358-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.689 DEBUG:boto:b'dghqvd-59obj1451000falseobj1452023-10-11T15:21:59.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:37.690 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.690 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.691 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.691 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.692 DEBUG:boto:Method: GET 2023-10-11T15:27:37.692 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.692 DEBUG:boto:Data: 2023-10-11T15:27:37.693 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.693 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.694 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.694 DEBUG:boto:Params: {} 2023-10-11T15:27:37.695 DEBUG:boto:Token: None 2023-10-11T15:27:37.695 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo= 2023-10-11T15:27:37.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo=', 'Content-Length': '0'} 2023-10-11T15:27:37.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57e5d72605c0613-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.746 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.747 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.747 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.748 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.748 DEBUG:boto:Method: GET 2023-10-11T15:27:37.749 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.749 DEBUG:boto:Data: 2023-10-11T15:27:37.750 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.750 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.751 DEBUG:boto:Params: {} 2023-10-11T15:27:37.751 DEBUG:boto:Token: None 2023-10-11T15:27:37.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo= 2023-10-11T15:27:37.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo=', 'Content-Length': '0'} 2023-10-11T15:27:37.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314a470f41d62ed6-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.756 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.757 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.757 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.757 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.758 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.758 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.759 DEBUG:boto:Data: 2023-10-11T15:27:37.759 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.760 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.760 DEBUG:boto:Params: {} 2023-10-11T15:27:37.761 DEBUG:boto:Token: None 2023-10-11T15:27:37.761 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G4v9Iv1cXkVsIdL6ObKPKNz5zBk= 2023-10-11T15:27:37.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G4v9Iv1cXkVsIdL6ObKPKNz5zBk=', 'Content-Length': '0'} 2023-10-11T15:27:37.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005905fa95caf69345-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.765 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.765 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.766 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.766 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.767 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.767 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.767 DEBUG:boto:Data: 2023-10-11T15:27:37.768 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.769 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.769 DEBUG:boto:Params: {} 2023-10-11T15:27:37.770 DEBUG:boto:Token: None 2023-10-11T15:27:37.770 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G4v9Iv1cXkVsIdL6ObKPKNz5zBk= 2023-10-11T15:27:37.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G4v9Iv1cXkVsIdL6ObKPKNz5zBk=', 'Content-Length': '0'} 2023-10-11T15:27:37.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067922fec3ec950b2-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:37.774 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.774 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.775 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.775 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.776 DEBUG:boto:Method: GET 2023-10-11T15:27:37.776 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.777 DEBUG:boto:Data: 2023-10-11T15:27:37.777 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.778 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.778 DEBUG:boto:Params: {} 2023-10-11T15:27:37.779 DEBUG:boto:Token: None 2023-10-11T15:27:37.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo= 2023-10-11T15:27:37.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo=', 'Content-Length': '0'} 2023-10-11T15:27:37.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d04322de8e560e4-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.783 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.783 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.784 DEBUG:boto:path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.784 DEBUG:boto:auth_path=/dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.785 DEBUG:boto:Method: GET 2023-10-11T15:27:37.785 DEBUG:boto:Path: /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.786 DEBUG:boto:Data: 2023-10-11T15:27:37.786 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.787 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.787 DEBUG:boto:Params: {} 2023-10-11T15:27:37.788 DEBUG:boto:Token: None 2023-10-11T15:27:37.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148?versionId=null 2023-10-11T15:27:37.789 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo= 2023-10-11T15:27:37.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oL+iP78DMwK1cmfNZBtCKHitnDo=', 'Content-Length': '0'} 2023-10-11T15:27:37.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028d6641d8334b636-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.792 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.792 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.793 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.793 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.794 DEBUG:boto:Method: GET 2023-10-11T15:27:37.794 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.795 DEBUG:boto:Data: 2023-10-11T15:27:37.795 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.796 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.796 DEBUG:boto:Params: {} 2023-10-11T15:27:37.797 DEBUG:boto:Token: None 2023-10-11T15:27:37.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b12914453dbe69-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.802 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.802 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.803 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.803 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.804 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.804 DEBUG:boto:Method: GET 2023-10-11T15:27:37.805 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.805 DEBUG:boto:Data: 2023-10-11T15:27:37.805 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.806 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.807 DEBUG:boto:Params: {} 2023-10-11T15:27:37.807 DEBUG:boto:Token: None 2023-10-11T15:27:37.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.808 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca30b4e7deaa62f2-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.858 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.861 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.862 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.862 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.862 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:37.863 DEBUG:boto:Data: 2023-10-11T15:27:37.863 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.864 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.865 DEBUG:boto:Params: {} 2023-10-11T15:27:37.865 DEBUG:boto:Token: None 2023-10-11T15:27:37.866 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148 2023-10-11T15:27:37.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6r5FKe7YE9rvifwglri6o60yYRo= 2023-10-11T15:27:37.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6r5FKe7YE9rvifwglri6o60yYRo=', 'Content-Length': '0'} 2023-10-11T15:27:37.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deec37ab50c84f39-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.869 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.870 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.870 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.871 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:37.871 DEBUG:boto:Data: 2023-10-11T15:27:37.871 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.872 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.873 DEBUG:boto:Params: {} 2023-10-11T15:27:37.873 DEBUG:boto:Token: None 2023-10-11T15:27:37.874 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148 2023-10-11T15:27:37.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6r5FKe7YE9rvifwglri6o60yYRo= 2023-10-11T15:27:37.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6r5FKe7YE9rvifwglri6o60yYRo=', 'Content-Length': '0'} 2023-10-11T15:27:37.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf92ec30ef920ef-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:27:37.878 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.879 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.879 DEBUG:boto:Method: GET 2023-10-11T15:27:37.880 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:37.880 DEBUG:boto:Data: 2023-10-11T15:27:37.880 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.881 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.882 DEBUG:boto:Params: {} 2023-10-11T15:27:37.882 DEBUG:boto:Token: None 2023-10-11T15:27:37.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148 2023-10-11T15:27:37.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ubRGgyGom5HZYrai+0hXXYF5QFA= 2023-10-11T15:27:37.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ubRGgyGom5HZYrai+0hXXYF5QFA=', 'Content-Length': '0'} 2023-10-11T15:27:37.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f30f6e9b2818fb2-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.887 DEBUG:boto:path=/dghqvd-59/obj148 2023-10-11T15:27:37.887 DEBUG:boto:auth_path=/dghqvd-59/obj148 2023-10-11T15:27:37.887 DEBUG:boto:Method: GET 2023-10-11T15:27:37.888 DEBUG:boto:Path: /dghqvd-59/obj148 2023-10-11T15:27:37.888 DEBUG:boto:Data: 2023-10-11T15:27:37.889 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.890 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.890 DEBUG:boto:Params: {} 2023-10-11T15:27:37.891 DEBUG:boto:Token: None 2023-10-11T15:27:37.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj148 2023-10-11T15:27:37.891 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ubRGgyGom5HZYrai+0hXXYF5QFA= 2023-10-11T15:27:37.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ubRGgyGom5HZYrai+0hXXYF5QFA=', 'Content-Length': '0'} 2023-10-11T15:27:37.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097d777b6268d658c-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.895 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.895 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.896 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.896 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.896 DEBUG:boto:Method: GET 2023-10-11T15:27:37.897 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.897 DEBUG:boto:Data: 2023-10-11T15:27:37.898 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.899 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.899 DEBUG:boto:Params: {} 2023-10-11T15:27:37.899 DEBUG:boto:Token: None 2023-10-11T15:27:37.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f048efe647dc14bc-006526bee9-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.904 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.905 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:37.905 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:37.906 DEBUG:boto:path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.906 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.907 DEBUG:boto:Method: GET 2023-10-11T15:27:37.907 DEBUG:boto:Path: /dghqvd-59/?prefix=obj148 2023-10-11T15:27:37.907 DEBUG:boto:Data: 2023-10-11T15:27:37.908 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.909 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.909 DEBUG:boto:Params: {} 2023-10-11T15:27:37.910 DEBUG:boto:Token: None 2023-10-11T15:27:37.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/ 2023-10-11T15:27:37.911 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40= 2023-10-11T15:27:37.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cEEssIRpu7FDF8tp79OuMzGtJ40=', 'Content-Length': '0'} 2023-10-11T15:27:37.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004381cc026eeb4a6a-006526bee9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.916 DEBUG:boto:b'dghqvd-59obj1481000falseobj1482023-10-11T15:25:38.302Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:37.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:37.917 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:37.918 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:37.918 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.919 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.919 DEBUG:boto:Method: GET 2023-10-11T15:27:37.920 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.920 DEBUG:boto:Data: 2023-10-11T15:27:37.921 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.921 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.922 DEBUG:boto:Params: {} 2023-10-11T15:27:37.922 DEBUG:boto:Token: None 2023-10-11T15:27:37.923 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8= 2023-10-11T15:27:37.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8=', 'Content-Length': '0'} 2023-10-11T15:27:37.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7d392d0e74df8b-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.966 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:37.967 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:37.968 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.968 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.968 DEBUG:boto:Method: GET 2023-10-11T15:27:37.969 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.969 DEBUG:boto:Data: 2023-10-11T15:27:37.970 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.971 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.971 DEBUG:boto:Params: {} 2023-10-11T15:27:37.972 DEBUG:boto:Token: None 2023-10-11T15:27:37.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8= 2023-10-11T15:27:37.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8=', 'Content-Length': '0'} 2023-10-11T15:27:37.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136604e04ffc9ff1-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.976 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:37.976 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:37.977 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.977 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.978 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.978 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.979 DEBUG:boto:Data: 2023-10-11T15:27:37.979 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.980 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.980 DEBUG:boto:Params: {} 2023-10-11T15:27:37.981 DEBUG:boto:Token: None 2023-10-11T15:27:37.981 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.982 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4QFmjVSaTtcq4cqgzcM5lTm0KPY= 2023-10-11T15:27:37.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4QFmjVSaTtcq4cqgzcM5lTm0KPY=', 'Content-Length': '0'} 2023-10-11T15:27:37.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f8c506aebc99d9-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.985 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:37.985 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:37.986 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.986 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.987 DEBUG:boto:Method: HEAD 2023-10-11T15:27:37.987 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.987 DEBUG:boto:Data: 2023-10-11T15:27:37.988 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:37.989 DEBUG:boto:Port: 8000 2023-10-11T15:27:37.989 DEBUG:boto:Params: {} 2023-10-11T15:27:37.990 DEBUG:boto:Token: None 2023-10-11T15:27:37.990 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4QFmjVSaTtcq4cqgzcM5lTm0KPY= 2023-10-11T15:27:37.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4QFmjVSaTtcq4cqgzcM5lTm0KPY=', 'Content-Length': '0'} 2023-10-11T15:27:37.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212687684c6e015c-006526bee9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:37.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:37.994 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:37.995 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:37.995 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.995 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.996 DEBUG:boto:Method: GET 2023-10-11T15:27:37.996 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:37.997 DEBUG:boto:Data: 2023-10-11T15:27:37.997 DEBUG:boto:Headers: {} 2023-10-11T15:27:37.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:37.998 DEBUG:boto:Port: 8001 2023-10-11T15:27:37.998 DEBUG:boto:Params: {} 2023-10-11T15:27:37.999 DEBUG:boto:Token: None 2023-10-11T15:27:37.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:37 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:38.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8= 2023-10-11T15:27:38.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:37 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fyHwRyNTLHYFHS4/m5OvdCg+GG8=', 'Content-Length': '0'} 2023-10-11T15:27:38.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071438938ccbf91e6-006526bee9-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.003 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:38.003 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:38.004 DEBUG:boto:path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:38.004 DEBUG:boto:auth_path=/dghqvd-59/obj155?versionId=null 2023-10-11T15:27:38.005 DEBUG:boto:Method: GET 2023-10-11T15:27:38.005 DEBUG:boto:Path: /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:38.006 DEBUG:boto:Data: 2023-10-11T15:27:38.006 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.007 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.007 DEBUG:boto:Params: {} 2023-10-11T15:27:38.008 DEBUG:boto:Token: None 2023-10-11T15:27:38.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj155?versionId=null 2023-10-11T15:27:38.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zRXFalZG1Xx8PpALYV36Tkr9eos= 2023-10-11T15:27:38.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zRXFalZG1Xx8PpALYV36Tkr9eos=', 'Content-Length': '0'} 2023-10-11T15:27:38.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e15fff91493d50-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.054 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.055 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.056 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.056 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.056 DEBUG:boto:Method: GET 2023-10-11T15:27:38.057 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.057 DEBUG:boto:Data: 2023-10-11T15:27:38.058 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.059 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.059 DEBUG:boto:Params: {} 2023-10-11T15:27:38.060 DEBUG:boto:Token: None 2023-10-11T15:27:38.060 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.061 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f2bf50261167207-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.066 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.067 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.067 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.068 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.068 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.068 DEBUG:boto:Method: GET 2023-10-11T15:27:38.069 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.069 DEBUG:boto:Data: 2023-10-11T15:27:38.070 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.071 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.071 DEBUG:boto:Params: {} 2023-10-11T15:27:38.072 DEBUG:boto:Token: None 2023-10-11T15:27:38.072 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.073 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5a72ead8f6ee85-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.078 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.079 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:38.079 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:38.080 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.080 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:38.081 DEBUG:boto:Data: 2023-10-11T15:27:38.081 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.081 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.082 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.082 DEBUG:boto:Params: {} 2023-10-11T15:27:38.083 DEBUG:boto:Token: None 2023-10-11T15:27:38.083 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj155 2023-10-11T15:27:38.084 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2x8FPWzAZ/1Z4bomwrzoSFxennA= 2023-10-11T15:27:38.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2x8FPWzAZ/1Z4bomwrzoSFxennA=', 'Content-Length': '0'} 2023-10-11T15:27:38.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e1a9b23b11452e0-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.087 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:38.087 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:38.088 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.088 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:38.088 DEBUG:boto:Data: 2023-10-11T15:27:38.089 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.090 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.090 DEBUG:boto:Params: {} 2023-10-11T15:27:38.091 DEBUG:boto:Token: None 2023-10-11T15:27:38.091 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj155 2023-10-11T15:27:38.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2x8FPWzAZ/1Z4bomwrzoSFxennA= 2023-10-11T15:27:38.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2x8FPWzAZ/1Z4bomwrzoSFxennA=', 'Content-Length': '0'} 2023-10-11T15:27:38.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1c5b07a2c7c6cc-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:27:38.095 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:38.095 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:38.096 DEBUG:boto:Method: GET 2023-10-11T15:27:38.096 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:38.096 DEBUG:boto:Data: 2023-10-11T15:27:38.097 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.097 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.098 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.098 DEBUG:boto:Params: {} 2023-10-11T15:27:38.099 DEBUG:boto:Token: None 2023-10-11T15:27:38.099 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj155 2023-10-11T15:27:38.100 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mvSlxl5SAqLcnteyPiGdkccSYAU= 2023-10-11T15:27:38.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mvSlxl5SAqLcnteyPiGdkccSYAU=', 'Content-Length': '0'} 2023-10-11T15:27:38.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84f872bb92c0516-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.103 DEBUG:boto:path=/dghqvd-59/obj155 2023-10-11T15:27:38.103 DEBUG:boto:auth_path=/dghqvd-59/obj155 2023-10-11T15:27:38.104 DEBUG:boto:Method: GET 2023-10-11T15:27:38.104 DEBUG:boto:Path: /dghqvd-59/obj155 2023-10-11T15:27:38.105 DEBUG:boto:Data: 2023-10-11T15:27:38.105 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.105 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.106 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.106 DEBUG:boto:Params: {} 2023-10-11T15:27:38.107 DEBUG:boto:Token: None 2023-10-11T15:27:38.107 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj155 2023-10-11T15:27:38.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mvSlxl5SAqLcnteyPiGdkccSYAU= 2023-10-11T15:27:38.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mvSlxl5SAqLcnteyPiGdkccSYAU=', 'Content-Length': '0'} 2023-10-11T15:27:38.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbeb1c25a17b0e5-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.112 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.112 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.112 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.113 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.113 DEBUG:boto:Method: GET 2023-10-11T15:27:38.114 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.114 DEBUG:boto:Data: 2023-10-11T15:27:38.115 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.116 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.116 DEBUG:boto:Params: {} 2023-10-11T15:27:38.117 DEBUG:boto:Token: None 2023-10-11T15:27:38.117 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd0deb75c5864138-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.162 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.163 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.164 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.165 DEBUG:boto:path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.166 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.167 DEBUG:boto:Method: GET 2023-10-11T15:27:38.168 DEBUG:boto:Path: /dghqvd-59/?prefix=obj155 2023-10-11T15:27:38.168 DEBUG:boto:Data: 2023-10-11T15:27:38.169 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.170 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.170 DEBUG:boto:Params: {} 2023-10-11T15:27:38.171 DEBUG:boto:Token: None 2023-10-11T15:27:38.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.172 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cda6a9ba3b0aa132-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.177 DEBUG:boto:b'dghqvd-59obj1551000falseobj1552023-10-11T15:24:15.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:38.179 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.179 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.180 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.180 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.180 DEBUG:boto:Method: GET 2023-10-11T15:27:38.181 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.181 DEBUG:boto:Data: 2023-10-11T15:27:38.182 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.183 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.184 DEBUG:boto:Params: {} 2023-10-11T15:27:38.184 DEBUG:boto:Token: None 2023-10-11T15:27:38.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo= 2023-10-11T15:27:38.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo=', 'Content-Length': '0'} 2023-10-11T15:27:38.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007795928d1b3c46a9-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.190 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.190 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.191 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.191 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.192 DEBUG:boto:Method: GET 2023-10-11T15:27:38.192 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.193 DEBUG:boto:Data: 2023-10-11T15:27:38.193 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.195 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.195 DEBUG:boto:Params: {} 2023-10-11T15:27:38.196 DEBUG:boto:Token: None 2023-10-11T15:27:38.197 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo= 2023-10-11T15:27:38.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo=', 'Content-Length': '0'} 2023-10-11T15:27:38.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17b76333b2fecc0-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.242 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.243 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.243 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.244 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.245 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.245 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.246 DEBUG:boto:Data: 2023-10-11T15:27:38.246 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.247 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.248 DEBUG:boto:Params: {} 2023-10-11T15:27:38.249 DEBUG:boto:Token: None 2023-10-11T15:27:38.249 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.250 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+N13KuAXlMJEn17RLoJAta4i7pk= 2023-10-11T15:27:38.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+N13KuAXlMJEn17RLoJAta4i7pk=', 'Content-Length': '0'} 2023-10-11T15:27:38.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647791f4ba41fd78-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.253 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.254 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.254 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.255 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.255 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.255 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.256 DEBUG:boto:Data: 2023-10-11T15:27:38.256 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.257 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.258 DEBUG:boto:Params: {} 2023-10-11T15:27:38.258 DEBUG:boto:Token: None 2023-10-11T15:27:38.259 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.259 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+N13KuAXlMJEn17RLoJAta4i7pk= 2023-10-11T15:27:38.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+N13KuAXlMJEn17RLoJAta4i7pk=', 'Content-Length': '0'} 2023-10-11T15:27:38.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ec0bc9de4724f9-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:38.263 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.264 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.264 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.265 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.265 DEBUG:boto:Method: GET 2023-10-11T15:27:38.266 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.266 DEBUG:boto:Data: 2023-10-11T15:27:38.267 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.268 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.268 DEBUG:boto:Params: {} 2023-10-11T15:27:38.269 DEBUG:boto:Token: None 2023-10-11T15:27:38.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo= 2023-10-11T15:27:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo=', 'Content-Length': '0'} 2023-10-11T15:27:38.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034fbc609a71ed4e3-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.274 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.274 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.275 DEBUG:boto:path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.275 DEBUG:boto:auth_path=/dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.276 DEBUG:boto:Method: GET 2023-10-11T15:27:38.276 DEBUG:boto:Path: /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.277 DEBUG:boto:Data: 2023-10-11T15:27:38.277 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.278 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.279 DEBUG:boto:Params: {} 2023-10-11T15:27:38.279 DEBUG:boto:Token: None 2023-10-11T15:27:38.280 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17?versionId=null 2023-10-11T15:27:38.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo= 2023-10-11T15:27:38.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/pX1v9809D9Aj3PQby4QxeVsJo=', 'Content-Length': '0'} 2023-10-11T15:27:38.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a8ce1958451763-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.284 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.284 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.285 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.285 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.286 DEBUG:boto:Method: GET 2023-10-11T15:27:38.286 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.287 DEBUG:boto:Data: 2023-10-11T15:27:38.287 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.288 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.288 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.289 DEBUG:boto:Params: {} 2023-10-11T15:27:38.289 DEBUG:boto:Token: None 2023-10-11T15:27:38.290 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.290 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000635834455a401916-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.294 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.295 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.295 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.296 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.296 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.297 DEBUG:boto:Method: GET 2023-10-11T15:27:38.297 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.298 DEBUG:boto:Data: 2023-10-11T15:27:38.298 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.300 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.300 DEBUG:boto:Params: {} 2023-10-11T15:27:38.301 DEBUG:boto:Token: None 2023-10-11T15:27:38.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.302 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a754cf60c66612-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.350 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.351 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.352 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.352 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.353 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:38.353 DEBUG:boto:Data: 2023-10-11T15:27:38.354 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.354 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.355 DEBUG:boto:Params: {} 2023-10-11T15:27:38.355 DEBUG:boto:Token: None 2023-10-11T15:27:38.356 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17 2023-10-11T15:27:38.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SD+JLwCNu8RHTt2Q13EtuPepj/M= 2023-10-11T15:27:38.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SD+JLwCNu8RHTt2Q13EtuPepj/M=', 'Content-Length': '0'} 2023-10-11T15:27:38.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da56de5990e782d1-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.360 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.360 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.360 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.361 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:38.361 DEBUG:boto:Data: 2023-10-11T15:27:38.362 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.363 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.363 DEBUG:boto:Params: {} 2023-10-11T15:27:38.363 DEBUG:boto:Token: None 2023-10-11T15:27:38.364 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17 2023-10-11T15:27:38.364 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SD+JLwCNu8RHTt2Q13EtuPepj/M= 2023-10-11T15:27:38.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SD+JLwCNu8RHTt2Q13EtuPepj/M=', 'Content-Length': '0'} 2023-10-11T15:27:38.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fabca3bf058c4c-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:27:38.368 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.368 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.369 DEBUG:boto:Method: GET 2023-10-11T15:27:38.369 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:38.370 DEBUG:boto:Data: 2023-10-11T15:27:38.370 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.371 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.371 DEBUG:boto:Params: {} 2023-10-11T15:27:38.372 DEBUG:boto:Token: None 2023-10-11T15:27:38.372 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17 2023-10-11T15:27:38.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aTsKyKDGZA1JlhaPmRBi8P4XoN4= 2023-10-11T15:27:38.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aTsKyKDGZA1JlhaPmRBi8P4XoN4=', 'Content-Length': '0'} 2023-10-11T15:27:38.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96494214efabba-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.376 DEBUG:boto:path=/dghqvd-59/obj17 2023-10-11T15:27:38.376 DEBUG:boto:auth_path=/dghqvd-59/obj17 2023-10-11T15:27:38.377 DEBUG:boto:Method: GET 2023-10-11T15:27:38.377 DEBUG:boto:Path: /dghqvd-59/obj17 2023-10-11T15:27:38.377 DEBUG:boto:Data: 2023-10-11T15:27:38.378 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.379 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.379 DEBUG:boto:Params: {} 2023-10-11T15:27:38.380 DEBUG:boto:Token: None 2023-10-11T15:27:38.380 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj17 2023-10-11T15:27:38.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aTsKyKDGZA1JlhaPmRBi8P4XoN4= 2023-10-11T15:27:38.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aTsKyKDGZA1JlhaPmRBi8P4XoN4=', 'Content-Length': '0'} 2023-10-11T15:27:38.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc121e3e42e46b5-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.384 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.384 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.385 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.385 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.385 DEBUG:boto:Method: GET 2023-10-11T15:27:38.386 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.386 DEBUG:boto:Data: 2023-10-11T15:27:38.387 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.387 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.388 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.388 DEBUG:boto:Params: {} 2023-10-11T15:27:38.388 DEBUG:boto:Token: None 2023-10-11T15:27:38.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.389 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f7c463cf1ea982-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.438 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.439 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.440 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.440 DEBUG:boto:path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.440 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.441 DEBUG:boto:Method: GET 2023-10-11T15:27:38.441 DEBUG:boto:Path: /dghqvd-59/?prefix=obj17 2023-10-11T15:27:38.441 DEBUG:boto:Data: 2023-10-11T15:27:38.442 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.442 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.443 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.443 DEBUG:boto:Params: {} 2023-10-11T15:27:38.444 DEBUG:boto:Token: None 2023-10-11T15:27:38.444 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.444 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005817dfc92e6c2db4-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.450 DEBUG:boto:b'dghqvd-59obj171000falseobj172023-10-11T15:16:32.599Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:38.451 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.451 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.452 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.452 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.452 DEBUG:boto:Method: GET 2023-10-11T15:27:38.453 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.453 DEBUG:boto:Data: 2023-10-11T15:27:38.454 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.454 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.455 DEBUG:boto:Params: {} 2023-10-11T15:27:38.455 DEBUG:boto:Token: None 2023-10-11T15:27:38.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA= 2023-10-11T15:27:38.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA=', 'Content-Length': '0'} 2023-10-11T15:27:38.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357902d0616c3bea-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.460 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.460 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.461 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.461 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.462 DEBUG:boto:Method: GET 2023-10-11T15:27:38.462 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.463 DEBUG:boto:Data: 2023-10-11T15:27:38.463 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.464 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.464 DEBUG:boto:Params: {} 2023-10-11T15:27:38.465 DEBUG:boto:Token: None 2023-10-11T15:27:38.465 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.466 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA= 2023-10-11T15:27:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA=', 'Content-Length': '0'} 2023-10-11T15:27:38.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002132e9e62668f3d5-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.514 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.515 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.515 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.516 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.516 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.517 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.517 DEBUG:boto:Data: 2023-10-11T15:27:38.517 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.518 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.519 DEBUG:boto:Params: {} 2023-10-11T15:27:38.519 DEBUG:boto:Token: None 2023-10-11T15:27:38.520 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3tFAOtqk5hYFVivN9hPTQUJvL/0= 2023-10-11T15:27:38.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3tFAOtqk5hYFVivN9hPTQUJvL/0=', 'Content-Length': '0'} 2023-10-11T15:27:38.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba1d2edf3b6eb733-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.524 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.524 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.525 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.525 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.525 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.526 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.526 DEBUG:boto:Data: 2023-10-11T15:27:38.526 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.527 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.528 DEBUG:boto:Params: {} 2023-10-11T15:27:38.528 DEBUG:boto:Token: None 2023-10-11T15:27:38.529 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3tFAOtqk5hYFVivN9hPTQUJvL/0= 2023-10-11T15:27:38.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3tFAOtqk5hYFVivN9hPTQUJvL/0=', 'Content-Length': '0'} 2023-10-11T15:27:38.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cabd6fb58aef8b69-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:38.534 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.534 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.534 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.535 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.535 DEBUG:boto:Method: GET 2023-10-11T15:27:38.536 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.536 DEBUG:boto:Data: 2023-10-11T15:27:38.537 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.537 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.538 DEBUG:boto:Params: {} 2023-10-11T15:27:38.538 DEBUG:boto:Token: None 2023-10-11T15:27:38.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA= 2023-10-11T15:27:38.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA=', 'Content-Length': '0'} 2023-10-11T15:27:38.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c40cff1352f821-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.542 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.543 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.543 DEBUG:boto:path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.543 DEBUG:boto:auth_path=/dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.544 DEBUG:boto:Method: GET 2023-10-11T15:27:38.544 DEBUG:boto:Path: /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.545 DEBUG:boto:Data: 2023-10-11T15:27:38.545 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.546 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.546 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.546 DEBUG:boto:Params: {} 2023-10-11T15:27:38.547 DEBUG:boto:Token: None 2023-10-11T15:27:38.547 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185?versionId=null 2023-10-11T15:27:38.548 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA= 2023-10-11T15:27:38.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jmqYtAT+GNzuJzk5dk3HBSFWxqA=', 'Content-Length': '0'} 2023-10-11T15:27:38.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c87828896fa42b4-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.551 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.551 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.552 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.552 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.552 DEBUG:boto:Method: GET 2023-10-11T15:27:38.553 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.553 DEBUG:boto:Data: 2023-10-11T15:27:38.554 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.555 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.555 DEBUG:boto:Params: {} 2023-10-11T15:27:38.555 DEBUG:boto:Token: None 2023-10-11T15:27:38.556 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008764c69134bfef55-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.560 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.560 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.561 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.561 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.561 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.562 DEBUG:boto:Method: GET 2023-10-11T15:27:38.562 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.563 DEBUG:boto:Data: 2023-10-11T15:27:38.563 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.564 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.564 DEBUG:boto:Params: {} 2023-10-11T15:27:38.565 DEBUG:boto:Token: None 2023-10-11T15:27:38.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d1f00c3685af497-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.571 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.572 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.572 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.572 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.573 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:38.573 DEBUG:boto:Data: 2023-10-11T15:27:38.574 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.574 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.574 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.575 DEBUG:boto:Params: {} 2023-10-11T15:27:38.575 DEBUG:boto:Token: None 2023-10-11T15:27:38.576 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185 2023-10-11T15:27:38.576 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o62QFF9okxEYDrNPTp3JiFKKLKk= 2023-10-11T15:27:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o62QFF9okxEYDrNPTp3JiFKKLKk=', 'Content-Length': '0'} 2023-10-11T15:27:38.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21d7ad4326354e1-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.579 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.580 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.580 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.580 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:38.581 DEBUG:boto:Data: 2023-10-11T15:27:38.581 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.581 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.582 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.582 DEBUG:boto:Params: {} 2023-10-11T15:27:38.583 DEBUG:boto:Token: None 2023-10-11T15:27:38.583 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185 2023-10-11T15:27:38.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o62QFF9okxEYDrNPTp3JiFKKLKk= 2023-10-11T15:27:38.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o62QFF9okxEYDrNPTp3JiFKKLKk=', 'Content-Length': '0'} 2023-10-11T15:27:38.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198ddec8fc039cf6-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:27:38.587 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.587 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.588 DEBUG:boto:Method: GET 2023-10-11T15:27:38.588 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:38.588 DEBUG:boto:Data: 2023-10-11T15:27:38.589 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.590 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.590 DEBUG:boto:Params: {} 2023-10-11T15:27:38.591 DEBUG:boto:Token: None 2023-10-11T15:27:38.591 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185 2023-10-11T15:27:38.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1sDGmBPjISr4nrIz6uXCTJt2Vac= 2023-10-11T15:27:38.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1sDGmBPjISr4nrIz6uXCTJt2Vac=', 'Content-Length': '0'} 2023-10-11T15:27:38.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b32cddc1c081742-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.638 DEBUG:boto:path=/dghqvd-59/obj185 2023-10-11T15:27:38.639 DEBUG:boto:auth_path=/dghqvd-59/obj185 2023-10-11T15:27:38.640 DEBUG:boto:Method: GET 2023-10-11T15:27:38.640 DEBUG:boto:Path: /dghqvd-59/obj185 2023-10-11T15:27:38.640 DEBUG:boto:Data: 2023-10-11T15:27:38.641 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.641 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.642 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.642 DEBUG:boto:Params: {} 2023-10-11T15:27:38.643 DEBUG:boto:Token: None 2023-10-11T15:27:38.643 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj185 2023-10-11T15:27:38.644 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1sDGmBPjISr4nrIz6uXCTJt2Vac= 2023-10-11T15:27:38.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1sDGmBPjISr4nrIz6uXCTJt2Vac=', 'Content-Length': '0'} 2023-10-11T15:27:38.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a87ee411a4a6ed-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.647 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.647 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.648 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.648 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.648 DEBUG:boto:Method: GET 2023-10-11T15:27:38.649 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.649 DEBUG:boto:Data: 2023-10-11T15:27:38.650 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.651 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.651 DEBUG:boto:Params: {} 2023-10-11T15:27:38.651 DEBUG:boto:Token: None 2023-10-11T15:27:38.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a683a3aefed29593-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.656 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.657 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.657 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.657 DEBUG:boto:path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.658 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.658 DEBUG:boto:Method: GET 2023-10-11T15:27:38.659 DEBUG:boto:Path: /dghqvd-59/?prefix=obj185 2023-10-11T15:27:38.659 DEBUG:boto:Data: 2023-10-11T15:27:38.660 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.660 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.661 DEBUG:boto:Params: {} 2023-10-11T15:27:38.661 DEBUG:boto:Token: None 2023-10-11T15:27:38.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd46c9c26fc3f82-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.667 DEBUG:boto:b'dghqvd-59obj1851000falseobj1852023-10-11T15:25:38.362Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:38.668 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.668 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.669 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.669 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.669 DEBUG:boto:Method: GET 2023-10-11T15:27:38.670 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.670 DEBUG:boto:Data: 2023-10-11T15:27:38.671 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.672 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.672 DEBUG:boto:Params: {} 2023-10-11T15:27:38.672 DEBUG:boto:Token: None 2023-10-11T15:27:38.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84= 2023-10-11T15:27:38.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84=', 'Content-Length': '0'} 2023-10-11T15:27:38.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfaca91b4274c63f-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.676 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.677 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.677 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.678 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.678 DEBUG:boto:Method: GET 2023-10-11T15:27:38.679 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.679 DEBUG:boto:Data: 2023-10-11T15:27:38.679 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.680 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.681 DEBUG:boto:Params: {} 2023-10-11T15:27:38.681 DEBUG:boto:Token: None 2023-10-11T15:27:38.682 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.682 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84= 2023-10-11T15:27:38.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84=', 'Content-Length': '0'} 2023-10-11T15:27:38.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d366f8b0cd997a-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.685 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.686 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.686 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.687 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.687 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.687 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.688 DEBUG:boto:Data: 2023-10-11T15:27:38.688 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.689 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.690 DEBUG:boto:Params: {} 2023-10-11T15:27:38.690 DEBUG:boto:Token: None 2023-10-11T15:27:38.691 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.691 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ezVqcG9vMVQHszY4rhQbdsBjHEk= 2023-10-11T15:27:38.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ezVqcG9vMVQHszY4rhQbdsBjHEk=', 'Content-Length': '0'} 2023-10-11T15:27:38.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b100a7a37ffbe282-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.694 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.695 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.695 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.695 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.696 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.696 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.697 DEBUG:boto:Data: 2023-10-11T15:27:38.697 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.697 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.698 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.698 DEBUG:boto:Params: {} 2023-10-11T15:27:38.699 DEBUG:boto:Token: None 2023-10-11T15:27:38.699 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ezVqcG9vMVQHszY4rhQbdsBjHEk= 2023-10-11T15:27:38.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ezVqcG9vMVQHszY4rhQbdsBjHEk=', 'Content-Length': '0'} 2023-10-11T15:27:38.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051093b0e25e495c3-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:38.703 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.703 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.704 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.704 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.705 DEBUG:boto:Method: GET 2023-10-11T15:27:38.705 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.705 DEBUG:boto:Data: 2023-10-11T15:27:38.706 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.706 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.707 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.707 DEBUG:boto:Params: {} 2023-10-11T15:27:38.708 DEBUG:boto:Token: None 2023-10-11T15:27:38.708 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84= 2023-10-11T15:27:38.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84=', 'Content-Length': '0'} 2023-10-11T15:27:38.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40dbd4493d68fc5-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.712 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.712 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.712 DEBUG:boto:path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.713 DEBUG:boto:auth_path=/dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.713 DEBUG:boto:Method: GET 2023-10-11T15:27:38.714 DEBUG:boto:Path: /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.714 DEBUG:boto:Data: 2023-10-11T15:27:38.715 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.715 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.715 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.716 DEBUG:boto:Params: {} 2023-10-11T15:27:38.716 DEBUG:boto:Token: None 2023-10-11T15:27:38.717 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19?versionId=null 2023-10-11T15:27:38.717 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84= 2023-10-11T15:27:38.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wbQSXq1XmA1Z8igZZHKuvRqZm84=', 'Content-Length': '0'} 2023-10-11T15:27:38.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000481bd46895be3044-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.720 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.721 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.721 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.722 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.722 DEBUG:boto:Method: GET 2023-10-11T15:27:38.723 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.723 DEBUG:boto:Data: 2023-10-11T15:27:38.723 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.724 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.725 DEBUG:boto:Params: {} 2023-10-11T15:27:38.725 DEBUG:boto:Token: None 2023-10-11T15:27:38.725 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e54ccf31163db480-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.770 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.771 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.772 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.772 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.772 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.773 DEBUG:boto:Method: GET 2023-10-11T15:27:38.773 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.774 DEBUG:boto:Data: 2023-10-11T15:27:38.774 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.775 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.775 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.775 DEBUG:boto:Params: {} 2023-10-11T15:27:38.776 DEBUG:boto:Token: None 2023-10-11T15:27:38.776 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1680ed5c7af3325-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.782 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.783 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.783 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.783 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.784 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:38.784 DEBUG:boto:Data: 2023-10-11T15:27:38.785 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.785 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.786 DEBUG:boto:Params: {} 2023-10-11T15:27:38.786 DEBUG:boto:Token: None 2023-10-11T15:27:38.787 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19 2023-10-11T15:27:38.787 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WPrVnvP28vk3GL9Y8htHecVuo4U= 2023-10-11T15:27:38.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WPrVnvP28vk3GL9Y8htHecVuo4U=', 'Content-Length': '0'} 2023-10-11T15:27:38.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fccc7d45b75dff8-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.790 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.791 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.791 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.792 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:38.792 DEBUG:boto:Data: 2023-10-11T15:27:38.792 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.793 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.794 DEBUG:boto:Params: {} 2023-10-11T15:27:38.794 DEBUG:boto:Token: None 2023-10-11T15:27:38.794 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19 2023-10-11T15:27:38.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WPrVnvP28vk3GL9Y8htHecVuo4U= 2023-10-11T15:27:38.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WPrVnvP28vk3GL9Y8htHecVuo4U=', 'Content-Length': '0'} 2023-10-11T15:27:38.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146001790481ae86-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:27:38.799 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.799 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.800 DEBUG:boto:Method: GET 2023-10-11T15:27:38.800 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:38.800 DEBUG:boto:Data: 2023-10-11T15:27:38.801 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.802 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.802 DEBUG:boto:Params: {} 2023-10-11T15:27:38.803 DEBUG:boto:Token: None 2023-10-11T15:27:38.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19 2023-10-11T15:27:38.804 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7qAvYgBSiH2e+oYaN4CdK0yuqxE= 2023-10-11T15:27:38.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7qAvYgBSiH2e+oYaN4CdK0yuqxE=', 'Content-Length': '0'} 2023-10-11T15:27:38.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8a8f20274c666a-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.807 DEBUG:boto:path=/dghqvd-59/obj19 2023-10-11T15:27:38.807 DEBUG:boto:auth_path=/dghqvd-59/obj19 2023-10-11T15:27:38.808 DEBUG:boto:Method: GET 2023-10-11T15:27:38.808 DEBUG:boto:Path: /dghqvd-59/obj19 2023-10-11T15:27:38.808 DEBUG:boto:Data: 2023-10-11T15:27:38.809 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.810 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.810 DEBUG:boto:Params: {} 2023-10-11T15:27:38.811 DEBUG:boto:Token: None 2023-10-11T15:27:38.811 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj19 2023-10-11T15:27:38.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7qAvYgBSiH2e+oYaN4CdK0yuqxE= 2023-10-11T15:27:38.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7qAvYgBSiH2e+oYaN4CdK0yuqxE=', 'Content-Length': '0'} 2023-10-11T15:27:38.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492bdcb0989d6e2b-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.854 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.855 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.855 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.856 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.856 DEBUG:boto:Method: GET 2023-10-11T15:27:38.857 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.857 DEBUG:boto:Data: 2023-10-11T15:27:38.858 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.859 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.859 DEBUG:boto:Params: {} 2023-10-11T15:27:38.859 DEBUG:boto:Token: None 2023-10-11T15:27:38.860 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b22e356926b4793-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.864 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.865 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.865 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.866 DEBUG:boto:path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.866 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.867 DEBUG:boto:Method: GET 2023-10-11T15:27:38.867 DEBUG:boto:Path: /dghqvd-59/?prefix=obj19 2023-10-11T15:27:38.868 DEBUG:boto:Data: 2023-10-11T15:27:38.868 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.869 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.869 DEBUG:boto:Params: {} 2023-10-11T15:27:38.870 DEBUG:boto:Token: None 2023-10-11T15:27:38.870 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804fa710d9860b6c-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.876 DEBUG:boto:b'dghqvd-59obj191000falseobj192023-10-11T15:18:55.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:38.877 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.877 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.878 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.878 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.879 DEBUG:boto:Method: GET 2023-10-11T15:27:38.879 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.879 DEBUG:boto:Data: 2023-10-11T15:27:38.880 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.881 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.881 DEBUG:boto:Params: {} 2023-10-11T15:27:38.882 DEBUG:boto:Token: None 2023-10-11T15:27:38.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic= 2023-10-11T15:27:38.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic=', 'Content-Length': '0'} 2023-10-11T15:27:38.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001252bccea28893a2-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.886 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.886 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.887 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.887 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.888 DEBUG:boto:Method: GET 2023-10-11T15:27:38.888 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.888 DEBUG:boto:Data: 2023-10-11T15:27:38.889 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.890 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.890 DEBUG:boto:Params: {} 2023-10-11T15:27:38.891 DEBUG:boto:Token: None 2023-10-11T15:27:38.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic= 2023-10-11T15:27:38.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic=', 'Content-Length': '0'} 2023-10-11T15:27:38.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a191a10e3989855-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.895 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.895 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.896 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.896 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.896 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.897 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.897 DEBUG:boto:Data: 2023-10-11T15:27:38.898 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.898 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.899 DEBUG:boto:Params: {} 2023-10-11T15:27:38.899 DEBUG:boto:Token: None 2023-10-11T15:27:38.900 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:udWga2RT61BEE1bAuPmU0JQmcnk= 2023-10-11T15:27:38.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:udWga2RT61BEE1bAuPmU0JQmcnk=', 'Content-Length': '0'} 2023-10-11T15:27:38.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c4cdbce02b45f69-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.903 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.904 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.904 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.904 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.905 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.905 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.906 DEBUG:boto:Data: 2023-10-11T15:27:38.906 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.907 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.907 DEBUG:boto:Params: {} 2023-10-11T15:27:38.908 DEBUG:boto:Token: None 2023-10-11T15:27:38.908 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:udWga2RT61BEE1bAuPmU0JQmcnk= 2023-10-11T15:27:38.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:udWga2RT61BEE1bAuPmU0JQmcnk=', 'Content-Length': '0'} 2023-10-11T15:27:38.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705056b41398e77a-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:38.912 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.912 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.912 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.913 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.913 DEBUG:boto:Method: GET 2023-10-11T15:27:38.914 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.914 DEBUG:boto:Data: 2023-10-11T15:27:38.915 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.915 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.916 DEBUG:boto:Params: {} 2023-10-11T15:27:38.916 DEBUG:boto:Token: None 2023-10-11T15:27:38.917 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.917 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic= 2023-10-11T15:27:38.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic=', 'Content-Length': '0'} 2023-10-11T15:27:38.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1bbc3f515c5b1e-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.962 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.963 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.963 DEBUG:boto:path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.964 DEBUG:boto:auth_path=/dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.964 DEBUG:boto:Method: GET 2023-10-11T15:27:38.965 DEBUG:boto:Path: /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.965 DEBUG:boto:Data: 2023-10-11T15:27:38.965 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.966 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.967 DEBUG:boto:Params: {} 2023-10-11T15:27:38.967 DEBUG:boto:Token: None 2023-10-11T15:27:38.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22?versionId=null 2023-10-11T15:27:38.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic= 2023-10-11T15:27:38.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pKArlsCK38j/HqGuptL/rFDyvic=', 'Content-Length': '0'} 2023-10-11T15:27:38.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2727b1e8a1ae622-006526beea-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.972 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.972 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.973 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.973 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.973 DEBUG:boto:Method: GET 2023-10-11T15:27:38.974 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.974 DEBUG:boto:Data: 2023-10-11T15:27:38.975 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.976 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.976 DEBUG:boto:Params: {} 2023-10-11T15:27:38.977 DEBUG:boto:Token: None 2023-10-11T15:27:38.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.978 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d3df7830993f149-006526beea-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.981 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.982 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:38.983 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:38.983 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.983 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.984 DEBUG:boto:Method: GET 2023-10-11T15:27:38.984 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:38.985 DEBUG:boto:Data: 2023-10-11T15:27:38.985 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.986 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:38.986 DEBUG:boto:Port: 8000 2023-10-11T15:27:38.986 DEBUG:boto:Params: {} 2023-10-11T15:27:38.987 DEBUG:boto:Token: None 2023-10-11T15:27:38.987 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/ 2023-10-11T15:27:38.988 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE= 2023-10-11T15:27:38.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eipPRmB1Zxhk/ZKwKwO2BuT/pSE=', 'Content-Length': '0'} 2023-10-11T15:27:38.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178d754331e5c9d2-006526beea-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:38.994 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:38.994 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:38.995 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:38.995 DEBUG:boto:Method: HEAD 2023-10-11T15:27:38.996 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:38.996 DEBUG:boto:Data: 2023-10-11T15:27:38.996 DEBUG:boto:Headers: {} 2023-10-11T15:27:38.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:38.997 DEBUG:boto:Port: 8001 2023-10-11T15:27:38.998 DEBUG:boto:Params: {} 2023-10-11T15:27:38.998 DEBUG:boto:Token: None 2023-10-11T15:27:38.999 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:38 GMT /dghqvd-59/obj22 2023-10-11T15:27:38.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2EaND9D1fq6YTrb5IVMdHs+vlgg= 2023-10-11T15:27:38.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2EaND9D1fq6YTrb5IVMdHs+vlgg=', 'Content-Length': '0'} 2023-10-11T15:27:39.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e25a71b5b792de-006526beea-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.002 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:39.003 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:39.003 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.004 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:39.004 DEBUG:boto:Data: 2023-10-11T15:27:39.004 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.005 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.006 DEBUG:boto:Params: {} 2023-10-11T15:27:39.006 DEBUG:boto:Token: None 2023-10-11T15:27:39.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj22 2023-10-11T15:27:39.007 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:g70+H2WkpiEujOxr1G18AQte7Ss= 2023-10-11T15:27:39.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:g70+H2WkpiEujOxr1G18AQte7Ss=', 'Content-Length': '0'} 2023-10-11T15:27:39.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd3fa3f18563bb2-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:27:39.011 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:39.011 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:39.011 DEBUG:boto:Method: GET 2023-10-11T15:27:39.012 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:39.012 DEBUG:boto:Data: 2023-10-11T15:27:39.013 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.013 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.014 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.014 DEBUG:boto:Params: {} 2023-10-11T15:27:39.014 DEBUG:boto:Token: None 2023-10-11T15:27:39.015 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj22 2023-10-11T15:27:39.015 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsmhcEYH2UgdSBeavFEawCbkfbk= 2023-10-11T15:27:39.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsmhcEYH2UgdSBeavFEawCbkfbk=', 'Content-Length': '0'} 2023-10-11T15:27:39.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f9b3f977496908-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.018 DEBUG:boto:path=/dghqvd-59/obj22 2023-10-11T15:27:39.019 DEBUG:boto:auth_path=/dghqvd-59/obj22 2023-10-11T15:27:39.019 DEBUG:boto:Method: GET 2023-10-11T15:27:39.020 DEBUG:boto:Path: /dghqvd-59/obj22 2023-10-11T15:27:39.020 DEBUG:boto:Data: 2023-10-11T15:27:39.021 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.021 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.022 DEBUG:boto:Params: {} 2023-10-11T15:27:39.022 DEBUG:boto:Token: None 2023-10-11T15:27:39.023 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj22 2023-10-11T15:27:39.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zsmhcEYH2UgdSBeavFEawCbkfbk= 2023-10-11T15:27:39.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zsmhcEYH2UgdSBeavFEawCbkfbk=', 'Content-Length': '0'} 2023-10-11T15:27:39.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba85c26e5d358c3-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.026 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.027 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.027 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.028 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.028 DEBUG:boto:Method: GET 2023-10-11T15:27:39.029 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.029 DEBUG:boto:Data: 2023-10-11T15:27:39.029 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.030 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.031 DEBUG:boto:Params: {} 2023-10-11T15:27:39.031 DEBUG:boto:Token: None 2023-10-11T15:27:39.032 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ee6ec5a99b551d7-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.036 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.036 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.037 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.037 DEBUG:boto:path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.038 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.038 DEBUG:boto:Method: GET 2023-10-11T15:27:39.039 DEBUG:boto:Path: /dghqvd-59/?prefix=obj22 2023-10-11T15:27:39.039 DEBUG:boto:Data: 2023-10-11T15:27:39.039 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.040 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.041 DEBUG:boto:Params: {} 2023-10-11T15:27:39.041 DEBUG:boto:Token: None 2023-10-11T15:27:39.042 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd090bc1c5a581c3-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.047 DEBUG:boto:b'dghqvd-59obj221000falseobj222023-10-11T15:16:05.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:39.048 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.049 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.049 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.050 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.050 DEBUG:boto:Method: GET 2023-10-11T15:27:39.051 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.051 DEBUG:boto:Data: 2023-10-11T15:27:39.052 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.052 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.053 DEBUG:boto:Params: {} 2023-10-11T15:27:39.053 DEBUG:boto:Token: None 2023-10-11T15:27:39.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc= 2023-10-11T15:27:39.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc=', 'Content-Length': '0'} 2023-10-11T15:27:39.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18e4b5863b37885-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.058 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.058 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.059 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.059 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.059 DEBUG:boto:Method: GET 2023-10-11T15:27:39.060 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.060 DEBUG:boto:Data: 2023-10-11T15:27:39.061 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.062 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.062 DEBUG:boto:Params: {} 2023-10-11T15:27:39.063 DEBUG:boto:Token: None 2023-10-11T15:27:39.063 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc= 2023-10-11T15:27:39.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc=', 'Content-Length': '0'} 2023-10-11T15:27:39.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb8d57f80b1cdcb-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.114 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.115 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.115 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.116 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.116 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.117 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.117 DEBUG:boto:Data: 2023-10-11T15:27:39.117 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.118 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.119 DEBUG:boto:Params: {} 2023-10-11T15:27:39.119 DEBUG:boto:Token: None 2023-10-11T15:27:39.120 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aI5vZfo/F59itEdBuWw5EXuLNmA= 2023-10-11T15:27:39.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aI5vZfo/F59itEdBuWw5EXuLNmA=', 'Content-Length': '0'} 2023-10-11T15:27:39.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9d5f002b1abb5ad-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.123 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.123 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.124 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.124 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.124 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.125 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.125 DEBUG:boto:Data: 2023-10-11T15:27:39.126 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.127 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.127 DEBUG:boto:Params: {} 2023-10-11T15:27:39.127 DEBUG:boto:Token: None 2023-10-11T15:27:39.128 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aI5vZfo/F59itEdBuWw5EXuLNmA= 2023-10-11T15:27:39.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aI5vZfo/F59itEdBuWw5EXuLNmA=', 'Content-Length': '0'} 2023-10-11T15:27:39.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f1cfb233323296-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:39.132 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.132 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.132 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.133 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.133 DEBUG:boto:Method: GET 2023-10-11T15:27:39.134 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.134 DEBUG:boto:Data: 2023-10-11T15:27:39.135 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.135 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.136 DEBUG:boto:Params: {} 2023-10-11T15:27:39.136 DEBUG:boto:Token: None 2023-10-11T15:27:39.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc= 2023-10-11T15:27:39.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc=', 'Content-Length': '0'} 2023-10-11T15:27:39.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590af4ded87a4764-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.182 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.183 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.183 DEBUG:boto:path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.184 DEBUG:boto:auth_path=/dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.184 DEBUG:boto:Method: GET 2023-10-11T15:27:39.185 DEBUG:boto:Path: /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.185 DEBUG:boto:Data: 2023-10-11T15:27:39.185 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.186 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.186 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.187 DEBUG:boto:Params: {} 2023-10-11T15:27:39.187 DEBUG:boto:Token: None 2023-10-11T15:27:39.188 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23?versionId=null 2023-10-11T15:27:39.188 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc= 2023-10-11T15:27:39.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IErTE4I9B472fLDy84xQhp/1ePc=', 'Content-Length': '0'} 2023-10-11T15:27:39.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b5b421a1b37160f-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.191 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.192 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.192 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.193 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.193 DEBUG:boto:Method: GET 2023-10-11T15:27:39.194 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.194 DEBUG:boto:Data: 2023-10-11T15:27:39.195 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.195 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.196 DEBUG:boto:Params: {} 2023-10-11T15:27:39.196 DEBUG:boto:Token: None 2023-10-11T15:27:39.197 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce71226d915f60a-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.201 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.202 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.202 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.203 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.203 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.203 DEBUG:boto:Method: GET 2023-10-11T15:27:39.204 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.204 DEBUG:boto:Data: 2023-10-11T15:27:39.204 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.205 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.205 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.206 DEBUG:boto:Params: {} 2023-10-11T15:27:39.206 DEBUG:boto:Token: None 2023-10-11T15:27:39.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4eb0f0274dbbf3-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.213 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.213 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.214 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.214 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.215 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:39.215 DEBUG:boto:Data: 2023-10-11T15:27:39.215 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.216 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.216 DEBUG:boto:Params: {} 2023-10-11T15:27:39.217 DEBUG:boto:Token: None 2023-10-11T15:27:39.217 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23 2023-10-11T15:27:39.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4opYJ6QavILBottqg/pKpcj8JM8= 2023-10-11T15:27:39.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4opYJ6QavILBottqg/pKpcj8JM8=', 'Content-Length': '0'} 2023-10-11T15:27:39.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c275080218f131d-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.221 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.221 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.222 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.222 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:39.223 DEBUG:boto:Data: 2023-10-11T15:27:39.223 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.223 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.224 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.224 DEBUG:boto:Params: {} 2023-10-11T15:27:39.225 DEBUG:boto:Token: None 2023-10-11T15:27:39.225 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23 2023-10-11T15:27:39.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4opYJ6QavILBottqg/pKpcj8JM8= 2023-10-11T15:27:39.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4opYJ6QavILBottqg/pKpcj8JM8=', 'Content-Length': '0'} 2023-10-11T15:27:39.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686ae308e114eef3-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:27:39.229 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.230 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.230 DEBUG:boto:Method: GET 2023-10-11T15:27:39.231 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:39.231 DEBUG:boto:Data: 2023-10-11T15:27:39.231 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.232 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.233 DEBUG:boto:Params: {} 2023-10-11T15:27:39.233 DEBUG:boto:Token: None 2023-10-11T15:27:39.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23 2023-10-11T15:27:39.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TTDEwagJCthjAYZzJ59DkhFNQvg= 2023-10-11T15:27:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TTDEwagJCthjAYZzJ59DkhFNQvg=', 'Content-Length': '0'} 2023-10-11T15:27:39.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ff8df58183d48b7-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.237 DEBUG:boto:path=/dghqvd-59/obj23 2023-10-11T15:27:39.238 DEBUG:boto:auth_path=/dghqvd-59/obj23 2023-10-11T15:27:39.238 DEBUG:boto:Method: GET 2023-10-11T15:27:39.239 DEBUG:boto:Path: /dghqvd-59/obj23 2023-10-11T15:27:39.239 DEBUG:boto:Data: 2023-10-11T15:27:39.239 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.240 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.241 DEBUG:boto:Params: {} 2023-10-11T15:27:39.241 DEBUG:boto:Token: None 2023-10-11T15:27:39.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj23 2023-10-11T15:27:39.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TTDEwagJCthjAYZzJ59DkhFNQvg= 2023-10-11T15:27:39.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TTDEwagJCthjAYZzJ59DkhFNQvg=', 'Content-Length': '0'} 2023-10-11T15:27:39.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e9f4c483e163c0d-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.286 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.287 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.287 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.288 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.288 DEBUG:boto:Method: GET 2023-10-11T15:27:39.289 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.289 DEBUG:boto:Data: 2023-10-11T15:27:39.290 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.290 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.291 DEBUG:boto:Params: {} 2023-10-11T15:27:39.293 DEBUG:boto:Token: None 2023-10-11T15:27:39.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7cb9886effd002-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.298 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.299 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.299 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.300 DEBUG:boto:path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.300 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.301 DEBUG:boto:Method: GET 2023-10-11T15:27:39.301 DEBUG:boto:Path: /dghqvd-59/?prefix=obj23 2023-10-11T15:27:39.302 DEBUG:boto:Data: 2023-10-11T15:27:39.302 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.303 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.303 DEBUG:boto:Params: {} 2023-10-11T15:27:39.304 DEBUG:boto:Token: None 2023-10-11T15:27:39.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ad7aece20cc9060-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.310 DEBUG:boto:b'dghqvd-59obj231000falseobj232023-10-11T15:21:14.324Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:39.311 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.311 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.312 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.312 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.313 DEBUG:boto:Method: GET 2023-10-11T15:27:39.313 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.314 DEBUG:boto:Data: 2023-10-11T15:27:39.314 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.315 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.315 DEBUG:boto:Params: {} 2023-10-11T15:27:39.316 DEBUG:boto:Token: None 2023-10-11T15:27:39.316 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA= 2023-10-11T15:27:39.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376d09a9517743a5-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.320 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.321 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.321 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.322 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.322 DEBUG:boto:Method: GET 2023-10-11T15:27:39.322 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.323 DEBUG:boto:Data: 2023-10-11T15:27:39.323 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.324 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.325 DEBUG:boto:Params: {} 2023-10-11T15:27:39.325 DEBUG:boto:Token: None 2023-10-11T15:27:39.326 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA= 2023-10-11T15:27:39.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0b2315442deb2b-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.330 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.330 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.331 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.331 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.331 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.332 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.332 DEBUG:boto:Data: 2023-10-11T15:27:39.333 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.333 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.333 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.334 DEBUG:boto:Params: {} 2023-10-11T15:27:39.334 DEBUG:boto:Token: None 2023-10-11T15:27:39.335 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jkOsmQcJ7+s48VdRAFRoAWOjJ8M= 2023-10-11T15:27:39.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jkOsmQcJ7+s48VdRAFRoAWOjJ8M=', 'Content-Length': '0'} 2023-10-11T15:27:39.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb0cbabb956b16e-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.338 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.339 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.339 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.339 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.340 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.340 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.341 DEBUG:boto:Data: 2023-10-11T15:27:39.341 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.342 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.342 DEBUG:boto:Params: {} 2023-10-11T15:27:39.343 DEBUG:boto:Token: None 2023-10-11T15:27:39.343 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jkOsmQcJ7+s48VdRAFRoAWOjJ8M= 2023-10-11T15:27:39.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jkOsmQcJ7+s48VdRAFRoAWOjJ8M=', 'Content-Length': '0'} 2023-10-11T15:27:39.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8e8cee93562ac0-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:39.347 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.348 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.348 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.349 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.349 DEBUG:boto:Method: GET 2023-10-11T15:27:39.349 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.350 DEBUG:boto:Data: 2023-10-11T15:27:39.350 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.351 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.351 DEBUG:boto:Params: {} 2023-10-11T15:27:39.352 DEBUG:boto:Token: None 2023-10-11T15:27:39.352 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA= 2023-10-11T15:27:39.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293c0fc8a770d70c-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.356 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.356 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.357 DEBUG:boto:path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.357 DEBUG:boto:auth_path=/dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.358 DEBUG:boto:Method: GET 2023-10-11T15:27:39.358 DEBUG:boto:Path: /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.359 DEBUG:boto:Data: 2023-10-11T15:27:39.359 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.360 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.360 DEBUG:boto:Params: {} 2023-10-11T15:27:39.361 DEBUG:boto:Token: None 2023-10-11T15:27:39.361 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29?versionId=null 2023-10-11T15:27:39.362 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA= 2023-10-11T15:27:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZxL3mPyhxlwV71bzB5nTGw4feHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5b07b96746e61a-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.406 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.407 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.408 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.408 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.408 DEBUG:boto:Method: GET 2023-10-11T15:27:39.409 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.409 DEBUG:boto:Data: 2023-10-11T15:27:39.410 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.411 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.411 DEBUG:boto:Params: {} 2023-10-11T15:27:39.411 DEBUG:boto:Token: None 2023-10-11T15:27:39.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003560de807fdcccb1-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.416 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.417 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.417 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.418 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.418 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.419 DEBUG:boto:Method: GET 2023-10-11T15:27:39.419 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.420 DEBUG:boto:Data: 2023-10-11T15:27:39.420 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.421 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.421 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.421 DEBUG:boto:Params: {} 2023-10-11T15:27:39.422 DEBUG:boto:Token: None 2023-10-11T15:27:39.423 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb540ffb2a7d1e8a-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.429 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.429 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.430 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.430 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.431 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:39.431 DEBUG:boto:Data: 2023-10-11T15:27:39.432 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.432 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.432 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.433 DEBUG:boto:Params: {} 2023-10-11T15:27:39.433 DEBUG:boto:Token: None 2023-10-11T15:27:39.434 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29 2023-10-11T15:27:39.434 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tPo+PqOX/bsfMrpVLD7QaSUdBtQ= 2023-10-11T15:27:39.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tPo+PqOX/bsfMrpVLD7QaSUdBtQ=', 'Content-Length': '0'} 2023-10-11T15:27:39.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f840956ed098da-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.438 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.438 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.439 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.439 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:39.440 DEBUG:boto:Data: 2023-10-11T15:27:39.440 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.440 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.441 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.441 DEBUG:boto:Params: {} 2023-10-11T15:27:39.442 DEBUG:boto:Token: None 2023-10-11T15:27:39.442 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29 2023-10-11T15:27:39.443 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tPo+PqOX/bsfMrpVLD7QaSUdBtQ= 2023-10-11T15:27:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tPo+PqOX/bsfMrpVLD7QaSUdBtQ=', 'Content-Length': '0'} 2023-10-11T15:27:39.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b6cfcaa92f662d-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:27:39.446 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.447 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.447 DEBUG:boto:Method: GET 2023-10-11T15:27:39.447 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:39.448 DEBUG:boto:Data: 2023-10-11T15:27:39.448 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.450 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.451 DEBUG:boto:Params: {} 2023-10-11T15:27:39.452 DEBUG:boto:Token: None 2023-10-11T15:27:39.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29 2023-10-11T15:27:39.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ipt3yl8R+n4uTGIRjRph3SAYAHA= 2023-10-11T15:27:39.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ipt3yl8R+n4uTGIRjRph3SAYAHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000718d6c9f31787fad-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.458 DEBUG:boto:path=/dghqvd-59/obj29 2023-10-11T15:27:39.459 DEBUG:boto:auth_path=/dghqvd-59/obj29 2023-10-11T15:27:39.459 DEBUG:boto:Method: GET 2023-10-11T15:27:39.460 DEBUG:boto:Path: /dghqvd-59/obj29 2023-10-11T15:27:39.461 DEBUG:boto:Data: 2023-10-11T15:27:39.462 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.463 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.464 DEBUG:boto:Params: {} 2023-10-11T15:27:39.465 DEBUG:boto:Token: None 2023-10-11T15:27:39.466 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj29 2023-10-11T15:27:39.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ipt3yl8R+n4uTGIRjRph3SAYAHA= 2023-10-11T15:27:39.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ipt3yl8R+n4uTGIRjRph3SAYAHA=', 'Content-Length': '0'} 2023-10-11T15:27:39.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4b7f88c4170ddc-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.471 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.472 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.473 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.473 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.474 DEBUG:boto:Method: GET 2023-10-11T15:27:39.475 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.475 DEBUG:boto:Data: 2023-10-11T15:27:39.476 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.477 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.478 DEBUG:boto:Params: {} 2023-10-11T15:27:39.478 DEBUG:boto:Token: None 2023-10-11T15:27:39.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029360b8f918a738c-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.485 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.486 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.487 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.487 DEBUG:boto:path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.488 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.488 DEBUG:boto:Method: GET 2023-10-11T15:27:39.489 DEBUG:boto:Path: /dghqvd-59/?prefix=obj29 2023-10-11T15:27:39.490 DEBUG:boto:Data: 2023-10-11T15:27:39.491 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.492 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.492 DEBUG:boto:Params: {} 2023-10-11T15:27:39.493 DEBUG:boto:Token: None 2023-10-11T15:27:39.493 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.494 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002e718351831ab3-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.543 DEBUG:boto:b'dghqvd-59obj291000falseobj292023-10-11T15:21:59.330Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:39.544 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.545 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.545 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.546 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.546 DEBUG:boto:Method: GET 2023-10-11T15:27:39.546 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.547 DEBUG:boto:Data: 2023-10-11T15:27:39.547 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.548 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.549 DEBUG:boto:Params: {} 2023-10-11T15:27:39.549 DEBUG:boto:Token: None 2023-10-11T15:27:39.550 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.550 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw= 2023-10-11T15:27:39.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw=', 'Content-Length': '0'} 2023-10-11T15:27:39.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e3d2bf44b2e9be8-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.554 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.554 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.555 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.555 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.555 DEBUG:boto:Method: GET 2023-10-11T15:27:39.556 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.556 DEBUG:boto:Data: 2023-10-11T15:27:39.557 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.558 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.558 DEBUG:boto:Params: {} 2023-10-11T15:27:39.559 DEBUG:boto:Token: None 2023-10-11T15:27:39.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw= 2023-10-11T15:27:39.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw=', 'Content-Length': '0'} 2023-10-11T15:27:39.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3657bf74a0235c-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.563 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.564 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.564 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.564 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.565 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.565 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.566 DEBUG:boto:Data: 2023-10-11T15:27:39.566 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.567 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.567 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.567 DEBUG:boto:Params: {} 2023-10-11T15:27:39.568 DEBUG:boto:Token: None 2023-10-11T15:27:39.569 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.569 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/DRkoqbPcCIayiwmwTQ9cPWOsUk= 2023-10-11T15:27:39.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/DRkoqbPcCIayiwmwTQ9cPWOsUk=', 'Content-Length': '0'} 2023-10-11T15:27:39.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdb8c5690472357-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.573 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.573 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.574 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.574 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.575 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.575 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.575 DEBUG:boto:Data: 2023-10-11T15:27:39.576 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.576 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.577 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.577 DEBUG:boto:Params: {} 2023-10-11T15:27:39.578 DEBUG:boto:Token: None 2023-10-11T15:27:39.578 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.579 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/DRkoqbPcCIayiwmwTQ9cPWOsUk= 2023-10-11T15:27:39.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/DRkoqbPcCIayiwmwTQ9cPWOsUk=', 'Content-Length': '0'} 2023-10-11T15:27:39.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb09e7c962a44d6-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:39.582 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.583 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.583 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.583 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.584 DEBUG:boto:Method: GET 2023-10-11T15:27:39.584 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.585 DEBUG:boto:Data: 2023-10-11T15:27:39.585 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.586 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.587 DEBUG:boto:Params: {} 2023-10-11T15:27:39.587 DEBUG:boto:Token: None 2023-10-11T15:27:39.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw= 2023-10-11T15:27:39.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw=', 'Content-Length': '0'} 2023-10-11T15:27:39.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d0b8d134ec2996-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.591 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.592 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.592 DEBUG:boto:path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.593 DEBUG:boto:auth_path=/dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.593 DEBUG:boto:Method: GET 2023-10-11T15:27:39.593 DEBUG:boto:Path: /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.594 DEBUG:boto:Data: 2023-10-11T15:27:39.594 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.595 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.596 DEBUG:boto:Params: {} 2023-10-11T15:27:39.596 DEBUG:boto:Token: None 2023-10-11T15:27:39.597 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31?versionId=null 2023-10-11T15:27:39.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw= 2023-10-11T15:27:39.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pgb7ww342Hnua0sgCJpmEnrJwRw=', 'Content-Length': '0'} 2023-10-11T15:27:39.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010df9f0b4f072ec8-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.600 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.601 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.601 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.602 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.602 DEBUG:boto:Method: GET 2023-10-11T15:27:39.603 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.603 DEBUG:boto:Data: 2023-10-11T15:27:39.603 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.604 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.605 DEBUG:boto:Params: {} 2023-10-11T15:27:39.605 DEBUG:boto:Token: None 2023-10-11T15:27:39.606 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdc4676adbb475c5-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.654 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.655 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.656 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.656 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.656 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.657 DEBUG:boto:Method: GET 2023-10-11T15:27:39.657 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.658 DEBUG:boto:Data: 2023-10-11T15:27:39.658 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.659 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.659 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.659 DEBUG:boto:Params: {} 2023-10-11T15:27:39.660 DEBUG:boto:Token: None 2023-10-11T15:27:39.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb06bbc02208e63d-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.667 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.667 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.668 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.668 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.669 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:39.669 DEBUG:boto:Data: 2023-10-11T15:27:39.670 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.671 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.671 DEBUG:boto:Params: {} 2023-10-11T15:27:39.672 DEBUG:boto:Token: None 2023-10-11T15:27:39.672 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31 2023-10-11T15:27:39.672 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yjjgJDLvykffdAxOS8FuiAG0ug= 2023-10-11T15:27:39.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yjjgJDLvykffdAxOS8FuiAG0ug=', 'Content-Length': '0'} 2023-10-11T15:27:39.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf4035f7c45a71c-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.676 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.676 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.677 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.677 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:39.678 DEBUG:boto:Data: 2023-10-11T15:27:39.678 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.679 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.679 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.679 DEBUG:boto:Params: {} 2023-10-11T15:27:39.680 DEBUG:boto:Token: None 2023-10-11T15:27:39.680 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31 2023-10-11T15:27:39.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yjjgJDLvykffdAxOS8FuiAG0ug= 2023-10-11T15:27:39.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yjjgJDLvykffdAxOS8FuiAG0ug=', 'Content-Length': '0'} 2023-10-11T15:27:39.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000019381e6f01d46b-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:27:39.684 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.685 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.685 DEBUG:boto:Method: GET 2023-10-11T15:27:39.685 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:39.686 DEBUG:boto:Data: 2023-10-11T15:27:39.686 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.687 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.688 DEBUG:boto:Params: {} 2023-10-11T15:27:39.688 DEBUG:boto:Token: None 2023-10-11T15:27:39.689 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31 2023-10-11T15:27:39.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xJdcS1TwkGJ31ty3zkZZUtVFPs= 2023-10-11T15:27:39.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xJdcS1TwkGJ31ty3zkZZUtVFPs=', 'Content-Length': '0'} 2023-10-11T15:27:39.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d529732bd68fa2-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.692 DEBUG:boto:path=/dghqvd-59/obj31 2023-10-11T15:27:39.693 DEBUG:boto:auth_path=/dghqvd-59/obj31 2023-10-11T15:27:39.693 DEBUG:boto:Method: GET 2023-10-11T15:27:39.694 DEBUG:boto:Path: /dghqvd-59/obj31 2023-10-11T15:27:39.694 DEBUG:boto:Data: 2023-10-11T15:27:39.695 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.695 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.696 DEBUG:boto:Params: {} 2023-10-11T15:27:39.696 DEBUG:boto:Token: None 2023-10-11T15:27:39.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj31 2023-10-11T15:27:39.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/xJdcS1TwkGJ31ty3zkZZUtVFPs= 2023-10-11T15:27:39.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/xJdcS1TwkGJ31ty3zkZZUtVFPs=', 'Content-Length': '0'} 2023-10-11T15:27:39.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25b2c5e3612b7e6-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.742 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.743 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.743 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.744 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.744 DEBUG:boto:Method: GET 2023-10-11T15:27:39.745 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.745 DEBUG:boto:Data: 2023-10-11T15:27:39.746 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.746 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.747 DEBUG:boto:Params: {} 2023-10-11T15:27:39.747 DEBUG:boto:Token: None 2023-10-11T15:27:39.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc50ad6b7813960c-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.752 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.753 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.753 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.754 DEBUG:boto:path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.754 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.755 DEBUG:boto:Method: GET 2023-10-11T15:27:39.755 DEBUG:boto:Path: /dghqvd-59/?prefix=obj31 2023-10-11T15:27:39.756 DEBUG:boto:Data: 2023-10-11T15:27:39.756 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.757 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.757 DEBUG:boto:Params: {} 2023-10-11T15:27:39.758 DEBUG:boto:Token: None 2023-10-11T15:27:39.758 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.759 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d70f5ff9089de52-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.764 DEBUG:boto:b'dghqvd-59obj311000falseobj312023-10-11T15:24:15.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:39.765 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.765 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.766 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.766 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.767 DEBUG:boto:Method: GET 2023-10-11T15:27:39.767 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.767 DEBUG:boto:Data: 2023-10-11T15:27:39.768 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.769 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.769 DEBUG:boto:Params: {} 2023-10-11T15:27:39.770 DEBUG:boto:Token: None 2023-10-11T15:27:39.770 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE= 2023-10-11T15:27:39.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE=', 'Content-Length': '0'} 2023-10-11T15:27:39.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002768b638f9848332-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.814 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.815 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.815 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.816 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.816 DEBUG:boto:Method: GET 2023-10-11T15:27:39.817 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.817 DEBUG:boto:Data: 2023-10-11T15:27:39.817 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.818 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.818 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.819 DEBUG:boto:Params: {} 2023-10-11T15:27:39.819 DEBUG:boto:Token: None 2023-10-11T15:27:39.820 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.820 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE= 2023-10-11T15:27:39.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE=', 'Content-Length': '0'} 2023-10-11T15:27:39.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618d00f9da43a0ec-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.824 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.824 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.825 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.825 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.826 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.826 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.826 DEBUG:boto:Data: 2023-10-11T15:27:39.827 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.828 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.828 DEBUG:boto:Params: {} 2023-10-11T15:27:39.829 DEBUG:boto:Token: None 2023-10-11T15:27:39.829 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rc1ueWuS3jSN5LmC3zWPvXkExU0= 2023-10-11T15:27:39.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rc1ueWuS3jSN5LmC3zWPvXkExU0=', 'Content-Length': '0'} 2023-10-11T15:27:39.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245be33207d2eb0b-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.832 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.833 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.833 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.834 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.834 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.834 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.835 DEBUG:boto:Data: 2023-10-11T15:27:39.835 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.836 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.836 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.836 DEBUG:boto:Params: {} 2023-10-11T15:27:39.837 DEBUG:boto:Token: None 2023-10-11T15:27:39.838 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rc1ueWuS3jSN5LmC3zWPvXkExU0= 2023-10-11T15:27:39.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rc1ueWuS3jSN5LmC3zWPvXkExU0=', 'Content-Length': '0'} 2023-10-11T15:27:39.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a1d0b5d69388b26-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:39.841 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.842 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.842 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.843 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.843 DEBUG:boto:Method: GET 2023-10-11T15:27:39.844 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.844 DEBUG:boto:Data: 2023-10-11T15:27:39.844 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.845 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.846 DEBUG:boto:Params: {} 2023-10-11T15:27:39.846 DEBUG:boto:Token: None 2023-10-11T15:27:39.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.847 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE= 2023-10-11T15:27:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE=', 'Content-Length': '0'} 2023-10-11T15:27:39.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007640d129d9c6b5ae-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.850 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.850 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.851 DEBUG:boto:path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.851 DEBUG:boto:auth_path=/dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.852 DEBUG:boto:Method: GET 2023-10-11T15:27:39.852 DEBUG:boto:Path: /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.853 DEBUG:boto:Data: 2023-10-11T15:27:39.853 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.854 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.854 DEBUG:boto:Params: {} 2023-10-11T15:27:39.855 DEBUG:boto:Token: None 2023-10-11T15:27:39.855 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33?versionId=null 2023-10-11T15:27:39.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE= 2023-10-11T15:27:39.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WZwLYuyo2rPQywx3ZLEFi8VhuUE=', 'Content-Length': '0'} 2023-10-11T15:27:39.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad698ebba48ff3c3-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.859 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.859 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.860 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.860 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.861 DEBUG:boto:Method: GET 2023-10-11T15:27:39.861 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.861 DEBUG:boto:Data: 2023-10-11T15:27:39.862 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.863 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.863 DEBUG:boto:Params: {} 2023-10-11T15:27:39.864 DEBUG:boto:Token: None 2023-10-11T15:27:39.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d8d5cfb21735c3a-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.868 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.869 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.869 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.870 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.870 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.871 DEBUG:boto:Method: GET 2023-10-11T15:27:39.871 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.872 DEBUG:boto:Data: 2023-10-11T15:27:39.872 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.873 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.873 DEBUG:boto:Params: {} 2023-10-11T15:27:39.874 DEBUG:boto:Token: None 2023-10-11T15:27:39.874 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aef56dfc2a5209f-006526beeb-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.926 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:39.927 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.928 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.928 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.929 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:39.929 DEBUG:boto:Data: 2023-10-11T15:27:39.929 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.930 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.931 DEBUG:boto:Params: {} 2023-10-11T15:27:39.931 DEBUG:boto:Token: None 2023-10-11T15:27:39.932 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33 2023-10-11T15:27:39.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+q8qM8ts+xbkU/KErv1dKEnBtvM= 2023-10-11T15:27:39.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+q8qM8ts+xbkU/KErv1dKEnBtvM=', 'Content-Length': '0'} 2023-10-11T15:27:39.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca1f6b54811a6fd-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.935 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.936 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.936 DEBUG:boto:Method: HEAD 2023-10-11T15:27:39.937 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:39.937 DEBUG:boto:Data: 2023-10-11T15:27:39.938 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.938 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.939 DEBUG:boto:Params: {} 2023-10-11T15:27:39.939 DEBUG:boto:Token: None 2023-10-11T15:27:39.940 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33 2023-10-11T15:27:39.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+q8qM8ts+xbkU/KErv1dKEnBtvM= 2023-10-11T15:27:39.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+q8qM8ts+xbkU/KErv1dKEnBtvM=', 'Content-Length': '0'} 2023-10-11T15:27:39.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010cf701d8255d3eb-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:27:39.944 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.944 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.945 DEBUG:boto:Method: GET 2023-10-11T15:27:39.945 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:39.946 DEBUG:boto:Data: 2023-10-11T15:27:39.946 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.947 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.947 DEBUG:boto:Params: {} 2023-10-11T15:27:39.948 DEBUG:boto:Token: None 2023-10-11T15:27:39.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33 2023-10-11T15:27:39.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nSJH52J7Xsd1ZwBERwG9Bz6f3RI= 2023-10-11T15:27:39.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nSJH52J7Xsd1ZwBERwG9Bz6f3RI=', 'Content-Length': '0'} 2023-10-11T15:27:39.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204e41cfe24cd8e9-006526beeb-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.952 DEBUG:boto:path=/dghqvd-59/obj33 2023-10-11T15:27:39.952 DEBUG:boto:auth_path=/dghqvd-59/obj33 2023-10-11T15:27:39.953 DEBUG:boto:Method: GET 2023-10-11T15:27:39.953 DEBUG:boto:Path: /dghqvd-59/obj33 2023-10-11T15:27:39.953 DEBUG:boto:Data: 2023-10-11T15:27:39.954 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:39.955 DEBUG:boto:Port: 8000 2023-10-11T15:27:39.955 DEBUG:boto:Params: {} 2023-10-11T15:27:39.956 DEBUG:boto:Token: None 2023-10-11T15:27:39.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/obj33 2023-10-11T15:27:39.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nSJH52J7Xsd1ZwBERwG9Bz6f3RI= 2023-10-11T15:27:39.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nSJH52J7Xsd1ZwBERwG9Bz6f3RI=', 'Content-Length': '0'} 2023-10-11T15:27:39.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012af3b511201ed3a-006526beeb-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:39.960 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:39.960 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:39.960 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.961 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.961 DEBUG:boto:Method: GET 2023-10-11T15:27:39.962 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:39.962 DEBUG:boto:Data: 2023-10-11T15:27:39.963 DEBUG:boto:Headers: {} 2023-10-11T15:27:39.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:39.964 DEBUG:boto:Port: 8001 2023-10-11T15:27:39.964 DEBUG:boto:Params: {} 2023-10-11T15:27:39.965 DEBUG:boto:Token: None 2023-10-11T15:27:39.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:39 GMT /dghqvd-59/ 2023-10-11T15:27:39.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc= 2023-10-11T15:27:39.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aPMESg+8L1L2bKpziZ9ZB3miuVc=', 'Content-Length': '0'} 2023-10-11T15:27:39.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7eeeb846d97aba8-006526beeb-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.014 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.015 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.015 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.016 DEBUG:boto:path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:40.016 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj33 2023-10-11T15:27:40.017 DEBUG:boto:Method: GET 2023-10-11T15:27:40.017 DEBUG:boto:Path: /dghqvd-59/?prefix=obj33 2023-10-11T15:27:40.018 DEBUG:boto:Data: 2023-10-11T15:27:40.018 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.019 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.019 DEBUG:boto:Params: {} 2023-10-11T15:27:40.020 DEBUG:boto:Token: None 2023-10-11T15:27:40.020 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000286bd931df520b36-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.026 DEBUG:boto:b'dghqvd-59obj331000falseobj332023-10-11T15:16:05.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:40.027 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.028 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.028 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.028 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.029 DEBUG:boto:Method: GET 2023-10-11T15:27:40.029 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.030 DEBUG:boto:Data: 2023-10-11T15:27:40.030 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.031 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.032 DEBUG:boto:Params: {} 2023-10-11T15:27:40.032 DEBUG:boto:Token: None 2023-10-11T15:27:40.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4= 2023-10-11T15:27:40.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4=', 'Content-Length': '0'} 2023-10-11T15:27:40.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f9a8316f3c074c7-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.036 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.037 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.037 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.038 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.038 DEBUG:boto:Method: GET 2023-10-11T15:27:40.039 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.039 DEBUG:boto:Data: 2023-10-11T15:27:40.039 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.040 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.041 DEBUG:boto:Params: {} 2023-10-11T15:27:40.041 DEBUG:boto:Token: None 2023-10-11T15:27:40.042 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4= 2023-10-11T15:27:40.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4=', 'Content-Length': '0'} 2023-10-11T15:27:40.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341e71e92517e11b-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.045 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.046 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.046 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.047 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.047 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.048 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.048 DEBUG:boto:Data: 2023-10-11T15:27:40.049 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.049 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.050 DEBUG:boto:Params: {} 2023-10-11T15:27:40.050 DEBUG:boto:Token: None 2023-10-11T15:27:40.051 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o47OM2UEBUUh+38Q3bgfqk2JpD8= 2023-10-11T15:27:40.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o47OM2UEBUUh+38Q3bgfqk2JpD8=', 'Content-Length': '0'} 2023-10-11T15:27:40.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002071ac3d8bffe6da-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.054 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.054 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.055 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.055 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.055 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.056 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.056 DEBUG:boto:Data: 2023-10-11T15:27:40.057 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.057 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.057 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.058 DEBUG:boto:Params: {} 2023-10-11T15:27:40.058 DEBUG:boto:Token: None 2023-10-11T15:27:40.059 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.059 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o47OM2UEBUUh+38Q3bgfqk2JpD8= 2023-10-11T15:27:40.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o47OM2UEBUUh+38Q3bgfqk2JpD8=', 'Content-Length': '0'} 2023-10-11T15:27:40.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be8f34445d2bd94-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:40.063 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.063 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.064 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.064 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.064 DEBUG:boto:Method: GET 2023-10-11T15:27:40.065 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.065 DEBUG:boto:Data: 2023-10-11T15:27:40.066 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.066 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.067 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.067 DEBUG:boto:Params: {} 2023-10-11T15:27:40.068 DEBUG:boto:Token: None 2023-10-11T15:27:40.068 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.068 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4= 2023-10-11T15:27:40.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4=', 'Content-Length': '0'} 2023-10-11T15:27:40.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013893b1cdeec8b5-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.071 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.072 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.072 DEBUG:boto:path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.073 DEBUG:boto:auth_path=/dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.073 DEBUG:boto:Method: GET 2023-10-11T15:27:40.074 DEBUG:boto:Path: /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.074 DEBUG:boto:Data: 2023-10-11T15:27:40.075 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.075 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.075 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.076 DEBUG:boto:Params: {} 2023-10-11T15:27:40.076 DEBUG:boto:Token: None 2023-10-11T15:27:40.077 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34?versionId=null 2023-10-11T15:27:40.077 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4= 2023-10-11T15:27:40.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQzgo5fkuWMgbyODl3YpPeIOUo4=', 'Content-Length': '0'} 2023-10-11T15:27:40.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006adabc0851742828-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.080 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.081 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.081 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.082 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.082 DEBUG:boto:Method: GET 2023-10-11T15:27:40.083 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.083 DEBUG:boto:Data: 2023-10-11T15:27:40.084 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.085 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.086 DEBUG:boto:Params: {} 2023-10-11T15:27:40.086 DEBUG:boto:Token: None 2023-10-11T15:27:40.087 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e65525a22708bad-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.091 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.091 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.092 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.092 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.093 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.093 DEBUG:boto:Method: GET 2023-10-11T15:27:40.094 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.094 DEBUG:boto:Data: 2023-10-11T15:27:40.095 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.096 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.096 DEBUG:boto:Params: {} 2023-10-11T15:27:40.097 DEBUG:boto:Token: None 2023-10-11T15:27:40.097 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000a1c10ef48c7bfc-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.103 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.104 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.104 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.105 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.105 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:40.106 DEBUG:boto:Data: 2023-10-11T15:27:40.106 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.107 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.107 DEBUG:boto:Params: {} 2023-10-11T15:27:40.108 DEBUG:boto:Token: None 2023-10-11T15:27:40.108 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34 2023-10-11T15:27:40.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jY3rFTNwHIz9sj3e0aMeZszwP9k= 2023-10-11T15:27:40.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jY3rFTNwHIz9sj3e0aMeZszwP9k=', 'Content-Length': '0'} 2023-10-11T15:27:40.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001095a864606e6d36-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.114 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.114 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.115 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.115 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:40.116 DEBUG:boto:Data: 2023-10-11T15:27:40.116 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.117 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.117 DEBUG:boto:Params: {} 2023-10-11T15:27:40.118 DEBUG:boto:Token: None 2023-10-11T15:27:40.118 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34 2023-10-11T15:27:40.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jY3rFTNwHIz9sj3e0aMeZszwP9k= 2023-10-11T15:27:40.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jY3rFTNwHIz9sj3e0aMeZszwP9k=', 'Content-Length': '0'} 2023-10-11T15:27:40.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031610ca8ef88d9cc-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:27:40.122 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.123 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.123 DEBUG:boto:Method: GET 2023-10-11T15:27:40.124 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:40.124 DEBUG:boto:Data: 2023-10-11T15:27:40.125 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.126 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.126 DEBUG:boto:Params: {} 2023-10-11T15:27:40.127 DEBUG:boto:Token: None 2023-10-11T15:27:40.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34 2023-10-11T15:27:40.129 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3nugcFPU1j9vTymA0QKFFbA1MQ4= 2023-10-11T15:27:40.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3nugcFPU1j9vTymA0QKFFbA1MQ4=', 'Content-Length': '0'} 2023-10-11T15:27:40.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c098af98ce01219f-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.174 DEBUG:boto:path=/dghqvd-59/obj34 2023-10-11T15:27:40.175 DEBUG:boto:auth_path=/dghqvd-59/obj34 2023-10-11T15:27:40.175 DEBUG:boto:Method: GET 2023-10-11T15:27:40.176 DEBUG:boto:Path: /dghqvd-59/obj34 2023-10-11T15:27:40.176 DEBUG:boto:Data: 2023-10-11T15:27:40.177 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.178 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.178 DEBUG:boto:Params: {} 2023-10-11T15:27:40.179 DEBUG:boto:Token: None 2023-10-11T15:27:40.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj34 2023-10-11T15:27:40.180 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3nugcFPU1j9vTymA0QKFFbA1MQ4= 2023-10-11T15:27:40.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3nugcFPU1j9vTymA0QKFFbA1MQ4=', 'Content-Length': '0'} 2023-10-11T15:27:40.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cb76ef16acd6e7a-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.183 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.183 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.184 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.184 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.185 DEBUG:boto:Method: GET 2023-10-11T15:27:40.185 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.186 DEBUG:boto:Data: 2023-10-11T15:27:40.186 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.187 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.187 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.188 DEBUG:boto:Params: {} 2023-10-11T15:27:40.188 DEBUG:boto:Token: None 2023-10-11T15:27:40.189 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.189 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7f2caf60a0deab1-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.192 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.193 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.194 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.194 DEBUG:boto:path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.194 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.195 DEBUG:boto:Method: GET 2023-10-11T15:27:40.195 DEBUG:boto:Path: /dghqvd-59/?prefix=obj34 2023-10-11T15:27:40.196 DEBUG:boto:Data: 2023-10-11T15:27:40.196 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.197 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.198 DEBUG:boto:Params: {} 2023-10-11T15:27:40.198 DEBUG:boto:Token: None 2023-10-11T15:27:40.199 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d157712c436af3-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.204 DEBUG:boto:b'dghqvd-59obj341000falseobj342023-10-11T15:16:32.660Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:40.205 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.206 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.206 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.207 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.207 DEBUG:boto:Method: GET 2023-10-11T15:27:40.208 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.208 DEBUG:boto:Data: 2023-10-11T15:27:40.209 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.210 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.210 DEBUG:boto:Params: {} 2023-10-11T15:27:40.210 DEBUG:boto:Token: None 2023-10-11T15:27:40.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow= 2023-10-11T15:27:40.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow=', 'Content-Length': '0'} 2023-10-11T15:27:40.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71b8ae8e60e1af3-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.215 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.216 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.216 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.217 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.217 DEBUG:boto:Method: GET 2023-10-11T15:27:40.217 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.218 DEBUG:boto:Data: 2023-10-11T15:27:40.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.219 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.220 DEBUG:boto:Params: {} 2023-10-11T15:27:40.220 DEBUG:boto:Token: None 2023-10-11T15:27:40.221 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow= 2023-10-11T15:27:40.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow=', 'Content-Length': '0'} 2023-10-11T15:27:40.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e90a7c93bd77ef-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.224 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.225 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.225 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.225 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.226 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.226 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.227 DEBUG:boto:Data: 2023-10-11T15:27:40.227 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.228 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.228 DEBUG:boto:Params: {} 2023-10-11T15:27:40.229 DEBUG:boto:Token: None 2023-10-11T15:27:40.229 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cdgGCho1AwzphP7i5E9kDfGsPQ8= 2023-10-11T15:27:40.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cdgGCho1AwzphP7i5E9kDfGsPQ8=', 'Content-Length': '0'} 2023-10-11T15:27:40.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042834be669cbed4d-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.233 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.233 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.234 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.234 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.235 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.235 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.235 DEBUG:boto:Data: 2023-10-11T15:27:40.236 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.237 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.237 DEBUG:boto:Params: {} 2023-10-11T15:27:40.237 DEBUG:boto:Token: None 2023-10-11T15:27:40.238 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cdgGCho1AwzphP7i5E9kDfGsPQ8= 2023-10-11T15:27:40.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cdgGCho1AwzphP7i5E9kDfGsPQ8=', 'Content-Length': '0'} 2023-10-11T15:27:40.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002049980efe4fb7-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:40.242 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.242 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.243 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.243 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.244 DEBUG:boto:Method: GET 2023-10-11T15:27:40.244 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.245 DEBUG:boto:Data: 2023-10-11T15:27:40.245 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.245 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.246 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.246 DEBUG:boto:Params: {} 2023-10-11T15:27:40.247 DEBUG:boto:Token: None 2023-10-11T15:27:40.247 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow= 2023-10-11T15:27:40.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow=', 'Content-Length': '0'} 2023-10-11T15:27:40.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050de00f5faab82a7-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.251 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.251 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.252 DEBUG:boto:path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.252 DEBUG:boto:auth_path=/dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.253 DEBUG:boto:Method: GET 2023-10-11T15:27:40.253 DEBUG:boto:Path: /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.254 DEBUG:boto:Data: 2023-10-11T15:27:40.254 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.255 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.255 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.255 DEBUG:boto:Params: {} 2023-10-11T15:27:40.256 DEBUG:boto:Token: None 2023-10-11T15:27:40.257 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37?versionId=null 2023-10-11T15:27:40.257 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow= 2023-10-11T15:27:40.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3b2vhvhFbOUjf1+5F9kedtEc4Ow=', 'Content-Length': '0'} 2023-10-11T15:27:40.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8869cca3928c318-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.260 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.260 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.261 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.261 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.262 DEBUG:boto:Method: GET 2023-10-11T15:27:40.262 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.262 DEBUG:boto:Data: 2023-10-11T15:27:40.263 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.263 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.264 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.264 DEBUG:boto:Params: {} 2023-10-11T15:27:40.265 DEBUG:boto:Token: None 2023-10-11T15:27:40.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.266 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc376d3cc4ef9f7a-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.310 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.311 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.312 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.312 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.313 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.313 DEBUG:boto:Method: GET 2023-10-11T15:27:40.314 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.314 DEBUG:boto:Data: 2023-10-11T15:27:40.317 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.317 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.318 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.318 DEBUG:boto:Params: {} 2023-10-11T15:27:40.319 DEBUG:boto:Token: None 2023-10-11T15:27:40.319 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a8f687970101cc-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.325 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.326 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.326 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.327 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.327 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:40.327 DEBUG:boto:Data: 2023-10-11T15:27:40.328 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.329 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.329 DEBUG:boto:Params: {} 2023-10-11T15:27:40.329 DEBUG:boto:Token: None 2023-10-11T15:27:40.330 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37 2023-10-11T15:27:40.330 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POO+7d5XhkCCgRzKNX431tEGvVY= 2023-10-11T15:27:40.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POO+7d5XhkCCgRzKNX431tEGvVY=', 'Content-Length': '0'} 2023-10-11T15:27:40.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008279b629ff687784-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.333 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.334 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.334 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.335 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:40.335 DEBUG:boto:Data: 2023-10-11T15:27:40.336 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.336 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.336 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.337 DEBUG:boto:Params: {} 2023-10-11T15:27:40.337 DEBUG:boto:Token: None 2023-10-11T15:27:40.338 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37 2023-10-11T15:27:40.338 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:POO+7d5XhkCCgRzKNX431tEGvVY= 2023-10-11T15:27:40.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:POO+7d5XhkCCgRzKNX431tEGvVY=', 'Content-Length': '0'} 2023-10-11T15:27:40.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7c9c00e9dc5a6e-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:27:40.342 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.342 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.343 DEBUG:boto:Method: GET 2023-10-11T15:27:40.343 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:40.344 DEBUG:boto:Data: 2023-10-11T15:27:40.344 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.345 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.346 DEBUG:boto:Params: {} 2023-10-11T15:27:40.346 DEBUG:boto:Token: None 2023-10-11T15:27:40.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37 2023-10-11T15:27:40.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XRf0TofMrlWCFYzB5mvj6VFDgyQ= 2023-10-11T15:27:40.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XRf0TofMrlWCFYzB5mvj6VFDgyQ=', 'Content-Length': '0'} 2023-10-11T15:27:40.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f1bfc36d03c75a-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.350 DEBUG:boto:path=/dghqvd-59/obj37 2023-10-11T15:27:40.351 DEBUG:boto:auth_path=/dghqvd-59/obj37 2023-10-11T15:27:40.351 DEBUG:boto:Method: GET 2023-10-11T15:27:40.352 DEBUG:boto:Path: /dghqvd-59/obj37 2023-10-11T15:27:40.352 DEBUG:boto:Data: 2023-10-11T15:27:40.353 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.354 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.354 DEBUG:boto:Params: {} 2023-10-11T15:27:40.355 DEBUG:boto:Token: None 2023-10-11T15:27:40.355 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj37 2023-10-11T15:27:40.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XRf0TofMrlWCFYzB5mvj6VFDgyQ= 2023-10-11T15:27:40.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XRf0TofMrlWCFYzB5mvj6VFDgyQ=', 'Content-Length': '0'} 2023-10-11T15:27:40.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000677393058ba60272-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.398 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.399 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.400 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.400 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.400 DEBUG:boto:Method: GET 2023-10-11T15:27:40.401 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.401 DEBUG:boto:Data: 2023-10-11T15:27:40.402 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.403 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.403 DEBUG:boto:Params: {} 2023-10-11T15:27:40.404 DEBUG:boto:Token: None 2023-10-11T15:27:40.404 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002148c225da61f4c6-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.408 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.409 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.409 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.410 DEBUG:boto:path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.410 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.410 DEBUG:boto:Method: GET 2023-10-11T15:27:40.411 DEBUG:boto:Path: /dghqvd-59/?prefix=obj37 2023-10-11T15:27:40.411 DEBUG:boto:Data: 2023-10-11T15:27:40.412 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.412 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.412 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.413 DEBUG:boto:Params: {} 2023-10-11T15:27:40.413 DEBUG:boto:Token: None 2023-10-11T15:27:40.414 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004280ae5012669def-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.419 DEBUG:boto:b'dghqvd-59obj371000falseobj372023-10-11T15:25:38.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:40.421 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.421 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.421 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.422 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.422 DEBUG:boto:Method: GET 2023-10-11T15:27:40.423 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.423 DEBUG:boto:Data: 2023-10-11T15:27:40.424 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.424 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.425 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.425 DEBUG:boto:Params: {} 2023-10-11T15:27:40.425 DEBUG:boto:Token: None 2023-10-11T15:27:40.426 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.426 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU= 2023-10-11T15:27:40.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU=', 'Content-Length': '0'} 2023-10-11T15:27:40.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000685675f1e75aa138-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.430 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.430 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.431 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.431 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.432 DEBUG:boto:Method: GET 2023-10-11T15:27:40.432 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.432 DEBUG:boto:Data: 2023-10-11T15:27:40.433 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.433 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.434 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.434 DEBUG:boto:Params: {} 2023-10-11T15:27:40.435 DEBUG:boto:Token: None 2023-10-11T15:27:40.435 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.436 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU= 2023-10-11T15:27:40.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU=', 'Content-Length': '0'} 2023-10-11T15:27:40.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000759a777809cd83b-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.439 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.440 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.440 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.441 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.441 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.441 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.442 DEBUG:boto:Data: 2023-10-11T15:27:40.442 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.443 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.444 DEBUG:boto:Params: {} 2023-10-11T15:27:40.444 DEBUG:boto:Token: None 2023-10-11T15:27:40.444 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0oR7/fhcvZiMJ3bguZRME7KV5GI= 2023-10-11T15:27:40.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0oR7/fhcvZiMJ3bguZRME7KV5GI=', 'Content-Length': '0'} 2023-10-11T15:27:40.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e17ca765f7c90dd-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.448 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.448 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.449 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.449 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.450 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.450 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.451 DEBUG:boto:Data: 2023-10-11T15:27:40.451 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.451 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.452 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.452 DEBUG:boto:Params: {} 2023-10-11T15:27:40.453 DEBUG:boto:Token: None 2023-10-11T15:27:40.453 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0oR7/fhcvZiMJ3bguZRME7KV5GI= 2023-10-11T15:27:40.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0oR7/fhcvZiMJ3bguZRME7KV5GI=', 'Content-Length': '0'} 2023-10-11T15:27:40.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7581b83158ec61e-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:40.457 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.458 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.458 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.459 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.459 DEBUG:boto:Method: GET 2023-10-11T15:27:40.460 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.460 DEBUG:boto:Data: 2023-10-11T15:27:40.461 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.462 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.462 DEBUG:boto:Params: {} 2023-10-11T15:27:40.463 DEBUG:boto:Token: None 2023-10-11T15:27:40.463 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU= 2023-10-11T15:27:40.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU=', 'Content-Length': '0'} 2023-10-11T15:27:40.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eb4625a2e791ece-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.510 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.511 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.512 DEBUG:boto:path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.512 DEBUG:boto:auth_path=/dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.512 DEBUG:boto:Method: GET 2023-10-11T15:27:40.513 DEBUG:boto:Path: /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.513 DEBUG:boto:Data: 2023-10-11T15:27:40.514 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.515 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.515 DEBUG:boto:Params: {} 2023-10-11T15:27:40.516 DEBUG:boto:Token: None 2023-10-11T15:27:40.516 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38?versionId=null 2023-10-11T15:27:40.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU= 2023-10-11T15:27:40.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I458McuPw3uH0SEs3ScAHkAeMlU=', 'Content-Length': '0'} 2023-10-11T15:27:40.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000226ea5289283734-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.522 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.522 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.523 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.523 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.524 DEBUG:boto:Method: GET 2023-10-11T15:27:40.524 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.525 DEBUG:boto:Data: 2023-10-11T15:27:40.525 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.526 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.527 DEBUG:boto:Params: {} 2023-10-11T15:27:40.527 DEBUG:boto:Token: None 2023-10-11T15:27:40.528 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d31f9e803555b40-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.531 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.532 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.533 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.533 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.534 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.534 DEBUG:boto:Method: GET 2023-10-11T15:27:40.535 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.535 DEBUG:boto:Data: 2023-10-11T15:27:40.536 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.537 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.537 DEBUG:boto:Params: {} 2023-10-11T15:27:40.538 DEBUG:boto:Token: None 2023-10-11T15:27:40.538 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882c5fec11ebd570-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.544 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.544 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.545 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.545 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.546 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:40.546 DEBUG:boto:Data: 2023-10-11T15:27:40.547 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.548 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.548 DEBUG:boto:Params: {} 2023-10-11T15:27:40.549 DEBUG:boto:Token: None 2023-10-11T15:27:40.549 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38 2023-10-11T15:27:40.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:20JQNPkmsDi5Fo0tyV7zOSloy6A= 2023-10-11T15:27:40.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:20JQNPkmsDi5Fo0tyV7zOSloy6A=', 'Content-Length': '0'} 2023-10-11T15:27:40.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006461429ac935894-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.552 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.553 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.553 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.554 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:40.554 DEBUG:boto:Data: 2023-10-11T15:27:40.554 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.555 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.555 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.556 DEBUG:boto:Params: {} 2023-10-11T15:27:40.556 DEBUG:boto:Token: None 2023-10-11T15:27:40.557 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38 2023-10-11T15:27:40.557 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:20JQNPkmsDi5Fo0tyV7zOSloy6A= 2023-10-11T15:27:40.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:20JQNPkmsDi5Fo0tyV7zOSloy6A=', 'Content-Length': '0'} 2023-10-11T15:27:40.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e21196fa5b074b-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:27:40.560 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.561 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.561 DEBUG:boto:Method: GET 2023-10-11T15:27:40.562 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:40.562 DEBUG:boto:Data: 2023-10-11T15:27:40.563 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.563 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.564 DEBUG:boto:Params: {} 2023-10-11T15:27:40.564 DEBUG:boto:Token: None 2023-10-11T15:27:40.565 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38 2023-10-11T15:27:40.565 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqLVTcaVXZbI3NRsSWzWSXxyh+s= 2023-10-11T15:27:40.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqLVTcaVXZbI3NRsSWzWSXxyh+s=', 'Content-Length': '0'} 2023-10-11T15:27:40.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba28736a82facad-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.568 DEBUG:boto:path=/dghqvd-59/obj38 2023-10-11T15:27:40.569 DEBUG:boto:auth_path=/dghqvd-59/obj38 2023-10-11T15:27:40.569 DEBUG:boto:Method: GET 2023-10-11T15:27:40.570 DEBUG:boto:Path: /dghqvd-59/obj38 2023-10-11T15:27:40.570 DEBUG:boto:Data: 2023-10-11T15:27:40.571 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.571 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.571 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.572 DEBUG:boto:Params: {} 2023-10-11T15:27:40.572 DEBUG:boto:Token: None 2023-10-11T15:27:40.573 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj38 2023-10-11T15:27:40.573 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hqLVTcaVXZbI3NRsSWzWSXxyh+s= 2023-10-11T15:27:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hqLVTcaVXZbI3NRsSWzWSXxyh+s=', 'Content-Length': '0'} 2023-10-11T15:27:40.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d4c5efd2cc62ef-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.576 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.577 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.577 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.578 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.578 DEBUG:boto:Method: GET 2023-10-11T15:27:40.579 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.579 DEBUG:boto:Data: 2023-10-11T15:27:40.579 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.580 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.580 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.581 DEBUG:boto:Params: {} 2023-10-11T15:27:40.581 DEBUG:boto:Token: None 2023-10-11T15:27:40.582 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b645d1e06c67bd-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.586 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.586 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.587 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.587 DEBUG:boto:path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.588 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.588 DEBUG:boto:Method: GET 2023-10-11T15:27:40.588 DEBUG:boto:Path: /dghqvd-59/?prefix=obj38 2023-10-11T15:27:40.589 DEBUG:boto:Data: 2023-10-11T15:27:40.589 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.590 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.591 DEBUG:boto:Params: {} 2023-10-11T15:27:40.591 DEBUG:boto:Token: None 2023-10-11T15:27:40.592 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096adaddf6e300734-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.597 DEBUG:boto:b'dghqvd-59obj381000falseobj382023-10-11T15:18:55.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:40.598 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.599 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.599 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.600 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.600 DEBUG:boto:Method: GET 2023-10-11T15:27:40.600 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.601 DEBUG:boto:Data: 2023-10-11T15:27:40.601 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.602 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.602 DEBUG:boto:Params: {} 2023-10-11T15:27:40.603 DEBUG:boto:Token: None 2023-10-11T15:27:40.603 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA= 2023-10-11T15:27:40.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA=', 'Content-Length': '0'} 2023-10-11T15:27:40.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f25d357541505f34-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.607 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.608 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.608 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.609 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.609 DEBUG:boto:Method: GET 2023-10-11T15:27:40.609 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.610 DEBUG:boto:Data: 2023-10-11T15:27:40.610 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.611 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.611 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.612 DEBUG:boto:Params: {} 2023-10-11T15:27:40.612 DEBUG:boto:Token: None 2023-10-11T15:27:40.613 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.613 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA= 2023-10-11T15:27:40.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA=', 'Content-Length': '0'} 2023-10-11T15:27:40.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eea3214a753485f-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.616 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.617 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.617 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.618 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.618 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.618 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.619 DEBUG:boto:Data: 2023-10-11T15:27:40.619 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.620 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.620 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.620 DEBUG:boto:Params: {} 2023-10-11T15:27:40.621 DEBUG:boto:Token: None 2023-10-11T15:27:40.621 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.622 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPEhjjfNFpEUXpfoajvWTX0LOMY= 2023-10-11T15:27:40.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPEhjjfNFpEUXpfoajvWTX0LOMY=', 'Content-Length': '0'} 2023-10-11T15:27:40.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bc869acf6299cf-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.625 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.625 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.626 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.626 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.626 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.627 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.627 DEBUG:boto:Data: 2023-10-11T15:27:40.628 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.629 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.629 DEBUG:boto:Params: {} 2023-10-11T15:27:40.629 DEBUG:boto:Token: None 2023-10-11T15:27:40.630 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uPEhjjfNFpEUXpfoajvWTX0LOMY= 2023-10-11T15:27:40.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uPEhjjfNFpEUXpfoajvWTX0LOMY=', 'Content-Length': '0'} 2023-10-11T15:27:40.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdeb9ee3187de3d-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:40.633 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.634 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.634 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.635 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.635 DEBUG:boto:Method: GET 2023-10-11T15:27:40.636 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.636 DEBUG:boto:Data: 2023-10-11T15:27:40.637 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.637 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.638 DEBUG:boto:Params: {} 2023-10-11T15:27:40.638 DEBUG:boto:Token: None 2023-10-11T15:27:40.639 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA= 2023-10-11T15:27:40.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA=', 'Content-Length': '0'} 2023-10-11T15:27:40.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005109aa22dedc70d6-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.682 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.683 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.683 DEBUG:boto:path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.684 DEBUG:boto:auth_path=/dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.684 DEBUG:boto:Method: GET 2023-10-11T15:27:40.685 DEBUG:boto:Path: /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.685 DEBUG:boto:Data: 2023-10-11T15:27:40.685 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.686 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.687 DEBUG:boto:Params: {} 2023-10-11T15:27:40.687 DEBUG:boto:Token: None 2023-10-11T15:27:40.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44?versionId=null 2023-10-11T15:27:40.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA= 2023-10-11T15:27:40.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v2q0NFyA22Wy7xXMnltOq523MpA=', 'Content-Length': '0'} 2023-10-11T15:27:40.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c4ec8a71f7f6e8-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.692 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.692 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.693 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.693 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.693 DEBUG:boto:Method: GET 2023-10-11T15:27:40.694 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.694 DEBUG:boto:Data: 2023-10-11T15:27:40.695 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.695 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.696 DEBUG:boto:Params: {} 2023-10-11T15:27:40.696 DEBUG:boto:Token: None 2023-10-11T15:27:40.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a120f68ef1b4e327-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.701 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.702 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.702 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.703 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.703 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.704 DEBUG:boto:Method: GET 2023-10-11T15:27:40.704 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.704 DEBUG:boto:Data: 2023-10-11T15:27:40.705 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.705 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.706 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.706 DEBUG:boto:Params: {} 2023-10-11T15:27:40.707 DEBUG:boto:Token: None 2023-10-11T15:27:40.707 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f941a05aa215c3f-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.758 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.759 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.760 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.760 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.761 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:40.761 DEBUG:boto:Data: 2023-10-11T15:27:40.761 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.762 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.763 DEBUG:boto:Params: {} 2023-10-11T15:27:40.763 DEBUG:boto:Token: None 2023-10-11T15:27:40.763 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44 2023-10-11T15:27:40.764 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PzHO4YIQx8qZfj5lxIahtWUwIuc= 2023-10-11T15:27:40.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PzHO4YIQx8qZfj5lxIahtWUwIuc=', 'Content-Length': '0'} 2023-10-11T15:27:40.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d729d9df9c99c64b-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.767 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.768 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.768 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.769 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:40.769 DEBUG:boto:Data: 2023-10-11T15:27:40.769 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.770 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.771 DEBUG:boto:Params: {} 2023-10-11T15:27:40.771 DEBUG:boto:Token: None 2023-10-11T15:27:40.772 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44 2023-10-11T15:27:40.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PzHO4YIQx8qZfj5lxIahtWUwIuc= 2023-10-11T15:27:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PzHO4YIQx8qZfj5lxIahtWUwIuc=', 'Content-Length': '0'} 2023-10-11T15:27:40.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb55d5be93345a27-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:27:40.775 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.776 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.776 DEBUG:boto:Method: GET 2023-10-11T15:27:40.777 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:40.777 DEBUG:boto:Data: 2023-10-11T15:27:40.778 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.779 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.779 DEBUG:boto:Params: {} 2023-10-11T15:27:40.779 DEBUG:boto:Token: None 2023-10-11T15:27:40.780 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44 2023-10-11T15:27:40.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lnvhssnpQN8zYgL8d245A48mizg= 2023-10-11T15:27:40.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lnvhssnpQN8zYgL8d245A48mizg=', 'Content-Length': '0'} 2023-10-11T15:27:40.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321adfa0bc34be3b-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.783 DEBUG:boto:path=/dghqvd-59/obj44 2023-10-11T15:27:40.784 DEBUG:boto:auth_path=/dghqvd-59/obj44 2023-10-11T15:27:40.784 DEBUG:boto:Method: GET 2023-10-11T15:27:40.785 DEBUG:boto:Path: /dghqvd-59/obj44 2023-10-11T15:27:40.785 DEBUG:boto:Data: 2023-10-11T15:27:40.786 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.787 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.787 DEBUG:boto:Params: {} 2023-10-11T15:27:40.787 DEBUG:boto:Token: None 2023-10-11T15:27:40.788 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj44 2023-10-11T15:27:40.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lnvhssnpQN8zYgL8d245A48mizg= 2023-10-11T15:27:40.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lnvhssnpQN8zYgL8d245A48mizg=', 'Content-Length': '0'} 2023-10-11T15:27:40.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9bc68767a01d34-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.792 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.792 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.792 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.793 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.793 DEBUG:boto:Method: GET 2023-10-11T15:27:40.794 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.794 DEBUG:boto:Data: 2023-10-11T15:27:40.795 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.795 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.796 DEBUG:boto:Params: {} 2023-10-11T15:27:40.796 DEBUG:boto:Token: None 2023-10-11T15:27:40.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.797 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278dc351f9ee1792-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.846 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.847 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.848 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.848 DEBUG:boto:path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.849 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.849 DEBUG:boto:Method: GET 2023-10-11T15:27:40.849 DEBUG:boto:Path: /dghqvd-59/?prefix=obj44 2023-10-11T15:27:40.850 DEBUG:boto:Data: 2023-10-11T15:27:40.850 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.851 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.852 DEBUG:boto:Params: {} 2023-10-11T15:27:40.852 DEBUG:boto:Token: None 2023-10-11T15:27:40.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb08e1a47c0c2a45-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.859 DEBUG:boto:b'dghqvd-59obj441000falseobj442023-10-11T15:16:06.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:40.860 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.860 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.861 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.861 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.861 DEBUG:boto:Method: GET 2023-10-11T15:27:40.862 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.862 DEBUG:boto:Data: 2023-10-11T15:27:40.863 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.864 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.864 DEBUG:boto:Params: {} 2023-10-11T15:27:40.865 DEBUG:boto:Token: None 2023-10-11T15:27:40.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw= 2023-10-11T15:27:40.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw=', 'Content-Length': '0'} 2023-10-11T15:27:40.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654f321b4f89fcf6-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.869 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.870 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.870 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.870 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.871 DEBUG:boto:Method: GET 2023-10-11T15:27:40.871 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.872 DEBUG:boto:Data: 2023-10-11T15:27:40.872 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.873 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.873 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.874 DEBUG:boto:Params: {} 2023-10-11T15:27:40.874 DEBUG:boto:Token: None 2023-10-11T15:27:40.874 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.875 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw= 2023-10-11T15:27:40.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw=', 'Content-Length': '0'} 2023-10-11T15:27:40.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5842c7f9b37ddd-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.878 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.879 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.879 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.879 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.880 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.880 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.881 DEBUG:boto:Data: 2023-10-11T15:27:40.881 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.882 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.882 DEBUG:boto:Params: {} 2023-10-11T15:27:40.883 DEBUG:boto:Token: None 2023-10-11T15:27:40.883 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:40qczw6Cq2g/o1RTn19XjomDs70= 2023-10-11T15:27:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:40qczw6Cq2g/o1RTn19XjomDs70=', 'Content-Length': '0'} 2023-10-11T15:27:40.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d03fb19ccb8f17-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.887 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.887 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.888 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.888 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.888 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.889 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.889 DEBUG:boto:Data: 2023-10-11T15:27:40.890 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.890 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.891 DEBUG:boto:Params: {} 2023-10-11T15:27:40.891 DEBUG:boto:Token: None 2023-10-11T15:27:40.892 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:40qczw6Cq2g/o1RTn19XjomDs70= 2023-10-11T15:27:40.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:40qczw6Cq2g/o1RTn19XjomDs70=', 'Content-Length': '0'} 2023-10-11T15:27:40.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe6d3df57260612-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:40.896 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.896 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.896 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.897 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.897 DEBUG:boto:Method: GET 2023-10-11T15:27:40.898 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.898 DEBUG:boto:Data: 2023-10-11T15:27:40.899 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.900 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.900 DEBUG:boto:Params: {} 2023-10-11T15:27:40.900 DEBUG:boto:Token: None 2023-10-11T15:27:40.901 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw= 2023-10-11T15:27:40.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw=', 'Content-Length': '0'} 2023-10-11T15:27:40.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24c858fd27a61c4-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.905 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.905 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.905 DEBUG:boto:path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.906 DEBUG:boto:auth_path=/dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.906 DEBUG:boto:Method: GET 2023-10-11T15:27:40.907 DEBUG:boto:Path: /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.907 DEBUG:boto:Data: 2023-10-11T15:27:40.908 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.909 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.909 DEBUG:boto:Params: {} 2023-10-11T15:27:40.909 DEBUG:boto:Token: None 2023-10-11T15:27:40.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46?versionId=null 2023-10-11T15:27:40.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw= 2023-10-11T15:27:40.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HUK9JsA62E1kfZtNTSVcf1qdubw=', 'Content-Length': '0'} 2023-10-11T15:27:40.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a079cab2b60d17d-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.954 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.955 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.955 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.956 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.956 DEBUG:boto:Method: GET 2023-10-11T15:27:40.957 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.957 DEBUG:boto:Data: 2023-10-11T15:27:40.958 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.959 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.959 DEBUG:boto:Params: {} 2023-10-11T15:27:40.960 DEBUG:boto:Token: None 2023-10-11T15:27:40.960 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.961 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66933b22fe06375-006526beec-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.964 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.965 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:40.966 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:40.966 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.967 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.967 DEBUG:boto:Method: GET 2023-10-11T15:27:40.967 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:40.968 DEBUG:boto:Data: 2023-10-11T15:27:40.968 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.969 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.970 DEBUG:boto:Params: {} 2023-10-11T15:27:40.970 DEBUG:boto:Token: None 2023-10-11T15:27:40.970 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/ 2023-10-11T15:27:40.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0= 2023-10-11T15:27:40.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ijh9MJUCOXnzixhugd+uNsi9sa0=', 'Content-Length': '0'} 2023-10-11T15:27:40.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434684e68865a9bb-006526beec-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.976 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:40.977 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.977 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.978 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.978 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:40.979 DEBUG:boto:Data: 2023-10-11T15:27:40.979 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.980 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.980 DEBUG:boto:Params: {} 2023-10-11T15:27:40.980 DEBUG:boto:Token: None 2023-10-11T15:27:40.981 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46 2023-10-11T15:27:40.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dH9xhtkJ/C14tB6XF+RFr01b8aw= 2023-10-11T15:27:40.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dH9xhtkJ/C14tB6XF+RFr01b8aw=', 'Content-Length': '0'} 2023-10-11T15:27:40.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062de2f44466be84-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.985 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.985 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.986 DEBUG:boto:Method: HEAD 2023-10-11T15:27:40.986 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:40.987 DEBUG:boto:Data: 2023-10-11T15:27:40.987 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:40.988 DEBUG:boto:Port: 8000 2023-10-11T15:27:40.988 DEBUG:boto:Params: {} 2023-10-11T15:27:40.989 DEBUG:boto:Token: None 2023-10-11T15:27:40.989 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46 2023-10-11T15:27:40.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dH9xhtkJ/C14tB6XF+RFr01b8aw= 2023-10-11T15:27:40.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dH9xhtkJ/C14tB6XF+RFr01b8aw=', 'Content-Length': '0'} 2023-10-11T15:27:40.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940ade9c5b10d222-006526beec-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:40.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:27:40.993 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:40.993 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:40.994 DEBUG:boto:Method: GET 2023-10-11T15:27:40.994 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:40.995 DEBUG:boto:Data: 2023-10-11T15:27:40.995 DEBUG:boto:Headers: {} 2023-10-11T15:27:40.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:40.996 DEBUG:boto:Port: 8001 2023-10-11T15:27:40.996 DEBUG:boto:Params: {} 2023-10-11T15:27:40.997 DEBUG:boto:Token: None 2023-10-11T15:27:40.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:40 GMT /dghqvd-59/obj46 2023-10-11T15:27:40.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qz7CKsWU9txSf9DDjYk1HbKDYbs= 2023-10-11T15:27:40.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qz7CKsWU9txSf9DDjYk1HbKDYbs=', 'Content-Length': '0'} 2023-10-11T15:27:41.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ed82710c6946d1-006526beec-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.042 DEBUG:boto:path=/dghqvd-59/obj46 2023-10-11T15:27:41.043 DEBUG:boto:auth_path=/dghqvd-59/obj46 2023-10-11T15:27:41.044 DEBUG:boto:Method: GET 2023-10-11T15:27:41.044 DEBUG:boto:Path: /dghqvd-59/obj46 2023-10-11T15:27:41.044 DEBUG:boto:Data: 2023-10-11T15:27:41.045 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.045 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.046 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.046 DEBUG:boto:Params: {} 2023-10-11T15:27:41.047 DEBUG:boto:Token: None 2023-10-11T15:27:41.047 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj46 2023-10-11T15:27:41.048 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:90aMhEGKZt+0wMJDjUtm0ualOe0= 2023-10-11T15:27:41.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:90aMhEGKZt+0wMJDjUtm0ualOe0=', 'Content-Length': '0'} 2023-10-11T15:27:41.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c44a12fca528685-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.051 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.051 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.052 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.052 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.053 DEBUG:boto:Method: GET 2023-10-11T15:27:41.053 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.054 DEBUG:boto:Data: 2023-10-11T15:27:41.054 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.055 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.055 DEBUG:boto:Params: {} 2023-10-11T15:27:41.056 DEBUG:boto:Token: None 2023-10-11T15:27:41.056 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.057 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe124c30ce4aefe9-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.060 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.061 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.062 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.062 DEBUG:boto:path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.063 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.063 DEBUG:boto:Method: GET 2023-10-11T15:27:41.064 DEBUG:boto:Path: /dghqvd-59/?prefix=obj46 2023-10-11T15:27:41.064 DEBUG:boto:Data: 2023-10-11T15:27:41.065 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.065 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.065 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.066 DEBUG:boto:Params: {} 2023-10-11T15:27:41.066 DEBUG:boto:Token: None 2023-10-11T15:27:41.067 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.067 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000517ce3ca76abd237-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.114 DEBUG:boto:b'dghqvd-59obj461000falseobj462023-10-11T15:21:14.344Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:41.116 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.116 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.117 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.117 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.117 DEBUG:boto:Method: GET 2023-10-11T15:27:41.118 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.118 DEBUG:boto:Data: 2023-10-11T15:27:41.119 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.120 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.120 DEBUG:boto:Params: {} 2023-10-11T15:27:41.121 DEBUG:boto:Token: None 2023-10-11T15:27:41.121 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY= 2023-10-11T15:27:41.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY=', 'Content-Length': '0'} 2023-10-11T15:27:41.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009358451dc2e4b75a-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.125 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.125 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.126 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.126 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.127 DEBUG:boto:Method: GET 2023-10-11T15:27:41.127 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.128 DEBUG:boto:Data: 2023-10-11T15:27:41.128 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.129 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.129 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.130 DEBUG:boto:Params: {} 2023-10-11T15:27:41.130 DEBUG:boto:Token: None 2023-10-11T15:27:41.131 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.131 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY= 2023-10-11T15:27:41.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY=', 'Content-Length': '0'} 2023-10-11T15:27:41.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c45eb5b33bdf21-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.135 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.135 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.135 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.136 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.136 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.136 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.137 DEBUG:boto:Data: 2023-10-11T15:27:41.137 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.138 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.138 DEBUG:boto:Params: {} 2023-10-11T15:27:41.139 DEBUG:boto:Token: None 2023-10-11T15:27:41.139 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5KRVM1SF+jYTb7CefQT2qYy43Rs= 2023-10-11T15:27:41.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5KRVM1SF+jYTb7CefQT2qYy43Rs=', 'Content-Length': '0'} 2023-10-11T15:27:41.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e7867050bb84c8-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.143 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.143 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.144 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.144 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.144 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.145 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.145 DEBUG:boto:Data: 2023-10-11T15:27:41.146 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.147 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.147 DEBUG:boto:Params: {} 2023-10-11T15:27:41.148 DEBUG:boto:Token: None 2023-10-11T15:27:41.148 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5KRVM1SF+jYTb7CefQT2qYy43Rs= 2023-10-11T15:27:41.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5KRVM1SF+jYTb7CefQT2qYy43Rs=', 'Content-Length': '0'} 2023-10-11T15:27:41.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d089cd454be34cfc-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:41.152 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.152 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.153 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.153 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.154 DEBUG:boto:Method: GET 2023-10-11T15:27:41.154 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.155 DEBUG:boto:Data: 2023-10-11T15:27:41.155 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.156 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.156 DEBUG:boto:Params: {} 2023-10-11T15:27:41.157 DEBUG:boto:Token: None 2023-10-11T15:27:41.157 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY= 2023-10-11T15:27:41.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY=', 'Content-Length': '0'} 2023-10-11T15:27:41.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db52dd52cd494df-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.161 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.161 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.162 DEBUG:boto:path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.162 DEBUG:boto:auth_path=/dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.162 DEBUG:boto:Method: GET 2023-10-11T15:27:41.163 DEBUG:boto:Path: /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.163 DEBUG:boto:Data: 2023-10-11T15:27:41.164 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.165 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.165 DEBUG:boto:Params: {} 2023-10-11T15:27:41.165 DEBUG:boto:Token: None 2023-10-11T15:27:41.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51?versionId=null 2023-10-11T15:27:41.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY= 2023-10-11T15:27:41.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Z37bfrYXLKKoaunKjjVv1ebuugY=', 'Content-Length': '0'} 2023-10-11T15:27:41.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9910c921d9cd02-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.170 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.170 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.171 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.171 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.172 DEBUG:boto:Method: GET 2023-10-11T15:27:41.172 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.172 DEBUG:boto:Data: 2023-10-11T15:27:41.173 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.174 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.174 DEBUG:boto:Params: {} 2023-10-11T15:27:41.175 DEBUG:boto:Token: None 2023-10-11T15:27:41.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046cec149c4adcaf8-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.180 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.180 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.181 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.181 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.182 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.182 DEBUG:boto:Method: GET 2023-10-11T15:27:41.183 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.183 DEBUG:boto:Data: 2023-10-11T15:27:41.183 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.184 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.184 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.185 DEBUG:boto:Params: {} 2023-10-11T15:27:41.185 DEBUG:boto:Token: None 2023-10-11T15:27:41.186 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.186 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423a4b5879695b9c-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.191 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.192 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.192 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.193 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.193 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:41.194 DEBUG:boto:Data: 2023-10-11T15:27:41.194 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.195 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.195 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.196 DEBUG:boto:Params: {} 2023-10-11T15:27:41.196 DEBUG:boto:Token: None 2023-10-11T15:27:41.197 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51 2023-10-11T15:27:41.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AMPTVIMJO1ZLUPbgP5YxkOtl9R8= 2023-10-11T15:27:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AMPTVIMJO1ZLUPbgP5YxkOtl9R8=', 'Content-Length': '0'} 2023-10-11T15:27:41.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4c9b34a191bd78-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.200 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.201 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.201 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.202 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:41.202 DEBUG:boto:Data: 2023-10-11T15:27:41.203 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.203 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.204 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.204 DEBUG:boto:Params: {} 2023-10-11T15:27:41.205 DEBUG:boto:Token: None 2023-10-11T15:27:41.205 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51 2023-10-11T15:27:41.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AMPTVIMJO1ZLUPbgP5YxkOtl9R8= 2023-10-11T15:27:41.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AMPTVIMJO1ZLUPbgP5YxkOtl9R8=', 'Content-Length': '0'} 2023-10-11T15:27:41.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127c55d7f7dead0c-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:27:41.209 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.210 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.210 DEBUG:boto:Method: GET 2023-10-11T15:27:41.210 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:41.211 DEBUG:boto:Data: 2023-10-11T15:27:41.211 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.212 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.212 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.213 DEBUG:boto:Params: {} 2023-10-11T15:27:41.213 DEBUG:boto:Token: None 2023-10-11T15:27:41.214 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51 2023-10-11T15:27:41.214 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QO4PpBbfFpV1J8IuF0ZsGhvI6xA= 2023-10-11T15:27:41.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QO4PpBbfFpV1J8IuF0ZsGhvI6xA=', 'Content-Length': '0'} 2023-10-11T15:27:41.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000febcd0a7c15e1ffe-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.217 DEBUG:boto:path=/dghqvd-59/obj51 2023-10-11T15:27:41.218 DEBUG:boto:auth_path=/dghqvd-59/obj51 2023-10-11T15:27:41.218 DEBUG:boto:Method: GET 2023-10-11T15:27:41.219 DEBUG:boto:Path: /dghqvd-59/obj51 2023-10-11T15:27:41.219 DEBUG:boto:Data: 2023-10-11T15:27:41.220 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.221 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.221 DEBUG:boto:Params: {} 2023-10-11T15:27:41.222 DEBUG:boto:Token: None 2023-10-11T15:27:41.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj51 2023-10-11T15:27:41.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QO4PpBbfFpV1J8IuF0ZsGhvI6xA= 2023-10-11T15:27:41.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QO4PpBbfFpV1J8IuF0ZsGhvI6xA=', 'Content-Length': '0'} 2023-10-11T15:27:41.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073336c7d1d63118a-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.266 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.267 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.267 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.268 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.268 DEBUG:boto:Method: GET 2023-10-11T15:27:41.269 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.269 DEBUG:boto:Data: 2023-10-11T15:27:41.270 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.270 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.271 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.271 DEBUG:boto:Params: {} 2023-10-11T15:27:41.271 DEBUG:boto:Token: None 2023-10-11T15:27:41.272 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000669a0374f84b904e-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.276 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.277 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.277 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.278 DEBUG:boto:path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.278 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.278 DEBUG:boto:Method: GET 2023-10-11T15:27:41.279 DEBUG:boto:Path: /dghqvd-59/?prefix=obj51 2023-10-11T15:27:41.279 DEBUG:boto:Data: 2023-10-11T15:27:41.280 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.280 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.281 DEBUG:boto:Params: {} 2023-10-11T15:27:41.281 DEBUG:boto:Token: None 2023-10-11T15:27:41.282 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.282 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005287c08c52c6e95d-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.287 DEBUG:boto:b'dghqvd-59obj511000falseobj512023-10-11T15:16:32.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:41.288 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.289 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.289 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.290 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.290 DEBUG:boto:Method: GET 2023-10-11T15:27:41.291 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.291 DEBUG:boto:Data: 2023-10-11T15:27:41.292 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.292 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.293 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.293 DEBUG:boto:Params: {} 2023-10-11T15:27:41.293 DEBUG:boto:Token: None 2023-10-11T15:27:41.294 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc= 2023-10-11T15:27:41.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc=', 'Content-Length': '0'} 2023-10-11T15:27:41.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f65bea7840eaed-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.338 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.339 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.339 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.340 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.340 DEBUG:boto:Method: GET 2023-10-11T15:27:41.340 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.341 DEBUG:boto:Data: 2023-10-11T15:27:41.341 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.342 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.343 DEBUG:boto:Params: {} 2023-10-11T15:27:41.343 DEBUG:boto:Token: None 2023-10-11T15:27:41.343 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc= 2023-10-11T15:27:41.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc=', 'Content-Length': '0'} 2023-10-11T15:27:41.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec36dc8bdabb28e6-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.347 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.348 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.348 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.349 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.349 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.349 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.350 DEBUG:boto:Data: 2023-10-11T15:27:41.350 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.351 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.351 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.351 DEBUG:boto:Params: {} 2023-10-11T15:27:41.352 DEBUG:boto:Token: None 2023-10-11T15:27:41.352 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.353 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xCC4SridUh+/BfbGlGz4YoHPDbc= 2023-10-11T15:27:41.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xCC4SridUh+/BfbGlGz4YoHPDbc=', 'Content-Length': '0'} 2023-10-11T15:27:41.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7f0d0b5e6872d8-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.356 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.356 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.356 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.357 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.357 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.358 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.358 DEBUG:boto:Data: 2023-10-11T15:27:41.358 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.359 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.359 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.360 DEBUG:boto:Params: {} 2023-10-11T15:27:41.360 DEBUG:boto:Token: None 2023-10-11T15:27:41.361 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.361 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xCC4SridUh+/BfbGlGz4YoHPDbc= 2023-10-11T15:27:41.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xCC4SridUh+/BfbGlGz4YoHPDbc=', 'Content-Length': '0'} 2023-10-11T15:27:41.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0f6a00d1b345196-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:41.364 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.365 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.365 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.365 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.366 DEBUG:boto:Method: GET 2023-10-11T15:27:41.366 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.367 DEBUG:boto:Data: 2023-10-11T15:27:41.367 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.368 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.368 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.368 DEBUG:boto:Params: {} 2023-10-11T15:27:41.369 DEBUG:boto:Token: None 2023-10-11T15:27:41.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc= 2023-10-11T15:27:41.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc=', 'Content-Length': '0'} 2023-10-11T15:27:41.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6209b6f14fa2736-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.373 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.373 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.374 DEBUG:boto:path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.374 DEBUG:boto:auth_path=/dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.375 DEBUG:boto:Method: GET 2023-10-11T15:27:41.375 DEBUG:boto:Path: /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.375 DEBUG:boto:Data: 2023-10-11T15:27:41.376 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.377 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.377 DEBUG:boto:Params: {} 2023-10-11T15:27:41.378 DEBUG:boto:Token: None 2023-10-11T15:27:41.378 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55?versionId=null 2023-10-11T15:27:41.379 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc= 2023-10-11T15:27:41.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bWoxlbf54IlAnvHa2b78mlWrCxc=', 'Content-Length': '0'} 2023-10-11T15:27:41.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001993083d21b83629-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.382 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.382 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.383 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.383 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.384 DEBUG:boto:Method: GET 2023-10-11T15:27:41.384 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.385 DEBUG:boto:Data: 2023-10-11T15:27:41.385 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.386 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.386 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.386 DEBUG:boto:Params: {} 2023-10-11T15:27:41.387 DEBUG:boto:Token: None 2023-10-11T15:27:41.387 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.388 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a48cf85e33da9b25-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.392 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.393 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.393 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.394 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.394 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.395 DEBUG:boto:Method: GET 2023-10-11T15:27:41.395 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.395 DEBUG:boto:Data: 2023-10-11T15:27:41.396 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.396 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.397 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.397 DEBUG:boto:Params: {} 2023-10-11T15:27:41.398 DEBUG:boto:Token: None 2023-10-11T15:27:41.398 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000346135c9e2a0a36a-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.404 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.404 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.405 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.405 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.406 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:41.407 DEBUG:boto:Data: 2023-10-11T15:27:41.407 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.408 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.409 DEBUG:boto:Params: {} 2023-10-11T15:27:41.409 DEBUG:boto:Token: None 2023-10-11T15:27:41.410 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55 2023-10-11T15:27:41.410 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xByR4CQ10MURW1yTQ+W5UNI3plE= 2023-10-11T15:27:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xByR4CQ10MURW1yTQ+W5UNI3plE=', 'Content-Length': '0'} 2023-10-11T15:27:41.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000383cae9fcee7173b-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.413 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.414 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.414 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.415 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:41.415 DEBUG:boto:Data: 2023-10-11T15:27:41.415 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.416 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.417 DEBUG:boto:Params: {} 2023-10-11T15:27:41.417 DEBUG:boto:Token: None 2023-10-11T15:27:41.418 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55 2023-10-11T15:27:41.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xByR4CQ10MURW1yTQ+W5UNI3plE= 2023-10-11T15:27:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xByR4CQ10MURW1yTQ+W5UNI3plE=', 'Content-Length': '0'} 2023-10-11T15:27:41.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc8cd968940319f-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:27:41.422 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.422 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.423 DEBUG:boto:Method: GET 2023-10-11T15:27:41.423 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:41.424 DEBUG:boto:Data: 2023-10-11T15:27:41.424 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.425 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.426 DEBUG:boto:Params: {} 2023-10-11T15:27:41.427 DEBUG:boto:Token: None 2023-10-11T15:27:41.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55 2023-10-11T15:27:41.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EXhxhMFPtAixPUNKtE0cIM6q+Zk= 2023-10-11T15:27:41.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EXhxhMFPtAixPUNKtE0cIM6q+Zk=', 'Content-Length': '0'} 2023-10-11T15:27:41.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2766d48b05dbe6-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.474 DEBUG:boto:path=/dghqvd-59/obj55 2023-10-11T15:27:41.475 DEBUG:boto:auth_path=/dghqvd-59/obj55 2023-10-11T15:27:41.476 DEBUG:boto:Method: GET 2023-10-11T15:27:41.476 DEBUG:boto:Path: /dghqvd-59/obj55 2023-10-11T15:27:41.476 DEBUG:boto:Data: 2023-10-11T15:27:41.477 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.478 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.478 DEBUG:boto:Params: {} 2023-10-11T15:27:41.479 DEBUG:boto:Token: None 2023-10-11T15:27:41.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj55 2023-10-11T15:27:41.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EXhxhMFPtAixPUNKtE0cIM6q+Zk= 2023-10-11T15:27:41.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EXhxhMFPtAixPUNKtE0cIM6q+Zk=', 'Content-Length': '0'} 2023-10-11T15:27:41.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007000c12cc3139923-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.483 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.484 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.484 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.484 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.485 DEBUG:boto:Method: GET 2023-10-11T15:27:41.485 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.486 DEBUG:boto:Data: 2023-10-11T15:27:41.486 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.487 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.487 DEBUG:boto:Params: {} 2023-10-11T15:27:41.488 DEBUG:boto:Token: None 2023-10-11T15:27:41.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5eb6f34effa024b-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.492 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.493 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.494 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.494 DEBUG:boto:path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.494 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.495 DEBUG:boto:Method: GET 2023-10-11T15:27:41.495 DEBUG:boto:Path: /dghqvd-59/?prefix=obj55 2023-10-11T15:27:41.496 DEBUG:boto:Data: 2023-10-11T15:27:41.496 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.496 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.497 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.497 DEBUG:boto:Params: {} 2023-10-11T15:27:41.498 DEBUG:boto:Token: None 2023-10-11T15:27:41.498 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177f495bed55a258-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.504 DEBUG:boto:b'dghqvd-59obj551000falseobj552023-10-11T15:16:06.032Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:41.505 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.505 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.506 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.506 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.507 DEBUG:boto:Method: GET 2023-10-11T15:27:41.507 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.507 DEBUG:boto:Data: 2023-10-11T15:27:41.508 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.509 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.509 DEBUG:boto:Params: {} 2023-10-11T15:27:41.509 DEBUG:boto:Token: None 2023-10-11T15:27:41.510 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas= 2023-10-11T15:27:41.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas=', 'Content-Length': '0'} 2023-10-11T15:27:41.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000155903aab89e36d-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.514 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.515 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.515 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.515 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.516 DEBUG:boto:Method: GET 2023-10-11T15:27:41.516 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.517 DEBUG:boto:Data: 2023-10-11T15:27:41.517 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.518 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.518 DEBUG:boto:Params: {} 2023-10-11T15:27:41.519 DEBUG:boto:Token: None 2023-10-11T15:27:41.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas= 2023-10-11T15:27:41.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas=', 'Content-Length': '0'} 2023-10-11T15:27:41.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a52529eec135e106-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.523 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.523 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.524 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.524 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.525 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.525 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.525 DEBUG:boto:Data: 2023-10-11T15:27:41.526 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.527 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.527 DEBUG:boto:Params: {} 2023-10-11T15:27:41.528 DEBUG:boto:Token: None 2023-10-11T15:27:41.528 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:spUTXbP929VKwDXR+CQrnoVLEtQ= 2023-10-11T15:27:41.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:spUTXbP929VKwDXR+CQrnoVLEtQ=', 'Content-Length': '0'} 2023-10-11T15:27:41.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ea2f13f7c91d72-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.532 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.532 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.533 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.533 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.533 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.534 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.534 DEBUG:boto:Data: 2023-10-11T15:27:41.535 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.536 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.536 DEBUG:boto:Params: {} 2023-10-11T15:27:41.537 DEBUG:boto:Token: None 2023-10-11T15:27:41.537 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.538 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:spUTXbP929VKwDXR+CQrnoVLEtQ= 2023-10-11T15:27:41.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:spUTXbP929VKwDXR+CQrnoVLEtQ=', 'Content-Length': '0'} 2023-10-11T15:27:41.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002399babaf4fe52de-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:41.543 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.544 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.544 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.545 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.545 DEBUG:boto:Method: GET 2023-10-11T15:27:41.545 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.546 DEBUG:boto:Data: 2023-10-11T15:27:41.546 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.547 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.547 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.548 DEBUG:boto:Params: {} 2023-10-11T15:27:41.548 DEBUG:boto:Token: None 2023-10-11T15:27:41.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.549 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas= 2023-10-11T15:27:41.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas=', 'Content-Length': '0'} 2023-10-11T15:27:41.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e402c9cd5736fa-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.594 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.595 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.595 DEBUG:boto:path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.596 DEBUG:boto:auth_path=/dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.596 DEBUG:boto:Method: GET 2023-10-11T15:27:41.597 DEBUG:boto:Path: /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.597 DEBUG:boto:Data: 2023-10-11T15:27:41.598 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.598 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.599 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.599 DEBUG:boto:Params: {} 2023-10-11T15:27:41.599 DEBUG:boto:Token: None 2023-10-11T15:27:41.600 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57?versionId=null 2023-10-11T15:27:41.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas= 2023-10-11T15:27:41.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U1OZ5M0hFmeYm6hdEexHJtXgcas=', 'Content-Length': '0'} 2023-10-11T15:27:41.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000149256dc54c047fd-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.604 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.604 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.605 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.605 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.606 DEBUG:boto:Method: GET 2023-10-11T15:27:41.606 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.606 DEBUG:boto:Data: 2023-10-11T15:27:41.607 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.608 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.608 DEBUG:boto:Params: {} 2023-10-11T15:27:41.609 DEBUG:boto:Token: None 2023-10-11T15:27:41.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0a1c8b4d87c6c48-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.613 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.614 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.614 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.615 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.615 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.616 DEBUG:boto:Method: GET 2023-10-11T15:27:41.616 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.617 DEBUG:boto:Data: 2023-10-11T15:27:41.617 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.618 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.618 DEBUG:boto:Params: {} 2023-10-11T15:27:41.619 DEBUG:boto:Token: None 2023-10-11T15:27:41.619 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16f3a9435247490-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.625 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.626 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.626 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.627 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.627 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:41.628 DEBUG:boto:Data: 2023-10-11T15:27:41.628 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.629 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.629 DEBUG:boto:Params: {} 2023-10-11T15:27:41.630 DEBUG:boto:Token: None 2023-10-11T15:27:41.630 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57 2023-10-11T15:27:41.631 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6v3M1iNMq2O4SLBctyXL9eBYKE= 2023-10-11T15:27:41.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6v3M1iNMq2O4SLBctyXL9eBYKE=', 'Content-Length': '0'} 2023-10-11T15:27:41.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008874a43ca1c4c67f-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.634 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.634 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.634 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.635 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:41.635 DEBUG:boto:Data: 2023-10-11T15:27:41.636 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.636 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.636 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.637 DEBUG:boto:Params: {} 2023-10-11T15:27:41.637 DEBUG:boto:Token: None 2023-10-11T15:27:41.638 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57 2023-10-11T15:27:41.638 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L6v3M1iNMq2O4SLBctyXL9eBYKE= 2023-10-11T15:27:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L6v3M1iNMq2O4SLBctyXL9eBYKE=', 'Content-Length': '0'} 2023-10-11T15:27:41.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9da067aea96b51-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:27:41.642 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.642 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.643 DEBUG:boto:Method: GET 2023-10-11T15:27:41.643 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:41.644 DEBUG:boto:Data: 2023-10-11T15:27:41.644 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.645 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.646 DEBUG:boto:Params: {} 2023-10-11T15:27:41.646 DEBUG:boto:Token: None 2023-10-11T15:27:41.646 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57 2023-10-11T15:27:41.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vHzo9FxdMfgk1TXiQIY7GlD/Bo8= 2023-10-11T15:27:41.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vHzo9FxdMfgk1TXiQIY7GlD/Bo8=', 'Content-Length': '0'} 2023-10-11T15:27:41.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f611fb8ad4eea5b8-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.650 DEBUG:boto:path=/dghqvd-59/obj57 2023-10-11T15:27:41.651 DEBUG:boto:auth_path=/dghqvd-59/obj57 2023-10-11T15:27:41.651 DEBUG:boto:Method: GET 2023-10-11T15:27:41.652 DEBUG:boto:Path: /dghqvd-59/obj57 2023-10-11T15:27:41.652 DEBUG:boto:Data: 2023-10-11T15:27:41.653 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.653 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.653 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.654 DEBUG:boto:Params: {} 2023-10-11T15:27:41.654 DEBUG:boto:Token: None 2023-10-11T15:27:41.655 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj57 2023-10-11T15:27:41.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vHzo9FxdMfgk1TXiQIY7GlD/Bo8= 2023-10-11T15:27:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vHzo9FxdMfgk1TXiQIY7GlD/Bo8=', 'Content-Length': '0'} 2023-10-11T15:27:41.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015a73977d4bf5c8c-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.659 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.659 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.659 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.660 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.660 DEBUG:boto:Method: GET 2023-10-11T15:27:41.661 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.661 DEBUG:boto:Data: 2023-10-11T15:27:41.662 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.662 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.663 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.663 DEBUG:boto:Params: {} 2023-10-11T15:27:41.663 DEBUG:boto:Token: None 2023-10-11T15:27:41.664 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.664 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0e71d0e06e3b36-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.668 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.669 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.669 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.669 DEBUG:boto:path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.670 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.670 DEBUG:boto:Method: GET 2023-10-11T15:27:41.671 DEBUG:boto:Path: /dghqvd-59/?prefix=obj57 2023-10-11T15:27:41.671 DEBUG:boto:Data: 2023-10-11T15:27:41.672 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.673 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.674 DEBUG:boto:Params: {} 2023-10-11T15:27:41.674 DEBUG:boto:Token: None 2023-10-11T15:27:41.675 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6dbf1c6a87a33b-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.681 DEBUG:boto:b'dghqvd-59obj571000falseobj572023-10-11T15:18:55.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:41.682 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.682 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.683 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.683 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.684 DEBUG:boto:Method: GET 2023-10-11T15:27:41.684 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.685 DEBUG:boto:Data: 2023-10-11T15:27:41.685 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.686 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.687 DEBUG:boto:Params: {} 2023-10-11T15:27:41.687 DEBUG:boto:Token: None 2023-10-11T15:27:41.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE= 2023-10-11T15:27:41.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE=', 'Content-Length': '0'} 2023-10-11T15:27:41.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2a1601876d5b91f-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.692 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.692 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.693 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.693 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.694 DEBUG:boto:Method: GET 2023-10-11T15:27:41.694 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.695 DEBUG:boto:Data: 2023-10-11T15:27:41.695 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.695 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.696 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.696 DEBUG:boto:Params: {} 2023-10-11T15:27:41.697 DEBUG:boto:Token: None 2023-10-11T15:27:41.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE= 2023-10-11T15:27:41.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE=', 'Content-Length': '0'} 2023-10-11T15:27:41.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e876fbb195345664-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.742 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.743 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.743 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.744 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.744 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.744 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.745 DEBUG:boto:Data: 2023-10-11T15:27:41.745 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.746 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.746 DEBUG:boto:Params: {} 2023-10-11T15:27:41.747 DEBUG:boto:Token: None 2023-10-11T15:27:41.747 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yq28D5qZeBMqKLtzomli8Kurf/c= 2023-10-11T15:27:41.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yq28D5qZeBMqKLtzomli8Kurf/c=', 'Content-Length': '0'} 2023-10-11T15:27:41.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f354a2662594ef36-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.752 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.752 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.753 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.753 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.754 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.754 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.755 DEBUG:boto:Data: 2023-10-11T15:27:41.755 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.755 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.756 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.756 DEBUG:boto:Params: {} 2023-10-11T15:27:41.756 DEBUG:boto:Token: None 2023-10-11T15:27:41.757 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.757 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Yq28D5qZeBMqKLtzomli8Kurf/c= 2023-10-11T15:27:41.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Yq28D5qZeBMqKLtzomli8Kurf/c=', 'Content-Length': '0'} 2023-10-11T15:27:41.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2974a796adb2c88-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:41.761 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.761 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.762 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.762 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.763 DEBUG:boto:Method: GET 2023-10-11T15:27:41.763 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.763 DEBUG:boto:Data: 2023-10-11T15:27:41.764 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.764 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.765 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.765 DEBUG:boto:Params: {} 2023-10-11T15:27:41.766 DEBUG:boto:Token: None 2023-10-11T15:27:41.766 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE= 2023-10-11T15:27:41.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE=', 'Content-Length': '0'} 2023-10-11T15:27:41.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074233e5c1b1ad9f6-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.810 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.811 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.811 DEBUG:boto:path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.812 DEBUG:boto:auth_path=/dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.812 DEBUG:boto:Method: GET 2023-10-11T15:27:41.812 DEBUG:boto:Path: /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.813 DEBUG:boto:Data: 2023-10-11T15:27:41.813 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.814 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.815 DEBUG:boto:Params: {} 2023-10-11T15:27:41.815 DEBUG:boto:Token: None 2023-10-11T15:27:41.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58?versionId=null 2023-10-11T15:27:41.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE= 2023-10-11T15:27:41.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQN/s4jP9H06q6YNtejiyT2KPsE=', 'Content-Length': '0'} 2023-10-11T15:27:41.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec8a44e521781ab-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.820 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.820 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.821 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.821 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.821 DEBUG:boto:Method: GET 2023-10-11T15:27:41.822 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.822 DEBUG:boto:Data: 2023-10-11T15:27:41.823 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.824 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.824 DEBUG:boto:Params: {} 2023-10-11T15:27:41.824 DEBUG:boto:Token: None 2023-10-11T15:27:41.825 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590e739fa359ae32-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.829 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.830 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.830 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.831 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.831 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.831 DEBUG:boto:Method: GET 2023-10-11T15:27:41.832 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.832 DEBUG:boto:Data: 2023-10-11T15:27:41.833 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.834 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.834 DEBUG:boto:Params: {} 2023-10-11T15:27:41.835 DEBUG:boto:Token: None 2023-10-11T15:27:41.835 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f93142bc3d8b788-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.841 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.842 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.842 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.842 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.843 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:41.843 DEBUG:boto:Data: 2023-10-11T15:27:41.844 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.844 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.845 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.845 DEBUG:boto:Params: {} 2023-10-11T15:27:41.845 DEBUG:boto:Token: None 2023-10-11T15:27:41.846 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58 2023-10-11T15:27:41.846 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cl3UMB7zpldEU9n0YoT6/ic/L84= 2023-10-11T15:27:41.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cl3UMB7zpldEU9n0YoT6/ic/L84=', 'Content-Length': '0'} 2023-10-11T15:27:41.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019edfb22464d9bb8-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.850 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.850 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.851 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.851 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:41.852 DEBUG:boto:Data: 2023-10-11T15:27:41.852 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.852 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.853 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.853 DEBUG:boto:Params: {} 2023-10-11T15:27:41.854 DEBUG:boto:Token: None 2023-10-11T15:27:41.854 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58 2023-10-11T15:27:41.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cl3UMB7zpldEU9n0YoT6/ic/L84= 2023-10-11T15:27:41.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cl3UMB7zpldEU9n0YoT6/ic/L84=', 'Content-Length': '0'} 2023-10-11T15:27:41.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006dde4bf083224e7-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:27:41.858 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.858 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.859 DEBUG:boto:Method: GET 2023-10-11T15:27:41.859 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:41.860 DEBUG:boto:Data: 2023-10-11T15:27:41.860 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.861 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.862 DEBUG:boto:Params: {} 2023-10-11T15:27:41.862 DEBUG:boto:Token: None 2023-10-11T15:27:41.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58 2023-10-11T15:27:41.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xuaanbbDdtMTxH2asXbBvA1cocw= 2023-10-11T15:27:41.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xuaanbbDdtMTxH2asXbBvA1cocw=', 'Content-Length': '0'} 2023-10-11T15:27:41.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a9fa1f10203d1ae-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.866 DEBUG:boto:path=/dghqvd-59/obj58 2023-10-11T15:27:41.867 DEBUG:boto:auth_path=/dghqvd-59/obj58 2023-10-11T15:27:41.867 DEBUG:boto:Method: GET 2023-10-11T15:27:41.868 DEBUG:boto:Path: /dghqvd-59/obj58 2023-10-11T15:27:41.868 DEBUG:boto:Data: 2023-10-11T15:27:41.868 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.869 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.870 DEBUG:boto:Params: {} 2023-10-11T15:27:41.870 DEBUG:boto:Token: None 2023-10-11T15:27:41.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj58 2023-10-11T15:27:41.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xuaanbbDdtMTxH2asXbBvA1cocw= 2023-10-11T15:27:41.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xuaanbbDdtMTxH2asXbBvA1cocw=', 'Content-Length': '0'} 2023-10-11T15:27:41.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003275e00e67bae6bf-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.914 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.915 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.915 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.916 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.916 DEBUG:boto:Method: GET 2023-10-11T15:27:41.917 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.917 DEBUG:boto:Data: 2023-10-11T15:27:41.918 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.919 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.919 DEBUG:boto:Params: {} 2023-10-11T15:27:41.919 DEBUG:boto:Token: None 2023-10-11T15:27:41.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bee5c2dec127d2f-006526beed-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.924 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.925 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:41.926 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:41.926 DEBUG:boto:path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.927 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.927 DEBUG:boto:Method: GET 2023-10-11T15:27:41.927 DEBUG:boto:Path: /dghqvd-59/?prefix=obj58 2023-10-11T15:27:41.928 DEBUG:boto:Data: 2023-10-11T15:27:41.928 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.929 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.930 DEBUG:boto:Params: {} 2023-10-11T15:27:41.930 DEBUG:boto:Token: None 2023-10-11T15:27:41.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/ 2023-10-11T15:27:41.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0= 2023-10-11T15:27:41.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fXuRh7c/iOAMZFFFZAcKOObSjQ0=', 'Content-Length': '0'} 2023-10-11T15:27:41.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df08cc1b790087b6-006526beed-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.937 DEBUG:boto:b'dghqvd-59obj581000falseobj582023-10-11T15:21:59.390Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:41.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:41.938 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:41.938 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:41.939 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.939 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.940 DEBUG:boto:Method: GET 2023-10-11T15:27:41.940 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.941 DEBUG:boto:Data: 2023-10-11T15:27:41.941 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.942 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.942 DEBUG:boto:Params: {} 2023-10-11T15:27:41.943 DEBUG:boto:Token: None 2023-10-11T15:27:41.943 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8= 2023-10-11T15:27:41.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8=', 'Content-Length': '0'} 2023-10-11T15:27:41.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000450948c2b4ebdd21-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.947 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:41.947 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:41.948 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.948 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.949 DEBUG:boto:Method: GET 2023-10-11T15:27:41.949 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.950 DEBUG:boto:Data: 2023-10-11T15:27:41.950 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.951 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.951 DEBUG:boto:Params: {} 2023-10-11T15:27:41.953 DEBUG:boto:Token: None 2023-10-11T15:27:41.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.954 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8= 2023-10-11T15:27:41.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8=', 'Content-Length': '0'} 2023-10-11T15:27:41.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c27bf4ada89c0e-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.957 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:41.958 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:41.958 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.959 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.959 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.959 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.960 DEBUG:boto:Data: 2023-10-11T15:27:41.960 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.961 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.961 DEBUG:boto:Params: {} 2023-10-11T15:27:41.962 DEBUG:boto:Token: None 2023-10-11T15:27:41.962 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.963 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eZ3e6oFRXVUQIwxpIN1Xal3WMfY= 2023-10-11T15:27:41.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eZ3e6oFRXVUQIwxpIN1Xal3WMfY=', 'Content-Length': '0'} 2023-10-11T15:27:41.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d1d96d98ece267-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.966 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:41.966 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:41.967 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.967 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.968 DEBUG:boto:Method: HEAD 2023-10-11T15:27:41.968 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.969 DEBUG:boto:Data: 2023-10-11T15:27:41.969 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.969 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:41.970 DEBUG:boto:Port: 8000 2023-10-11T15:27:41.970 DEBUG:boto:Params: {} 2023-10-11T15:27:41.970 DEBUG:boto:Token: None 2023-10-11T15:27:41.971 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.971 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eZ3e6oFRXVUQIwxpIN1Xal3WMfY= 2023-10-11T15:27:41.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eZ3e6oFRXVUQIwxpIN1Xal3WMfY=', 'Content-Length': '0'} 2023-10-11T15:27:41.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aef43d498e0ac62-006526beed-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:41.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:41.975 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:41.975 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:41.976 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.976 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.977 DEBUG:boto:Method: GET 2023-10-11T15:27:41.977 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.978 DEBUG:boto:Data: 2023-10-11T15:27:41.978 DEBUG:boto:Headers: {} 2023-10-11T15:27:41.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:41.979 DEBUG:boto:Port: 8001 2023-10-11T15:27:41.980 DEBUG:boto:Params: {} 2023-10-11T15:27:41.980 DEBUG:boto:Token: None 2023-10-11T15:27:41.981 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:41 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:41.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8= 2023-10-11T15:27:41.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:41 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PZgFFQJ64QATAbJeqX86tAvwFf8=', 'Content-Length': '0'} 2023-10-11T15:27:41.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000207a5633fd79e9fe-006526beed-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.026 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:42.027 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:42.027 DEBUG:boto:path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:42.027 DEBUG:boto:auth_path=/dghqvd-59/obj62?versionId=null 2023-10-11T15:27:42.028 DEBUG:boto:Method: GET 2023-10-11T15:27:42.028 DEBUG:boto:Path: /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:42.029 DEBUG:boto:Data: 2023-10-11T15:27:42.029 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.030 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.030 DEBUG:boto:Params: {} 2023-10-11T15:27:42.031 DEBUG:boto:Token: None 2023-10-11T15:27:42.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj62?versionId=null 2023-10-11T15:27:42.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZdZ7jBlYWlDDtIdTzqhum7g5UGo= 2023-10-11T15:27:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZdZ7jBlYWlDDtIdTzqhum7g5UGo=', 'Content-Length': '0'} 2023-10-11T15:27:42.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000366ccadde742f94b-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.035 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.035 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.036 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.036 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.036 DEBUG:boto:Method: GET 2023-10-11T15:27:42.037 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.037 DEBUG:boto:Data: 2023-10-11T15:27:42.038 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.038 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.039 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.039 DEBUG:boto:Params: {} 2023-10-11T15:27:42.040 DEBUG:boto:Token: None 2023-10-11T15:27:42.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ab73ef61e5d630-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.044 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.045 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.046 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.046 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.047 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.047 DEBUG:boto:Method: GET 2023-10-11T15:27:42.047 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.048 DEBUG:boto:Data: 2023-10-11T15:27:42.048 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.049 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.049 DEBUG:boto:Params: {} 2023-10-11T15:27:42.050 DEBUG:boto:Token: None 2023-10-11T15:27:42.050 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e27b7684a8ad2b-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.098 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.099 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:42.099 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:42.100 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.100 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:42.101 DEBUG:boto:Data: 2023-10-11T15:27:42.101 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.102 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.102 DEBUG:boto:Params: {} 2023-10-11T15:27:42.103 DEBUG:boto:Token: None 2023-10-11T15:27:42.103 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj62 2023-10-11T15:27:42.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OzQlR0jHxeHtf8NTCzGXr4SgeVM= 2023-10-11T15:27:42.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OzQlR0jHxeHtf8NTCzGXr4SgeVM=', 'Content-Length': '0'} 2023-10-11T15:27:42.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e05bc586d09ea4a5-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.107 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:42.107 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:42.107 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.108 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:42.108 DEBUG:boto:Data: 2023-10-11T15:27:42.109 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.109 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.109 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.110 DEBUG:boto:Params: {} 2023-10-11T15:27:42.111 DEBUG:boto:Token: None 2023-10-11T15:27:42.111 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj62 2023-10-11T15:27:42.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OzQlR0jHxeHtf8NTCzGXr4SgeVM= 2023-10-11T15:27:42.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OzQlR0jHxeHtf8NTCzGXr4SgeVM=', 'Content-Length': '0'} 2023-10-11T15:27:42.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040173154e4f3c842-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:27:42.115 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:42.115 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:42.116 DEBUG:boto:Method: GET 2023-10-11T15:27:42.116 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:42.117 DEBUG:boto:Data: 2023-10-11T15:27:42.117 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.118 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.118 DEBUG:boto:Params: {} 2023-10-11T15:27:42.119 DEBUG:boto:Token: None 2023-10-11T15:27:42.119 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj62 2023-10-11T15:27:42.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ZZbyGtLr1pMJdJzwyNUNSiIFII= 2023-10-11T15:27:42.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ZZbyGtLr1pMJdJzwyNUNSiIFII=', 'Content-Length': '0'} 2023-10-11T15:27:42.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ab5a559ec29582-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.123 DEBUG:boto:path=/dghqvd-59/obj62 2023-10-11T15:27:42.123 DEBUG:boto:auth_path=/dghqvd-59/obj62 2023-10-11T15:27:42.124 DEBUG:boto:Method: GET 2023-10-11T15:27:42.124 DEBUG:boto:Path: /dghqvd-59/obj62 2023-10-11T15:27:42.125 DEBUG:boto:Data: 2023-10-11T15:27:42.125 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.125 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.126 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.126 DEBUG:boto:Params: {} 2023-10-11T15:27:42.127 DEBUG:boto:Token: None 2023-10-11T15:27:42.127 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj62 2023-10-11T15:27:42.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ZZbyGtLr1pMJdJzwyNUNSiIFII= 2023-10-11T15:27:42.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ZZbyGtLr1pMJdJzwyNUNSiIFII=', 'Content-Length': '0'} 2023-10-11T15:27:42.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddbe1f4462f88b3d-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.131 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.132 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.132 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.133 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.133 DEBUG:boto:Method: GET 2023-10-11T15:27:42.134 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.134 DEBUG:boto:Data: 2023-10-11T15:27:42.135 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.135 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.136 DEBUG:boto:Params: {} 2023-10-11T15:27:42.136 DEBUG:boto:Token: None 2023-10-11T15:27:42.137 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373fabc78d4f7a22-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.141 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.142 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.142 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.143 DEBUG:boto:path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.143 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.144 DEBUG:boto:Method: GET 2023-10-11T15:27:42.144 DEBUG:boto:Path: /dghqvd-59/?prefix=obj62 2023-10-11T15:27:42.144 DEBUG:boto:Data: 2023-10-11T15:27:42.145 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.146 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.146 DEBUG:boto:Params: {} 2023-10-11T15:27:42.146 DEBUG:boto:Token: None 2023-10-11T15:27:42.147 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c8e4d8f72d8ccc-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.153 DEBUG:boto:b'dghqvd-59obj621000falseobj622023-10-11T15:24:15.825Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:42.154 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.154 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.155 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.155 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.156 DEBUG:boto:Method: GET 2023-10-11T15:27:42.158 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.158 DEBUG:boto:Data: 2023-10-11T15:27:42.159 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.159 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.160 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.160 DEBUG:boto:Params: {} 2023-10-11T15:27:42.161 DEBUG:boto:Token: None 2023-10-11T15:27:42.161 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk= 2023-10-11T15:27:42.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk=', 'Content-Length': '0'} 2023-10-11T15:27:42.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002714dfd1fa91d416-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.165 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.166 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.166 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.166 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.167 DEBUG:boto:Method: GET 2023-10-11T15:27:42.167 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.168 DEBUG:boto:Data: 2023-10-11T15:27:42.168 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.169 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.169 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.170 DEBUG:boto:Params: {} 2023-10-11T15:27:42.170 DEBUG:boto:Token: None 2023-10-11T15:27:42.171 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.171 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk= 2023-10-11T15:27:42.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk=', 'Content-Length': '0'} 2023-10-11T15:27:42.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808c66ab8121bf79-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.214 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.215 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.215 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.216 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.216 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.217 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.217 DEBUG:boto:Data: 2023-10-11T15:27:42.218 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.218 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.218 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.219 DEBUG:boto:Params: {} 2023-10-11T15:27:42.219 DEBUG:boto:Token: None 2023-10-11T15:27:42.220 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bZXzfBdocrmSrfRPsp4XYUZonVU= 2023-10-11T15:27:42.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bZXzfBdocrmSrfRPsp4XYUZonVU=', 'Content-Length': '0'} 2023-10-11T15:27:42.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005998ec942745c2ed-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.224 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.224 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.225 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.225 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.225 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.226 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.226 DEBUG:boto:Data: 2023-10-11T15:27:42.227 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.227 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.228 DEBUG:boto:Params: {} 2023-10-11T15:27:42.228 DEBUG:boto:Token: None 2023-10-11T15:27:42.229 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bZXzfBdocrmSrfRPsp4XYUZonVU= 2023-10-11T15:27:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bZXzfBdocrmSrfRPsp4XYUZonVU=', 'Content-Length': '0'} 2023-10-11T15:27:42.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e7f44f47130eaa-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:42.233 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.234 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.234 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.235 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.235 DEBUG:boto:Method: GET 2023-10-11T15:27:42.235 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.236 DEBUG:boto:Data: 2023-10-11T15:27:42.236 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.237 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.238 DEBUG:boto:Params: {} 2023-10-11T15:27:42.238 DEBUG:boto:Token: None 2023-10-11T15:27:42.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk= 2023-10-11T15:27:42.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk=', 'Content-Length': '0'} 2023-10-11T15:27:42.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf63ac3bf405c8c-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.282 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.283 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.283 DEBUG:boto:path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.284 DEBUG:boto:auth_path=/dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.284 DEBUG:boto:Method: GET 2023-10-11T15:27:42.285 DEBUG:boto:Path: /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.285 DEBUG:boto:Data: 2023-10-11T15:27:42.285 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.286 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.287 DEBUG:boto:Params: {} 2023-10-11T15:27:42.287 DEBUG:boto:Token: None 2023-10-11T15:27:42.288 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68?versionId=null 2023-10-11T15:27:42.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk= 2023-10-11T15:27:42.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYSj/ZBeHCwpnIOcjB2UZbY/Fyk=', 'Content-Length': '0'} 2023-10-11T15:27:42.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045539d96d3d0deb5-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.292 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.293 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.293 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.293 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.294 DEBUG:boto:Method: GET 2023-10-11T15:27:42.294 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.295 DEBUG:boto:Data: 2023-10-11T15:27:42.295 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.296 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.296 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.296 DEBUG:boto:Params: {} 2023-10-11T15:27:42.297 DEBUG:boto:Token: None 2023-10-11T15:27:42.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be175293e2026af4-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.302 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.302 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.303 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.303 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.304 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.304 DEBUG:boto:Method: GET 2023-10-11T15:27:42.305 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.305 DEBUG:boto:Data: 2023-10-11T15:27:42.305 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.306 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.307 DEBUG:boto:Params: {} 2023-10-11T15:27:42.307 DEBUG:boto:Token: None 2023-10-11T15:27:42.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e545747050acb5ae-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.315 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.315 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.316 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.316 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.317 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:42.317 DEBUG:boto:Data: 2023-10-11T15:27:42.317 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.318 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.318 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.319 DEBUG:boto:Params: {} 2023-10-11T15:27:42.319 DEBUG:boto:Token: None 2023-10-11T15:27:42.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68 2023-10-11T15:27:42.320 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8apZE/ktzRwp8ROq2gbqm9l5E+A= 2023-10-11T15:27:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8apZE/ktzRwp8ROq2gbqm9l5E+A=', 'Content-Length': '0'} 2023-10-11T15:27:42.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d701399dc6dd85c-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.323 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.324 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.324 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.325 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:42.325 DEBUG:boto:Data: 2023-10-11T15:27:42.326 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.326 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.327 DEBUG:boto:Params: {} 2023-10-11T15:27:42.327 DEBUG:boto:Token: None 2023-10-11T15:27:42.328 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68 2023-10-11T15:27:42.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8apZE/ktzRwp8ROq2gbqm9l5E+A= 2023-10-11T15:27:42.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8apZE/ktzRwp8ROq2gbqm9l5E+A=', 'Content-Length': '0'} 2023-10-11T15:27:42.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002daad2e19d4b0bd2-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:27:42.331 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.332 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.332 DEBUG:boto:Method: GET 2023-10-11T15:27:42.333 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:42.333 DEBUG:boto:Data: 2023-10-11T15:27:42.334 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.335 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.335 DEBUG:boto:Params: {} 2023-10-11T15:27:42.335 DEBUG:boto:Token: None 2023-10-11T15:27:42.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68 2023-10-11T15:27:42.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N4RPe3CJkU0emOSJ+ay0Mu1crdU= 2023-10-11T15:27:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N4RPe3CJkU0emOSJ+ay0Mu1crdU=', 'Content-Length': '0'} 2023-10-11T15:27:42.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eddb5e7382c22e00-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.339 DEBUG:boto:path=/dghqvd-59/obj68 2023-10-11T15:27:42.340 DEBUG:boto:auth_path=/dghqvd-59/obj68 2023-10-11T15:27:42.340 DEBUG:boto:Method: GET 2023-10-11T15:27:42.341 DEBUG:boto:Path: /dghqvd-59/obj68 2023-10-11T15:27:42.341 DEBUG:boto:Data: 2023-10-11T15:27:42.342 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.342 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.343 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.343 DEBUG:boto:Params: {} 2023-10-11T15:27:42.343 DEBUG:boto:Token: None 2023-10-11T15:27:42.344 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj68 2023-10-11T15:27:42.344 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N4RPe3CJkU0emOSJ+ay0Mu1crdU= 2023-10-11T15:27:42.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N4RPe3CJkU0emOSJ+ay0Mu1crdU=', 'Content-Length': '0'} 2023-10-11T15:27:42.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd95286ca0d04629-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.394 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.395 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.395 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.396 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.396 DEBUG:boto:Method: GET 2023-10-11T15:27:42.397 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.397 DEBUG:boto:Data: 2023-10-11T15:27:42.398 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.399 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.399 DEBUG:boto:Params: {} 2023-10-11T15:27:42.399 DEBUG:boto:Token: None 2023-10-11T15:27:42.400 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304b29caec1daa83-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.404 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.405 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.405 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.406 DEBUG:boto:path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.406 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.407 DEBUG:boto:Method: GET 2023-10-11T15:27:42.407 DEBUG:boto:Path: /dghqvd-59/?prefix=obj68 2023-10-11T15:27:42.407 DEBUG:boto:Data: 2023-10-11T15:27:42.408 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.409 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.409 DEBUG:boto:Params: {} 2023-10-11T15:27:42.410 DEBUG:boto:Token: None 2023-10-11T15:27:42.410 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c3221af8de767c-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.416 DEBUG:boto:b'dghqvd-59obj681000falseobj682023-10-11T15:16:32.746Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:42.417 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.417 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.418 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.418 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.419 DEBUG:boto:Method: GET 2023-10-11T15:27:42.419 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.419 DEBUG:boto:Data: 2023-10-11T15:27:42.420 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.420 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.421 DEBUG:boto:Params: {} 2023-10-11T15:27:42.421 DEBUG:boto:Token: None 2023-10-11T15:27:42.422 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.422 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI= 2023-10-11T15:27:42.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI=', 'Content-Length': '0'} 2023-10-11T15:27:42.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012040f1348e3aa44-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.466 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.467 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.467 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.468 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.468 DEBUG:boto:Method: GET 2023-10-11T15:27:42.468 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.469 DEBUG:boto:Data: 2023-10-11T15:27:42.469 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.470 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.471 DEBUG:boto:Params: {} 2023-10-11T15:27:42.471 DEBUG:boto:Token: None 2023-10-11T15:27:42.471 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI= 2023-10-11T15:27:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI=', 'Content-Length': '0'} 2023-10-11T15:27:42.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9e96de4d0e988b-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.475 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.476 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.476 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.477 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.477 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.477 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.478 DEBUG:boto:Data: 2023-10-11T15:27:42.478 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.479 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.479 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.479 DEBUG:boto:Params: {} 2023-10-11T15:27:42.480 DEBUG:boto:Token: None 2023-10-11T15:27:42.480 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1Jwf0JD2Djnn7H7tn3czFud6aRk= 2023-10-11T15:27:42.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1Jwf0JD2Djnn7H7tn3czFud6aRk=', 'Content-Length': '0'} 2023-10-11T15:27:42.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086808350a45a7c46-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.484 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.484 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.484 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.485 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.485 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.485 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.486 DEBUG:boto:Data: 2023-10-11T15:27:42.486 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.487 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.487 DEBUG:boto:Params: {} 2023-10-11T15:27:42.488 DEBUG:boto:Token: None 2023-10-11T15:27:42.488 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1Jwf0JD2Djnn7H7tn3czFud6aRk= 2023-10-11T15:27:42.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1Jwf0JD2Djnn7H7tn3czFud6aRk=', 'Content-Length': '0'} 2023-10-11T15:27:42.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e28bd9ed0cff004-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:42.492 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.492 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.493 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.493 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.494 DEBUG:boto:Method: GET 2023-10-11T15:27:42.494 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.495 DEBUG:boto:Data: 2023-10-11T15:27:42.495 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.495 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.496 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.496 DEBUG:boto:Params: {} 2023-10-11T15:27:42.497 DEBUG:boto:Token: None 2023-10-11T15:27:42.497 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.498 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI= 2023-10-11T15:27:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI=', 'Content-Length': '0'} 2023-10-11T15:27:42.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48638edbdf0b319-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.501 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.501 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.502 DEBUG:boto:path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.502 DEBUG:boto:auth_path=/dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.502 DEBUG:boto:Method: GET 2023-10-11T15:27:42.503 DEBUG:boto:Path: /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.503 DEBUG:boto:Data: 2023-10-11T15:27:42.504 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.505 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.505 DEBUG:boto:Params: {} 2023-10-11T15:27:42.505 DEBUG:boto:Token: None 2023-10-11T15:27:42.506 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69?versionId=null 2023-10-11T15:27:42.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI= 2023-10-11T15:27:42.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0qw73l0NyXuv0JIMJiQKYLwJsTI=', 'Content-Length': '0'} 2023-10-11T15:27:42.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050ea0759256a0451-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.550 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.551 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.551 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.551 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.552 DEBUG:boto:Method: GET 2023-10-11T15:27:42.552 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.553 DEBUG:boto:Data: 2023-10-11T15:27:42.553 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.553 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.554 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.554 DEBUG:boto:Params: {} 2023-10-11T15:27:42.555 DEBUG:boto:Token: None 2023-10-11T15:27:42.555 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.555 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b072eedad175de7c-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.559 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.561 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.562 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.562 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.563 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.563 DEBUG:boto:Method: GET 2023-10-11T15:27:42.563 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.564 DEBUG:boto:Data: 2023-10-11T15:27:42.564 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.565 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.565 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.565 DEBUG:boto:Params: {} 2023-10-11T15:27:42.566 DEBUG:boto:Token: None 2023-10-11T15:27:42.566 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6e8a13e49c40e50-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.572 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.573 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.573 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.573 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.574 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:42.574 DEBUG:boto:Data: 2023-10-11T15:27:42.575 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.575 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.576 DEBUG:boto:Params: {} 2023-10-11T15:27:42.576 DEBUG:boto:Token: None 2023-10-11T15:27:42.577 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69 2023-10-11T15:27:42.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o07NQI9lQr2oUqoiDHDOtpsNKD0= 2023-10-11T15:27:42.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o07NQI9lQr2oUqoiDHDOtpsNKD0=', 'Content-Length': '0'} 2023-10-11T15:27:42.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa0872ff79866ad-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.580 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.580 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.581 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.581 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:42.582 DEBUG:boto:Data: 2023-10-11T15:27:42.582 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.583 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.583 DEBUG:boto:Params: {} 2023-10-11T15:27:42.584 DEBUG:boto:Token: None 2023-10-11T15:27:42.584 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69 2023-10-11T15:27:42.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:o07NQI9lQr2oUqoiDHDOtpsNKD0= 2023-10-11T15:27:42.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o07NQI9lQr2oUqoiDHDOtpsNKD0=', 'Content-Length': '0'} 2023-10-11T15:27:42.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d0dbf626eea2004-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:27:42.588 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.589 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.589 DEBUG:boto:Method: GET 2023-10-11T15:27:42.589 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:42.590 DEBUG:boto:Data: 2023-10-11T15:27:42.590 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.591 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.592 DEBUG:boto:Params: {} 2023-10-11T15:27:42.592 DEBUG:boto:Token: None 2023-10-11T15:27:42.593 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69 2023-10-11T15:27:42.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yGjW7TwWttiujdqRMttOT/lde3E= 2023-10-11T15:27:42.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yGjW7TwWttiujdqRMttOT/lde3E=', 'Content-Length': '0'} 2023-10-11T15:27:42.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076befbee0e215ed7-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.638 DEBUG:boto:path=/dghqvd-59/obj69 2023-10-11T15:27:42.639 DEBUG:boto:auth_path=/dghqvd-59/obj69 2023-10-11T15:27:42.639 DEBUG:boto:Method: GET 2023-10-11T15:27:42.640 DEBUG:boto:Path: /dghqvd-59/obj69 2023-10-11T15:27:42.640 DEBUG:boto:Data: 2023-10-11T15:27:42.641 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.641 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.641 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.642 DEBUG:boto:Params: {} 2023-10-11T15:27:42.642 DEBUG:boto:Token: None 2023-10-11T15:27:42.643 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj69 2023-10-11T15:27:42.643 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yGjW7TwWttiujdqRMttOT/lde3E= 2023-10-11T15:27:42.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yGjW7TwWttiujdqRMttOT/lde3E=', 'Content-Length': '0'} 2023-10-11T15:27:42.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c1ddef7e9e1ae2-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.647 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.647 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.647 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.648 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.648 DEBUG:boto:Method: GET 2023-10-11T15:27:42.649 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.649 DEBUG:boto:Data: 2023-10-11T15:27:42.649 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.650 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.651 DEBUG:boto:Params: {} 2023-10-11T15:27:42.651 DEBUG:boto:Token: None 2023-10-11T15:27:42.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb91f1e5e6fb6c12-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.656 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.657 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.657 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.657 DEBUG:boto:path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.658 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.658 DEBUG:boto:Method: GET 2023-10-11T15:27:42.659 DEBUG:boto:Path: /dghqvd-59/?prefix=obj69 2023-10-11T15:27:42.659 DEBUG:boto:Data: 2023-10-11T15:27:42.660 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.660 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.661 DEBUG:boto:Params: {} 2023-10-11T15:27:42.661 DEBUG:boto:Token: None 2023-10-11T15:27:42.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.662 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091dde30f3556b07f-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.667 DEBUG:boto:b'dghqvd-59obj691000falseobj692023-10-11T15:21:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:42.668 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.669 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.669 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.670 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.670 DEBUG:boto:Method: GET 2023-10-11T15:27:42.671 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.671 DEBUG:boto:Data: 2023-10-11T15:27:42.672 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.672 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.672 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.673 DEBUG:boto:Params: {} 2023-10-11T15:27:42.673 DEBUG:boto:Token: None 2023-10-11T15:27:42.674 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg= 2023-10-11T15:27:42.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg=', 'Content-Length': '0'} 2023-10-11T15:27:42.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e5e58a50ea3b7bc-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.678 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.678 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.679 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.679 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.680 DEBUG:boto:Method: GET 2023-10-11T15:27:42.680 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.681 DEBUG:boto:Data: 2023-10-11T15:27:42.681 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.682 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.682 DEBUG:boto:Params: {} 2023-10-11T15:27:42.683 DEBUG:boto:Token: None 2023-10-11T15:27:42.683 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg= 2023-10-11T15:27:42.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg=', 'Content-Length': '0'} 2023-10-11T15:27:42.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004787fd914411cee1-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.734 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.735 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.735 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.736 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.736 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.737 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.737 DEBUG:boto:Data: 2023-10-11T15:27:42.737 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.738 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.739 DEBUG:boto:Params: {} 2023-10-11T15:27:42.739 DEBUG:boto:Token: None 2023-10-11T15:27:42.740 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bx+Zg/ZsusUIk4W52sNDLcdCoNM= 2023-10-11T15:27:42.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bx+Zg/ZsusUIk4W52sNDLcdCoNM=', 'Content-Length': '0'} 2023-10-11T15:27:42.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003661d42047d4adf9-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.744 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.744 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.744 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.745 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.745 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.745 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.746 DEBUG:boto:Data: 2023-10-11T15:27:42.746 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.747 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.747 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.747 DEBUG:boto:Params: {} 2023-10-11T15:27:42.748 DEBUG:boto:Token: None 2023-10-11T15:27:42.748 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bx+Zg/ZsusUIk4W52sNDLcdCoNM= 2023-10-11T15:27:42.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bx+Zg/ZsusUIk4W52sNDLcdCoNM=', 'Content-Length': '0'} 2023-10-11T15:27:42.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000204acd3a1ca4a52f-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:42.752 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.753 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.753 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.754 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.754 DEBUG:boto:Method: GET 2023-10-11T15:27:42.754 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.755 DEBUG:boto:Data: 2023-10-11T15:27:42.755 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.756 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.756 DEBUG:boto:Params: {} 2023-10-11T15:27:42.757 DEBUG:boto:Token: None 2023-10-11T15:27:42.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg= 2023-10-11T15:27:42.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg=', 'Content-Length': '0'} 2023-10-11T15:27:42.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbfe566bfde221e-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.802 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.803 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.803 DEBUG:boto:path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.804 DEBUG:boto:auth_path=/dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.804 DEBUG:boto:Method: GET 2023-10-11T15:27:42.805 DEBUG:boto:Path: /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.805 DEBUG:boto:Data: 2023-10-11T15:27:42.806 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.806 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.807 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.807 DEBUG:boto:Params: {} 2023-10-11T15:27:42.808 DEBUG:boto:Token: None 2023-10-11T15:27:42.808 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74?versionId=null 2023-10-11T15:27:42.809 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg= 2023-10-11T15:27:42.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:k2Rm/GElnC99h13WJ+41swJ8Cgg=', 'Content-Length': '0'} 2023-10-11T15:27:42.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424011de6fab0eb8-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.813 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.813 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.814 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.814 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.815 DEBUG:boto:Method: GET 2023-10-11T15:27:42.815 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.815 DEBUG:boto:Data: 2023-10-11T15:27:42.816 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.817 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.817 DEBUG:boto:Params: {} 2023-10-11T15:27:42.818 DEBUG:boto:Token: None 2023-10-11T15:27:42.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.819 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d7bc417fb455b3-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.822 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.823 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.824 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.824 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.824 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.825 DEBUG:boto:Method: GET 2023-10-11T15:27:42.825 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.826 DEBUG:boto:Data: 2023-10-11T15:27:42.826 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.827 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.827 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.827 DEBUG:boto:Params: {} 2023-10-11T15:27:42.828 DEBUG:boto:Token: None 2023-10-11T15:27:42.829 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000659b2989c51d0eb5-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.835 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.835 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.836 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.836 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.837 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:42.837 DEBUG:boto:Data: 2023-10-11T15:27:42.838 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.838 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.839 DEBUG:boto:Params: {} 2023-10-11T15:27:42.839 DEBUG:boto:Token: None 2023-10-11T15:27:42.840 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74 2023-10-11T15:27:42.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uZAJPcuPTeDTw+X530AS/JOoEto= 2023-10-11T15:27:42.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uZAJPcuPTeDTw+X530AS/JOoEto=', 'Content-Length': '0'} 2023-10-11T15:27:42.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158480df863ff6d3-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.844 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.844 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.845 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.845 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:42.846 DEBUG:boto:Data: 2023-10-11T15:27:42.846 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.847 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.848 DEBUG:boto:Params: {} 2023-10-11T15:27:42.848 DEBUG:boto:Token: None 2023-10-11T15:27:42.849 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74 2023-10-11T15:27:42.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uZAJPcuPTeDTw+X530AS/JOoEto= 2023-10-11T15:27:42.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uZAJPcuPTeDTw+X530AS/JOoEto=', 'Content-Length': '0'} 2023-10-11T15:27:42.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f647fe141e41d-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:27:42.853 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.853 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.854 DEBUG:boto:Method: GET 2023-10-11T15:27:42.854 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:42.855 DEBUG:boto:Data: 2023-10-11T15:27:42.855 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.856 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.856 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.857 DEBUG:boto:Params: {} 2023-10-11T15:27:42.857 DEBUG:boto:Token: None 2023-10-11T15:27:42.858 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74 2023-10-11T15:27:42.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mWT9Ynobbxy4DppLpOTKKgorvtY= 2023-10-11T15:27:42.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mWT9Ynobbxy4DppLpOTKKgorvtY=', 'Content-Length': '0'} 2023-10-11T15:27:42.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b7cf5dae423121-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.862 DEBUG:boto:path=/dghqvd-59/obj74 2023-10-11T15:27:42.863 DEBUG:boto:auth_path=/dghqvd-59/obj74 2023-10-11T15:27:42.863 DEBUG:boto:Method: GET 2023-10-11T15:27:42.863 DEBUG:boto:Path: /dghqvd-59/obj74 2023-10-11T15:27:42.864 DEBUG:boto:Data: 2023-10-11T15:27:42.864 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.865 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.866 DEBUG:boto:Params: {} 2023-10-11T15:27:42.867 DEBUG:boto:Token: None 2023-10-11T15:27:42.867 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj74 2023-10-11T15:27:42.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mWT9Ynobbxy4DppLpOTKKgorvtY= 2023-10-11T15:27:42.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mWT9Ynobbxy4DppLpOTKKgorvtY=', 'Content-Length': '0'} 2023-10-11T15:27:42.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:25:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b63983ac1b24aed-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.871 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.872 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.872 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.873 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.873 DEBUG:boto:Method: GET 2023-10-11T15:27:42.874 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.874 DEBUG:boto:Data: 2023-10-11T15:27:42.875 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.875 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.876 DEBUG:boto:Params: {} 2023-10-11T15:27:42.876 DEBUG:boto:Token: None 2023-10-11T15:27:42.877 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bcd6db048837519-006526beee-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.881 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.882 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:42.882 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:42.883 DEBUG:boto:path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.883 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.883 DEBUG:boto:Method: GET 2023-10-11T15:27:42.884 DEBUG:boto:Path: /dghqvd-59/?prefix=obj74 2023-10-11T15:27:42.884 DEBUG:boto:Data: 2023-10-11T15:27:42.885 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.886 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.886 DEBUG:boto:Params: {} 2023-10-11T15:27:42.887 DEBUG:boto:Token: None 2023-10-11T15:27:42.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/ 2023-10-11T15:27:42.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg= 2023-10-11T15:27:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zz3WnoKwYImu85aFrgcNxE2uyxg=', 'Content-Length': '0'} 2023-10-11T15:27:42.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001164c9679a3281e8-006526beee-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.934 DEBUG:boto:b'dghqvd-59obj741000falseobj742023-10-11T15:25:38.223Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:42.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:42.936 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:42.936 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:42.937 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.937 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.938 DEBUG:boto:Method: GET 2023-10-11T15:27:42.938 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.939 DEBUG:boto:Data: 2023-10-11T15:27:42.939 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.940 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.940 DEBUG:boto:Params: {} 2023-10-11T15:27:42.941 DEBUG:boto:Token: None 2023-10-11T15:27:42.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8= 2023-10-11T15:27:42.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8=', 'Content-Length': '0'} 2023-10-11T15:27:42.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261a31ef6b20a81c-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.946 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:42.946 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:42.946 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.947 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.947 DEBUG:boto:Method: GET 2023-10-11T15:27:42.948 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.948 DEBUG:boto:Data: 2023-10-11T15:27:42.949 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.949 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.950 DEBUG:boto:Params: {} 2023-10-11T15:27:42.950 DEBUG:boto:Token: None 2023-10-11T15:27:42.951 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8= 2023-10-11T15:27:42.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8=', 'Content-Length': '0'} 2023-10-11T15:27:42.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004de4d0d89b047e93-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.955 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:42.955 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:42.956 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.956 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.957 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.957 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.957 DEBUG:boto:Data: 2023-10-11T15:27:42.958 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.959 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.959 DEBUG:boto:Params: {} 2023-10-11T15:27:42.959 DEBUG:boto:Token: None 2023-10-11T15:27:42.960 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xv/UhVIUTqoOKVZSNV5+k1noCGo= 2023-10-11T15:27:42.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xv/UhVIUTqoOKVZSNV5+k1noCGo=', 'Content-Length': '0'} 2023-10-11T15:27:42.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007062784b600723d8-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.963 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:42.964 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:42.964 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.965 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.965 DEBUG:boto:Method: HEAD 2023-10-11T15:27:42.965 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.966 DEBUG:boto:Data: 2023-10-11T15:27:42.966 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:42.967 DEBUG:boto:Port: 8000 2023-10-11T15:27:42.968 DEBUG:boto:Params: {} 2023-10-11T15:27:42.968 DEBUG:boto:Token: None 2023-10-11T15:27:42.969 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Xv/UhVIUTqoOKVZSNV5+k1noCGo= 2023-10-11T15:27:42.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Xv/UhVIUTqoOKVZSNV5+k1noCGo=', 'Content-Length': '0'} 2023-10-11T15:27:42.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602347cbd7f18e2d-006526beee-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:42.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:42.973 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:42.973 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:42.973 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.974 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.974 DEBUG:boto:Method: GET 2023-10-11T15:27:42.975 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.975 DEBUG:boto:Data: 2023-10-11T15:27:42.977 DEBUG:boto:Headers: {} 2023-10-11T15:27:42.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:42.978 DEBUG:boto:Port: 8001 2023-10-11T15:27:42.978 DEBUG:boto:Params: {} 2023-10-11T15:27:42.979 DEBUG:boto:Token: None 2023-10-11T15:27:42.979 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:42 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:42.980 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8= 2023-10-11T15:27:42.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLd6WXdDCQS6BF70aU8eNWWIHm8=', 'Content-Length': '0'} 2023-10-11T15:27:42.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed00e7c9f8ff174-006526beee-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.026 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:43.027 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:43.027 DEBUG:boto:path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:43.028 DEBUG:boto:auth_path=/dghqvd-59/obj76?versionId=null 2023-10-11T15:27:43.028 DEBUG:boto:Method: GET 2023-10-11T15:27:43.029 DEBUG:boto:Path: /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:43.029 DEBUG:boto:Data: 2023-10-11T15:27:43.030 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.031 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.031 DEBUG:boto:Params: {} 2023-10-11T15:27:43.031 DEBUG:boto:Token: None 2023-10-11T15:27:43.032 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj76?versionId=null 2023-10-11T15:27:43.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ED2EY6mBSqP+gkHpsh34x23rxB0= 2023-10-11T15:27:43.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ED2EY6mBSqP+gkHpsh34x23rxB0=', 'Content-Length': '0'} 2023-10-11T15:27:43.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002280e34c6d923ea7-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.036 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.036 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.037 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.037 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.038 DEBUG:boto:Method: GET 2023-10-11T15:27:43.038 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.039 DEBUG:boto:Data: 2023-10-11T15:27:43.039 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.040 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.040 DEBUG:boto:Params: {} 2023-10-11T15:27:43.041 DEBUG:boto:Token: None 2023-10-11T15:27:43.041 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a1c2cb597067b74-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.045 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.046 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.046 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.047 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.047 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.048 DEBUG:boto:Method: GET 2023-10-11T15:27:43.048 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.048 DEBUG:boto:Data: 2023-10-11T15:27:43.049 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.049 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.050 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.050 DEBUG:boto:Params: {} 2023-10-11T15:27:43.051 DEBUG:boto:Token: None 2023-10-11T15:27:43.051 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f0bd159458d1aee-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.102 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.103 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:43.104 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:43.104 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.104 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:43.105 DEBUG:boto:Data: 2023-10-11T15:27:43.105 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.106 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.106 DEBUG:boto:Params: {} 2023-10-11T15:27:43.107 DEBUG:boto:Token: None 2023-10-11T15:27:43.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj76 2023-10-11T15:27:43.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yCGrfNdrarlakSwe3fFbks7bcKg= 2023-10-11T15:27:43.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yCGrfNdrarlakSwe3fFbks7bcKg=', 'Content-Length': '0'} 2023-10-11T15:27:43.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce1bef9ddfb818a-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.113 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:43.114 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:43.114 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.115 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:43.115 DEBUG:boto:Data: 2023-10-11T15:27:43.115 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.116 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.117 DEBUG:boto:Params: {} 2023-10-11T15:27:43.117 DEBUG:boto:Token: None 2023-10-11T15:27:43.118 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj76 2023-10-11T15:27:43.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yCGrfNdrarlakSwe3fFbks7bcKg= 2023-10-11T15:27:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yCGrfNdrarlakSwe3fFbks7bcKg=', 'Content-Length': '0'} 2023-10-11T15:27:43.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019aaa7d8bf38ebb4-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:27:43.123 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:43.123 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:43.124 DEBUG:boto:Method: GET 2023-10-11T15:27:43.124 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:43.125 DEBUG:boto:Data: 2023-10-11T15:27:43.125 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.127 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.128 DEBUG:boto:Params: {} 2023-10-11T15:27:43.129 DEBUG:boto:Token: None 2023-10-11T15:27:43.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj76 2023-10-11T15:27:43.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nwesdWT7cBi0MeeZxUsBYQN1S8s= 2023-10-11T15:27:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nwesdWT7cBi0MeeZxUsBYQN1S8s=', 'Content-Length': '0'} 2023-10-11T15:27:43.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c06fc40ed3ed57-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.134 DEBUG:boto:path=/dghqvd-59/obj76 2023-10-11T15:27:43.135 DEBUG:boto:auth_path=/dghqvd-59/obj76 2023-10-11T15:27:43.135 DEBUG:boto:Method: GET 2023-10-11T15:27:43.136 DEBUG:boto:Path: /dghqvd-59/obj76 2023-10-11T15:27:43.136 DEBUG:boto:Data: 2023-10-11T15:27:43.137 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.138 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.138 DEBUG:boto:Params: {} 2023-10-11T15:27:43.139 DEBUG:boto:Token: None 2023-10-11T15:27:43.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj76 2023-10-11T15:27:43.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nwesdWT7cBi0MeeZxUsBYQN1S8s= 2023-10-11T15:27:43.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nwesdWT7cBi0MeeZxUsBYQN1S8s=', 'Content-Length': '0'} 2023-10-11T15:27:43.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000492c639addce6d75-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.143 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.144 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.144 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.145 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.145 DEBUG:boto:Method: GET 2023-10-11T15:27:43.145 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.146 DEBUG:boto:Data: 2023-10-11T15:27:43.146 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.147 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.148 DEBUG:boto:Params: {} 2023-10-11T15:27:43.148 DEBUG:boto:Token: None 2023-10-11T15:27:43.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3a903bc0bddaf4-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.153 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.154 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.154 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.155 DEBUG:boto:path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.155 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.156 DEBUG:boto:Method: GET 2023-10-11T15:27:43.156 DEBUG:boto:Path: /dghqvd-59/?prefix=obj76 2023-10-11T15:27:43.156 DEBUG:boto:Data: 2023-10-11T15:27:43.157 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.157 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.158 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.158 DEBUG:boto:Params: {} 2023-10-11T15:27:43.159 DEBUG:boto:Token: None 2023-10-11T15:27:43.159 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.160 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3a23e6992d96027-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.165 DEBUG:boto:b'dghqvd-59obj761000falseobj762023-10-11T15:18:55.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:43.166 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.167 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.167 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.168 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.169 DEBUG:boto:Method: GET 2023-10-11T15:27:43.169 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.170 DEBUG:boto:Data: 2023-10-11T15:27:43.171 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.172 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.173 DEBUG:boto:Params: {} 2023-10-11T15:27:43.174 DEBUG:boto:Token: None 2023-10-11T15:27:43.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ= 2023-10-11T15:27:43.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ec063b8b6d9613-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.179 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.179 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.179 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.180 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.180 DEBUG:boto:Method: GET 2023-10-11T15:27:43.181 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.181 DEBUG:boto:Data: 2023-10-11T15:27:43.181 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.182 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.183 DEBUG:boto:Params: {} 2023-10-11T15:27:43.183 DEBUG:boto:Token: None 2023-10-11T15:27:43.184 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ= 2023-10-11T15:27:43.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfd0abb1da3ce8e-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.230 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.231 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.231 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.232 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.233 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.233 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.233 DEBUG:boto:Data: 2023-10-11T15:27:43.234 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.234 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.235 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.235 DEBUG:boto:Params: {} 2023-10-11T15:27:43.236 DEBUG:boto:Token: None 2023-10-11T15:27:43.236 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:McW/mqGjAmrVdin/IWlWWgb8ecc= 2023-10-11T15:27:43.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:McW/mqGjAmrVdin/IWlWWgb8ecc=', 'Content-Length': '0'} 2023-10-11T15:27:43.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a1366107711cd1-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.241 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.241 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.242 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.243 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.244 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.244 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.245 DEBUG:boto:Data: 2023-10-11T15:27:43.246 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.247 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.247 DEBUG:boto:Params: {} 2023-10-11T15:27:43.248 DEBUG:boto:Token: None 2023-10-11T15:27:43.248 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:McW/mqGjAmrVdin/IWlWWgb8ecc= 2023-10-11T15:27:43.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:McW/mqGjAmrVdin/IWlWWgb8ecc=', 'Content-Length': '0'} 2023-10-11T15:27:43.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d77fda31379fd458-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:43.253 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.253 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.253 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.254 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.254 DEBUG:boto:Method: GET 2023-10-11T15:27:43.255 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.255 DEBUG:boto:Data: 2023-10-11T15:27:43.256 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.256 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.256 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.257 DEBUG:boto:Params: {} 2023-10-11T15:27:43.257 DEBUG:boto:Token: None 2023-10-11T15:27:43.258 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.258 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ= 2023-10-11T15:27:43.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f01421c7b33ce5-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.262 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.263 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.263 DEBUG:boto:path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.264 DEBUG:boto:auth_path=/dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.264 DEBUG:boto:Method: GET 2023-10-11T15:27:43.265 DEBUG:boto:Path: /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.265 DEBUG:boto:Data: 2023-10-11T15:27:43.265 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.266 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.267 DEBUG:boto:Params: {} 2023-10-11T15:27:43.267 DEBUG:boto:Token: None 2023-10-11T15:27:43.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85?versionId=null 2023-10-11T15:27:43.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ= 2023-10-11T15:27:43.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:N86IeJsLGsmHWCJ8xjUZ4z54RKQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ea83acc12e0dc3-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.272 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.272 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.273 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.273 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.273 DEBUG:boto:Method: GET 2023-10-11T15:27:43.274 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.274 DEBUG:boto:Data: 2023-10-11T15:27:43.275 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.276 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.276 DEBUG:boto:Params: {} 2023-10-11T15:27:43.277 DEBUG:boto:Token: None 2023-10-11T15:27:43.277 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca8003240ab4bbc-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.281 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.282 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.283 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.283 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.284 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.284 DEBUG:boto:Method: GET 2023-10-11T15:27:43.284 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.285 DEBUG:boto:Data: 2023-10-11T15:27:43.285 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.286 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.287 DEBUG:boto:Params: {} 2023-10-11T15:27:43.287 DEBUG:boto:Token: None 2023-10-11T15:27:43.287 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.288 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052e92a88c82bc23e-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.334 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.335 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.335 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.336 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.336 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:43.337 DEBUG:boto:Data: 2023-10-11T15:27:43.337 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.338 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.339 DEBUG:boto:Params: {} 2023-10-11T15:27:43.339 DEBUG:boto:Token: None 2023-10-11T15:27:43.340 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85 2023-10-11T15:27:43.340 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IH+G5yW16Y0f9r3TlEcObK14/XI= 2023-10-11T15:27:43.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IH+G5yW16Y0f9r3TlEcObK14/XI=', 'Content-Length': '0'} 2023-10-11T15:27:43.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d76b1ed815dc016-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.344 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.344 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.345 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.345 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:43.346 DEBUG:boto:Data: 2023-10-11T15:27:43.346 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.347 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.347 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.347 DEBUG:boto:Params: {} 2023-10-11T15:27:43.348 DEBUG:boto:Token: None 2023-10-11T15:27:43.348 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85 2023-10-11T15:27:43.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IH+G5yW16Y0f9r3TlEcObK14/XI= 2023-10-11T15:27:43.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IH+G5yW16Y0f9r3TlEcObK14/XI=', 'Content-Length': '0'} 2023-10-11T15:27:43.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d7edf2ef31df23-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:27:43.353 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.353 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.353 DEBUG:boto:Method: GET 2023-10-11T15:27:43.354 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:43.354 DEBUG:boto:Data: 2023-10-11T15:27:43.355 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.356 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.356 DEBUG:boto:Params: {} 2023-10-11T15:27:43.357 DEBUG:boto:Token: None 2023-10-11T15:27:43.357 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85 2023-10-11T15:27:43.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8A9ai9HaYsAkBd44Hll5KMysp+A= 2023-10-11T15:27:43.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8A9ai9HaYsAkBd44Hll5KMysp+A=', 'Content-Length': '0'} 2023-10-11T15:27:43.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051cc6e4ec18baa56-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.361 DEBUG:boto:path=/dghqvd-59/obj85 2023-10-11T15:27:43.361 DEBUG:boto:auth_path=/dghqvd-59/obj85 2023-10-11T15:27:43.362 DEBUG:boto:Method: GET 2023-10-11T15:27:43.362 DEBUG:boto:Path: /dghqvd-59/obj85 2023-10-11T15:27:43.363 DEBUG:boto:Data: 2023-10-11T15:27:43.363 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.364 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.364 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.364 DEBUG:boto:Params: {} 2023-10-11T15:27:43.365 DEBUG:boto:Token: None 2023-10-11T15:27:43.365 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj85 2023-10-11T15:27:43.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8A9ai9HaYsAkBd44Hll5KMysp+A= 2023-10-11T15:27:43.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8A9ai9HaYsAkBd44Hll5KMysp+A=', 'Content-Length': '0'} 2023-10-11T15:27:43.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:16:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b25c9bcef28c799-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.369 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.370 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.370 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.371 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.371 DEBUG:boto:Method: GET 2023-10-11T15:27:43.372 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.372 DEBUG:boto:Data: 2023-10-11T15:27:43.373 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.373 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.374 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.374 DEBUG:boto:Params: {} 2023-10-11T15:27:43.375 DEBUG:boto:Token: None 2023-10-11T15:27:43.375 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082930c837deb39fd-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.379 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.380 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.381 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.381 DEBUG:boto:path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.382 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.382 DEBUG:boto:Method: GET 2023-10-11T15:27:43.383 DEBUG:boto:Path: /dghqvd-59/?prefix=obj85 2023-10-11T15:27:43.383 DEBUG:boto:Data: 2023-10-11T15:27:43.383 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.385 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.385 DEBUG:boto:Params: {} 2023-10-11T15:27:43.385 DEBUG:boto:Token: None 2023-10-11T15:27:43.386 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034f3304dc0049a28-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.395 DEBUG:boto:b'dghqvd-59obj851000falseobj852023-10-11T15:16:32.765Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:43.396 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.396 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.397 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.397 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.398 DEBUG:boto:Method: GET 2023-10-11T15:27:43.398 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.399 DEBUG:boto:Data: 2023-10-11T15:27:43.399 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.400 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.400 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.400 DEBUG:boto:Params: {} 2023-10-11T15:27:43.401 DEBUG:boto:Token: None 2023-10-11T15:27:43.402 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.402 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ= 2023-10-11T15:27:43.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015cee8272e9ba9f0-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.450 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.451 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.451 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.452 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.452 DEBUG:boto:Method: GET 2023-10-11T15:27:43.453 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.453 DEBUG:boto:Data: 2023-10-11T15:27:43.454 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.454 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.455 DEBUG:boto:Params: {} 2023-10-11T15:27:43.455 DEBUG:boto:Token: None 2023-10-11T15:27:43.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ= 2023-10-11T15:27:43.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035540e656daad3d6-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.460 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.460 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.460 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.461 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.461 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.462 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.462 DEBUG:boto:Data: 2023-10-11T15:27:43.462 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.463 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.464 DEBUG:boto:Params: {} 2023-10-11T15:27:43.464 DEBUG:boto:Token: None 2023-10-11T15:27:43.465 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAiWYz60p14rYT+5iyJ8nRe9yn8= 2023-10-11T15:27:43.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAiWYz60p14rYT+5iyJ8nRe9yn8=', 'Content-Length': '0'} 2023-10-11T15:27:43.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000774ef940e6621269-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.468 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.469 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.469 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.469 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.470 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.470 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.471 DEBUG:boto:Data: 2023-10-11T15:27:43.471 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.472 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.472 DEBUG:boto:Params: {} 2023-10-11T15:27:43.473 DEBUG:boto:Token: None 2023-10-11T15:27:43.473 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zAiWYz60p14rYT+5iyJ8nRe9yn8= 2023-10-11T15:27:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zAiWYz60p14rYT+5iyJ8nRe9yn8=', 'Content-Length': '0'} 2023-10-11T15:27:43.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006832371650b5e17a-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:43.477 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.478 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.478 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.479 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.479 DEBUG:boto:Method: GET 2023-10-11T15:27:43.479 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.480 DEBUG:boto:Data: 2023-10-11T15:27:43.480 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.481 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.482 DEBUG:boto:Params: {} 2023-10-11T15:27:43.482 DEBUG:boto:Token: None 2023-10-11T15:27:43.483 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ= 2023-10-11T15:27:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8dda12ab9a8c8c-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.486 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.487 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.487 DEBUG:boto:path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.488 DEBUG:boto:auth_path=/dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.488 DEBUG:boto:Method: GET 2023-10-11T15:27:43.488 DEBUG:boto:Path: /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.489 DEBUG:boto:Data: 2023-10-11T15:27:43.489 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.490 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.491 DEBUG:boto:Params: {} 2023-10-11T15:27:43.491 DEBUG:boto:Token: None 2023-10-11T15:27:43.492 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87?versionId=null 2023-10-11T15:27:43.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ= 2023-10-11T15:27:43.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OEZyirXmmnLQqngHnxm/+SDE+LQ=', 'Content-Length': '0'} 2023-10-11T15:27:43.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777d7e7541fe7146-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.495 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.496 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.496 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.497 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.497 DEBUG:boto:Method: GET 2023-10-11T15:27:43.498 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.498 DEBUG:boto:Data: 2023-10-11T15:27:43.499 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.499 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.500 DEBUG:boto:Params: {} 2023-10-11T15:27:43.500 DEBUG:boto:Token: None 2023-10-11T15:27:43.501 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdc65a405b87e7fe-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.505 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.506 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.506 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.506 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.507 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.507 DEBUG:boto:Method: GET 2023-10-11T15:27:43.508 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.508 DEBUG:boto:Data: 2023-10-11T15:27:43.509 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.509 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.510 DEBUG:boto:Params: {} 2023-10-11T15:27:43.510 DEBUG:boto:Token: None 2023-10-11T15:27:43.511 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065eb64dc6aa9ae06-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.558 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.559 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.560 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.560 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.560 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:43.561 DEBUG:boto:Data: 2023-10-11T15:27:43.561 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.562 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.562 DEBUG:boto:Params: {} 2023-10-11T15:27:43.563 DEBUG:boto:Token: None 2023-10-11T15:27:43.563 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87 2023-10-11T15:27:43.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3bUB7Fw/adrnAOItE2kC33VnBxY= 2023-10-11T15:27:43.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3bUB7Fw/adrnAOItE2kC33VnBxY=', 'Content-Length': '0'} 2023-10-11T15:27:43.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b831d06a3c0f8869-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.567 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.567 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.568 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.568 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:43.568 DEBUG:boto:Data: 2023-10-11T15:27:43.569 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.569 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.570 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.570 DEBUG:boto:Params: {} 2023-10-11T15:27:43.571 DEBUG:boto:Token: None 2023-10-11T15:27:43.571 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87 2023-10-11T15:27:43.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3bUB7Fw/adrnAOItE2kC33VnBxY= 2023-10-11T15:27:43.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3bUB7Fw/adrnAOItE2kC33VnBxY=', 'Content-Length': '0'} 2023-10-11T15:27:43.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e56fcf1c898688-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:27:43.575 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.575 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.576 DEBUG:boto:Method: GET 2023-10-11T15:27:43.576 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:43.577 DEBUG:boto:Data: 2023-10-11T15:27:43.577 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.578 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.579 DEBUG:boto:Params: {} 2023-10-11T15:27:43.579 DEBUG:boto:Token: None 2023-10-11T15:27:43.580 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87 2023-10-11T15:27:43.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:agVUFrXKUhEK0Zqc2v1TzfCpF+M= 2023-10-11T15:27:43.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:agVUFrXKUhEK0Zqc2v1TzfCpF+M=', 'Content-Length': '0'} 2023-10-11T15:27:43.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f890fb540a887361-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.583 DEBUG:boto:path=/dghqvd-59/obj87 2023-10-11T15:27:43.584 DEBUG:boto:auth_path=/dghqvd-59/obj87 2023-10-11T15:27:43.584 DEBUG:boto:Method: GET 2023-10-11T15:27:43.585 DEBUG:boto:Path: /dghqvd-59/obj87 2023-10-11T15:27:43.585 DEBUG:boto:Data: 2023-10-11T15:27:43.586 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.586 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.587 DEBUG:boto:Params: {} 2023-10-11T15:27:43.587 DEBUG:boto:Token: None 2023-10-11T15:27:43.588 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj87 2023-10-11T15:27:43.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:agVUFrXKUhEK0Zqc2v1TzfCpF+M= 2023-10-11T15:27:43.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:agVUFrXKUhEK0Zqc2v1TzfCpF+M=', 'Content-Length': '0'} 2023-10-11T15:27:43.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e18ef073590f39-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.593 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.594 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.594 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.595 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.595 DEBUG:boto:Method: GET 2023-10-11T15:27:43.596 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.596 DEBUG:boto:Data: 2023-10-11T15:27:43.596 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.597 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.598 DEBUG:boto:Params: {} 2023-10-11T15:27:43.598 DEBUG:boto:Token: None 2023-10-11T15:27:43.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000580bb418708bf97d-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.603 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.604 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.604 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.605 DEBUG:boto:path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.605 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.605 DEBUG:boto:Method: GET 2023-10-11T15:27:43.606 DEBUG:boto:Path: /dghqvd-59/?prefix=obj87 2023-10-11T15:27:43.606 DEBUG:boto:Data: 2023-10-11T15:27:43.607 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.608 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.608 DEBUG:boto:Params: {} 2023-10-11T15:27:43.608 DEBUG:boto:Token: None 2023-10-11T15:27:43.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed09d73c8e3a6a1-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.616 DEBUG:boto:b'dghqvd-59obj871000falseobj872023-10-11T15:21:59.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:43.617 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.618 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.618 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.619 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.619 DEBUG:boto:Method: GET 2023-10-11T15:27:43.620 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.620 DEBUG:boto:Data: 2023-10-11T15:27:43.621 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.622 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.622 DEBUG:boto:Params: {} 2023-10-11T15:27:43.623 DEBUG:boto:Token: None 2023-10-11T15:27:43.623 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4= 2023-10-11T15:27:43.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4=', 'Content-Length': '0'} 2023-10-11T15:27:43.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70f8506bb928685-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.666 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.667 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.667 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.668 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.668 DEBUG:boto:Method: GET 2023-10-11T15:27:43.669 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.669 DEBUG:boto:Data: 2023-10-11T15:27:43.670 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.670 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.671 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.671 DEBUG:boto:Params: {} 2023-10-11T15:27:43.672 DEBUG:boto:Token: None 2023-10-11T15:27:43.672 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4= 2023-10-11T15:27:43.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4=', 'Content-Length': '0'} 2023-10-11T15:27:43.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db61aa9d648d5b8b-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.677 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.677 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.678 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.678 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.678 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.679 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.679 DEBUG:boto:Data: 2023-10-11T15:27:43.680 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.681 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.681 DEBUG:boto:Params: {} 2023-10-11T15:27:43.682 DEBUG:boto:Token: None 2023-10-11T15:27:43.682 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Knqbe8v7GbBLgsi+PAOxt9wJi4A= 2023-10-11T15:27:43.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Knqbe8v7GbBLgsi+PAOxt9wJi4A=', 'Content-Length': '0'} 2023-10-11T15:27:43.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43e12f95da9eeb1-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.686 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.686 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.687 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.687 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.688 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.688 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.688 DEBUG:boto:Data: 2023-10-11T15:27:43.689 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.690 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.690 DEBUG:boto:Params: {} 2023-10-11T15:27:43.691 DEBUG:boto:Token: None 2023-10-11T15:27:43.691 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Knqbe8v7GbBLgsi+PAOxt9wJi4A= 2023-10-11T15:27:43.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Knqbe8v7GbBLgsi+PAOxt9wJi4A=', 'Content-Length': '0'} 2023-10-11T15:27:43.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8abbc608340cf7-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:43.695 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.696 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.696 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.696 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.697 DEBUG:boto:Method: GET 2023-10-11T15:27:43.697 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.698 DEBUG:boto:Data: 2023-10-11T15:27:43.698 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.699 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.700 DEBUG:boto:Params: {} 2023-10-11T15:27:43.700 DEBUG:boto:Token: None 2023-10-11T15:27:43.701 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4= 2023-10-11T15:27:43.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4=', 'Content-Length': '0'} 2023-10-11T15:27:43.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acec26925402dc69-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.705 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.705 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.706 DEBUG:boto:path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.706 DEBUG:boto:auth_path=/dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.706 DEBUG:boto:Method: GET 2023-10-11T15:27:43.707 DEBUG:boto:Path: /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.707 DEBUG:boto:Data: 2023-10-11T15:27:43.708 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.708 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.708 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.709 DEBUG:boto:Params: {} 2023-10-11T15:27:43.709 DEBUG:boto:Token: None 2023-10-11T15:27:43.710 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92?versionId=null 2023-10-11T15:27:43.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4= 2023-10-11T15:27:43.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uKtzUXRZ7NunTDZO75mpnQczpk4=', 'Content-Length': '0'} 2023-10-11T15:27:43.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e57cafe52419be6-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.713 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.714 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.714 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.715 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.715 DEBUG:boto:Method: GET 2023-10-11T15:27:43.716 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.716 DEBUG:boto:Data: 2023-10-11T15:27:43.716 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.717 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.718 DEBUG:boto:Params: {} 2023-10-11T15:27:43.718 DEBUG:boto:Token: None 2023-10-11T15:27:43.719 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.719 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff32b45136f1abf6-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.723 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.723 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.724 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.724 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.725 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.725 DEBUG:boto:Method: GET 2023-10-11T15:27:43.725 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.726 DEBUG:boto:Data: 2023-10-11T15:27:43.726 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.727 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.727 DEBUG:boto:Params: {} 2023-10-11T15:27:43.728 DEBUG:boto:Token: None 2023-10-11T15:27:43.728 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002523a7c1fb276d78-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.734 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.735 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.735 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.736 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.736 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:43.736 DEBUG:boto:Data: 2023-10-11T15:27:43.737 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.738 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.738 DEBUG:boto:Params: {} 2023-10-11T15:27:43.738 DEBUG:boto:Token: None 2023-10-11T15:27:43.739 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92 2023-10-11T15:27:43.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ydK9QdQjECvpA69kA1lgXKkMGIo= 2023-10-11T15:27:43.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ydK9QdQjECvpA69kA1lgXKkMGIo=', 'Content-Length': '0'} 2023-10-11T15:27:43.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a58d52772c32d5e-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.742 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.743 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.743 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.743 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:43.744 DEBUG:boto:Data: 2023-10-11T15:27:43.744 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.745 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.745 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.745 DEBUG:boto:Params: {} 2023-10-11T15:27:43.746 DEBUG:boto:Token: None 2023-10-11T15:27:43.746 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92 2023-10-11T15:27:43.746 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ydK9QdQjECvpA69kA1lgXKkMGIo= 2023-10-11T15:27:43.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ydK9QdQjECvpA69kA1lgXKkMGIo=', 'Content-Length': '0'} 2023-10-11T15:27:43.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4436e689d013064-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:27:43.750 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.750 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.751 DEBUG:boto:Method: GET 2023-10-11T15:27:43.751 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:43.751 DEBUG:boto:Data: 2023-10-11T15:27:43.752 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.752 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.752 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.753 DEBUG:boto:Params: {} 2023-10-11T15:27:43.753 DEBUG:boto:Token: None 2023-10-11T15:27:43.754 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92 2023-10-11T15:27:43.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAkXsnLJ85nvIC8/ZvQLyAf2jgk= 2023-10-11T15:27:43.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAkXsnLJ85nvIC8/ZvQLyAf2jgk=', 'Content-Length': '0'} 2023-10-11T15:27:43.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a84bd20f7512af-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.757 DEBUG:boto:path=/dghqvd-59/obj92 2023-10-11T15:27:43.758 DEBUG:boto:auth_path=/dghqvd-59/obj92 2023-10-11T15:27:43.758 DEBUG:boto:Method: GET 2023-10-11T15:27:43.759 DEBUG:boto:Path: /dghqvd-59/obj92 2023-10-11T15:27:43.759 DEBUG:boto:Data: 2023-10-11T15:27:43.759 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.760 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.760 DEBUG:boto:Params: {} 2023-10-11T15:27:43.761 DEBUG:boto:Token: None 2023-10-11T15:27:43.761 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj92 2023-10-11T15:27:43.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZAkXsnLJ85nvIC8/ZvQLyAf2jgk= 2023-10-11T15:27:43.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZAkXsnLJ85nvIC8/ZvQLyAf2jgk=', 'Content-Length': '0'} 2023-10-11T15:27:43.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:21:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdac3291f1ab2a8a-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.806 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.807 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.807 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.808 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.808 DEBUG:boto:Method: GET 2023-10-11T15:27:43.808 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.809 DEBUG:boto:Data: 2023-10-11T15:27:43.809 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.810 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.810 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.811 DEBUG:boto:Params: {} 2023-10-11T15:27:43.811 DEBUG:boto:Token: None 2023-10-11T15:27:43.811 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cece610f636896a-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.816 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.817 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.817 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.817 DEBUG:boto:path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.818 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.818 DEBUG:boto:Method: GET 2023-10-11T15:27:43.819 DEBUG:boto:Path: /dghqvd-59/?prefix=obj92 2023-10-11T15:27:43.819 DEBUG:boto:Data: 2023-10-11T15:27:43.819 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.820 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.821 DEBUG:boto:Params: {} 2023-10-11T15:27:43.821 DEBUG:boto:Token: None 2023-10-11T15:27:43.821 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.822 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002584be1345a35445-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.827 DEBUG:boto:b'dghqvd-59obj921000falseobj922023-10-11T15:21:14.424Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:43.828 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.828 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.829 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.829 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.830 DEBUG:boto:Method: GET 2023-10-11T15:27:43.830 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.831 DEBUG:boto:Data: 2023-10-11T15:27:43.831 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.831 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.832 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.832 DEBUG:boto:Params: {} 2023-10-11T15:27:43.833 DEBUG:boto:Token: None 2023-10-11T15:27:43.833 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM= 2023-10-11T15:27:43.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM=', 'Content-Length': '0'} 2023-10-11T15:27:43.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483ae3d9039f3869-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.882 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.883 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.883 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.884 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.884 DEBUG:boto:Method: GET 2023-10-11T15:27:43.884 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.885 DEBUG:boto:Data: 2023-10-11T15:27:43.885 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.886 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.886 DEBUG:boto:Params: {} 2023-10-11T15:27:43.887 DEBUG:boto:Token: None 2023-10-11T15:27:43.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM= 2023-10-11T15:27:43.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM=', 'Content-Length': '0'} 2023-10-11T15:27:43.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd2b34338a36e4bc-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.891 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.891 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.892 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.892 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.893 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.893 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.893 DEBUG:boto:Data: 2023-10-11T15:27:43.894 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.895 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.895 DEBUG:boto:Params: {} 2023-10-11T15:27:43.896 DEBUG:boto:Token: None 2023-10-11T15:27:43.896 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BGnI4Y9c3x93h/YSntwM6q4bX64= 2023-10-11T15:27:43.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BGnI4Y9c3x93h/YSntwM6q4bX64=', 'Content-Length': '0'} 2023-10-11T15:27:43.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a78d66cfca7e9f88-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.899 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.900 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.900 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.901 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.901 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.901 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.902 DEBUG:boto:Data: 2023-10-11T15:27:43.902 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.903 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.903 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.904 DEBUG:boto:Params: {} 2023-10-11T15:27:43.904 DEBUG:boto:Token: None 2023-10-11T15:27:43.905 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.905 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BGnI4Y9c3x93h/YSntwM6q4bX64= 2023-10-11T15:27:43.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BGnI4Y9c3x93h/YSntwM6q4bX64=', 'Content-Length': '0'} 2023-10-11T15:27:43.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b61481812057972-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:43.908 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.909 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.909 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.910 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.910 DEBUG:boto:Method: GET 2023-10-11T15:27:43.910 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.911 DEBUG:boto:Data: 2023-10-11T15:27:43.911 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.912 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.912 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.913 DEBUG:boto:Params: {} 2023-10-11T15:27:43.913 DEBUG:boto:Token: None 2023-10-11T15:27:43.913 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.914 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM= 2023-10-11T15:27:43.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM=', 'Content-Length': '0'} 2023-10-11T15:27:43.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5895260fedcde8f-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.917 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.918 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.918 DEBUG:boto:path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.919 DEBUG:boto:auth_path=/dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.919 DEBUG:boto:Method: GET 2023-10-11T15:27:43.919 DEBUG:boto:Path: /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.920 DEBUG:boto:Data: 2023-10-11T15:27:43.920 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.921 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.922 DEBUG:boto:Params: {} 2023-10-11T15:27:43.922 DEBUG:boto:Token: None 2023-10-11T15:27:43.923 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93?versionId=null 2023-10-11T15:27:43.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM= 2023-10-11T15:27:43.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HNIJsDN5Yv3mI76vSFgR/7dahiM=', 'Content-Length': '0'} 2023-10-11T15:27:43.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f06cc0bf26180f1-006526beef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.966 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.967 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.967 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.968 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.968 DEBUG:boto:Method: GET 2023-10-11T15:27:43.969 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.969 DEBUG:boto:Data: 2023-10-11T15:27:43.970 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.971 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.971 DEBUG:boto:Params: {} 2023-10-11T15:27:43.971 DEBUG:boto:Token: None 2023-10-11T15:27:43.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.973 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d951874189cb0cd5-006526beef-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.976 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.977 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:43.978 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:43.978 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.979 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.979 DEBUG:boto:Method: GET 2023-10-11T15:27:43.979 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:43.980 DEBUG:boto:Data: 2023-10-11T15:27:43.980 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:43.981 DEBUG:boto:Port: 8000 2023-10-11T15:27:43.982 DEBUG:boto:Params: {} 2023-10-11T15:27:43.982 DEBUG:boto:Token: None 2023-10-11T15:27:43.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/ 2023-10-11T15:27:43.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8= 2023-10-11T15:27:43.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajpwDCIRfvtyF//GMnBkNCBrhY8=', 'Content-Length': '0'} 2023-10-11T15:27:43.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f59e1cbeeaefb8-006526beef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.988 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:43.989 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.989 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.990 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.990 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:43.991 DEBUG:boto:Data: 2023-10-11T15:27:43.991 DEBUG:boto:Headers: {} 2023-10-11T15:27:43.992 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:43.992 DEBUG:boto:Port: 8001 2023-10-11T15:27:43.992 DEBUG:boto:Params: {} 2023-10-11T15:27:43.993 DEBUG:boto:Token: None 2023-10-11T15:27:43.993 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:43 GMT /dghqvd-59/obj93 2023-10-11T15:27:43.994 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AJhV3hNwGNQIy988hKkYG48KTUg= 2023-10-11T15:27:43.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AJhV3hNwGNQIy988hKkYG48KTUg=', 'Content-Length': '0'} 2023-10-11T15:27:43.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad6b8ebe2b2f7de-006526beef-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:43.997 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:43.997 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:43.998 DEBUG:boto:Method: HEAD 2023-10-11T15:27:43.998 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:43.999 DEBUG:boto:Data: 2023-10-11T15:27:43.999 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.000 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.000 DEBUG:boto:Params: {} 2023-10-11T15:27:44.001 DEBUG:boto:Token: None 2023-10-11T15:27:44.001 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj93 2023-10-11T15:27:44.002 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bNBGoXBs1qoy/uDHXz93AmIL6AQ= 2023-10-11T15:27:44.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bNBGoXBs1qoy/uDHXz93AmIL6AQ=', 'Content-Length': '0'} 2023-10-11T15:27:44.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c9a036a745dec6-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:27:44.008 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:44.009 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:44.009 DEBUG:boto:Method: GET 2023-10-11T15:27:44.010 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:44.010 DEBUG:boto:Data: 2023-10-11T15:27:44.011 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.011 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.011 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.012 DEBUG:boto:Params: {} 2023-10-11T15:27:44.012 DEBUG:boto:Token: None 2023-10-11T15:27:44.013 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj93 2023-10-11T15:27:44.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RzFfmYrWee4sXL0R/BkzMYvjURw= 2023-10-11T15:27:44.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RzFfmYrWee4sXL0R/BkzMYvjURw=', 'Content-Length': '0'} 2023-10-11T15:27:44.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6cbb3299393d5c7-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.058 DEBUG:boto:path=/dghqvd-59/obj93 2023-10-11T15:27:44.059 DEBUG:boto:auth_path=/dghqvd-59/obj93 2023-10-11T15:27:44.059 DEBUG:boto:Method: GET 2023-10-11T15:27:44.060 DEBUG:boto:Path: /dghqvd-59/obj93 2023-10-11T15:27:44.060 DEBUG:boto:Data: 2023-10-11T15:27:44.060 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.061 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.061 DEBUG:boto:Params: {} 2023-10-11T15:27:44.062 DEBUG:boto:Token: None 2023-10-11T15:27:44.062 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj93 2023-10-11T15:27:44.063 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RzFfmYrWee4sXL0R/BkzMYvjURw= 2023-10-11T15:27:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RzFfmYrWee4sXL0R/BkzMYvjURw=', 'Content-Length': '0'} 2023-10-11T15:27:44.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:24:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026ca90c6dea417f6-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.066 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.067 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.067 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.068 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.068 DEBUG:boto:Method: GET 2023-10-11T15:27:44.068 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.069 DEBUG:boto:Data: 2023-10-11T15:27:44.069 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.070 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.071 DEBUG:boto:Params: {} 2023-10-11T15:27:44.071 DEBUG:boto:Token: None 2023-10-11T15:27:44.072 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f066c2b4137f5bc-006526bef0-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.076 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.076 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.077 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.077 DEBUG:boto:path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.078 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.078 DEBUG:boto:Method: GET 2023-10-11T15:27:44.079 DEBUG:boto:Path: /dghqvd-59/?prefix=obj93 2023-10-11T15:27:44.079 DEBUG:boto:Data: 2023-10-11T15:27:44.079 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.080 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.080 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.081 DEBUG:boto:Params: {} 2023-10-11T15:27:44.081 DEBUG:boto:Token: None 2023-10-11T15:27:44.082 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.082 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375fbd9dffb565c4-006526bef0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.088 DEBUG:boto:b'dghqvd-59obj931000falseobj932023-10-11T15:24:15.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:44.089 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.089 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.090 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.090 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.091 DEBUG:boto:Method: GET 2023-10-11T15:27:44.091 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.091 DEBUG:boto:Data: 2023-10-11T15:27:44.092 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.092 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.093 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.093 DEBUG:boto:Params: {} 2023-10-11T15:27:44.094 DEBUG:boto:Token: None 2023-10-11T15:27:44.094 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.095 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc= 2023-10-11T15:27:44.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc=', 'Content-Length': '0'} 2023-10-11T15:27:44.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f98e1605dc0e49-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.098 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.099 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.099 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.100 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.100 DEBUG:boto:Method: GET 2023-10-11T15:27:44.100 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.101 DEBUG:boto:Data: 2023-10-11T15:27:44.102 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.103 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.103 DEBUG:boto:Params: {} 2023-10-11T15:27:44.104 DEBUG:boto:Token: None 2023-10-11T15:27:44.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc= 2023-10-11T15:27:44.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc=', 'Content-Length': '0'} 2023-10-11T15:27:44.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b186b4f3403841-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.150 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.151 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.151 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.152 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.152 DEBUG:boto:Method: HEAD 2023-10-11T15:27:44.153 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.153 DEBUG:boto:Data: 2023-10-11T15:27:44.153 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.154 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.155 DEBUG:boto:Params: {} 2023-10-11T15:27:44.155 DEBUG:boto:Token: None 2023-10-11T15:27:44.156 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vurFOCkhEX0h+3CDgMoLWA1mxHs= 2023-10-11T15:27:44.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vurFOCkhEX0h+3CDgMoLWA1mxHs=', 'Content-Length': '0'} 2023-10-11T15:27:44.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c69ac4e29c4560-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.159 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.160 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.160 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.161 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.161 DEBUG:boto:Method: HEAD 2023-10-11T15:27:44.161 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.162 DEBUG:boto:Data: 2023-10-11T15:27:44.162 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.163 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.164 DEBUG:boto:Params: {} 2023-10-11T15:27:44.164 DEBUG:boto:Token: None 2023-10-11T15:27:44.165 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vurFOCkhEX0h+3CDgMoLWA1mxHs= 2023-10-11T15:27:44.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vurFOCkhEX0h+3CDgMoLWA1mxHs=', 'Content-Length': '0'} 2023-10-11T15:27:44.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268f5e6502bfa30b-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:44.169 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.169 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.169 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.170 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.170 DEBUG:boto:Method: GET 2023-10-11T15:27:44.171 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.171 DEBUG:boto:Data: 2023-10-11T15:27:44.172 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.172 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.172 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.173 DEBUG:boto:Params: {} 2023-10-11T15:27:44.173 DEBUG:boto:Token: None 2023-10-11T15:27:44.174 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.174 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc= 2023-10-11T15:27:44.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc=', 'Content-Length': '0'} 2023-10-11T15:27:44.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c42e815884d0a8-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.218 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.219 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.219 DEBUG:boto:path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.220 DEBUG:boto:auth_path=/dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.220 DEBUG:boto:Method: GET 2023-10-11T15:27:44.220 DEBUG:boto:Path: /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.221 DEBUG:boto:Data: 2023-10-11T15:27:44.221 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.222 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.222 DEBUG:boto:Params: {} 2023-10-11T15:27:44.223 DEBUG:boto:Token: None 2023-10-11T15:27:44.223 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95?versionId=null 2023-10-11T15:27:44.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc= 2023-10-11T15:27:44.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4H/7QMYazAhQ2Vk+ykI5r6mhdcc=', 'Content-Length': '0'} 2023-10-11T15:27:44.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d07739a9489bdb-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.227 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.228 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.228 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.229 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.229 DEBUG:boto:Method: GET 2023-10-11T15:27:44.229 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.230 DEBUG:boto:Data: 2023-10-11T15:27:44.230 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.231 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.232 DEBUG:boto:Params: {} 2023-10-11T15:27:44.232 DEBUG:boto:Token: None 2023-10-11T15:27:44.233 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d8ea7184ae92aa-006526bef0-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.237 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.237 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.238 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.238 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.239 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.239 DEBUG:boto:Method: GET 2023-10-11T15:27:44.240 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.240 DEBUG:boto:Data: 2023-10-11T15:27:44.241 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.242 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.242 DEBUG:boto:Params: {} 2023-10-11T15:27:44.243 DEBUG:boto:Token: None 2023-10-11T15:27:44.243 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.244 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0cafb557980fd5-006526bef0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.249 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.250 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.250 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.251 DEBUG:boto:Method: HEAD 2023-10-11T15:27:44.251 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:44.252 DEBUG:boto:Data: 2023-10-11T15:27:44.252 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.253 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.253 DEBUG:boto:Params: {} 2023-10-11T15:27:44.254 DEBUG:boto:Token: None 2023-10-11T15:27:44.254 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95 2023-10-11T15:27:44.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B0F9c2X1YmTP3JymsA8t5kyIaIM= 2023-10-11T15:27:44.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B0F9c2X1YmTP3JymsA8t5kyIaIM=', 'Content-Length': '0'} 2023-10-11T15:27:44.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e5b5587a32bc283-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.258 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.258 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.259 DEBUG:boto:Method: HEAD 2023-10-11T15:27:44.259 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:44.260 DEBUG:boto:Data: 2023-10-11T15:27:44.260 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.260 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.261 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.261 DEBUG:boto:Params: {} 2023-10-11T15:27:44.262 DEBUG:boto:Token: None 2023-10-11T15:27:44.262 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95 2023-10-11T15:27:44.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B0F9c2X1YmTP3JymsA8t5kyIaIM= 2023-10-11T15:27:44.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B0F9c2X1YmTP3JymsA8t5kyIaIM=', 'Content-Length': '0'} 2023-10-11T15:27:44.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cbdd65a939ecfd-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:27:44.266 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.267 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.267 DEBUG:boto:Method: GET 2023-10-11T15:27:44.267 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:44.268 DEBUG:boto:Data: 2023-10-11T15:27:44.268 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.269 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.269 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.269 DEBUG:boto:Params: {} 2023-10-11T15:27:44.270 DEBUG:boto:Token: None 2023-10-11T15:27:44.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95 2023-10-11T15:27:44.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jRfrkQFfwcgopuvkhU8nI2gRaE= 2023-10-11T15:27:44.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jRfrkQFfwcgopuvkhU8nI2gRaE=', 'Content-Length': '0'} 2023-10-11T15:27:44.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-59:daa316e9-869c-4074-a756-a8adfb4a3573.5751.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e5e0457836a936-006526bef0-1e24-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.274 DEBUG:boto:path=/dghqvd-59/obj95 2023-10-11T15:27:44.274 DEBUG:boto:auth_path=/dghqvd-59/obj95 2023-10-11T15:27:44.275 DEBUG:boto:Method: GET 2023-10-11T15:27:44.275 DEBUG:boto:Path: /dghqvd-59/obj95 2023-10-11T15:27:44.276 DEBUG:boto:Data: 2023-10-11T15:27:44.276 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.277 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.277 DEBUG:boto:Params: {} 2023-10-11T15:27:44.278 DEBUG:boto:Token: None 2023-10-11T15:27:44.278 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/obj95 2023-10-11T15:27:44.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jRfrkQFfwcgopuvkhU8nI2gRaE= 2023-10-11T15:27:44.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jRfrkQFfwcgopuvkhU8nI2gRaE=', 'Content-Length': '0'} 2023-10-11T15:27:44.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:18:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019300ae35d75ebbf-006526bef0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.326 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.327 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.328 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.328 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.328 DEBUG:boto:Method: GET 2023-10-11T15:27:44.329 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.329 DEBUG:boto:Data: 2023-10-11T15:27:44.330 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:27:44.330 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.331 DEBUG:boto:Params: {} 2023-10-11T15:27:44.331 DEBUG:boto:Token: None 2023-10-11T15:27:44.332 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000707d3ac899495315-006526bef0-1e24-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.336 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.337 DEBUG:boto:path=/dghqvd-59/ 2023-10-11T15:27:44.337 DEBUG:boto:auth_path=/dghqvd-59/ 2023-10-11T15:27:44.338 DEBUG:boto:path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.338 DEBUG:boto:auth_path=/dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.338 DEBUG:boto:Method: GET 2023-10-11T15:27:44.339 DEBUG:boto:Path: /dghqvd-59/?prefix=obj95 2023-10-11T15:27:44.339 DEBUG:boto:Data: 2023-10-11T15:27:44.340 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.340 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:27:44.341 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.341 DEBUG:boto:Params: {} 2023-10-11T15:27:44.341 DEBUG:boto:Token: None 2023-10-11T15:27:44.342 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:27:44 GMT /dghqvd-59/ 2023-10-11T15:27:44.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo= 2023-10-11T15:27:44.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:27:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xgDg/21dC+AitTGmBVuRZlE2neo=', 'Content-Length': '0'} 2023-10-11T15:27:44.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e95eeee109b7d77-006526bef0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.348 DEBUG:boto:b'dghqvd-59obj951000falseobj952023-10-11T15:18:55.464Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:27:44.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-59 zones={a2, a1} 2023-10-11T15:27:44.349 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-10-11T15:27:44.354 INFO:rgw_multi.tests:create role zone=a1 name=roles-dghqvd-1 2023-10-11T15:27:44.354 DEBUG:boto:Method: POST 2023-10-11T15:27:44.355 DEBUG:boto:Path: / 2023-10-11T15:27:44.355 DEBUG:boto:Data: 2023-10-11T15:27:44.355 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:27:44.356 DEBUG:boto:Port: 8000 2023-10-11T15:27:44.357 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-11T15:27:44.357 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:27:44.358 DEBUG:boto:Token: None 2023-10-11T15:27:44.359 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8000 x-amz-date:20231011T152744Z host;x-amz-date 12cad3a01d92299eac49b7c5ce7d55d223889459b261bfc23c33e6989a7ec209 2023-10-11T15:27:44.359 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152744Z 20231011/front/smithi158/aws4_request a4c682db61d7699153b67d6060230f5dfbdaecf30a53128b310a9a2cfbe9f94c 2023-10-11T15:27:44.360 DEBUG:boto:Signature: b9e30669a9a18e3bfc0c480a08dd8b4cf2787a7c4b842be8372f7e51f626bc7b 2023-10-11T15:27:44.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152744Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=b9e30669a9a18e3bfc0c480a08dd8b4cf2787a7c4b842be8372f7e51f626bc7b', 'Host': 'smithi158.front.sepia.ceph.com:8000'} 2023-10-11T15:27:44.372 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002196add450826107-006526bef0-1675-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.373 DEBUG:boto:b'0a5e5029-8009-489a-bc9a-4a015f7ecb11roles-dghqvd-1/arn:aws:iam:::role/roles-dghqvd-12023-10-11T15:27:44.364Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000002196add450826107-006526bef0-1675-a1' 2023-10-11T15:27:44.374 INFO:rgw_multi.tests:create role zone=a2 name=roles-dghqvd-2 2023-10-11T15:27:44.374 DEBUG:boto:Method: POST 2023-10-11T15:27:44.375 DEBUG:boto:Path: / 2023-10-11T15:27:44.375 DEBUG:boto:Data: 2023-10-11T15:27:44.376 DEBUG:boto:Headers: {} 2023-10-11T15:27:44.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:27:44.376 DEBUG:boto:Port: 8001 2023-10-11T15:27:44.377 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-10-11T15:27:44.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:27:44.378 DEBUG:boto:Token: None 2023-10-11T15:27:44.378 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8001 x-amz-date:20231011T152744Z host;x-amz-date 90908b6f8e78c848aac18690ed94ffc09643aab237755a9418176f9c09cc6550 2023-10-11T15:27:44.379 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152744Z 20231011/front/smithi158/aws4_request 6cfb8a98ec0bb0482e24f22b26184871dbffda53068f4f0861d2b5c7d91c6185 2023-10-11T15:27:44.379 DEBUG:boto:Signature: 9102c5504a45ea6dab5c3d22970078d05973446b78343af109ba8176f45688f3 2023-10-11T15:27:44.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152744Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=9102c5504a45ea6dab5c3d22970078d05973446b78343af109ba8176f45688f3', 'Host': 'smithi158.front.sepia.ceph.com:8001'} 2023-10-11T15:27:44.399 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a29c327a6efdb93f-006526bef0-1e24-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 11 Oct 2023 15:27:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:27:44.399 DEBUG:boto:b'da10beda-fcd8-4f5e-a662-580f3fdb27f2roles-dghqvd-2/arn:aws:iam:::role/roles-dghqvd-22023-10-11T15:27:44.393Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000a29c327a6efdb93f-006526bef0-1e24-a2' 2023-10-11T15:27:44.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:44.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:44.402 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:44.457 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:44.457 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:44.489 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:27:44.503 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:27:44.503 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.503 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:27:44.503 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:27:44.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.504 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:27:44.504 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:27:44.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.505 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:27:44.505 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:27:44.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.506 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:27:44.506 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:25:38.128846Z" 2023-10-11T15:27:44.506 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.506 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:27:44.507 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:44.508 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:27:44.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:44.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:44.509 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:44.560 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:44.560 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:44.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:44.587+0000 7f65e83d9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:44.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:44.588+0000 7f65e83d9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:44.603 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:27:44.603 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:27:44.604 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:27:44.604 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:27:44.604 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:27:44.604 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:27:44.605 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:44.605 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.605 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:27:44.605 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.605 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:27:44.606 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:44.606 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:44.606 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:27:44.606 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:44.607 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:27:44.607 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:44.607 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:27:44.607 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:44.608 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.608 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:27:44.608 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:44.609 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:44.609 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:27:44.609 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:44.609 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:44.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:44.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:27:44.610 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:44.610 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.610 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.611 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.611 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:27:44.611 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:44.611 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:44.611 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:27:44.612 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:44.612 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:44.612 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:44.612 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:27:44.612 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:44.613 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.613 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:44.613 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:27:44.613 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:44.614 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:44.614 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:27:44.614 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:44.614 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:44.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:44.615 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:27:44.615 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:44.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.615 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:44.616 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:27:44.616 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.616 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:27:44.616 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:27:44.616 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:27:44.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:44.617 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:27:44Z" 2023-10-11T15:27:44.617 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:27:44.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:27:44Z" } 2023-10-11T15:27:44.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:27:44.620 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:44.620 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:44.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:27:49.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:49.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:49.623 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:49.683 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:49.683 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:49.715 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:49.713+0000 7f29a54e4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:49.716 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:49.714+0000 7f29a54e4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:49.729 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:27:49.730 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:27:49.730 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:27:49.730 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:27:49.731 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:27:49.731 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:27:49.731 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:49.731 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.731 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:27:49.732 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:49.732 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:27:49.732 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:49.732 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:49.733 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:27:49.733 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:49.733 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:27:49.733 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:49.733 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:27:49.734 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:49.734 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:49.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.734 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:49.735 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:27:49.735 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:49.735 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:49.735 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:27:49.736 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:49.736 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:49.736 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:49.736 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:27:49.736 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:49.737 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:49.737 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.737 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:49.737 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:27:49.738 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:49.738 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:49.738 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:27:49.738 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:49.738 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:49.739 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:49.739 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:27:49.739 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:49.739 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:49.739 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.740 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:49.740 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:27:49.740 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:49.740 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:49.741 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:27:49.741 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:49.741 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:49.741 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:49.742 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:27:49.742 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:49.742 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:49.742 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:49.742 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:27:49.745 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.745 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:27:49.746 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:27:49.746 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:27:49.746 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:49.746 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:27:49Z" 2023-10-11T15:27:49.746 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:27:49.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:27:49Z" } 2023-10-11T15:27:49.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:27:49.749 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:49.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:49.750 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:27:54.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:54.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:54.753 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:54.810 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:54.811 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:54.845 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:54.843+0000 7fa8ad3b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:54.846 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:54.844+0000 7fa8ad3b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:54.860 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:27:54.860 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:27:54.860 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:27:54.860 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:27:54.861 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:27:54.861 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:27:54.861 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:54.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.861 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:27:54.862 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:54.862 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:27:54.862 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:54.862 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:54.863 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:27:54.863 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:54.863 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:27:54.863 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:54.864 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:27:54.864 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:54.864 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:54.864 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.864 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:54.865 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:27:54.865 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:54.865 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:54.865 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:27:54.865 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:54.866 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:54.866 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:54.866 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:27:54.866 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:54.867 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:54.867 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.867 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:54.867 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:27:54.868 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:54.868 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:54.868 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:27:54.868 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:54.868 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:54.869 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:54.869 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:27:54.869 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:54.869 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:54.870 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.870 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:54.870 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:27:54.870 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:54.871 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:54.871 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:27:54.871 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:54.871 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:54.872 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:54.872 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:27:54.872 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:54.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:54.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:54.873 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:27:54.873 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.873 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:27:54.873 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:27:54.873 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:27:54.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:54.874 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:27:54Z" 2023-10-11T15:27:54.874 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:27:54.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:27:54Z" } 2023-10-11T15:27:54.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:27:54.876 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:54.877 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:27:54.877 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:27:59.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:59.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:27:59.879 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:27:59.937 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:27:59.938 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:27:59.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:59.967+0000 7fc850798780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:59.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:27:59.968+0000 7fc850798780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:27:59.985 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:27:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:27:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:27:59.985 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:27:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:27:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:27:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:59.986 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:27:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:27:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:59.987 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:27:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:27:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:59.988 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:27:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:59.989 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:27:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:27:59.990 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:27:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:59.991 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:27:59.992 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:27:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:59.993 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:27:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.994 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:27:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:27:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:27:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:27:59.995 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:27:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:27:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:27:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:27:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:27:59.996 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:27:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:27:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:27:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.997 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:27:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:27:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:27:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:27:59.998 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:27:59Z" 2023-10-11T15:27:59.998 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:00.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:27:59Z" } 2023-10-11T15:28:00.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:00.001 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:00.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:00.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:05.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:05.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:05.004 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:05.061 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:05.062 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:05.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:05.092+0000 7f113a43d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:05.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:05.093+0000 7f113a43d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:05.109 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:05.109 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:05.109 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:05.109 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:05.110 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:05.110 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:05.110 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:05.110 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.111 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:05.111 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:05.111 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:05.111 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:05.111 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:05.112 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:05.113 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:05.113 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:05.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.113 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:05.113 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:05.114 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:05.114 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:05.114 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:05.114 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:05.115 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:05.115 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:05.115 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:05.115 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:05.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:05.116 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:05.116 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:05.116 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:05.116 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:05.117 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:05.117 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:05.117 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:05.117 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:05.118 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:05.118 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:05.118 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:05.118 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:05.119 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:05.119 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:05.119 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:05.119 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:05.120 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:05.120 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:05.120 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:05.120 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:05.120 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:05.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:05.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:05.121 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:05.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.122 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:05.122 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:05.122 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:05.122 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:05.122 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:05Z" 2023-10-11T15:28:05.123 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:05.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:05Z" } 2023-10-11T15:28:05.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:05.125 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:05.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:05.125 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:10.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:10.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:10.128 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:10.187 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:10.187 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:10.219 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:10.216+0000 7ffa324b6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:10.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:10.218+0000 7ffa324b6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:10.233 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:10.234 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:10.234 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:10.234 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:10.234 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:10.235 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:10.235 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:10.235 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.235 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:10.235 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:10.236 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:10.236 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:10.236 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:10.236 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:10.237 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:10.237 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:10.237 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:10.237 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:10.237 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:10.238 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:10.238 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.238 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:10.238 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:10.239 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:10.239 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:10.239 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:10.239 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:10.239 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:10.240 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:10.240 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:10.240 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:10.240 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:10.241 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.241 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:10.241 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:10.241 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:10.241 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:10.242 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:10.242 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:10.242 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:10.242 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:10.242 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:10.243 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:10.243 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:10.243 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.243 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:10.244 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:10.244 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:10.244 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:10.244 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:10.244 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:10.245 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:10.245 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:10.245 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:10.245 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:10.245 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:10.246 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:10.246 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:10.246 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.246 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:10.246 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:10.247 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:10.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:10.247 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:10Z" 2023-10-11T15:28:10.247 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:10.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:10Z" } 2023-10-11T15:28:10.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:10.249 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:10.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:10.250 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:15.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:15.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:15.253 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:15.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:15.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:15.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:15.342+0000 7f71c648c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:15.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:15.343+0000 7f71c648c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:15.358 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:15.359 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:15.359 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:15.359 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:15.360 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:15.360 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:15.360 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:15.360 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.360 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:15.361 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:15.361 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:15.361 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:15.361 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:15.361 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:15.362 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:15.362 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:15.362 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:15.362 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:15.363 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:15.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:15.364 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:15.364 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:15.364 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:15.365 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:15.365 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:15.365 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:15.365 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:15.365 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:15.366 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:15.366 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:15.366 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:15.366 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:15.367 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:15.367 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:15.367 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:15.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:15.367 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:15.368 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:15.368 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:15.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.368 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:15.369 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:15.369 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:15.369 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:15.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:15.369 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:15.370 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:15.370 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:15.370 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:15.370 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:15.371 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:15.371 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:15.371 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:15.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.371 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:15.372 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:15.372 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:15.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:15.372 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:15Z" 2023-10-11T15:28:15.372 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:15.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:15Z" } 2023-10-11T15:28:15.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:15.375 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:15.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:15.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:20.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:20.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:20.378 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:20.435 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:20.436 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:20.469 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:20.468+0000 7f8fce312780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:20.471 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:20.470+0000 7f8fce312780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:20.484 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:20.485 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:20.485 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:20.485 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:20.485 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:20.486 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:20.486 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:20.486 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.486 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:20.487 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:20.487 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:20.487 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:20.487 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:20.488 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:20.488 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:20.488 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:20.488 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:20.488 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:20.489 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:20.489 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:20.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.489 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:20.489 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:20.490 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:20.490 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:20.490 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:20.490 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:20.490 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:20.491 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:20.491 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:20.491 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:20.491 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:20.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.492 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:20.492 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:20.492 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:20.492 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:20.493 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:20.493 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:20.493 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:20.493 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:20.493 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:20.494 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:20.494 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:20.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.494 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:20.495 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:20.495 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:20.495 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:20.495 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:20.495 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:20.496 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:20.496 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:20.496 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:20.496 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:20.496 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:20.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:20.497 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:20.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.497 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:20.497 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:20.498 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:20.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:20.498 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:20Z" 2023-10-11T15:28:20.498 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:20.499 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:20Z" } 2023-10-11T15:28:20.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:20.500 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:20.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:20.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:25.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:25.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:25.505 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:25.563 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:25.563 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:25.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:25.594+0000 7f576b2f4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:25.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:25.595+0000 7f576b2f4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:25.610 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:25.611 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:25.611 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:25.611 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:25.611 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:25.611 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:25.612 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:25.612 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.612 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:25.612 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:25.612 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:25.613 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:25.613 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:25.613 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:25.613 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:25.613 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:25.614 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:25.614 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:25.614 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:25.614 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:25.615 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.615 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:25.615 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:25.615 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:25.615 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:25.616 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:25.616 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:25.616 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:25.616 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:25.617 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:25.617 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:25.617 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:25.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.617 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:25.618 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:25.618 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:25.618 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:25.618 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:25.619 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:25.619 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:25.619 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:25.619 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:25.619 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:25.620 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:25.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.620 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:25.620 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:25.620 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:25.621 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:25.621 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:25.621 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:25.621 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:25.621 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:25.622 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:25.622 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:25.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:25.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:25.623 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:25.623 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.623 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:25.623 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:25.623 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:25.624 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:25.624 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:25Z" 2023-10-11T15:28:25.624 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:25.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:25Z" } 2023-10-11T15:28:25.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:25.626 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:25.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:25.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:30.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:30.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:30.629 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:30.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:30.687 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:30.718 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:30.717+0000 7f89a88a8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:30.719 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:30.718+0000 7f89a88a8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:30.733 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:30.733 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:30.733 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:30.734 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:30.734 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:30.734 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:30.735 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:30.735 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.735 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:30.735 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:30.735 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:30.736 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:30.736 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:30.736 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:30.736 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:30.737 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:30.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.738 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:30.738 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:30.738 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:30.739 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:30.739 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:30.739 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:30.739 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:30.740 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:30.740 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:30.740 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:30.741 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:30.741 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.741 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:30.741 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:30.742 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:30.742 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:30.742 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:30.742 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:30.742 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:30.743 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:30.743 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:30.743 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:30.744 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:30.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.744 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:30.744 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:30.744 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:30.745 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:30.745 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:30.745 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:30.745 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:30.746 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:30.746 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:30.746 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:30.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:30.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:30.747 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:30.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.747 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:30.748 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:30.748 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:30.748 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:30.748 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:30Z" 2023-10-11T15:28:30.749 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:30.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:30Z" } 2023-10-11T15:28:30.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:30.751 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:30.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:30.752 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:35.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:35.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:35.754 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:35.811 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:35.811 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:35.841+0000 7f8ac002b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:35.843 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:35.842+0000 7f8ac002b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:35.857 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:35.857 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:35.857 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:35.858 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:35.858 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:35.858 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:35.858 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:35.859 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:35.860 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:35.860 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:35.860 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:35.860 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:35.860 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:35.861 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:35.861 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:35.861 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:35.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.861 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:35.862 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:35.862 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:35.862 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:35.862 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:35.863 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:35.863 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:35.863 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:35.863 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:35.863 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:35.864 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:35.864 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.864 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:35.864 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:35.864 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:35.865 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:35.865 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:35.865 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:35.865 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:35.865 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:35.867 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:35.867 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:35.869 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:35.869 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:35.869 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:35.870 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:35.870 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:35.870 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:35.870 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:35.870 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:35.871 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:35.871 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:35.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.871 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:35.871 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:35.872 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:35.872 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:35.872 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:35Z" 2023-10-11T15:28:35.872 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:35.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:35Z" } 2023-10-11T15:28:35.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:35.874 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:35.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:35.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:40.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:40.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:40.878 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:40.936 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:40.936 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:40.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:40.967+0000 7f11f85e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:40.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:40.968+0000 7f11f85e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:40.982 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:40.983 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:40.983 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:40.983 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:40.984 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:40.984 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:40.984 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:40.984 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.984 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:40.985 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:40.985 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:40.985 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:40.985 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:40.986 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:40.986 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:40.986 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:40.986 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:40.986 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:40.987 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:40.987 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:40.987 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:40.988 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:40.988 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:40.988 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:40.988 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:40.988 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:40.989 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:40.989 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:40.989 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:40.989 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:40.989 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:40.990 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:40.991 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:40.992 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:40.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:40.994 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:40.995 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:40.996 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:40.997 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:40.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:40.997 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:40Z" 2023-10-11T15:28:40.997 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:40.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:40Z" } 2023-10-11T15:28:40.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:40.999 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:41.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:41.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:46.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:46.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:46.002 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:46.061 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:46.062 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:46.093 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:46.092+0000 7f8e31c88780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:46.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:46.093+0000 7f8e31c88780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:46.107 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:46.108 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:46.108 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:46.108 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:46.109 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:46.109 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:46.109 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:46.109 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.109 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:46.110 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:46.110 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:46.110 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:46.110 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:46.111 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:46.111 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:46.111 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:46.111 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:46.111 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:46.113 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:46.113 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:46.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.114 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:46.114 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:46.114 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:46.114 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:46.115 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:46.115 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:46.115 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:46.115 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:46.115 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:46.116 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:46.116 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:46.116 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.116 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:46.117 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:46.117 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:46.117 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:46.117 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034974.544814_114.1", 2023-10-11T15:28:46.117 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:46.118 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:46.118 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:46.118 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T15:28:46.118 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:46.119 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:46.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.119 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:46.119 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:46.119 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:46.120 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:46.120 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:46.120 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:46.120 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:46.121 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:46.121 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:46.121 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:46.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:46.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:46.122 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:46.122 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.122 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:46.122 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:46.123 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:46.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:46.123 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:46Z" 2023-10-11T15:28:46.123 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:46.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697034974.544814_114.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T14:36:14.544814Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:46Z" } 2023-10-11T15:28:46.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:46.126 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:46.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:46.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697034974.544814_114.1 2023-10-11T15:28:51.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:51.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:51.129 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:51.187 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:51.187 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:51.220 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:51.219+0000 7f14a6ed3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:51.221 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:51.220+0000 7f14a6ed3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:51.235 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:51.235 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:51.236 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:51.236 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:51.236 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:51.236 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:51.236 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:51.237 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.237 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:51.237 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:51.237 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:51.237 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:51.238 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:51.238 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:51.238 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:51.238 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:51.238 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:51.239 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:51.239 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:51.239 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:51.239 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.240 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:51.240 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:51.240 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:51.240 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:51.240 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:51.241 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:51.241 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:51.241 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:51.241 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:51.242 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:51.242 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:51.242 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.242 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:51.242 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:51.243 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:51.243 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:51.243 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.364027_376.1", 2023-10-11T15:28:51.243 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:51.244 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:51.244 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:51.244 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.364027Z", 2023-10-11T15:28:51.244 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:51.244 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:51.245 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.245 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:51.245 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:51.245 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:51.245 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:51.246 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:51.246 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:51.246 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:51.246 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:51.246 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:51.247 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:51.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:51.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:51.247 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:51.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.248 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:51.248 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:51.248 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:51.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:51.248 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:51Z" 2023-10-11T15:28:51.249 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:51.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.364027_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.364027Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:51Z" } 2023-10-11T15:28:51.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.364027_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.364027Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:51.251 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:51.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.364027_376.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:51.251 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1697038064.387448_379.1 target=1_1697038064.364027_376.1 2023-10-11T15:28:56.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.254 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:56.312 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:56.312 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:56.345 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:56.344+0000 7f9b7c25b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:56.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:56.345+0000 7f9b7c25b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:56.360 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:56.360 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:56.360 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:56.360 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:56.361 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:56.361 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:56.361 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.361 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.362 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:56.362 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.362 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:56.362 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.362 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.363 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:56.363 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.363 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:56.363 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.364 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:56.364 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.364 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.364 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.364 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.365 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:56.365 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.365 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.365 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:56.365 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:56.366 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:56.367 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.368 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.369 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.370 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697037938.128846_495.1", 2023-10-11T15:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.128846Z", 2023-10-11T15:28:56.371 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.372 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:56Z" 2023-10-11T15:28:56.373 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:56.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697037938.128846_495.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.128846Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:56Z" } 2023-10-11T15:28:56.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697037938.128846_495.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.128846Z', 'realm_epoch': 2}}] 2023-10-11T15:28:56.376 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:56.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697037938.128846_495.1'} 2023-10-11T15:28:56.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:28:56.377 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dghqvd-1 zones={a2, a1} 2023-10-11T15:28:56.377 DEBUG:boto:Method: POST 2023-10-11T15:28:56.378 DEBUG:boto:Path: / 2023-10-11T15:28:56.378 DEBUG:boto:Data: 2023-10-11T15:28:56.379 DEBUG:boto:Headers: {} 2023-10-11T15:28:56.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:28:56.379 DEBUG:boto:Port: 8001 2023-10-11T15:28:56.380 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-11T15:28:56.380 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:28:56.381 DEBUG:boto:Token: None 2023-10-11T15:28:56.382 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8001 x-amz-date:20231011T152856Z host;x-amz-date f3a5ddd264bcaccbc0d34c2da743d986af8ed564ca56f602f30615817c67f0c1 2023-10-11T15:28:56.382 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152856Z 20231011/front/smithi158/aws4_request 983e3513aa1fd0489976fc5cb16a2d9e3597671256bd89ff2cf65e92f30aa75c 2023-10-11T15:28:56.383 DEBUG:boto:Signature: 788b6a76a5a2841efb47baf0c3e84c950d2bb90d0c71464596d38afeb04621e6 2023-10-11T15:28:56.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152856Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=788b6a76a5a2841efb47baf0c3e84c950d2bb90d0c71464596d38afeb04621e6', 'Host': 'smithi158.front.sepia.ceph.com:8001'} 2023-10-11T15:28:56.387 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ee004d38f3a429a-006526bf38-1e24-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 11 Oct 2023 15:28:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:28:56.387 DEBUG:boto:b'tx000001ee004d38f3a429a-006526bf38-1e24-a20a5e5029-8009-489a-bc9a-4a015f7ecb11roles-dghqvd-1/arn:aws:iam:::role/roles-dghqvd-12023-10-11T15:27:44.364Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-11T15:28:56.389 DEBUG:boto:Method: POST 2023-10-11T15:28:56.389 DEBUG:boto:Path: / 2023-10-11T15:28:56.389 DEBUG:boto:Data: 2023-10-11T15:28:56.390 DEBUG:boto:Headers: {} 2023-10-11T15:28:56.390 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:28:56.391 DEBUG:boto:Port: 8000 2023-10-11T15:28:56.391 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-11T15:28:56.391 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:28:56.392 DEBUG:boto:Token: None 2023-10-11T15:28:56.392 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8000 x-amz-date:20231011T152856Z host;x-amz-date f3a5ddd264bcaccbc0d34c2da743d986af8ed564ca56f602f30615817c67f0c1 2023-10-11T15:28:56.393 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152856Z 20231011/front/smithi158/aws4_request feefcd2cc29675e2f7762f7cee7f82d6140b689090cc922cb93cef67e17eaed6 2023-10-11T15:28:56.393 DEBUG:boto:Signature: 5edd6c19ae70a81c33ce2072e5888022c4712a1433e81e12439b0cabb9822276 2023-10-11T15:28:56.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152856Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=5edd6c19ae70a81c33ce2072e5888022c4712a1433e81e12439b0cabb9822276', 'Host': 'smithi158.front.sepia.ceph.com:8000'} 2023-10-11T15:28:56.397 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047879b33919f875a-006526bf38-1675-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 11 Oct 2023 15:28:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:28:56.397 DEBUG:boto:b'tx0000047879b33919f875a-006526bf38-1675-a10a5e5029-8009-489a-bc9a-4a015f7ecb11roles-dghqvd-1/arn:aws:iam:::role/roles-dghqvd-12023-10-11T15:27:44.364Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-11T15:28:56.398 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dghqvd-1 2023-10-11T15:28:56.399 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dghqvd-1 zones={a2, a1} 2023-10-11T15:28:56.399 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dghqvd-2 zones={a1, a2} 2023-10-11T15:28:56.399 DEBUG:boto:Method: POST 2023-10-11T15:28:56.400 DEBUG:boto:Path: / 2023-10-11T15:28:56.400 DEBUG:boto:Data: 2023-10-11T15:28:56.401 DEBUG:boto:Headers: {} 2023-10-11T15:28:56.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:28:56.402 DEBUG:boto:Port: 8000 2023-10-11T15:28:56.402 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-11T15:28:56.403 DEBUG:boto:Token: None 2023-10-11T15:28:56.403 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8000 x-amz-date:20231011T152856Z host;x-amz-date e811aa11596d6038e9f73169781a1e95cc23b5bd35f91749eda3f9d40d7985e1 2023-10-11T15:28:56.403 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152856Z 20231011/front/smithi158/aws4_request 773c420d7240394d48d854b29ef5ad1450b87c366cb4cc3b75440eafe62933b7 2023-10-11T15:28:56.404 DEBUG:boto:Signature: 87ddb92058c871ef5c84ab1fe5f0fb71b5c59ef8fa1088cf7442adf8928a4b69 2023-10-11T15:28:56.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152856Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=87ddb92058c871ef5c84ab1fe5f0fb71b5c59ef8fa1088cf7442adf8928a4b69', 'Host': 'smithi158.front.sepia.ceph.com:8000'} 2023-10-11T15:28:56.406 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b3c4d3a3af4a697-006526bf38-1675-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 11 Oct 2023 15:28:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:28:56.406 DEBUG:boto:b'tx000005b3c4d3a3af4a697-006526bf38-1675-a1da10beda-fcd8-4f5e-a662-580f3fdb27f2roles-dghqvd-2/arn:aws:iam:::role/roles-dghqvd-22023-10-11T15:27:44.384Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-11T15:28:56.407 DEBUG:boto:Method: POST 2023-10-11T15:28:56.408 DEBUG:boto:Path: / 2023-10-11T15:28:56.408 DEBUG:boto:Data: 2023-10-11T15:28:56.408 DEBUG:boto:Headers: {} 2023-10-11T15:28:56.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com 2023-10-11T15:28:56.409 DEBUG:boto:Port: 8001 2023-10-11T15:28:56.409 DEBUG:boto:Params: {'RoleName': 'roles-dghqvd-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-10-11T15:28:56.410 DEBUG:boto:Token: None 2023-10-11T15:28:56.410 DEBUG:boto:CanonicalRequest: POST / host:smithi158.front.sepia.ceph.com:8001 x-amz-date:20231011T152856Z host;x-amz-date e811aa11596d6038e9f73169781a1e95cc23b5bd35f91749eda3f9d40d7985e1 2023-10-11T15:28:56.411 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20231011T152856Z 20231011/front/smithi158/aws4_request e6f334078eb2eda1f4441368e0461bf94ad90feaadd77ea7c1ddfb7814e1d78e 2023-10-11T15:28:56.411 DEBUG:boto:Signature: 8735d539d6bbcb935d4fdd91d4d83e62d7ef070d057cb648d7be0372d692a1ab 2023-10-11T15:28:56.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20231011T152856Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=MWUFUL61ESHN8BNC6ISZ/20231011/front/smithi158/aws4_request,SignedHeaders=host;x-amz-date,Signature=8735d539d6bbcb935d4fdd91d4d83e62d7ef070d057cb648d7be0372d692a1ab', 'Host': 'smithi158.front.sepia.ceph.com:8001'} 2023-10-11T15:28:56.414 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003eb4376858734152-006526bf38-1e24-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 11 Oct 2023 15:28:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:28:56.415 DEBUG:boto:b'tx000003eb4376858734152-006526bf38-1e24-a2da10beda-fcd8-4f5e-a662-580f3fdb27f2roles-dghqvd-2/arn:aws:iam:::role/roles-dghqvd-22023-10-11T15:27:44.384Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-10-11T15:28:56.415 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dghqvd-2 2023-10-11T15:28:56.416 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dghqvd-2 zones={a1, a2} 2023-10-11T15:28:56.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-10-11T15:28:56.417 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:28:56.418 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:28:56.418 DEBUG:boto:Method: PUT 2023-10-11T15:28:56.419 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:28:56.419 DEBUG:boto:Data: 2023-10-11T15:28:56.420 DEBUG:boto:Headers: {} 2023-10-11T15:28:56.420 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:28:56.420 DEBUG:boto:Port: 8000 2023-10-11T15:28:56.421 DEBUG:boto:Params: {} 2023-10-11T15:28:56.421 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:28:56.422 DEBUG:boto:Token: None 2023-10-11T15:28:56.422 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:28:56 GMT /dghqvd-60/ 2023-10-11T15:28:56.423 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8IY9wxCCH9l5fPdvmbRnmSLpVNA= 2023-10-11T15:28:56.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:28:56 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8IY9wxCCH9l5fPdvmbRnmSLpVNA=', 'Content-Length': '0'} 2023-10-11T15:28:56.441 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009cdc771ba8b02e89-006526bf38-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:28:56 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:28:56.442 DEBUG:rgw_multi.tests:created bucket=dghqvd-60 2023-10-11T15:28:56.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.443 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:56.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:56.493 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:56.523 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:28:56.536 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:28:56.536 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.536 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:56.536 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:28:56.537 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.537 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.537 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:56.537 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:28:56.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.538 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.538 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:28:56.538 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:28:56.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.539 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.539 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038136.439138_499.1", 2023-10-11T15:28:56.539 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:28:56.439138Z" 2023-10-11T15:28:56.539 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.539 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:28:56.540 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038136.439138_499.1'} 2023-10-11T15:28:56.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:28:56.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:28:56.542 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:28:56.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:28:56.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:28:56.622 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:56.621+0000 7effbf7a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:56.623 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:28:56.622+0000 7effbf7a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:28:56.638 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:28:56.639 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:28:56.639 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:28:56.639 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:28:56.639 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:28:56.640 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:28:56.640 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.640 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.640 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:28:56.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.641 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:28:56.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.641 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:28:56.642 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.642 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:28:56.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.642 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:28:56.643 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.643 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.643 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:28:56.644 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.644 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.644 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:28:56.644 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.644 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.645 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.645 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:28:56.645 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.645 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.645 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.646 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.646 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:28:56.646 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.646 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.647 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:28:56.647 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.647 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.647 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.648 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:28:56.648 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.648 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.648 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:28:56.649 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:28:56.649 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:28:56.649 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:28:56.649 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038136.433120_496.1", 2023-10-11T15:28:56.649 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:28:56.650 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:28:56.650 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:28:56.650 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:28:56.433120Z", 2023-10-11T15:28:56.650 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:28:56.651 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.651 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:28:56.651 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:28:56.651 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.651 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:28:56.652 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:28:56.652 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:28:56.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:28:56.652 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:28:56Z" 2023-10-11T15:28:56.652 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:28:56.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038136.433120_496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:28:56.433120Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:28:56Z" } 2023-10-11T15:28:56.654 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038136.433120_496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:28:56.433120Z', 'realm_epoch': 2}}] 2023-10-11T15:28:56.655 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038136.439138_499.1'} 2023-10-11T15:28:56.655 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038136.433120_496.1'} 2023-10-11T15:28:56.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038136.439138_499.1 target=1_1697038136.433120_496.1 2023-10-11T15:29:01.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:29:01.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:29:01.659 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:29:01.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:29:01.718 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:29:01.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:01.748+0000 7f189aa2d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:29:01.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:01.749+0000 7f189aa2d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:29:01.764 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:29:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:29:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:29:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:29:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:29:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:29:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:29:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:29:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:29:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:29:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:29:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:29:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:29:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:29:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:29:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:29:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:29:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:29:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:29:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:29:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:29:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:29:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:29:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:29:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:29:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:29:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:29:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:29:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:29:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:29:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:29:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:29:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:29:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:29:01.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:29:01.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:29:01.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:29:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:29:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:29:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:29:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:29:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.775 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:29:01.775 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:29:01.775 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:29:01.775 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:29:01.776 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038136.439138_499.1", 2023-10-11T15:29:01.776 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:29:01.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:29:01.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:29:01.777 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:28:56.439138Z", 2023-10-11T15:29:01.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:29:01.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:29:01.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:29:01.777 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:29:01.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.778 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:29:01.778 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:29:01.778 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:29:01.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:29:01.779 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:29:01Z" 2023-10-11T15:29:01.779 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:29:01.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038136.439138_499.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:28:56.439138Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:29:01Z" } 2023-10-11T15:29:01.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038136.439138_499.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:28:56.439138Z', 'realm_epoch': 2}}] 2023-10-11T15:29:01.781 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038136.439138_499.1'} 2023-10-11T15:29:01.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038136.439138_499.1'} 2023-10-11T15:29:01.782 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:29:01.782 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:29:01.783 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:29:01.784 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:29:01.782+0000 7f10b496b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 170306) UID: 0 2023-10-11T15:29:01.784 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:29:01.782+0000 7f10b8bd47c0 -1 shutting down 2023-10-11T15:29:06.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:29:07.885 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:29:07.886 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:29:07.886 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:29:07.887 DEBUG:boto:Method: HEAD 2023-10-11T15:29:07.887 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:29:07.887 DEBUG:boto:Data: 2023-10-11T15:29:07.888 DEBUG:boto:Headers: {} 2023-10-11T15:29:07.888 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.889 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.889 DEBUG:boto:Params: {} 2023-10-11T15:29:07.889 DEBUG:boto:Token: None 2023-10-11T15:29:07.890 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/ 2023-10-11T15:29:07.890 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU= 2023-10-11T15:29:07.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU=', 'Content-Length': '0'} 2023-10-11T15:29:07.894 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009696dc50ba5b886c-006526bf43-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.895 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:29:07.896 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:29:07.896 DEBUG:boto:Method: PUT 2023-10-11T15:29:07.896 DEBUG:boto:Path: /dghqvd-60/obj11 2023-10-11T15:29:07.897 DEBUG:boto:Data: 2023-10-11T15:29:07.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:29:07.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.898 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.898 DEBUG:boto:Params: {} 2023-10-11T15:29:07.899 DEBUG:boto:Token: None 2023-10-11T15:29:07.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/obj11 2023-10-11T15:29:07.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gxpjp5dq/WScxKmtKsBonZNCgsk= 2023-10-11T15:29:07.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gxpjp5dq/WScxKmtKsBonZNCgsk='} 2023-10-11T15:29:07.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f83ae789e5865ca5-006526bf43-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.909 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:29:07.909 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:29:07.910 DEBUG:boto:Method: HEAD 2023-10-11T15:29:07.910 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:29:07.910 DEBUG:boto:Data: 2023-10-11T15:29:07.911 DEBUG:boto:Headers: {} 2023-10-11T15:29:07.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.912 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.912 DEBUG:boto:Params: {} 2023-10-11T15:29:07.912 DEBUG:boto:Token: None 2023-10-11T15:29:07.913 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/ 2023-10-11T15:29:07.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU= 2023-10-11T15:29:07.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU=', 'Content-Length': '0'} 2023-10-11T15:29:07.916 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064a0d31962847faf-006526bf43-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.917 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:29:07.917 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:29:07.918 DEBUG:boto:Method: PUT 2023-10-11T15:29:07.918 DEBUG:boto:Path: /dghqvd-60/obj22 2023-10-11T15:29:07.919 DEBUG:boto:Data: 2023-10-11T15:29:07.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:29:07.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.920 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.920 DEBUG:boto:Params: {} 2023-10-11T15:29:07.921 DEBUG:boto:Token: None 2023-10-11T15:29:07.921 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/obj22 2023-10-11T15:29:07.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pI6FIZFIBTQob7k25Pd85phbCN4= 2023-10-11T15:29:07.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pI6FIZFIBTQob7k25Pd85phbCN4='} 2023-10-11T15:29:07.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce300b0c2f8db973-006526bf43-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.927 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:29:07.928 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:29:07.928 DEBUG:boto:Method: HEAD 2023-10-11T15:29:07.929 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:29:07.929 DEBUG:boto:Data: 2023-10-11T15:29:07.929 DEBUG:boto:Headers: {} 2023-10-11T15:29:07.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.930 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.931 DEBUG:boto:Params: {} 2023-10-11T15:29:07.931 DEBUG:boto:Token: None 2023-10-11T15:29:07.932 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/ 2023-10-11T15:29:07.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU= 2023-10-11T15:29:07.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU=', 'Content-Length': '0'} 2023-10-11T15:29:07.935 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039ce2eb83782a96c-006526bf43-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.935 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:29:07.936 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:29:07.936 DEBUG:boto:Method: PUT 2023-10-11T15:29:07.937 DEBUG:boto:Path: /dghqvd-60/obj33 2023-10-11T15:29:07.937 DEBUG:boto:Data: 2023-10-11T15:29:07.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:29:07.938 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.938 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.939 DEBUG:boto:Params: {} 2023-10-11T15:29:07.939 DEBUG:boto:Token: None 2023-10-11T15:29:07.940 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/obj33 2023-10-11T15:29:07.940 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DwAYz8+w9w9ukwCK3t/uEU20bKg= 2023-10-11T15:29:07.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DwAYz8+w9w9ukwCK3t/uEU20bKg='} 2023-10-11T15:29:07.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001eda09e17e15a0fa-006526bf43-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.987 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:29:07.988 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:29:07.988 DEBUG:boto:Method: HEAD 2023-10-11T15:29:07.989 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:29:07.989 DEBUG:boto:Data: 2023-10-11T15:29:07.989 DEBUG:boto:Headers: {} 2023-10-11T15:29:07.990 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.990 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.990 DEBUG:boto:Params: {} 2023-10-11T15:29:07.991 DEBUG:boto:Token: None 2023-10-11T15:29:07.991 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:29:07 GMT /dghqvd-60/ 2023-10-11T15:29:07.992 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU= 2023-10-11T15:29:07.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:29:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9jgG2v0hY8Qgl9hYVef8DO59MPU=', 'Content-Length': '0'} 2023-10-11T15:29:07.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008f97fd7c48a85a5-006526bf43-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:07.996 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:29:07.996 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:29:07.996 DEBUG:boto:Method: PUT 2023-10-11T15:29:07.997 DEBUG:boto:Path: /dghqvd-60/obj44 2023-10-11T15:29:07.997 DEBUG:boto:Data: 2023-10-11T15:29:07.998 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:29:07.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:07.998 DEBUG:boto:Port: 8000 2023-10-11T15:29:07.999 DEBUG:boto:Params: {} 2023-10-11T15:29:07.999 DEBUG:boto:Token: None 2023-10-11T15:29:08.000 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:29:08 GMT /dghqvd-60/obj44 2023-10-11T15:29:08.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6ORfqZknvsIzW3Op1YpsXF45s2Q= 2023-10-11T15:29:08.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:29:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6ORfqZknvsIzW3Op1YpsXF45s2Q='} 2023-10-11T15:29:08.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013eb84c54705c7ac-006526bf44-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:08.007 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:29:08.008 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:29:08.008 DEBUG:boto:Method: HEAD 2023-10-11T15:29:08.008 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:29:08.009 DEBUG:boto:Data: 2023-10-11T15:29:08.009 DEBUG:boto:Headers: {} 2023-10-11T15:29:08.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:08.010 DEBUG:boto:Port: 8000 2023-10-11T15:29:08.010 DEBUG:boto:Params: {} 2023-10-11T15:29:08.011 DEBUG:boto:Token: None 2023-10-11T15:29:08.011 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:29:08 GMT /dghqvd-60/ 2023-10-11T15:29:08.012 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DSGiV+tgPuNZrQlpdkE7AfOdWj4= 2023-10-11T15:29:08.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:29:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DSGiV+tgPuNZrQlpdkE7AfOdWj4=', 'Content-Length': '0'} 2023-10-11T15:29:08.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001be8a2471c49dd72-006526bf44-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:08.015 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:29:08.016 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:29:08.016 DEBUG:boto:Method: PUT 2023-10-11T15:29:08.016 DEBUG:boto:Path: /dghqvd-60/obj55 2023-10-11T15:29:08.017 DEBUG:boto:Data: 2023-10-11T15:29:08.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:29:08.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:29:08.018 DEBUG:boto:Port: 8000 2023-10-11T15:29:08.018 DEBUG:boto:Params: {} 2023-10-11T15:29:08.019 DEBUG:boto:Token: None 2023-10-11T15:29:08.019 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:29:08 GMT /dghqvd-60/obj55 2023-10-11T15:29:08.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D9OG/k3gHuUxaeTlwOXdOY2I5vI= 2023-10-11T15:29:08.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:29:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D9OG/k3gHuUxaeTlwOXdOY2I5vI='} 2023-10-11T15:29:08.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066c8e5f403c00634-006526bf44-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:29:08.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:29:08.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:29:08.068 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:29:08.126 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:29:08.127 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:29:08.161 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:08.158+0000 7f8457f10780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-11T15:29:08.162 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:08.159+0000 7f8457f10780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-11T15:29:08.183 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:29:08.184 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:29:08.186 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:29:08.186 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:29:08.208 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:29:08.208 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:29:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:29:08.210 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:29:09.211 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:29:09.235 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:29:09.235 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:29:09.235 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:29:09.236 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:29:09.236 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-60 2023-10-11T15:29:09.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:29:09.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:29:09.238 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:29:09.332 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:29:09.333 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:29:09.366 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:09.364+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:11.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:29:14.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:14.364+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:19.367 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:19.365+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:24.368 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:24.366+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:29.369 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:29.367+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:34.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:34.367+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:39.370 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:39.368+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:44.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:44.368+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:49.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:49.370+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:54.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:54.370+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:29:59.372 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:29:59.371+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:04.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:04.371+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:09.373 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:09.372+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:14.374 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:14.373+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:19.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:19.373+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:24.375 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:24.374+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:29.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:29.374+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:34.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:34.375+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:39.377 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:39.376+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:44.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:44.375+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:49.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:49.376+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:54.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:54.376+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:30:59.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:30:59.377+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:04.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:04.377+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:09.380 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:09.378+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:14.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:14.379+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:19.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:19.380+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:24.382 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:24.381+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:29.383 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:29.382+0000 7f281ec21780 1 waiting to reach incremental sync.. 2023-10-11T15:31:34.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:34.383+0000 7f281ec21780 1 bucket sync caught up with source: 2023-10-11T15:31:34.385 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.79.6, 00000000002.62.6, 00000000002.107.6] 2023-10-11T15:31:34.385 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.79.6, 00000000002.62.6, 00000000002.107.6] 2023-10-11T15:31:34.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:34.383+0000 7f281ec21780 0 bucket checkpoint complete 2023-10-11T15:31:34.401 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-60 2023-10-11T15:31:34.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:31:34.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:31:34.403 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:31:34.466 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:34.466 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:34.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:34.502+0000 7f14800e8780 1 bucket sync caught up with source: 2023-10-11T15:31:34.504 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.148.6, 00000000002.68.6, 00000000002.326.6] 2023-10-11T15:31:34.504 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.148.6, 00000000002.68.6, 00000000002.326.6] 2023-10-11T15:31:34.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:34.502+0000 7f14800e8780 0 bucket checkpoint complete 2023-10-11T15:31:34.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-60 zones={a2, a1} 2023-10-11T15:31:34.517 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.518 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.518 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.519 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:31:34.519 DEBUG:boto:Data: 2023-10-11T15:31:34.520 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.520 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.520 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.521 DEBUG:boto:Params: {} 2023-10-11T15:31:34.521 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:31:34.522 DEBUG:boto:Token: None 2023-10-11T15:31:34.522 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.523 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1bxL4kbA6/WWatxRpDVhDcoGvWY= 2023-10-11T15:31:34.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1bxL4kbA6/WWatxRpDVhDcoGvWY=', 'Content-Length': '0'} 2023-10-11T15:31:34.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eeba9b887f91bd8-006526bfd6-1edc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.530 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.530 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.530 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.531 DEBUG:boto:Path: /dghqvd-60/ 2023-10-11T15:31:34.531 DEBUG:boto:Data: 2023-10-11T15:31:34.532 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.533 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.533 DEBUG:boto:Params: {} 2023-10-11T15:31:34.534 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:31:34.534 DEBUG:boto:Token: None 2023-10-11T15:31:34.534 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1bxL4kbA6/WWatxRpDVhDcoGvWY= 2023-10-11T15:31:34.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1bxL4kbA6/WWatxRpDVhDcoGvWY=', 'Content-Length': '0'} 2023-10-11T15:31:34.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023dd7ad833e2b285-006526bfd6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:31:34.540 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.541 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.542 DEBUG:boto:path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.542 DEBUG:boto:auth_path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.543 DEBUG:boto:Method: GET 2023-10-11T15:31:34.543 DEBUG:boto:Path: /dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.544 DEBUG:boto:Data: 2023-10-11T15:31:34.544 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.545 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.545 DEBUG:boto:Params: {} 2023-10-11T15:31:34.546 DEBUG:boto:Token: None 2023-10-11T15:31:34.546 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/?versions 2023-10-11T15:31:34.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws= 2023-10-11T15:31:34.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws=', 'Content-Length': '0'} 2023-10-11T15:31:34.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097ce7887895e94ec-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.551 DEBUG:boto:b'dghqvd-60999falseobj11nulltrue2023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:31:34.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:31:34.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:31:34.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:31:34.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:31:34.554 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:31:34.554 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.555 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.555 DEBUG:boto:path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.555 DEBUG:boto:auth_path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.556 DEBUG:boto:Method: GET 2023-10-11T15:31:34.556 DEBUG:boto:Path: /dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.557 DEBUG:boto:Data: 2023-10-11T15:31:34.557 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.558 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.558 DEBUG:boto:Params: {} 2023-10-11T15:31:34.559 DEBUG:boto:Token: None 2023-10-11T15:31:34.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/?versions 2023-10-11T15:31:34.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws= 2023-10-11T15:31:34.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws=', 'Content-Length': '0'} 2023-10-11T15:31:34.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a70c97913b35bb73-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.563 DEBUG:boto:b'dghqvd-60999falseobj11nulltrue2023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:31:34.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:31:34.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:31:34.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:31:34.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:31:34.569 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.570 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.571 DEBUG:boto:path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.571 DEBUG:boto:auth_path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.572 DEBUG:boto:Method: GET 2023-10-11T15:31:34.572 DEBUG:boto:Path: /dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.572 DEBUG:boto:Data: 2023-10-11T15:31:34.573 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.573 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.574 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.574 DEBUG:boto:Params: {} 2023-10-11T15:31:34.575 DEBUG:boto:Token: None 2023-10-11T15:31:34.575 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/?versions 2023-10-11T15:31:34.576 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws= 2023-10-11T15:31:34.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws=', 'Content-Length': '0'} 2023-10-11T15:31:34.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e6e8377f9391b9c-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.579 DEBUG:boto:b'dghqvd-60999falseobj11nulltrue2023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.580 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.581 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.581 DEBUG:boto:path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.582 DEBUG:boto:auth_path=/dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.582 DEBUG:boto:Method: GET 2023-10-11T15:31:34.582 DEBUG:boto:Path: /dghqvd-60/?versions&max-keys=999 2023-10-11T15:31:34.583 DEBUG:boto:Data: 2023-10-11T15:31:34.583 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.584 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.584 DEBUG:boto:Params: {} 2023-10-11T15:31:34.585 DEBUG:boto:Token: None 2023-10-11T15:31:34.585 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/?versions 2023-10-11T15:31:34.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws= 2023-10-11T15:31:34.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6ixazG550CFDY55ckhBNuhRTws=', 'Content-Length': '0'} 2023-10-11T15:31:34.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca276e721ceda2cc-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.590 DEBUG:boto:b'dghqvd-60999falseobj11nulltrue2023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:31:34.591 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.591 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.592 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.592 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.593 DEBUG:boto:Method: GET 2023-10-11T15:31:34.593 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.594 DEBUG:boto:Data: 2023-10-11T15:31:34.594 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.595 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.595 DEBUG:boto:Params: {} 2023-10-11T15:31:34.596 DEBUG:boto:Token: None 2023-10-11T15:31:34.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M= 2023-10-11T15:31:34.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M=', 'Content-Length': '0'} 2023-10-11T15:31:34.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257bcdd8b01ff3d3-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.600 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.600 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.601 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.601 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.602 DEBUG:boto:Method: GET 2023-10-11T15:31:34.602 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.603 DEBUG:boto:Data: 2023-10-11T15:31:34.603 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.604 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.604 DEBUG:boto:Params: {} 2023-10-11T15:31:34.605 DEBUG:boto:Token: None 2023-10-11T15:31:34.605 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M= 2023-10-11T15:31:34.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M=', 'Content-Length': '0'} 2023-10-11T15:31:34.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000877abe2778cb846d-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.609 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.609 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.610 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.610 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.610 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.611 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.611 DEBUG:boto:Data: 2023-10-11T15:31:34.612 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.613 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.613 DEBUG:boto:Params: {} 2023-10-11T15:31:34.613 DEBUG:boto:Token: None 2023-10-11T15:31:34.614 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQzA/Wxy+jcTYrekAGO2RW5Sk7I= 2023-10-11T15:31:34.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQzA/Wxy+jcTYrekAGO2RW5Sk7I=', 'Content-Length': '0'} 2023-10-11T15:31:34.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b789f1dea4a7d0ea-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.617 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.618 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.618 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.619 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.619 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.619 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.620 DEBUG:boto:Data: 2023-10-11T15:31:34.620 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.621 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.621 DEBUG:boto:Params: {} 2023-10-11T15:31:34.622 DEBUG:boto:Token: None 2023-10-11T15:31:34.622 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQzA/Wxy+jcTYrekAGO2RW5Sk7I= 2023-10-11T15:31:34.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQzA/Wxy+jcTYrekAGO2RW5Sk7I=', 'Content-Length': '0'} 2023-10-11T15:31:34.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004940b361487fcec5-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:31:34.626 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.627 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.627 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.628 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.628 DEBUG:boto:Method: GET 2023-10-11T15:31:34.628 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.629 DEBUG:boto:Data: 2023-10-11T15:31:34.629 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.630 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.631 DEBUG:boto:Params: {} 2023-10-11T15:31:34.631 DEBUG:boto:Token: None 2023-10-11T15:31:34.632 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M= 2023-10-11T15:31:34.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M=', 'Content-Length': '0'} 2023-10-11T15:31:34.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002475ec26e1ee9f52-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.678 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.679 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.679 DEBUG:boto:path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.680 DEBUG:boto:auth_path=/dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.680 DEBUG:boto:Method: GET 2023-10-11T15:31:34.681 DEBUG:boto:Path: /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.681 DEBUG:boto:Data: 2023-10-11T15:31:34.682 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.683 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.683 DEBUG:boto:Params: {} 2023-10-11T15:31:34.684 DEBUG:boto:Token: None 2023-10-11T15:31:34.684 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11?versionId=null 2023-10-11T15:31:34.685 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M= 2023-10-11T15:31:34.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rrdtBLuq6npBjDX/V/qYwYQOc8M=', 'Content-Length': '0'} 2023-10-11T15:31:34.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd509d3d79c7c23f-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.688 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.688 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.689 DEBUG:boto:path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.689 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.690 DEBUG:boto:Method: GET 2023-10-11T15:31:34.690 DEBUG:boto:Path: /dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.690 DEBUG:boto:Data: 2023-10-11T15:31:34.691 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.691 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.692 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.692 DEBUG:boto:Params: {} 2023-10-11T15:31:34.693 DEBUG:boto:Token: None 2023-10-11T15:31:34.693 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.694 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d00b0422c6508101-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.697 DEBUG:boto:b'dghqvd-60obj111000falseobj112023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.698 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.698 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.699 DEBUG:boto:path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.699 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.700 DEBUG:boto:Method: GET 2023-10-11T15:31:34.700 DEBUG:boto:Path: /dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.701 DEBUG:boto:Data: 2023-10-11T15:31:34.701 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.702 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.702 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.703 DEBUG:boto:Params: {} 2023-10-11T15:31:34.703 DEBUG:boto:Token: None 2023-10-11T15:31:34.704 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.704 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dfbcd7f64f1f559-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.708 DEBUG:boto:b'dghqvd-60obj111000falseobj112023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.709 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.709 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.709 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.710 DEBUG:boto:Path: /dghqvd-60/obj11 2023-10-11T15:31:34.711 DEBUG:boto:Data: 2023-10-11T15:31:34.711 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.712 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.712 DEBUG:boto:Params: {} 2023-10-11T15:31:34.713 DEBUG:boto:Token: None 2023-10-11T15:31:34.713 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11 2023-10-11T15:31:34.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DWOMNUCnRQDx8aDqUGNoC+0hD/A= 2023-10-11T15:31:34.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DWOMNUCnRQDx8aDqUGNoC+0hD/A=', 'Content-Length': '0'} 2023-10-11T15:31:34.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfd83e95bc95fda-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.717 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.717 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.717 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.718 DEBUG:boto:Path: /dghqvd-60/obj11 2023-10-11T15:31:34.718 DEBUG:boto:Data: 2023-10-11T15:31:34.719 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.720 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.720 DEBUG:boto:Params: {} 2023-10-11T15:31:34.721 DEBUG:boto:Token: None 2023-10-11T15:31:34.721 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11 2023-10-11T15:31:34.722 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DWOMNUCnRQDx8aDqUGNoC+0hD/A= 2023-10-11T15:31:34.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DWOMNUCnRQDx8aDqUGNoC+0hD/A=', 'Content-Length': '0'} 2023-10-11T15:31:34.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f56789b4eb8de6e-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:31:34.725 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.725 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.726 DEBUG:boto:Method: GET 2023-10-11T15:31:34.726 DEBUG:boto:Path: /dghqvd-60/obj11 2023-10-11T15:31:34.727 DEBUG:boto:Data: 2023-10-11T15:31:34.727 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.728 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.729 DEBUG:boto:Params: {} 2023-10-11T15:31:34.729 DEBUG:boto:Token: None 2023-10-11T15:31:34.730 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11 2023-10-11T15:31:34.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czB3SELbMubXZWCwYfSFGYfF8+w= 2023-10-11T15:31:34.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czB3SELbMubXZWCwYfSFGYfF8+w=', 'Content-Length': '0'} 2023-10-11T15:31:34.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6a67555fa2fcfc-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.733 DEBUG:boto:path=/dghqvd-60/obj11 2023-10-11T15:31:34.734 DEBUG:boto:auth_path=/dghqvd-60/obj11 2023-10-11T15:31:34.734 DEBUG:boto:Method: GET 2023-10-11T15:31:34.735 DEBUG:boto:Path: /dghqvd-60/obj11 2023-10-11T15:31:34.735 DEBUG:boto:Data: 2023-10-11T15:31:34.735 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.736 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.736 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.737 DEBUG:boto:Params: {} 2023-10-11T15:31:34.737 DEBUG:boto:Token: None 2023-10-11T15:31:34.738 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj11 2023-10-11T15:31:34.738 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czB3SELbMubXZWCwYfSFGYfF8+w= 2023-10-11T15:31:34.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czB3SELbMubXZWCwYfSFGYfF8+w=', 'Content-Length': '0'} 2023-10-11T15:31:34.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17d4f7f8dea9050-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.786 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.787 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.787 DEBUG:boto:path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.788 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.788 DEBUG:boto:Method: GET 2023-10-11T15:31:34.789 DEBUG:boto:Path: /dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.789 DEBUG:boto:Data: 2023-10-11T15:31:34.790 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.791 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.791 DEBUG:boto:Params: {} 2023-10-11T15:31:34.792 DEBUG:boto:Token: None 2023-10-11T15:31:34.792 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c90430553fff182-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.796 DEBUG:boto:b'dghqvd-60obj111000falseobj112023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.797 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.798 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.798 DEBUG:boto:path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.799 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.799 DEBUG:boto:Method: GET 2023-10-11T15:31:34.800 DEBUG:boto:Path: /dghqvd-60/?prefix=obj11 2023-10-11T15:31:34.800 DEBUG:boto:Data: 2023-10-11T15:31:34.800 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.801 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.802 DEBUG:boto:Params: {} 2023-10-11T15:31:34.802 DEBUG:boto:Token: None 2023-10-11T15:31:34.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e0ea535cec71626-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.807 DEBUG:boto:b'dghqvd-60obj111000falseobj112023-10-11T15:29:07.902Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:31:34.808 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.808 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.809 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.809 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.810 DEBUG:boto:Method: GET 2023-10-11T15:31:34.810 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.811 DEBUG:boto:Data: 2023-10-11T15:31:34.811 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.811 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.812 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.812 DEBUG:boto:Params: {} 2023-10-11T15:31:34.813 DEBUG:boto:Token: None 2023-10-11T15:31:34.813 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA= 2023-10-11T15:31:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA=', 'Content-Length': '0'} 2023-10-11T15:31:34.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b67e0a833fd31ce4-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.817 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.817 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.818 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.818 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.819 DEBUG:boto:Method: GET 2023-10-11T15:31:34.819 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.820 DEBUG:boto:Data: 2023-10-11T15:31:34.820 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.820 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.821 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.821 DEBUG:boto:Params: {} 2023-10-11T15:31:34.822 DEBUG:boto:Token: None 2023-10-11T15:31:34.822 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA= 2023-10-11T15:31:34.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA=', 'Content-Length': '0'} 2023-10-11T15:31:34.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f278312b17eb0246-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.826 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.827 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.827 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.827 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.828 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.828 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.829 DEBUG:boto:Data: 2023-10-11T15:31:34.829 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.830 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.830 DEBUG:boto:Params: {} 2023-10-11T15:31:34.831 DEBUG:boto:Token: None 2023-10-11T15:31:34.831 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/hA4FX2Qc9CoPb7OMJYKXRHSIgg= 2023-10-11T15:31:34.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/hA4FX2Qc9CoPb7OMJYKXRHSIgg=', 'Content-Length': '0'} 2023-10-11T15:31:34.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468216e47352da78-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.835 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.835 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.835 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.836 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.836 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.836 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.837 DEBUG:boto:Data: 2023-10-11T15:31:34.837 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.838 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.838 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.839 DEBUG:boto:Params: {} 2023-10-11T15:31:34.839 DEBUG:boto:Token: None 2023-10-11T15:31:34.840 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.840 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/hA4FX2Qc9CoPb7OMJYKXRHSIgg= 2023-10-11T15:31:34.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/hA4FX2Qc9CoPb7OMJYKXRHSIgg=', 'Content-Length': '0'} 2023-10-11T15:31:34.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad832a324690d6b-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:31:34.843 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.844 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.844 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.844 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.845 DEBUG:boto:Method: GET 2023-10-11T15:31:34.845 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.846 DEBUG:boto:Data: 2023-10-11T15:31:34.846 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.847 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.847 DEBUG:boto:Params: {} 2023-10-11T15:31:34.848 DEBUG:boto:Token: None 2023-10-11T15:31:34.848 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA= 2023-10-11T15:31:34.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA=', 'Content-Length': '0'} 2023-10-11T15:31:34.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e71934523312d4e2-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.852 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.852 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.853 DEBUG:boto:path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.853 DEBUG:boto:auth_path=/dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.854 DEBUG:boto:Method: GET 2023-10-11T15:31:34.854 DEBUG:boto:Path: /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.855 DEBUG:boto:Data: 2023-10-11T15:31:34.855 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.856 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.856 DEBUG:boto:Params: {} 2023-10-11T15:31:34.857 DEBUG:boto:Token: None 2023-10-11T15:31:34.857 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22?versionId=null 2023-10-11T15:31:34.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA= 2023-10-11T15:31:34.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARw3fDqu4SiawfW3BRhXHEHedyA=', 'Content-Length': '0'} 2023-10-11T15:31:34.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a1d53fbceb308a-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.861 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.861 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.862 DEBUG:boto:path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.862 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.863 DEBUG:boto:Method: GET 2023-10-11T15:31:34.863 DEBUG:boto:Path: /dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.864 DEBUG:boto:Data: 2023-10-11T15:31:34.864 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.864 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.865 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.865 DEBUG:boto:Params: {} 2023-10-11T15:31:34.866 DEBUG:boto:Token: None 2023-10-11T15:31:34.866 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad93cf88d0185c4-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.870 DEBUG:boto:b'dghqvd-60obj221000falseobj222023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.871 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.872 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.872 DEBUG:boto:path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.873 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.873 DEBUG:boto:Method: GET 2023-10-11T15:31:34.873 DEBUG:boto:Path: /dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.874 DEBUG:boto:Data: 2023-10-11T15:31:34.874 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.875 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.876 DEBUG:boto:Params: {} 2023-10-11T15:31:34.876 DEBUG:boto:Token: None 2023-10-11T15:31:34.877 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b34bb4fd953af7-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.881 DEBUG:boto:b'dghqvd-60obj221000falseobj222023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.881 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.882 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.882 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.883 DEBUG:boto:Path: /dghqvd-60/obj22 2023-10-11T15:31:34.883 DEBUG:boto:Data: 2023-10-11T15:31:34.883 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.884 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.884 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.885 DEBUG:boto:Params: {} 2023-10-11T15:31:34.885 DEBUG:boto:Token: None 2023-10-11T15:31:34.885 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22 2023-10-11T15:31:34.886 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JUbM+6cjC7akj768pwYUmu+ZQh0= 2023-10-11T15:31:34.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JUbM+6cjC7akj768pwYUmu+ZQh0=', 'Content-Length': '0'} 2023-10-11T15:31:34.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f64fe426187ed8-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.889 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.890 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.890 DEBUG:boto:Method: HEAD 2023-10-11T15:31:34.890 DEBUG:boto:Path: /dghqvd-60/obj22 2023-10-11T15:31:34.891 DEBUG:boto:Data: 2023-10-11T15:31:34.891 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.892 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.892 DEBUG:boto:Params: {} 2023-10-11T15:31:34.893 DEBUG:boto:Token: None 2023-10-11T15:31:34.893 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22 2023-10-11T15:31:34.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JUbM+6cjC7akj768pwYUmu+ZQh0= 2023-10-11T15:31:34.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JUbM+6cjC7akj768pwYUmu+ZQh0=', 'Content-Length': '0'} 2023-10-11T15:31:34.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de44d81f7a9547b-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:31:34.897 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.898 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.898 DEBUG:boto:Method: GET 2023-10-11T15:31:34.899 DEBUG:boto:Path: /dghqvd-60/obj22 2023-10-11T15:31:34.899 DEBUG:boto:Data: 2023-10-11T15:31:34.899 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.900 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.900 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.901 DEBUG:boto:Params: {} 2023-10-11T15:31:34.901 DEBUG:boto:Token: None 2023-10-11T15:31:34.902 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22 2023-10-11T15:31:34.902 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ujFNqVRO50uh1v25Q5G3OIgwvGw= 2023-10-11T15:31:34.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ujFNqVRO50uh1v25Q5G3OIgwvGw=', 'Content-Length': '0'} 2023-10-11T15:31:34.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b764f3e4abff7880-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.905 DEBUG:boto:path=/dghqvd-60/obj22 2023-10-11T15:31:34.906 DEBUG:boto:auth_path=/dghqvd-60/obj22 2023-10-11T15:31:34.906 DEBUG:boto:Method: GET 2023-10-11T15:31:34.906 DEBUG:boto:Path: /dghqvd-60/obj22 2023-10-11T15:31:34.907 DEBUG:boto:Data: 2023-10-11T15:31:34.907 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.908 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.908 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.909 DEBUG:boto:Params: {} 2023-10-11T15:31:34.909 DEBUG:boto:Token: None 2023-10-11T15:31:34.910 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj22 2023-10-11T15:31:34.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ujFNqVRO50uh1v25Q5G3OIgwvGw= 2023-10-11T15:31:34.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ujFNqVRO50uh1v25Q5G3OIgwvGw=', 'Content-Length': '0'} 2023-10-11T15:31:34.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bba7ffc536888c3-006526bfd6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.913 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.914 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.915 DEBUG:boto:path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.915 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.915 DEBUG:boto:Method: GET 2023-10-11T15:31:34.916 DEBUG:boto:Path: /dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.916 DEBUG:boto:Data: 2023-10-11T15:31:34.917 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.918 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.918 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.919 DEBUG:boto:Params: {} 2023-10-11T15:31:34.919 DEBUG:boto:Token: None 2023-10-11T15:31:34.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973c5dfb1b8d0239-006526bfd6-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.924 DEBUG:boto:b'dghqvd-60obj221000falseobj222023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.925 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:34.926 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:34.926 DEBUG:boto:path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.926 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.927 DEBUG:boto:Method: GET 2023-10-11T15:31:34.927 DEBUG:boto:Path: /dghqvd-60/?prefix=obj22 2023-10-11T15:31:34.928 DEBUG:boto:Data: 2023-10-11T15:31:34.928 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.929 DEBUG:boto:Port: 8000 2023-10-11T15:31:34.930 DEBUG:boto:Params: {} 2023-10-11T15:31:34.930 DEBUG:boto:Token: None 2023-10-11T15:31:34.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/ 2023-10-11T15:31:34.932 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY= 2023-10-11T15:31:34.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5YNSE0EEErOpu1grn/sRjufcOnY=', 'Content-Length': '0'} 2023-10-11T15:31:34.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000597cb6d22f9bf2bc-006526bfd6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.936 DEBUG:boto:b'dghqvd-60obj221000falseobj222023-10-11T15:29:07.923Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:34.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:31:34.938 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:34.939 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:34.939 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.940 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.940 DEBUG:boto:Method: GET 2023-10-11T15:31:34.941 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.941 DEBUG:boto:Data: 2023-10-11T15:31:34.941 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.942 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:34.942 DEBUG:boto:Port: 8001 2023-10-11T15:31:34.943 DEBUG:boto:Params: {} 2023-10-11T15:31:34.943 DEBUG:boto:Token: None 2023-10-11T15:31:34.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:34 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.944 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RqeaXMZ16pgYR/kTnKX+yRtTWiA= 2023-10-11T15:31:34.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RqeaXMZ16pgYR/kTnKX+yRtTWiA=', 'Content-Length': '0'} 2023-10-11T15:31:34.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86b0e623751ac64-006526bfd6-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:34.995 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:34.995 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:34.996 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.996 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.997 DEBUG:boto:Method: GET 2023-10-11T15:31:34.997 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:34.998 DEBUG:boto:Data: 2023-10-11T15:31:34.998 DEBUG:boto:Headers: {} 2023-10-11T15:31:34.999 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:34.999 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.000 DEBUG:boto:Params: {} 2023-10-11T15:31:35.000 DEBUG:boto:Token: None 2023-10-11T15:31:35.001 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA= 2023-10-11T15:31:35.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA=', 'Content-Length': '0'} 2023-10-11T15:31:35.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de47aec0ef28bb47-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.004 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.005 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.005 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.006 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.006 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.007 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.007 DEBUG:boto:Data: 2023-10-11T15:31:35.008 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.009 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.010 DEBUG:boto:Params: {} 2023-10-11T15:31:35.010 DEBUG:boto:Token: None 2023-10-11T15:31:35.011 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RVxOYdpxRDY3IiO3V/N07uWAStU= 2023-10-11T15:31:35.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RVxOYdpxRDY3IiO3V/N07uWAStU=', 'Content-Length': '0'} 2023-10-11T15:31:35.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872276e6b2d0a848-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.014 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.014 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.015 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.015 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.016 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.016 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.016 DEBUG:boto:Data: 2023-10-11T15:31:35.017 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.018 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.018 DEBUG:boto:Params: {} 2023-10-11T15:31:35.019 DEBUG:boto:Token: None 2023-10-11T15:31:35.019 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RVxOYdpxRDY3IiO3V/N07uWAStU= 2023-10-11T15:31:35.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RVxOYdpxRDY3IiO3V/N07uWAStU=', 'Content-Length': '0'} 2023-10-11T15:31:35.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9138c8ea096eb6e-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:31:35.023 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.023 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.024 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.024 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.025 DEBUG:boto:Method: GET 2023-10-11T15:31:35.025 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.025 DEBUG:boto:Data: 2023-10-11T15:31:35.026 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.027 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.027 DEBUG:boto:Params: {} 2023-10-11T15:31:35.028 DEBUG:boto:Token: None 2023-10-11T15:31:35.028 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA= 2023-10-11T15:31:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA=', 'Content-Length': '0'} 2023-10-11T15:31:35.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7cf3d816e9ca2c-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.032 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.032 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.033 DEBUG:boto:path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.033 DEBUG:boto:auth_path=/dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.034 DEBUG:boto:Method: GET 2023-10-11T15:31:35.034 DEBUG:boto:Path: /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.034 DEBUG:boto:Data: 2023-10-11T15:31:35.035 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.035 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.036 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.036 DEBUG:boto:Params: {} 2023-10-11T15:31:35.037 DEBUG:boto:Token: None 2023-10-11T15:31:35.037 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33?versionId=null 2023-10-11T15:31:35.038 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA= 2023-10-11T15:31:35.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TraAOYCMNX1GlOKiXG89gol25RA=', 'Content-Length': '0'} 2023-10-11T15:31:35.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009abaeaa1601ae170-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.041 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.041 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.042 DEBUG:boto:path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.042 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.042 DEBUG:boto:Method: GET 2023-10-11T15:31:35.043 DEBUG:boto:Path: /dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.043 DEBUG:boto:Data: 2023-10-11T15:31:35.044 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.045 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.045 DEBUG:boto:Params: {} 2023-10-11T15:31:35.045 DEBUG:boto:Token: None 2023-10-11T15:31:35.046 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e655b89472e856-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.050 DEBUG:boto:b'dghqvd-60obj331000falseobj332023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.050 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.051 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.051 DEBUG:boto:path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.052 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.052 DEBUG:boto:Method: GET 2023-10-11T15:31:35.053 DEBUG:boto:Path: /dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.053 DEBUG:boto:Data: 2023-10-11T15:31:35.053 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.054 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.054 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.055 DEBUG:boto:Params: {} 2023-10-11T15:31:35.055 DEBUG:boto:Token: None 2023-10-11T15:31:35.056 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.056 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a9bbc33626cc32-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.060 DEBUG:boto:b'dghqvd-60obj331000falseobj332023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.060 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.061 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.061 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.061 DEBUG:boto:Path: /dghqvd-60/obj33 2023-10-11T15:31:35.062 DEBUG:boto:Data: 2023-10-11T15:31:35.062 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.063 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.063 DEBUG:boto:Params: {} 2023-10-11T15:31:35.064 DEBUG:boto:Token: None 2023-10-11T15:31:35.064 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33 2023-10-11T15:31:35.065 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wO04gpqTiME0EjJzJ5/LuPZW3p0= 2023-10-11T15:31:35.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wO04gpqTiME0EjJzJ5/LuPZW3p0=', 'Content-Length': '0'} 2023-10-11T15:31:35.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653837ea86ded95a-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.067 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.068 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.068 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.069 DEBUG:boto:Path: /dghqvd-60/obj33 2023-10-11T15:31:35.069 DEBUG:boto:Data: 2023-10-11T15:31:35.070 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.070 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.071 DEBUG:boto:Params: {} 2023-10-11T15:31:35.071 DEBUG:boto:Token: None 2023-10-11T15:31:35.072 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33 2023-10-11T15:31:35.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wO04gpqTiME0EjJzJ5/LuPZW3p0= 2023-10-11T15:31:35.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wO04gpqTiME0EjJzJ5/LuPZW3p0=', 'Content-Length': '0'} 2023-10-11T15:31:35.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008020037aeb0affdf-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:31:35.075 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.076 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.076 DEBUG:boto:Method: GET 2023-10-11T15:31:35.076 DEBUG:boto:Path: /dghqvd-60/obj33 2023-10-11T15:31:35.077 DEBUG:boto:Data: 2023-10-11T15:31:35.077 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.078 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.079 DEBUG:boto:Params: {} 2023-10-11T15:31:35.079 DEBUG:boto:Token: None 2023-10-11T15:31:35.080 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33 2023-10-11T15:31:35.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HK9UR/Z8TQSySa3lhlv2rpT9On0= 2023-10-11T15:31:35.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HK9UR/Z8TQSySa3lhlv2rpT9On0=', 'Content-Length': '0'} 2023-10-11T15:31:35.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfca06ab91d367c2-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.131 DEBUG:boto:path=/dghqvd-60/obj33 2023-10-11T15:31:35.131 DEBUG:boto:auth_path=/dghqvd-60/obj33 2023-10-11T15:31:35.132 DEBUG:boto:Method: GET 2023-10-11T15:31:35.132 DEBUG:boto:Path: /dghqvd-60/obj33 2023-10-11T15:31:35.133 DEBUG:boto:Data: 2023-10-11T15:31:35.133 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.134 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.134 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.135 DEBUG:boto:Params: {} 2023-10-11T15:31:35.135 DEBUG:boto:Token: None 2023-10-11T15:31:35.136 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj33 2023-10-11T15:31:35.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HK9UR/Z8TQSySa3lhlv2rpT9On0= 2023-10-11T15:31:35.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HK9UR/Z8TQSySa3lhlv2rpT9On0=', 'Content-Length': '0'} 2023-10-11T15:31:35.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b18c29340ba790d-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.140 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.140 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.141 DEBUG:boto:path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.141 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.142 DEBUG:boto:Method: GET 2023-10-11T15:31:35.142 DEBUG:boto:Path: /dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.143 DEBUG:boto:Data: 2023-10-11T15:31:35.144 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.144 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.145 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.145 DEBUG:boto:Params: {} 2023-10-11T15:31:35.146 DEBUG:boto:Token: None 2023-10-11T15:31:35.146 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2838253e5b76c94-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.150 DEBUG:boto:b'dghqvd-60obj331000falseobj332023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.151 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.152 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.152 DEBUG:boto:path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.153 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.153 DEBUG:boto:Method: GET 2023-10-11T15:31:35.154 DEBUG:boto:Path: /dghqvd-60/?prefix=obj33 2023-10-11T15:31:35.154 DEBUG:boto:Data: 2023-10-11T15:31:35.155 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.156 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.157 DEBUG:boto:Params: {} 2023-10-11T15:31:35.157 DEBUG:boto:Token: None 2023-10-11T15:31:35.158 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9f573a2db2887f-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.162 DEBUG:boto:b'dghqvd-60obj331000falseobj332023-10-11T15:29:07.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:31:35.163 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.163 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.164 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.164 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.165 DEBUG:boto:Method: GET 2023-10-11T15:31:35.165 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.165 DEBUG:boto:Data: 2023-10-11T15:31:35.166 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.166 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.167 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.167 DEBUG:boto:Params: {} 2023-10-11T15:31:35.168 DEBUG:boto:Token: None 2023-10-11T15:31:35.168 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk= 2023-10-11T15:31:35.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk=', 'Content-Length': '0'} 2023-10-11T15:31:35.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9084349d6d56e70-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.172 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.173 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.173 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.174 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.174 DEBUG:boto:Method: GET 2023-10-11T15:31:35.175 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.175 DEBUG:boto:Data: 2023-10-11T15:31:35.176 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.177 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.177 DEBUG:boto:Params: {} 2023-10-11T15:31:35.177 DEBUG:boto:Token: None 2023-10-11T15:31:35.178 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk= 2023-10-11T15:31:35.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk=', 'Content-Length': '0'} 2023-10-11T15:31:35.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f5e2f84e3a2106-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.182 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.183 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.186 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.186 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.187 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.187 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.188 DEBUG:boto:Data: 2023-10-11T15:31:35.188 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.188 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.189 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.189 DEBUG:boto:Params: {} 2023-10-11T15:31:35.190 DEBUG:boto:Token: None 2023-10-11T15:31:35.190 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LfFMb+bE5Dg2YU7kO9T6Imk6m84= 2023-10-11T15:31:35.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LfFMb+bE5Dg2YU7kO9T6Imk6m84=', 'Content-Length': '0'} 2023-10-11T15:31:35.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b440471b02b395-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.194 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.194 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.195 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.195 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.196 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.196 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.196 DEBUG:boto:Data: 2023-10-11T15:31:35.197 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.197 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.198 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.198 DEBUG:boto:Params: {} 2023-10-11T15:31:35.198 DEBUG:boto:Token: None 2023-10-11T15:31:35.199 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.199 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LfFMb+bE5Dg2YU7kO9T6Imk6m84= 2023-10-11T15:31:35.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LfFMb+bE5Dg2YU7kO9T6Imk6m84=', 'Content-Length': '0'} 2023-10-11T15:31:35.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e79b458ce77d0176-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:31:35.203 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.203 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.204 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.204 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.204 DEBUG:boto:Method: GET 2023-10-11T15:31:35.205 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.205 DEBUG:boto:Data: 2023-10-11T15:31:35.206 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.206 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.207 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.207 DEBUG:boto:Params: {} 2023-10-11T15:31:35.208 DEBUG:boto:Token: None 2023-10-11T15:31:35.208 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.209 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk= 2023-10-11T15:31:35.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk=', 'Content-Length': '0'} 2023-10-11T15:31:35.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087f5d9469b4238e1-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.212 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.212 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.212 DEBUG:boto:path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.213 DEBUG:boto:auth_path=/dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.213 DEBUG:boto:Method: GET 2023-10-11T15:31:35.214 DEBUG:boto:Path: /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.214 DEBUG:boto:Data: 2023-10-11T15:31:35.215 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.215 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.216 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.216 DEBUG:boto:Params: {} 2023-10-11T15:31:35.217 DEBUG:boto:Token: None 2023-10-11T15:31:35.217 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44?versionId=null 2023-10-11T15:31:35.218 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk= 2023-10-11T15:31:35.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NttebNfV7VT+Uub2vOcwCmd0EMk=', 'Content-Length': '0'} 2023-10-11T15:31:35.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfee5f2e759f2100-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.221 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.221 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.222 DEBUG:boto:path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.222 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.223 DEBUG:boto:Method: GET 2023-10-11T15:31:35.223 DEBUG:boto:Path: /dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.223 DEBUG:boto:Data: 2023-10-11T15:31:35.224 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.224 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.225 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.225 DEBUG:boto:Params: {} 2023-10-11T15:31:35.226 DEBUG:boto:Token: None 2023-10-11T15:31:35.226 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.227 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008951cb8c0ccfd2dd-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.270 DEBUG:boto:b'dghqvd-60obj441000falseobj442023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.271 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.272 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.273 DEBUG:boto:path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.273 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.274 DEBUG:boto:Method: GET 2023-10-11T15:31:35.274 DEBUG:boto:Path: /dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.275 DEBUG:boto:Data: 2023-10-11T15:31:35.276 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.277 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.278 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.279 DEBUG:boto:Params: {} 2023-10-11T15:31:35.280 DEBUG:boto:Token: None 2023-10-11T15:31:35.280 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.281 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003671d05d7a86ceab-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.285 DEBUG:boto:b'dghqvd-60obj441000falseobj442023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.286 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.287 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.287 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.288 DEBUG:boto:Path: /dghqvd-60/obj44 2023-10-11T15:31:35.288 DEBUG:boto:Data: 2023-10-11T15:31:35.288 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.289 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.289 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.290 DEBUG:boto:Params: {} 2023-10-11T15:31:35.290 DEBUG:boto:Token: None 2023-10-11T15:31:35.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44 2023-10-11T15:31:35.291 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ya3JdsymmSKZvoujfLgTPsBnQGU= 2023-10-11T15:31:35.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ya3JdsymmSKZvoujfLgTPsBnQGU=', 'Content-Length': '0'} 2023-10-11T15:31:35.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c27c6f7de6b239-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.295 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.295 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.295 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.296 DEBUG:boto:Path: /dghqvd-60/obj44 2023-10-11T15:31:35.296 DEBUG:boto:Data: 2023-10-11T15:31:35.297 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.298 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.298 DEBUG:boto:Params: {} 2023-10-11T15:31:35.298 DEBUG:boto:Token: None 2023-10-11T15:31:35.299 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44 2023-10-11T15:31:35.299 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ya3JdsymmSKZvoujfLgTPsBnQGU= 2023-10-11T15:31:35.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ya3JdsymmSKZvoujfLgTPsBnQGU=', 'Content-Length': '0'} 2023-10-11T15:31:35.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f5bb8396003f9a-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:31:35.303 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.304 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.304 DEBUG:boto:Method: GET 2023-10-11T15:31:35.305 DEBUG:boto:Path: /dghqvd-60/obj44 2023-10-11T15:31:35.306 DEBUG:boto:Data: 2023-10-11T15:31:35.306 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.307 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.307 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.308 DEBUG:boto:Params: {} 2023-10-11T15:31:35.308 DEBUG:boto:Token: None 2023-10-11T15:31:35.309 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44 2023-10-11T15:31:35.309 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UjjGm8bOPtYoKcnc5iaI1dD507k= 2023-10-11T15:31:35.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UjjGm8bOPtYoKcnc5iaI1dD507k=', 'Content-Length': '0'} 2023-10-11T15:31:35.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007463666a8225d9e9-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.313 DEBUG:boto:path=/dghqvd-60/obj44 2023-10-11T15:31:35.314 DEBUG:boto:auth_path=/dghqvd-60/obj44 2023-10-11T15:31:35.314 DEBUG:boto:Method: GET 2023-10-11T15:31:35.315 DEBUG:boto:Path: /dghqvd-60/obj44 2023-10-11T15:31:35.315 DEBUG:boto:Data: 2023-10-11T15:31:35.316 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.316 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.316 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.317 DEBUG:boto:Params: {} 2023-10-11T15:31:35.318 DEBUG:boto:Token: None 2023-10-11T15:31:35.318 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj44 2023-10-11T15:31:35.319 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UjjGm8bOPtYoKcnc5iaI1dD507k= 2023-10-11T15:31:35.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UjjGm8bOPtYoKcnc5iaI1dD507k=', 'Content-Length': '0'} 2023-10-11T15:31:35.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007553035564b577cd-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.321 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.322 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.322 DEBUG:boto:path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.323 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.323 DEBUG:boto:Method: GET 2023-10-11T15:31:35.324 DEBUG:boto:Path: /dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.324 DEBUG:boto:Data: 2023-10-11T15:31:35.325 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.325 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.325 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.326 DEBUG:boto:Params: {} 2023-10-11T15:31:35.326 DEBUG:boto:Token: None 2023-10-11T15:31:35.327 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.327 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043e11e87d1b6cb5-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.331 DEBUG:boto:b'dghqvd-60obj441000falseobj442023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.331 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.332 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.332 DEBUG:boto:path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.333 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.333 DEBUG:boto:Method: GET 2023-10-11T15:31:35.334 DEBUG:boto:Path: /dghqvd-60/?prefix=obj44 2023-10-11T15:31:35.334 DEBUG:boto:Data: 2023-10-11T15:31:35.335 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.335 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.336 DEBUG:boto:Params: {} 2023-10-11T15:31:35.336 DEBUG:boto:Token: None 2023-10-11T15:31:35.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1462dd327befb04-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.341 DEBUG:boto:b'dghqvd-60obj441000falseobj442023-10-11T15:29:08.002Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:31:35.342 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.342 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.343 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.343 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.344 DEBUG:boto:Method: GET 2023-10-11T15:31:35.344 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.344 DEBUG:boto:Data: 2023-10-11T15:31:35.345 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.346 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.346 DEBUG:boto:Params: {} 2023-10-11T15:31:35.347 DEBUG:boto:Token: None 2023-10-11T15:31:35.347 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.348 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE= 2023-10-11T15:31:35.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE=', 'Content-Length': '0'} 2023-10-11T15:31:35.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003341e6a28a128a58-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.351 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.351 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.352 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.352 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.353 DEBUG:boto:Method: GET 2023-10-11T15:31:35.353 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.354 DEBUG:boto:Data: 2023-10-11T15:31:35.354 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.355 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.355 DEBUG:boto:Params: {} 2023-10-11T15:31:35.356 DEBUG:boto:Token: None 2023-10-11T15:31:35.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE= 2023-10-11T15:31:35.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE=', 'Content-Length': '0'} 2023-10-11T15:31:35.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052579a3e2141aafa-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.360 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.360 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.360 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.361 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.361 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.362 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.362 DEBUG:boto:Data: 2023-10-11T15:31:35.363 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.363 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.364 DEBUG:boto:Params: {} 2023-10-11T15:31:35.364 DEBUG:boto:Token: None 2023-10-11T15:31:35.365 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G1sWTGqWQbMykQ2oHbQbUrvq97E= 2023-10-11T15:31:35.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G1sWTGqWQbMykQ2oHbQbUrvq97E=', 'Content-Length': '0'} 2023-10-11T15:31:35.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20ca84bd1cb0253-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.368 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.368 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.369 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.369 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.370 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.370 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.371 DEBUG:boto:Data: 2023-10-11T15:31:35.371 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.372 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.372 DEBUG:boto:Params: {} 2023-10-11T15:31:35.373 DEBUG:boto:Token: None 2023-10-11T15:31:35.373 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.374 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G1sWTGqWQbMykQ2oHbQbUrvq97E= 2023-10-11T15:31:35.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G1sWTGqWQbMykQ2oHbQbUrvq97E=', 'Content-Length': '0'} 2023-10-11T15:31:35.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c973a2bce7b720c2-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:31:35.377 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.378 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.378 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.379 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.379 DEBUG:boto:Method: GET 2023-10-11T15:31:35.380 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.380 DEBUG:boto:Data: 2023-10-11T15:31:35.380 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.381 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.381 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.382 DEBUG:boto:Params: {} 2023-10-11T15:31:35.382 DEBUG:boto:Token: None 2023-10-11T15:31:35.383 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.383 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE= 2023-10-11T15:31:35.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE=', 'Content-Length': '0'} 2023-10-11T15:31:35.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419a715dfea9290f-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.430 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.431 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.432 DEBUG:boto:path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.432 DEBUG:boto:auth_path=/dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.432 DEBUG:boto:Method: GET 2023-10-11T15:31:35.433 DEBUG:boto:Path: /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.433 DEBUG:boto:Data: 2023-10-11T15:31:35.434 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.435 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.435 DEBUG:boto:Params: {} 2023-10-11T15:31:35.435 DEBUG:boto:Token: None 2023-10-11T15:31:35.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55?versionId=null 2023-10-11T15:31:35.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE= 2023-10-11T15:31:35.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wb4KnxMre87gR6nDSGHM8z9eDpE=', 'Content-Length': '0'} 2023-10-11T15:31:35.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bd787df36da558-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.440 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.440 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.441 DEBUG:boto:path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.441 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.441 DEBUG:boto:Method: GET 2023-10-11T15:31:35.442 DEBUG:boto:Path: /dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.442 DEBUG:boto:Data: 2023-10-11T15:31:35.443 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.444 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.444 DEBUG:boto:Params: {} 2023-10-11T15:31:35.444 DEBUG:boto:Token: None 2023-10-11T15:31:35.445 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4f9fd8e1d0900e-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.449 DEBUG:boto:b'dghqvd-60obj551000falseobj552023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.450 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.450 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.451 DEBUG:boto:path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.451 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.452 DEBUG:boto:Method: GET 2023-10-11T15:31:35.452 DEBUG:boto:Path: /dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.452 DEBUG:boto:Data: 2023-10-11T15:31:35.453 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.453 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.454 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.454 DEBUG:boto:Params: {} 2023-10-11T15:31:35.455 DEBUG:boto:Token: None 2023-10-11T15:31:35.455 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.456 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc91e5b66aefc85-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.459 DEBUG:boto:b'dghqvd-60obj551000falseobj552023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.460 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.460 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.461 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.461 DEBUG:boto:Path: /dghqvd-60/obj55 2023-10-11T15:31:35.462 DEBUG:boto:Data: 2023-10-11T15:31:35.462 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.463 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.463 DEBUG:boto:Params: {} 2023-10-11T15:31:35.464 DEBUG:boto:Token: None 2023-10-11T15:31:35.464 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55 2023-10-11T15:31:35.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HPQB8+2tyZeYcmqK0e2Wphqw5lk= 2023-10-11T15:31:35.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HPQB8+2tyZeYcmqK0e2Wphqw5lk=', 'Content-Length': '0'} 2023-10-11T15:31:35.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99b3c3996b3a0b1-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.468 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.468 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.468 DEBUG:boto:Method: HEAD 2023-10-11T15:31:35.469 DEBUG:boto:Path: /dghqvd-60/obj55 2023-10-11T15:31:35.469 DEBUG:boto:Data: 2023-10-11T15:31:35.470 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.470 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.470 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.471 DEBUG:boto:Params: {} 2023-10-11T15:31:35.471 DEBUG:boto:Token: None 2023-10-11T15:31:35.472 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55 2023-10-11T15:31:35.472 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HPQB8+2tyZeYcmqK0e2Wphqw5lk= 2023-10-11T15:31:35.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HPQB8+2tyZeYcmqK0e2Wphqw5lk=', 'Content-Length': '0'} 2023-10-11T15:31:35.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b3ed1604589678-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:31:35.476 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.476 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.476 DEBUG:boto:Method: GET 2023-10-11T15:31:35.477 DEBUG:boto:Path: /dghqvd-60/obj55 2023-10-11T15:31:35.477 DEBUG:boto:Data: 2023-10-11T15:31:35.478 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.479 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.479 DEBUG:boto:Params: {} 2023-10-11T15:31:35.480 DEBUG:boto:Token: None 2023-10-11T15:31:35.480 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55 2023-10-11T15:31:35.481 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JBJCHEziHylmNfiAVEJ/memCgl4= 2023-10-11T15:31:35.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JBJCHEziHylmNfiAVEJ/memCgl4=', 'Content-Length': '0'} 2023-10-11T15:31:35.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-60:daa316e9-869c-4074-a756-a8adfb4a3573.5751.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e868c432cac7d91-006526bfd7-1edc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.484 DEBUG:boto:path=/dghqvd-60/obj55 2023-10-11T15:31:35.484 DEBUG:boto:auth_path=/dghqvd-60/obj55 2023-10-11T15:31:35.485 DEBUG:boto:Method: GET 2023-10-11T15:31:35.485 DEBUG:boto:Path: /dghqvd-60/obj55 2023-10-11T15:31:35.486 DEBUG:boto:Data: 2023-10-11T15:31:35.486 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.487 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.487 DEBUG:boto:Params: {} 2023-10-11T15:31:35.488 DEBUG:boto:Token: None 2023-10-11T15:31:35.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/obj55 2023-10-11T15:31:35.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JBJCHEziHylmNfiAVEJ/memCgl4= 2023-10-11T15:31:35.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JBJCHEziHylmNfiAVEJ/memCgl4=', 'Content-Length': '0'} 2023-10-11T15:31:35.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:29:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56cb6f5730f3a56-006526bfd7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.534 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.535 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.535 DEBUG:boto:path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.536 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.536 DEBUG:boto:Method: GET 2023-10-11T15:31:35.537 DEBUG:boto:Path: /dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.537 DEBUG:boto:Data: 2023-10-11T15:31:35.538 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.538 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:35.538 DEBUG:boto:Port: 8001 2023-10-11T15:31:35.539 DEBUG:boto:Params: {} 2023-10-11T15:31:35.539 DEBUG:boto:Token: None 2023-10-11T15:31:35.540 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd8043ea29d814fa-006526bfd7-1edc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.544 DEBUG:boto:b'dghqvd-60obj551000falseobj552023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.545 DEBUG:boto:path=/dghqvd-60/ 2023-10-11T15:31:35.545 DEBUG:boto:auth_path=/dghqvd-60/ 2023-10-11T15:31:35.546 DEBUG:boto:path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.546 DEBUG:boto:auth_path=/dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.547 DEBUG:boto:Method: GET 2023-10-11T15:31:35.547 DEBUG:boto:Path: /dghqvd-60/?prefix=obj55 2023-10-11T15:31:35.548 DEBUG:boto:Data: 2023-10-11T15:31:35.548 DEBUG:boto:Headers: {} 2023-10-11T15:31:35.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:35.549 DEBUG:boto:Port: 8000 2023-10-11T15:31:35.549 DEBUG:boto:Params: {} 2023-10-11T15:31:35.550 DEBUG:boto:Token: None 2023-10-11T15:31:35.550 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:31:35 GMT /dghqvd-60/ 2023-10-11T15:31:35.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E= 2023-10-11T15:31:35.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:35 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rEmBOjBuYowUhrp6VpDqqsczb2E=', 'Content-Length': '0'} 2023-10-11T15:31:35.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6942682168b76a3-006526bfd7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:31:35 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:35.554 DEBUG:boto:b'dghqvd-60obj551000falseobj552023-10-11T15:29:08.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:31:35.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-60 zones={a2, a1} 2023-10-11T15:31:35.556 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T15:31:35.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.557 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:35.608 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:35.608 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:35.657 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:31:35.657 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.658 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000025975", 2023-10-11T15:31:35.658 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:23.113227Z" 2023-10-11T15:31:35.658 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.658 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.659 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000019038", 2023-10-11T15:31:35.659 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:25:38.162896Z" 2023-10-11T15:31:35.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.659 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.659 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000030738", 2023-10-11T15:31:35.660 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:08.005385Z" 2023-10-11T15:31:35.660 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.660 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.660 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000026636", 2023-10-11T15:31:35.660 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:23.115000Z" 2023-10-11T15:31:35.661 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.661 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:31:35.662 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000025975', 1: '00000000000000000000:00000000000000019038', 2: '00000000000000000000:00000000000000030738', 3: '00000000000000000000:00000000000000026636'} 2023-10-11T15:31:35.663 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:31:35.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.664 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:35.716 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:35.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:35.760 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:31:35.760 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:31:35.761 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:31:35.761 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:31:35.761 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:31:35.761 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 15524158432587076073 2023-10-11T15:31:35.761 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.762 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:31:35.762 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.762 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:31:35.762 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.763 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.763 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000025975", 2023-10-11T15:31:35.763 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.764 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.764 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:23.113227Z" 2023-10-11T15:31:35.764 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.765 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:31:35.765 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.765 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000019038", 2023-10-11T15:31:35.766 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.766 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:25:38.162896Z" 2023-10-11T15:31:35.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:31:35.767 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.768 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000030738", 2023-10-11T15:31:35.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 51, 2023-10-11T15:31:35.768 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:08.005385Z" 2023-10-11T15:31:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:31:35.770 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.770 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000026636", 2023-10-11T15:31:35.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:31:35.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:23.115000Z" 2023-10-11T15:31:35.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.772 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:31:35.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.772 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:31:35.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 60, 2023-10-11T15:31:35.772 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 60 2023-10-11T15:31:35.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.773 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:31:35Z" 2023-10-11T15:31:35.773 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:31:35.774 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15524158432587076073 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025975", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:29:23.113227Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019038", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:25:38.162896Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030738", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-10-11T15:29:08.005385Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026636", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:29:23.115000Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-10-11T15:31:35Z" } 2023-10-11T15:31:35.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025975', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:29:23.113227Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019038', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:25:38.162896Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030738', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-10-11T15:29:08.005385Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026636', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:29:23.115000Z'}}] 2023-10-11T15:31:35.775 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000025975', 1: '00000000000000000000:00000000000000019038', 2: '00000000000000000000:00000000000000030738', 3: '00000000000000000000:00000000000000026636'} 2023-10-11T15:31:35.776 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000025975', 1: '00000000000000000000:00000000000000019038', 2: '00000000000000000000:00000000000000030738', 3: '00000000000000000000:00000000000000026636'} 2023-10-11T15:31:35.776 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:31:35.777 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T15:31:35.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:35.826 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:35.827 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:35.874 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:31:35.875 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.875 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000010545", 2023-10-11T15:31:35.875 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:52.296442Z" 2023-10-11T15:31:35.876 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.876 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.876 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005163", 2023-10-11T15:31:35.876 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:33.239861Z" 2023-10-11T15:31:35.876 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.877 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.877 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-10-11T15:31:35.877 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:11.561342Z" 2023-10-11T15:31:35.877 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.878 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.878 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000013053", 2023-10-11T15:31:35.878 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:29:30.294809Z" 2023-10-11T15:31:35.878 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.879 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:31:35.880 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000010545', 1: '00000000000000000000:00000000000000005163', 2: '00000000000000000000:00000000000000015024', 3: '00000000000000000000:00000000000000013053'} 2023-10-11T15:31:35.880 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:31:35.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:35.881 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:35.932 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:35.933 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:35.976 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:31:35.976 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:31:35.976 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:31:35.977 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:31:35.977 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:31:35.977 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T15:31:35.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.978 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:31:35.978 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.978 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:31:35.978 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.979 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.979 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000010545", 2023-10-11T15:31:35.979 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.979 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.980 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.980 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:52.296442Z" 2023-10-11T15:31:35.980 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.981 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.981 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.981 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:31:35.982 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.982 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.982 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005163", 2023-10-11T15:31:35.982 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.982 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.983 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.983 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:33.239861Z" 2023-10-11T15:31:35.983 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.983 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.984 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:31:35.984 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.984 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.984 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-10-11T15:31:35.984 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.985 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.985 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.985 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:11.561342Z" 2023-10-11T15:31:35.986 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.986 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.986 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:35.986 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:31:35.986 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:35.987 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:31:35.987 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000013053", 2023-10-11T15:31:35.987 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:35.987 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:35.988 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:35.988 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:29:30.294809Z" 2023-10-11T15:31:35.988 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.989 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:35.989 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:31:35.989 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.989 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:31:35.990 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T15:31:35.990 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T15:31:35.990 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:35.991 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:31:35Z" 2023-10-11T15:31:35.991 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:31:35.993 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010545", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:29:52.296442Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005163", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:33.239861Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:29:11.561342Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013053", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:29:30.294809Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T15:31:35Z" } 2023-10-11T15:31:35.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010545', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:29:52.296442Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005163', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:33.239861Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:29:11.561342Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013053', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:29:30.294809Z'}}] 2023-10-11T15:31:35.994 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010545', 1: '00000000000000000000:00000000000000005163', 2: '00000000000000000000:00000000000000015024', 3: '00000000000000000000:00000000000000013053'} 2023-10-11T15:31:35.995 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010545', 1: '00000000000000000000:00000000000000005163', 2: '00000000000000000000:00000000000000015024', 3: '00000000000000000000:00000000000000013053'} 2023-10-11T15:31:35.995 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:31:35.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-10-11T15:31:35.998 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:35.999 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:35.999 DEBUG:boto:Method: PUT 2023-10-11T15:31:36.000 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:36.000 DEBUG:boto:Data: 2023-10-11T15:31:36.001 DEBUG:boto:Headers: {} 2023-10-11T15:31:36.001 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:36.002 DEBUG:boto:Port: 8000 2023-10-11T15:31:36.003 DEBUG:boto:Params: {} 2023-10-11T15:31:36.003 DEBUG:boto:Token: None 2023-10-11T15:31:36.003 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:31:36 GMT /dghqvd-61/ 2023-10-11T15:31:36.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vrS2jbIbCjUzgkSSbQKNTsT8MmA= 2023-10-11T15:31:36.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:36 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vrS2jbIbCjUzgkSSbQKNTsT8MmA=', 'Content-Length': '0'} 2023-10-11T15:31:36.019 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000043a97e7b4ab8cc8d-006526bfd8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:36 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:36.020 DEBUG:rgw_multi.tests:created bucket=dghqvd-61 2023-10-11T15:31:36.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:36.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:36.022 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:36.072 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:36.073 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:36.102 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:31:36.118 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:31:36.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:31:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:31:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.119 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:31:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:31:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:31:36.120 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:31:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038296.017041_503.1", 2023-10-11T15:31:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:36.017041Z" 2023-10-11T15:31:36.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.122 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:31:36.123 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038296.017041_503.1'} 2023-10-11T15:31:36.123 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:31:36.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:36.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:31:36.125 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:31:36.175 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:36.176 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:36.205 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:36.204+0000 7fa77f710780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:31:36.206 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:36.205+0000 7fa77f710780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:31:36.219 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:31:36.219 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:31:36.219 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:31:36.220 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:31:36.220 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:31:36.220 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:31:36.220 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:31:36.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.221 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:31:36.221 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.222 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:31:36.222 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:36.222 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:31:36.223 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:31:36.223 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:36.223 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:31:36.223 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:36.224 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:31:36.224 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:31:36.224 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.224 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.225 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.225 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:31:36.225 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:36.225 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:31:36.225 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:31:36.226 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:36.226 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:36.226 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:36.226 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:31:36.227 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:31:36.227 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.227 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.228 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:31:36.228 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:36.228 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:31:36.229 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:31:36.229 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:36.229 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:36.229 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:36.230 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:31:36.230 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:31:36.230 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.230 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.230 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:31:36.231 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:31:36.231 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:31:36.231 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:31:36.232 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038296.017041_503.1", 2023-10-11T15:31:36.232 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:31:36.232 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:31:36.232 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:31:36.233 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:36.017041Z", 2023-10-11T15:31:36.233 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:31:36.233 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.233 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:31:36.234 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:31:36.234 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.234 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:31:36.234 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:31:36.234 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:31:36.235 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:31:36.235 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:31:36Z" 2023-10-11T15:31:36.235 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:31:36.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038296.017041_503.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:36.017041Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:31:36Z" } 2023-10-11T15:31:36.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038296.017041_503.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:36.017041Z', 'realm_epoch': 2}}] 2023-10-11T15:31:36.237 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038296.017041_503.1'} 2023-10-11T15:31:36.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038296.017041_503.1'} 2023-10-11T15:31:36.238 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:31:36.239 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:31:36.239 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:31:36.240 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:31:36.239+0000 7fe8c668b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 173400) UID: 0 2023-10-11T15:31:36.241 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:31:36.239+0000 7fe8ca8f47c0 -1 shutting down 2023-10-11T15:31:40.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:31:42.341 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:31:42.341 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.342 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.342 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.343 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.343 DEBUG:boto:Data: 2023-10-11T15:31:42.343 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.344 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.344 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.344 DEBUG:boto:Params: {} 2023-10-11T15:31:42.345 DEBUG:boto:Token: None 2023-10-11T15:31:42.345 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.346 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000060c47fa886245c3-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.353 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:31:42.354 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:31:42.354 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.355 DEBUG:boto:Path: /dghqvd-61/obj11 2023-10-11T15:31:42.355 DEBUG:boto:Data: 2023-10-11T15:31:42.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.356 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.357 DEBUG:boto:Params: {} 2023-10-11T15:31:42.357 DEBUG:boto:Token: None 2023-10-11T15:31:42.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj11 2023-10-11T15:31:42.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9xNOnr8SEatkp1/gJvahgPUP1xk= 2023-10-11T15:31:42.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9xNOnr8SEatkp1/gJvahgPUP1xk='} 2023-10-11T15:31:42.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000040cfe7bd1fd153-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.367 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.367 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.368 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.368 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.369 DEBUG:boto:Data: 2023-10-11T15:31:42.369 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.370 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.370 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.370 DEBUG:boto:Params: {} 2023-10-11T15:31:42.371 DEBUG:boto:Token: None 2023-10-11T15:31:42.371 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.372 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2417672668b4692-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.375 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:31:42.376 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:31:42.376 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.376 DEBUG:boto:Path: /dghqvd-61/obj22 2023-10-11T15:31:42.377 DEBUG:boto:Data: 2023-10-11T15:31:42.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.378 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.378 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.379 DEBUG:boto:Params: {} 2023-10-11T15:31:42.379 DEBUG:boto:Token: None 2023-10-11T15:31:42.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj22 2023-10-11T15:31:42.380 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VAipDPWDwZAe30aqDia0pMTN9HU= 2023-10-11T15:31:42.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VAipDPWDwZAe30aqDia0pMTN9HU='} 2023-10-11T15:31:42.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a6ad9a41042949f-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.423 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.424 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.424 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.425 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.425 DEBUG:boto:Data: 2023-10-11T15:31:42.425 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.426 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.427 DEBUG:boto:Params: {} 2023-10-11T15:31:42.427 DEBUG:boto:Token: None 2023-10-11T15:31:42.428 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083ee18a4f971e190-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.432 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:31:42.432 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:31:42.433 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.433 DEBUG:boto:Path: /dghqvd-61/obj33 2023-10-11T15:31:42.434 DEBUG:boto:Data: 2023-10-11T15:31:42.434 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.435 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.435 DEBUG:boto:Params: {} 2023-10-11T15:31:42.436 DEBUG:boto:Token: None 2023-10-11T15:31:42.436 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj33 2023-10-11T15:31:42.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9WolksdMS5QN5nkbBXgSqB10aUQ= 2023-10-11T15:31:42.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9WolksdMS5QN5nkbBXgSqB10aUQ='} 2023-10-11T15:31:42.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003455cb7187e89c49-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.444 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.445 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.445 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.446 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.446 DEBUG:boto:Data: 2023-10-11T15:31:42.446 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.447 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.448 DEBUG:boto:Params: {} 2023-10-11T15:31:42.448 DEBUG:boto:Token: None 2023-10-11T15:31:42.448 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9fb6465c70eb7fa-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.452 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:31:42.453 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:31:42.454 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.454 DEBUG:boto:Path: /dghqvd-61/obj44 2023-10-11T15:31:42.454 DEBUG:boto:Data: 2023-10-11T15:31:42.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.456 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.456 DEBUG:boto:Params: {} 2023-10-11T15:31:42.457 DEBUG:boto:Token: None 2023-10-11T15:31:42.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj44 2023-10-11T15:31:42.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fjARf/YG2f6f6v9SQRsw8y7/S3Y= 2023-10-11T15:31:42.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fjARf/YG2f6f6v9SQRsw8y7/S3Y='} 2023-10-11T15:31:42.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a34b05dc97cdac4-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.507 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.508 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.508 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.509 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.509 DEBUG:boto:Data: 2023-10-11T15:31:42.509 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.510 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.511 DEBUG:boto:Params: {} 2023-10-11T15:31:42.511 DEBUG:boto:Token: None 2023-10-11T15:31:42.511 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006837eb31bd8603f3-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.516 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:31:42.516 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:31:42.516 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.517 DEBUG:boto:Path: /dghqvd-61/obj55 2023-10-11T15:31:42.517 DEBUG:boto:Data: 2023-10-11T15:31:42.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.518 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.519 DEBUG:boto:Params: {} 2023-10-11T15:31:42.519 DEBUG:boto:Token: None 2023-10-11T15:31:42.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj55 2023-10-11T15:31:42.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Vm4XcN5mgfVyqwIvXMppw2te01U= 2023-10-11T15:31:42.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vm4XcN5mgfVyqwIvXMppw2te01U='} 2023-10-11T15:31:42.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081b80c40ad7adcbe-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:42.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:42.527 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:42.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:42.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:42.633 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:42.633 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:42.637 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 5 2023-10-11T15:31:42.674 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.675 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.675 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.675 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.676 DEBUG:boto:Data: 2023-10-11T15:31:42.676 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.677 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.677 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.678 DEBUG:boto:Params: {} 2023-10-11T15:31:42.678 DEBUG:boto:Token: None 2023-10-11T15:31:42.679 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f3096ee21497726-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.683 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:31:42.684 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:31:42.684 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.685 DEBUG:boto:Path: /dghqvd-61/obj17 2023-10-11T15:31:42.685 DEBUG:boto:Data: 2023-10-11T15:31:42.685 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.686 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.686 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.687 DEBUG:boto:Params: {} 2023-10-11T15:31:42.687 DEBUG:boto:Token: None 2023-10-11T15:31:42.688 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj17 2023-10-11T15:31:42.688 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PY6gX4ueZxneOge3wDKvrwk6fzA= 2023-10-11T15:31:42.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PY6gX4ueZxneOge3wDKvrwk6fzA='} 2023-10-11T15:31:42.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078c4a05286e25377-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.735 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.736 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.736 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.737 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.737 DEBUG:boto:Data: 2023-10-11T15:31:42.737 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.738 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.739 DEBUG:boto:Params: {} 2023-10-11T15:31:42.739 DEBUG:boto:Token: None 2023-10-11T15:31:42.740 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000699e0389e37e5878-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.744 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:31:42.745 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:31:42.746 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.747 DEBUG:boto:Path: /dghqvd-61/obj34 2023-10-11T15:31:42.747 DEBUG:boto:Data: 2023-10-11T15:31:42.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.749 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.750 DEBUG:boto:Params: {} 2023-10-11T15:31:42.750 DEBUG:boto:Token: None 2023-10-11T15:31:42.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj34 2023-10-11T15:31:42.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cbVyX2xbxlt/ONy+kSBqHCLD0ps= 2023-10-11T15:31:42.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cbVyX2xbxlt/ONy+kSBqHCLD0ps='} 2023-10-11T15:31:42.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4657725914f6791-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.758 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.758 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.759 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.759 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.759 DEBUG:boto:Data: 2023-10-11T15:31:42.760 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.761 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.761 DEBUG:boto:Params: {} 2023-10-11T15:31:42.761 DEBUG:boto:Token: None 2023-10-11T15:31:42.762 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6a8a1273738e688-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.768 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:31:42.769 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:31:42.769 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.770 DEBUG:boto:Path: /dghqvd-61/obj51 2023-10-11T15:31:42.770 DEBUG:boto:Data: 2023-10-11T15:31:42.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.771 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.772 DEBUG:boto:Params: {} 2023-10-11T15:31:42.772 DEBUG:boto:Token: None 2023-10-11T15:31:42.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj51 2023-10-11T15:31:42.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SKveH0QpqQRlumRrdxwvqDTLa+E= 2023-10-11T15:31:42.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SKveH0QpqQRlumRrdxwvqDTLa+E='} 2023-10-11T15:31:42.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003484ebdab31df779-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.820 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.821 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.822 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.823 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.823 DEBUG:boto:Data: 2023-10-11T15:31:42.824 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.825 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.826 DEBUG:boto:Params: {} 2023-10-11T15:31:42.827 DEBUG:boto:Token: None 2023-10-11T15:31:42.828 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.829 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.833 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8faae251f9706ce-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.834 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:31:42.835 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:31:42.835 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.837 DEBUG:boto:Path: /dghqvd-61/obj68 2023-10-11T15:31:42.837 DEBUG:boto:Data: 2023-10-11T15:31:42.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.839 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.839 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.840 DEBUG:boto:Params: {} 2023-10-11T15:31:42.841 DEBUG:boto:Token: None 2023-10-11T15:31:42.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj68 2023-10-11T15:31:42.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WSEHvaLQgE7kUwMyxISZuIFKngs= 2023-10-11T15:31:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WSEHvaLQgE7kUwMyxISZuIFKngs='} 2023-10-11T15:31:42.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c3f916111efca8-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.850 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:42.851 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:42.852 DEBUG:boto:Method: HEAD 2023-10-11T15:31:42.852 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:42.853 DEBUG:boto:Data: 2023-10-11T15:31:42.854 DEBUG:boto:Headers: {} 2023-10-11T15:31:42.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.855 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.856 DEBUG:boto:Params: {} 2023-10-11T15:31:42.856 DEBUG:boto:Token: None 2023-10-11T15:31:42.857 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/ 2023-10-11T15:31:42.858 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk= 2023-10-11T15:31:42.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PgzFv+N2mHeZqxoxNdvDS4gFCgk=', 'Content-Length': '0'} 2023-10-11T15:31:42.862 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b058221b12f06ee0-006526bfde-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.862 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:31:42.863 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:31:42.863 DEBUG:boto:Method: PUT 2023-10-11T15:31:42.864 DEBUG:boto:Path: /dghqvd-61/obj85 2023-10-11T15:31:42.864 DEBUG:boto:Data: 2023-10-11T15:31:42.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:42.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:42.865 DEBUG:boto:Port: 8000 2023-10-11T15:31:42.866 DEBUG:boto:Params: {} 2023-10-11T15:31:42.866 DEBUG:boto:Token: None 2023-10-11T15:31:42.867 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:42 GMT /dghqvd-61/obj85 2023-10-11T15:31:42.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gcPUJrfg05RI5jVU1Kck2iXbj+U= 2023-10-11T15:31:42.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gcPUJrfg05RI5jVU1Kck2iXbj+U='} 2023-10-11T15:31:42.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6beff931001e8b9-006526bfde-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:42.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:42.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:42.875 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:42.928 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:42.971 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:42.971 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:42.977 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 10 2023-10-11T15:31:43.020 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.021 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.021 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.022 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.022 DEBUG:boto:Data: 2023-10-11T15:31:43.023 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.024 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.024 DEBUG:boto:Params: {} 2023-10-11T15:31:43.024 DEBUG:boto:Token: None 2023-10-11T15:31:43.025 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.025 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebd73d7dd0b30478-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.029 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:31:43.030 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:31:43.030 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.031 DEBUG:boto:Path: /dghqvd-61/obj19 2023-10-11T15:31:43.032 DEBUG:boto:Data: 2023-10-11T15:31:43.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.034 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.035 DEBUG:boto:Params: {} 2023-10-11T15:31:43.035 DEBUG:boto:Token: None 2023-10-11T15:31:43.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj19 2023-10-11T15:31:43.037 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:voy0u1z5iSY+2k33gPgESJ/n+oM= 2023-10-11T15:31:43.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:voy0u1z5iSY+2k33gPgESJ/n+oM='} 2023-10-11T15:31:43.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058321e88241e0e05-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.044 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.045 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.045 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.046 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.046 DEBUG:boto:Data: 2023-10-11T15:31:43.046 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.047 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.047 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.048 DEBUG:boto:Params: {} 2023-10-11T15:31:43.048 DEBUG:boto:Token: None 2023-10-11T15:31:43.049 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.049 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005efabd72436dc29e-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.053 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:31:43.053 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:31:43.054 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.054 DEBUG:boto:Path: /dghqvd-61/obj38 2023-10-11T15:31:43.055 DEBUG:boto:Data: 2023-10-11T15:31:43.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.056 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.056 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.056 DEBUG:boto:Params: {} 2023-10-11T15:31:43.057 DEBUG:boto:Token: None 2023-10-11T15:31:43.057 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj38 2023-10-11T15:31:43.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/uAnsnKxDnZ1XURBJ2YCE0fCFis= 2023-10-11T15:31:43.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/uAnsnKxDnZ1XURBJ2YCE0fCFis='} 2023-10-11T15:31:43.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ad90cc0d444cffa-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.107 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.108 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.108 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.109 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.109 DEBUG:boto:Data: 2023-10-11T15:31:43.109 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.110 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.110 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.111 DEBUG:boto:Params: {} 2023-10-11T15:31:43.111 DEBUG:boto:Token: None 2023-10-11T15:31:43.112 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.112 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000927a03aa4727b093-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.116 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:31:43.116 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:31:43.117 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.117 DEBUG:boto:Path: /dghqvd-61/obj57 2023-10-11T15:31:43.117 DEBUG:boto:Data: 2023-10-11T15:31:43.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.118 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.119 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.119 DEBUG:boto:Params: {} 2023-10-11T15:31:43.119 DEBUG:boto:Token: None 2023-10-11T15:31:43.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj57 2023-10-11T15:31:43.120 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EQo3q6jCQ7i9jxDPPsxgFX1jfwE= 2023-10-11T15:31:43.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EQo3q6jCQ7i9jxDPPsxgFX1jfwE='} 2023-10-11T15:31:43.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000192fb67cb22d129e-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.125 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.126 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.126 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.127 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.127 DEBUG:boto:Data: 2023-10-11T15:31:43.128 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.129 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.129 DEBUG:boto:Params: {} 2023-10-11T15:31:43.129 DEBUG:boto:Token: None 2023-10-11T15:31:43.130 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072660314c51260a8-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.134 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:31:43.134 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:31:43.135 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.135 DEBUG:boto:Path: /dghqvd-61/obj76 2023-10-11T15:31:43.135 DEBUG:boto:Data: 2023-10-11T15:31:43.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.137 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.137 DEBUG:boto:Params: {} 2023-10-11T15:31:43.137 DEBUG:boto:Token: None 2023-10-11T15:31:43.138 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj76 2023-10-11T15:31:43.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wjbOYZ8uYyuOOBgO0ch9qTpUog8= 2023-10-11T15:31:43.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wjbOYZ8uYyuOOBgO0ch9qTpUog8='} 2023-10-11T15:31:43.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6000e87a3775822-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.187 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.188 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.188 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.189 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.189 DEBUG:boto:Data: 2023-10-11T15:31:43.189 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.190 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.190 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.191 DEBUG:boto:Params: {} 2023-10-11T15:31:43.191 DEBUG:boto:Token: None 2023-10-11T15:31:43.192 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.192 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0c6a4f65fd8e755-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.196 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:31:43.197 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:31:43.197 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.198 DEBUG:boto:Path: /dghqvd-61/obj95 2023-10-11T15:31:43.198 DEBUG:boto:Data: 2023-10-11T15:31:43.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.199 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.199 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.200 DEBUG:boto:Params: {} 2023-10-11T15:31:43.200 DEBUG:boto:Token: None 2023-10-11T15:31:43.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj95 2023-10-11T15:31:43.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fP6FnYJQNI69lsVuDAjd27z4yAA= 2023-10-11T15:31:43.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fP6FnYJQNI69lsVuDAjd27z4yAA='} 2023-10-11T15:31:43.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcfd4a9f3e6b063e-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.208 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:43.260 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:43.261 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:43.306 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:43.306 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:43.312 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 15 2023-10-11T15:31:43.359 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.360 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.361 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.361 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.361 DEBUG:boto:Data: 2023-10-11T15:31:43.362 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.362 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.362 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.363 DEBUG:boto:Params: {} 2023-10-11T15:31:43.363 DEBUG:boto:Token: None 2023-10-11T15:31:43.364 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.365 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af38208b5ee38bbe-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.369 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:31:43.369 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:31:43.370 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.370 DEBUG:boto:Path: /dghqvd-61/obj23 2023-10-11T15:31:43.371 DEBUG:boto:Data: 2023-10-11T15:31:43.371 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.372 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.373 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.373 DEBUG:boto:Params: {} 2023-10-11T15:31:43.374 DEBUG:boto:Token: None 2023-10-11T15:31:43.375 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj23 2023-10-11T15:31:43.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4QAvRjhcubjHdmQYZMv4xZMRo9U= 2023-10-11T15:31:43.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4QAvRjhcubjHdmQYZMv4xZMRo9U='} 2023-10-11T15:31:43.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0392bad31e3ef8c-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.428 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.428 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.429 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.429 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.429 DEBUG:boto:Data: 2023-10-11T15:31:43.430 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.430 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.431 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.431 DEBUG:boto:Params: {} 2023-10-11T15:31:43.432 DEBUG:boto:Token: None 2023-10-11T15:31:43.432 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db6a1a153807fc44-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.437 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:31:43.437 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:31:43.438 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.438 DEBUG:boto:Path: /dghqvd-61/obj46 2023-10-11T15:31:43.439 DEBUG:boto:Data: 2023-10-11T15:31:43.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.440 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.440 DEBUG:boto:Params: {} 2023-10-11T15:31:43.441 DEBUG:boto:Token: None 2023-10-11T15:31:43.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj46 2023-10-11T15:31:43.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wCsqne4JoY+YpW1epxIrXLgdNRk= 2023-10-11T15:31:43.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wCsqne4JoY+YpW1epxIrXLgdNRk='} 2023-10-11T15:31:43.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eea066fa0f13792b-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.447 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.448 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.448 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.449 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.449 DEBUG:boto:Data: 2023-10-11T15:31:43.449 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.450 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.450 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.451 DEBUG:boto:Params: {} 2023-10-11T15:31:43.451 DEBUG:boto:Token: None 2023-10-11T15:31:43.452 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.452 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aa922a7c1a0598f-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.456 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:31:43.457 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:31:43.457 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.458 DEBUG:boto:Path: /dghqvd-61/obj69 2023-10-11T15:31:43.458 DEBUG:boto:Data: 2023-10-11T15:31:43.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.459 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.460 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.460 DEBUG:boto:Params: {} 2023-10-11T15:31:43.460 DEBUG:boto:Token: None 2023-10-11T15:31:43.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj69 2023-10-11T15:31:43.462 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:r1NPDqN72w8JJHHl2odx+cxeY9M= 2023-10-11T15:31:43.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:r1NPDqN72w8JJHHl2odx+cxeY9M='} 2023-10-11T15:31:43.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5a000a6c0981ab4-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.512 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.513 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.513 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.513 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.514 DEBUG:boto:Data: 2023-10-11T15:31:43.514 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.515 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.516 DEBUG:boto:Params: {} 2023-10-11T15:31:43.516 DEBUG:boto:Token: None 2023-10-11T15:31:43.517 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016c98f67baa60516-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.521 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:31:43.521 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:31:43.522 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.522 DEBUG:boto:Path: /dghqvd-61/obj92 2023-10-11T15:31:43.523 DEBUG:boto:Data: 2023-10-11T15:31:43.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.524 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.525 DEBUG:boto:Params: {} 2023-10-11T15:31:43.525 DEBUG:boto:Token: None 2023-10-11T15:31:43.526 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj92 2023-10-11T15:31:43.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:XzF7W7kBHaQSRq4xKoVK7ZYhZMg= 2023-10-11T15:31:43.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:XzF7W7kBHaQSRq4xKoVK7ZYhZMg='} 2023-10-11T15:31:43.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065b3da566eb91111-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.533 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.533 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.534 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.534 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.534 DEBUG:boto:Data: 2023-10-11T15:31:43.535 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.535 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.536 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.536 DEBUG:boto:Params: {} 2023-10-11T15:31:43.536 DEBUG:boto:Token: None 2023-10-11T15:31:43.537 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8ba49d2d9a36cba-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.541 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:31:43.542 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:31:43.542 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.542 DEBUG:boto:Path: /dghqvd-61/obj115 2023-10-11T15:31:43.543 DEBUG:boto:Data: 2023-10-11T15:31:43.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.544 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.544 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.544 DEBUG:boto:Params: {} 2023-10-11T15:31:43.545 DEBUG:boto:Token: None 2023-10-11T15:31:43.545 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj115 2023-10-11T15:31:43.546 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u/t7I86r8gZXPaSgflCP3R1Tvgg= 2023-10-11T15:31:43.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u/t7I86r8gZXPaSgflCP3R1Tvgg='} 2023-10-11T15:31:43.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9a21a999c7b4a1e-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.552 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:43.605 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:43.606 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:43.653 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:43.653 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:43.661 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 20 2023-10-11T15:31:43.711 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.712 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.712 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.713 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.713 DEBUG:boto:Data: 2023-10-11T15:31:43.714 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.714 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.714 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.715 DEBUG:boto:Params: {} 2023-10-11T15:31:43.715 DEBUG:boto:Token: None 2023-10-11T15:31:43.716 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009854175290405d23-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.721 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:31:43.721 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:31:43.721 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.722 DEBUG:boto:Path: /dghqvd-61/obj29 2023-10-11T15:31:43.722 DEBUG:boto:Data: 2023-10-11T15:31:43.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.723 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.723 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.724 DEBUG:boto:Params: {} 2023-10-11T15:31:43.724 DEBUG:boto:Token: None 2023-10-11T15:31:43.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj29 2023-10-11T15:31:43.725 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NJSe7o6GPgLDmdrmO7/LHtzRcKw= 2023-10-11T15:31:43.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NJSe7o6GPgLDmdrmO7/LHtzRcKw='} 2023-10-11T15:31:43.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000086116991aa11d78-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.775 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.776 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.777 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.777 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.777 DEBUG:boto:Data: 2023-10-11T15:31:43.778 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.779 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.779 DEBUG:boto:Params: {} 2023-10-11T15:31:43.779 DEBUG:boto:Token: None 2023-10-11T15:31:43.780 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6300e65a145d816-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.785 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:31:43.785 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:31:43.785 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.786 DEBUG:boto:Path: /dghqvd-61/obj58 2023-10-11T15:31:43.786 DEBUG:boto:Data: 2023-10-11T15:31:43.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.787 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.789 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.790 DEBUG:boto:Params: {} 2023-10-11T15:31:43.790 DEBUG:boto:Token: None 2023-10-11T15:31:43.791 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj58 2023-10-11T15:31:43.791 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RwxC9VMWKRuyO8OF1UDHltPoTDo= 2023-10-11T15:31:43.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RwxC9VMWKRuyO8OF1UDHltPoTDo='} 2023-10-11T15:31:43.797 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e38ca164bd5553f-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.797 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.798 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.798 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.799 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.799 DEBUG:boto:Data: 2023-10-11T15:31:43.800 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.800 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.800 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.801 DEBUG:boto:Params: {} 2023-10-11T15:31:43.801 DEBUG:boto:Token: None 2023-10-11T15:31:43.802 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.802 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023790fabb403f1c1-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.807 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:31:43.807 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:31:43.807 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.808 DEBUG:boto:Path: /dghqvd-61/obj87 2023-10-11T15:31:43.808 DEBUG:boto:Data: 2023-10-11T15:31:43.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.809 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.810 DEBUG:boto:Params: {} 2023-10-11T15:31:43.811 DEBUG:boto:Token: None 2023-10-11T15:31:43.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj87 2023-10-11T15:31:43.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TJV4Jq87f99WNRwD1t/05gfjUUU= 2023-10-11T15:31:43.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TJV4Jq87f99WNRwD1t/05gfjUUU='} 2023-10-11T15:31:43.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b62b933274ee2c25-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.859 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.860 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.860 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.860 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.861 DEBUG:boto:Data: 2023-10-11T15:31:43.861 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.862 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.863 DEBUG:boto:Params: {} 2023-10-11T15:31:43.863 DEBUG:boto:Token: None 2023-10-11T15:31:43.864 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026e155738bdd7910-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.868 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:31:43.869 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:31:43.869 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.870 DEBUG:boto:Path: /dghqvd-61/obj116 2023-10-11T15:31:43.870 DEBUG:boto:Data: 2023-10-11T15:31:43.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.871 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.872 DEBUG:boto:Params: {} 2023-10-11T15:31:43.872 DEBUG:boto:Token: None 2023-10-11T15:31:43.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj116 2023-10-11T15:31:43.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+ELhxfGXUPLC96JX5v04KCqZ7Cg= 2023-10-11T15:31:43.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+ELhxfGXUPLC96JX5v04KCqZ7Cg='} 2023-10-11T15:31:43.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000700440240e0eb18c-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.879 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:43.879 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:43.880 DEBUG:boto:Method: HEAD 2023-10-11T15:31:43.880 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:43.881 DEBUG:boto:Data: 2023-10-11T15:31:43.881 DEBUG:boto:Headers: {} 2023-10-11T15:31:43.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.882 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.882 DEBUG:boto:Params: {} 2023-10-11T15:31:43.883 DEBUG:boto:Token: None 2023-10-11T15:31:43.883 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/ 2023-10-11T15:31:43.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww= 2023-10-11T15:31:43.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OJsVZ2sh5qol2/y5jtZGCm1Rvww=', 'Content-Length': '0'} 2023-10-11T15:31:43.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b80aff46d3daa18-006526bfdf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.888 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:31:43.888 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:31:43.888 DEBUG:boto:Method: PUT 2023-10-11T15:31:43.889 DEBUG:boto:Path: /dghqvd-61/obj145 2023-10-11T15:31:43.889 DEBUG:boto:Data: 2023-10-11T15:31:43.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:43.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:43.891 DEBUG:boto:Port: 8000 2023-10-11T15:31:43.891 DEBUG:boto:Params: {} 2023-10-11T15:31:43.891 DEBUG:boto:Token: None 2023-10-11T15:31:43.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:43 GMT /dghqvd-61/obj145 2023-10-11T15:31:43.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rnRe1jMWeGiiM357//jl+h0/VRw= 2023-10-11T15:31:43.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rnRe1jMWeGiiM357//jl+h0/VRw='} 2023-10-11T15:31:43.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c280a3360cb7279b-006526bfdf-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:43.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:43.936 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:43.989 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:44.037 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:44.037 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:44.047 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 25 2023-10-11T15:31:44.105 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.105 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.106 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.106 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.106 DEBUG:boto:Data: 2023-10-11T15:31:44.107 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.108 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.108 DEBUG:boto:Params: {} 2023-10-11T15:31:44.109 DEBUG:boto:Token: None 2023-10-11T15:31:44.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1ae2bfb3cddd0c4-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.114 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:31:44.115 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:31:44.115 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.116 DEBUG:boto:Path: /dghqvd-61/obj31 2023-10-11T15:31:44.116 DEBUG:boto:Data: 2023-10-11T15:31:44.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.117 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.117 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.118 DEBUG:boto:Params: {} 2023-10-11T15:31:44.118 DEBUG:boto:Token: None 2023-10-11T15:31:44.119 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj31 2023-10-11T15:31:44.119 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:paOksgtvBvn78N8Pni/C7AlaIpw= 2023-10-11T15:31:44.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:paOksgtvBvn78N8Pni/C7AlaIpw='} 2023-10-11T15:31:44.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c8aecb57749fdf3-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.125 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.126 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.126 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.127 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.127 DEBUG:boto:Data: 2023-10-11T15:31:44.127 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.128 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.129 DEBUG:boto:Params: {} 2023-10-11T15:31:44.129 DEBUG:boto:Token: None 2023-10-11T15:31:44.129 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092202a35b2e7f564-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.135 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:31:44.135 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:31:44.135 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.136 DEBUG:boto:Path: /dghqvd-61/obj62 2023-10-11T15:31:44.136 DEBUG:boto:Data: 2023-10-11T15:31:44.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.137 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.138 DEBUG:boto:Params: {} 2023-10-11T15:31:44.138 DEBUG:boto:Token: None 2023-10-11T15:31:44.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj62 2023-10-11T15:31:44.139 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cPkIEaWw1rc+Q9tS709xXOcc2E0= 2023-10-11T15:31:44.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cPkIEaWw1rc+Q9tS709xXOcc2E0='} 2023-10-11T15:31:44.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000745b180cbfa55a5a-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.145 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.146 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.146 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.146 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.147 DEBUG:boto:Data: 2023-10-11T15:31:44.147 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.148 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.148 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.148 DEBUG:boto:Params: {} 2023-10-11T15:31:44.149 DEBUG:boto:Token: None 2023-10-11T15:31:44.149 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000226ffecf43c43b74-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.154 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:31:44.155 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:31:44.155 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.156 DEBUG:boto:Path: /dghqvd-61/obj93 2023-10-11T15:31:44.156 DEBUG:boto:Data: 2023-10-11T15:31:44.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.157 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.157 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.158 DEBUG:boto:Params: {} 2023-10-11T15:31:44.158 DEBUG:boto:Token: None 2023-10-11T15:31:44.159 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj93 2023-10-11T15:31:44.159 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tvhki++gtXdZoHw6E0nFEWBjR8E= 2023-10-11T15:31:44.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tvhki++gtXdZoHw6E0nFEWBjR8E='} 2023-10-11T15:31:44.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023ed68ea5f50b9c9-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.165 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.165 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.166 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.166 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.167 DEBUG:boto:Data: 2023-10-11T15:31:44.167 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.168 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.168 DEBUG:boto:Params: {} 2023-10-11T15:31:44.169 DEBUG:boto:Token: None 2023-10-11T15:31:44.169 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.170 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6d99fb717a68ed3-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.174 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:31:44.175 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:31:44.175 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.175 DEBUG:boto:Path: /dghqvd-61/obj124 2023-10-11T15:31:44.176 DEBUG:boto:Data: 2023-10-11T15:31:44.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.177 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.177 DEBUG:boto:Params: {} 2023-10-11T15:31:44.178 DEBUG:boto:Token: None 2023-10-11T15:31:44.178 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj124 2023-10-11T15:31:44.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:X6dpOFfPmKrthqHE838d+9osJY4= 2023-10-11T15:31:44.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X6dpOFfPmKrthqHE838d+9osJY4='} 2023-10-11T15:31:44.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fd9b4c63c714e34-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.223 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.224 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.224 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.225 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.225 DEBUG:boto:Data: 2023-10-11T15:31:44.225 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.226 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.227 DEBUG:boto:Params: {} 2023-10-11T15:31:44.227 DEBUG:boto:Token: None 2023-10-11T15:31:44.227 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d15e7fbce0b20ae1-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.232 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:31:44.233 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:31:44.233 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.234 DEBUG:boto:Path: /dghqvd-61/obj155 2023-10-11T15:31:44.234 DEBUG:boto:Data: 2023-10-11T15:31:44.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.236 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.236 DEBUG:boto:Params: {} 2023-10-11T15:31:44.237 DEBUG:boto:Token: None 2023-10-11T15:31:44.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj155 2023-10-11T15:31:44.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:u3OWSdh4wDyo5G4nxlN2QjbojGY= 2023-10-11T15:31:44.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u3OWSdh4wDyo5G4nxlN2QjbojGY='} 2023-10-11T15:31:44.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dfa03b8fa2f1ba6-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:44.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:44.245 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:44.352 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:31:44.353 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-61 2023-10-11T15:31:44.364 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 30 2023-10-11T15:31:44.424 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.424 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.425 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.425 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.425 DEBUG:boto:Data: 2023-10-11T15:31:44.426 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.427 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.427 DEBUG:boto:Params: {} 2023-10-11T15:31:44.427 DEBUG:boto:Token: None 2023-10-11T15:31:44.428 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a0149f2ad4f308-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.433 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:31:44.433 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:31:44.434 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.434 DEBUG:boto:Path: /dghqvd-61/obj37 2023-10-11T15:31:44.434 DEBUG:boto:Data: 2023-10-11T15:31:44.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.436 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.436 DEBUG:boto:Params: {} 2023-10-11T15:31:44.436 DEBUG:boto:Token: None 2023-10-11T15:31:44.437 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj37 2023-10-11T15:31:44.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uzTb7YiaIZb2nDzEykIVD+kmVHo= 2023-10-11T15:31:44.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uzTb7YiaIZb2nDzEykIVD+kmVHo='} 2023-10-11T15:31:44.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004551705c360adc4a-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.487 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.488 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.488 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.489 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.489 DEBUG:boto:Data: 2023-10-11T15:31:44.489 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.490 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.490 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.491 DEBUG:boto:Params: {} 2023-10-11T15:31:44.491 DEBUG:boto:Token: None 2023-10-11T15:31:44.491 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.492 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d531cdb79cadc8b6-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.497 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:31:44.497 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:31:44.497 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.498 DEBUG:boto:Path: /dghqvd-61/obj74 2023-10-11T15:31:44.498 DEBUG:boto:Data: 2023-10-11T15:31:44.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.499 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.500 DEBUG:boto:Params: {} 2023-10-11T15:31:44.500 DEBUG:boto:Token: None 2023-10-11T15:31:44.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj74 2023-10-11T15:31:44.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qYA9kDRFGrFqfVLW1ft1iFvm+VE= 2023-10-11T15:31:44.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qYA9kDRFGrFqfVLW1ft1iFvm+VE='} 2023-10-11T15:31:44.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b17ef5a04666e526-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.507 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.507 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.508 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.508 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.508 DEBUG:boto:Data: 2023-10-11T15:31:44.509 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.510 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.510 DEBUG:boto:Params: {} 2023-10-11T15:31:44.511 DEBUG:boto:Token: None 2023-10-11T15:31:44.511 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.511 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d000d9de8fd2fa9-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.516 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:31:44.516 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:31:44.517 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.517 DEBUG:boto:Path: /dghqvd-61/obj111 2023-10-11T15:31:44.518 DEBUG:boto:Data: 2023-10-11T15:31:44.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.519 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.519 DEBUG:boto:Params: {} 2023-10-11T15:31:44.520 DEBUG:boto:Token: None 2023-10-11T15:31:44.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj111 2023-10-11T15:31:44.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nPcwyQG7NrDdFIQ8DY6oSv124PI= 2023-10-11T15:31:44.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nPcwyQG7NrDdFIQ8DY6oSv124PI='} 2023-10-11T15:31:44.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dc6f70f2bf2223d-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.567 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.568 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.568 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.569 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.569 DEBUG:boto:Data: 2023-10-11T15:31:44.569 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.570 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.571 DEBUG:boto:Params: {} 2023-10-11T15:31:44.571 DEBUG:boto:Token: None 2023-10-11T15:31:44.571 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000752b0bcfe57e9a8d-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.576 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:31:44.577 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:31:44.577 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.578 DEBUG:boto:Path: /dghqvd-61/obj148 2023-10-11T15:31:44.578 DEBUG:boto:Data: 2023-10-11T15:31:44.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.579 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.580 DEBUG:boto:Params: {} 2023-10-11T15:31:44.580 DEBUG:boto:Token: None 2023-10-11T15:31:44.580 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj148 2023-10-11T15:31:44.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uEaCeXmySfq1DLSlwocSZkAA9JU= 2023-10-11T15:31:44.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uEaCeXmySfq1DLSlwocSZkAA9JU='} 2023-10-11T15:31:44.587 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006afe8d99b00c30dc-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.587 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:31:44.588 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:31:44.588 DEBUG:boto:Method: HEAD 2023-10-11T15:31:44.588 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:31:44.589 DEBUG:boto:Data: 2023-10-11T15:31:44.589 DEBUG:boto:Headers: {} 2023-10-11T15:31:44.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.590 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.590 DEBUG:boto:Params: {} 2023-10-11T15:31:44.591 DEBUG:boto:Token: None 2023-10-11T15:31:44.591 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/ 2023-10-11T15:31:44.592 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI= 2023-10-11T15:31:44.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Mb4DqtlEGhhw/ggbYaC8+rzldDI=', 'Content-Length': '0'} 2023-10-11T15:31:44.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2267d68d1501157-006526bfe0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.596 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:31:44.597 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:31:44.597 DEBUG:boto:Method: PUT 2023-10-11T15:31:44.598 DEBUG:boto:Path: /dghqvd-61/obj185 2023-10-11T15:31:44.598 DEBUG:boto:Data: 2023-10-11T15:31:44.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:31:44.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:31:44.599 DEBUG:boto:Port: 8000 2023-10-11T15:31:44.600 DEBUG:boto:Params: {} 2023-10-11T15:31:44.600 DEBUG:boto:Token: None 2023-10-11T15:31:44.601 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:31:44 GMT /dghqvd-61/obj185 2023-10-11T15:31:44.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qh1FiPENAILSXE0R17tdAKuYWV8= 2023-10-11T15:31:44.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:31:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qh1FiPENAILSXE0R17tdAKuYWV8='} 2023-10-11T15:31:44.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5a4135a1758013b-006526bfe0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:31:44.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:44.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:31:44.648 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:31:44.699 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:44.699 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:44.731 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:44.730+0000 7f8155cf5780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-11T15:31:44.732 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:44.731+0000 7f8155cf5780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-11T15:31:44.753 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:31:44.754 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:31:44.756 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:31:44.756 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:44.778 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:31:44.778 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:31:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:31:44.781 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:31:45.782 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:31:45.806 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:31:45.806 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:31:45.806 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:31:45.807 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:31:45.808 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-61 2023-10-11T15:31:45.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:31:45.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:31:45.809 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:31:45.905 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:31:45.905 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:31:45.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:31:45.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:45.943+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:31:50.944 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:50.943+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:31:55.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:31:55.944+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:00.945 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:00.944+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:05.946 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:05.945+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:10.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:10.945+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:15.947 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:15.945+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:20.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:20.946+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:25.948 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:25.946+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:30.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:30.946+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:35.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:35.947+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:40.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:40.947+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:45.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:45.948+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:50.951 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:50.949+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:32:55.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:32:55.950+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:00.952 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:00.951+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:05.953 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:05.952+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:10.954 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:10.952+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:15.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:15.953+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:20.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:20.953+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:25.955 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:25.954+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:30.956 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:30.955+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:35.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:35.955+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:40.957 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:40.956+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:45.958 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:45.957+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:50.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:50.956+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:33:55.959 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:33:55.957+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:00.960 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:00.958+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:05.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:05.958+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:10.961 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:10.959+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:15.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:15.960+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:20.962 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:20.960+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:25.963 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:25.961+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:30.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:30.962+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:35.964 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:35.963+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:40.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:40.963+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:45.965 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:45.964+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:50.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:50.965+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:34:55.966 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:34:55.965+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:00.967 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:00.966+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:05.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:05.966+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:10.968 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:10.967+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:15.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:15.967+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:20.969 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:20.967+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:25.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:25.968+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:30.970 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:30.968+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:35.971 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:35.969+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:40.972 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:40.970+0000 7f4a477f4780 1 waiting to reach incremental sync.. 2023-10-11T15:35:45.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:45.973+0000 7f4a477f4780 1 bucket sync caught up with source: 2023-10-11T15:35:45.975 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.96.6, , , , , , 00000000002.96.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.93.6, , , , 00000000001.71.6] 2023-10-11T15:35:45.976 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.96.6, , , , , , 00000000002.96.6, , , , , 00000000002.138.6, , , , , , , , , , , , , , , 00000000001.93.6, , , , 00000000001.71.6] 2023-10-11T15:35:45.976 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:45.973+0000 7f4a477f4780 0 bucket checkpoint complete 2023-10-11T15:35:45.992 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-61 2023-10-11T15:35:45.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:45.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:45.994 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:35:46.057 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:46.057 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:46.094 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:46.092+0000 7f3618122780 1 bucket sync caught up with source: 2023-10-11T15:35:46.094 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, 00000000003.271.6, 00000000002.269.6, 00000000002.243.6, 00000000005.210.6, 00000000004.317.6, 00000000001.199.6, 00000000001.284.6, 00000000004.155.6, 00000000007.354.6, 00000000006.252.6] 2023-10-11T15:35:46.094 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, 00000000003.271.6, 00000000002.269.6, 00000000002.243.6, 00000000005.210.6, 00000000004.317.6, 00000000001.199.6, 00000000001.284.6, 00000000004.155.6, 00000000007.354.6, 00000000006.252.6] 2023-10-11T15:35:46.095 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:46.092+0000 7f3618122780 0 bucket checkpoint complete 2023-10-11T15:35:46.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-61 zones={a2, a1} 2023-10-11T15:35:46.108 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.109 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.109 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.110 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:35:46.110 DEBUG:boto:Data: 2023-10-11T15:35:46.110 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.111 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.112 DEBUG:boto:Params: {} 2023-10-11T15:35:46.112 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:35:46.113 DEBUG:boto:Token: None 2023-10-11T15:35:46.114 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.114 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Wmdd7M2V6SXaiuf78JODZbEO7g= 2023-10-11T15:35:46.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Wmdd7M2V6SXaiuf78JODZbEO7g=', 'Content-Length': '0'} 2023-10-11T15:35:46.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c1f2a9d73d1381c-006526c0d2-1f5c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.121 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.121 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.122 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.122 DEBUG:boto:Path: /dghqvd-61/ 2023-10-11T15:35:46.122 DEBUG:boto:Data: 2023-10-11T15:35:46.123 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.123 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.124 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.124 DEBUG:boto:Params: {} 2023-10-11T15:35:46.125 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:35:46.125 DEBUG:boto:Token: None 2023-10-11T15:35:46.126 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Wmdd7M2V6SXaiuf78JODZbEO7g= 2023-10-11T15:35:46.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Wmdd7M2V6SXaiuf78JODZbEO7g=', 'Content-Length': '0'} 2023-10-11T15:35:46.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055a7f85236a15dce-006526c0d2-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:35:46.133 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.133 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.134 DEBUG:boto:path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.134 DEBUG:boto:auth_path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.134 DEBUG:boto:Method: GET 2023-10-11T15:35:46.135 DEBUG:boto:Path: /dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.135 DEBUG:boto:Data: 2023-10-11T15:35:46.136 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.137 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.137 DEBUG:boto:Params: {} 2023-10-11T15:35:46.137 DEBUG:boto:Token: None 2023-10-11T15:35:46.138 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/?versions 2023-10-11T15:35:46.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI= 2023-10-11T15:35:46.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI=', 'Content-Length': '0'} 2023-10-11T15:35:46.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063404c8c17c37284-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.142 DEBUG:boto:b'dghqvd-61999falseobj11nulltrue2023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:35:46.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:35:46.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:35:46.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:35:46.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:35:46.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:35:46.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:35:46.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:35:46.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:35:46.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:35:46.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:35:46.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:35:46.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:35:46.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:35:46.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:35:46.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:35:46.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:35:46.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:35:46.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:35:46.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:35:46.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:35:46.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:35:46.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:35:46.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:35:46.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:35:46.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:35:46.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:35:46.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:35:46.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:35:46.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:35:46.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:35:46.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:35:46.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:35:46.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:35:46.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:35:46.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:35:46.159 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.159 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.160 DEBUG:boto:path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.160 DEBUG:boto:auth_path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.161 DEBUG:boto:Method: GET 2023-10-11T15:35:46.161 DEBUG:boto:Path: /dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.162 DEBUG:boto:Data: 2023-10-11T15:35:46.162 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.163 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.163 DEBUG:boto:Params: {} 2023-10-11T15:35:46.164 DEBUG:boto:Token: None 2023-10-11T15:35:46.164 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/?versions 2023-10-11T15:35:46.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI= 2023-10-11T15:35:46.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI=', 'Content-Length': '0'} 2023-10-11T15:35:46.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b91f0e76677e069-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.170 DEBUG:boto:b'dghqvd-61999falseobj11nulltrue2023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:35:46.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-10-11T15:35:46.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-10-11T15:35:46.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-10-11T15:35:46.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-10-11T15:35:46.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-10-11T15:35:46.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-10-11T15:35:46.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-10-11T15:35:46.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-10-11T15:35:46.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-10-11T15:35:46.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-10-11T15:35:46.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:35:46.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-10-11T15:35:46.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-10-11T15:35:46.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-10-11T15:35:46.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:35:46.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-10-11T15:35:46.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-10-11T15:35:46.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-10-11T15:35:46.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:35:46.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-10-11T15:35:46.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-10-11T15:35:46.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:35:46.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-10-11T15:35:46.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-10-11T15:35:46.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-10-11T15:35:46.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-10-11T15:35:46.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-10-11T15:35:46.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-10-11T15:35:46.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-10-11T15:35:46.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-10-11T15:35:46.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-10-11T15:35:46.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-10-11T15:35:46.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-10-11T15:35:46.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-10-11T15:35:46.185 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.186 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.186 DEBUG:boto:path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.186 DEBUG:boto:auth_path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.187 DEBUG:boto:Method: GET 2023-10-11T15:35:46.187 DEBUG:boto:Path: /dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.188 DEBUG:boto:Data: 2023-10-11T15:35:46.188 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.189 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.189 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.189 DEBUG:boto:Params: {} 2023-10-11T15:35:46.190 DEBUG:boto:Token: None 2023-10-11T15:35:46.190 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/?versions 2023-10-11T15:35:46.191 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI= 2023-10-11T15:35:46.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI=', 'Content-Length': '0'} 2023-10-11T15:35:46.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb39816f0be0d67-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.195 DEBUG:boto:b'dghqvd-61999falseobj11nulltrue2023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.197 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.197 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.197 DEBUG:boto:path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.198 DEBUG:boto:auth_path=/dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.198 DEBUG:boto:Method: GET 2023-10-11T15:35:46.199 DEBUG:boto:Path: /dghqvd-61/?versions&max-keys=999 2023-10-11T15:35:46.199 DEBUG:boto:Data: 2023-10-11T15:35:46.199 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.200 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.200 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.201 DEBUG:boto:Params: {} 2023-10-11T15:35:46.201 DEBUG:boto:Token: None 2023-10-11T15:35:46.201 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/?versions 2023-10-11T15:35:46.202 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI= 2023-10-11T15:35:46.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nZ0SXEeemoiXjJ6CEgsS1EvtVyI=', 'Content-Length': '0'} 2023-10-11T15:35:46.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098da6ac2f3c70db2-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.207 DEBUG:boto:b'dghqvd-61999falseobj11nulltrue2023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:35:46.210 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.211 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.211 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.211 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.212 DEBUG:boto:Method: GET 2023-10-11T15:35:46.212 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.213 DEBUG:boto:Data: 2023-10-11T15:35:46.213 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.214 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.214 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.215 DEBUG:boto:Params: {} 2023-10-11T15:35:46.215 DEBUG:boto:Token: None 2023-10-11T15:35:46.216 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.216 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8= 2023-10-11T15:35:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8=', 'Content-Length': '0'} 2023-10-11T15:35:46.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000023326694f711941-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.220 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.220 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.221 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.221 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.221 DEBUG:boto:Method: GET 2023-10-11T15:35:46.222 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.222 DEBUG:boto:Data: 2023-10-11T15:35:46.223 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.223 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.224 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.224 DEBUG:boto:Params: {} 2023-10-11T15:35:46.225 DEBUG:boto:Token: None 2023-10-11T15:35:46.225 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.226 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8= 2023-10-11T15:35:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8=', 'Content-Length': '0'} 2023-10-11T15:35:46.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b185048c61ced8b-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.275 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.275 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.275 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.276 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.276 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.277 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.277 DEBUG:boto:Data: 2023-10-11T15:35:46.278 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.278 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.279 DEBUG:boto:Params: {} 2023-10-11T15:35:46.279 DEBUG:boto:Token: None 2023-10-11T15:35:46.280 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EEYgbucexNEXuHKmOSeJKLG5vco= 2023-10-11T15:35:46.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EEYgbucexNEXuHKmOSeJKLG5vco=', 'Content-Length': '0'} 2023-10-11T15:35:46.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd365e4c2416b7a-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.283 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.284 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.284 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.284 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.285 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.285 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.286 DEBUG:boto:Data: 2023-10-11T15:35:46.286 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.286 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.287 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.287 DEBUG:boto:Params: {} 2023-10-11T15:35:46.288 DEBUG:boto:Token: None 2023-10-11T15:35:46.288 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EEYgbucexNEXuHKmOSeJKLG5vco= 2023-10-11T15:35:46.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EEYgbucexNEXuHKmOSeJKLG5vco=', 'Content-Length': '0'} 2023-10-11T15:35:46.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdeb76d283e1740e-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:35:46.292 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.292 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.293 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.293 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.294 DEBUG:boto:Method: GET 2023-10-11T15:35:46.294 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.294 DEBUG:boto:Data: 2023-10-11T15:35:46.295 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.296 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.296 DEBUG:boto:Params: {} 2023-10-11T15:35:46.297 DEBUG:boto:Token: None 2023-10-11T15:35:46.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8= 2023-10-11T15:35:46.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8=', 'Content-Length': '0'} 2023-10-11T15:35:46.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d63f96327a261e-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.342 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.343 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.343 DEBUG:boto:path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.344 DEBUG:boto:auth_path=/dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.344 DEBUG:boto:Method: GET 2023-10-11T15:35:46.345 DEBUG:boto:Path: /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.345 DEBUG:boto:Data: 2023-10-11T15:35:46.346 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.347 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.347 DEBUG:boto:Params: {} 2023-10-11T15:35:46.348 DEBUG:boto:Token: None 2023-10-11T15:35:46.348 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11?versionId=null 2023-10-11T15:35:46.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8= 2023-10-11T15:35:46.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m2DwneW5c44TY2vvtotSMMyDmc8=', 'Content-Length': '0'} 2023-10-11T15:35:46.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74bd7abab4cd975-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.352 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.352 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.353 DEBUG:boto:path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.353 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.354 DEBUG:boto:Method: GET 2023-10-11T15:35:46.354 DEBUG:boto:Path: /dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.354 DEBUG:boto:Data: 2023-10-11T15:35:46.355 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.356 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.356 DEBUG:boto:Params: {} 2023-10-11T15:35:46.357 DEBUG:boto:Token: None 2023-10-11T15:35:46.357 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010c064844b0d2ea2-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.361 DEBUG:boto:b'dghqvd-61obj111000falseobj112023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.362 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.363 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.363 DEBUG:boto:path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.363 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.364 DEBUG:boto:Method: GET 2023-10-11T15:35:46.364 DEBUG:boto:Path: /dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.365 DEBUG:boto:Data: 2023-10-11T15:35:46.365 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.366 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.366 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.367 DEBUG:boto:Params: {} 2023-10-11T15:35:46.367 DEBUG:boto:Token: None 2023-10-11T15:35:46.368 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096779d97b43abbd2-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.373 DEBUG:boto:b'dghqvd-61obj111000falseobj112023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.373 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.374 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.374 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.375 DEBUG:boto:Path: /dghqvd-61/obj11 2023-10-11T15:35:46.375 DEBUG:boto:Data: 2023-10-11T15:35:46.376 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.376 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.377 DEBUG:boto:Params: {} 2023-10-11T15:35:46.377 DEBUG:boto:Token: None 2023-10-11T15:35:46.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11 2023-10-11T15:35:46.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JXT+WXcGq0bJVDhkHKEmNKSLj4Y= 2023-10-11T15:35:46.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JXT+WXcGq0bJVDhkHKEmNKSLj4Y=', 'Content-Length': '0'} 2023-10-11T15:35:46.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f5139e39e22bcc-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.381 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.381 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.382 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.382 DEBUG:boto:Path: /dghqvd-61/obj11 2023-10-11T15:35:46.383 DEBUG:boto:Data: 2023-10-11T15:35:46.383 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.383 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.384 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.384 DEBUG:boto:Params: {} 2023-10-11T15:35:46.385 DEBUG:boto:Token: None 2023-10-11T15:35:46.385 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11 2023-10-11T15:35:46.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JXT+WXcGq0bJVDhkHKEmNKSLj4Y= 2023-10-11T15:35:46.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JXT+WXcGq0bJVDhkHKEmNKSLj4Y=', 'Content-Length': '0'} 2023-10-11T15:35:46.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0a7163d30e54b0-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:35:46.389 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.389 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.390 DEBUG:boto:Method: GET 2023-10-11T15:35:46.390 DEBUG:boto:Path: /dghqvd-61/obj11 2023-10-11T15:35:46.391 DEBUG:boto:Data: 2023-10-11T15:35:46.391 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.392 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.392 DEBUG:boto:Params: {} 2023-10-11T15:35:46.393 DEBUG:boto:Token: None 2023-10-11T15:35:46.393 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11 2023-10-11T15:35:46.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:insSV9Lj4DFjBwd/jrSiaHbrfrw= 2023-10-11T15:35:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:insSV9Lj4DFjBwd/jrSiaHbrfrw=', 'Content-Length': '0'} 2023-10-11T15:35:46.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a3b96dab37fd83-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.397 DEBUG:boto:path=/dghqvd-61/obj11 2023-10-11T15:35:46.397 DEBUG:boto:auth_path=/dghqvd-61/obj11 2023-10-11T15:35:46.398 DEBUG:boto:Method: GET 2023-10-11T15:35:46.398 DEBUG:boto:Path: /dghqvd-61/obj11 2023-10-11T15:35:46.399 DEBUG:boto:Data: 2023-10-11T15:35:46.402 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.403 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.404 DEBUG:boto:Params: {} 2023-10-11T15:35:46.404 DEBUG:boto:Token: None 2023-10-11T15:35:46.405 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj11 2023-10-11T15:35:46.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:insSV9Lj4DFjBwd/jrSiaHbrfrw= 2023-10-11T15:35:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:insSV9Lj4DFjBwd/jrSiaHbrfrw=', 'Content-Length': '0'} 2023-10-11T15:35:46.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e044027c20f1e404-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.450 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.451 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.451 DEBUG:boto:path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.452 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.452 DEBUG:boto:Method: GET 2023-10-11T15:35:46.453 DEBUG:boto:Path: /dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.453 DEBUG:boto:Data: 2023-10-11T15:35:46.454 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.455 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.455 DEBUG:boto:Params: {} 2023-10-11T15:35:46.456 DEBUG:boto:Token: None 2023-10-11T15:35:46.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd368cbb9bccb836-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.461 DEBUG:boto:b'dghqvd-61obj111000falseobj112023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.461 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.462 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.462 DEBUG:boto:path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.463 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.463 DEBUG:boto:Method: GET 2023-10-11T15:35:46.464 DEBUG:boto:Path: /dghqvd-61/?prefix=obj11 2023-10-11T15:35:46.464 DEBUG:boto:Data: 2023-10-11T15:35:46.465 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.465 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.465 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.466 DEBUG:boto:Params: {} 2023-10-11T15:35:46.466 DEBUG:boto:Token: None 2023-10-11T15:35:46.467 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.467 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0f50981eac146f-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.472 DEBUG:boto:b'dghqvd-61obj111000falseobj112023-10-11T15:31:42.360Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:35:46.474 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.474 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.474 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.475 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.475 DEBUG:boto:Method: GET 2023-10-11T15:35:46.476 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.476 DEBUG:boto:Data: 2023-10-11T15:35:46.477 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.477 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.478 DEBUG:boto:Params: {} 2023-10-11T15:35:46.478 DEBUG:boto:Token: None 2023-10-11T15:35:46.479 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI= 2023-10-11T15:35:46.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI=', 'Content-Length': '0'} 2023-10-11T15:35:46.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5bada37c5e3dde-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.483 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.483 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.484 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.484 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.484 DEBUG:boto:Method: GET 2023-10-11T15:35:46.485 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.485 DEBUG:boto:Data: 2023-10-11T15:35:46.486 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.487 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.487 DEBUG:boto:Params: {} 2023-10-11T15:35:46.487 DEBUG:boto:Token: None 2023-10-11T15:35:46.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI= 2023-10-11T15:35:46.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI=', 'Content-Length': '0'} 2023-10-11T15:35:46.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000425f5707f9198afd-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.492 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.492 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.492 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.493 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.493 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.494 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.494 DEBUG:boto:Data: 2023-10-11T15:35:46.494 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.495 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.495 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.496 DEBUG:boto:Params: {} 2023-10-11T15:35:46.496 DEBUG:boto:Token: None 2023-10-11T15:35:46.497 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zNtZn74yVR9S8jydlBckoAs0XQs= 2023-10-11T15:35:46.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zNtZn74yVR9S8jydlBckoAs0XQs=', 'Content-Length': '0'} 2023-10-11T15:35:46.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929c024636451dce-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.500 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.501 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.501 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.502 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.502 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.502 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.503 DEBUG:boto:Data: 2023-10-11T15:35:46.503 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.504 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.505 DEBUG:boto:Params: {} 2023-10-11T15:35:46.505 DEBUG:boto:Token: None 2023-10-11T15:35:46.505 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.506 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zNtZn74yVR9S8jydlBckoAs0XQs= 2023-10-11T15:35:46.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zNtZn74yVR9S8jydlBckoAs0XQs=', 'Content-Length': '0'} 2023-10-11T15:35:46.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8d00bd3e24a431-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:35:46.509 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.510 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.510 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.510 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.511 DEBUG:boto:Method: GET 2023-10-11T15:35:46.511 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.512 DEBUG:boto:Data: 2023-10-11T15:35:46.512 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.513 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.513 DEBUG:boto:Params: {} 2023-10-11T15:35:46.514 DEBUG:boto:Token: None 2023-10-11T15:35:46.514 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.515 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI= 2023-10-11T15:35:46.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI=', 'Content-Length': '0'} 2023-10-11T15:35:46.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c282b63f1d822c-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.518 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.519 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.519 DEBUG:boto:path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.519 DEBUG:boto:auth_path=/dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.520 DEBUG:boto:Method: GET 2023-10-11T15:35:46.520 DEBUG:boto:Path: /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.521 DEBUG:boto:Data: 2023-10-11T15:35:46.521 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.522 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.522 DEBUG:boto:Params: {} 2023-10-11T15:35:46.523 DEBUG:boto:Token: None 2023-10-11T15:35:46.523 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111?versionId=null 2023-10-11T15:35:46.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI= 2023-10-11T15:35:46.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1GY8aQRe6ymYPhxJmJ/tzgrOOI=', 'Content-Length': '0'} 2023-10-11T15:35:46.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d9c2fbc1ba5a5d-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.574 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.575 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.576 DEBUG:boto:path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.576 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.577 DEBUG:boto:Method: GET 2023-10-11T15:35:46.577 DEBUG:boto:Path: /dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.577 DEBUG:boto:Data: 2023-10-11T15:35:46.578 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.579 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.579 DEBUG:boto:Params: {} 2023-10-11T15:35:46.580 DEBUG:boto:Token: None 2023-10-11T15:35:46.580 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.581 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5ad20e99be5c789-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.584 DEBUG:boto:b'dghqvd-61obj1111000falseobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.585 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.585 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.586 DEBUG:boto:path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.586 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.587 DEBUG:boto:Method: GET 2023-10-11T15:35:46.587 DEBUG:boto:Path: /dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.587 DEBUG:boto:Data: 2023-10-11T15:35:46.588 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.588 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.589 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.589 DEBUG:boto:Params: {} 2023-10-11T15:35:46.590 DEBUG:boto:Token: None 2023-10-11T15:35:46.590 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006579a0b492668201-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.595 DEBUG:boto:b'dghqvd-61obj1111000falseobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.596 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.596 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.597 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.597 DEBUG:boto:Path: /dghqvd-61/obj111 2023-10-11T15:35:46.598 DEBUG:boto:Data: 2023-10-11T15:35:46.598 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.599 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.599 DEBUG:boto:Params: {} 2023-10-11T15:35:46.600 DEBUG:boto:Token: None 2023-10-11T15:35:46.600 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111 2023-10-11T15:35:46.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L5ezIGeK+0D/ubTSmNTkjyzPfm8= 2023-10-11T15:35:46.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L5ezIGeK+0D/ubTSmNTkjyzPfm8=', 'Content-Length': '0'} 2023-10-11T15:35:46.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cff6cfe369c2cdf-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.603 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.604 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.604 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.605 DEBUG:boto:Path: /dghqvd-61/obj111 2023-10-11T15:35:46.605 DEBUG:boto:Data: 2023-10-11T15:35:46.605 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.606 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.606 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.607 DEBUG:boto:Params: {} 2023-10-11T15:35:46.607 DEBUG:boto:Token: None 2023-10-11T15:35:46.608 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111 2023-10-11T15:35:46.608 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L5ezIGeK+0D/ubTSmNTkjyzPfm8= 2023-10-11T15:35:46.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L5ezIGeK+0D/ubTSmNTkjyzPfm8=', 'Content-Length': '0'} 2023-10-11T15:35:46.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435d6be34737a44f-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-10-11T15:35:46.611 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.612 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.612 DEBUG:boto:Method: GET 2023-10-11T15:35:46.613 DEBUG:boto:Path: /dghqvd-61/obj111 2023-10-11T15:35:46.613 DEBUG:boto:Data: 2023-10-11T15:35:46.614 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.614 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.615 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.615 DEBUG:boto:Params: {} 2023-10-11T15:35:46.615 DEBUG:boto:Token: None 2023-10-11T15:35:46.616 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111 2023-10-11T15:35:46.616 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ne17E696zh6oPwrEGufRM2/Andg= 2023-10-11T15:35:46.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ne17E696zh6oPwrEGufRM2/Andg=', 'Content-Length': '0'} 2023-10-11T15:35:46.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060268343804f26e1-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.662 DEBUG:boto:path=/dghqvd-61/obj111 2023-10-11T15:35:46.663 DEBUG:boto:auth_path=/dghqvd-61/obj111 2023-10-11T15:35:46.663 DEBUG:boto:Method: GET 2023-10-11T15:35:46.664 DEBUG:boto:Path: /dghqvd-61/obj111 2023-10-11T15:35:46.664 DEBUG:boto:Data: 2023-10-11T15:35:46.665 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.665 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.666 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.666 DEBUG:boto:Params: {} 2023-10-11T15:35:46.667 DEBUG:boto:Token: None 2023-10-11T15:35:46.667 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj111 2023-10-11T15:35:46.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ne17E696zh6oPwrEGufRM2/Andg= 2023-10-11T15:35:46.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ne17E696zh6oPwrEGufRM2/Andg=', 'Content-Length': '0'} 2023-10-11T15:35:46.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8aa372d5b72b433-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.671 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.671 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.672 DEBUG:boto:path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.672 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.673 DEBUG:boto:Method: GET 2023-10-11T15:35:46.673 DEBUG:boto:Path: /dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.674 DEBUG:boto:Data: 2023-10-11T15:35:46.674 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.675 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.675 DEBUG:boto:Params: {} 2023-10-11T15:35:46.676 DEBUG:boto:Token: None 2023-10-11T15:35:46.676 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.677 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16c89b1b3d95b2b-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.680 DEBUG:boto:b'dghqvd-61obj1111000falseobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.681 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.681 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.682 DEBUG:boto:path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.682 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.683 DEBUG:boto:Method: GET 2023-10-11T15:35:46.683 DEBUG:boto:Path: /dghqvd-61/?prefix=obj111 2023-10-11T15:35:46.684 DEBUG:boto:Data: 2023-10-11T15:35:46.684 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.684 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.685 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.685 DEBUG:boto:Params: {} 2023-10-11T15:35:46.686 DEBUG:boto:Token: None 2023-10-11T15:35:46.686 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.687 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000666f18e075d9c3b5-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.738 DEBUG:boto:b'dghqvd-61obj1111000falseobj1112023-10-11T15:31:44.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:35:46.739 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.740 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.740 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.741 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.741 DEBUG:boto:Method: GET 2023-10-11T15:35:46.742 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.742 DEBUG:boto:Data: 2023-10-11T15:35:46.742 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.743 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.743 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.744 DEBUG:boto:Params: {} 2023-10-11T15:35:46.744 DEBUG:boto:Token: None 2023-10-11T15:35:46.744 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.745 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ= 2023-10-11T15:35:46.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ=', 'Content-Length': '0'} 2023-10-11T15:35:46.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000297058ff78052628-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.748 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.749 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.749 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.749 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.750 DEBUG:boto:Method: GET 2023-10-11T15:35:46.750 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.751 DEBUG:boto:Data: 2023-10-11T15:35:46.751 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.752 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.752 DEBUG:boto:Params: {} 2023-10-11T15:35:46.753 DEBUG:boto:Token: None 2023-10-11T15:35:46.753 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.754 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ= 2023-10-11T15:35:46.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ=', 'Content-Length': '0'} 2023-10-11T15:35:46.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559de06e897cc9db-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.756 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.757 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.757 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.758 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.758 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.759 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.759 DEBUG:boto:Data: 2023-10-11T15:35:46.759 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.760 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.760 DEBUG:boto:Params: {} 2023-10-11T15:35:46.761 DEBUG:boto:Token: None 2023-10-11T15:35:46.761 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HR4xxuJOZpgeQ6/vkxqmQ6PoXbg= 2023-10-11T15:35:46.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HR4xxuJOZpgeQ6/vkxqmQ6PoXbg=', 'Content-Length': '0'} 2023-10-11T15:35:46.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b110e102d20f018-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.765 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.765 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.766 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.766 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.766 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.767 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.767 DEBUG:boto:Data: 2023-10-11T15:35:46.768 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.769 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.769 DEBUG:boto:Params: {} 2023-10-11T15:35:46.769 DEBUG:boto:Token: None 2023-10-11T15:35:46.770 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.770 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HR4xxuJOZpgeQ6/vkxqmQ6PoXbg= 2023-10-11T15:35:46.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HR4xxuJOZpgeQ6/vkxqmQ6PoXbg=', 'Content-Length': '0'} 2023-10-11T15:35:46.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0d00d903320a9e-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:35:46.774 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.774 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.774 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.775 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.775 DEBUG:boto:Method: GET 2023-10-11T15:35:46.776 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.776 DEBUG:boto:Data: 2023-10-11T15:35:46.776 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.777 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.778 DEBUG:boto:Params: {} 2023-10-11T15:35:46.778 DEBUG:boto:Token: None 2023-10-11T15:35:46.779 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ= 2023-10-11T15:35:46.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ=', 'Content-Length': '0'} 2023-10-11T15:35:46.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021711b6b8bba952c-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.782 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.783 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.783 DEBUG:boto:path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.784 DEBUG:boto:auth_path=/dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.784 DEBUG:boto:Method: GET 2023-10-11T15:35:46.784 DEBUG:boto:Path: /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.785 DEBUG:boto:Data: 2023-10-11T15:35:46.785 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.786 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.786 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.786 DEBUG:boto:Params: {} 2023-10-11T15:35:46.787 DEBUG:boto:Token: None 2023-10-11T15:35:46.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115?versionId=null 2023-10-11T15:35:46.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ= 2023-10-11T15:35:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PE9iMh7H+u/JwSq0l13P5o43fdQ=', 'Content-Length': '0'} 2023-10-11T15:35:46.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d173bfb9c6950119-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.791 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.791 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.792 DEBUG:boto:path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.792 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.793 DEBUG:boto:Method: GET 2023-10-11T15:35:46.793 DEBUG:boto:Path: /dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.793 DEBUG:boto:Data: 2023-10-11T15:35:46.794 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.795 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.795 DEBUG:boto:Params: {} 2023-10-11T15:35:46.795 DEBUG:boto:Token: None 2023-10-11T15:35:46.796 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec741b21c3818ca-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.799 DEBUG:boto:b'dghqvd-61obj1151000falseobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.800 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.800 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.801 DEBUG:boto:path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.801 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.802 DEBUG:boto:Method: GET 2023-10-11T15:35:46.802 DEBUG:boto:Path: /dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.802 DEBUG:boto:Data: 2023-10-11T15:35:46.803 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.803 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.804 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.804 DEBUG:boto:Params: {} 2023-10-11T15:35:46.805 DEBUG:boto:Token: None 2023-10-11T15:35:46.805 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7d23d983860180-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.810 DEBUG:boto:b'dghqvd-61obj1151000falseobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.811 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.811 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.812 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.812 DEBUG:boto:Path: /dghqvd-61/obj115 2023-10-11T15:35:46.812 DEBUG:boto:Data: 2023-10-11T15:35:46.813 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.813 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.814 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.814 DEBUG:boto:Params: {} 2023-10-11T15:35:46.815 DEBUG:boto:Token: None 2023-10-11T15:35:46.815 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115 2023-10-11T15:35:46.815 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JTxEY47LNgZEcd2UCJXog+l+3ew= 2023-10-11T15:35:46.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JTxEY47LNgZEcd2UCJXog+l+3ew=', 'Content-Length': '0'} 2023-10-11T15:35:46.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfd0d57a8fb4a71-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.819 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.819 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.819 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.820 DEBUG:boto:Path: /dghqvd-61/obj115 2023-10-11T15:35:46.820 DEBUG:boto:Data: 2023-10-11T15:35:46.821 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.821 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.821 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.822 DEBUG:boto:Params: {} 2023-10-11T15:35:46.822 DEBUG:boto:Token: None 2023-10-11T15:35:46.823 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115 2023-10-11T15:35:46.823 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JTxEY47LNgZEcd2UCJXog+l+3ew= 2023-10-11T15:35:46.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JTxEY47LNgZEcd2UCJXog+l+3ew=', 'Content-Length': '0'} 2023-10-11T15:35:46.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb930a2e56e48f0-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-10-11T15:35:46.826 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.827 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.827 DEBUG:boto:Method: GET 2023-10-11T15:35:46.828 DEBUG:boto:Path: /dghqvd-61/obj115 2023-10-11T15:35:46.828 DEBUG:boto:Data: 2023-10-11T15:35:46.828 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.829 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.829 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.830 DEBUG:boto:Params: {} 2023-10-11T15:35:46.830 DEBUG:boto:Token: None 2023-10-11T15:35:46.831 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115 2023-10-11T15:35:46.831 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D+i+X0p34U2HmWQ4kXF4lCfqFh4= 2023-10-11T15:35:46.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D+i+X0p34U2HmWQ4kXF4lCfqFh4=', 'Content-Length': '0'} 2023-10-11T15:35:46.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0db3db12ff5e5e-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.834 DEBUG:boto:path=/dghqvd-61/obj115 2023-10-11T15:35:46.835 DEBUG:boto:auth_path=/dghqvd-61/obj115 2023-10-11T15:35:46.835 DEBUG:boto:Method: GET 2023-10-11T15:35:46.836 DEBUG:boto:Path: /dghqvd-61/obj115 2023-10-11T15:35:46.836 DEBUG:boto:Data: 2023-10-11T15:35:46.836 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.837 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.838 DEBUG:boto:Params: {} 2023-10-11T15:35:46.838 DEBUG:boto:Token: None 2023-10-11T15:35:46.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj115 2023-10-11T15:35:46.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D+i+X0p34U2HmWQ4kXF4lCfqFh4= 2023-10-11T15:35:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D+i+X0p34U2HmWQ4kXF4lCfqFh4=', 'Content-Length': '0'} 2023-10-11T15:35:46.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022fb84e4d1b5feba-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.842 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.842 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.843 DEBUG:boto:path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.843 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.844 DEBUG:boto:Method: GET 2023-10-11T15:35:46.844 DEBUG:boto:Path: /dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.845 DEBUG:boto:Data: 2023-10-11T15:35:46.845 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.845 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.846 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.846 DEBUG:boto:Params: {} 2023-10-11T15:35:46.847 DEBUG:boto:Token: None 2023-10-11T15:35:46.847 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.848 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0355d015e07baee-006526c0d2-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.851 DEBUG:boto:b'dghqvd-61obj1151000falseobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.854 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:46.854 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:46.855 DEBUG:boto:path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.855 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.855 DEBUG:boto:Method: GET 2023-10-11T15:35:46.856 DEBUG:boto:Path: /dghqvd-61/?prefix=obj115 2023-10-11T15:35:46.856 DEBUG:boto:Data: 2023-10-11T15:35:46.857 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.857 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.858 DEBUG:boto:Params: {} 2023-10-11T15:35:46.858 DEBUG:boto:Token: None 2023-10-11T15:35:46.859 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/ 2023-10-11T15:35:46.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58= 2023-10-11T15:35:46.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9u4VD9gAULHoiJT7PMVG1i/tq58=', 'Content-Length': '0'} 2023-10-11T15:35:46.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afe0772e33e1bee3-006526c0d2-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.864 DEBUG:boto:b'dghqvd-61obj1151000falseobj1152023-10-11T15:31:43.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:46.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:35:46.865 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.866 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.866 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.867 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.867 DEBUG:boto:Method: GET 2023-10-11T15:35:46.868 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.868 DEBUG:boto:Data: 2023-10-11T15:35:46.868 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.869 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.870 DEBUG:boto:Params: {} 2023-10-11T15:35:46.870 DEBUG:boto:Token: None 2023-10-11T15:35:46.871 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE= 2023-10-11T15:35:46.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE=', 'Content-Length': '0'} 2023-10-11T15:35:46.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff56f839506a0d7-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.918 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.919 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.919 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.920 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.920 DEBUG:boto:Method: GET 2023-10-11T15:35:46.921 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.921 DEBUG:boto:Data: 2023-10-11T15:35:46.922 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.922 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.923 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.923 DEBUG:boto:Params: {} 2023-10-11T15:35:46.923 DEBUG:boto:Token: None 2023-10-11T15:35:46.924 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.924 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE= 2023-10-11T15:35:46.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE=', 'Content-Length': '0'} 2023-10-11T15:35:46.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003779d725e2515f2f-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.928 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.928 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.928 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.929 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.929 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.930 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.930 DEBUG:boto:Data: 2023-10-11T15:35:46.930 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.931 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.931 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.932 DEBUG:boto:Params: {} 2023-10-11T15:35:46.932 DEBUG:boto:Token: None 2023-10-11T15:35:46.933 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7v8/3mdOkUlXv4DuiF+N/GxZTao= 2023-10-11T15:35:46.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7v8/3mdOkUlXv4DuiF+N/GxZTao=', 'Content-Length': '0'} 2023-10-11T15:35:46.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73920fe72693251-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.936 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.936 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.937 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.937 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.937 DEBUG:boto:Method: HEAD 2023-10-11T15:35:46.938 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.938 DEBUG:boto:Data: 2023-10-11T15:35:46.939 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.939 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.940 DEBUG:boto:Params: {} 2023-10-11T15:35:46.940 DEBUG:boto:Token: None 2023-10-11T15:35:46.941 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.941 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7v8/3mdOkUlXv4DuiF+N/GxZTao= 2023-10-11T15:35:46.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7v8/3mdOkUlXv4DuiF+N/GxZTao=', 'Content-Length': '0'} 2023-10-11T15:35:46.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccf98e7af415aad-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:35:46.945 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.945 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.945 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.946 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.946 DEBUG:boto:Method: GET 2023-10-11T15:35:46.947 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.947 DEBUG:boto:Data: 2023-10-11T15:35:46.948 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:46.948 DEBUG:boto:Port: 8001 2023-10-11T15:35:46.949 DEBUG:boto:Params: {} 2023-10-11T15:35:46.949 DEBUG:boto:Token: None 2023-10-11T15:35:46.950 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE= 2023-10-11T15:35:46.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE=', 'Content-Length': '0'} 2023-10-11T15:35:46.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058074179289b7a83-006526c0d2-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:46.953 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:46.954 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:46.954 DEBUG:boto:path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.955 DEBUG:boto:auth_path=/dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.955 DEBUG:boto:Method: GET 2023-10-11T15:35:46.956 DEBUG:boto:Path: /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.956 DEBUG:boto:Data: 2023-10-11T15:35:46.956 DEBUG:boto:Headers: {} 2023-10-11T15:35:46.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:46.957 DEBUG:boto:Port: 8000 2023-10-11T15:35:46.958 DEBUG:boto:Params: {} 2023-10-11T15:35:46.958 DEBUG:boto:Token: None 2023-10-11T15:35:46.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:46 GMT /dghqvd-61/obj116?versionId=null 2023-10-11T15:35:46.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE= 2023-10-11T15:35:46.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oqCdCWpiKJhLfT9s+9XTuza1yvE=', 'Content-Length': '0'} 2023-10-11T15:35:46.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006992f656b88b2d09-006526c0d2-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.002 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.003 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.003 DEBUG:boto:path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.004 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.004 DEBUG:boto:Method: GET 2023-10-11T15:35:47.005 DEBUG:boto:Path: /dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.005 DEBUG:boto:Data: 2023-10-11T15:35:47.006 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.006 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.006 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.007 DEBUG:boto:Params: {} 2023-10-11T15:35:47.007 DEBUG:boto:Token: None 2023-10-11T15:35:47.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc522e58fd7cc5ad-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.012 DEBUG:boto:b'dghqvd-61obj1161000falseobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.013 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.013 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.014 DEBUG:boto:path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.014 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.015 DEBUG:boto:Method: GET 2023-10-11T15:35:47.015 DEBUG:boto:Path: /dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.015 DEBUG:boto:Data: 2023-10-11T15:35:47.016 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.017 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.017 DEBUG:boto:Params: {} 2023-10-11T15:35:47.018 DEBUG:boto:Token: None 2023-10-11T15:35:47.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006017bc0fec1d099-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.024 DEBUG:boto:b'dghqvd-61obj1161000falseobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.025 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:47.025 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:47.025 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.026 DEBUG:boto:Path: /dghqvd-61/obj116 2023-10-11T15:35:47.026 DEBUG:boto:Data: 2023-10-11T15:35:47.027 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.027 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.028 DEBUG:boto:Params: {} 2023-10-11T15:35:47.028 DEBUG:boto:Token: None 2023-10-11T15:35:47.029 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj116 2023-10-11T15:35:47.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sYB2T+aIIDSmKTKU19TIxhhYs6I= 2023-10-11T15:35:47.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sYB2T+aIIDSmKTKU19TIxhhYs6I=', 'Content-Length': '0'} 2023-10-11T15:35:47.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f0875beccea087-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.032 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:47.033 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:47.033 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.034 DEBUG:boto:Path: /dghqvd-61/obj116 2023-10-11T15:35:47.034 DEBUG:boto:Data: 2023-10-11T15:35:47.034 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.035 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.035 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.036 DEBUG:boto:Params: {} 2023-10-11T15:35:47.036 DEBUG:boto:Token: None 2023-10-11T15:35:47.036 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj116 2023-10-11T15:35:47.037 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sYB2T+aIIDSmKTKU19TIxhhYs6I= 2023-10-11T15:35:47.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sYB2T+aIIDSmKTKU19TIxhhYs6I=', 'Content-Length': '0'} 2023-10-11T15:35:47.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caad4a2b430ed1cf-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-10-11T15:35:47.040 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:47.041 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:47.041 DEBUG:boto:Method: GET 2023-10-11T15:35:47.042 DEBUG:boto:Path: /dghqvd-61/obj116 2023-10-11T15:35:47.042 DEBUG:boto:Data: 2023-10-11T15:35:47.043 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.043 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.044 DEBUG:boto:Params: {} 2023-10-11T15:35:47.044 DEBUG:boto:Token: None 2023-10-11T15:35:47.045 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj116 2023-10-11T15:35:47.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n+W6DrmMsa7R0gApjExKxy71Btg= 2023-10-11T15:35:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n+W6DrmMsa7R0gApjExKxy71Btg=', 'Content-Length': '0'} 2023-10-11T15:35:47.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000542c7e23196b45f-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.049 DEBUG:boto:path=/dghqvd-61/obj116 2023-10-11T15:35:47.049 DEBUG:boto:auth_path=/dghqvd-61/obj116 2023-10-11T15:35:47.050 DEBUG:boto:Method: GET 2023-10-11T15:35:47.050 DEBUG:boto:Path: /dghqvd-61/obj116 2023-10-11T15:35:47.050 DEBUG:boto:Data: 2023-10-11T15:35:47.051 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.051 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.052 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.052 DEBUG:boto:Params: {} 2023-10-11T15:35:47.053 DEBUG:boto:Token: None 2023-10-11T15:35:47.053 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj116 2023-10-11T15:35:47.054 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n+W6DrmMsa7R0gApjExKxy71Btg= 2023-10-11T15:35:47.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n+W6DrmMsa7R0gApjExKxy71Btg=', 'Content-Length': '0'} 2023-10-11T15:35:47.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408300372efb7daa-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.058 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.058 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.059 DEBUG:boto:path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.059 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.060 DEBUG:boto:Method: GET 2023-10-11T15:35:47.060 DEBUG:boto:Path: /dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.060 DEBUG:boto:Data: 2023-10-11T15:35:47.061 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.061 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.062 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.062 DEBUG:boto:Params: {} 2023-10-11T15:35:47.063 DEBUG:boto:Token: None 2023-10-11T15:35:47.063 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.064 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038c3d1311ca748af-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.067 DEBUG:boto:b'dghqvd-61obj1161000falseobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.068 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.068 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.069 DEBUG:boto:path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.069 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.070 DEBUG:boto:Method: GET 2023-10-11T15:35:47.070 DEBUG:boto:Path: /dghqvd-61/?prefix=obj116 2023-10-11T15:35:47.071 DEBUG:boto:Data: 2023-10-11T15:35:47.071 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.071 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.072 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.072 DEBUG:boto:Params: {} 2023-10-11T15:35:47.073 DEBUG:boto:Token: None 2023-10-11T15:35:47.073 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.074 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033538eb818302121-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.079 DEBUG:boto:b'dghqvd-61obj1161000falseobj1162023-10-11T15:31:43.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:35:47.080 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.080 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.081 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.081 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.081 DEBUG:boto:Method: GET 2023-10-11T15:35:47.082 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.082 DEBUG:boto:Data: 2023-10-11T15:35:47.083 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.083 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.084 DEBUG:boto:Params: {} 2023-10-11T15:35:47.084 DEBUG:boto:Token: None 2023-10-11T15:35:47.085 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w= 2023-10-11T15:35:47.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w=', 'Content-Length': '0'} 2023-10-11T15:35:47.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44f44be8f1b16e3-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.134 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.135 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.135 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.136 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.136 DEBUG:boto:Method: GET 2023-10-11T15:35:47.137 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.137 DEBUG:boto:Data: 2023-10-11T15:35:47.138 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.138 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.138 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.139 DEBUG:boto:Params: {} 2023-10-11T15:35:47.139 DEBUG:boto:Token: None 2023-10-11T15:35:47.140 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w= 2023-10-11T15:35:47.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w=', 'Content-Length': '0'} 2023-10-11T15:35:47.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000841935549595faf7-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.143 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.144 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.144 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.145 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.145 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.145 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.146 DEBUG:boto:Data: 2023-10-11T15:35:47.146 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.147 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.148 DEBUG:boto:Params: {} 2023-10-11T15:35:47.148 DEBUG:boto:Token: None 2023-10-11T15:35:47.149 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.149 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0trTz1AJQa5TbBSyVkqEtMYOyU4= 2023-10-11T15:35:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0trTz1AJQa5TbBSyVkqEtMYOyU4=', 'Content-Length': '0'} 2023-10-11T15:35:47.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e70cdc605e934b3-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.152 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.152 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.153 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.153 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.154 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.154 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.155 DEBUG:boto:Data: 2023-10-11T15:35:47.155 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.155 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.156 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.156 DEBUG:boto:Params: {} 2023-10-11T15:35:47.157 DEBUG:boto:Token: None 2023-10-11T15:35:47.157 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0trTz1AJQa5TbBSyVkqEtMYOyU4= 2023-10-11T15:35:47.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0trTz1AJQa5TbBSyVkqEtMYOyU4=', 'Content-Length': '0'} 2023-10-11T15:35:47.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003216f85fbbbda26d-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:35:47.161 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.161 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.162 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.162 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.163 DEBUG:boto:Method: GET 2023-10-11T15:35:47.163 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.163 DEBUG:boto:Data: 2023-10-11T15:35:47.164 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.164 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.165 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.165 DEBUG:boto:Params: {} 2023-10-11T15:35:47.166 DEBUG:boto:Token: None 2023-10-11T15:35:47.166 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.167 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w= 2023-10-11T15:35:47.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w=', 'Content-Length': '0'} 2023-10-11T15:35:47.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f92f6dbd4cf5839-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.169 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.170 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.170 DEBUG:boto:path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.171 DEBUG:boto:auth_path=/dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.171 DEBUG:boto:Method: GET 2023-10-11T15:35:47.172 DEBUG:boto:Path: /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.172 DEBUG:boto:Data: 2023-10-11T15:35:47.172 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.173 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.174 DEBUG:boto:Params: {} 2023-10-11T15:35:47.174 DEBUG:boto:Token: None 2023-10-11T15:35:47.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124?versionId=null 2023-10-11T15:35:47.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w= 2023-10-11T15:35:47.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pmu43u2hB7Tj/IYEPtR7GX+wZ8w=', 'Content-Length': '0'} 2023-10-11T15:35:47.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c106f70d5c72673d-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.222 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.223 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.223 DEBUG:boto:path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.224 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.224 DEBUG:boto:Method: GET 2023-10-11T15:35:47.225 DEBUG:boto:Path: /dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.225 DEBUG:boto:Data: 2023-10-11T15:35:47.225 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.226 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.227 DEBUG:boto:Params: {} 2023-10-11T15:35:47.227 DEBUG:boto:Token: None 2023-10-11T15:35:47.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd139e0db18fa92e-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.232 DEBUG:boto:b'dghqvd-61obj1241000falseobj1242023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.232 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.233 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.233 DEBUG:boto:path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.234 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.234 DEBUG:boto:Method: GET 2023-10-11T15:35:47.235 DEBUG:boto:Path: /dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.235 DEBUG:boto:Data: 2023-10-11T15:35:47.235 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.236 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.237 DEBUG:boto:Params: {} 2023-10-11T15:35:47.237 DEBUG:boto:Token: None 2023-10-11T15:35:47.238 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.238 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002681606755bb3d9-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.243 DEBUG:boto:b'dghqvd-61obj1241000falseobj1242023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.243 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.244 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.244 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.245 DEBUG:boto:Path: /dghqvd-61/obj124 2023-10-11T15:35:47.245 DEBUG:boto:Data: 2023-10-11T15:35:47.246 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.246 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.246 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.247 DEBUG:boto:Params: {} 2023-10-11T15:35:47.247 DEBUG:boto:Token: None 2023-10-11T15:35:47.248 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124 2023-10-11T15:35:47.248 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7OI9NdEaIffCRz+MubhDi/+JPVg= 2023-10-11T15:35:47.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7OI9NdEaIffCRz+MubhDi/+JPVg=', 'Content-Length': '0'} 2023-10-11T15:35:47.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab1c918314e14cd-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.251 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.251 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.252 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.252 DEBUG:boto:Path: /dghqvd-61/obj124 2023-10-11T15:35:47.252 DEBUG:boto:Data: 2023-10-11T15:35:47.253 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.253 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.254 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.254 DEBUG:boto:Params: {} 2023-10-11T15:35:47.255 DEBUG:boto:Token: None 2023-10-11T15:35:47.255 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124 2023-10-11T15:35:47.255 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7OI9NdEaIffCRz+MubhDi/+JPVg= 2023-10-11T15:35:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7OI9NdEaIffCRz+MubhDi/+JPVg=', 'Content-Length': '0'} 2023-10-11T15:35:47.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e14babb0db4b0d-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-10-11T15:35:47.259 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.259 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.260 DEBUG:boto:Method: GET 2023-10-11T15:35:47.260 DEBUG:boto:Path: /dghqvd-61/obj124 2023-10-11T15:35:47.260 DEBUG:boto:Data: 2023-10-11T15:35:47.261 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.261 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.262 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.262 DEBUG:boto:Params: {} 2023-10-11T15:35:47.263 DEBUG:boto:Token: None 2023-10-11T15:35:47.263 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124 2023-10-11T15:35:47.263 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cgpXlZSWtwErVlrnDf0s0cgzIyo= 2023-10-11T15:35:47.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cgpXlZSWtwErVlrnDf0s0cgzIyo=', 'Content-Length': '0'} 2023-10-11T15:35:47.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f187194ff317c35-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.310 DEBUG:boto:path=/dghqvd-61/obj124 2023-10-11T15:35:47.311 DEBUG:boto:auth_path=/dghqvd-61/obj124 2023-10-11T15:35:47.312 DEBUG:boto:Method: GET 2023-10-11T15:35:47.312 DEBUG:boto:Path: /dghqvd-61/obj124 2023-10-11T15:35:47.312 DEBUG:boto:Data: 2023-10-11T15:35:47.313 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.313 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.314 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.314 DEBUG:boto:Params: {} 2023-10-11T15:35:47.315 DEBUG:boto:Token: None 2023-10-11T15:35:47.315 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj124 2023-10-11T15:35:47.316 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cgpXlZSWtwErVlrnDf0s0cgzIyo= 2023-10-11T15:35:47.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cgpXlZSWtwErVlrnDf0s0cgzIyo=', 'Content-Length': '0'} 2023-10-11T15:35:47.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1209a0e226faaa-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.319 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.320 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.320 DEBUG:boto:path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.320 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.321 DEBUG:boto:Method: GET 2023-10-11T15:35:47.321 DEBUG:boto:Path: /dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.322 DEBUG:boto:Data: 2023-10-11T15:35:47.322 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.323 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.323 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.323 DEBUG:boto:Params: {} 2023-10-11T15:35:47.324 DEBUG:boto:Token: None 2023-10-11T15:35:47.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.325 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005554707ddffa5c8b-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.328 DEBUG:boto:b'dghqvd-61obj1241000falseobj1242023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.329 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.329 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.330 DEBUG:boto:path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.330 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.331 DEBUG:boto:Method: GET 2023-10-11T15:35:47.331 DEBUG:boto:Path: /dghqvd-61/?prefix=obj124 2023-10-11T15:35:47.332 DEBUG:boto:Data: 2023-10-11T15:35:47.332 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.332 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.333 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.333 DEBUG:boto:Params: {} 2023-10-11T15:35:47.334 DEBUG:boto:Token: None 2023-10-11T15:35:47.334 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.335 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc25102159fa9c5c-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.382 DEBUG:boto:b'dghqvd-61obj1241000falseobj1242023-10-11T15:31:44.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:35:47.384 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.385 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.385 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.386 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.386 DEBUG:boto:Method: GET 2023-10-11T15:35:47.387 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.387 DEBUG:boto:Data: 2023-10-11T15:35:47.387 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.388 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.389 DEBUG:boto:Params: {} 2023-10-11T15:35:47.389 DEBUG:boto:Token: None 2023-10-11T15:35:47.390 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.390 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg= 2023-10-11T15:35:47.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg=', 'Content-Length': '0'} 2023-10-11T15:35:47.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a3a166b2e5ba30-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.394 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.394 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.395 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.395 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.396 DEBUG:boto:Method: GET 2023-10-11T15:35:47.396 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.397 DEBUG:boto:Data: 2023-10-11T15:35:47.397 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.398 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.399 DEBUG:boto:Params: {} 2023-10-11T15:35:47.399 DEBUG:boto:Token: None 2023-10-11T15:35:47.399 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg= 2023-10-11T15:35:47.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg=', 'Content-Length': '0'} 2023-10-11T15:35:47.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060617e72bfe4cfb3-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.403 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.404 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.404 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.404 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.405 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.405 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.406 DEBUG:boto:Data: 2023-10-11T15:35:47.406 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.407 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.407 DEBUG:boto:Params: {} 2023-10-11T15:35:47.408 DEBUG:boto:Token: None 2023-10-11T15:35:47.408 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9hT8pY8xlVbPC/RiPrlshGo/GK8= 2023-10-11T15:35:47.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9hT8pY8xlVbPC/RiPrlshGo/GK8=', 'Content-Length': '0'} 2023-10-11T15:35:47.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440ef5a95409ca41-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.412 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.413 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.413 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.413 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.414 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.414 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.415 DEBUG:boto:Data: 2023-10-11T15:35:47.415 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.417 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.417 DEBUG:boto:Params: {} 2023-10-11T15:35:47.417 DEBUG:boto:Token: None 2023-10-11T15:35:47.418 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9hT8pY8xlVbPC/RiPrlshGo/GK8= 2023-10-11T15:35:47.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9hT8pY8xlVbPC/RiPrlshGo/GK8=', 'Content-Length': '0'} 2023-10-11T15:35:47.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c10ec14a5b516a3-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:35:47.422 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.422 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.423 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.423 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.424 DEBUG:boto:Method: GET 2023-10-11T15:35:47.424 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.425 DEBUG:boto:Data: 2023-10-11T15:35:47.425 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.425 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.426 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.426 DEBUG:boto:Params: {} 2023-10-11T15:35:47.427 DEBUG:boto:Token: None 2023-10-11T15:35:47.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg= 2023-10-11T15:35:47.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg=', 'Content-Length': '0'} 2023-10-11T15:35:47.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa50a83c81910bc-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.431 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.431 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.432 DEBUG:boto:path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.432 DEBUG:boto:auth_path=/dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.433 DEBUG:boto:Method: GET 2023-10-11T15:35:47.433 DEBUG:boto:Path: /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.433 DEBUG:boto:Data: 2023-10-11T15:35:47.434 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.435 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.435 DEBUG:boto:Params: {} 2023-10-11T15:35:47.436 DEBUG:boto:Token: None 2023-10-11T15:35:47.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145?versionId=null 2023-10-11T15:35:47.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg= 2023-10-11T15:35:47.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pfaQ27GjNeiKfiYUU0BEuyGP0Fg=', 'Content-Length': '0'} 2023-10-11T15:35:47.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008baa7be01cadc98d-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.439 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.440 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.440 DEBUG:boto:path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.441 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.441 DEBUG:boto:Method: GET 2023-10-11T15:35:47.442 DEBUG:boto:Path: /dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.442 DEBUG:boto:Data: 2023-10-11T15:35:47.442 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.443 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.443 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.444 DEBUG:boto:Params: {} 2023-10-11T15:35:47.444 DEBUG:boto:Token: None 2023-10-11T15:35:47.445 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.445 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a3182cac83dba23-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.449 DEBUG:boto:b'dghqvd-61obj1451000falseobj1452023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.449 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.450 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.450 DEBUG:boto:path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.451 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.451 DEBUG:boto:Method: GET 2023-10-11T15:35:47.451 DEBUG:boto:Path: /dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.452 DEBUG:boto:Data: 2023-10-11T15:35:47.452 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.453 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.453 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.454 DEBUG:boto:Params: {} 2023-10-11T15:35:47.454 DEBUG:boto:Token: None 2023-10-11T15:35:47.455 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbf9d257ad175ce-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.502 DEBUG:boto:b'dghqvd-61obj1451000falseobj1452023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.503 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.503 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.504 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.504 DEBUG:boto:Path: /dghqvd-61/obj145 2023-10-11T15:35:47.505 DEBUG:boto:Data: 2023-10-11T15:35:47.505 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.506 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.506 DEBUG:boto:Params: {} 2023-10-11T15:35:47.507 DEBUG:boto:Token: None 2023-10-11T15:35:47.507 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145 2023-10-11T15:35:47.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cbmR6dWfW5YzSz9r2RjanbtTA1E= 2023-10-11T15:35:47.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cbmR6dWfW5YzSz9r2RjanbtTA1E=', 'Content-Length': '0'} 2023-10-11T15:35:47.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a4328e51564409-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.511 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.511 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.512 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.512 DEBUG:boto:Path: /dghqvd-61/obj145 2023-10-11T15:35:47.512 DEBUG:boto:Data: 2023-10-11T15:35:47.513 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.513 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.514 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.514 DEBUG:boto:Params: {} 2023-10-11T15:35:47.515 DEBUG:boto:Token: None 2023-10-11T15:35:47.515 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145 2023-10-11T15:35:47.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cbmR6dWfW5YzSz9r2RjanbtTA1E= 2023-10-11T15:35:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cbmR6dWfW5YzSz9r2RjanbtTA1E=', 'Content-Length': '0'} 2023-10-11T15:35:47.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015538ddfebc4d30-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-10-11T15:35:47.519 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.519 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.520 DEBUG:boto:Method: GET 2023-10-11T15:35:47.520 DEBUG:boto:Path: /dghqvd-61/obj145 2023-10-11T15:35:47.520 DEBUG:boto:Data: 2023-10-11T15:35:47.521 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.521 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.522 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.522 DEBUG:boto:Params: {} 2023-10-11T15:35:47.523 DEBUG:boto:Token: None 2023-10-11T15:35:47.523 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145 2023-10-11T15:35:47.524 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TUzbEKVydId0q3PQyY95KEiDM2w= 2023-10-11T15:35:47.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TUzbEKVydId0q3PQyY95KEiDM2w=', 'Content-Length': '0'} 2023-10-11T15:35:47.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000841a3c831c8ed9e9-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.526 DEBUG:boto:path=/dghqvd-61/obj145 2023-10-11T15:35:47.527 DEBUG:boto:auth_path=/dghqvd-61/obj145 2023-10-11T15:35:47.527 DEBUG:boto:Method: GET 2023-10-11T15:35:47.528 DEBUG:boto:Path: /dghqvd-61/obj145 2023-10-11T15:35:47.528 DEBUG:boto:Data: 2023-10-11T15:35:47.529 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.529 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.529 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.530 DEBUG:boto:Params: {} 2023-10-11T15:35:47.530 DEBUG:boto:Token: None 2023-10-11T15:35:47.531 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj145 2023-10-11T15:35:47.531 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TUzbEKVydId0q3PQyY95KEiDM2w= 2023-10-11T15:35:47.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TUzbEKVydId0q3PQyY95KEiDM2w=', 'Content-Length': '0'} 2023-10-11T15:35:47.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319da14d811bff17-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.534 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.534 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.535 DEBUG:boto:path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.535 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.536 DEBUG:boto:Method: GET 2023-10-11T15:35:47.536 DEBUG:boto:Path: /dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.536 DEBUG:boto:Data: 2023-10-11T15:35:47.537 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.538 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.538 DEBUG:boto:Params: {} 2023-10-11T15:35:47.539 DEBUG:boto:Token: None 2023-10-11T15:35:47.539 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.540 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b839a2873ed71bd3-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.543 DEBUG:boto:b'dghqvd-61obj1451000falseobj1452023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.544 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.544 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.545 DEBUG:boto:path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.545 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.546 DEBUG:boto:Method: GET 2023-10-11T15:35:47.546 DEBUG:boto:Path: /dghqvd-61/?prefix=obj145 2023-10-11T15:35:47.547 DEBUG:boto:Data: 2023-10-11T15:35:47.547 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.548 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.548 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.548 DEBUG:boto:Params: {} 2023-10-11T15:35:47.549 DEBUG:boto:Token: None 2023-10-11T15:35:47.549 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.550 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005203e08fb109b2b3-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.554 DEBUG:boto:b'dghqvd-61obj1451000falseobj1452023-10-11T15:31:43.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:35:47.555 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.556 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.556 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.557 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.557 DEBUG:boto:Method: GET 2023-10-11T15:35:47.557 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.558 DEBUG:boto:Data: 2023-10-11T15:35:47.558 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.559 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.560 DEBUG:boto:Params: {} 2023-10-11T15:35:47.560 DEBUG:boto:Token: None 2023-10-11T15:35:47.561 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.561 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI= 2023-10-11T15:35:47.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI=', 'Content-Length': '0'} 2023-10-11T15:35:47.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a62b5e3439c6b63-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.606 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.607 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.608 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.608 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.608 DEBUG:boto:Method: GET 2023-10-11T15:35:47.609 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.609 DEBUG:boto:Data: 2023-10-11T15:35:47.610 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.611 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.611 DEBUG:boto:Params: {} 2023-10-11T15:35:47.611 DEBUG:boto:Token: None 2023-10-11T15:35:47.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI= 2023-10-11T15:35:47.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI=', 'Content-Length': '0'} 2023-10-11T15:35:47.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05605721daa3eba-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.616 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.616 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.617 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.617 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.617 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.618 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.618 DEBUG:boto:Data: 2023-10-11T15:35:47.619 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.620 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.620 DEBUG:boto:Params: {} 2023-10-11T15:35:47.620 DEBUG:boto:Token: None 2023-10-11T15:35:47.621 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pd26H2lpbjvfxhaWKw0jMZXfiXw= 2023-10-11T15:35:47.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pd26H2lpbjvfxhaWKw0jMZXfiXw=', 'Content-Length': '0'} 2023-10-11T15:35:47.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df1ff7905aeff6a0-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.624 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.625 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.625 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.625 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.626 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.626 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.627 DEBUG:boto:Data: 2023-10-11T15:35:47.627 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.628 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.628 DEBUG:boto:Params: {} 2023-10-11T15:35:47.629 DEBUG:boto:Token: None 2023-10-11T15:35:47.629 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.629 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pd26H2lpbjvfxhaWKw0jMZXfiXw= 2023-10-11T15:35:47.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pd26H2lpbjvfxhaWKw0jMZXfiXw=', 'Content-Length': '0'} 2023-10-11T15:35:47.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000315a6627866c4c9-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:35:47.633 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.633 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.634 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.634 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.635 DEBUG:boto:Method: GET 2023-10-11T15:35:47.635 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.635 DEBUG:boto:Data: 2023-10-11T15:35:47.636 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.636 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.637 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.637 DEBUG:boto:Params: {} 2023-10-11T15:35:47.637 DEBUG:boto:Token: None 2023-10-11T15:35:47.638 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.638 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI= 2023-10-11T15:35:47.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI=', 'Content-Length': '0'} 2023-10-11T15:35:47.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c091313ff8afe8-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.641 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.642 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.642 DEBUG:boto:path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.643 DEBUG:boto:auth_path=/dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.643 DEBUG:boto:Method: GET 2023-10-11T15:35:47.644 DEBUG:boto:Path: /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.644 DEBUG:boto:Data: 2023-10-11T15:35:47.644 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.645 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.646 DEBUG:boto:Params: {} 2023-10-11T15:35:47.646 DEBUG:boto:Token: None 2023-10-11T15:35:47.647 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148?versionId=null 2023-10-11T15:35:47.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI= 2023-10-11T15:35:47.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gwSLqLtl32pN8uuJFv3ENGFjHwI=', 'Content-Length': '0'} 2023-10-11T15:35:47.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004633a17d3690f3bc-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.650 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.651 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.651 DEBUG:boto:path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.651 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.652 DEBUG:boto:Method: GET 2023-10-11T15:35:47.652 DEBUG:boto:Path: /dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.653 DEBUG:boto:Data: 2023-10-11T15:35:47.653 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.654 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.654 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.654 DEBUG:boto:Params: {} 2023-10-11T15:35:47.655 DEBUG:boto:Token: None 2023-10-11T15:35:47.655 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.656 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49978b71d253fec-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.659 DEBUG:boto:b'dghqvd-61obj1481000falseobj1482023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.660 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.661 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.661 DEBUG:boto:path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.662 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.662 DEBUG:boto:Method: GET 2023-10-11T15:35:47.662 DEBUG:boto:Path: /dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.663 DEBUG:boto:Data: 2023-10-11T15:35:47.663 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.664 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.664 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.665 DEBUG:boto:Params: {} 2023-10-11T15:35:47.665 DEBUG:boto:Token: None 2023-10-11T15:35:47.665 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.666 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9e281e5e3f83ef-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.671 DEBUG:boto:b'dghqvd-61obj1481000falseobj1482023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.671 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.672 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.672 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.673 DEBUG:boto:Path: /dghqvd-61/obj148 2023-10-11T15:35:47.673 DEBUG:boto:Data: 2023-10-11T15:35:47.674 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.674 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.674 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.675 DEBUG:boto:Params: {} 2023-10-11T15:35:47.675 DEBUG:boto:Token: None 2023-10-11T15:35:47.676 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148 2023-10-11T15:35:47.676 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYakP43HIpKofuLGFSRjsPnatzY= 2023-10-11T15:35:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYakP43HIpKofuLGFSRjsPnatzY=', 'Content-Length': '0'} 2023-10-11T15:35:47.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac03d1b9ad9685f-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.679 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.679 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.680 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.680 DEBUG:boto:Path: /dghqvd-61/obj148 2023-10-11T15:35:47.681 DEBUG:boto:Data: 2023-10-11T15:35:47.681 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.682 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.682 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.682 DEBUG:boto:Params: {} 2023-10-11T15:35:47.683 DEBUG:boto:Token: None 2023-10-11T15:35:47.683 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148 2023-10-11T15:35:47.684 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pYakP43HIpKofuLGFSRjsPnatzY= 2023-10-11T15:35:47.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pYakP43HIpKofuLGFSRjsPnatzY=', 'Content-Length': '0'} 2023-10-11T15:35:47.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ed9be35976dd8a-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-10-11T15:35:47.687 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.688 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.688 DEBUG:boto:Method: GET 2023-10-11T15:35:47.688 DEBUG:boto:Path: /dghqvd-61/obj148 2023-10-11T15:35:47.689 DEBUG:boto:Data: 2023-10-11T15:35:47.689 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.690 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.690 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.691 DEBUG:boto:Params: {} 2023-10-11T15:35:47.691 DEBUG:boto:Token: None 2023-10-11T15:35:47.692 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148 2023-10-11T15:35:47.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vfbx83rIHIJaJvnvfCT/JTIjoI= 2023-10-11T15:35:47.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vfbx83rIHIJaJvnvfCT/JTIjoI=', 'Content-Length': '0'} 2023-10-11T15:35:47.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec2bd838b46bbf4-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.738 DEBUG:boto:path=/dghqvd-61/obj148 2023-10-11T15:35:47.739 DEBUG:boto:auth_path=/dghqvd-61/obj148 2023-10-11T15:35:47.739 DEBUG:boto:Method: GET 2023-10-11T15:35:47.740 DEBUG:boto:Path: /dghqvd-61/obj148 2023-10-11T15:35:47.740 DEBUG:boto:Data: 2023-10-11T15:35:47.741 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.742 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.742 DEBUG:boto:Params: {} 2023-10-11T15:35:47.743 DEBUG:boto:Token: None 2023-10-11T15:35:47.743 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj148 2023-10-11T15:35:47.744 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2vfbx83rIHIJaJvnvfCT/JTIjoI= 2023-10-11T15:35:47.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2vfbx83rIHIJaJvnvfCT/JTIjoI=', 'Content-Length': '0'} 2023-10-11T15:35:47.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ef352b355e6e73-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.747 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.748 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.748 DEBUG:boto:path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.748 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.749 DEBUG:boto:Method: GET 2023-10-11T15:35:47.749 DEBUG:boto:Path: /dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.750 DEBUG:boto:Data: 2023-10-11T15:35:47.750 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.751 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.751 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.751 DEBUG:boto:Params: {} 2023-10-11T15:35:47.752 DEBUG:boto:Token: None 2023-10-11T15:35:47.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.753 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dc97dbf3798bea7-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.756 DEBUG:boto:b'dghqvd-61obj1481000falseobj1482023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.757 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.758 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.758 DEBUG:boto:path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.758 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.759 DEBUG:boto:Method: GET 2023-10-11T15:35:47.759 DEBUG:boto:Path: /dghqvd-61/?prefix=obj148 2023-10-11T15:35:47.760 DEBUG:boto:Data: 2023-10-11T15:35:47.760 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.761 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.761 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.761 DEBUG:boto:Params: {} 2023-10-11T15:35:47.762 DEBUG:boto:Token: None 2023-10-11T15:35:47.762 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.763 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008efb2f065dae4995-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.810 DEBUG:boto:b'dghqvd-61obj1481000falseobj1482023-10-11T15:31:44.583Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:35:47.812 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.812 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.813 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.813 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.813 DEBUG:boto:Method: GET 2023-10-11T15:35:47.814 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.814 DEBUG:boto:Data: 2023-10-11T15:35:47.815 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.815 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.816 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.816 DEBUG:boto:Params: {} 2023-10-11T15:35:47.817 DEBUG:boto:Token: None 2023-10-11T15:35:47.817 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0= 2023-10-11T15:35:47.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0=', 'Content-Length': '0'} 2023-10-11T15:35:47.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6c1db11673dd91-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.821 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.822 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.822 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.822 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.823 DEBUG:boto:Method: GET 2023-10-11T15:35:47.823 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.824 DEBUG:boto:Data: 2023-10-11T15:35:47.824 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.825 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.825 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.826 DEBUG:boto:Params: {} 2023-10-11T15:35:47.826 DEBUG:boto:Token: None 2023-10-11T15:35:47.827 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.827 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0= 2023-10-11T15:35:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0=', 'Content-Length': '0'} 2023-10-11T15:35:47.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004864f9b898fae9af-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.830 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.831 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.831 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.831 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.832 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.832 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.833 DEBUG:boto:Data: 2023-10-11T15:35:47.833 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.834 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.834 DEBUG:boto:Params: {} 2023-10-11T15:35:47.835 DEBUG:boto:Token: None 2023-10-11T15:35:47.835 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.836 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wMDXDwR3yfw0xni2UfASnKjNiWw= 2023-10-11T15:35:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wMDXDwR3yfw0xni2UfASnKjNiWw=', 'Content-Length': '0'} 2023-10-11T15:35:47.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000613a8df9119520f3-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.839 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.839 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.839 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.840 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.840 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.841 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.841 DEBUG:boto:Data: 2023-10-11T15:35:47.841 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.842 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.843 DEBUG:boto:Params: {} 2023-10-11T15:35:47.843 DEBUG:boto:Token: None 2023-10-11T15:35:47.844 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wMDXDwR3yfw0xni2UfASnKjNiWw= 2023-10-11T15:35:47.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wMDXDwR3yfw0xni2UfASnKjNiWw=', 'Content-Length': '0'} 2023-10-11T15:35:47.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd9992bd04b8fcd-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:35:47.848 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.849 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.849 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.849 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.850 DEBUG:boto:Method: GET 2023-10-11T15:35:47.850 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.851 DEBUG:boto:Data: 2023-10-11T15:35:47.851 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.852 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.852 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.852 DEBUG:boto:Params: {} 2023-10-11T15:35:47.853 DEBUG:boto:Token: None 2023-10-11T15:35:47.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0= 2023-10-11T15:35:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0=', 'Content-Length': '0'} 2023-10-11T15:35:47.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624609b53f381a5d-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.857 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.857 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.858 DEBUG:boto:path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.858 DEBUG:boto:auth_path=/dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.859 DEBUG:boto:Method: GET 2023-10-11T15:35:47.859 DEBUG:boto:Path: /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.860 DEBUG:boto:Data: 2023-10-11T15:35:47.860 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.861 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.862 DEBUG:boto:Params: {} 2023-10-11T15:35:47.862 DEBUG:boto:Token: None 2023-10-11T15:35:47.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155?versionId=null 2023-10-11T15:35:47.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0= 2023-10-11T15:35:47.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aMrpVka1nU9vEhPmeOBxnWeluG0=', 'Content-Length': '0'} 2023-10-11T15:35:47.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa8cedbe8a6b14e-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.866 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.867 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.867 DEBUG:boto:path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.867 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.868 DEBUG:boto:Method: GET 2023-10-11T15:35:47.868 DEBUG:boto:Path: /dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.869 DEBUG:boto:Data: 2023-10-11T15:35:47.869 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.870 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.871 DEBUG:boto:Params: {} 2023-10-11T15:35:47.871 DEBUG:boto:Token: None 2023-10-11T15:35:47.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b752958c421fad-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.877 DEBUG:boto:b'dghqvd-61obj1551000falseobj1552023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.878 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.878 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.879 DEBUG:boto:path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.879 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.880 DEBUG:boto:Method: GET 2023-10-11T15:35:47.880 DEBUG:boto:Path: /dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.881 DEBUG:boto:Data: 2023-10-11T15:35:47.881 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.882 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.883 DEBUG:boto:Params: {} 2023-10-11T15:35:47.883 DEBUG:boto:Token: None 2023-10-11T15:35:47.884 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.884 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d4f1c12743e01bf-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.889 DEBUG:boto:b'dghqvd-61obj1551000falseobj1552023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.889 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.890 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.890 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.891 DEBUG:boto:Path: /dghqvd-61/obj155 2023-10-11T15:35:47.891 DEBUG:boto:Data: 2023-10-11T15:35:47.891 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.892 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.892 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.893 DEBUG:boto:Params: {} 2023-10-11T15:35:47.893 DEBUG:boto:Token: None 2023-10-11T15:35:47.894 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155 2023-10-11T15:35:47.894 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UmZR/7sUAQLkQEMR0ntVG04wJ4o= 2023-10-11T15:35:47.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UmZR/7sUAQLkQEMR0ntVG04wJ4o=', 'Content-Length': '0'} 2023-10-11T15:35:47.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e76a12622c75e3-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.897 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.897 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.898 DEBUG:boto:Method: HEAD 2023-10-11T15:35:47.898 DEBUG:boto:Path: /dghqvd-61/obj155 2023-10-11T15:35:47.898 DEBUG:boto:Data: 2023-10-11T15:35:47.899 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.899 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.900 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.900 DEBUG:boto:Params: {} 2023-10-11T15:35:47.900 DEBUG:boto:Token: None 2023-10-11T15:35:47.901 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155 2023-10-11T15:35:47.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UmZR/7sUAQLkQEMR0ntVG04wJ4o= 2023-10-11T15:35:47.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UmZR/7sUAQLkQEMR0ntVG04wJ4o=', 'Content-Length': '0'} 2023-10-11T15:35:47.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8935e88c3fd3fb6-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-10-11T15:35:47.905 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.905 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.905 DEBUG:boto:Method: GET 2023-10-11T15:35:47.906 DEBUG:boto:Path: /dghqvd-61/obj155 2023-10-11T15:35:47.906 DEBUG:boto:Data: 2023-10-11T15:35:47.907 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.908 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.908 DEBUG:boto:Params: {} 2023-10-11T15:35:47.908 DEBUG:boto:Token: None 2023-10-11T15:35:47.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155 2023-10-11T15:35:47.909 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I1MAvLYzbgtDnTv6we8SZmTvJtE= 2023-10-11T15:35:47.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I1MAvLYzbgtDnTv6we8SZmTvJtE=', 'Content-Length': '0'} 2023-10-11T15:35:47.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007abef8333438e62-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.912 DEBUG:boto:path=/dghqvd-61/obj155 2023-10-11T15:35:47.913 DEBUG:boto:auth_path=/dghqvd-61/obj155 2023-10-11T15:35:47.913 DEBUG:boto:Method: GET 2023-10-11T15:35:47.914 DEBUG:boto:Path: /dghqvd-61/obj155 2023-10-11T15:35:47.914 DEBUG:boto:Data: 2023-10-11T15:35:47.915 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.915 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.916 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.916 DEBUG:boto:Params: {} 2023-10-11T15:35:47.917 DEBUG:boto:Token: None 2023-10-11T15:35:47.917 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj155 2023-10-11T15:35:47.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I1MAvLYzbgtDnTv6we8SZmTvJtE= 2023-10-11T15:35:47.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I1MAvLYzbgtDnTv6we8SZmTvJtE=', 'Content-Length': '0'} 2023-10-11T15:35:47.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150bcc4b1ef3c5ea-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.962 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.963 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.963 DEBUG:boto:path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.964 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.964 DEBUG:boto:Method: GET 2023-10-11T15:35:47.964 DEBUG:boto:Path: /dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.965 DEBUG:boto:Data: 2023-10-11T15:35:47.965 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.966 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.967 DEBUG:boto:Params: {} 2023-10-11T15:35:47.967 DEBUG:boto:Token: None 2023-10-11T15:35:47.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff0ebe22c592b6e0-006526c0d3-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.972 DEBUG:boto:b'dghqvd-61obj1551000falseobj1552023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.973 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:47.973 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:47.974 DEBUG:boto:path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.974 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.975 DEBUG:boto:Method: GET 2023-10-11T15:35:47.975 DEBUG:boto:Path: /dghqvd-61/?prefix=obj155 2023-10-11T15:35:47.976 DEBUG:boto:Data: 2023-10-11T15:35:47.976 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.977 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.977 DEBUG:boto:Params: {} 2023-10-11T15:35:47.978 DEBUG:boto:Token: None 2023-10-11T15:35:47.978 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/ 2023-10-11T15:35:47.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y= 2023-10-11T15:35:47.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yVCmiW9ZOLt17JhsCs9l6HEJt4Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d49baee61f83ab0d-006526c0d3-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.983 DEBUG:boto:b'dghqvd-61obj1551000falseobj1552023-10-11T15:31:44.239Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:47.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:35:47.985 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:47.985 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:47.986 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.986 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.986 DEBUG:boto:Method: GET 2023-10-11T15:35:47.987 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.987 DEBUG:boto:Data: 2023-10-11T15:35:47.988 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:47.989 DEBUG:boto:Port: 8001 2023-10-11T15:35:47.989 DEBUG:boto:Params: {} 2023-10-11T15:35:47.989 DEBUG:boto:Token: None 2023-10-11T15:35:47.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qequwr9uOFejfKKtIn5KOe1t0/Y= 2023-10-11T15:35:47.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qequwr9uOFejfKKtIn5KOe1t0/Y=', 'Content-Length': '0'} 2023-10-11T15:35:47.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f9acccef98edb8e-006526c0d3-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:47.994 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:47.994 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:47.994 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.995 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.995 DEBUG:boto:Method: GET 2023-10-11T15:35:47.996 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.996 DEBUG:boto:Data: 2023-10-11T15:35:47.997 DEBUG:boto:Headers: {} 2023-10-11T15:35:47.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:47.997 DEBUG:boto:Port: 8000 2023-10-11T15:35:47.998 DEBUG:boto:Params: {} 2023-10-11T15:35:47.998 DEBUG:boto:Token: None 2023-10-11T15:35:47.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:47 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:47.999 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Qequwr9uOFejfKKtIn5KOe1t0/Y= 2023-10-11T15:35:48.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Qequwr9uOFejfKKtIn5KOe1t0/Y=', 'Content-Length': '0'} 2023-10-11T15:35:48.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b2e4cbc2ca81886-006526c0d3-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.002 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.003 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.003 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.003 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.004 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.004 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.005 DEBUG:boto:Data: 2023-10-11T15:35:48.005 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.006 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.006 DEBUG:boto:Params: {} 2023-10-11T15:35:48.007 DEBUG:boto:Token: None 2023-10-11T15:35:48.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G2hnK0F1qobrt3fTIb5QUH+wVPo= 2023-10-11T15:35:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G2hnK0F1qobrt3fTIb5QUH+wVPo=', 'Content-Length': '0'} 2023-10-11T15:35:48.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f276013af8c156e4-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.011 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.011 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.012 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.012 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.012 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.013 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.013 DEBUG:boto:Data: 2023-10-11T15:35:48.014 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.015 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.015 DEBUG:boto:Params: {} 2023-10-11T15:35:48.016 DEBUG:boto:Token: None 2023-10-11T15:35:48.016 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G2hnK0F1qobrt3fTIb5QUH+wVPo= 2023-10-11T15:35:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G2hnK0F1qobrt3fTIb5QUH+wVPo=', 'Content-Length': '0'} 2023-10-11T15:35:48.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240c269fa34770e0-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:35:48.020 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.020 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.021 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.021 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.022 DEBUG:boto:Method: GET 2023-10-11T15:35:48.022 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.022 DEBUG:boto:Data: 2023-10-11T15:35:48.023 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.024 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.024 DEBUG:boto:Params: {} 2023-10-11T15:35:48.025 DEBUG:boto:Token: None 2023-10-11T15:35:48.025 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7R8PoKx3xhx0LegFoBPyxzYY4zk= 2023-10-11T15:35:48.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7R8PoKx3xhx0LegFoBPyxzYY4zk=', 'Content-Length': '0'} 2023-10-11T15:35:48.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799a248a87433482-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.029 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.029 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.029 DEBUG:boto:path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.030 DEBUG:boto:auth_path=/dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.030 DEBUG:boto:Method: GET 2023-10-11T15:35:48.031 DEBUG:boto:Path: /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.031 DEBUG:boto:Data: 2023-10-11T15:35:48.032 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.032 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.033 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.033 DEBUG:boto:Params: {} 2023-10-11T15:35:48.034 DEBUG:boto:Token: None 2023-10-11T15:35:48.034 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17?versionId=null 2023-10-11T15:35:48.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7R8PoKx3xhx0LegFoBPyxzYY4zk= 2023-10-11T15:35:48.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7R8PoKx3xhx0LegFoBPyxzYY4zk=', 'Content-Length': '0'} 2023-10-11T15:35:48.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd07dbc9e9c15999-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.082 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.083 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.083 DEBUG:boto:path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.084 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.084 DEBUG:boto:Method: GET 2023-10-11T15:35:48.085 DEBUG:boto:Path: /dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.085 DEBUG:boto:Data: 2023-10-11T15:35:48.086 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.086 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.087 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.087 DEBUG:boto:Params: {} 2023-10-11T15:35:48.088 DEBUG:boto:Token: None 2023-10-11T15:35:48.088 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.089 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2f95adcb5b65ef-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.092 DEBUG:boto:b'dghqvd-61obj171000falseobj172023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.093 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.093 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.094 DEBUG:boto:path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.094 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.095 DEBUG:boto:Method: GET 2023-10-11T15:35:48.095 DEBUG:boto:Path: /dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.095 DEBUG:boto:Data: 2023-10-11T15:35:48.096 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.096 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.097 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.097 DEBUG:boto:Params: {} 2023-10-11T15:35:48.098 DEBUG:boto:Token: None 2023-10-11T15:35:48.098 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.099 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12e56c74dd0a3a2-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.104 DEBUG:boto:b'dghqvd-61obj171000falseobj172023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.104 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.105 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.105 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.105 DEBUG:boto:Path: /dghqvd-61/obj17 2023-10-11T15:35:48.106 DEBUG:boto:Data: 2023-10-11T15:35:48.106 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.107 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.108 DEBUG:boto:Params: {} 2023-10-11T15:35:48.108 DEBUG:boto:Token: None 2023-10-11T15:35:48.109 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17 2023-10-11T15:35:48.109 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fsnJ4bJAe6dXvEh82E/xx38X52M= 2023-10-11T15:35:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fsnJ4bJAe6dXvEh82E/xx38X52M=', 'Content-Length': '0'} 2023-10-11T15:35:48.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc42d0649bf7c116-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.112 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.112 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.113 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.113 DEBUG:boto:Path: /dghqvd-61/obj17 2023-10-11T15:35:48.114 DEBUG:boto:Data: 2023-10-11T15:35:48.114 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.115 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.115 DEBUG:boto:Params: {} 2023-10-11T15:35:48.116 DEBUG:boto:Token: None 2023-10-11T15:35:48.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17 2023-10-11T15:35:48.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fsnJ4bJAe6dXvEh82E/xx38X52M= 2023-10-11T15:35:48.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fsnJ4bJAe6dXvEh82E/xx38X52M=', 'Content-Length': '0'} 2023-10-11T15:35:48.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db9964db78f6ebb0-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-10-11T15:35:48.120 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.120 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.121 DEBUG:boto:Method: GET 2023-10-11T15:35:48.121 DEBUG:boto:Path: /dghqvd-61/obj17 2023-10-11T15:35:48.122 DEBUG:boto:Data: 2023-10-11T15:35:48.122 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.123 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.123 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.123 DEBUG:boto:Params: {} 2023-10-11T15:35:48.124 DEBUG:boto:Token: None 2023-10-11T15:35:48.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17 2023-10-11T15:35:48.125 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8UWl/RQ8BqvcIKK4Mnec9p6nuHM= 2023-10-11T15:35:48.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8UWl/RQ8BqvcIKK4Mnec9p6nuHM=', 'Content-Length': '0'} 2023-10-11T15:35:48.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e728beb02b5eee1e-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.170 DEBUG:boto:path=/dghqvd-61/obj17 2023-10-11T15:35:48.171 DEBUG:boto:auth_path=/dghqvd-61/obj17 2023-10-11T15:35:48.172 DEBUG:boto:Method: GET 2023-10-11T15:35:48.172 DEBUG:boto:Path: /dghqvd-61/obj17 2023-10-11T15:35:48.172 DEBUG:boto:Data: 2023-10-11T15:35:48.173 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.174 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.174 DEBUG:boto:Params: {} 2023-10-11T15:35:48.175 DEBUG:boto:Token: None 2023-10-11T15:35:48.175 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj17 2023-10-11T15:35:48.176 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8UWl/RQ8BqvcIKK4Mnec9p6nuHM= 2023-10-11T15:35:48.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8UWl/RQ8BqvcIKK4Mnec9p6nuHM=', 'Content-Length': '0'} 2023-10-11T15:35:48.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351d4208534029bc-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.179 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.180 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.180 DEBUG:boto:path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.181 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.181 DEBUG:boto:Method: GET 2023-10-11T15:35:48.182 DEBUG:boto:Path: /dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.182 DEBUG:boto:Data: 2023-10-11T15:35:48.182 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.183 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.184 DEBUG:boto:Params: {} 2023-10-11T15:35:48.184 DEBUG:boto:Token: None 2023-10-11T15:35:48.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ea53b84e938a487-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.189 DEBUG:boto:b'dghqvd-61obj171000falseobj172023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.190 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.190 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.191 DEBUG:boto:path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.191 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.191 DEBUG:boto:Method: GET 2023-10-11T15:35:48.192 DEBUG:boto:Path: /dghqvd-61/?prefix=obj17 2023-10-11T15:35:48.192 DEBUG:boto:Data: 2023-10-11T15:35:48.193 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.194 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.194 DEBUG:boto:Params: {} 2023-10-11T15:35:48.194 DEBUG:boto:Token: None 2023-10-11T15:35:48.195 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647b10c5852998e6-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.242 DEBUG:boto:b'dghqvd-61obj171000falseobj172023-10-11T15:31:42.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:35:48.244 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.244 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.244 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.245 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.245 DEBUG:boto:Method: GET 2023-10-11T15:35:48.246 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.246 DEBUG:boto:Data: 2023-10-11T15:35:48.247 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.247 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.247 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.248 DEBUG:boto:Params: {} 2023-10-11T15:35:48.248 DEBUG:boto:Token: None 2023-10-11T15:35:48.249 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.249 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo= 2023-10-11T15:35:48.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo=', 'Content-Length': '0'} 2023-10-11T15:35:48.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9c7032d0a33be0-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.253 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.253 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.253 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.254 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.254 DEBUG:boto:Method: GET 2023-10-11T15:35:48.255 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.255 DEBUG:boto:Data: 2023-10-11T15:35:48.256 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.256 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.256 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.257 DEBUG:boto:Params: {} 2023-10-11T15:35:48.257 DEBUG:boto:Token: None 2023-10-11T15:35:48.258 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.258 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo= 2023-10-11T15:35:48.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo=', 'Content-Length': '0'} 2023-10-11T15:35:48.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aecca3a8bee948b1-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.261 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.262 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.262 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.263 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.263 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.264 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.264 DEBUG:boto:Data: 2023-10-11T15:35:48.264 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.265 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.265 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.265 DEBUG:boto:Params: {} 2023-10-11T15:35:48.266 DEBUG:boto:Token: None 2023-10-11T15:35:48.266 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.267 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GA+G89qCZ4bHcaibgCS8OkDIOSY= 2023-10-11T15:35:48.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GA+G89qCZ4bHcaibgCS8OkDIOSY=', 'Content-Length': '0'} 2023-10-11T15:35:48.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18e39483a6fbdd6-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.270 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.270 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.270 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.271 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.271 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.272 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.272 DEBUG:boto:Data: 2023-10-11T15:35:48.272 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.273 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.273 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.274 DEBUG:boto:Params: {} 2023-10-11T15:35:48.274 DEBUG:boto:Token: None 2023-10-11T15:35:48.275 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.275 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GA+G89qCZ4bHcaibgCS8OkDIOSY= 2023-10-11T15:35:48.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GA+G89qCZ4bHcaibgCS8OkDIOSY=', 'Content-Length': '0'} 2023-10-11T15:35:48.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090fb580306b35a1f-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:35:48.278 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.279 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.279 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.279 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.280 DEBUG:boto:Method: GET 2023-10-11T15:35:48.280 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.281 DEBUG:boto:Data: 2023-10-11T15:35:48.281 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.282 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.283 DEBUG:boto:Params: {} 2023-10-11T15:35:48.283 DEBUG:boto:Token: None 2023-10-11T15:35:48.283 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo= 2023-10-11T15:35:48.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo=', 'Content-Length': '0'} 2023-10-11T15:35:48.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c475defd1b3a817-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.287 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.287 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.288 DEBUG:boto:path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.288 DEBUG:boto:auth_path=/dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.289 DEBUG:boto:Method: GET 2023-10-11T15:35:48.289 DEBUG:boto:Path: /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.289 DEBUG:boto:Data: 2023-10-11T15:35:48.290 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.291 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.291 DEBUG:boto:Params: {} 2023-10-11T15:35:48.292 DEBUG:boto:Token: None 2023-10-11T15:35:48.292 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185?versionId=null 2023-10-11T15:35:48.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo= 2023-10-11T15:35:48.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y72uen76l5BU6fCPs1+9LpgE3Uo=', 'Content-Length': '0'} 2023-10-11T15:35:48.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a865188668ed500-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.296 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.296 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.296 DEBUG:boto:path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.297 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.297 DEBUG:boto:Method: GET 2023-10-11T15:35:48.298 DEBUG:boto:Path: /dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.298 DEBUG:boto:Data: 2023-10-11T15:35:48.299 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.299 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.299 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.300 DEBUG:boto:Params: {} 2023-10-11T15:35:48.300 DEBUG:boto:Token: None 2023-10-11T15:35:48.301 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f07691a02ed7376-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.305 DEBUG:boto:b'dghqvd-61obj1851000falseobj1852023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.305 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.306 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.306 DEBUG:boto:path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.307 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.307 DEBUG:boto:Method: GET 2023-10-11T15:35:48.308 DEBUG:boto:Path: /dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.308 DEBUG:boto:Data: 2023-10-11T15:35:48.308 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.309 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.309 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.310 DEBUG:boto:Params: {} 2023-10-11T15:35:48.310 DEBUG:boto:Token: None 2023-10-11T15:35:48.311 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.311 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009657dfa86cdc45f6-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.315 DEBUG:boto:b'dghqvd-61obj1851000falseobj1852023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.316 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.316 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.317 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.317 DEBUG:boto:Path: /dghqvd-61/obj185 2023-10-11T15:35:48.318 DEBUG:boto:Data: 2023-10-11T15:35:48.318 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.319 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.319 DEBUG:boto:Params: {} 2023-10-11T15:35:48.320 DEBUG:boto:Token: None 2023-10-11T15:35:48.320 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185 2023-10-11T15:35:48.321 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ojd4ceBmEkH0fnaxdnov9lbuu0= 2023-10-11T15:35:48.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ojd4ceBmEkH0fnaxdnov9lbuu0=', 'Content-Length': '0'} 2023-10-11T15:35:48.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f18d2eab11da7aa-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.323 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.324 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.324 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.325 DEBUG:boto:Path: /dghqvd-61/obj185 2023-10-11T15:35:48.325 DEBUG:boto:Data: 2023-10-11T15:35:48.325 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.326 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.326 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.327 DEBUG:boto:Params: {} 2023-10-11T15:35:48.327 DEBUG:boto:Token: None 2023-10-11T15:35:48.327 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185 2023-10-11T15:35:48.328 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ojd4ceBmEkH0fnaxdnov9lbuu0= 2023-10-11T15:35:48.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ojd4ceBmEkH0fnaxdnov9lbuu0=', 'Content-Length': '0'} 2023-10-11T15:35:48.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054dd60e2d0938843-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-10-11T15:35:48.331 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.332 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.332 DEBUG:boto:Method: GET 2023-10-11T15:35:48.333 DEBUG:boto:Path: /dghqvd-61/obj185 2023-10-11T15:35:48.333 DEBUG:boto:Data: 2023-10-11T15:35:48.333 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.334 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.335 DEBUG:boto:Params: {} 2023-10-11T15:35:48.335 DEBUG:boto:Token: None 2023-10-11T15:35:48.336 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185 2023-10-11T15:35:48.336 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Amkbeu2xv+SrG4hEcZE0x49Dqio= 2023-10-11T15:35:48.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Amkbeu2xv+SrG4hEcZE0x49Dqio=', 'Content-Length': '0'} 2023-10-11T15:35:48.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0d8e1cf5a49ce4-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.382 DEBUG:boto:path=/dghqvd-61/obj185 2023-10-11T15:35:48.383 DEBUG:boto:auth_path=/dghqvd-61/obj185 2023-10-11T15:35:48.383 DEBUG:boto:Method: GET 2023-10-11T15:35:48.384 DEBUG:boto:Path: /dghqvd-61/obj185 2023-10-11T15:35:48.384 DEBUG:boto:Data: 2023-10-11T15:35:48.385 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.385 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.386 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.386 DEBUG:boto:Params: {} 2023-10-11T15:35:48.387 DEBUG:boto:Token: None 2023-10-11T15:35:48.387 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj185 2023-10-11T15:35:48.387 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Amkbeu2xv+SrG4hEcZE0x49Dqio= 2023-10-11T15:35:48.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Amkbeu2xv+SrG4hEcZE0x49Dqio=', 'Content-Length': '0'} 2023-10-11T15:35:48.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb5477027f540f46-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.391 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.391 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.391 DEBUG:boto:path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.392 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.392 DEBUG:boto:Method: GET 2023-10-11T15:35:48.393 DEBUG:boto:Path: /dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.393 DEBUG:boto:Data: 2023-10-11T15:35:48.394 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.394 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.394 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.395 DEBUG:boto:Params: {} 2023-10-11T15:35:48.395 DEBUG:boto:Token: None 2023-10-11T15:35:48.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.396 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f67748de48eb9af-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.400 DEBUG:boto:b'dghqvd-61obj1851000falseobj1852023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.400 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.401 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.401 DEBUG:boto:path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.402 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.402 DEBUG:boto:Method: GET 2023-10-11T15:35:48.403 DEBUG:boto:Path: /dghqvd-61/?prefix=obj185 2023-10-11T15:35:48.403 DEBUG:boto:Data: 2023-10-11T15:35:48.403 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.404 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.405 DEBUG:boto:Params: {} 2023-10-11T15:35:48.405 DEBUG:boto:Token: None 2023-10-11T15:35:48.406 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058bf87a029c5b786-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.413 DEBUG:boto:b'dghqvd-61obj1851000falseobj1852023-10-11T15:31:44.603Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:35:48.415 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.416 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.416 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.417 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.417 DEBUG:boto:Method: GET 2023-10-11T15:35:48.417 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.418 DEBUG:boto:Data: 2023-10-11T15:35:48.418 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.419 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.420 DEBUG:boto:Params: {} 2023-10-11T15:35:48.421 DEBUG:boto:Token: None 2023-10-11T15:35:48.421 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.422 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg= 2023-10-11T15:35:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg=', 'Content-Length': '0'} 2023-10-11T15:35:48.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037a4c4746629238c-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.425 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.426 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.426 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.427 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.427 DEBUG:boto:Method: GET 2023-10-11T15:35:48.428 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.428 DEBUG:boto:Data: 2023-10-11T15:35:48.429 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.429 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.430 DEBUG:boto:Params: {} 2023-10-11T15:35:48.431 DEBUG:boto:Token: None 2023-10-11T15:35:48.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.432 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg= 2023-10-11T15:35:48.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg=', 'Content-Length': '0'} 2023-10-11T15:35:48.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd187c60d0c3217-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.435 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.435 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.436 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.436 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.436 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.437 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.437 DEBUG:boto:Data: 2023-10-11T15:35:48.438 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.439 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.439 DEBUG:boto:Params: {} 2023-10-11T15:35:48.439 DEBUG:boto:Token: None 2023-10-11T15:35:48.440 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.440 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9NsPFQn0D3/OmVVlruMa8F6AfOs= 2023-10-11T15:35:48.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9NsPFQn0D3/OmVVlruMa8F6AfOs=', 'Content-Length': '0'} 2023-10-11T15:35:48.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261de78f5223c6ba-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.443 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.444 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.444 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.445 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.445 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.445 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.446 DEBUG:boto:Data: 2023-10-11T15:35:48.446 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.447 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.447 DEBUG:boto:Params: {} 2023-10-11T15:35:48.448 DEBUG:boto:Token: None 2023-10-11T15:35:48.448 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9NsPFQn0D3/OmVVlruMa8F6AfOs= 2023-10-11T15:35:48.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9NsPFQn0D3/OmVVlruMa8F6AfOs=', 'Content-Length': '0'} 2023-10-11T15:35:48.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f666bf41cdacb0a-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:35:48.452 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.452 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.453 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.453 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.454 DEBUG:boto:Method: GET 2023-10-11T15:35:48.454 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.455 DEBUG:boto:Data: 2023-10-11T15:35:48.455 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.455 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.456 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.456 DEBUG:boto:Params: {} 2023-10-11T15:35:48.457 DEBUG:boto:Token: None 2023-10-11T15:35:48.457 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg= 2023-10-11T15:35:48.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg=', 'Content-Length': '0'} 2023-10-11T15:35:48.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006acabf799e983b67-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.506 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.507 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.507 DEBUG:boto:path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.508 DEBUG:boto:auth_path=/dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.508 DEBUG:boto:Method: GET 2023-10-11T15:35:48.509 DEBUG:boto:Path: /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.509 DEBUG:boto:Data: 2023-10-11T15:35:48.510 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.510 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.510 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.511 DEBUG:boto:Params: {} 2023-10-11T15:35:48.511 DEBUG:boto:Token: None 2023-10-11T15:35:48.512 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19?versionId=null 2023-10-11T15:35:48.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg= 2023-10-11T15:35:48.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UPwsfhvTGUuMMe9nQNGyIISOPXg=', 'Content-Length': '0'} 2023-10-11T15:35:48.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e7150afe936e91-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.516 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.516 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.517 DEBUG:boto:path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.517 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.518 DEBUG:boto:Method: GET 2023-10-11T15:35:48.518 DEBUG:boto:Path: /dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.518 DEBUG:boto:Data: 2023-10-11T15:35:48.519 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.519 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.520 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.520 DEBUG:boto:Params: {} 2023-10-11T15:35:48.520 DEBUG:boto:Token: None 2023-10-11T15:35:48.521 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.521 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b88f08608e21c5-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.525 DEBUG:boto:b'dghqvd-61obj191000falseobj192023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.526 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.526 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.527 DEBUG:boto:path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.527 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.528 DEBUG:boto:Method: GET 2023-10-11T15:35:48.528 DEBUG:boto:Path: /dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.529 DEBUG:boto:Data: 2023-10-11T15:35:48.529 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.530 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.530 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.531 DEBUG:boto:Params: {} 2023-10-11T15:35:48.531 DEBUG:boto:Token: None 2023-10-11T15:35:48.532 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.532 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712591056e137864-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.582 DEBUG:boto:b'dghqvd-61obj191000falseobj192023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.583 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.584 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.584 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.585 DEBUG:boto:Path: /dghqvd-61/obj19 2023-10-11T15:35:48.585 DEBUG:boto:Data: 2023-10-11T15:35:48.585 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.586 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.587 DEBUG:boto:Params: {} 2023-10-11T15:35:48.587 DEBUG:boto:Token: None 2023-10-11T15:35:48.588 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19 2023-10-11T15:35:48.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lcAwzB+y34KID+fhAKVSIpUCw/0= 2023-10-11T15:35:48.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lcAwzB+y34KID+fhAKVSIpUCw/0=', 'Content-Length': '0'} 2023-10-11T15:35:48.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003934edaa9eebf268-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.591 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.592 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.592 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.593 DEBUG:boto:Path: /dghqvd-61/obj19 2023-10-11T15:35:48.593 DEBUG:boto:Data: 2023-10-11T15:35:48.593 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.594 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.595 DEBUG:boto:Params: {} 2023-10-11T15:35:48.595 DEBUG:boto:Token: None 2023-10-11T15:35:48.596 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19 2023-10-11T15:35:48.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lcAwzB+y34KID+fhAKVSIpUCw/0= 2023-10-11T15:35:48.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lcAwzB+y34KID+fhAKVSIpUCw/0=', 'Content-Length': '0'} 2023-10-11T15:35:48.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fe7314ae74d49c-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-10-11T15:35:48.599 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.600 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.600 DEBUG:boto:Method: GET 2023-10-11T15:35:48.601 DEBUG:boto:Path: /dghqvd-61/obj19 2023-10-11T15:35:48.601 DEBUG:boto:Data: 2023-10-11T15:35:48.601 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.602 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.602 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.603 DEBUG:boto:Params: {} 2023-10-11T15:35:48.603 DEBUG:boto:Token: None 2023-10-11T15:35:48.604 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19 2023-10-11T15:35:48.604 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3sVsRGOpww1PYuqqXWVZ+KL2D8= 2023-10-11T15:35:48.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3sVsRGOpww1PYuqqXWVZ+KL2D8=', 'Content-Length': '0'} 2023-10-11T15:35:48.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df83214735f2aaec-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.607 DEBUG:boto:path=/dghqvd-61/obj19 2023-10-11T15:35:48.608 DEBUG:boto:auth_path=/dghqvd-61/obj19 2023-10-11T15:35:48.608 DEBUG:boto:Method: GET 2023-10-11T15:35:48.608 DEBUG:boto:Path: /dghqvd-61/obj19 2023-10-11T15:35:48.609 DEBUG:boto:Data: 2023-10-11T15:35:48.609 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.610 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.611 DEBUG:boto:Params: {} 2023-10-11T15:35:48.611 DEBUG:boto:Token: None 2023-10-11T15:35:48.612 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj19 2023-10-11T15:35:48.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:O3sVsRGOpww1PYuqqXWVZ+KL2D8= 2023-10-11T15:35:48.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:O3sVsRGOpww1PYuqqXWVZ+KL2D8=', 'Content-Length': '0'} 2023-10-11T15:35:48.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068ef5517cdbb532b-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.615 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.615 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.616 DEBUG:boto:path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.616 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.617 DEBUG:boto:Method: GET 2023-10-11T15:35:48.617 DEBUG:boto:Path: /dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.618 DEBUG:boto:Data: 2023-10-11T15:35:48.618 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.619 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.619 DEBUG:boto:Params: {} 2023-10-11T15:35:48.620 DEBUG:boto:Token: None 2023-10-11T15:35:48.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000393e6fd6ff481729-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.666 DEBUG:boto:b'dghqvd-61obj191000falseobj192023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.667 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.668 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.668 DEBUG:boto:path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.669 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.669 DEBUG:boto:Method: GET 2023-10-11T15:35:48.669 DEBUG:boto:Path: /dghqvd-61/?prefix=obj19 2023-10-11T15:35:48.670 DEBUG:boto:Data: 2023-10-11T15:35:48.670 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.671 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.672 DEBUG:boto:Params: {} 2023-10-11T15:35:48.672 DEBUG:boto:Token: None 2023-10-11T15:35:48.673 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b86842a82842488d-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.678 DEBUG:boto:b'dghqvd-61obj191000falseobj192023-10-11T15:31:43.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:35:48.679 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.679 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.680 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.680 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.681 DEBUG:boto:Method: GET 2023-10-11T15:35:48.681 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.682 DEBUG:boto:Data: 2023-10-11T15:35:48.682 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.684 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.684 DEBUG:boto:Params: {} 2023-10-11T15:35:48.685 DEBUG:boto:Token: None 2023-10-11T15:35:48.685 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ= 2023-10-11T15:35:48.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015af8aff98df443f-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.692 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.692 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.693 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.693 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.694 DEBUG:boto:Method: GET 2023-10-11T15:35:48.694 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.695 DEBUG:boto:Data: 2023-10-11T15:35:48.695 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.696 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.696 DEBUG:boto:Params: {} 2023-10-11T15:35:48.697 DEBUG:boto:Token: None 2023-10-11T15:35:48.697 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.698 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ= 2023-10-11T15:35:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec92891dde89c178-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.701 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.702 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.702 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.702 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.703 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.703 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.704 DEBUG:boto:Data: 2023-10-11T15:35:48.704 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.704 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.705 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.705 DEBUG:boto:Params: {} 2023-10-11T15:35:48.706 DEBUG:boto:Token: None 2023-10-11T15:35:48.706 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.707 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DQCP70RIr4+AnPqsW3jtu4ZNfKM= 2023-10-11T15:35:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DQCP70RIr4+AnPqsW3jtu4ZNfKM=', 'Content-Length': '0'} 2023-10-11T15:35:48.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ae4442ef4eb164-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.709 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.710 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.710 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.711 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.711 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.712 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.712 DEBUG:boto:Data: 2023-10-11T15:35:48.712 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.713 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.714 DEBUG:boto:Params: {} 2023-10-11T15:35:48.714 DEBUG:boto:Token: None 2023-10-11T15:35:48.715 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.715 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DQCP70RIr4+AnPqsW3jtu4ZNfKM= 2023-10-11T15:35:48.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DQCP70RIr4+AnPqsW3jtu4ZNfKM=', 'Content-Length': '0'} 2023-10-11T15:35:48.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c602282657633c-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:35:48.718 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.719 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.719 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.719 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.720 DEBUG:boto:Method: GET 2023-10-11T15:35:48.720 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.721 DEBUG:boto:Data: 2023-10-11T15:35:48.721 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.722 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.722 DEBUG:boto:Params: {} 2023-10-11T15:35:48.723 DEBUG:boto:Token: None 2023-10-11T15:35:48.723 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.724 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ= 2023-10-11T15:35:48.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004259c87f1f8e3c5d-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.727 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.727 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.728 DEBUG:boto:path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.728 DEBUG:boto:auth_path=/dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.729 DEBUG:boto:Method: GET 2023-10-11T15:35:48.729 DEBUG:boto:Path: /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.729 DEBUG:boto:Data: 2023-10-11T15:35:48.730 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.731 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.731 DEBUG:boto:Params: {} 2023-10-11T15:35:48.732 DEBUG:boto:Token: None 2023-10-11T15:35:48.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22?versionId=null 2023-10-11T15:35:48.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ= 2023-10-11T15:35:48.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nBw6hNjqzqFillfHljkWsheKZDQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000672d8e2422e9347b-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.735 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.736 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.736 DEBUG:boto:path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.736 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.737 DEBUG:boto:Method: GET 2023-10-11T15:35:48.737 DEBUG:boto:Path: /dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.738 DEBUG:boto:Data: 2023-10-11T15:35:48.738 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.739 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.739 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.740 DEBUG:boto:Params: {} 2023-10-11T15:35:48.740 DEBUG:boto:Token: None 2023-10-11T15:35:48.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.741 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc2131da9e2a351-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.786 DEBUG:boto:b'dghqvd-61obj221000falseobj222023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.787 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.788 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.788 DEBUG:boto:path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.789 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.789 DEBUG:boto:Method: GET 2023-10-11T15:35:48.789 DEBUG:boto:Path: /dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.790 DEBUG:boto:Data: 2023-10-11T15:35:48.790 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.791 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.792 DEBUG:boto:Params: {} 2023-10-11T15:35:48.792 DEBUG:boto:Token: None 2023-10-11T15:35:48.793 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.793 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079fc05d2b9d66d5c-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.797 DEBUG:boto:b'dghqvd-61obj221000falseobj222023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.798 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.799 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.799 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.799 DEBUG:boto:Path: /dghqvd-61/obj22 2023-10-11T15:35:48.800 DEBUG:boto:Data: 2023-10-11T15:35:48.800 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.801 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.801 DEBUG:boto:Params: {} 2023-10-11T15:35:48.802 DEBUG:boto:Token: None 2023-10-11T15:35:48.802 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22 2023-10-11T15:35:48.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1PIbbqcXU+dbBRUWohbuyDA62GA= 2023-10-11T15:35:48.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1PIbbqcXU+dbBRUWohbuyDA62GA=', 'Content-Length': '0'} 2023-10-11T15:35:48.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce72b4be5fa91ed1-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.806 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.806 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.806 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.807 DEBUG:boto:Path: /dghqvd-61/obj22 2023-10-11T15:35:48.807 DEBUG:boto:Data: 2023-10-11T15:35:48.808 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.808 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.809 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.809 DEBUG:boto:Params: {} 2023-10-11T15:35:48.809 DEBUG:boto:Token: None 2023-10-11T15:35:48.810 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22 2023-10-11T15:35:48.810 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1PIbbqcXU+dbBRUWohbuyDA62GA= 2023-10-11T15:35:48.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1PIbbqcXU+dbBRUWohbuyDA62GA=', 'Content-Length': '0'} 2023-10-11T15:35:48.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aca7aeb7b6fc05b-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:35:48.814 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.814 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.814 DEBUG:boto:Method: GET 2023-10-11T15:35:48.815 DEBUG:boto:Path: /dghqvd-61/obj22 2023-10-11T15:35:48.815 DEBUG:boto:Data: 2023-10-11T15:35:48.816 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.816 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.817 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.817 DEBUG:boto:Params: {} 2023-10-11T15:35:48.817 DEBUG:boto:Token: None 2023-10-11T15:35:48.818 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22 2023-10-11T15:35:48.818 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SwWRsUR6ri07f+I3Nd8W+e/fpIA= 2023-10-11T15:35:48.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SwWRsUR6ri07f+I3Nd8W+e/fpIA=', 'Content-Length': '0'} 2023-10-11T15:35:48.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fbe9a1035f06313-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.821 DEBUG:boto:path=/dghqvd-61/obj22 2023-10-11T15:35:48.822 DEBUG:boto:auth_path=/dghqvd-61/obj22 2023-10-11T15:35:48.822 DEBUG:boto:Method: GET 2023-10-11T15:35:48.823 DEBUG:boto:Path: /dghqvd-61/obj22 2023-10-11T15:35:48.823 DEBUG:boto:Data: 2023-10-11T15:35:48.824 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.824 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.824 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.825 DEBUG:boto:Params: {} 2023-10-11T15:35:48.825 DEBUG:boto:Token: None 2023-10-11T15:35:48.826 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj22 2023-10-11T15:35:48.826 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SwWRsUR6ri07f+I3Nd8W+e/fpIA= 2023-10-11T15:35:48.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SwWRsUR6ri07f+I3Nd8W+e/fpIA=', 'Content-Length': '0'} 2023-10-11T15:35:48.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64931a8c594f015-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.829 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.830 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.830 DEBUG:boto:path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.831 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.831 DEBUG:boto:Method: GET 2023-10-11T15:35:48.832 DEBUG:boto:Path: /dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.832 DEBUG:boto:Data: 2023-10-11T15:35:48.832 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.833 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.834 DEBUG:boto:Params: {} 2023-10-11T15:35:48.834 DEBUG:boto:Token: None 2023-10-11T15:35:48.835 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002416e462a57d1449-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.838 DEBUG:boto:b'dghqvd-61obj221000falseobj222023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.839 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.840 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.840 DEBUG:boto:path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.841 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.841 DEBUG:boto:Method: GET 2023-10-11T15:35:48.842 DEBUG:boto:Path: /dghqvd-61/?prefix=obj22 2023-10-11T15:35:48.842 DEBUG:boto:Data: 2023-10-11T15:35:48.843 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.843 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.843 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.844 DEBUG:boto:Params: {} 2023-10-11T15:35:48.844 DEBUG:boto:Token: None 2023-10-11T15:35:48.845 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.845 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c66cabc4f777bf5-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.850 DEBUG:boto:b'dghqvd-61obj221000falseobj222023-10-11T15:31:42.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:35:48.851 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.852 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.852 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.852 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.853 DEBUG:boto:Method: GET 2023-10-11T15:35:48.853 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.854 DEBUG:boto:Data: 2023-10-11T15:35:48.854 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.855 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.855 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.855 DEBUG:boto:Params: {} 2023-10-11T15:35:48.856 DEBUG:boto:Token: None 2023-10-11T15:35:48.856 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.857 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg= 2023-10-11T15:35:48.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg=', 'Content-Length': '0'} 2023-10-11T15:35:48.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e8d3b3b7a23e8de-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.860 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.860 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.861 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.861 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.862 DEBUG:boto:Method: GET 2023-10-11T15:35:48.862 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.863 DEBUG:boto:Data: 2023-10-11T15:35:48.863 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.863 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.864 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.864 DEBUG:boto:Params: {} 2023-10-11T15:35:48.865 DEBUG:boto:Token: None 2023-10-11T15:35:48.865 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.866 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg= 2023-10-11T15:35:48.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg=', 'Content-Length': '0'} 2023-10-11T15:35:48.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eefd175a9824f7e-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.869 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.869 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.870 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.870 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.871 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.871 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.871 DEBUG:boto:Data: 2023-10-11T15:35:48.872 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.872 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.873 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.873 DEBUG:boto:Params: {} 2023-10-11T15:35:48.873 DEBUG:boto:Token: None 2023-10-11T15:35:48.874 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x2Me/NuRBndcRsdk6q48iabHeas= 2023-10-11T15:35:48.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x2Me/NuRBndcRsdk6q48iabHeas=', 'Content-Length': '0'} 2023-10-11T15:35:48.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3334209e79ea7d5-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.877 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.878 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.878 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.879 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.879 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.879 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.880 DEBUG:boto:Data: 2023-10-11T15:35:48.880 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.881 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.881 DEBUG:boto:Params: {} 2023-10-11T15:35:48.882 DEBUG:boto:Token: None 2023-10-11T15:35:48.882 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x2Me/NuRBndcRsdk6q48iabHeas= 2023-10-11T15:35:48.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x2Me/NuRBndcRsdk6q48iabHeas=', 'Content-Length': '0'} 2023-10-11T15:35:48.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15911d4b7e29e35-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:35:48.886 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.886 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.887 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.887 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.888 DEBUG:boto:Method: GET 2023-10-11T15:35:48.888 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.889 DEBUG:boto:Data: 2023-10-11T15:35:48.889 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.889 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.890 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.890 DEBUG:boto:Params: {} 2023-10-11T15:35:48.891 DEBUG:boto:Token: None 2023-10-11T15:35:48.891 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg= 2023-10-11T15:35:48.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg=', 'Content-Length': '0'} 2023-10-11T15:35:48.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096f8b44768271991-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.895 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.895 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.896 DEBUG:boto:path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.896 DEBUG:boto:auth_path=/dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.897 DEBUG:boto:Method: GET 2023-10-11T15:35:48.897 DEBUG:boto:Path: /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.897 DEBUG:boto:Data: 2023-10-11T15:35:48.898 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.899 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.899 DEBUG:boto:Params: {} 2023-10-11T15:35:48.900 DEBUG:boto:Token: None 2023-10-11T15:35:48.900 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23?versionId=null 2023-10-11T15:35:48.901 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg= 2023-10-11T15:35:48.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DBGAVB691JHXKaiSmw7/E+TCywg=', 'Content-Length': '0'} 2023-10-11T15:35:48.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24c520ef8111f98-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.904 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.904 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.905 DEBUG:boto:path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.905 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.906 DEBUG:boto:Method: GET 2023-10-11T15:35:48.906 DEBUG:boto:Path: /dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.907 DEBUG:boto:Data: 2023-10-11T15:35:48.907 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.907 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.908 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.908 DEBUG:boto:Params: {} 2023-10-11T15:35:48.909 DEBUG:boto:Token: None 2023-10-11T15:35:48.909 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.910 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086acf23eedf698e5-006526c0d4-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.913 DEBUG:boto:b'dghqvd-61obj231000falseobj232023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.914 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:48.914 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:48.915 DEBUG:boto:path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.915 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.915 DEBUG:boto:Method: GET 2023-10-11T15:35:48.916 DEBUG:boto:Path: /dghqvd-61/?prefix=obj23 2023-10-11T15:35:48.916 DEBUG:boto:Data: 2023-10-11T15:35:48.917 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.917 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.918 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.918 DEBUG:boto:Params: {} 2023-10-11T15:35:48.919 DEBUG:boto:Token: None 2023-10-11T15:35:48.919 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/ 2023-10-11T15:35:48.920 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw= 2023-10-11T15:35:48.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aYZCQDSM4pgG5rwhKMClb0+OwJw=', 'Content-Length': '0'} 2023-10-11T15:35:48.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e39f1924387e8c-006526c0d4-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.966 DEBUG:boto:b'dghqvd-61obj231000falseobj232023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:48.967 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.968 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.968 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.969 DEBUG:boto:Path: /dghqvd-61/obj23 2023-10-11T15:35:48.969 DEBUG:boto:Data: 2023-10-11T15:35:48.969 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.970 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.971 DEBUG:boto:Params: {} 2023-10-11T15:35:48.971 DEBUG:boto:Token: None 2023-10-11T15:35:48.972 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23 2023-10-11T15:35:48.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c3TGrNG+4N0wu8egZEAG9WXX7AQ= 2023-10-11T15:35:48.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c3TGrNG+4N0wu8egZEAG9WXX7AQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd8bb0314594823-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.976 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.976 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.977 DEBUG:boto:Method: HEAD 2023-10-11T15:35:48.977 DEBUG:boto:Path: /dghqvd-61/obj23 2023-10-11T15:35:48.978 DEBUG:boto:Data: 2023-10-11T15:35:48.978 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.978 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:48.979 DEBUG:boto:Port: 8000 2023-10-11T15:35:48.979 DEBUG:boto:Params: {} 2023-10-11T15:35:48.980 DEBUG:boto:Token: None 2023-10-11T15:35:48.980 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23 2023-10-11T15:35:48.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:c3TGrNG+4N0wu8egZEAG9WXX7AQ= 2023-10-11T15:35:48.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:c3TGrNG+4N0wu8egZEAG9WXX7AQ=', 'Content-Length': '0'} 2023-10-11T15:35:48.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b15fa46db6cc5347-006526c0d4-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:48.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-10-11T15:35:48.984 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:48.985 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:48.985 DEBUG:boto:Method: GET 2023-10-11T15:35:48.986 DEBUG:boto:Path: /dghqvd-61/obj23 2023-10-11T15:35:48.986 DEBUG:boto:Data: 2023-10-11T15:35:48.987 DEBUG:boto:Headers: {} 2023-10-11T15:35:48.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:48.988 DEBUG:boto:Port: 8001 2023-10-11T15:35:48.988 DEBUG:boto:Params: {} 2023-10-11T15:35:48.989 DEBUG:boto:Token: None 2023-10-11T15:35:48.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:48 GMT /dghqvd-61/obj23 2023-10-11T15:35:48.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JQFIdFPsxT8jzL8gy+rJQlWGWqE= 2023-10-11T15:35:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:48 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JQFIdFPsxT8jzL8gy+rJQlWGWqE=', 'Content-Length': '0'} 2023-10-11T15:35:48.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006790624f520c8a4e-006526c0d4-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:48 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.038 DEBUG:boto:path=/dghqvd-61/obj23 2023-10-11T15:35:49.039 DEBUG:boto:auth_path=/dghqvd-61/obj23 2023-10-11T15:35:49.040 DEBUG:boto:Method: GET 2023-10-11T15:35:49.040 DEBUG:boto:Path: /dghqvd-61/obj23 2023-10-11T15:35:49.040 DEBUG:boto:Data: 2023-10-11T15:35:49.041 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.041 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.042 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.042 DEBUG:boto:Params: {} 2023-10-11T15:35:49.043 DEBUG:boto:Token: None 2023-10-11T15:35:49.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj23 2023-10-11T15:35:49.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FBwfRsG/VGXu6RkobWepSvOIr3A= 2023-10-11T15:35:49.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FBwfRsG/VGXu6RkobWepSvOIr3A=', 'Content-Length': '0'} 2023-10-11T15:35:49.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad00a583571b804-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.048 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.048 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.049 DEBUG:boto:path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.049 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.050 DEBUG:boto:Method: GET 2023-10-11T15:35:49.050 DEBUG:boto:Path: /dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.051 DEBUG:boto:Data: 2023-10-11T15:35:49.051 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.052 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.053 DEBUG:boto:Params: {} 2023-10-11T15:35:49.054 DEBUG:boto:Token: None 2023-10-11T15:35:49.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f41122b7d1bbfdac-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.058 DEBUG:boto:b'dghqvd-61obj231000falseobj232023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.059 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.059 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.060 DEBUG:boto:path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.060 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.061 DEBUG:boto:Method: GET 2023-10-11T15:35:49.061 DEBUG:boto:Path: /dghqvd-61/?prefix=obj23 2023-10-11T15:35:49.062 DEBUG:boto:Data: 2023-10-11T15:35:49.062 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.063 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.063 DEBUG:boto:Params: {} 2023-10-11T15:35:49.064 DEBUG:boto:Token: None 2023-10-11T15:35:49.064 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.065 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db5639f0869db24-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.070 DEBUG:boto:b'dghqvd-61obj231000falseobj232023-10-11T15:31:43.379Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:35:49.071 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.071 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.072 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.072 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.072 DEBUG:boto:Method: GET 2023-10-11T15:35:49.073 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.073 DEBUG:boto:Data: 2023-10-11T15:35:49.074 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.075 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.075 DEBUG:boto:Params: {} 2023-10-11T15:35:49.076 DEBUG:boto:Token: None 2023-10-11T15:35:49.076 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.077 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU= 2023-10-11T15:35:49.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU=', 'Content-Length': '0'} 2023-10-11T15:35:49.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9655840f37897bd-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.080 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.080 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.080 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.081 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.082 DEBUG:boto:Method: GET 2023-10-11T15:35:49.083 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.083 DEBUG:boto:Data: 2023-10-11T15:35:49.084 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.084 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.085 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.085 DEBUG:boto:Params: {} 2023-10-11T15:35:49.086 DEBUG:boto:Token: None 2023-10-11T15:35:49.086 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU= 2023-10-11T15:35:49.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU=', 'Content-Length': '0'} 2023-10-11T15:35:49.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3232a0806e26dc-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.090 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.091 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.091 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.091 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.092 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.092 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.093 DEBUG:boto:Data: 2023-10-11T15:35:49.093 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.094 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.094 DEBUG:boto:Params: {} 2023-10-11T15:35:49.097 DEBUG:boto:Token: None 2023-10-11T15:35:49.098 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xMf1PoEojWpy77/md9DZrlYuJvY= 2023-10-11T15:35:49.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xMf1PoEojWpy77/md9DZrlYuJvY=', 'Content-Length': '0'} 2023-10-11T15:35:49.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8223d92ff6b73ad-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.101 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.101 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.102 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.102 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.103 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.103 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.104 DEBUG:boto:Data: 2023-10-11T15:35:49.104 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.105 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.105 DEBUG:boto:Params: {} 2023-10-11T15:35:49.106 DEBUG:boto:Token: None 2023-10-11T15:35:49.106 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.107 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xMf1PoEojWpy77/md9DZrlYuJvY= 2023-10-11T15:35:49.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xMf1PoEojWpy77/md9DZrlYuJvY=', 'Content-Length': '0'} 2023-10-11T15:35:49.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfd3a43ec03c892a-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:35:49.110 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.111 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.111 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.111 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.112 DEBUG:boto:Method: GET 2023-10-11T15:35:49.112 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.113 DEBUG:boto:Data: 2023-10-11T15:35:49.113 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.114 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.114 DEBUG:boto:Params: {} 2023-10-11T15:35:49.115 DEBUG:boto:Token: None 2023-10-11T15:35:49.115 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.116 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU= 2023-10-11T15:35:49.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU=', 'Content-Length': '0'} 2023-10-11T15:35:49.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d82d2e4d9be81e4-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.119 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.119 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.120 DEBUG:boto:path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.120 DEBUG:boto:auth_path=/dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.121 DEBUG:boto:Method: GET 2023-10-11T15:35:49.121 DEBUG:boto:Path: /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.122 DEBUG:boto:Data: 2023-10-11T15:35:49.122 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.123 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.123 DEBUG:boto:Params: {} 2023-10-11T15:35:49.124 DEBUG:boto:Token: None 2023-10-11T15:35:49.124 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29?versionId=null 2023-10-11T15:35:49.125 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU= 2023-10-11T15:35:49.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:flXfFxJbjgKHlxOEmekIFLDEqHU=', 'Content-Length': '0'} 2023-10-11T15:35:49.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d5a9dd39b0d338-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.128 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.128 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.129 DEBUG:boto:path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.129 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.130 DEBUG:boto:Method: GET 2023-10-11T15:35:49.130 DEBUG:boto:Path: /dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.131 DEBUG:boto:Data: 2023-10-11T15:35:49.131 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.131 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.132 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.132 DEBUG:boto:Params: {} 2023-10-11T15:35:49.133 DEBUG:boto:Token: None 2023-10-11T15:35:49.133 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b27829eb1449378-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.178 DEBUG:boto:b'dghqvd-61obj291000falseobj292023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.179 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.180 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.180 DEBUG:boto:path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.181 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.181 DEBUG:boto:Method: GET 2023-10-11T15:35:49.182 DEBUG:boto:Path: /dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.182 DEBUG:boto:Data: 2023-10-11T15:35:49.183 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.184 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.184 DEBUG:boto:Params: {} 2023-10-11T15:35:49.184 DEBUG:boto:Token: None 2023-10-11T15:35:49.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b438cc98291fb1fc-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.190 DEBUG:boto:b'dghqvd-61obj291000falseobj292023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.191 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.191 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.192 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.192 DEBUG:boto:Path: /dghqvd-61/obj29 2023-10-11T15:35:49.192 DEBUG:boto:Data: 2023-10-11T15:35:49.193 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.194 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.194 DEBUG:boto:Params: {} 2023-10-11T15:35:49.195 DEBUG:boto:Token: None 2023-10-11T15:35:49.195 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29 2023-10-11T15:35:49.195 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Wku2Z4fZqTYBOEzEgZtrZHRJvE= 2023-10-11T15:35:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Wku2Z4fZqTYBOEzEgZtrZHRJvE=', 'Content-Length': '0'} 2023-10-11T15:35:49.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000609cc06423c5393-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.199 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.199 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.200 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.200 DEBUG:boto:Path: /dghqvd-61/obj29 2023-10-11T15:35:49.200 DEBUG:boto:Data: 2023-10-11T15:35:49.201 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.201 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.202 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.202 DEBUG:boto:Params: {} 2023-10-11T15:35:49.202 DEBUG:boto:Token: None 2023-10-11T15:35:49.203 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29 2023-10-11T15:35:49.203 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Wku2Z4fZqTYBOEzEgZtrZHRJvE= 2023-10-11T15:35:49.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Wku2Z4fZqTYBOEzEgZtrZHRJvE=', 'Content-Length': '0'} 2023-10-11T15:35:49.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069e69e6f9fc1a36-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-10-11T15:35:49.207 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.207 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.208 DEBUG:boto:Method: GET 2023-10-11T15:35:49.208 DEBUG:boto:Path: /dghqvd-61/obj29 2023-10-11T15:35:49.209 DEBUG:boto:Data: 2023-10-11T15:35:49.209 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.210 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.210 DEBUG:boto:Params: {} 2023-10-11T15:35:49.211 DEBUG:boto:Token: None 2023-10-11T15:35:49.211 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29 2023-10-11T15:35:49.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81LrKFKnt65QKFKTpZKnHIGCfeY= 2023-10-11T15:35:49.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81LrKFKnt65QKFKTpZKnHIGCfeY=', 'Content-Length': '0'} 2023-10-11T15:35:49.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001749ef7610e8f3a5-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.215 DEBUG:boto:path=/dghqvd-61/obj29 2023-10-11T15:35:49.215 DEBUG:boto:auth_path=/dghqvd-61/obj29 2023-10-11T15:35:49.215 DEBUG:boto:Method: GET 2023-10-11T15:35:49.216 DEBUG:boto:Path: /dghqvd-61/obj29 2023-10-11T15:35:49.216 DEBUG:boto:Data: 2023-10-11T15:35:49.217 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.217 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.218 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.218 DEBUG:boto:Params: {} 2023-10-11T15:35:49.219 DEBUG:boto:Token: None 2023-10-11T15:35:49.219 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj29 2023-10-11T15:35:49.220 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:81LrKFKnt65QKFKTpZKnHIGCfeY= 2023-10-11T15:35:49.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:81LrKFKnt65QKFKTpZKnHIGCfeY=', 'Content-Length': '0'} 2023-10-11T15:35:49.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac67d9667a470929-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.223 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.223 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.224 DEBUG:boto:path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.224 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.224 DEBUG:boto:Method: GET 2023-10-11T15:35:49.225 DEBUG:boto:Path: /dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.225 DEBUG:boto:Data: 2023-10-11T15:35:49.226 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.226 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.227 DEBUG:boto:Params: {} 2023-10-11T15:35:49.228 DEBUG:boto:Token: None 2023-10-11T15:35:49.228 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.229 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fcaf8886dda02ce-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.232 DEBUG:boto:b'dghqvd-61obj291000falseobj292023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.233 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.233 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.234 DEBUG:boto:path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.234 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.235 DEBUG:boto:Method: GET 2023-10-11T15:35:49.235 DEBUG:boto:Path: /dghqvd-61/?prefix=obj29 2023-10-11T15:35:49.236 DEBUG:boto:Data: 2023-10-11T15:35:49.236 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.236 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.237 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.237 DEBUG:boto:Params: {} 2023-10-11T15:35:49.238 DEBUG:boto:Token: None 2023-10-11T15:35:49.239 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbafa6eaa3334ea3-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.244 DEBUG:boto:b'dghqvd-61obj291000falseobj292023-10-11T15:31:43.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:35:49.245 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.246 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.246 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.247 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.247 DEBUG:boto:Method: GET 2023-10-11T15:35:49.248 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.248 DEBUG:boto:Data: 2023-10-11T15:35:49.249 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.249 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.250 DEBUG:boto:Params: {} 2023-10-11T15:35:49.250 DEBUG:boto:Token: None 2023-10-11T15:35:49.251 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA= 2023-10-11T15:35:49.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009276d642260016da-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.255 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.255 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.256 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.256 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.256 DEBUG:boto:Method: GET 2023-10-11T15:35:49.257 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.257 DEBUG:boto:Data: 2023-10-11T15:35:49.258 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.258 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.259 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.259 DEBUG:boto:Params: {} 2023-10-11T15:35:49.259 DEBUG:boto:Token: None 2023-10-11T15:35:49.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA= 2023-10-11T15:35:49.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11fff1dd2cf7fab-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.263 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.264 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.264 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.265 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.265 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.265 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.266 DEBUG:boto:Data: 2023-10-11T15:35:49.266 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.267 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.267 DEBUG:boto:Params: {} 2023-10-11T15:35:49.268 DEBUG:boto:Token: None 2023-10-11T15:35:49.268 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9EayxBQpCFEN4fibgPmLyF7UIKE= 2023-10-11T15:35:49.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9EayxBQpCFEN4fibgPmLyF7UIKE=', 'Content-Length': '0'} 2023-10-11T15:35:49.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0aad113ba608f0-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.272 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.272 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.273 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.273 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.274 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.274 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.275 DEBUG:boto:Data: 2023-10-11T15:35:49.275 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.276 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.276 DEBUG:boto:Params: {} 2023-10-11T15:35:49.277 DEBUG:boto:Token: None 2023-10-11T15:35:49.277 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9EayxBQpCFEN4fibgPmLyF7UIKE= 2023-10-11T15:35:49.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9EayxBQpCFEN4fibgPmLyF7UIKE=', 'Content-Length': '0'} 2023-10-11T15:35:49.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5e150583f6b071-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:35:49.281 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.282 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.282 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.283 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.283 DEBUG:boto:Method: GET 2023-10-11T15:35:49.284 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.284 DEBUG:boto:Data: 2023-10-11T15:35:49.285 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.285 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.286 DEBUG:boto:Params: {} 2023-10-11T15:35:49.286 DEBUG:boto:Token: None 2023-10-11T15:35:49.287 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA= 2023-10-11T15:35:49.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e915e124928c8345-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.334 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.335 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.335 DEBUG:boto:path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.336 DEBUG:boto:auth_path=/dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.336 DEBUG:boto:Method: GET 2023-10-11T15:35:49.337 DEBUG:boto:Path: /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.337 DEBUG:boto:Data: 2023-10-11T15:35:49.338 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.338 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.339 DEBUG:boto:Params: {} 2023-10-11T15:35:49.339 DEBUG:boto:Token: None 2023-10-11T15:35:49.340 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31?versionId=null 2023-10-11T15:35:49.340 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA= 2023-10-11T15:35:49.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1ohAyPZ+zLAeDoEvAuvwhv1x1PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d2beb90e8591ad-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.343 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.344 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.344 DEBUG:boto:path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.345 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.345 DEBUG:boto:Method: GET 2023-10-11T15:35:49.346 DEBUG:boto:Path: /dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.346 DEBUG:boto:Data: 2023-10-11T15:35:49.346 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.347 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.347 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.348 DEBUG:boto:Params: {} 2023-10-11T15:35:49.348 DEBUG:boto:Token: None 2023-10-11T15:35:49.349 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067aa650e28f9129f-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.353 DEBUG:boto:b'dghqvd-61obj311000falseobj312023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.354 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.354 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.355 DEBUG:boto:path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.355 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.355 DEBUG:boto:Method: GET 2023-10-11T15:35:49.356 DEBUG:boto:Path: /dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.356 DEBUG:boto:Data: 2023-10-11T15:35:49.357 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.357 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.357 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.358 DEBUG:boto:Params: {} 2023-10-11T15:35:49.358 DEBUG:boto:Token: None 2023-10-11T15:35:49.359 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.359 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6263606b4b501a5-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.365 DEBUG:boto:b'dghqvd-61obj311000falseobj312023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.366 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.366 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.367 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.367 DEBUG:boto:Path: /dghqvd-61/obj31 2023-10-11T15:35:49.368 DEBUG:boto:Data: 2023-10-11T15:35:49.368 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.369 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.369 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.370 DEBUG:boto:Params: {} 2023-10-11T15:35:49.371 DEBUG:boto:Token: None 2023-10-11T15:35:49.372 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31 2023-10-11T15:35:49.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3qNAPnFZla2Rw3SP/C/TiRnC19Y= 2023-10-11T15:35:49.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3qNAPnFZla2Rw3SP/C/TiRnC19Y=', 'Content-Length': '0'} 2023-10-11T15:35:49.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049254fee3cb1366b-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.377 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.377 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.378 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.378 DEBUG:boto:Path: /dghqvd-61/obj31 2023-10-11T15:35:49.379 DEBUG:boto:Data: 2023-10-11T15:35:49.379 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.380 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.380 DEBUG:boto:Params: {} 2023-10-11T15:35:49.381 DEBUG:boto:Token: None 2023-10-11T15:35:49.381 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31 2023-10-11T15:35:49.382 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3qNAPnFZla2Rw3SP/C/TiRnC19Y= 2023-10-11T15:35:49.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3qNAPnFZla2Rw3SP/C/TiRnC19Y=', 'Content-Length': '0'} 2023-10-11T15:35:49.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cf5098ce2e0ddf-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-10-11T15:35:49.385 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.385 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.386 DEBUG:boto:Method: GET 2023-10-11T15:35:49.386 DEBUG:boto:Path: /dghqvd-61/obj31 2023-10-11T15:35:49.387 DEBUG:boto:Data: 2023-10-11T15:35:49.387 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.388 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.388 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.389 DEBUG:boto:Params: {} 2023-10-11T15:35:49.389 DEBUG:boto:Token: None 2023-10-11T15:35:49.389 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31 2023-10-11T15:35:49.390 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+58MoIoJ+26q63QIyrODSPeB0PA= 2023-10-11T15:35:49.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+58MoIoJ+26q63QIyrODSPeB0PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e84a6091b4ee1ef6-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.393 DEBUG:boto:path=/dghqvd-61/obj31 2023-10-11T15:35:49.393 DEBUG:boto:auth_path=/dghqvd-61/obj31 2023-10-11T15:35:49.394 DEBUG:boto:Method: GET 2023-10-11T15:35:49.394 DEBUG:boto:Path: /dghqvd-61/obj31 2023-10-11T15:35:49.395 DEBUG:boto:Data: 2023-10-11T15:35:49.395 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.395 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.396 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.396 DEBUG:boto:Params: {} 2023-10-11T15:35:49.397 DEBUG:boto:Token: None 2023-10-11T15:35:49.397 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj31 2023-10-11T15:35:49.398 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+58MoIoJ+26q63QIyrODSPeB0PA= 2023-10-11T15:35:49.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+58MoIoJ+26q63QIyrODSPeB0PA=', 'Content-Length': '0'} 2023-10-11T15:35:49.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004662a9d1556c5834-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.401 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.401 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.401 DEBUG:boto:path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.402 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.402 DEBUG:boto:Method: GET 2023-10-11T15:35:49.403 DEBUG:boto:Path: /dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.403 DEBUG:boto:Data: 2023-10-11T15:35:49.404 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.404 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.404 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.405 DEBUG:boto:Params: {} 2023-10-11T15:35:49.405 DEBUG:boto:Token: None 2023-10-11T15:35:49.406 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.406 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5b5ebfdfe80d5f-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.410 DEBUG:boto:b'dghqvd-61obj311000falseobj312023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.411 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.411 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.412 DEBUG:boto:path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.412 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.413 DEBUG:boto:Method: GET 2023-10-11T15:35:49.413 DEBUG:boto:Path: /dghqvd-61/?prefix=obj31 2023-10-11T15:35:49.413 DEBUG:boto:Data: 2023-10-11T15:35:49.414 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.414 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.415 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.415 DEBUG:boto:Params: {} 2023-10-11T15:35:49.416 DEBUG:boto:Token: None 2023-10-11T15:35:49.416 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005303b71de5c8a46b-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.421 DEBUG:boto:b'dghqvd-61obj311000falseobj312023-10-11T15:31:44.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:35:49.422 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.423 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.423 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.423 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.424 DEBUG:boto:Method: GET 2023-10-11T15:35:49.424 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.425 DEBUG:boto:Data: 2023-10-11T15:35:49.425 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.426 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.426 DEBUG:boto:Params: {} 2023-10-11T15:35:49.427 DEBUG:boto:Token: None 2023-10-11T15:35:49.427 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ= 2023-10-11T15:35:49.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b21766437436d18-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.431 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.431 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.432 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.432 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.433 DEBUG:boto:Method: GET 2023-10-11T15:35:49.433 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.434 DEBUG:boto:Data: 2023-10-11T15:35:49.434 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.434 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.435 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.435 DEBUG:boto:Params: {} 2023-10-11T15:35:49.436 DEBUG:boto:Token: None 2023-10-11T15:35:49.436 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.437 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ= 2023-10-11T15:35:49.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b983eaf2216ada8-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.482 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.483 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.484 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.484 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.484 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.485 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.485 DEBUG:boto:Data: 2023-10-11T15:35:49.486 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.486 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.486 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.487 DEBUG:boto:Params: {} 2023-10-11T15:35:49.487 DEBUG:boto:Token: None 2023-10-11T15:35:49.488 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.488 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hwd0Bv7UHQPfm9Viy3gYg9jayGw= 2023-10-11T15:35:49.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hwd0Bv7UHQPfm9Viy3gYg9jayGw=', 'Content-Length': '0'} 2023-10-11T15:35:49.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cab300bb31214775-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.491 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.492 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.492 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.492 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.493 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.493 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.494 DEBUG:boto:Data: 2023-10-11T15:35:49.494 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.494 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.495 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.495 DEBUG:boto:Params: {} 2023-10-11T15:35:49.496 DEBUG:boto:Token: None 2023-10-11T15:35:49.496 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.497 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hwd0Bv7UHQPfm9Viy3gYg9jayGw= 2023-10-11T15:35:49.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hwd0Bv7UHQPfm9Viy3gYg9jayGw=', 'Content-Length': '0'} 2023-10-11T15:35:49.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f0f654e7b4c2e5a-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:35:49.500 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.500 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.501 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.501 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.502 DEBUG:boto:Method: GET 2023-10-11T15:35:49.502 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.502 DEBUG:boto:Data: 2023-10-11T15:35:49.505 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.505 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.506 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.506 DEBUG:boto:Params: {} 2023-10-11T15:35:49.507 DEBUG:boto:Token: None 2023-10-11T15:35:49.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ= 2023-10-11T15:35:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc6685522c05698-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.511 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.512 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.512 DEBUG:boto:path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.513 DEBUG:boto:auth_path=/dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.513 DEBUG:boto:Method: GET 2023-10-11T15:35:49.514 DEBUG:boto:Path: /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.514 DEBUG:boto:Data: 2023-10-11T15:35:49.515 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.515 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.516 DEBUG:boto:Params: {} 2023-10-11T15:35:49.516 DEBUG:boto:Token: None 2023-10-11T15:35:49.517 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33?versionId=null 2023-10-11T15:35:49.517 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ= 2023-10-11T15:35:49.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:F2ozdev3djXkYnw1k2c32jwvgjQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021dd4912065bfb3c-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.520 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.521 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.521 DEBUG:boto:path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.522 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.522 DEBUG:boto:Method: GET 2023-10-11T15:35:49.523 DEBUG:boto:Path: /dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.523 DEBUG:boto:Data: 2023-10-11T15:35:49.523 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.524 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.525 DEBUG:boto:Params: {} 2023-10-11T15:35:49.525 DEBUG:boto:Token: None 2023-10-11T15:35:49.525 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60d5d92f6c31403-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.570 DEBUG:boto:b'dghqvd-61obj331000falseobj332023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.572 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.573 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.573 DEBUG:boto:path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.574 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.575 DEBUG:boto:Method: GET 2023-10-11T15:35:49.576 DEBUG:boto:Path: /dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.576 DEBUG:boto:Data: 2023-10-11T15:35:49.577 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.578 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.579 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.579 DEBUG:boto:Params: {} 2023-10-11T15:35:49.580 DEBUG:boto:Token: None 2023-10-11T15:35:49.581 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea3a4d018a6c409-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.588 DEBUG:boto:b'dghqvd-61obj331000falseobj332023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.589 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.589 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.590 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.590 DEBUG:boto:Path: /dghqvd-61/obj33 2023-10-11T15:35:49.590 DEBUG:boto:Data: 2023-10-11T15:35:49.591 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.591 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.592 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.592 DEBUG:boto:Params: {} 2023-10-11T15:35:49.593 DEBUG:boto:Token: None 2023-10-11T15:35:49.593 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33 2023-10-11T15:35:49.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e7A1TWTTBhtxUBtyt9IGK/Q9LuQ= 2023-10-11T15:35:49.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e7A1TWTTBhtxUBtyt9IGK/Q9LuQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084df70433f0b1033-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.597 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.597 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.597 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.598 DEBUG:boto:Path: /dghqvd-61/obj33 2023-10-11T15:35:49.598 DEBUG:boto:Data: 2023-10-11T15:35:49.599 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.599 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.600 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.600 DEBUG:boto:Params: {} 2023-10-11T15:35:49.601 DEBUG:boto:Token: None 2023-10-11T15:35:49.601 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33 2023-10-11T15:35:49.601 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e7A1TWTTBhtxUBtyt9IGK/Q9LuQ= 2023-10-11T15:35:49.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e7A1TWTTBhtxUBtyt9IGK/Q9LuQ=', 'Content-Length': '0'} 2023-10-11T15:35:49.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9889d528a028370-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:35:49.604 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.605 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.605 DEBUG:boto:Method: GET 2023-10-11T15:35:49.606 DEBUG:boto:Path: /dghqvd-61/obj33 2023-10-11T15:35:49.606 DEBUG:boto:Data: 2023-10-11T15:35:49.607 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.607 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.608 DEBUG:boto:Params: {} 2023-10-11T15:35:49.608 DEBUG:boto:Token: None 2023-10-11T15:35:49.609 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33 2023-10-11T15:35:49.609 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fgugsOkttscDOvlJsxu3RmOyt3g= 2023-10-11T15:35:49.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fgugsOkttscDOvlJsxu3RmOyt3g=', 'Content-Length': '0'} 2023-10-11T15:35:49.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d181c26cf60361e5-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.612 DEBUG:boto:path=/dghqvd-61/obj33 2023-10-11T15:35:49.613 DEBUG:boto:auth_path=/dghqvd-61/obj33 2023-10-11T15:35:49.613 DEBUG:boto:Method: GET 2023-10-11T15:35:49.613 DEBUG:boto:Path: /dghqvd-61/obj33 2023-10-11T15:35:49.614 DEBUG:boto:Data: 2023-10-11T15:35:49.614 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.615 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.615 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.615 DEBUG:boto:Params: {} 2023-10-11T15:35:49.616 DEBUG:boto:Token: None 2023-10-11T15:35:49.616 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj33 2023-10-11T15:35:49.617 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fgugsOkttscDOvlJsxu3RmOyt3g= 2023-10-11T15:35:49.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fgugsOkttscDOvlJsxu3RmOyt3g=', 'Content-Length': '0'} 2023-10-11T15:35:49.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400a7103dadabdcb-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.662 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.663 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.663 DEBUG:boto:path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.664 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.664 DEBUG:boto:Method: GET 2023-10-11T15:35:49.665 DEBUG:boto:Path: /dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.665 DEBUG:boto:Data: 2023-10-11T15:35:49.665 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.666 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.667 DEBUG:boto:Params: {} 2023-10-11T15:35:49.667 DEBUG:boto:Token: None 2023-10-11T15:35:49.668 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c529e4f86e4ab40f-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.672 DEBUG:boto:b'dghqvd-61obj331000falseobj332023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.672 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.673 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.673 DEBUG:boto:path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.674 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.674 DEBUG:boto:Method: GET 2023-10-11T15:35:49.675 DEBUG:boto:Path: /dghqvd-61/?prefix=obj33 2023-10-11T15:35:49.675 DEBUG:boto:Data: 2023-10-11T15:35:49.676 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.677 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.677 DEBUG:boto:Params: {} 2023-10-11T15:35:49.678 DEBUG:boto:Token: None 2023-10-11T15:35:49.678 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ec44a4fd6d014d-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.683 DEBUG:boto:b'dghqvd-61obj331000falseobj332023-10-11T15:31:42.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:35:49.685 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.685 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.686 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.686 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.687 DEBUG:boto:Method: GET 2023-10-11T15:35:49.688 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.688 DEBUG:boto:Data: 2023-10-11T15:35:49.689 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.690 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.690 DEBUG:boto:Params: {} 2023-10-11T15:35:49.691 DEBUG:boto:Token: None 2023-10-11T15:35:49.691 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE= 2023-10-11T15:35:49.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE=', 'Content-Length': '0'} 2023-10-11T15:35:49.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c3e803585c976c-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.695 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.695 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.696 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.696 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.697 DEBUG:boto:Method: GET 2023-10-11T15:35:49.697 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.698 DEBUG:boto:Data: 2023-10-11T15:35:49.698 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.699 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.700 DEBUG:boto:Params: {} 2023-10-11T15:35:49.700 DEBUG:boto:Token: None 2023-10-11T15:35:49.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.701 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE= 2023-10-11T15:35:49.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE=', 'Content-Length': '0'} 2023-10-11T15:35:49.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3172e426162cfbc-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.704 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.704 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.705 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.705 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.705 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.706 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.706 DEBUG:boto:Data: 2023-10-11T15:35:49.707 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.709 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.710 DEBUG:boto:Params: {} 2023-10-11T15:35:49.710 DEBUG:boto:Token: None 2023-10-11T15:35:49.711 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.711 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LqzR8jO9uL0oOhkhCyHm7USPTbM= 2023-10-11T15:35:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LqzR8jO9uL0oOhkhCyHm7USPTbM=', 'Content-Length': '0'} 2023-10-11T15:35:49.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fbee4901c24c7f4-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.714 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.714 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.715 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.715 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.715 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.716 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.716 DEBUG:boto:Data: 2023-10-11T15:35:49.717 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.717 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.718 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.718 DEBUG:boto:Params: {} 2023-10-11T15:35:49.719 DEBUG:boto:Token: None 2023-10-11T15:35:49.719 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LqzR8jO9uL0oOhkhCyHm7USPTbM= 2023-10-11T15:35:49.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LqzR8jO9uL0oOhkhCyHm7USPTbM=', 'Content-Length': '0'} 2023-10-11T15:35:49.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea194c6c9d9a740-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:35:49.723 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.723 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.724 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.724 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.725 DEBUG:boto:Method: GET 2023-10-11T15:35:49.725 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.726 DEBUG:boto:Data: 2023-10-11T15:35:49.726 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.727 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.727 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.727 DEBUG:boto:Params: {} 2023-10-11T15:35:49.728 DEBUG:boto:Token: None 2023-10-11T15:35:49.729 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.729 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE= 2023-10-11T15:35:49.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE=', 'Content-Length': '0'} 2023-10-11T15:35:49.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003027d817cf1973a1-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.778 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.779 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.779 DEBUG:boto:path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.780 DEBUG:boto:auth_path=/dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.780 DEBUG:boto:Method: GET 2023-10-11T15:35:49.781 DEBUG:boto:Path: /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.781 DEBUG:boto:Data: 2023-10-11T15:35:49.782 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.782 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.782 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.783 DEBUG:boto:Params: {} 2023-10-11T15:35:49.783 DEBUG:boto:Token: None 2023-10-11T15:35:49.784 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34?versionId=null 2023-10-11T15:35:49.784 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE= 2023-10-11T15:35:49.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Li98CuwEVtyLCsdhLXkAtw6fOYE=', 'Content-Length': '0'} 2023-10-11T15:35:49.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6d1847d2f5f9f8e-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.787 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.788 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.788 DEBUG:boto:path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.789 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.789 DEBUG:boto:Method: GET 2023-10-11T15:35:49.789 DEBUG:boto:Path: /dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.790 DEBUG:boto:Data: 2023-10-11T15:35:49.790 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.791 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.791 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.792 DEBUG:boto:Params: {} 2023-10-11T15:35:49.792 DEBUG:boto:Token: None 2023-10-11T15:35:49.793 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.794 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fcb002a11c42e0c-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.797 DEBUG:boto:b'dghqvd-61obj341000falseobj342023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.798 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.798 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.799 DEBUG:boto:path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.799 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.800 DEBUG:boto:Method: GET 2023-10-11T15:35:49.800 DEBUG:boto:Path: /dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.801 DEBUG:boto:Data: 2023-10-11T15:35:49.801 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.802 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.804 DEBUG:boto:Params: {} 2023-10-11T15:35:49.804 DEBUG:boto:Token: None 2023-10-11T15:35:49.806 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003feeabf21cfb9079-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.854 DEBUG:boto:b'dghqvd-61obj341000falseobj342023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.855 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.856 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.856 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.857 DEBUG:boto:Path: /dghqvd-61/obj34 2023-10-11T15:35:49.857 DEBUG:boto:Data: 2023-10-11T15:35:49.858 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.858 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.858 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.859 DEBUG:boto:Params: {} 2023-10-11T15:35:49.859 DEBUG:boto:Token: None 2023-10-11T15:35:49.860 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34 2023-10-11T15:35:49.860 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbfvwKpDDtLjlzuJTCL4D30hUsg= 2023-10-11T15:35:49.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbfvwKpDDtLjlzuJTCL4D30hUsg=', 'Content-Length': '0'} 2023-10-11T15:35:49.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10917dd33dab5d8-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.864 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.864 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.864 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.865 DEBUG:boto:Path: /dghqvd-61/obj34 2023-10-11T15:35:49.865 DEBUG:boto:Data: 2023-10-11T15:35:49.866 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.866 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.867 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.867 DEBUG:boto:Params: {} 2023-10-11T15:35:49.868 DEBUG:boto:Token: None 2023-10-11T15:35:49.868 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34 2023-10-11T15:35:49.868 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbfvwKpDDtLjlzuJTCL4D30hUsg= 2023-10-11T15:35:49.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbfvwKpDDtLjlzuJTCL4D30hUsg=', 'Content-Length': '0'} 2023-10-11T15:35:49.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7f3b0bf600007d-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-10-11T15:35:49.872 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.872 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.873 DEBUG:boto:Method: GET 2023-10-11T15:35:49.873 DEBUG:boto:Path: /dghqvd-61/obj34 2023-10-11T15:35:49.874 DEBUG:boto:Data: 2023-10-11T15:35:49.874 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.875 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.875 DEBUG:boto:Params: {} 2023-10-11T15:35:49.876 DEBUG:boto:Token: None 2023-10-11T15:35:49.876 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34 2023-10-11T15:35:49.877 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMi+XTzacqhiYTtgH5EBkLcSuw8= 2023-10-11T15:35:49.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMi+XTzacqhiYTtgH5EBkLcSuw8=', 'Content-Length': '0'} 2023-10-11T15:35:49.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c98925436fb9af39-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.880 DEBUG:boto:path=/dghqvd-61/obj34 2023-10-11T15:35:49.880 DEBUG:boto:auth_path=/dghqvd-61/obj34 2023-10-11T15:35:49.881 DEBUG:boto:Method: GET 2023-10-11T15:35:49.881 DEBUG:boto:Path: /dghqvd-61/obj34 2023-10-11T15:35:49.881 DEBUG:boto:Data: 2023-10-11T15:35:49.882 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.883 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.883 DEBUG:boto:Params: {} 2023-10-11T15:35:49.884 DEBUG:boto:Token: None 2023-10-11T15:35:49.884 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj34 2023-10-11T15:35:49.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMi+XTzacqhiYTtgH5EBkLcSuw8= 2023-10-11T15:35:49.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMi+XTzacqhiYTtgH5EBkLcSuw8=', 'Content-Length': '0'} 2023-10-11T15:35:49.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d13a77ab675933-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.887 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.888 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.888 DEBUG:boto:path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.889 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.889 DEBUG:boto:Method: GET 2023-10-11T15:35:49.889 DEBUG:boto:Path: /dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.890 DEBUG:boto:Data: 2023-10-11T15:35:49.890 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.891 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.891 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.892 DEBUG:boto:Params: {} 2023-10-11T15:35:49.892 DEBUG:boto:Token: None 2023-10-11T15:35:49.893 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.893 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deac79d1a0352199-006526c0d5-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.938 DEBUG:boto:b'dghqvd-61obj341000falseobj342023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.939 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:49.940 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:49.940 DEBUG:boto:path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.941 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.941 DEBUG:boto:Method: GET 2023-10-11T15:35:49.941 DEBUG:boto:Path: /dghqvd-61/?prefix=obj34 2023-10-11T15:35:49.942 DEBUG:boto:Data: 2023-10-11T15:35:49.943 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.943 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.944 DEBUG:boto:Params: {} 2023-10-11T15:35:49.944 DEBUG:boto:Token: None 2023-10-11T15:35:49.945 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/ 2023-10-11T15:35:49.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g= 2023-10-11T15:35:49.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:62SHk48Mb/WbULh5WrnyjSTGU2g=', 'Content-Length': '0'} 2023-10-11T15:35:49.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db2fff61eb9269a5-006526c0d5-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.950 DEBUG:boto:b'dghqvd-61obj341000falseobj342023-10-11T15:31:42.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:49.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:35:49.951 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.951 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.952 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.952 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.953 DEBUG:boto:Method: GET 2023-10-11T15:35:49.953 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.954 DEBUG:boto:Data: 2023-10-11T15:35:49.954 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.955 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.955 DEBUG:boto:Params: {} 2023-10-11T15:35:49.956 DEBUG:boto:Token: None 2023-10-11T15:35:49.956 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY= 2023-10-11T15:35:49.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY=', 'Content-Length': '0'} 2023-10-11T15:35:49.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000001d03a8e14289f8-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.960 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.961 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.961 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.961 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.962 DEBUG:boto:Method: GET 2023-10-11T15:35:49.962 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.963 DEBUG:boto:Data: 2023-10-11T15:35:49.963 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.964 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.964 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.964 DEBUG:boto:Params: {} 2023-10-11T15:35:49.965 DEBUG:boto:Token: None 2023-10-11T15:35:49.965 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.966 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY= 2023-10-11T15:35:49.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY=', 'Content-Length': '0'} 2023-10-11T15:35:49.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e393de23736e1c55-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.969 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.970 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.970 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.970 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.971 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.971 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.972 DEBUG:boto:Data: 2023-10-11T15:35:49.972 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.973 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.973 DEBUG:boto:Params: {} 2023-10-11T15:35:49.974 DEBUG:boto:Token: None 2023-10-11T15:35:49.974 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sCiN2Tm6bSA3Zo85MPr9pNuDmc4= 2023-10-11T15:35:49.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sCiN2Tm6bSA3Zo85MPr9pNuDmc4=', 'Content-Length': '0'} 2023-10-11T15:35:49.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004416d4555d422c01-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.977 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.978 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.978 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.979 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.979 DEBUG:boto:Method: HEAD 2023-10-11T15:35:49.979 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.980 DEBUG:boto:Data: 2023-10-11T15:35:49.980 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.981 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.981 DEBUG:boto:Params: {} 2023-10-11T15:35:49.982 DEBUG:boto:Token: None 2023-10-11T15:35:49.982 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sCiN2Tm6bSA3Zo85MPr9pNuDmc4= 2023-10-11T15:35:49.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sCiN2Tm6bSA3Zo85MPr9pNuDmc4=', 'Content-Length': '0'} 2023-10-11T15:35:49.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034c82e3c31870c6d-006526c0d5-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:35:49.986 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.986 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.987 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.987 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.987 DEBUG:boto:Method: GET 2023-10-11T15:35:49.988 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.988 DEBUG:boto:Data: 2023-10-11T15:35:49.989 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:49.990 DEBUG:boto:Port: 8001 2023-10-11T15:35:49.990 DEBUG:boto:Params: {} 2023-10-11T15:35:49.991 DEBUG:boto:Token: None 2023-10-11T15:35:49.991 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:49 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY= 2023-10-11T15:35:49.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:49 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iNgBaH0bO9vu4YA1ums6LSiCAIY=', 'Content-Length': '0'} 2023-10-11T15:35:49.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041911d2d35e1bfdb-006526c0d5-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:49 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:49.995 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:49.995 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:49.995 DEBUG:boto:path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.996 DEBUG:boto:auth_path=/dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.996 DEBUG:boto:Method: GET 2023-10-11T15:35:49.997 DEBUG:boto:Path: /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:49.997 DEBUG:boto:Data: 2023-10-11T15:35:49.998 DEBUG:boto:Headers: {} 2023-10-11T15:35:49.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:49.998 DEBUG:boto:Port: 8000 2023-10-11T15:35:49.999 DEBUG:boto:Params: {} 2023-10-11T15:35:49.999 DEBUG:boto:Token: None 2023-10-11T15:35:50.000 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj37?versionId=null 2023-10-11T15:35:50.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4vadjBSkmz8lQeDfN+jGVy2d9Is= 2023-10-11T15:35:50.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4vadjBSkmz8lQeDfN+jGVy2d9Is=', 'Content-Length': '0'} 2023-10-11T15:35:50.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a0a7bb6a4582d53-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.003 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.004 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.004 DEBUG:boto:path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.005 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.005 DEBUG:boto:Method: GET 2023-10-11T15:35:50.005 DEBUG:boto:Path: /dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.006 DEBUG:boto:Data: 2023-10-11T15:35:50.006 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.007 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.007 DEBUG:boto:Params: {} 2023-10-11T15:35:50.008 DEBUG:boto:Token: None 2023-10-11T15:35:50.008 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000038ff6dcb2f0768a-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.012 DEBUG:boto:b'dghqvd-61obj371000falseobj372023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.013 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.013 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.014 DEBUG:boto:path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.014 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.015 DEBUG:boto:Method: GET 2023-10-11T15:35:50.015 DEBUG:boto:Path: /dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.016 DEBUG:boto:Data: 2023-10-11T15:35:50.016 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.017 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.017 DEBUG:boto:Params: {} 2023-10-11T15:35:50.018 DEBUG:boto:Token: None 2023-10-11T15:35:50.018 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.019 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe79c4795ec0cec8-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.066 DEBUG:boto:b'dghqvd-61obj371000falseobj372023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.067 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:50.068 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:50.068 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.069 DEBUG:boto:Path: /dghqvd-61/obj37 2023-10-11T15:35:50.069 DEBUG:boto:Data: 2023-10-11T15:35:50.069 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.070 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.070 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.071 DEBUG:boto:Params: {} 2023-10-11T15:35:50.071 DEBUG:boto:Token: None 2023-10-11T15:35:50.072 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj37 2023-10-11T15:35:50.072 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8WEn912CCTHUlMD/pk/8ycj3SyQ= 2023-10-11T15:35:50.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8WEn912CCTHUlMD/pk/8ycj3SyQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3edca64115168d6-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.075 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:50.076 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:50.076 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.076 DEBUG:boto:Path: /dghqvd-61/obj37 2023-10-11T15:35:50.077 DEBUG:boto:Data: 2023-10-11T15:35:50.077 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.078 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.078 DEBUG:boto:Params: {} 2023-10-11T15:35:50.079 DEBUG:boto:Token: None 2023-10-11T15:35:50.079 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj37 2023-10-11T15:35:50.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8WEn912CCTHUlMD/pk/8ycj3SyQ= 2023-10-11T15:35:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8WEn912CCTHUlMD/pk/8ycj3SyQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8020cd0bd2a77f1-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-10-11T15:35:50.083 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:50.084 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:50.084 DEBUG:boto:Method: GET 2023-10-11T15:35:50.084 DEBUG:boto:Path: /dghqvd-61/obj37 2023-10-11T15:35:50.085 DEBUG:boto:Data: 2023-10-11T15:35:50.085 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.086 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.086 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.087 DEBUG:boto:Params: {} 2023-10-11T15:35:50.087 DEBUG:boto:Token: None 2023-10-11T15:35:50.088 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj37 2023-10-11T15:35:50.088 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b20cnYjZe+Fsay4KCP6TIuQg7rY= 2023-10-11T15:35:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b20cnYjZe+Fsay4KCP6TIuQg7rY=', 'Content-Length': '0'} 2023-10-11T15:35:50.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005322a164c09ab211-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.134 DEBUG:boto:path=/dghqvd-61/obj37 2023-10-11T15:35:50.135 DEBUG:boto:auth_path=/dghqvd-61/obj37 2023-10-11T15:35:50.136 DEBUG:boto:Method: GET 2023-10-11T15:35:50.136 DEBUG:boto:Path: /dghqvd-61/obj37 2023-10-11T15:35:50.136 DEBUG:boto:Data: 2023-10-11T15:35:50.137 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.138 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.138 DEBUG:boto:Params: {} 2023-10-11T15:35:50.139 DEBUG:boto:Token: None 2023-10-11T15:35:50.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj37 2023-10-11T15:35:50.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:b20cnYjZe+Fsay4KCP6TIuQg7rY= 2023-10-11T15:35:50.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:b20cnYjZe+Fsay4KCP6TIuQg7rY=', 'Content-Length': '0'} 2023-10-11T15:35:50.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a1263ed729c0a5-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.143 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.143 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.144 DEBUG:boto:path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.144 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.144 DEBUG:boto:Method: GET 2023-10-11T15:35:50.145 DEBUG:boto:Path: /dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.145 DEBUG:boto:Data: 2023-10-11T15:35:50.146 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.147 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.147 DEBUG:boto:Params: {} 2023-10-11T15:35:50.147 DEBUG:boto:Token: None 2023-10-11T15:35:50.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73213fcf4b6f6d2-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.152 DEBUG:boto:b'dghqvd-61obj371000falseobj372023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.152 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.153 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.153 DEBUG:boto:path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.154 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.154 DEBUG:boto:Method: GET 2023-10-11T15:35:50.154 DEBUG:boto:Path: /dghqvd-61/?prefix=obj37 2023-10-11T15:35:50.155 DEBUG:boto:Data: 2023-10-11T15:35:50.155 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.156 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.157 DEBUG:boto:Params: {} 2023-10-11T15:35:50.157 DEBUG:boto:Token: None 2023-10-11T15:35:50.158 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c544a9332b5c1c-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.163 DEBUG:boto:b'dghqvd-61obj371000falseobj372023-10-11T15:31:44.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:35:50.164 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.165 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.165 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.166 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.166 DEBUG:boto:Method: GET 2023-10-11T15:35:50.166 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.167 DEBUG:boto:Data: 2023-10-11T15:35:50.167 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.168 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.169 DEBUG:boto:Params: {} 2023-10-11T15:35:50.169 DEBUG:boto:Token: None 2023-10-11T15:35:50.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.170 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko= 2023-10-11T15:35:50.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko=', 'Content-Length': '0'} 2023-10-11T15:35:50.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2783675faa0d3f3-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.173 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.173 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.174 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.174 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.175 DEBUG:boto:Method: GET 2023-10-11T15:35:50.175 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.176 DEBUG:boto:Data: 2023-10-11T15:35:50.176 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.176 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.177 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.177 DEBUG:boto:Params: {} 2023-10-11T15:35:50.178 DEBUG:boto:Token: None 2023-10-11T15:35:50.178 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko= 2023-10-11T15:35:50.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko=', 'Content-Length': '0'} 2023-10-11T15:35:50.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b39a39f9d78ece-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.182 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.182 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.183 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.183 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.183 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.184 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.184 DEBUG:boto:Data: 2023-10-11T15:35:50.185 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.185 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.186 DEBUG:boto:Params: {} 2023-10-11T15:35:50.186 DEBUG:boto:Token: None 2023-10-11T15:35:50.187 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ThASUWk/NvJw1D1mDBR9PaVGRhg= 2023-10-11T15:35:50.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ThASUWk/NvJw1D1mDBR9PaVGRhg=', 'Content-Length': '0'} 2023-10-11T15:35:50.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c61d215a2f00a3-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.190 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.191 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.191 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.191 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.192 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.192 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.193 DEBUG:boto:Data: 2023-10-11T15:35:50.193 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.194 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.194 DEBUG:boto:Params: {} 2023-10-11T15:35:50.195 DEBUG:boto:Token: None 2023-10-11T15:35:50.195 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ThASUWk/NvJw1D1mDBR9PaVGRhg= 2023-10-11T15:35:50.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ThASUWk/NvJw1D1mDBR9PaVGRhg=', 'Content-Length': '0'} 2023-10-11T15:35:50.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a11235e2e45033-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:35:50.199 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.199 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.200 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.200 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.201 DEBUG:boto:Method: GET 2023-10-11T15:35:50.201 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.201 DEBUG:boto:Data: 2023-10-11T15:35:50.202 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.203 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.203 DEBUG:boto:Params: {} 2023-10-11T15:35:50.204 DEBUG:boto:Token: None 2023-10-11T15:35:50.204 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko= 2023-10-11T15:35:50.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko=', 'Content-Length': '0'} 2023-10-11T15:35:50.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec98aa8503671e44-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.207 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.208 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.208 DEBUG:boto:path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.209 DEBUG:boto:auth_path=/dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.209 DEBUG:boto:Method: GET 2023-10-11T15:35:50.210 DEBUG:boto:Path: /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.210 DEBUG:boto:Data: 2023-10-11T15:35:50.211 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.211 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.211 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.212 DEBUG:boto:Params: {} 2023-10-11T15:35:50.212 DEBUG:boto:Token: None 2023-10-11T15:35:50.213 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38?versionId=null 2023-10-11T15:35:50.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko= 2023-10-11T15:35:50.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:S8K25/glHboIVbUXoXEOmO/pgko=', 'Content-Length': '0'} 2023-10-11T15:35:50.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf7ad0a63248553a-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.217 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.217 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.218 DEBUG:boto:path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.218 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.218 DEBUG:boto:Method: GET 2023-10-11T15:35:50.219 DEBUG:boto:Path: /dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.219 DEBUG:boto:Data: 2023-10-11T15:35:50.220 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.221 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.221 DEBUG:boto:Params: {} 2023-10-11T15:35:50.221 DEBUG:boto:Token: None 2023-10-11T15:35:50.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e98a49cb45cafe3-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.266 DEBUG:boto:b'dghqvd-61obj381000falseobj382023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.267 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.268 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.268 DEBUG:boto:path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.269 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.269 DEBUG:boto:Method: GET 2023-10-11T15:35:50.269 DEBUG:boto:Path: /dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.270 DEBUG:boto:Data: 2023-10-11T15:35:50.270 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.271 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.272 DEBUG:boto:Params: {} 2023-10-11T15:35:50.272 DEBUG:boto:Token: None 2023-10-11T15:35:50.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.273 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000899805cb90eb2800-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.278 DEBUG:boto:b'dghqvd-61obj381000falseobj382023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.279 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.280 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.280 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.281 DEBUG:boto:Path: /dghqvd-61/obj38 2023-10-11T15:35:50.281 DEBUG:boto:Data: 2023-10-11T15:35:50.282 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.282 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.282 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.283 DEBUG:boto:Params: {} 2023-10-11T15:35:50.283 DEBUG:boto:Token: None 2023-10-11T15:35:50.284 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38 2023-10-11T15:35:50.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXvBGJHUkyVYhmWRpTrH+iEt8NU= 2023-10-11T15:35:50.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXvBGJHUkyVYhmWRpTrH+iEt8NU=', 'Content-Length': '0'} 2023-10-11T15:35:50.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000053e9f308f93b3a-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.287 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.288 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.288 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.289 DEBUG:boto:Path: /dghqvd-61/obj38 2023-10-11T15:35:50.289 DEBUG:boto:Data: 2023-10-11T15:35:50.289 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.290 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.290 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.291 DEBUG:boto:Params: {} 2023-10-11T15:35:50.291 DEBUG:boto:Token: None 2023-10-11T15:35:50.291 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38 2023-10-11T15:35:50.292 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wXvBGJHUkyVYhmWRpTrH+iEt8NU= 2023-10-11T15:35:50.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wXvBGJHUkyVYhmWRpTrH+iEt8NU=', 'Content-Length': '0'} 2023-10-11T15:35:50.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008200d09dfeff7890-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-10-11T15:35:50.295 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.296 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.296 DEBUG:boto:Method: GET 2023-10-11T15:35:50.297 DEBUG:boto:Path: /dghqvd-61/obj38 2023-10-11T15:35:50.297 DEBUG:boto:Data: 2023-10-11T15:35:50.298 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.299 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.299 DEBUG:boto:Params: {} 2023-10-11T15:35:50.300 DEBUG:boto:Token: None 2023-10-11T15:35:50.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38 2023-10-11T15:35:50.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9/hOPSEZ3GlfyeyuZ2iF8MIqcXE= 2023-10-11T15:35:50.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9/hOPSEZ3GlfyeyuZ2iF8MIqcXE=', 'Content-Length': '0'} 2023-10-11T15:35:50.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefc974dd6746dd2-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.303 DEBUG:boto:path=/dghqvd-61/obj38 2023-10-11T15:35:50.304 DEBUG:boto:auth_path=/dghqvd-61/obj38 2023-10-11T15:35:50.304 DEBUG:boto:Method: GET 2023-10-11T15:35:50.305 DEBUG:boto:Path: /dghqvd-61/obj38 2023-10-11T15:35:50.305 DEBUG:boto:Data: 2023-10-11T15:35:50.306 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.306 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.307 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.307 DEBUG:boto:Params: {} 2023-10-11T15:35:50.307 DEBUG:boto:Token: None 2023-10-11T15:35:50.308 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj38 2023-10-11T15:35:50.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9/hOPSEZ3GlfyeyuZ2iF8MIqcXE= 2023-10-11T15:35:50.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9/hOPSEZ3GlfyeyuZ2iF8MIqcXE=', 'Content-Length': '0'} 2023-10-11T15:35:50.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c9c65ea4d97cf6-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.311 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.312 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.312 DEBUG:boto:path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.313 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.313 DEBUG:boto:Method: GET 2023-10-11T15:35:50.314 DEBUG:boto:Path: /dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.314 DEBUG:boto:Data: 2023-10-11T15:35:50.315 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.315 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.315 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.316 DEBUG:boto:Params: {} 2023-10-11T15:35:50.316 DEBUG:boto:Token: None 2023-10-11T15:35:50.317 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca77b6e7c95a5ab-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.362 DEBUG:boto:b'dghqvd-61obj381000falseobj382023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.364 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.364 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.365 DEBUG:boto:path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.365 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.366 DEBUG:boto:Method: GET 2023-10-11T15:35:50.366 DEBUG:boto:Path: /dghqvd-61/?prefix=obj38 2023-10-11T15:35:50.367 DEBUG:boto:Data: 2023-10-11T15:35:50.367 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.368 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.368 DEBUG:boto:Params: {} 2023-10-11T15:35:50.369 DEBUG:boto:Token: None 2023-10-11T15:35:50.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.370 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046adff19f7d75a9c-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.375 DEBUG:boto:b'dghqvd-61obj381000falseobj382023-10-11T15:31:43.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:35:50.376 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.376 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.377 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.377 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.377 DEBUG:boto:Method: GET 2023-10-11T15:35:50.378 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.378 DEBUG:boto:Data: 2023-10-11T15:35:50.379 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.379 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.380 DEBUG:boto:Params: {} 2023-10-11T15:35:50.380 DEBUG:boto:Token: None 2023-10-11T15:35:50.381 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98= 2023-10-11T15:35:50.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98=', 'Content-Length': '0'} 2023-10-11T15:35:50.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021debe1c45abee21-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.386 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.387 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.388 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.388 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.389 DEBUG:boto:Method: GET 2023-10-11T15:35:50.390 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.391 DEBUG:boto:Data: 2023-10-11T15:35:50.392 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.393 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.394 DEBUG:boto:Params: {} 2023-10-11T15:35:50.395 DEBUG:boto:Token: None 2023-10-11T15:35:50.396 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.397 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98= 2023-10-11T15:35:50.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98=', 'Content-Length': '0'} 2023-10-11T15:35:50.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cb98f3be21850f-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.401 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.402 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.402 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.403 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.404 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.404 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.406 DEBUG:boto:Data: 2023-10-11T15:35:50.406 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.407 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.408 DEBUG:boto:Params: {} 2023-10-11T15:35:50.408 DEBUG:boto:Token: None 2023-10-11T15:35:50.409 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.410 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z6g+bXdH5TDkxH7AziaV1P5fW1w= 2023-10-11T15:35:50.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z6g+bXdH5TDkxH7AziaV1P5fW1w=', 'Content-Length': '0'} 2023-10-11T15:35:50.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daacd54c271f3e68-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.413 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.414 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.414 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.415 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.416 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.416 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.417 DEBUG:boto:Data: 2023-10-11T15:35:50.418 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.419 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.420 DEBUG:boto:Params: {} 2023-10-11T15:35:50.421 DEBUG:boto:Token: None 2023-10-11T15:35:50.422 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.422 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z6g+bXdH5TDkxH7AziaV1P5fW1w= 2023-10-11T15:35:50.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z6g+bXdH5TDkxH7AziaV1P5fW1w=', 'Content-Length': '0'} 2023-10-11T15:35:50.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1bbcc9a5a1b9fd-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:35:50.427 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.428 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.428 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.429 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.430 DEBUG:boto:Method: GET 2023-10-11T15:35:50.431 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.434 DEBUG:boto:Data: 2023-10-11T15:35:50.435 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.435 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.436 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.436 DEBUG:boto:Params: {} 2023-10-11T15:35:50.437 DEBUG:boto:Token: None 2023-10-11T15:35:50.438 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98= 2023-10-11T15:35:50.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98=', 'Content-Length': '0'} 2023-10-11T15:35:50.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4146b9bc12c125-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.443 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.444 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.445 DEBUG:boto:path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.445 DEBUG:boto:auth_path=/dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.446 DEBUG:boto:Method: GET 2023-10-11T15:35:50.447 DEBUG:boto:Path: /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.447 DEBUG:boto:Data: 2023-10-11T15:35:50.448 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.450 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.450 DEBUG:boto:Params: {} 2023-10-11T15:35:50.451 DEBUG:boto:Token: None 2023-10-11T15:35:50.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44?versionId=null 2023-10-11T15:35:50.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98= 2023-10-11T15:35:50.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/1v/sCk/bPKCE1AarXHIl9xNX98=', 'Content-Length': '0'} 2023-10-11T15:35:50.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012eba6bb0e70381c-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.457 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.458 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.458 DEBUG:boto:path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.459 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.459 DEBUG:boto:Method: GET 2023-10-11T15:35:50.460 DEBUG:boto:Path: /dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.461 DEBUG:boto:Data: 2023-10-11T15:35:50.461 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.462 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.462 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.463 DEBUG:boto:Params: {} 2023-10-11T15:35:50.463 DEBUG:boto:Token: None 2023-10-11T15:35:50.464 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.464 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3bf58beffbcbd7c-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.468 DEBUG:boto:b'dghqvd-61obj441000falseobj442023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.469 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.469 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.469 DEBUG:boto:path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.470 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.470 DEBUG:boto:Method: GET 2023-10-11T15:35:50.471 DEBUG:boto:Path: /dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.471 DEBUG:boto:Data: 2023-10-11T15:35:50.472 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.473 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.473 DEBUG:boto:Params: {} 2023-10-11T15:35:50.473 DEBUG:boto:Token: None 2023-10-11T15:35:50.474 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486c3c92d249caba-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.479 DEBUG:boto:b'dghqvd-61obj441000falseobj442023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.480 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.481 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.481 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.482 DEBUG:boto:Path: /dghqvd-61/obj44 2023-10-11T15:35:50.482 DEBUG:boto:Data: 2023-10-11T15:35:50.483 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.483 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.483 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.484 DEBUG:boto:Params: {} 2023-10-11T15:35:50.484 DEBUG:boto:Token: None 2023-10-11T15:35:50.485 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44 2023-10-11T15:35:50.485 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cd+7zkjQ1tS2pjX6JR6aHH31URQ= 2023-10-11T15:35:50.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cd+7zkjQ1tS2pjX6JR6aHH31URQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3a864739a16e95b-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.488 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.489 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.489 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.489 DEBUG:boto:Path: /dghqvd-61/obj44 2023-10-11T15:35:50.490 DEBUG:boto:Data: 2023-10-11T15:35:50.490 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.491 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.491 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.492 DEBUG:boto:Params: {} 2023-10-11T15:35:50.493 DEBUG:boto:Token: None 2023-10-11T15:35:50.494 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44 2023-10-11T15:35:50.495 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cd+7zkjQ1tS2pjX6JR6aHH31URQ= 2023-10-11T15:35:50.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cd+7zkjQ1tS2pjX6JR6aHH31URQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b871cd218cf70d4-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:35:50.499 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.500 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.501 DEBUG:boto:Method: GET 2023-10-11T15:35:50.502 DEBUG:boto:Path: /dghqvd-61/obj44 2023-10-11T15:35:50.502 DEBUG:boto:Data: 2023-10-11T15:35:50.503 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.504 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.505 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.505 DEBUG:boto:Params: {} 2023-10-11T15:35:50.506 DEBUG:boto:Token: None 2023-10-11T15:35:50.507 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44 2023-10-11T15:35:50.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GSABsTnKLiD4X6GE1enFHo3ZHnU= 2023-10-11T15:35:50.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GSABsTnKLiD4X6GE1enFHo3ZHnU=', 'Content-Length': '0'} 2023-10-11T15:35:50.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001341c0ba36a17ad0-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.558 DEBUG:boto:path=/dghqvd-61/obj44 2023-10-11T15:35:50.559 DEBUG:boto:auth_path=/dghqvd-61/obj44 2023-10-11T15:35:50.560 DEBUG:boto:Method: GET 2023-10-11T15:35:50.560 DEBUG:boto:Path: /dghqvd-61/obj44 2023-10-11T15:35:50.560 DEBUG:boto:Data: 2023-10-11T15:35:50.561 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.561 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.562 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.562 DEBUG:boto:Params: {} 2023-10-11T15:35:50.563 DEBUG:boto:Token: None 2023-10-11T15:35:50.564 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj44 2023-10-11T15:35:50.564 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:GSABsTnKLiD4X6GE1enFHo3ZHnU= 2023-10-11T15:35:50.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GSABsTnKLiD4X6GE1enFHo3ZHnU=', 'Content-Length': '0'} 2023-10-11T15:35:50.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b323e036a41d7831-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.568 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.568 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.569 DEBUG:boto:path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.569 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.570 DEBUG:boto:Method: GET 2023-10-11T15:35:50.571 DEBUG:boto:Path: /dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.571 DEBUG:boto:Data: 2023-10-11T15:35:50.572 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.572 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.573 DEBUG:boto:Params: {} 2023-10-11T15:35:50.573 DEBUG:boto:Token: None 2023-10-11T15:35:50.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089770199a23a5b6a-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.578 DEBUG:boto:b'dghqvd-61obj441000falseobj442023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.579 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.579 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.580 DEBUG:boto:path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.580 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.581 DEBUG:boto:Method: GET 2023-10-11T15:35:50.581 DEBUG:boto:Path: /dghqvd-61/?prefix=obj44 2023-10-11T15:35:50.582 DEBUG:boto:Data: 2023-10-11T15:35:50.582 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.583 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.583 DEBUG:boto:Params: {} 2023-10-11T15:35:50.584 DEBUG:boto:Token: None 2023-10-11T15:35:50.584 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c9fbc2ac3782d81-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.590 DEBUG:boto:b'dghqvd-61obj441000falseobj442023-10-11T15:31:42.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:35:50.592 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.592 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.592 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.593 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.593 DEBUG:boto:Method: GET 2023-10-11T15:35:50.594 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.594 DEBUG:boto:Data: 2023-10-11T15:35:50.595 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.595 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.596 DEBUG:boto:Params: {} 2023-10-11T15:35:50.596 DEBUG:boto:Token: None 2023-10-11T15:35:50.597 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4= 2023-10-11T15:35:50.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4=', 'Content-Length': '0'} 2023-10-11T15:35:50.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029266efd0fe839a3-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.601 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.601 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.602 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.602 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.603 DEBUG:boto:Method: GET 2023-10-11T15:35:50.603 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.604 DEBUG:boto:Data: 2023-10-11T15:35:50.604 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.605 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.605 DEBUG:boto:Params: {} 2023-10-11T15:35:50.606 DEBUG:boto:Token: None 2023-10-11T15:35:50.607 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.608 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4= 2023-10-11T15:35:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4=', 'Content-Length': '0'} 2023-10-11T15:35:50.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec465afccce946fa-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.611 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.611 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.612 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.612 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.612 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.613 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.614 DEBUG:boto:Data: 2023-10-11T15:35:50.614 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.614 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.615 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.615 DEBUG:boto:Params: {} 2023-10-11T15:35:50.616 DEBUG:boto:Token: None 2023-10-11T15:35:50.616 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.617 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5PawhFHPmjtl+wcqV3w68K7XApE= 2023-10-11T15:35:50.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5PawhFHPmjtl+wcqV3w68K7XApE=', 'Content-Length': '0'} 2023-10-11T15:35:50.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b959707dd86dc9bb-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.621 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.621 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.621 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.622 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.622 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.623 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.623 DEBUG:boto:Data: 2023-10-11T15:35:50.624 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.624 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.625 DEBUG:boto:Params: {} 2023-10-11T15:35:50.626 DEBUG:boto:Token: None 2023-10-11T15:35:50.626 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.627 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5PawhFHPmjtl+wcqV3w68K7XApE= 2023-10-11T15:35:50.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5PawhFHPmjtl+wcqV3w68K7XApE=', 'Content-Length': '0'} 2023-10-11T15:35:50.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c116a3a268fe68-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:35:50.630 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.631 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.631 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.631 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.632 DEBUG:boto:Method: GET 2023-10-11T15:35:50.632 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.633 DEBUG:boto:Data: 2023-10-11T15:35:50.633 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.634 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.634 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.634 DEBUG:boto:Params: {} 2023-10-11T15:35:50.635 DEBUG:boto:Token: None 2023-10-11T15:35:50.636 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4= 2023-10-11T15:35:50.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4=', 'Content-Length': '0'} 2023-10-11T15:35:50.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f21e2b3d6ce059d-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.639 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.640 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.640 DEBUG:boto:path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.641 DEBUG:boto:auth_path=/dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.641 DEBUG:boto:Method: GET 2023-10-11T15:35:50.641 DEBUG:boto:Path: /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.642 DEBUG:boto:Data: 2023-10-11T15:35:50.642 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.643 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.643 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.644 DEBUG:boto:Params: {} 2023-10-11T15:35:50.644 DEBUG:boto:Token: None 2023-10-11T15:35:50.644 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46?versionId=null 2023-10-11T15:35:50.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4= 2023-10-11T15:35:50.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Aq4zjmziQXnuAKXEwCthsbCOXm4=', 'Content-Length': '0'} 2023-10-11T15:35:50.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9c6786c23324cb-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.649 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.649 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.649 DEBUG:boto:path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.650 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.650 DEBUG:boto:Method: GET 2023-10-11T15:35:50.651 DEBUG:boto:Path: /dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.651 DEBUG:boto:Data: 2023-10-11T15:35:50.651 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.652 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.652 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.653 DEBUG:boto:Params: {} 2023-10-11T15:35:50.653 DEBUG:boto:Token: None 2023-10-11T15:35:50.654 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.654 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c69f35ace27f60a-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.698 DEBUG:boto:b'dghqvd-61obj461000falseobj462023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.699 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.700 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.700 DEBUG:boto:path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.701 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.701 DEBUG:boto:Method: GET 2023-10-11T15:35:50.701 DEBUG:boto:Path: /dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.702 DEBUG:boto:Data: 2023-10-11T15:35:50.702 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.703 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.703 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.704 DEBUG:boto:Params: {} 2023-10-11T15:35:50.704 DEBUG:boto:Token: None 2023-10-11T15:35:50.705 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.705 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfae92ba5db0539-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.710 DEBUG:boto:b'dghqvd-61obj461000falseobj462023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.711 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.711 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.712 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.712 DEBUG:boto:Path: /dghqvd-61/obj46 2023-10-11T15:35:50.713 DEBUG:boto:Data: 2023-10-11T15:35:50.713 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.714 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.714 DEBUG:boto:Params: {} 2023-10-11T15:35:50.715 DEBUG:boto:Token: None 2023-10-11T15:35:50.715 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46 2023-10-11T15:35:50.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gpWyTC1oYjtIC7WwtuMzt4NO2t0= 2023-10-11T15:35:50.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gpWyTC1oYjtIC7WwtuMzt4NO2t0=', 'Content-Length': '0'} 2023-10-11T15:35:50.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09411024f84fb21-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.719 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.719 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.720 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.720 DEBUG:boto:Path: /dghqvd-61/obj46 2023-10-11T15:35:50.720 DEBUG:boto:Data: 2023-10-11T15:35:50.721 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.721 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.722 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.722 DEBUG:boto:Params: {} 2023-10-11T15:35:50.722 DEBUG:boto:Token: None 2023-10-11T15:35:50.723 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46 2023-10-11T15:35:50.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gpWyTC1oYjtIC7WwtuMzt4NO2t0= 2023-10-11T15:35:50.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gpWyTC1oYjtIC7WwtuMzt4NO2t0=', 'Content-Length': '0'} 2023-10-11T15:35:50.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bd001dcbb373b2-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-10-11T15:35:50.727 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.730 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.730 DEBUG:boto:Method: GET 2023-10-11T15:35:50.731 DEBUG:boto:Path: /dghqvd-61/obj46 2023-10-11T15:35:50.731 DEBUG:boto:Data: 2023-10-11T15:35:50.732 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.732 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.733 DEBUG:boto:Params: {} 2023-10-11T15:35:50.733 DEBUG:boto:Token: None 2023-10-11T15:35:50.734 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46 2023-10-11T15:35:50.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YemXxyYXKay4ETvDd6k1m9W2Lhw= 2023-10-11T15:35:50.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YemXxyYXKay4ETvDd6k1m9W2Lhw=', 'Content-Length': '0'} 2023-10-11T15:35:50.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab4ad8fcd0d5644-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.738 DEBUG:boto:path=/dghqvd-61/obj46 2023-10-11T15:35:50.738 DEBUG:boto:auth_path=/dghqvd-61/obj46 2023-10-11T15:35:50.739 DEBUG:boto:Method: GET 2023-10-11T15:35:50.740 DEBUG:boto:Path: /dghqvd-61/obj46 2023-10-11T15:35:50.740 DEBUG:boto:Data: 2023-10-11T15:35:50.741 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.741 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.742 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.742 DEBUG:boto:Params: {} 2023-10-11T15:35:50.742 DEBUG:boto:Token: None 2023-10-11T15:35:50.743 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj46 2023-10-11T15:35:50.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YemXxyYXKay4ETvDd6k1m9W2Lhw= 2023-10-11T15:35:50.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YemXxyYXKay4ETvDd6k1m9W2Lhw=', 'Content-Length': '0'} 2023-10-11T15:35:50.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018742b5083bf1180-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.747 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.747 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.748 DEBUG:boto:path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.748 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.748 DEBUG:boto:Method: GET 2023-10-11T15:35:50.749 DEBUG:boto:Path: /dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.749 DEBUG:boto:Data: 2023-10-11T15:35:50.750 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.750 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.751 DEBUG:boto:Params: {} 2023-10-11T15:35:50.751 DEBUG:boto:Token: None 2023-10-11T15:35:50.752 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f45c39a3d6dcf2-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.756 DEBUG:boto:b'dghqvd-61obj461000falseobj462023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.756 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.757 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.757 DEBUG:boto:path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.758 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.758 DEBUG:boto:Method: GET 2023-10-11T15:35:50.758 DEBUG:boto:Path: /dghqvd-61/?prefix=obj46 2023-10-11T15:35:50.759 DEBUG:boto:Data: 2023-10-11T15:35:50.759 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.760 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.760 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.760 DEBUG:boto:Params: {} 2023-10-11T15:35:50.761 DEBUG:boto:Token: None 2023-10-11T15:35:50.761 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.762 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d126499d106fb7-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.767 DEBUG:boto:b'dghqvd-61obj461000falseobj462023-10-11T15:31:43.443Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:35:50.768 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.768 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.769 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.769 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.769 DEBUG:boto:Method: GET 2023-10-11T15:35:50.770 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.770 DEBUG:boto:Data: 2023-10-11T15:35:50.771 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.771 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.772 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.772 DEBUG:boto:Params: {} 2023-10-11T15:35:50.772 DEBUG:boto:Token: None 2023-10-11T15:35:50.773 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.773 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk= 2023-10-11T15:35:50.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk=', 'Content-Length': '0'} 2023-10-11T15:35:50.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08929ab57fbe268-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.776 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.777 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.777 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.778 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.778 DEBUG:boto:Method: GET 2023-10-11T15:35:50.779 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.779 DEBUG:boto:Data: 2023-10-11T15:35:50.779 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.780 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.780 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.781 DEBUG:boto:Params: {} 2023-10-11T15:35:50.781 DEBUG:boto:Token: None 2023-10-11T15:35:50.782 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk= 2023-10-11T15:35:50.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk=', 'Content-Length': '0'} 2023-10-11T15:35:50.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc096c47cd4e0b0-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.785 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.785 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.786 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.786 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.787 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.787 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.787 DEBUG:boto:Data: 2023-10-11T15:35:50.788 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.788 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.789 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.789 DEBUG:boto:Params: {} 2023-10-11T15:35:50.789 DEBUG:boto:Token: None 2023-10-11T15:35:50.790 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.790 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMUOC/rwcZbeUAPbMDW8R2sRGfI= 2023-10-11T15:35:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMUOC/rwcZbeUAPbMDW8R2sRGfI=', 'Content-Length': '0'} 2023-10-11T15:35:50.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033135c0e2bb47dbb-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.793 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.793 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.794 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.794 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.795 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.795 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.795 DEBUG:boto:Data: 2023-10-11T15:35:50.796 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.797 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.797 DEBUG:boto:Params: {} 2023-10-11T15:35:50.797 DEBUG:boto:Token: None 2023-10-11T15:35:50.798 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WMUOC/rwcZbeUAPbMDW8R2sRGfI= 2023-10-11T15:35:50.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WMUOC/rwcZbeUAPbMDW8R2sRGfI=', 'Content-Length': '0'} 2023-10-11T15:35:50.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdfefc6166dda8c2-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:35:50.802 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.802 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.803 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.803 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.803 DEBUG:boto:Method: GET 2023-10-11T15:35:50.804 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.804 DEBUG:boto:Data: 2023-10-11T15:35:50.805 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.805 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.806 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.806 DEBUG:boto:Params: {} 2023-10-11T15:35:50.807 DEBUG:boto:Token: None 2023-10-11T15:35:50.807 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.807 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk= 2023-10-11T15:35:50.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk=', 'Content-Length': '0'} 2023-10-11T15:35:50.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c3412ad6cb32eb-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.810 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.811 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.812 DEBUG:boto:path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.812 DEBUG:boto:auth_path=/dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.812 DEBUG:boto:Method: GET 2023-10-11T15:35:50.813 DEBUG:boto:Path: /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.813 DEBUG:boto:Data: 2023-10-11T15:35:50.814 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.814 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.815 DEBUG:boto:Params: {} 2023-10-11T15:35:50.815 DEBUG:boto:Token: None 2023-10-11T15:35:50.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51?versionId=null 2023-10-11T15:35:50.816 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk= 2023-10-11T15:35:50.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PLKNA2aD0UHTLZ/rZCl++K1W3Zk=', 'Content-Length': '0'} 2023-10-11T15:35:50.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf3060e665cff3c-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.819 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.820 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.820 DEBUG:boto:path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.821 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.821 DEBUG:boto:Method: GET 2023-10-11T15:35:50.821 DEBUG:boto:Path: /dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.822 DEBUG:boto:Data: 2023-10-11T15:35:50.822 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.823 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.824 DEBUG:boto:Params: {} 2023-10-11T15:35:50.824 DEBUG:boto:Token: None 2023-10-11T15:35:50.825 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8133db64e31255-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.828 DEBUG:boto:b'dghqvd-61obj511000falseobj512023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.829 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.830 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.830 DEBUG:boto:path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.831 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.831 DEBUG:boto:Method: GET 2023-10-11T15:35:50.831 DEBUG:boto:Path: /dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.832 DEBUG:boto:Data: 2023-10-11T15:35:50.832 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.833 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.833 DEBUG:boto:Params: {} 2023-10-11T15:35:50.834 DEBUG:boto:Token: None 2023-10-11T15:35:50.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dba9e55d6f7e9755-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.886 DEBUG:boto:b'dghqvd-61obj511000falseobj512023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.887 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.888 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.888 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.888 DEBUG:boto:Path: /dghqvd-61/obj51 2023-10-11T15:35:50.889 DEBUG:boto:Data: 2023-10-11T15:35:50.889 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.890 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.890 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.891 DEBUG:boto:Params: {} 2023-10-11T15:35:50.891 DEBUG:boto:Token: None 2023-10-11T15:35:50.892 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51 2023-10-11T15:35:50.892 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fOwPO9yG1mJfQvqs1U/uFsZ21ho= 2023-10-11T15:35:50.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fOwPO9yG1mJfQvqs1U/uFsZ21ho=', 'Content-Length': '0'} 2023-10-11T15:35:50.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d395598495fe1751-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.895 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.896 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.896 DEBUG:boto:Method: HEAD 2023-10-11T15:35:50.896 DEBUG:boto:Path: /dghqvd-61/obj51 2023-10-11T15:35:50.897 DEBUG:boto:Data: 2023-10-11T15:35:50.897 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.898 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.898 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.898 DEBUG:boto:Params: {} 2023-10-11T15:35:50.899 DEBUG:boto:Token: None 2023-10-11T15:35:50.899 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51 2023-10-11T15:35:50.900 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fOwPO9yG1mJfQvqs1U/uFsZ21ho= 2023-10-11T15:35:50.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fOwPO9yG1mJfQvqs1U/uFsZ21ho=', 'Content-Length': '0'} 2023-10-11T15:35:50.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002149d892692e862-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-10-11T15:35:50.903 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.904 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.904 DEBUG:boto:Method: GET 2023-10-11T15:35:50.905 DEBUG:boto:Path: /dghqvd-61/obj51 2023-10-11T15:35:50.905 DEBUG:boto:Data: 2023-10-11T15:35:50.905 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.906 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.906 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.907 DEBUG:boto:Params: {} 2023-10-11T15:35:50.907 DEBUG:boto:Token: None 2023-10-11T15:35:50.908 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51 2023-10-11T15:35:50.908 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YZrKR4GE6Wv34QUNVuEcbVP8wHQ= 2023-10-11T15:35:50.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YZrKR4GE6Wv34QUNVuEcbVP8wHQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee29a9d62a0b1f3-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.958 DEBUG:boto:path=/dghqvd-61/obj51 2023-10-11T15:35:50.959 DEBUG:boto:auth_path=/dghqvd-61/obj51 2023-10-11T15:35:50.960 DEBUG:boto:Method: GET 2023-10-11T15:35:50.960 DEBUG:boto:Path: /dghqvd-61/obj51 2023-10-11T15:35:50.961 DEBUG:boto:Data: 2023-10-11T15:35:50.961 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.961 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.962 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.962 DEBUG:boto:Params: {} 2023-10-11T15:35:50.963 DEBUG:boto:Token: None 2023-10-11T15:35:50.963 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj51 2023-10-11T15:35:50.964 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YZrKR4GE6Wv34QUNVuEcbVP8wHQ= 2023-10-11T15:35:50.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YZrKR4GE6Wv34QUNVuEcbVP8wHQ=', 'Content-Length': '0'} 2023-10-11T15:35:50.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf59189e875fed5-006526c0d6-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.967 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.968 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.968 DEBUG:boto:path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.969 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.969 DEBUG:boto:Method: GET 2023-10-11T15:35:50.970 DEBUG:boto:Path: /dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.971 DEBUG:boto:Data: 2023-10-11T15:35:50.971 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.972 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.972 DEBUG:boto:Params: {} 2023-10-11T15:35:50.973 DEBUG:boto:Token: None 2023-10-11T15:35:50.973 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005023b22bee7f5d84-006526c0d6-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.978 DEBUG:boto:b'dghqvd-61obj511000falseobj512023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.979 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:50.979 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:50.980 DEBUG:boto:path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.980 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.981 DEBUG:boto:Method: GET 2023-10-11T15:35:50.981 DEBUG:boto:Path: /dghqvd-61/?prefix=obj51 2023-10-11T15:35:50.982 DEBUG:boto:Data: 2023-10-11T15:35:50.982 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.982 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:50.983 DEBUG:boto:Port: 8000 2023-10-11T15:35:50.983 DEBUG:boto:Params: {} 2023-10-11T15:35:50.984 DEBUG:boto:Token: None 2023-10-11T15:35:50.984 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/ 2023-10-11T15:35:50.985 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk= 2023-10-11T15:35:50.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:709PXsZ+QPRXgjvr10Bm/tNxbvk=', 'Content-Length': '0'} 2023-10-11T15:35:50.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baeef8895245012e-006526c0d6-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:50.990 DEBUG:boto:b'dghqvd-61obj511000falseobj512023-10-11T15:31:42.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:50.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:35:50.991 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:50.992 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:50.992 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:50.993 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:50.993 DEBUG:boto:Method: GET 2023-10-11T15:35:50.994 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:50.994 DEBUG:boto:Data: 2023-10-11T15:35:50.994 DEBUG:boto:Headers: {} 2023-10-11T15:35:50.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:50.996 DEBUG:boto:Port: 8001 2023-10-11T15:35:50.997 DEBUG:boto:Params: {} 2023-10-11T15:35:50.997 DEBUG:boto:Token: None 2023-10-11T15:35:50.998 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:50 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:50.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jf7unKlComFI6/7ln6BRMN1kwJU= 2023-10-11T15:35:50.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:50 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jf7unKlComFI6/7ln6BRMN1kwJU=', 'Content-Length': '0'} 2023-10-11T15:35:51.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8ccd2cc3b31cfb-006526c0d6-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:50 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.002 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.002 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.003 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.003 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.003 DEBUG:boto:Method: GET 2023-10-11T15:35:51.004 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.004 DEBUG:boto:Data: 2023-10-11T15:35:51.005 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.006 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.006 DEBUG:boto:Params: {} 2023-10-11T15:35:51.007 DEBUG:boto:Token: None 2023-10-11T15:35:51.007 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw= 2023-10-11T15:35:51.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw=', 'Content-Length': '0'} 2023-10-11T15:35:51.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd4fed089f7be8f-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.010 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.011 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.011 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.012 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.012 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.012 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.013 DEBUG:boto:Data: 2023-10-11T15:35:51.013 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.014 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.014 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.014 DEBUG:boto:Params: {} 2023-10-11T15:35:51.015 DEBUG:boto:Token: None 2023-10-11T15:35:51.015 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.016 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChrSknYMFR4fPsYc/UvRr8KnzRc= 2023-10-11T15:35:51.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChrSknYMFR4fPsYc/UvRr8KnzRc=', 'Content-Length': '0'} 2023-10-11T15:35:51.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a611e4c72940ae0e-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.019 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.019 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.019 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.020 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.020 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.021 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.021 DEBUG:boto:Data: 2023-10-11T15:35:51.021 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.022 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.022 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.023 DEBUG:boto:Params: {} 2023-10-11T15:35:51.023 DEBUG:boto:Token: None 2023-10-11T15:35:51.024 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ChrSknYMFR4fPsYc/UvRr8KnzRc= 2023-10-11T15:35:51.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ChrSknYMFR4fPsYc/UvRr8KnzRc=', 'Content-Length': '0'} 2023-10-11T15:35:51.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9d1e3d5fe4e963-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:35:51.028 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.028 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.029 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.029 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.030 DEBUG:boto:Method: GET 2023-10-11T15:35:51.030 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.031 DEBUG:boto:Data: 2023-10-11T15:35:51.031 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.032 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.032 DEBUG:boto:Params: {} 2023-10-11T15:35:51.033 DEBUG:boto:Token: None 2023-10-11T15:35:51.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw= 2023-10-11T15:35:51.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw=', 'Content-Length': '0'} 2023-10-11T15:35:51.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087736cfbb82489ad-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.037 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.037 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.038 DEBUG:boto:path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.038 DEBUG:boto:auth_path=/dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.038 DEBUG:boto:Method: GET 2023-10-11T15:35:51.039 DEBUG:boto:Path: /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.039 DEBUG:boto:Data: 2023-10-11T15:35:51.040 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.040 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.041 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.041 DEBUG:boto:Params: {} 2023-10-11T15:35:51.041 DEBUG:boto:Token: None 2023-10-11T15:35:51.042 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55?versionId=null 2023-10-11T15:35:51.042 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw= 2023-10-11T15:35:51.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4F9znK2BIytzX+u1YH/YrP+rRFw=', 'Content-Length': '0'} 2023-10-11T15:35:51.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000791b01ccd3bc30b9-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.086 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.087 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.088 DEBUG:boto:path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.088 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.088 DEBUG:boto:Method: GET 2023-10-11T15:35:51.089 DEBUG:boto:Path: /dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.089 DEBUG:boto:Data: 2023-10-11T15:35:51.090 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.090 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.091 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.091 DEBUG:boto:Params: {} 2023-10-11T15:35:51.092 DEBUG:boto:Token: None 2023-10-11T15:35:51.092 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.093 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002587d5955b8fa5dc-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.096 DEBUG:boto:b'dghqvd-61obj551000falseobj552023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.097 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.098 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.098 DEBUG:boto:path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.099 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.099 DEBUG:boto:Method: GET 2023-10-11T15:35:51.099 DEBUG:boto:Path: /dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.100 DEBUG:boto:Data: 2023-10-11T15:35:51.100 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.101 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.101 DEBUG:boto:Params: {} 2023-10-11T15:35:51.102 DEBUG:boto:Token: None 2023-10-11T15:35:51.102 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012e0d4fa81fa14ec-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.108 DEBUG:boto:b'dghqvd-61obj551000falseobj552023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.109 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.109 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.110 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.110 DEBUG:boto:Path: /dghqvd-61/obj55 2023-10-11T15:35:51.110 DEBUG:boto:Data: 2023-10-11T15:35:51.111 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.112 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.112 DEBUG:boto:Params: {} 2023-10-11T15:35:51.113 DEBUG:boto:Token: None 2023-10-11T15:35:51.113 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55 2023-10-11T15:35:51.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E7sCVxlvCnLcImT1rWm/3YVfcjY= 2023-10-11T15:35:51.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E7sCVxlvCnLcImT1rWm/3YVfcjY=', 'Content-Length': '0'} 2023-10-11T15:35:51.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d097d1dffc54693-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.117 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.117 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.118 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.118 DEBUG:boto:Path: /dghqvd-61/obj55 2023-10-11T15:35:51.118 DEBUG:boto:Data: 2023-10-11T15:35:51.119 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.120 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.120 DEBUG:boto:Params: {} 2023-10-11T15:35:51.121 DEBUG:boto:Token: None 2023-10-11T15:35:51.121 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55 2023-10-11T15:35:51.121 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E7sCVxlvCnLcImT1rWm/3YVfcjY= 2023-10-11T15:35:51.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E7sCVxlvCnLcImT1rWm/3YVfcjY=', 'Content-Length': '0'} 2023-10-11T15:35:51.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64820f0d9b844f3-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:35:51.125 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.125 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.126 DEBUG:boto:Method: GET 2023-10-11T15:35:51.126 DEBUG:boto:Path: /dghqvd-61/obj55 2023-10-11T15:35:51.127 DEBUG:boto:Data: 2023-10-11T15:35:51.127 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.127 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.128 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.128 DEBUG:boto:Params: {} 2023-10-11T15:35:51.129 DEBUG:boto:Token: None 2023-10-11T15:35:51.129 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55 2023-10-11T15:35:51.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M5yB0BjAKsUOnwyQBPdeEf/ADvY= 2023-10-11T15:35:51.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M5yB0BjAKsUOnwyQBPdeEf/ADvY=', 'Content-Length': '0'} 2023-10-11T15:35:51.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a165434ef8b952ed-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.133 DEBUG:boto:path=/dghqvd-61/obj55 2023-10-11T15:35:51.134 DEBUG:boto:auth_path=/dghqvd-61/obj55 2023-10-11T15:35:51.134 DEBUG:boto:Method: GET 2023-10-11T15:35:51.135 DEBUG:boto:Path: /dghqvd-61/obj55 2023-10-11T15:35:51.135 DEBUG:boto:Data: 2023-10-11T15:35:51.135 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.136 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.137 DEBUG:boto:Params: {} 2023-10-11T15:35:51.137 DEBUG:boto:Token: None 2023-10-11T15:35:51.138 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj55 2023-10-11T15:35:51.138 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:M5yB0BjAKsUOnwyQBPdeEf/ADvY= 2023-10-11T15:35:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:M5yB0BjAKsUOnwyQBPdeEf/ADvY=', 'Content-Length': '0'} 2023-10-11T15:35:51.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716457109f9058d2-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.141 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.141 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.142 DEBUG:boto:path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.142 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.143 DEBUG:boto:Method: GET 2023-10-11T15:35:51.143 DEBUG:boto:Path: /dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.145 DEBUG:boto:Data: 2023-10-11T15:35:51.146 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.146 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.147 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.147 DEBUG:boto:Params: {} 2023-10-11T15:35:51.147 DEBUG:boto:Token: None 2023-10-11T15:35:51.148 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011f939ceba9ff6ec-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.152 DEBUG:boto:b'dghqvd-61obj551000falseobj552023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.153 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.153 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.154 DEBUG:boto:path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.154 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.154 DEBUG:boto:Method: GET 2023-10-11T15:35:51.155 DEBUG:boto:Path: /dghqvd-61/?prefix=obj55 2023-10-11T15:35:51.155 DEBUG:boto:Data: 2023-10-11T15:35:51.156 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.157 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.157 DEBUG:boto:Params: {} 2023-10-11T15:35:51.158 DEBUG:boto:Token: None 2023-10-11T15:35:51.158 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.159 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beeae82d64b936ac-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.163 DEBUG:boto:b'dghqvd-61obj551000falseobj552023-10-11T15:31:42.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:35:51.164 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.165 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.165 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.166 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.166 DEBUG:boto:Method: GET 2023-10-11T15:35:51.167 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.167 DEBUG:boto:Data: 2023-10-11T15:35:51.167 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.168 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.168 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.169 DEBUG:boto:Params: {} 2023-10-11T15:35:51.169 DEBUG:boto:Token: None 2023-10-11T15:35:51.170 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.170 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw= 2023-10-11T15:35:51.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw=', 'Content-Length': '0'} 2023-10-11T15:35:51.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006877998ae6263694-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.173 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.173 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.174 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.174 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.175 DEBUG:boto:Method: GET 2023-10-11T15:35:51.175 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.176 DEBUG:boto:Data: 2023-10-11T15:35:51.176 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.177 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.177 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.178 DEBUG:boto:Params: {} 2023-10-11T15:35:51.178 DEBUG:boto:Token: None 2023-10-11T15:35:51.179 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.179 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw= 2023-10-11T15:35:51.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw=', 'Content-Length': '0'} 2023-10-11T15:35:51.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024cad7066eab661-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.222 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.223 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.223 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.224 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.224 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.225 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.225 DEBUG:boto:Data: 2023-10-11T15:35:51.226 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.227 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.227 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.228 DEBUG:boto:Params: {} 2023-10-11T15:35:51.229 DEBUG:boto:Token: None 2023-10-11T15:35:51.230 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjZW8KSOVYrN2Oc8XKIH+SM08iU= 2023-10-11T15:35:51.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjZW8KSOVYrN2Oc8XKIH+SM08iU=', 'Content-Length': '0'} 2023-10-11T15:35:51.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3903b969632f88f-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.233 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.234 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.234 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.235 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.235 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.236 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.236 DEBUG:boto:Data: 2023-10-11T15:35:51.237 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.238 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.238 DEBUG:boto:Params: {} 2023-10-11T15:35:51.239 DEBUG:boto:Token: None 2023-10-11T15:35:51.239 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.240 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KjZW8KSOVYrN2Oc8XKIH+SM08iU= 2023-10-11T15:35:51.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KjZW8KSOVYrN2Oc8XKIH+SM08iU=', 'Content-Length': '0'} 2023-10-11T15:35:51.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a6aa6959640162-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:35:51.243 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.244 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.245 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.245 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.246 DEBUG:boto:Method: GET 2023-10-11T15:35:51.247 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.248 DEBUG:boto:Data: 2023-10-11T15:35:51.248 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.249 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.250 DEBUG:boto:Params: {} 2023-10-11T15:35:51.250 DEBUG:boto:Token: None 2023-10-11T15:35:51.251 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw= 2023-10-11T15:35:51.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw=', 'Content-Length': '0'} 2023-10-11T15:35:51.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2b762b8e0a30a1-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.255 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.256 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.257 DEBUG:boto:path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.258 DEBUG:boto:auth_path=/dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.259 DEBUG:boto:Method: GET 2023-10-11T15:35:51.260 DEBUG:boto:Path: /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.260 DEBUG:boto:Data: 2023-10-11T15:35:51.261 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.262 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.263 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.263 DEBUG:boto:Params: {} 2023-10-11T15:35:51.264 DEBUG:boto:Token: None 2023-10-11T15:35:51.265 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57?versionId=null 2023-10-11T15:35:51.265 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw= 2023-10-11T15:35:51.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H1jDWLKbKEBeMFWN9tHyPLKPbtw=', 'Content-Length': '0'} 2023-10-11T15:35:51.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03b82fd453f0c45-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.269 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.269 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.270 DEBUG:boto:path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.270 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.270 DEBUG:boto:Method: GET 2023-10-11T15:35:51.271 DEBUG:boto:Path: /dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.271 DEBUG:boto:Data: 2023-10-11T15:35:51.272 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.272 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.273 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.273 DEBUG:boto:Params: {} 2023-10-11T15:35:51.273 DEBUG:boto:Token: None 2023-10-11T15:35:51.274 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b79d629c9d1d129-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.279 DEBUG:boto:b'dghqvd-61obj571000falseobj572023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.280 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.280 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.281 DEBUG:boto:path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.281 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.282 DEBUG:boto:Method: GET 2023-10-11T15:35:51.282 DEBUG:boto:Path: /dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.283 DEBUG:boto:Data: 2023-10-11T15:35:51.283 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.284 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.284 DEBUG:boto:Params: {} 2023-10-11T15:35:51.285 DEBUG:boto:Token: None 2023-10-11T15:35:51.285 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040cdecf49bc8e53e-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.291 DEBUG:boto:b'dghqvd-61obj571000falseobj572023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.291 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.292 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.292 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.293 DEBUG:boto:Path: /dghqvd-61/obj57 2023-10-11T15:35:51.293 DEBUG:boto:Data: 2023-10-11T15:35:51.293 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.294 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.295 DEBUG:boto:Params: {} 2023-10-11T15:35:51.296 DEBUG:boto:Token: None 2023-10-11T15:35:51.296 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57 2023-10-11T15:35:51.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I/mktqtgsil3Rbs729P9pWo6ENM= 2023-10-11T15:35:51.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I/mktqtgsil3Rbs729P9pWo6ENM=', 'Content-Length': '0'} 2023-10-11T15:35:51.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b719341a52f56829-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.301 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.302 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.302 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.302 DEBUG:boto:Path: /dghqvd-61/obj57 2023-10-11T15:35:51.303 DEBUG:boto:Data: 2023-10-11T15:35:51.303 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.304 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.305 DEBUG:boto:Params: {} 2023-10-11T15:35:51.305 DEBUG:boto:Token: None 2023-10-11T15:35:51.306 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57 2023-10-11T15:35:51.307 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I/mktqtgsil3Rbs729P9pWo6ENM= 2023-10-11T15:35:51.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I/mktqtgsil3Rbs729P9pWo6ENM=', 'Content-Length': '0'} 2023-10-11T15:35:51.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcac1df39d175340-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-10-11T15:35:51.310 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.310 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.311 DEBUG:boto:Method: GET 2023-10-11T15:35:51.311 DEBUG:boto:Path: /dghqvd-61/obj57 2023-10-11T15:35:51.311 DEBUG:boto:Data: 2023-10-11T15:35:51.312 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.313 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.313 DEBUG:boto:Params: {} 2023-10-11T15:35:51.313 DEBUG:boto:Token: None 2023-10-11T15:35:51.314 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57 2023-10-11T15:35:51.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RXu84t50d/M8snf7cmjWFh1uNzU= 2023-10-11T15:35:51.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RXu84t50d/M8snf7cmjWFh1uNzU=', 'Content-Length': '0'} 2023-10-11T15:35:51.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb99a9fddc8ed1d-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.317 DEBUG:boto:path=/dghqvd-61/obj57 2023-10-11T15:35:51.318 DEBUG:boto:auth_path=/dghqvd-61/obj57 2023-10-11T15:35:51.318 DEBUG:boto:Method: GET 2023-10-11T15:35:51.319 DEBUG:boto:Path: /dghqvd-61/obj57 2023-10-11T15:35:51.319 DEBUG:boto:Data: 2023-10-11T15:35:51.319 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.320 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.321 DEBUG:boto:Params: {} 2023-10-11T15:35:51.321 DEBUG:boto:Token: None 2023-10-11T15:35:51.322 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj57 2023-10-11T15:35:51.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RXu84t50d/M8snf7cmjWFh1uNzU= 2023-10-11T15:35:51.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RXu84t50d/M8snf7cmjWFh1uNzU=', 'Content-Length': '0'} 2023-10-11T15:35:51.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c83311b6036ab1-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.325 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.325 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.326 DEBUG:boto:path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.326 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.327 DEBUG:boto:Method: GET 2023-10-11T15:35:51.327 DEBUG:boto:Path: /dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.328 DEBUG:boto:Data: 2023-10-11T15:35:51.328 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.328 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.329 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.329 DEBUG:boto:Params: {} 2023-10-11T15:35:51.330 DEBUG:boto:Token: None 2023-10-11T15:35:51.330 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.331 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009990d3187fe4f7a4-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.334 DEBUG:boto:b'dghqvd-61obj571000falseobj572023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.335 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.335 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.336 DEBUG:boto:path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.336 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.337 DEBUG:boto:Method: GET 2023-10-11T15:35:51.337 DEBUG:boto:Path: /dghqvd-61/?prefix=obj57 2023-10-11T15:35:51.338 DEBUG:boto:Data: 2023-10-11T15:35:51.338 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.338 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.339 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.339 DEBUG:boto:Params: {} 2023-10-11T15:35:51.340 DEBUG:boto:Token: None 2023-10-11T15:35:51.340 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1a44467a8a8ea0-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.347 DEBUG:boto:b'dghqvd-61obj571000falseobj572023-10-11T15:31:43.122Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:35:51.349 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.350 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.351 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.351 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.352 DEBUG:boto:Method: GET 2023-10-11T15:35:51.353 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.353 DEBUG:boto:Data: 2023-10-11T15:35:51.354 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.356 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.356 DEBUG:boto:Params: {} 2023-10-11T15:35:51.357 DEBUG:boto:Token: None 2023-10-11T15:35:51.358 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.359 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y= 2023-10-11T15:35:51.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y=', 'Content-Length': '0'} 2023-10-11T15:35:51.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c336f86d8f1a6c8-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.363 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.364 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.364 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.365 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.365 DEBUG:boto:Method: GET 2023-10-11T15:35:51.366 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.366 DEBUG:boto:Data: 2023-10-11T15:35:51.367 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.367 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.368 DEBUG:boto:Params: {} 2023-10-11T15:35:51.368 DEBUG:boto:Token: None 2023-10-11T15:35:51.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y= 2023-10-11T15:35:51.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y=', 'Content-Length': '0'} 2023-10-11T15:35:51.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617cd1a18d5be96c-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.415 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.416 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.416 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.416 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.417 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.417 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.418 DEBUG:boto:Data: 2023-10-11T15:35:51.418 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.419 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.420 DEBUG:boto:Params: {} 2023-10-11T15:35:51.420 DEBUG:boto:Token: None 2023-10-11T15:35:51.420 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6TjSIH5JMlzLJunBC/F7hZP6Tc= 2023-10-11T15:35:51.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6TjSIH5JMlzLJunBC/F7hZP6Tc=', 'Content-Length': '0'} 2023-10-11T15:35:51.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cadadda6a2b3a631-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.424 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.424 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.425 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.425 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.426 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.426 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.426 DEBUG:boto:Data: 2023-10-11T15:35:51.427 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.428 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.428 DEBUG:boto:Params: {} 2023-10-11T15:35:51.429 DEBUG:boto:Token: None 2023-10-11T15:35:51.429 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.430 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:i6TjSIH5JMlzLJunBC/F7hZP6Tc= 2023-10-11T15:35:51.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:i6TjSIH5JMlzLJunBC/F7hZP6Tc=', 'Content-Length': '0'} 2023-10-11T15:35:51.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8cb168f5746142-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:35:51.433 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.433 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.434 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.434 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.435 DEBUG:boto:Method: GET 2023-10-11T15:35:51.435 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.436 DEBUG:boto:Data: 2023-10-11T15:35:51.436 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.437 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.438 DEBUG:boto:Params: {} 2023-10-11T15:35:51.438 DEBUG:boto:Token: None 2023-10-11T15:35:51.439 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.439 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y= 2023-10-11T15:35:51.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y=', 'Content-Length': '0'} 2023-10-11T15:35:51.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705b6349be3b0b3c-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.443 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.443 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.444 DEBUG:boto:path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.445 DEBUG:boto:auth_path=/dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.445 DEBUG:boto:Method: GET 2023-10-11T15:35:51.446 DEBUG:boto:Path: /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.447 DEBUG:boto:Data: 2023-10-11T15:35:51.448 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.449 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.449 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.450 DEBUG:boto:Params: {} 2023-10-11T15:35:51.451 DEBUG:boto:Token: None 2023-10-11T15:35:51.452 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58?versionId=null 2023-10-11T15:35:51.453 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y= 2023-10-11T15:35:51.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:czR1e6jwYJwwevtJdRaDBEAsw4Y=', 'Content-Length': '0'} 2023-10-11T15:35:51.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d171286b727d61b-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.457 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.457 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.458 DEBUG:boto:path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.458 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.459 DEBUG:boto:Method: GET 2023-10-11T15:35:51.459 DEBUG:boto:Path: /dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.460 DEBUG:boto:Data: 2023-10-11T15:35:51.460 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.460 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.461 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.461 DEBUG:boto:Params: {} 2023-10-11T15:35:51.462 DEBUG:boto:Token: None 2023-10-11T15:35:51.462 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ff6fb638008e749-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.466 DEBUG:boto:b'dghqvd-61obj581000falseobj582023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.467 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.468 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.468 DEBUG:boto:path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.468 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.469 DEBUG:boto:Method: GET 2023-10-11T15:35:51.469 DEBUG:boto:Path: /dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.470 DEBUG:boto:Data: 2023-10-11T15:35:51.470 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.471 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.471 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.471 DEBUG:boto:Params: {} 2023-10-11T15:35:51.472 DEBUG:boto:Token: None 2023-10-11T15:35:51.472 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.473 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb764885b764da1-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.478 DEBUG:boto:b'dghqvd-61obj581000falseobj582023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.479 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.479 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.480 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.480 DEBUG:boto:Path: /dghqvd-61/obj58 2023-10-11T15:35:51.480 DEBUG:boto:Data: 2023-10-11T15:35:51.481 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.481 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.482 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.482 DEBUG:boto:Params: {} 2023-10-11T15:35:51.483 DEBUG:boto:Token: None 2023-10-11T15:35:51.483 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58 2023-10-11T15:35:51.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7gKfrs4s10O8dV8NXZknk6Fh4G4= 2023-10-11T15:35:51.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7gKfrs4s10O8dV8NXZknk6Fh4G4=', 'Content-Length': '0'} 2023-10-11T15:35:51.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031298b4af84a5a67-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.486 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.487 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.487 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.488 DEBUG:boto:Path: /dghqvd-61/obj58 2023-10-11T15:35:51.488 DEBUG:boto:Data: 2023-10-11T15:35:51.488 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.489 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.490 DEBUG:boto:Params: {} 2023-10-11T15:35:51.490 DEBUG:boto:Token: None 2023-10-11T15:35:51.491 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58 2023-10-11T15:35:51.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:7gKfrs4s10O8dV8NXZknk6Fh4G4= 2023-10-11T15:35:51.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7gKfrs4s10O8dV8NXZknk6Fh4G4=', 'Content-Length': '0'} 2023-10-11T15:35:51.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026c56557b7ebbcdb-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-10-11T15:35:51.494 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.495 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.495 DEBUG:boto:Method: GET 2023-10-11T15:35:51.496 DEBUG:boto:Path: /dghqvd-61/obj58 2023-10-11T15:35:51.496 DEBUG:boto:Data: 2023-10-11T15:35:51.497 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.497 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.497 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.498 DEBUG:boto:Params: {} 2023-10-11T15:35:51.498 DEBUG:boto:Token: None 2023-10-11T15:35:51.499 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58 2023-10-11T15:35:51.499 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5QjZ2CWNpibABFLSij6m0eW/cW8= 2023-10-11T15:35:51.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5QjZ2CWNpibABFLSij6m0eW/cW8=', 'Content-Length': '0'} 2023-10-11T15:35:51.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222e1baef97f33f1-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.503 DEBUG:boto:path=/dghqvd-61/obj58 2023-10-11T15:35:51.504 DEBUG:boto:auth_path=/dghqvd-61/obj58 2023-10-11T15:35:51.504 DEBUG:boto:Method: GET 2023-10-11T15:35:51.504 DEBUG:boto:Path: /dghqvd-61/obj58 2023-10-11T15:35:51.505 DEBUG:boto:Data: 2023-10-11T15:35:51.505 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.506 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.506 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.507 DEBUG:boto:Params: {} 2023-10-11T15:35:51.507 DEBUG:boto:Token: None 2023-10-11T15:35:51.508 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj58 2023-10-11T15:35:51.508 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5QjZ2CWNpibABFLSij6m0eW/cW8= 2023-10-11T15:35:51.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5QjZ2CWNpibABFLSij6m0eW/cW8=', 'Content-Length': '0'} 2023-10-11T15:35:51.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f4223e2818577b-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.512 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.512 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.513 DEBUG:boto:path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.513 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.513 DEBUG:boto:Method: GET 2023-10-11T15:35:51.514 DEBUG:boto:Path: /dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.514 DEBUG:boto:Data: 2023-10-11T15:35:51.515 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.516 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.516 DEBUG:boto:Params: {} 2023-10-11T15:35:51.517 DEBUG:boto:Token: None 2023-10-11T15:35:51.518 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000205e4f019db4119-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.567 DEBUG:boto:b'dghqvd-61obj581000falseobj582023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.568 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.568 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.568 DEBUG:boto:path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.569 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.569 DEBUG:boto:Method: GET 2023-10-11T15:35:51.570 DEBUG:boto:Path: /dghqvd-61/?prefix=obj58 2023-10-11T15:35:51.571 DEBUG:boto:Data: 2023-10-11T15:35:51.571 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.572 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.573 DEBUG:boto:Params: {} 2023-10-11T15:35:51.573 DEBUG:boto:Token: None 2023-10-11T15:35:51.574 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000731fba23836db717-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.579 DEBUG:boto:b'dghqvd-61obj581000falseobj582023-10-11T15:31:43.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:35:51.580 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.581 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.581 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.582 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.582 DEBUG:boto:Method: GET 2023-10-11T15:35:51.582 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.583 DEBUG:boto:Data: 2023-10-11T15:35:51.583 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.584 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.584 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.585 DEBUG:boto:Params: {} 2023-10-11T15:35:51.585 DEBUG:boto:Token: None 2023-10-11T15:35:51.586 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.586 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA= 2023-10-11T15:35:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA=', 'Content-Length': '0'} 2023-10-11T15:35:51.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a37fd259adb831-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.590 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.592 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.593 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.593 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.595 DEBUG:boto:Method: GET 2023-10-11T15:35:51.596 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.596 DEBUG:boto:Data: 2023-10-11T15:35:51.597 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.597 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.598 DEBUG:boto:Params: {} 2023-10-11T15:35:51.599 DEBUG:boto:Token: None 2023-10-11T15:35:51.599 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.600 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA= 2023-10-11T15:35:51.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA=', 'Content-Length': '0'} 2023-10-11T15:35:51.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2732973c383bfe3-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.650 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.651 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.652 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.652 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.653 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.653 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.653 DEBUG:boto:Data: 2023-10-11T15:35:51.654 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.655 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.655 DEBUG:boto:Params: {} 2023-10-11T15:35:51.656 DEBUG:boto:Token: None 2023-10-11T15:35:51.657 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EWxtYPfnezayx6NQcULOiomZk3A= 2023-10-11T15:35:51.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EWxtYPfnezayx6NQcULOiomZk3A=', 'Content-Length': '0'} 2023-10-11T15:35:51.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001edc6de037f205b3-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.661 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.662 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.663 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.663 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.664 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.664 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.665 DEBUG:boto:Data: 2023-10-11T15:35:51.665 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.666 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.666 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.666 DEBUG:boto:Params: {} 2023-10-11T15:35:51.667 DEBUG:boto:Token: None 2023-10-11T15:35:51.667 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.668 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EWxtYPfnezayx6NQcULOiomZk3A= 2023-10-11T15:35:51.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EWxtYPfnezayx6NQcULOiomZk3A=', 'Content-Length': '0'} 2023-10-11T15:35:51.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d643becc9a6067e-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:35:51.671 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.672 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.672 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.672 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.673 DEBUG:boto:Method: GET 2023-10-11T15:35:51.673 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.674 DEBUG:boto:Data: 2023-10-11T15:35:51.674 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.676 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.676 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.677 DEBUG:boto:Params: {} 2023-10-11T15:35:51.678 DEBUG:boto:Token: None 2023-10-11T15:35:51.679 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.679 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA= 2023-10-11T15:35:51.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA=', 'Content-Length': '0'} 2023-10-11T15:35:51.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c4a3334a5b93e6-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.683 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.683 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.684 DEBUG:boto:path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.684 DEBUG:boto:auth_path=/dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.684 DEBUG:boto:Method: GET 2023-10-11T15:35:51.685 DEBUG:boto:Path: /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.686 DEBUG:boto:Data: 2023-10-11T15:35:51.686 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.687 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.687 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.688 DEBUG:boto:Params: {} 2023-10-11T15:35:51.688 DEBUG:boto:Token: None 2023-10-11T15:35:51.688 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62?versionId=null 2023-10-11T15:35:51.689 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA= 2023-10-11T15:35:51.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tZDfOzZWgNpf1xbsnAkUwK3BnTA=', 'Content-Length': '0'} 2023-10-11T15:35:51.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001727f066219008e2-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.692 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.693 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.694 DEBUG:boto:path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.695 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.695 DEBUG:boto:Method: GET 2023-10-11T15:35:51.696 DEBUG:boto:Path: /dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.697 DEBUG:boto:Data: 2023-10-11T15:35:51.698 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.699 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.699 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.700 DEBUG:boto:Params: {} 2023-10-11T15:35:51.701 DEBUG:boto:Token: None 2023-10-11T15:35:51.702 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.702 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000737f792a1ddb42da-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.706 DEBUG:boto:b'dghqvd-61obj621000falseobj622023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.707 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.708 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.709 DEBUG:boto:path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.709 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.710 DEBUG:boto:Method: GET 2023-10-11T15:35:51.711 DEBUG:boto:Path: /dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.712 DEBUG:boto:Data: 2023-10-11T15:35:51.712 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.713 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.713 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.714 DEBUG:boto:Params: {} 2023-10-11T15:35:51.715 DEBUG:boto:Token: None 2023-10-11T15:35:51.715 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.716 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b526f2fb4bdfab-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.720 DEBUG:boto:b'dghqvd-61obj621000falseobj622023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.721 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.722 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.722 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.723 DEBUG:boto:Path: /dghqvd-61/obj62 2023-10-11T15:35:51.723 DEBUG:boto:Data: 2023-10-11T15:35:51.723 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.724 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.725 DEBUG:boto:Params: {} 2023-10-11T15:35:51.725 DEBUG:boto:Token: None 2023-10-11T15:35:51.726 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62 2023-10-11T15:35:51.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qo1KqbWYs1uPJzr8Zb7jslxAyM4= 2023-10-11T15:35:51.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qo1KqbWYs1uPJzr8Zb7jslxAyM4=', 'Content-Length': '0'} 2023-10-11T15:35:51.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0dbb67a6de6e60b-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.729 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.730 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.730 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.731 DEBUG:boto:Path: /dghqvd-61/obj62 2023-10-11T15:35:51.731 DEBUG:boto:Data: 2023-10-11T15:35:51.731 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.732 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.732 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.733 DEBUG:boto:Params: {} 2023-10-11T15:35:51.733 DEBUG:boto:Token: None 2023-10-11T15:35:51.734 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62 2023-10-11T15:35:51.734 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qo1KqbWYs1uPJzr8Zb7jslxAyM4= 2023-10-11T15:35:51.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qo1KqbWYs1uPJzr8Zb7jslxAyM4=', 'Content-Length': '0'} 2023-10-11T15:35:51.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379b5ce293717d4e-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-10-11T15:35:51.738 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.738 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.739 DEBUG:boto:Method: GET 2023-10-11T15:35:51.739 DEBUG:boto:Path: /dghqvd-61/obj62 2023-10-11T15:35:51.740 DEBUG:boto:Data: 2023-10-11T15:35:51.740 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.741 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.741 DEBUG:boto:Params: {} 2023-10-11T15:35:51.742 DEBUG:boto:Token: None 2023-10-11T15:35:51.742 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62 2023-10-11T15:35:51.743 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kduilq9pQHCXST/t02TZgKbOoyk= 2023-10-11T15:35:51.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kduilq9pQHCXST/t02TZgKbOoyk=', 'Content-Length': '0'} 2023-10-11T15:35:51.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c563ae8111c808-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.746 DEBUG:boto:path=/dghqvd-61/obj62 2023-10-11T15:35:51.746 DEBUG:boto:auth_path=/dghqvd-61/obj62 2023-10-11T15:35:51.747 DEBUG:boto:Method: GET 2023-10-11T15:35:51.747 DEBUG:boto:Path: /dghqvd-61/obj62 2023-10-11T15:35:51.747 DEBUG:boto:Data: 2023-10-11T15:35:51.748 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.748 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.749 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.749 DEBUG:boto:Params: {} 2023-10-11T15:35:51.750 DEBUG:boto:Token: None 2023-10-11T15:35:51.750 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj62 2023-10-11T15:35:51.751 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kduilq9pQHCXST/t02TZgKbOoyk= 2023-10-11T15:35:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kduilq9pQHCXST/t02TZgKbOoyk=', 'Content-Length': '0'} 2023-10-11T15:35:51.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bb60d64eb7c5555-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.756 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.757 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.757 DEBUG:boto:path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.758 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.759 DEBUG:boto:Method: GET 2023-10-11T15:35:51.760 DEBUG:boto:Path: /dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.760 DEBUG:boto:Data: 2023-10-11T15:35:51.761 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.762 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.762 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.763 DEBUG:boto:Params: {} 2023-10-11T15:35:51.764 DEBUG:boto:Token: None 2023-10-11T15:35:51.765 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.765 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f7955204fa3f77b-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.770 DEBUG:boto:b'dghqvd-61obj621000falseobj622023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.770 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.771 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.771 DEBUG:boto:path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.772 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.772 DEBUG:boto:Method: GET 2023-10-11T15:35:51.773 DEBUG:boto:Path: /dghqvd-61/?prefix=obj62 2023-10-11T15:35:51.773 DEBUG:boto:Data: 2023-10-11T15:35:51.773 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.774 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.775 DEBUG:boto:Params: {} 2023-10-11T15:35:51.775 DEBUG:boto:Token: None 2023-10-11T15:35:51.776 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b77be6e971505d-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.826 DEBUG:boto:b'dghqvd-61obj621000falseobj622023-10-11T15:31:44.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:35:51.828 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.828 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.829 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.829 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.830 DEBUG:boto:Method: GET 2023-10-11T15:35:51.830 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.831 DEBUG:boto:Data: 2023-10-11T15:35:51.831 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.832 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.832 DEBUG:boto:Params: {} 2023-10-11T15:35:51.833 DEBUG:boto:Token: None 2023-10-11T15:35:51.833 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw= 2023-10-11T15:35:51.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw=', 'Content-Length': '0'} 2023-10-11T15:35:51.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616028d8dab44434-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.837 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.838 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.838 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.839 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.839 DEBUG:boto:Method: GET 2023-10-11T15:35:51.840 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.840 DEBUG:boto:Data: 2023-10-11T15:35:51.840 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.841 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.841 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.842 DEBUG:boto:Params: {} 2023-10-11T15:35:51.843 DEBUG:boto:Token: None 2023-10-11T15:35:51.843 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.844 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw= 2023-10-11T15:35:51.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw=', 'Content-Length': '0'} 2023-10-11T15:35:51.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf2b0a4de8ba4989-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.847 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.848 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.848 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.849 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.849 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.849 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.850 DEBUG:boto:Data: 2023-10-11T15:35:51.850 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.851 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.852 DEBUG:boto:Params: {} 2023-10-11T15:35:51.852 DEBUG:boto:Token: None 2023-10-11T15:35:51.853 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.854 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yz3Jl5LosxUL2gXq34XzcgyfuGI= 2023-10-11T15:35:51.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yz3Jl5LosxUL2gXq34XzcgyfuGI=', 'Content-Length': '0'} 2023-10-11T15:35:51.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007473f50e7fed0162-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.857 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.858 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.858 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.859 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.859 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.860 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.861 DEBUG:boto:Data: 2023-10-11T15:35:51.861 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.862 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.862 DEBUG:boto:Params: {} 2023-10-11T15:35:51.863 DEBUG:boto:Token: None 2023-10-11T15:35:51.864 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yz3Jl5LosxUL2gXq34XzcgyfuGI= 2023-10-11T15:35:51.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yz3Jl5LosxUL2gXq34XzcgyfuGI=', 'Content-Length': '0'} 2023-10-11T15:35:51.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a409f1280149e303-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:35:51.868 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.868 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.869 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.870 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.871 DEBUG:boto:Method: GET 2023-10-11T15:35:51.872 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.872 DEBUG:boto:Data: 2023-10-11T15:35:51.873 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.875 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.875 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.876 DEBUG:boto:Params: {} 2023-10-11T15:35:51.877 DEBUG:boto:Token: None 2023-10-11T15:35:51.877 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.878 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw= 2023-10-11T15:35:51.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw=', 'Content-Length': '0'} 2023-10-11T15:35:51.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2644868e235f1f-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.881 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.881 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.882 DEBUG:boto:path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.882 DEBUG:boto:auth_path=/dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.883 DEBUG:boto:Method: GET 2023-10-11T15:35:51.883 DEBUG:boto:Path: /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.884 DEBUG:boto:Data: 2023-10-11T15:35:51.885 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.886 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.886 DEBUG:boto:Params: {} 2023-10-11T15:35:51.887 DEBUG:boto:Token: None 2023-10-11T15:35:51.887 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68?versionId=null 2023-10-11T15:35:51.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw= 2023-10-11T15:35:51.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QfRPM0jr5VH7UubPkVTkrSSjxOw=', 'Content-Length': '0'} 2023-10-11T15:35:51.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a9aadc657a32bd1-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.935 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.935 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.936 DEBUG:boto:path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.937 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.938 DEBUG:boto:Method: GET 2023-10-11T15:35:51.939 DEBUG:boto:Path: /dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.940 DEBUG:boto:Data: 2023-10-11T15:35:51.940 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.941 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.942 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.943 DEBUG:boto:Params: {} 2023-10-11T15:35:51.944 DEBUG:boto:Token: None 2023-10-11T15:35:51.944 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0dde29f2b51b03-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.948 DEBUG:boto:b'dghqvd-61obj681000falseobj682023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.949 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.950 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.950 DEBUG:boto:path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.950 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.951 DEBUG:boto:Method: GET 2023-10-11T15:35:51.951 DEBUG:boto:Path: /dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.952 DEBUG:boto:Data: 2023-10-11T15:35:51.952 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.953 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.953 DEBUG:boto:Params: {} 2023-10-11T15:35:51.954 DEBUG:boto:Token: None 2023-10-11T15:35:51.954 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:51.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:51.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:51.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ebfb7511e3b2a41-006526c0d7-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.962 DEBUG:boto:b'dghqvd-61obj681000falseobj682023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:51.962 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.963 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.963 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.964 DEBUG:boto:Path: /dghqvd-61/obj68 2023-10-11T15:35:51.964 DEBUG:boto:Data: 2023-10-11T15:35:51.965 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.965 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.966 DEBUG:boto:Params: {} 2023-10-11T15:35:51.966 DEBUG:boto:Token: None 2023-10-11T15:35:51.967 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68 2023-10-11T15:35:51.967 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V/ct7BNY79mSYsHLQlf36k1dlDM= 2023-10-11T15:35:51.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V/ct7BNY79mSYsHLQlf36k1dlDM=', 'Content-Length': '0'} 2023-10-11T15:35:51.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbd23760dd1d72e2-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.970 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.970 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.971 DEBUG:boto:Method: HEAD 2023-10-11T15:35:51.971 DEBUG:boto:Path: /dghqvd-61/obj68 2023-10-11T15:35:51.972 DEBUG:boto:Data: 2023-10-11T15:35:51.972 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.973 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.973 DEBUG:boto:Params: {} 2023-10-11T15:35:51.974 DEBUG:boto:Token: None 2023-10-11T15:35:51.974 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68 2023-10-11T15:35:51.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V/ct7BNY79mSYsHLQlf36k1dlDM= 2023-10-11T15:35:51.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V/ct7BNY79mSYsHLQlf36k1dlDM=', 'Content-Length': '0'} 2023-10-11T15:35:51.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ace77f74718036-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-10-11T15:35:51.978 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.979 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.979 DEBUG:boto:Method: GET 2023-10-11T15:35:51.980 DEBUG:boto:Path: /dghqvd-61/obj68 2023-10-11T15:35:51.980 DEBUG:boto:Data: 2023-10-11T15:35:51.981 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.981 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.981 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.982 DEBUG:boto:Params: {} 2023-10-11T15:35:51.982 DEBUG:boto:Token: None 2023-10-11T15:35:51.983 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68 2023-10-11T15:35:51.983 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VWjsB5mqKqxR5jX7LB7XknDVrRk= 2023-10-11T15:35:51.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VWjsB5mqKqxR5jX7LB7XknDVrRk=', 'Content-Length': '0'} 2023-10-11T15:35:51.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c13dac8c6c6782-006526c0d7-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.986 DEBUG:boto:path=/dghqvd-61/obj68 2023-10-11T15:35:51.986 DEBUG:boto:auth_path=/dghqvd-61/obj68 2023-10-11T15:35:51.987 DEBUG:boto:Method: GET 2023-10-11T15:35:51.987 DEBUG:boto:Path: /dghqvd-61/obj68 2023-10-11T15:35:51.988 DEBUG:boto:Data: 2023-10-11T15:35:51.988 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:51.989 DEBUG:boto:Port: 8000 2023-10-11T15:35:51.989 DEBUG:boto:Params: {} 2023-10-11T15:35:51.990 DEBUG:boto:Token: None 2023-10-11T15:35:51.990 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/obj68 2023-10-11T15:35:51.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VWjsB5mqKqxR5jX7LB7XknDVrRk= 2023-10-11T15:35:51.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VWjsB5mqKqxR5jX7LB7XknDVrRk=', 'Content-Length': '0'} 2023-10-11T15:35:51.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000329478649886f1ba-006526c0d7-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:51 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:51.994 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:51.994 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:51.995 DEBUG:boto:path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.995 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.995 DEBUG:boto:Method: GET 2023-10-11T15:35:51.996 DEBUG:boto:Path: /dghqvd-61/?prefix=obj68 2023-10-11T15:35:51.996 DEBUG:boto:Data: 2023-10-11T15:35:51.997 DEBUG:boto:Headers: {} 2023-10-11T15:35:51.997 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:51.998 DEBUG:boto:Port: 8001 2023-10-11T15:35:51.998 DEBUG:boto:Params: {} 2023-10-11T15:35:51.999 DEBUG:boto:Token: None 2023-10-11T15:35:51.999 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:51 GMT /dghqvd-61/ 2023-10-11T15:35:52.000 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY= 2023-10-11T15:35:52.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:51 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oPTX4MFHGOPpnpzNV5k05bV5FfY=', 'Content-Length': '0'} 2023-10-11T15:35:52.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fb2b7099d546fa6-006526c0d7-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.003 DEBUG:boto:b'dghqvd-61obj681000falseobj682023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.004 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.004 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.004 DEBUG:boto:path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:52.005 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj68 2023-10-11T15:35:52.005 DEBUG:boto:Method: GET 2023-10-11T15:35:52.006 DEBUG:boto:Path: /dghqvd-61/?prefix=obj68 2023-10-11T15:35:52.006 DEBUG:boto:Data: 2023-10-11T15:35:52.007 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.008 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.008 DEBUG:boto:Params: {} 2023-10-11T15:35:52.008 DEBUG:boto:Token: None 2023-10-11T15:35:52.009 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb68046c0657ec59-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.014 DEBUG:boto:b'dghqvd-61obj681000falseobj682023-10-11T15:31:42.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:35:52.015 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.015 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.016 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.016 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.017 DEBUG:boto:Method: GET 2023-10-11T15:35:52.017 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.017 DEBUG:boto:Data: 2023-10-11T15:35:52.018 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.018 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.019 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.019 DEBUG:boto:Params: {} 2023-10-11T15:35:52.020 DEBUG:boto:Token: None 2023-10-11T15:35:52.020 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M= 2023-10-11T15:35:52.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M=', 'Content-Length': '0'} 2023-10-11T15:35:52.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f64cd7b6857b23-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.024 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.024 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.025 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.025 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.026 DEBUG:boto:Method: GET 2023-10-11T15:35:52.026 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.026 DEBUG:boto:Data: 2023-10-11T15:35:52.027 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.028 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.028 DEBUG:boto:Params: {} 2023-10-11T15:35:52.029 DEBUG:boto:Token: None 2023-10-11T15:35:52.029 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M= 2023-10-11T15:35:52.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M=', 'Content-Length': '0'} 2023-10-11T15:35:52.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfca4d9e3636015b-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.032 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.033 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.033 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.033 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.034 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.034 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.035 DEBUG:boto:Data: 2023-10-11T15:35:52.035 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.036 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.036 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.036 DEBUG:boto:Params: {} 2023-10-11T15:35:52.037 DEBUG:boto:Token: None 2023-10-11T15:35:52.037 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.038 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5LEU9XCuxk83P0R2xVRYPExN30U= 2023-10-11T15:35:52.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5LEU9XCuxk83P0R2xVRYPExN30U=', 'Content-Length': '0'} 2023-10-11T15:35:52.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a87580acd62343-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.041 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.041 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.041 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.042 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.042 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.043 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.043 DEBUG:boto:Data: 2023-10-11T15:35:52.043 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.044 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.044 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.045 DEBUG:boto:Params: {} 2023-10-11T15:35:52.045 DEBUG:boto:Token: None 2023-10-11T15:35:52.046 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.046 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5LEU9XCuxk83P0R2xVRYPExN30U= 2023-10-11T15:35:52.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5LEU9XCuxk83P0R2xVRYPExN30U=', 'Content-Length': '0'} 2023-10-11T15:35:52.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060624f8e2b22128-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:35:52.049 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.049 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.050 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.050 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.051 DEBUG:boto:Method: GET 2023-10-11T15:35:52.051 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.052 DEBUG:boto:Data: 2023-10-11T15:35:52.052 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.053 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.053 DEBUG:boto:Params: {} 2023-10-11T15:35:52.054 DEBUG:boto:Token: None 2023-10-11T15:35:52.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M= 2023-10-11T15:35:52.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M=', 'Content-Length': '0'} 2023-10-11T15:35:52.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099937f465ec646f3-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.098 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.099 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.099 DEBUG:boto:path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.100 DEBUG:boto:auth_path=/dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.100 DEBUG:boto:Method: GET 2023-10-11T15:35:52.101 DEBUG:boto:Path: /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.101 DEBUG:boto:Data: 2023-10-11T15:35:52.102 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.102 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.103 DEBUG:boto:Params: {} 2023-10-11T15:35:52.103 DEBUG:boto:Token: None 2023-10-11T15:35:52.104 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69?versionId=null 2023-10-11T15:35:52.104 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M= 2023-10-11T15:35:52.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TyVX/nktk6lnEJHldV9KWipaB/M=', 'Content-Length': '0'} 2023-10-11T15:35:52.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7b7943eac24c4d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.107 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.108 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.108 DEBUG:boto:path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.109 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.109 DEBUG:boto:Method: GET 2023-10-11T15:35:52.109 DEBUG:boto:Path: /dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.110 DEBUG:boto:Data: 2023-10-11T15:35:52.110 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.111 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.111 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.112 DEBUG:boto:Params: {} 2023-10-11T15:35:52.112 DEBUG:boto:Token: None 2023-10-11T15:35:52.113 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.113 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f768e06c2f51222-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.116 DEBUG:boto:b'dghqvd-61obj691000falseobj692023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.117 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.117 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.118 DEBUG:boto:path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.118 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.119 DEBUG:boto:Method: GET 2023-10-11T15:35:52.119 DEBUG:boto:Path: /dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.120 DEBUG:boto:Data: 2023-10-11T15:35:52.120 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.121 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.121 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.121 DEBUG:boto:Params: {} 2023-10-11T15:35:52.122 DEBUG:boto:Token: None 2023-10-11T15:35:52.122 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.123 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfe44f820cb2395e-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.128 DEBUG:boto:b'dghqvd-61obj691000falseobj692023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.128 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.129 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.130 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.130 DEBUG:boto:Path: /dghqvd-61/obj69 2023-10-11T15:35:52.131 DEBUG:boto:Data: 2023-10-11T15:35:52.131 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.132 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.133 DEBUG:boto:Params: {} 2023-10-11T15:35:52.133 DEBUG:boto:Token: None 2023-10-11T15:35:52.134 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69 2023-10-11T15:35:52.134 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:glEdjK6NqHJAF8iJZu4ZMk5f43E= 2023-10-11T15:35:52.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:glEdjK6NqHJAF8iJZu4ZMk5f43E=', 'Content-Length': '0'} 2023-10-11T15:35:52.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006762d47ef5501c00-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.137 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.137 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.138 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.138 DEBUG:boto:Path: /dghqvd-61/obj69 2023-10-11T15:35:52.139 DEBUG:boto:Data: 2023-10-11T15:35:52.139 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.140 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.140 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.140 DEBUG:boto:Params: {} 2023-10-11T15:35:52.141 DEBUG:boto:Token: None 2023-10-11T15:35:52.141 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69 2023-10-11T15:35:52.142 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:glEdjK6NqHJAF8iJZu4ZMk5f43E= 2023-10-11T15:35:52.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:glEdjK6NqHJAF8iJZu4ZMk5f43E=', 'Content-Length': '0'} 2023-10-11T15:35:52.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c566db36b1eb9c39-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-10-11T15:35:52.145 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.145 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.146 DEBUG:boto:Method: GET 2023-10-11T15:35:52.146 DEBUG:boto:Path: /dghqvd-61/obj69 2023-10-11T15:35:52.147 DEBUG:boto:Data: 2023-10-11T15:35:52.147 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.148 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.148 DEBUG:boto:Params: {} 2023-10-11T15:35:52.149 DEBUG:boto:Token: None 2023-10-11T15:35:52.149 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69 2023-10-11T15:35:52.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:llTHoOSGClnsdowbYv2lzek/Qyk= 2023-10-11T15:35:52.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:llTHoOSGClnsdowbYv2lzek/Qyk=', 'Content-Length': '0'} 2023-10-11T15:35:52.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008036a0f304292e4c-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.153 DEBUG:boto:path=/dghqvd-61/obj69 2023-10-11T15:35:52.153 DEBUG:boto:auth_path=/dghqvd-61/obj69 2023-10-11T15:35:52.154 DEBUG:boto:Method: GET 2023-10-11T15:35:52.154 DEBUG:boto:Path: /dghqvd-61/obj69 2023-10-11T15:35:52.155 DEBUG:boto:Data: 2023-10-11T15:35:52.155 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.156 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.157 DEBUG:boto:Params: {} 2023-10-11T15:35:52.157 DEBUG:boto:Token: None 2023-10-11T15:35:52.158 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj69 2023-10-11T15:35:52.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:llTHoOSGClnsdowbYv2lzek/Qyk= 2023-10-11T15:35:52.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:llTHoOSGClnsdowbYv2lzek/Qyk=', 'Content-Length': '0'} 2023-10-11T15:35:52.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004df5ba6fc770494a-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.206 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.207 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.208 DEBUG:boto:path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.208 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.209 DEBUG:boto:Method: GET 2023-10-11T15:35:52.209 DEBUG:boto:Path: /dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.209 DEBUG:boto:Data: 2023-10-11T15:35:52.210 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.211 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.211 DEBUG:boto:Params: {} 2023-10-11T15:35:52.212 DEBUG:boto:Token: None 2023-10-11T15:35:52.212 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.213 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f3ec60587b2ca0-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.216 DEBUG:boto:b'dghqvd-61obj691000falseobj692023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.217 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.217 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.218 DEBUG:boto:path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.218 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.219 DEBUG:boto:Method: GET 2023-10-11T15:35:52.219 DEBUG:boto:Path: /dghqvd-61/?prefix=obj69 2023-10-11T15:35:52.220 DEBUG:boto:Data: 2023-10-11T15:35:52.220 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.221 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.221 DEBUG:boto:Params: {} 2023-10-11T15:35:52.222 DEBUG:boto:Token: None 2023-10-11T15:35:52.222 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.223 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048284ff3fc924924-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.227 DEBUG:boto:b'dghqvd-61obj691000falseobj692023-10-11T15:31:43.463Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:35:52.228 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.229 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.229 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.230 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.230 DEBUG:boto:Method: GET 2023-10-11T15:35:52.231 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.231 DEBUG:boto:Data: 2023-10-11T15:35:52.232 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.232 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.232 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.233 DEBUG:boto:Params: {} 2023-10-11T15:35:52.233 DEBUG:boto:Token: None 2023-10-11T15:35:52.234 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.234 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls= 2023-10-11T15:35:52.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls=', 'Content-Length': '0'} 2023-10-11T15:35:52.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8acd35312328ec-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.237 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.238 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.238 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.239 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.239 DEBUG:boto:Method: GET 2023-10-11T15:35:52.239 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.240 DEBUG:boto:Data: 2023-10-11T15:35:52.240 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.241 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.241 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.241 DEBUG:boto:Params: {} 2023-10-11T15:35:52.242 DEBUG:boto:Token: None 2023-10-11T15:35:52.242 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.243 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls= 2023-10-11T15:35:52.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls=', 'Content-Length': '0'} 2023-10-11T15:35:52.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfad9a8a3e465d56-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.246 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.246 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.247 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.247 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.248 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.248 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.249 DEBUG:boto:Data: 2023-10-11T15:35:52.249 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.250 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.250 DEBUG:boto:Params: {} 2023-10-11T15:35:52.251 DEBUG:boto:Token: None 2023-10-11T15:35:52.251 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5kgxeryC0agFvs2lkPg6HJ5DN+g= 2023-10-11T15:35:52.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5kgxeryC0agFvs2lkPg6HJ5DN+g=', 'Content-Length': '0'} 2023-10-11T15:35:52.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1d3dda811d6f6b-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.254 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.255 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.255 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.256 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.256 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.256 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.257 DEBUG:boto:Data: 2023-10-11T15:35:52.257 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.258 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.258 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.258 DEBUG:boto:Params: {} 2023-10-11T15:35:52.259 DEBUG:boto:Token: None 2023-10-11T15:35:52.259 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5kgxeryC0agFvs2lkPg6HJ5DN+g= 2023-10-11T15:35:52.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5kgxeryC0agFvs2lkPg6HJ5DN+g=', 'Content-Length': '0'} 2023-10-11T15:35:52.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049c668dddb053c7-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:35:52.263 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.263 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.264 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.264 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.265 DEBUG:boto:Method: GET 2023-10-11T15:35:52.265 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.266 DEBUG:boto:Data: 2023-10-11T15:35:52.266 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.267 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.267 DEBUG:boto:Params: {} 2023-10-11T15:35:52.268 DEBUG:boto:Token: None 2023-10-11T15:35:52.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.269 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls= 2023-10-11T15:35:52.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls=', 'Content-Length': '0'} 2023-10-11T15:35:52.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d33c6a0efb175f-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.272 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.272 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.272 DEBUG:boto:path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.273 DEBUG:boto:auth_path=/dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.273 DEBUG:boto:Method: GET 2023-10-11T15:35:52.274 DEBUG:boto:Path: /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.274 DEBUG:boto:Data: 2023-10-11T15:35:52.275 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.275 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.276 DEBUG:boto:Params: {} 2023-10-11T15:35:52.276 DEBUG:boto:Token: None 2023-10-11T15:35:52.277 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74?versionId=null 2023-10-11T15:35:52.277 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls= 2023-10-11T15:35:52.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5GRfYIkbDJyv6rvUm8SneZX6Yls=', 'Content-Length': '0'} 2023-10-11T15:35:52.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037da43f6d5dfc6fc-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.280 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.281 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.281 DEBUG:boto:path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.281 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.282 DEBUG:boto:Method: GET 2023-10-11T15:35:52.282 DEBUG:boto:Path: /dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.283 DEBUG:boto:Data: 2023-10-11T15:35:52.283 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.284 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.284 DEBUG:boto:Params: {} 2023-10-11T15:35:52.285 DEBUG:boto:Token: None 2023-10-11T15:35:52.285 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e2565d459ec9417-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.330 DEBUG:boto:b'dghqvd-61obj741000falseobj742023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.331 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.332 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.332 DEBUG:boto:path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.333 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.333 DEBUG:boto:Method: GET 2023-10-11T15:35:52.333 DEBUG:boto:Path: /dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.334 DEBUG:boto:Data: 2023-10-11T15:35:52.334 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.335 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.335 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.336 DEBUG:boto:Params: {} 2023-10-11T15:35:52.336 DEBUG:boto:Token: None 2023-10-11T15:35:52.337 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006981d7a570e3bace-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.342 DEBUG:boto:b'dghqvd-61obj741000falseobj742023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.343 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.343 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.343 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.344 DEBUG:boto:Path: /dghqvd-61/obj74 2023-10-11T15:35:52.344 DEBUG:boto:Data: 2023-10-11T15:35:52.345 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.345 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.345 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.346 DEBUG:boto:Params: {} 2023-10-11T15:35:52.346 DEBUG:boto:Token: None 2023-10-11T15:35:52.347 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74 2023-10-11T15:35:52.347 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDTAdhb2McUnMYLfEedxRKyDZfI= 2023-10-11T15:35:52.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDTAdhb2McUnMYLfEedxRKyDZfI=', 'Content-Length': '0'} 2023-10-11T15:35:52.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c033eb8a209df65-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.350 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.351 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.351 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.351 DEBUG:boto:Path: /dghqvd-61/obj74 2023-10-11T15:35:52.352 DEBUG:boto:Data: 2023-10-11T15:35:52.352 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.353 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.353 DEBUG:boto:Params: {} 2023-10-11T15:35:52.354 DEBUG:boto:Token: None 2023-10-11T15:35:52.354 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74 2023-10-11T15:35:52.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EDTAdhb2McUnMYLfEedxRKyDZfI= 2023-10-11T15:35:52.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EDTAdhb2McUnMYLfEedxRKyDZfI=', 'Content-Length': '0'} 2023-10-11T15:35:52.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd18afbceb1e286-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-10-11T15:35:52.358 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.358 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.359 DEBUG:boto:Method: GET 2023-10-11T15:35:52.359 DEBUG:boto:Path: /dghqvd-61/obj74 2023-10-11T15:35:52.360 DEBUG:boto:Data: 2023-10-11T15:35:52.360 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.360 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.361 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.361 DEBUG:boto:Params: {} 2023-10-11T15:35:52.362 DEBUG:boto:Token: None 2023-10-11T15:35:52.362 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74 2023-10-11T15:35:52.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vQMGQmqlh3zvhBUMGqahsCzFvOY= 2023-10-11T15:35:52.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vQMGQmqlh3zvhBUMGqahsCzFvOY=', 'Content-Length': '0'} 2023-10-11T15:35:52.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087af1e22550c728b-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.368 DEBUG:boto:path=/dghqvd-61/obj74 2023-10-11T15:35:52.368 DEBUG:boto:auth_path=/dghqvd-61/obj74 2023-10-11T15:35:52.369 DEBUG:boto:Method: GET 2023-10-11T15:35:52.369 DEBUG:boto:Path: /dghqvd-61/obj74 2023-10-11T15:35:52.370 DEBUG:boto:Data: 2023-10-11T15:35:52.370 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.371 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.371 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.371 DEBUG:boto:Params: {} 2023-10-11T15:35:52.372 DEBUG:boto:Token: None 2023-10-11T15:35:52.372 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj74 2023-10-11T15:35:52.373 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vQMGQmqlh3zvhBUMGqahsCzFvOY= 2023-10-11T15:35:52.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vQMGQmqlh3zvhBUMGqahsCzFvOY=', 'Content-Length': '0'} 2023-10-11T15:35:52.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e445a2a39e83dab-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.376 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.376 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.377 DEBUG:boto:path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.377 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.377 DEBUG:boto:Method: GET 2023-10-11T15:35:52.378 DEBUG:boto:Path: /dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.378 DEBUG:boto:Data: 2023-10-11T15:35:52.379 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.379 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.379 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.380 DEBUG:boto:Params: {} 2023-10-11T15:35:52.380 DEBUG:boto:Token: None 2023-10-11T15:35:52.381 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.381 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968d981879c4e68d-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.385 DEBUG:boto:b'dghqvd-61obj741000falseobj742023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.386 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.386 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.387 DEBUG:boto:path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.387 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.388 DEBUG:boto:Method: GET 2023-10-11T15:35:52.388 DEBUG:boto:Path: /dghqvd-61/?prefix=obj74 2023-10-11T15:35:52.388 DEBUG:boto:Data: 2023-10-11T15:35:52.389 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.390 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.390 DEBUG:boto:Params: {} 2023-10-11T15:35:52.391 DEBUG:boto:Token: None 2023-10-11T15:35:52.391 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000938e14dddbc20511-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.396 DEBUG:boto:b'dghqvd-61obj741000falseobj742023-10-11T15:31:44.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:35:52.397 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.398 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.398 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.399 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.399 DEBUG:boto:Method: GET 2023-10-11T15:35:52.400 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.400 DEBUG:boto:Data: 2023-10-11T15:35:52.401 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.401 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.401 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.402 DEBUG:boto:Params: {} 2023-10-11T15:35:52.402 DEBUG:boto:Token: None 2023-10-11T15:35:52.403 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg= 2023-10-11T15:35:52.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg=', 'Content-Length': '0'} 2023-10-11T15:35:52.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f350593472e37263-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.407 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.407 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.408 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.408 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.409 DEBUG:boto:Method: GET 2023-10-11T15:35:52.409 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.409 DEBUG:boto:Data: 2023-10-11T15:35:52.410 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.410 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.411 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.411 DEBUG:boto:Params: {} 2023-10-11T15:35:52.412 DEBUG:boto:Token: None 2023-10-11T15:35:52.412 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.413 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg= 2023-10-11T15:35:52.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg=', 'Content-Length': '0'} 2023-10-11T15:35:52.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b03c2fb1bdd1a80-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.458 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.459 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.459 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.460 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.460 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.461 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.461 DEBUG:boto:Data: 2023-10-11T15:35:52.462 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.463 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.463 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.464 DEBUG:boto:Params: {} 2023-10-11T15:35:52.464 DEBUG:boto:Token: None 2023-10-11T15:35:52.465 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.465 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aN5sQSapYVU4xRkeAI7MFfGmIGk= 2023-10-11T15:35:52.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aN5sQSapYVU4xRkeAI7MFfGmIGk=', 'Content-Length': '0'} 2023-10-11T15:35:52.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005538f53f47f7c0de-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.468 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.469 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.469 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.470 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.470 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.470 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.471 DEBUG:boto:Data: 2023-10-11T15:35:52.471 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.472 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.472 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.472 DEBUG:boto:Params: {} 2023-10-11T15:35:52.473 DEBUG:boto:Token: None 2023-10-11T15:35:52.473 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.474 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:aN5sQSapYVU4xRkeAI7MFfGmIGk= 2023-10-11T15:35:52.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aN5sQSapYVU4xRkeAI7MFfGmIGk=', 'Content-Length': '0'} 2023-10-11T15:35:52.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049c14dcb3cb239d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:35:52.477 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.477 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.478 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.478 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.479 DEBUG:boto:Method: GET 2023-10-11T15:35:52.479 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.480 DEBUG:boto:Data: 2023-10-11T15:35:52.480 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.480 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.481 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.481 DEBUG:boto:Params: {} 2023-10-11T15:35:52.482 DEBUG:boto:Token: None 2023-10-11T15:35:52.482 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.483 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg= 2023-10-11T15:35:52.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg=', 'Content-Length': '0'} 2023-10-11T15:35:52.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062fc8408548ff016-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.486 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.486 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.487 DEBUG:boto:path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.487 DEBUG:boto:auth_path=/dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.487 DEBUG:boto:Method: GET 2023-10-11T15:35:52.488 DEBUG:boto:Path: /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.488 DEBUG:boto:Data: 2023-10-11T15:35:52.489 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.489 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.490 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.490 DEBUG:boto:Params: {} 2023-10-11T15:35:52.491 DEBUG:boto:Token: None 2023-10-11T15:35:52.491 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76?versionId=null 2023-10-11T15:35:52.491 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg= 2023-10-11T15:35:52.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ts90PTlmli2x/tQY4Tph9uuSCyg=', 'Content-Length': '0'} 2023-10-11T15:35:52.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c2215d9f30a33b4-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.494 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.495 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.495 DEBUG:boto:path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.496 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.496 DEBUG:boto:Method: GET 2023-10-11T15:35:52.496 DEBUG:boto:Path: /dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.497 DEBUG:boto:Data: 2023-10-11T15:35:52.497 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.498 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.499 DEBUG:boto:Params: {} 2023-10-11T15:35:52.499 DEBUG:boto:Token: None 2023-10-11T15:35:52.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f5cb7b102ce5639-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.504 DEBUG:boto:b'dghqvd-61obj761000falseobj762023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.505 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.505 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.505 DEBUG:boto:path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.506 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.506 DEBUG:boto:Method: GET 2023-10-11T15:35:52.507 DEBUG:boto:Path: /dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.507 DEBUG:boto:Data: 2023-10-11T15:35:52.508 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.508 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.509 DEBUG:boto:Params: {} 2023-10-11T15:35:52.509 DEBUG:boto:Token: None 2023-10-11T15:35:52.510 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.510 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673a31129df058f9-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.515 DEBUG:boto:b'dghqvd-61obj761000falseobj762023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.515 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.516 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.516 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.517 DEBUG:boto:Path: /dghqvd-61/obj76 2023-10-11T15:35:52.517 DEBUG:boto:Data: 2023-10-11T15:35:52.518 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.518 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.519 DEBUG:boto:Params: {} 2023-10-11T15:35:52.519 DEBUG:boto:Token: None 2023-10-11T15:35:52.520 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76 2023-10-11T15:35:52.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:boCYT/NbMjWs2oISBwFOdtCUYO8= 2023-10-11T15:35:52.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:boCYT/NbMjWs2oISBwFOdtCUYO8=', 'Content-Length': '0'} 2023-10-11T15:35:52.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060afc1e1e3061b19-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.523 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.524 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.524 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.525 DEBUG:boto:Path: /dghqvd-61/obj76 2023-10-11T15:35:52.525 DEBUG:boto:Data: 2023-10-11T15:35:52.525 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.526 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.526 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.527 DEBUG:boto:Params: {} 2023-10-11T15:35:52.527 DEBUG:boto:Token: None 2023-10-11T15:35:52.528 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76 2023-10-11T15:35:52.528 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:boCYT/NbMjWs2oISBwFOdtCUYO8= 2023-10-11T15:35:52.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:boCYT/NbMjWs2oISBwFOdtCUYO8=', 'Content-Length': '0'} 2023-10-11T15:35:52.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2373ad3271951d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-10-11T15:35:52.531 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.532 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.532 DEBUG:boto:Method: GET 2023-10-11T15:35:52.533 DEBUG:boto:Path: /dghqvd-61/obj76 2023-10-11T15:35:52.533 DEBUG:boto:Data: 2023-10-11T15:35:52.534 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.534 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.535 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.535 DEBUG:boto:Params: {} 2023-10-11T15:35:52.536 DEBUG:boto:Token: None 2023-10-11T15:35:52.536 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76 2023-10-11T15:35:52.537 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+cjyMrhO5QfYd9Zruxs3SQgfnqE= 2023-10-11T15:35:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+cjyMrhO5QfYd9Zruxs3SQgfnqE=', 'Content-Length': '0'} 2023-10-11T15:35:52.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f10533aa54f4f825-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.582 DEBUG:boto:path=/dghqvd-61/obj76 2023-10-11T15:35:52.583 DEBUG:boto:auth_path=/dghqvd-61/obj76 2023-10-11T15:35:52.584 DEBUG:boto:Method: GET 2023-10-11T15:35:52.584 DEBUG:boto:Path: /dghqvd-61/obj76 2023-10-11T15:35:52.584 DEBUG:boto:Data: 2023-10-11T15:35:52.585 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.585 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.586 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.586 DEBUG:boto:Params: {} 2023-10-11T15:35:52.587 DEBUG:boto:Token: None 2023-10-11T15:35:52.587 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj76 2023-10-11T15:35:52.588 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+cjyMrhO5QfYd9Zruxs3SQgfnqE= 2023-10-11T15:35:52.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+cjyMrhO5QfYd9Zruxs3SQgfnqE=', 'Content-Length': '0'} 2023-10-11T15:35:52.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa1ee77e1d8b58f-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.591 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.591 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.592 DEBUG:boto:path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.592 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.592 DEBUG:boto:Method: GET 2023-10-11T15:35:52.593 DEBUG:boto:Path: /dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.593 DEBUG:boto:Data: 2023-10-11T15:35:52.594 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.594 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.594 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.595 DEBUG:boto:Params: {} 2023-10-11T15:35:52.595 DEBUG:boto:Token: None 2023-10-11T15:35:52.596 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.596 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001610730fcba2cae8-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.599 DEBUG:boto:b'dghqvd-61obj761000falseobj762023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.600 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.601 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.601 DEBUG:boto:path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.601 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.602 DEBUG:boto:Method: GET 2023-10-11T15:35:52.602 DEBUG:boto:Path: /dghqvd-61/?prefix=obj76 2023-10-11T15:35:52.603 DEBUG:boto:Data: 2023-10-11T15:35:52.603 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.604 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.604 DEBUG:boto:Params: {} 2023-10-11T15:35:52.605 DEBUG:boto:Token: None 2023-10-11T15:35:52.605 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b83cf87474d6613-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.610 DEBUG:boto:b'dghqvd-61obj761000falseobj762023-10-11T15:31:43.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:35:52.612 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.612 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.612 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.613 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.613 DEBUG:boto:Method: GET 2023-10-11T15:35:52.614 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.614 DEBUG:boto:Data: 2023-10-11T15:35:52.615 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.615 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.615 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.616 DEBUG:boto:Params: {} 2023-10-11T15:35:52.616 DEBUG:boto:Token: None 2023-10-11T15:35:52.617 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.617 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk= 2023-10-11T15:35:52.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk=', 'Content-Length': '0'} 2023-10-11T15:35:52.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e13ac97e30c5ec-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.620 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.621 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.621 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.622 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.622 DEBUG:boto:Method: GET 2023-10-11T15:35:52.622 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.623 DEBUG:boto:Data: 2023-10-11T15:35:52.623 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.624 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.624 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.625 DEBUG:boto:Params: {} 2023-10-11T15:35:52.625 DEBUG:boto:Token: None 2023-10-11T15:35:52.626 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.626 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk= 2023-10-11T15:35:52.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk=', 'Content-Length': '0'} 2023-10-11T15:35:52.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003919ae6225a98dd7-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.629 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.630 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.630 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.630 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.631 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.631 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.632 DEBUG:boto:Data: 2023-10-11T15:35:52.632 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.633 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.633 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.633 DEBUG:boto:Params: {} 2023-10-11T15:35:52.634 DEBUG:boto:Token: None 2023-10-11T15:35:52.634 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.635 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nahiJwwpbw3rw43uksgwURLYKaY= 2023-10-11T15:35:52.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nahiJwwpbw3rw43uksgwURLYKaY=', 'Content-Length': '0'} 2023-10-11T15:35:52.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008adfce80ab61d9d-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.637 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.638 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.638 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.639 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.639 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.640 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.640 DEBUG:boto:Data: 2023-10-11T15:35:52.640 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.641 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.641 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.642 DEBUG:boto:Params: {} 2023-10-11T15:35:52.642 DEBUG:boto:Token: None 2023-10-11T15:35:52.643 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.643 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nahiJwwpbw3rw43uksgwURLYKaY= 2023-10-11T15:35:52.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nahiJwwpbw3rw43uksgwURLYKaY=', 'Content-Length': '0'} 2023-10-11T15:35:52.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06cac0f1b782b7d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:35:52.647 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.647 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.647 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.648 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.648 DEBUG:boto:Method: GET 2023-10-11T15:35:52.649 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.649 DEBUG:boto:Data: 2023-10-11T15:35:52.649 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.650 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.650 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.651 DEBUG:boto:Params: {} 2023-10-11T15:35:52.651 DEBUG:boto:Token: None 2023-10-11T15:35:52.652 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.652 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk= 2023-10-11T15:35:52.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk=', 'Content-Length': '0'} 2023-10-11T15:35:52.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d638b3247012e246-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.655 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.655 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.656 DEBUG:boto:path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.656 DEBUG:boto:auth_path=/dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.657 DEBUG:boto:Method: GET 2023-10-11T15:35:52.657 DEBUG:boto:Path: /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.658 DEBUG:boto:Data: 2023-10-11T15:35:52.658 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.659 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.659 DEBUG:boto:Params: {} 2023-10-11T15:35:52.660 DEBUG:boto:Token: None 2023-10-11T15:35:52.660 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85?versionId=null 2023-10-11T15:35:52.661 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk= 2023-10-11T15:35:52.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rg3q9EALJTE6ZkFf2+wc/Zof4dk=', 'Content-Length': '0'} 2023-10-11T15:35:52.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000482ec94d4e89e7e4-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.663 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.664 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.665 DEBUG:boto:path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.665 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.665 DEBUG:boto:Method: GET 2023-10-11T15:35:52.666 DEBUG:boto:Path: /dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.666 DEBUG:boto:Data: 2023-10-11T15:35:52.667 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.668 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.668 DEBUG:boto:Params: {} 2023-10-11T15:35:52.668 DEBUG:boto:Token: None 2023-10-11T15:35:52.669 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.669 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4356a490cd76dd9-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.714 DEBUG:boto:b'dghqvd-61obj851000falseobj852023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.715 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.716 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.716 DEBUG:boto:path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.717 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.717 DEBUG:boto:Method: GET 2023-10-11T15:35:52.718 DEBUG:boto:Path: /dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.718 DEBUG:boto:Data: 2023-10-11T15:35:52.719 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.720 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.721 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.721 DEBUG:boto:Params: {} 2023-10-11T15:35:52.722 DEBUG:boto:Token: None 2023-10-11T15:35:52.722 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.723 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c3e37cfc08d59cf-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.727 DEBUG:boto:b'dghqvd-61obj851000falseobj852023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.728 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.728 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.729 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.729 DEBUG:boto:Path: /dghqvd-61/obj85 2023-10-11T15:35:52.729 DEBUG:boto:Data: 2023-10-11T15:35:52.730 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.731 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.731 DEBUG:boto:Params: {} 2023-10-11T15:35:52.732 DEBUG:boto:Token: None 2023-10-11T15:35:52.732 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85 2023-10-11T15:35:52.733 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lPaNOwwFj/rqMJEwU6xt4UAjtNA= 2023-10-11T15:35:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lPaNOwwFj/rqMJEwU6xt4UAjtNA=', 'Content-Length': '0'} 2023-10-11T15:35:52.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e67037d2090ac06-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.735 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.736 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.736 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.737 DEBUG:boto:Path: /dghqvd-61/obj85 2023-10-11T15:35:52.737 DEBUG:boto:Data: 2023-10-11T15:35:52.738 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.738 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.739 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.739 DEBUG:boto:Params: {} 2023-10-11T15:35:52.739 DEBUG:boto:Token: None 2023-10-11T15:35:52.740 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85 2023-10-11T15:35:52.740 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lPaNOwwFj/rqMJEwU6xt4UAjtNA= 2023-10-11T15:35:52.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lPaNOwwFj/rqMJEwU6xt4UAjtNA=', 'Content-Length': '0'} 2023-10-11T15:35:52.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5484d808692837e-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-10-11T15:35:52.744 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.744 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.745 DEBUG:boto:Method: GET 2023-10-11T15:35:52.745 DEBUG:boto:Path: /dghqvd-61/obj85 2023-10-11T15:35:52.745 DEBUG:boto:Data: 2023-10-11T15:35:52.746 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.747 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.747 DEBUG:boto:Params: {} 2023-10-11T15:35:52.748 DEBUG:boto:Token: None 2023-10-11T15:35:52.748 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85 2023-10-11T15:35:52.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NmHoj0OEOAgLhAg6zxLxggXIYB0= 2023-10-11T15:35:52.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NmHoj0OEOAgLhAg6zxLxggXIYB0=', 'Content-Length': '0'} 2023-10-11T15:35:52.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12cf195d6a0ac59-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.751 DEBUG:boto:path=/dghqvd-61/obj85 2023-10-11T15:35:52.752 DEBUG:boto:auth_path=/dghqvd-61/obj85 2023-10-11T15:35:52.752 DEBUG:boto:Method: GET 2023-10-11T15:35:52.752 DEBUG:boto:Path: /dghqvd-61/obj85 2023-10-11T15:35:52.753 DEBUG:boto:Data: 2023-10-11T15:35:52.753 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.754 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.754 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.755 DEBUG:boto:Params: {} 2023-10-11T15:35:52.755 DEBUG:boto:Token: None 2023-10-11T15:35:52.756 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj85 2023-10-11T15:35:52.756 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NmHoj0OEOAgLhAg6zxLxggXIYB0= 2023-10-11T15:35:52.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NmHoj0OEOAgLhAg6zxLxggXIYB0=', 'Content-Length': '0'} 2023-10-11T15:35:52.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498431b6e5a855e7-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.798 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.799 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.799 DEBUG:boto:path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.800 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.801 DEBUG:boto:Method: GET 2023-10-11T15:35:52.801 DEBUG:boto:Path: /dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.801 DEBUG:boto:Data: 2023-10-11T15:35:52.802 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.802 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.803 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.803 DEBUG:boto:Params: {} 2023-10-11T15:35:52.804 DEBUG:boto:Token: None 2023-10-11T15:35:52.804 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.805 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ce6cf3d76f264f-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.809 DEBUG:boto:b'dghqvd-61obj851000falseobj852023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.810 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.810 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.811 DEBUG:boto:path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.811 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.812 DEBUG:boto:Method: GET 2023-10-11T15:35:52.812 DEBUG:boto:Path: /dghqvd-61/?prefix=obj85 2023-10-11T15:35:52.813 DEBUG:boto:Data: 2023-10-11T15:35:52.813 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.814 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.814 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.814 DEBUG:boto:Params: {} 2023-10-11T15:35:52.815 DEBUG:boto:Token: None 2023-10-11T15:35:52.816 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.817 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093cfe3cd9bac650f-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.822 DEBUG:boto:b'dghqvd-61obj851000falseobj852023-10-11T15:31:42.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:35:52.823 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.824 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.824 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.825 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.826 DEBUG:boto:Method: GET 2023-10-11T15:35:52.826 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.827 DEBUG:boto:Data: 2023-10-11T15:35:52.827 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.828 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.828 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.829 DEBUG:boto:Params: {} 2023-10-11T15:35:52.829 DEBUG:boto:Token: None 2023-10-11T15:35:52.830 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.830 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo= 2023-10-11T15:35:52.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo=', 'Content-Length': '0'} 2023-10-11T15:35:52.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce47d7a6e4222f5a-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.833 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.834 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.834 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.835 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.835 DEBUG:boto:Method: GET 2023-10-11T15:35:52.835 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.836 DEBUG:boto:Data: 2023-10-11T15:35:52.836 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.837 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.837 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.838 DEBUG:boto:Params: {} 2023-10-11T15:35:52.838 DEBUG:boto:Token: None 2023-10-11T15:35:52.839 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.839 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo= 2023-10-11T15:35:52.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo=', 'Content-Length': '0'} 2023-10-11T15:35:52.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf1379d4d32188d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.843 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.843 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.844 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.845 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.845 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.845 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.846 DEBUG:boto:Data: 2023-10-11T15:35:52.846 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.847 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.847 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.847 DEBUG:boto:Params: {} 2023-10-11T15:35:52.848 DEBUG:boto:Token: None 2023-10-11T15:35:52.848 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s1TQ5T4dNzAZR8/Lx+Icl2TclgU= 2023-10-11T15:35:52.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s1TQ5T4dNzAZR8/Lx+Icl2TclgU=', 'Content-Length': '0'} 2023-10-11T15:35:52.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095fdda2a4207352e-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.852 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.852 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.853 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.853 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.854 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.855 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.856 DEBUG:boto:Data: 2023-10-11T15:35:52.856 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.857 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.858 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.858 DEBUG:boto:Params: {} 2023-10-11T15:35:52.859 DEBUG:boto:Token: None 2023-10-11T15:35:52.860 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.861 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:s1TQ5T4dNzAZR8/Lx+Icl2TclgU= 2023-10-11T15:35:52.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:s1TQ5T4dNzAZR8/Lx+Icl2TclgU=', 'Content-Length': '0'} 2023-10-11T15:35:52.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7fc3cc94214096-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:35:52.864 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.865 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.865 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.866 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.866 DEBUG:boto:Method: GET 2023-10-11T15:35:52.867 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.867 DEBUG:boto:Data: 2023-10-11T15:35:52.868 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.868 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.869 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.869 DEBUG:boto:Params: {} 2023-10-11T15:35:52.870 DEBUG:boto:Token: None 2023-10-11T15:35:52.870 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo= 2023-10-11T15:35:52.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo=', 'Content-Length': '0'} 2023-10-11T15:35:52.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3c0026096c3c03-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.914 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.915 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.916 DEBUG:boto:path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.916 DEBUG:boto:auth_path=/dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.916 DEBUG:boto:Method: GET 2023-10-11T15:35:52.917 DEBUG:boto:Path: /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.917 DEBUG:boto:Data: 2023-10-11T15:35:52.918 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.919 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.919 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.919 DEBUG:boto:Params: {} 2023-10-11T15:35:52.920 DEBUG:boto:Token: None 2023-10-11T15:35:52.920 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87?versionId=null 2023-10-11T15:35:52.921 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo= 2023-10-11T15:35:52.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4TVBUNWw0/EjPx8LnDY4jW4pWQo=', 'Content-Length': '0'} 2023-10-11T15:35:52.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065369dbb3481f91d-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.924 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.925 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.925 DEBUG:boto:path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.926 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.926 DEBUG:boto:Method: GET 2023-10-11T15:35:52.927 DEBUG:boto:Path: /dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.927 DEBUG:boto:Data: 2023-10-11T15:35:52.928 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.929 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.929 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.930 DEBUG:boto:Params: {} 2023-10-11T15:35:52.930 DEBUG:boto:Token: None 2023-10-11T15:35:52.931 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.931 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000262f72cf5177c74c-006526c0d8-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.935 DEBUG:boto:b'dghqvd-61obj871000falseobj872023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.936 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:52.936 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:52.937 DEBUG:boto:path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.937 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.938 DEBUG:boto:Method: GET 2023-10-11T15:35:52.938 DEBUG:boto:Path: /dghqvd-61/?prefix=obj87 2023-10-11T15:35:52.939 DEBUG:boto:Data: 2023-10-11T15:35:52.939 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.940 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.940 DEBUG:boto:Params: {} 2023-10-11T15:35:52.941 DEBUG:boto:Token: None 2023-10-11T15:35:52.941 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/ 2023-10-11T15:35:52.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ= 2023-10-11T15:35:52.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tWMQ4i9NvpBceWnSnEge3MoB0NQ=', 'Content-Length': '0'} 2023-10-11T15:35:52.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4d0d20d7fdd5d3e-006526c0d8-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.947 DEBUG:boto:b'dghqvd-61obj871000falseobj872023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:52.948 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.948 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.949 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.949 DEBUG:boto:Path: /dghqvd-61/obj87 2023-10-11T15:35:52.950 DEBUG:boto:Data: 2023-10-11T15:35:52.950 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.951 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.951 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.951 DEBUG:boto:Params: {} 2023-10-11T15:35:52.952 DEBUG:boto:Token: None 2023-10-11T15:35:52.952 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87 2023-10-11T15:35:52.953 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:94Vfs4QkKTUu0WqHHUJiAvh61oY= 2023-10-11T15:35:52.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:94Vfs4QkKTUu0WqHHUJiAvh61oY=', 'Content-Length': '0'} 2023-10-11T15:35:52.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bddd4b559f8798cd-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.956 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.956 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.957 DEBUG:boto:Method: HEAD 2023-10-11T15:35:52.957 DEBUG:boto:Path: /dghqvd-61/obj87 2023-10-11T15:35:52.958 DEBUG:boto:Data: 2023-10-11T15:35:52.958 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.959 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.959 DEBUG:boto:Params: {} 2023-10-11T15:35:52.960 DEBUG:boto:Token: None 2023-10-11T15:35:52.960 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87 2023-10-11T15:35:52.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:94Vfs4QkKTUu0WqHHUJiAvh61oY= 2023-10-11T15:35:52.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:94Vfs4QkKTUu0WqHHUJiAvh61oY=', 'Content-Length': '0'} 2023-10-11T15:35:52.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e293af8a91885814-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-10-11T15:35:52.964 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.964 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.965 DEBUG:boto:Method: GET 2023-10-11T15:35:52.965 DEBUG:boto:Path: /dghqvd-61/obj87 2023-10-11T15:35:52.966 DEBUG:boto:Data: 2023-10-11T15:35:52.966 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:52.967 DEBUG:boto:Port: 8001 2023-10-11T15:35:52.967 DEBUG:boto:Params: {} 2023-10-11T15:35:52.968 DEBUG:boto:Token: None 2023-10-11T15:35:52.968 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87 2023-10-11T15:35:52.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/l9mfi7kCOAmrH0OSfA2Rhg4Fm4= 2023-10-11T15:35:52.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/l9mfi7kCOAmrH0OSfA2Rhg4Fm4=', 'Content-Length': '0'} 2023-10-11T15:35:52.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009387f36c36df45bc-006526c0d8-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:52.972 DEBUG:boto:path=/dghqvd-61/obj87 2023-10-11T15:35:52.973 DEBUG:boto:auth_path=/dghqvd-61/obj87 2023-10-11T15:35:52.974 DEBUG:boto:Method: GET 2023-10-11T15:35:52.974 DEBUG:boto:Path: /dghqvd-61/obj87 2023-10-11T15:35:52.975 DEBUG:boto:Data: 2023-10-11T15:35:52.975 DEBUG:boto:Headers: {} 2023-10-11T15:35:52.976 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:52.976 DEBUG:boto:Port: 8000 2023-10-11T15:35:52.976 DEBUG:boto:Params: {} 2023-10-11T15:35:52.977 DEBUG:boto:Token: None 2023-10-11T15:35:52.977 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:52 GMT /dghqvd-61/obj87 2023-10-11T15:35:52.978 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/l9mfi7kCOAmrH0OSfA2Rhg4Fm4= 2023-10-11T15:35:52.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/l9mfi7kCOAmrH0OSfA2Rhg4Fm4=', 'Content-Length': '0'} 2023-10-11T15:35:52.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ee26720086cc6a-006526c0d8-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.022 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.023 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.024 DEBUG:boto:path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.024 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.024 DEBUG:boto:Method: GET 2023-10-11T15:35:53.025 DEBUG:boto:Path: /dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.025 DEBUG:boto:Data: 2023-10-11T15:35:53.026 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.026 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.027 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.027 DEBUG:boto:Params: {} 2023-10-11T15:35:53.028 DEBUG:boto:Token: None 2023-10-11T15:35:53.028 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e74eb26624d083-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.032 DEBUG:boto:b'dghqvd-61obj871000falseobj872023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.033 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.034 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.035 DEBUG:boto:path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.036 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.036 DEBUG:boto:Method: GET 2023-10-11T15:35:53.037 DEBUG:boto:Path: /dghqvd-61/?prefix=obj87 2023-10-11T15:35:53.038 DEBUG:boto:Data: 2023-10-11T15:35:53.038 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.039 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.039 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.040 DEBUG:boto:Params: {} 2023-10-11T15:35:53.040 DEBUG:boto:Token: None 2023-10-11T15:35:53.040 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.041 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08fec392f7e35c5-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.046 DEBUG:boto:b'dghqvd-61obj871000falseobj872023-10-11T15:31:43.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:35:53.047 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.047 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.048 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.048 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.049 DEBUG:boto:Method: GET 2023-10-11T15:35:53.049 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.049 DEBUG:boto:Data: 2023-10-11T15:35:53.050 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.050 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.051 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.051 DEBUG:boto:Params: {} 2023-10-11T15:35:53.052 DEBUG:boto:Token: None 2023-10-11T15:35:53.052 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.053 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I= 2023-10-11T15:35:53.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I=', 'Content-Length': '0'} 2023-10-11T15:35:53.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823e76519c9c3932-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.098 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.099 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.100 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.100 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.101 DEBUG:boto:Method: GET 2023-10-11T15:35:53.101 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.102 DEBUG:boto:Data: 2023-10-11T15:35:53.102 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.103 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.103 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.103 DEBUG:boto:Params: {} 2023-10-11T15:35:53.104 DEBUG:boto:Token: None 2023-10-11T15:35:53.105 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I= 2023-10-11T15:35:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I=', 'Content-Length': '0'} 2023-10-11T15:35:53.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306c5ec5f3d36c33-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.110 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.110 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.111 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.111 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.112 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.112 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.113 DEBUG:boto:Data: 2023-10-11T15:35:53.113 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.113 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.114 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.114 DEBUG:boto:Params: {} 2023-10-11T15:35:53.115 DEBUG:boto:Token: None 2023-10-11T15:35:53.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.116 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v3+RphGmNj8t7EHLa4rk7sWwGdQ= 2023-10-11T15:35:53.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v3+RphGmNj8t7EHLa4rk7sWwGdQ=', 'Content-Length': '0'} 2023-10-11T15:35:53.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c808f69268de6e3-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.119 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.120 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.120 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.120 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.121 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.121 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.122 DEBUG:boto:Data: 2023-10-11T15:35:53.122 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.123 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.123 DEBUG:boto:Params: {} 2023-10-11T15:35:53.124 DEBUG:boto:Token: None 2023-10-11T15:35:53.124 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.125 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:v3+RphGmNj8t7EHLa4rk7sWwGdQ= 2023-10-11T15:35:53.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:v3+RphGmNj8t7EHLa4rk7sWwGdQ=', 'Content-Length': '0'} 2023-10-11T15:35:53.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f589cc307d6d3d2-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:35:53.128 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.129 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.129 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.130 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.130 DEBUG:boto:Method: GET 2023-10-11T15:35:53.131 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.131 DEBUG:boto:Data: 2023-10-11T15:35:53.132 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.132 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.133 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.133 DEBUG:boto:Params: {} 2023-10-11T15:35:53.134 DEBUG:boto:Token: None 2023-10-11T15:35:53.134 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.135 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I= 2023-10-11T15:35:53.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I=', 'Content-Length': '0'} 2023-10-11T15:35:53.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b9ab63686032be-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.139 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.140 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.140 DEBUG:boto:path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.141 DEBUG:boto:auth_path=/dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.142 DEBUG:boto:Method: GET 2023-10-11T15:35:53.143 DEBUG:boto:Path: /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.144 DEBUG:boto:Data: 2023-10-11T15:35:53.145 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.146 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.146 DEBUG:boto:Params: {} 2023-10-11T15:35:53.147 DEBUG:boto:Token: None 2023-10-11T15:35:53.147 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92?versionId=null 2023-10-11T15:35:53.148 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I= 2023-10-11T15:35:53.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+kvl/rhnWurzXlCGfb/WVvczV7I=', 'Content-Length': '0'} 2023-10-11T15:35:53.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2547d16288bc97-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.151 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.151 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.152 DEBUG:boto:path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.152 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.152 DEBUG:boto:Method: GET 2023-10-11T15:35:53.153 DEBUG:boto:Path: /dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.153 DEBUG:boto:Data: 2023-10-11T15:35:53.154 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.155 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.155 DEBUG:boto:Params: {} 2023-10-11T15:35:53.155 DEBUG:boto:Token: None 2023-10-11T15:35:53.156 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.157 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f416d171bbc9ad76-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.160 DEBUG:boto:b'dghqvd-61obj921000falseobj922023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.161 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.161 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.162 DEBUG:boto:path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.162 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.163 DEBUG:boto:Method: GET 2023-10-11T15:35:53.163 DEBUG:boto:Path: /dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.164 DEBUG:boto:Data: 2023-10-11T15:35:53.164 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.165 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.165 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.166 DEBUG:boto:Params: {} 2023-10-11T15:35:53.166 DEBUG:boto:Token: None 2023-10-11T15:35:53.167 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.168 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003797e065c9bbcbce-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.173 DEBUG:boto:b'dghqvd-61obj921000falseobj922023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.174 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.175 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.176 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.176 DEBUG:boto:Path: /dghqvd-61/obj92 2023-10-11T15:35:53.177 DEBUG:boto:Data: 2023-10-11T15:35:53.178 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.178 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.179 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.181 DEBUG:boto:Params: {} 2023-10-11T15:35:53.182 DEBUG:boto:Token: None 2023-10-11T15:35:53.183 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92 2023-10-11T15:35:53.184 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nQ0hAnd/exzllARlHCe+klXvgNo= 2023-10-11T15:35:53.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nQ0hAnd/exzllARlHCe+klXvgNo=', 'Content-Length': '0'} 2023-10-11T15:35:53.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285392aaec369317-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.187 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.187 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.188 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.188 DEBUG:boto:Path: /dghqvd-61/obj92 2023-10-11T15:35:53.189 DEBUG:boto:Data: 2023-10-11T15:35:53.189 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.190 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.190 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.190 DEBUG:boto:Params: {} 2023-10-11T15:35:53.191 DEBUG:boto:Token: None 2023-10-11T15:35:53.191 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92 2023-10-11T15:35:53.192 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nQ0hAnd/exzllARlHCe+klXvgNo= 2023-10-11T15:35:53.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nQ0hAnd/exzllARlHCe+klXvgNo=', 'Content-Length': '0'} 2023-10-11T15:35:53.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df795d2d33c6ad5e-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-10-11T15:35:53.196 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.196 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.196 DEBUG:boto:Method: GET 2023-10-11T15:35:53.197 DEBUG:boto:Path: /dghqvd-61/obj92 2023-10-11T15:35:53.197 DEBUG:boto:Data: 2023-10-11T15:35:53.198 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.199 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.199 DEBUG:boto:Params: {} 2023-10-11T15:35:53.200 DEBUG:boto:Token: None 2023-10-11T15:35:53.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92 2023-10-11T15:35:53.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2xMOefgaGdjE5FJ8cWmTEeCOYFs= 2023-10-11T15:35:53.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2xMOefgaGdjE5FJ8cWmTEeCOYFs=', 'Content-Length': '0'} 2023-10-11T15:35:53.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545901aced97601f-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.246 DEBUG:boto:path=/dghqvd-61/obj92 2023-10-11T15:35:53.247 DEBUG:boto:auth_path=/dghqvd-61/obj92 2023-10-11T15:35:53.247 DEBUG:boto:Method: GET 2023-10-11T15:35:53.248 DEBUG:boto:Path: /dghqvd-61/obj92 2023-10-11T15:35:53.248 DEBUG:boto:Data: 2023-10-11T15:35:53.249 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.249 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.250 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.250 DEBUG:boto:Params: {} 2023-10-11T15:35:53.251 DEBUG:boto:Token: None 2023-10-11T15:35:53.251 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj92 2023-10-11T15:35:53.252 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2xMOefgaGdjE5FJ8cWmTEeCOYFs= 2023-10-11T15:35:53.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2xMOefgaGdjE5FJ8cWmTEeCOYFs=', 'Content-Length': '0'} 2023-10-11T15:35:53.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051ffc400953c4459-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.255 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.255 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.256 DEBUG:boto:path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.256 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.257 DEBUG:boto:Method: GET 2023-10-11T15:35:53.257 DEBUG:boto:Path: /dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.258 DEBUG:boto:Data: 2023-10-11T15:35:53.258 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.259 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.260 DEBUG:boto:Params: {} 2023-10-11T15:35:53.260 DEBUG:boto:Token: None 2023-10-11T15:35:53.261 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f72fde94f2e40ae-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.265 DEBUG:boto:b'dghqvd-61obj921000falseobj922023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.266 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.267 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.268 DEBUG:boto:path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.268 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.269 DEBUG:boto:Method: GET 2023-10-11T15:35:53.269 DEBUG:boto:Path: /dghqvd-61/?prefix=obj92 2023-10-11T15:35:53.270 DEBUG:boto:Data: 2023-10-11T15:35:53.270 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.271 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.272 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.272 DEBUG:boto:Params: {} 2023-10-11T15:35:53.273 DEBUG:boto:Token: None 2023-10-11T15:35:53.273 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.274 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe1f56bcfe75a06e-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.279 DEBUG:boto:b'dghqvd-61obj921000falseobj922023-10-11T15:31:43.528Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:35:53.280 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.280 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.281 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.281 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.282 DEBUG:boto:Method: GET 2023-10-11T15:35:53.282 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.283 DEBUG:boto:Data: 2023-10-11T15:35:53.284 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.285 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.285 DEBUG:boto:Params: {} 2023-10-11T15:35:53.285 DEBUG:boto:Token: None 2023-10-11T15:35:53.286 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k= 2023-10-11T15:35:53.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k=', 'Content-Length': '0'} 2023-10-11T15:35:53.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e69c03ea82931a-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.290 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.290 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.290 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.291 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.291 DEBUG:boto:Method: GET 2023-10-11T15:35:53.292 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.292 DEBUG:boto:Data: 2023-10-11T15:35:53.293 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.293 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.294 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.294 DEBUG:boto:Params: {} 2023-10-11T15:35:53.295 DEBUG:boto:Token: None 2023-10-11T15:35:53.296 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k= 2023-10-11T15:35:53.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k=', 'Content-Length': '0'} 2023-10-11T15:35:53.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c9a2ddc6a2bf3f-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.299 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.300 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.300 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.301 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.301 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.301 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.302 DEBUG:boto:Data: 2023-10-11T15:35:53.302 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.303 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.303 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.304 DEBUG:boto:Params: {} 2023-10-11T15:35:53.304 DEBUG:boto:Token: None 2023-10-11T15:35:53.304 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gnpHy2030xwhJ7kfJrvmxWaMAA4= 2023-10-11T15:35:53.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gnpHy2030xwhJ7kfJrvmxWaMAA4=', 'Content-Length': '0'} 2023-10-11T15:35:53.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062daf90aa0bf941-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.308 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.309 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.309 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.310 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.310 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.310 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.311 DEBUG:boto:Data: 2023-10-11T15:35:53.311 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.312 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.312 DEBUG:boto:Params: {} 2023-10-11T15:35:53.313 DEBUG:boto:Token: None 2023-10-11T15:35:53.313 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gnpHy2030xwhJ7kfJrvmxWaMAA4= 2023-10-11T15:35:53.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gnpHy2030xwhJ7kfJrvmxWaMAA4=', 'Content-Length': '0'} 2023-10-11T15:35:53.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004781a19d812d9ff8-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:35:53.317 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.318 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.318 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.319 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.319 DEBUG:boto:Method: GET 2023-10-11T15:35:53.319 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.320 DEBUG:boto:Data: 2023-10-11T15:35:53.321 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.321 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.321 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.322 DEBUG:boto:Params: {} 2023-10-11T15:35:53.322 DEBUG:boto:Token: None 2023-10-11T15:35:53.323 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k= 2023-10-11T15:35:53.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k=', 'Content-Length': '0'} 2023-10-11T15:35:53.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c109f689e623462-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.326 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.327 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.327 DEBUG:boto:path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.328 DEBUG:boto:auth_path=/dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.328 DEBUG:boto:Method: GET 2023-10-11T15:35:53.329 DEBUG:boto:Path: /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.329 DEBUG:boto:Data: 2023-10-11T15:35:53.330 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.330 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.331 DEBUG:boto:Params: {} 2023-10-11T15:35:53.331 DEBUG:boto:Token: None 2023-10-11T15:35:53.332 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93?versionId=null 2023-10-11T15:35:53.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k= 2023-10-11T15:35:53.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Zikaq93FvyObKSFAPjzW2pZIr2k=', 'Content-Length': '0'} 2023-10-11T15:35:53.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009932f53fdc3d0d2f-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.336 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.336 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.337 DEBUG:boto:path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.337 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.338 DEBUG:boto:Method: GET 2023-10-11T15:35:53.338 DEBUG:boto:Path: /dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.339 DEBUG:boto:Data: 2023-10-11T15:35:53.339 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.340 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.340 DEBUG:boto:Params: {} 2023-10-11T15:35:53.341 DEBUG:boto:Token: None 2023-10-11T15:35:53.341 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c67a669202030e9-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.345 DEBUG:boto:b'dghqvd-61obj931000falseobj932023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.346 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.347 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.347 DEBUG:boto:path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.347 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.348 DEBUG:boto:Method: GET 2023-10-11T15:35:53.348 DEBUG:boto:Path: /dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.349 DEBUG:boto:Data: 2023-10-11T15:35:53.349 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.350 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.350 DEBUG:boto:Params: {} 2023-10-11T15:35:53.351 DEBUG:boto:Token: None 2023-10-11T15:35:53.351 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1754d96329fe8c1-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.398 DEBUG:boto:b'dghqvd-61obj931000falseobj932023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.399 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.400 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.400 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.400 DEBUG:boto:Path: /dghqvd-61/obj93 2023-10-11T15:35:53.401 DEBUG:boto:Data: 2023-10-11T15:35:53.401 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.402 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.402 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.402 DEBUG:boto:Params: {} 2023-10-11T15:35:53.403 DEBUG:boto:Token: None 2023-10-11T15:35:53.403 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93 2023-10-11T15:35:53.404 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sjnnES/eABW7r405SYqtTZbszCg= 2023-10-11T15:35:53.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sjnnES/eABW7r405SYqtTZbszCg=', 'Content-Length': '0'} 2023-10-11T15:35:53.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd2de51a6b8aa5b-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.407 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.407 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.408 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.408 DEBUG:boto:Path: /dghqvd-61/obj93 2023-10-11T15:35:53.409 DEBUG:boto:Data: 2023-10-11T15:35:53.409 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.410 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.410 DEBUG:boto:Params: {} 2023-10-11T15:35:53.411 DEBUG:boto:Token: None 2023-10-11T15:35:53.411 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93 2023-10-11T15:35:53.412 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sjnnES/eABW7r405SYqtTZbszCg= 2023-10-11T15:35:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sjnnES/eABW7r405SYqtTZbszCg=', 'Content-Length': '0'} 2023-10-11T15:35:53.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba5d48b39b3e55e-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-10-11T15:35:53.415 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.415 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.416 DEBUG:boto:Method: GET 2023-10-11T15:35:53.416 DEBUG:boto:Path: /dghqvd-61/obj93 2023-10-11T15:35:53.417 DEBUG:boto:Data: 2023-10-11T15:35:53.417 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.418 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.418 DEBUG:boto:Params: {} 2023-10-11T15:35:53.419 DEBUG:boto:Token: None 2023-10-11T15:35:53.419 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93 2023-10-11T15:35:53.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXkjRJWOTohIXkoWMyebu6InsQI= 2023-10-11T15:35:53.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXkjRJWOTohIXkoWMyebu6InsQI=', 'Content-Length': '0'} 2023-10-11T15:35:53.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088948894fcd611b5-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.423 DEBUG:boto:path=/dghqvd-61/obj93 2023-10-11T15:35:53.424 DEBUG:boto:auth_path=/dghqvd-61/obj93 2023-10-11T15:35:53.424 DEBUG:boto:Method: GET 2023-10-11T15:35:53.425 DEBUG:boto:Path: /dghqvd-61/obj93 2023-10-11T15:35:53.425 DEBUG:boto:Data: 2023-10-11T15:35:53.426 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.426 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.427 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.427 DEBUG:boto:Params: {} 2023-10-11T15:35:53.428 DEBUG:boto:Token: None 2023-10-11T15:35:53.428 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj93 2023-10-11T15:35:53.428 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NXkjRJWOTohIXkoWMyebu6InsQI= 2023-10-11T15:35:53.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NXkjRJWOTohIXkoWMyebu6InsQI=', 'Content-Length': '0'} 2023-10-11T15:35:53.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888e187e2f9de608-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.432 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.432 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.433 DEBUG:boto:path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.433 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.433 DEBUG:boto:Method: GET 2023-10-11T15:35:53.434 DEBUG:boto:Path: /dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.435 DEBUG:boto:Data: 2023-10-11T15:35:53.435 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.436 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.437 DEBUG:boto:Params: {} 2023-10-11T15:35:53.437 DEBUG:boto:Token: None 2023-10-11T15:35:53.438 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032bccb22468616d0-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.482 DEBUG:boto:b'dghqvd-61obj931000falseobj932023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.483 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.484 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.484 DEBUG:boto:path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.484 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.485 DEBUG:boto:Method: GET 2023-10-11T15:35:53.485 DEBUG:boto:Path: /dghqvd-61/?prefix=obj93 2023-10-11T15:35:53.486 DEBUG:boto:Data: 2023-10-11T15:35:53.486 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.487 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.487 DEBUG:boto:Params: {} 2023-10-11T15:35:53.488 DEBUG:boto:Token: None 2023-10-11T15:35:53.488 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc01d7bc49f1a69e-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.493 DEBUG:boto:b'dghqvd-61obj931000falseobj932023-10-11T15:31:44.160Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:35:53.495 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.495 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.495 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.496 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.496 DEBUG:boto:Method: GET 2023-10-11T15:35:53.496 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.497 DEBUG:boto:Data: 2023-10-11T15:35:53.497 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.498 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.498 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.499 DEBUG:boto:Params: {} 2023-10-11T15:35:53.499 DEBUG:boto:Token: None 2023-10-11T15:35:53.500 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.500 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0= 2023-10-11T15:35:53.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0=', 'Content-Length': '0'} 2023-10-11T15:35:53.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f25b89c4e6d2a0-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.503 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.504 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.504 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.505 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.505 DEBUG:boto:Method: GET 2023-10-11T15:35:53.505 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.506 DEBUG:boto:Data: 2023-10-11T15:35:53.506 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.507 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.507 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.507 DEBUG:boto:Params: {} 2023-10-11T15:35:53.508 DEBUG:boto:Token: None 2023-10-11T15:35:53.508 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.509 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0= 2023-10-11T15:35:53.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0=', 'Content-Length': '0'} 2023-10-11T15:35:53.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0ccdb6befa346c-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.512 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.513 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.513 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.513 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.514 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.514 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.515 DEBUG:boto:Data: 2023-10-11T15:35:53.515 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.515 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.516 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.516 DEBUG:boto:Params: {} 2023-10-11T15:35:53.517 DEBUG:boto:Token: None 2023-10-11T15:35:53.517 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.518 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1lm9eLd2n+islfrF4QWVUVuacOA= 2023-10-11T15:35:53.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1lm9eLd2n+islfrF4QWVUVuacOA=', 'Content-Length': '0'} 2023-10-11T15:35:53.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdcb520e07c645a0-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.521 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.521 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.521 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.522 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.522 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.523 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.523 DEBUG:boto:Data: 2023-10-11T15:35:53.523 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.524 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.524 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.525 DEBUG:boto:Params: {} 2023-10-11T15:35:53.525 DEBUG:boto:Token: None 2023-10-11T15:35:53.526 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.526 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1lm9eLd2n+islfrF4QWVUVuacOA= 2023-10-11T15:35:53.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1lm9eLd2n+islfrF4QWVUVuacOA=', 'Content-Length': '0'} 2023-10-11T15:35:53.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53dd17ee6160757-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:35:53.529 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.530 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.530 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.530 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.531 DEBUG:boto:Method: GET 2023-10-11T15:35:53.531 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.532 DEBUG:boto:Data: 2023-10-11T15:35:53.532 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.533 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.533 DEBUG:boto:Params: {} 2023-10-11T15:35:53.534 DEBUG:boto:Token: None 2023-10-11T15:35:53.534 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0= 2023-10-11T15:35:53.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0=', 'Content-Length': '0'} 2023-10-11T15:35:53.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d409036be779d9e-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.538 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.538 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.539 DEBUG:boto:path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.539 DEBUG:boto:auth_path=/dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.540 DEBUG:boto:Method: GET 2023-10-11T15:35:53.540 DEBUG:boto:Path: /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.541 DEBUG:boto:Data: 2023-10-11T15:35:53.541 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.542 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.543 DEBUG:boto:Params: {} 2023-10-11T15:35:53.543 DEBUG:boto:Token: None 2023-10-11T15:35:53.544 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95?versionId=null 2023-10-11T15:35:53.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0= 2023-10-11T15:35:53.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HRT436RjFzIocYk5O1yml4zmpo0=', 'Content-Length': '0'} 2023-10-11T15:35:53.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaab78e49471a265-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.547 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.547 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.548 DEBUG:boto:path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.548 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.548 DEBUG:boto:Method: GET 2023-10-11T15:35:53.549 DEBUG:boto:Path: /dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.549 DEBUG:boto:Data: 2023-10-11T15:35:53.550 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.550 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.551 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.551 DEBUG:boto:Params: {} 2023-10-11T15:35:53.551 DEBUG:boto:Token: None 2023-10-11T15:35:53.552 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.552 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020ba38ca40e46d41-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.556 DEBUG:boto:b'dghqvd-61obj951000falseobj952023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.556 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.557 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.557 DEBUG:boto:path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.557 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.558 DEBUG:boto:Method: GET 2023-10-11T15:35:53.558 DEBUG:boto:Path: /dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.559 DEBUG:boto:Data: 2023-10-11T15:35:53.559 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.560 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.560 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.560 DEBUG:boto:Params: {} 2023-10-11T15:35:53.561 DEBUG:boto:Token: None 2023-10-11T15:35:53.561 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.562 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d943b45275b972-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.567 DEBUG:boto:b'dghqvd-61obj951000falseobj952023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.567 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.568 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.568 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.569 DEBUG:boto:Path: /dghqvd-61/obj95 2023-10-11T15:35:53.569 DEBUG:boto:Data: 2023-10-11T15:35:53.569 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.570 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.570 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.571 DEBUG:boto:Params: {} 2023-10-11T15:35:53.571 DEBUG:boto:Token: None 2023-10-11T15:35:53.572 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95 2023-10-11T15:35:53.572 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8GaxWQVPwr9uNTpKjefE02QN5bQ= 2023-10-11T15:35:53.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8GaxWQVPwr9uNTpKjefE02QN5bQ=', 'Content-Length': '0'} 2023-10-11T15:35:53.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a516587827d7560-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.575 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.575 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.576 DEBUG:boto:Method: HEAD 2023-10-11T15:35:53.576 DEBUG:boto:Path: /dghqvd-61/obj95 2023-10-11T15:35:53.577 DEBUG:boto:Data: 2023-10-11T15:35:53.577 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.577 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.578 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.578 DEBUG:boto:Params: {} 2023-10-11T15:35:53.579 DEBUG:boto:Token: None 2023-10-11T15:35:53.579 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95 2023-10-11T15:35:53.580 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8GaxWQVPwr9uNTpKjefE02QN5bQ= 2023-10-11T15:35:53.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8GaxWQVPwr9uNTpKjefE02QN5bQ=', 'Content-Length': '0'} 2023-10-11T15:35:53.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91b37e7ef7cac1f-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-10-11T15:35:53.583 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.583 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.584 DEBUG:boto:Method: GET 2023-10-11T15:35:53.584 DEBUG:boto:Path: /dghqvd-61/obj95 2023-10-11T15:35:53.585 DEBUG:boto:Data: 2023-10-11T15:35:53.585 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.586 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.586 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.587 DEBUG:boto:Params: {} 2023-10-11T15:35:53.587 DEBUG:boto:Token: None 2023-10-11T15:35:53.587 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95 2023-10-11T15:35:53.589 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MfkEraM8wIxSr1g4lWEhPawWlOA= 2023-10-11T15:35:53.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MfkEraM8wIxSr1g4lWEhPawWlOA=', 'Content-Length': '0'} 2023-10-11T15:35:53.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-61:daa316e9-869c-4074-a756-a8adfb4a3573.5751.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d6a50f24851db5-006526c0d9-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.634 DEBUG:boto:path=/dghqvd-61/obj95 2023-10-11T15:35:53.635 DEBUG:boto:auth_path=/dghqvd-61/obj95 2023-10-11T15:35:53.635 DEBUG:boto:Method: GET 2023-10-11T15:35:53.636 DEBUG:boto:Path: /dghqvd-61/obj95 2023-10-11T15:35:53.636 DEBUG:boto:Data: 2023-10-11T15:35:53.637 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.638 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.638 DEBUG:boto:Params: {} 2023-10-11T15:35:53.639 DEBUG:boto:Token: None 2023-10-11T15:35:53.639 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/obj95 2023-10-11T15:35:53.640 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MfkEraM8wIxSr1g4lWEhPawWlOA= 2023-10-11T15:35:53.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MfkEraM8wIxSr1g4lWEhPawWlOA=', 'Content-Length': '0'} 2023-10-11T15:35:53.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:31:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009948800a687ccb4e-006526c0d9-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.643 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.643 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.644 DEBUG:boto:path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.644 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.644 DEBUG:boto:Method: GET 2023-10-11T15:35:53.645 DEBUG:boto:Path: /dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.645 DEBUG:boto:Data: 2023-10-11T15:35:53.646 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.646 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:35:53.647 DEBUG:boto:Port: 8001 2023-10-11T15:35:53.647 DEBUG:boto:Params: {} 2023-10-11T15:35:53.647 DEBUG:boto:Token: None 2023-10-11T15:35:53.648 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.648 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009daed00a88ffefd1-006526c0d9-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.651 DEBUG:boto:b'dghqvd-61obj951000falseobj952023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.652 DEBUG:boto:path=/dghqvd-61/ 2023-10-11T15:35:53.653 DEBUG:boto:auth_path=/dghqvd-61/ 2023-10-11T15:35:53.653 DEBUG:boto:path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.653 DEBUG:boto:auth_path=/dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.654 DEBUG:boto:Method: GET 2023-10-11T15:35:53.654 DEBUG:boto:Path: /dghqvd-61/?prefix=obj95 2023-10-11T15:35:53.655 DEBUG:boto:Data: 2023-10-11T15:35:53.655 DEBUG:boto:Headers: {} 2023-10-11T15:35:53.656 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:53.656 DEBUG:boto:Port: 8000 2023-10-11T15:35:53.657 DEBUG:boto:Params: {} 2023-10-11T15:35:53.657 DEBUG:boto:Token: None 2023-10-11T15:35:53.657 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:35:53 GMT /dghqvd-61/ 2023-10-11T15:35:53.658 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI= 2023-10-11T15:35:53.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:53 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:InxQibT8SIkBVo+0eR8nRFRZcfI=', 'Content-Length': '0'} 2023-10-11T15:35:53.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfdf05c4566eae6c-006526c0d9-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:35:53 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:53.662 DEBUG:boto:b'dghqvd-61obj951000falseobj952023-10-11T15:31:43.203Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:35:53.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-61 zones={a2, a1} 2023-10-11T15:35:53.664 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T15:35:53.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.666 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:53.716 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:53.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:53.766 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:35:53.766 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.766 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032347", 2023-10-11T15:35:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:57.117433Z" 2023-10-11T15:35:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:35:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:35:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037387", 2023-10-11T15:35:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:44.605809Z" 2023-10-11T15:35:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032294", 2023-10-11T15:35:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:57.120590Z" 2023-10-11T15:35:53.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.771 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:35:53.772 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032347', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037387', 3: '00000000000000000000:00000000000000032294'} 2023-10-11T15:35:53.772 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:35:53.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.774 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:53.825 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:53.825 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:53.869 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:35:53.870 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:35:53.870 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:35:53.870 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:35:53.870 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:35:53.871 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 5291180467943621742 2023-10-11T15:35:53.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.871 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:35:53.871 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.871 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:35:53.872 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:53.872 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:53.872 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032347", 2023-10-11T15:35:53.872 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:53.872 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:53.873 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:53.873 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.117433Z" 2023-10-11T15:35:53.873 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.873 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.873 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.874 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:35:53.874 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:53.874 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:53.874 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:35:53.875 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:53.875 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:53.875 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:53.875 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:35:53.875 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.876 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.876 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.876 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:35:53.876 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:53.876 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:53.877 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037387", 2023-10-11T15:35:53.877 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:53.877 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 52, 2023-10-11T15:35:53.877 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:53.877 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:44.605809Z" 2023-10-11T15:35:53.878 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.878 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.878 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:35:53.879 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:53.879 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:53.879 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032294", 2023-10-11T15:35:53.879 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:53.879 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:35:53.880 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:53.880 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.120590Z" 2023-10-11T15:35:53.880 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.880 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.880 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:35:53.881 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.881 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:35:53.881 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 61, 2023-10-11T15:35:53.881 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 61 2023-10-11T15:35:53.881 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.882 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:35:53Z" 2023-10-11T15:35:53.882 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:35:53.883 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5291180467943621742 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032347", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.117433Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023427", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037387", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-10-11T15:31:44.605809Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032294", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:31:57.120590Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-10-11T15:35:53Z" } 2023-10-11T15:35:53.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032347', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.117433Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023427', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037387', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-10-11T15:31:44.605809Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032294', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.120590Z'}}] 2023-10-11T15:35:53.884 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032347', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037387', 3: '00000000000000000000:00000000000000032294'} 2023-10-11T15:35:53.884 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032347', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037387', 3: '00000000000000000000:00000000000000032294'} 2023-10-11T15:35:53.885 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:35:53.885 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T15:35:53.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.886 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:53.935 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:53.982 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:35:53.983 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.983 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-10-11T15:35:53.983 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:33:34.867247Z" 2023-10-11T15:35:53.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.984 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005586", 2023-10-11T15:35:53.984 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:33:19.029468Z" 2023-10-11T15:35:53.984 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.985 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-10-11T15:35:53.985 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:32:50.864212Z" 2023-10-11T15:35:53.985 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:53.985 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:53.986 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000014181", 2023-10-11T15:35:53.986 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:33:48.624033Z" 2023-10-11T15:35:53.986 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:53.986 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:35:53.987 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000011394', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000014181'} 2023-10-11T15:35:53.988 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:35:53.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:53.989 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:54.041 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:54.041 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:54.084 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:35:54.084 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:35:54.084 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:35:54.084 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:35:54.085 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:35:54.085 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T15:35:54.085 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.085 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:35:54.086 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.086 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:35:54.086 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.086 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:54.086 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-10-11T15:35:54.087 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.087 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.087 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.087 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:33:34.867247Z" 2023-10-11T15:35:54.087 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.088 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.088 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.088 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:35:54.088 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.089 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:54.089 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005586", 2023-10-11T15:35:54.089 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.089 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.089 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.090 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:33:19.029468Z" 2023-10-11T15:35:54.090 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.090 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.090 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.090 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:35:54.091 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.091 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:54.091 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-10-11T15:35:54.091 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.091 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.092 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.092 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:32:50.864212Z" 2023-10-11T15:35:54.092 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.092 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.093 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.093 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:35:54.093 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.093 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:35:54.093 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000014181", 2023-10-11T15:35:54.094 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.094 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.094 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.094 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:33:48.624033Z" 2023-10-11T15:35:54.094 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.095 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.095 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:35:54.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.095 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:35:54.095 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T15:35:54.096 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T15:35:54.096 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.096 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:35:54Z" 2023-10-11T15:35:54.096 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:35:54.097 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011394", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:33:34.867247Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005586", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:33:19.029468Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:32:50.864212Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014181", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:33:48.624033Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T15:35:54Z" } 2023-10-11T15:35:54.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011394', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:33:34.867247Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005586', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:33:19.029468Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:32:50.864212Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014181', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:33:48.624033Z'}}] 2023-10-11T15:35:54.098 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011394', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000014181'} 2023-10-11T15:35:54.099 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011394', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015588', 3: '00000000000000000000:00000000000000014181'} 2023-10-11T15:35:54.099 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:35:54.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-10-11T15:35:54.101 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:35:54.102 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:35:54.102 DEBUG:boto:Method: PUT 2023-10-11T15:35:54.102 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:35:54.103 DEBUG:boto:Data: 2023-10-11T15:35:54.103 DEBUG:boto:Headers: {} 2023-10-11T15:35:54.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:54.104 DEBUG:boto:Port: 8000 2023-10-11T15:35:54.104 DEBUG:boto:Params: {} 2023-10-11T15:35:54.105 DEBUG:boto:Token: None 2023-10-11T15:35:54.105 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:35:54 GMT /dghqvd-62/ 2023-10-11T15:35:54.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rbD8p5DW4sPBD3E9+eg4Ohzwdo8= 2023-10-11T15:35:54.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rbD8p5DW4sPBD3E9+eg4Ohzwdo8=', 'Content-Length': '0'} 2023-10-11T15:35:54.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c2fb2c06e44887b-006526c0da-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:54.122 DEBUG:rgw_multi.tests:created bucket=dghqvd-62 2023-10-11T15:35:54.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:54.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:54.123 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:54.171 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:54.172 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:54.207 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:35:54.219 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:35:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:35:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:35:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:35:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:35:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:35:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:35:54.224 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.224 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.224 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:35:54.224 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:35:54.119201Z" 2023-10-11T15:35:54.225 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.225 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:35:54.228 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:35:54.229 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:35:54.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:54.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:35:54.232 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:35:54.287 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:54.287 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:54.315 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:54.313+0000 7fd4006a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:35:54.316 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:54.314+0000 7fd4006a5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:35:54.329 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:35:54.329 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:35:54.330 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:35:54.330 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:35:54.330 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:35:54.330 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:35:54.331 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:35:54.331 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.331 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:35:54.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.331 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:35:54.332 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.332 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:35:54.332 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:35:54.332 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.333 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:35:54.333 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.333 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:35:54.333 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:35:54.333 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.334 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.334 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.334 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:35:54.334 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.335 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:35:54.335 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:35:54.335 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.335 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.335 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.336 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:35:54.336 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:35:54.336 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.336 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.336 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.337 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:35:54.337 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.337 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:35:54.338 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:35:54.338 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.338 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.338 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.339 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:35:54.339 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:35:54.339 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.339 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.339 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:35:54.340 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:35:54.340 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:35:54.340 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:35:54.340 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:35:54.340 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:35:54.341 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:35:54.341 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:35:54.341 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:35:54.341 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:35:54.342 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.342 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:35:54.342 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:35:54.342 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.342 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:35:54.343 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:35:54.343 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:35:54.343 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:35:54.343 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:35:54Z" 2023-10-11T15:35:54.344 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:35:54.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:35:54Z" } 2023-10-11T15:35:54.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:35:54.346 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:35:54.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:35:54.346 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:35:54.347 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:35:54.347 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:35:54.348 DEBUG:boto:Method: HEAD 2023-10-11T15:35:54.348 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:35:54.349 DEBUG:boto:Data: 2023-10-11T15:35:54.349 DEBUG:boto:Headers: {} 2023-10-11T15:35:54.349 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:54.350 DEBUG:boto:Port: 8000 2023-10-11T15:35:54.350 DEBUG:boto:Params: {} 2023-10-11T15:35:54.351 DEBUG:boto:Token: None 2023-10-11T15:35:54.351 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:35:54 GMT /dghqvd-62/ 2023-10-11T15:35:54.351 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AwOeOTygcSpBwBlO4ekAGc9o0ac= 2023-10-11T15:35:54.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:35:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AwOeOTygcSpBwBlO4ekAGc9o0ac=', 'Content-Length': '0'} 2023-10-11T15:35:54.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002649caba54127255-006526c0da-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:54.355 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:35:54.356 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:35:54.356 DEBUG:boto:Method: PUT 2023-10-11T15:35:54.357 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:35:54.357 DEBUG:boto:Data: 2023-10-11T15:35:54.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:35:54.358 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:35:54.359 DEBUG:boto:Port: 8000 2023-10-11T15:35:54.359 DEBUG:boto:Params: {} 2023-10-11T15:35:54.360 DEBUG:boto:Token: None 2023-10-11T15:35:54.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:35:54 GMT /dghqvd-62/dummy 2023-10-11T15:35:54.360 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vcsexjeRD7+quAF2kqOFdyhs7i0= 2023-10-11T15:35:54.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:35:54 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vcsexjeRD7+quAF2kqOFdyhs7i0='} 2023-10-11T15:35:54.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e83faad5b6d66293-006526c0da-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:35:54.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-62 2023-10-11T15:35:54.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:54.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:54.369 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:35:54.428 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:54.428 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:35:54.463 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:54.461+0000 7ff1dec7f780 1 waiting to reach incremental sync.. 2023-10-11T15:35:59.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:59.463+0000 7ff1dec7f780 1 bucket sync caught up with source: 2023-10-11T15:35:59.465 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:35:59.466 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:35:59.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:35:59.463+0000 7ff1dec7f780 0 bucket checkpoint complete 2023-10-11T15:35:59.482 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-62 2023-10-11T15:35:59.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:59.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:35:59.484 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:35:59.546 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:35:59.546 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:36:00.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:00.584+0000 7ffbd8678780 1 waiting to reach incremental sync.. 2023-10-11T15:36:05.586 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:05.585+0000 7ffbd8678780 1 waiting to reach incremental sync.. 2023-10-11T15:36:10.587 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:10.586+0000 7ffbd8678780 1 waiting to reach incremental sync.. 2023-10-11T15:36:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:15.588+0000 7ffbd8678780 1 bucket sync caught up with source: 2023-10-11T15:36:15.589 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.237.6, , , , , ] 2023-10-11T15:36:15.590 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.237.6, , , , , ] 2023-10-11T15:36:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:15.588+0000 7ffbd8678780 0 bucket checkpoint complete 2023-10-11T15:36:15.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-62 zones={a2, a1} 2023-10-11T15:36:15.607 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.608 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.608 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.609 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:15.609 DEBUG:boto:Data: 2023-10-11T15:36:15.610 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.611 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.611 DEBUG:boto:Params: {} 2023-10-11T15:36:15.611 DEBUG:boto:Token: None 2023-10-11T15:36:15.612 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sQjs5nfojlUYtpomTrAGrDAK4So= 2023-10-11T15:36:15.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sQjs5nfojlUYtpomTrAGrDAK4So=', 'Content-Length': '0'} 2023-10-11T15:36:15.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7414edf6440fadc-006526c0ef-1f5c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.617 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.617 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.617 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.618 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:15.618 DEBUG:boto:Data: 2023-10-11T15:36:15.619 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.620 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.620 DEBUG:boto:Params: {} 2023-10-11T15:36:15.620 DEBUG:boto:Token: None 2023-10-11T15:36:15.621 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.621 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sQjs5nfojlUYtpomTrAGrDAK4So= 2023-10-11T15:36:15.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sQjs5nfojlUYtpomTrAGrDAK4So=', 'Content-Length': '0'} 2023-10-11T15:36:15.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c28a7bdd1980ba3-006526c0ef-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.626 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:36:15.627 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.628 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.629 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.629 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.630 DEBUG:boto:Method: GET 2023-10-11T15:36:15.631 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.631 DEBUG:boto:Data: 2023-10-11T15:36:15.632 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.633 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.634 DEBUG:boto:Params: {} 2023-10-11T15:36:15.634 DEBUG:boto:Token: None 2023-10-11T15:36:15.636 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/?versions 2023-10-11T15:36:15.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8= 2023-10-11T15:36:15.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8=', 'Content-Length': '0'} 2023-10-11T15:36:15.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053a6df05125d62d8-006526c0ef-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.640 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.641 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T15:36:15.641 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:36:15.642 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.642 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.642 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.643 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.643 DEBUG:boto:Method: GET 2023-10-11T15:36:15.644 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.644 DEBUG:boto:Data: 2023-10-11T15:36:15.644 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.645 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.646 DEBUG:boto:Params: {} 2023-10-11T15:36:15.646 DEBUG:boto:Token: None 2023-10-11T15:36:15.647 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/?versions 2023-10-11T15:36:15.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8= 2023-10-11T15:36:15.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8=', 'Content-Length': '0'} 2023-10-11T15:36:15.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009943db2c06c74778-006526c0ef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.651 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.651 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T15:36:15.652 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.652 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.653 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.653 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.653 DEBUG:boto:Method: GET 2023-10-11T15:36:15.654 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.654 DEBUG:boto:Data: 2023-10-11T15:36:15.655 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.655 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.656 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.656 DEBUG:boto:Params: {} 2023-10-11T15:36:15.656 DEBUG:boto:Token: None 2023-10-11T15:36:15.657 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/?versions 2023-10-11T15:36:15.657 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8= 2023-10-11T15:36:15.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8=', 'Content-Length': '0'} 2023-10-11T15:36:15.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075109989555f7e91-006526c0ef-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.702 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.703 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.704 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.704 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.705 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.705 DEBUG:boto:Method: GET 2023-10-11T15:36:15.706 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:36:15.706 DEBUG:boto:Data: 2023-10-11T15:36:15.706 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.708 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.708 DEBUG:boto:Params: {} 2023-10-11T15:36:15.709 DEBUG:boto:Token: None 2023-10-11T15:36:15.709 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/?versions 2023-10-11T15:36:15.710 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8= 2023-10-11T15:36:15.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B2lGgvbkpo1D0Md6ox9OM8L0WQ8=', 'Content-Length': '0'} 2023-10-11T15:36:15.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000974d9eb8f4aaf337-006526c0ef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.713 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:36:15.715 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.715 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.715 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.716 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.716 DEBUG:boto:Method: GET 2023-10-11T15:36:15.717 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.717 DEBUG:boto:Data: 2023-10-11T15:36:15.718 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.718 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.719 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.719 DEBUG:boto:Params: {} 2023-10-11T15:36:15.719 DEBUG:boto:Token: None 2023-10-11T15:36:15.720 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.720 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc= 2023-10-11T15:36:15.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc=', 'Content-Length': '0'} 2023-10-11T15:36:15.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24a5115f0fc245e-006526c0ef-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.724 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.724 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.725 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.725 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.726 DEBUG:boto:Method: GET 2023-10-11T15:36:15.726 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.726 DEBUG:boto:Data: 2023-10-11T15:36:15.727 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.728 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.728 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.728 DEBUG:boto:Params: {} 2023-10-11T15:36:15.729 DEBUG:boto:Token: None 2023-10-11T15:36:15.730 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.730 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc= 2023-10-11T15:36:15.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc=', 'Content-Length': '0'} 2023-10-11T15:36:15.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f239360ecb3717ee-006526c0ef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.734 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.734 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.735 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.735 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.735 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.736 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.736 DEBUG:boto:Data: 2023-10-11T15:36:15.737 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.737 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.738 DEBUG:boto:Params: {} 2023-10-11T15:36:15.738 DEBUG:boto:Token: None 2023-10-11T15:36:15.739 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:STUFH1suT/dnYwcu+ZO3HiojPeE= 2023-10-11T15:36:15.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:STUFH1suT/dnYwcu+ZO3HiojPeE=', 'Content-Length': '0'} 2023-10-11T15:36:15.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2de6948842d4d1e-006526c0ef-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.742 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.743 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.743 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.744 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.744 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.745 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.745 DEBUG:boto:Data: 2023-10-11T15:36:15.746 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.746 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.746 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.747 DEBUG:boto:Params: {} 2023-10-11T15:36:15.747 DEBUG:boto:Token: None 2023-10-11T15:36:15.748 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.748 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:STUFH1suT/dnYwcu+ZO3HiojPeE= 2023-10-11T15:36:15.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:STUFH1suT/dnYwcu+ZO3HiojPeE=', 'Content-Length': '0'} 2023-10-11T15:36:15.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba1b1cc9abc6bf9-006526c0ef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:36:15.752 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.752 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.753 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.753 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.754 DEBUG:boto:Method: GET 2023-10-11T15:36:15.754 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.755 DEBUG:boto:Data: 2023-10-11T15:36:15.755 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.756 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.756 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.756 DEBUG:boto:Params: {} 2023-10-11T15:36:15.757 DEBUG:boto:Token: None 2023-10-11T15:36:15.757 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.758 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc= 2023-10-11T15:36:15.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc=', 'Content-Length': '0'} 2023-10-11T15:36:15.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74c85492456f38c-006526c0ef-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.761 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.762 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.762 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.763 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.763 DEBUG:boto:Method: GET 2023-10-11T15:36:15.763 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.764 DEBUG:boto:Data: 2023-10-11T15:36:15.764 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.765 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.765 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.766 DEBUG:boto:Params: {} 2023-10-11T15:36:15.766 DEBUG:boto:Token: None 2023-10-11T15:36:15.767 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:36:15.767 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc= 2023-10-11T15:36:15.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T/XjswVueafj4vSIWY3GjC/PGZc=', 'Content-Length': '0'} 2023-10-11T15:36:15.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d5abb31aa0846d-006526c0ef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.771 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.771 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.771 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.772 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.772 DEBUG:boto:Method: GET 2023-10-11T15:36:15.773 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.773 DEBUG:boto:Data: 2023-10-11T15:36:15.774 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.775 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.775 DEBUG:boto:Params: {} 2023-10-11T15:36:15.776 DEBUG:boto:Token: None 2023-10-11T15:36:15.776 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.777 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak= 2023-10-11T15:36:15.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak=', 'Content-Length': '0'} 2023-10-11T15:36:15.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8b4be0ae182616c-006526c0ef-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.780 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.781 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.781 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.782 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.782 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.783 DEBUG:boto:Method: GET 2023-10-11T15:36:15.783 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.783 DEBUG:boto:Data: 2023-10-11T15:36:15.784 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.785 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.785 DEBUG:boto:Params: {} 2023-10-11T15:36:15.786 DEBUG:boto:Token: None 2023-10-11T15:36:15.786 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.787 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak= 2023-10-11T15:36:15.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak=', 'Content-Length': '0'} 2023-10-11T15:36:15.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584575c78f5a834e-006526c0ef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.790 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.791 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.791 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.792 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.792 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:36:15.793 DEBUG:boto:Data: 2023-10-11T15:36:15.793 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.794 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.794 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.794 DEBUG:boto:Params: {} 2023-10-11T15:36:15.795 DEBUG:boto:Token: None 2023-10-11T15:36:15.795 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy 2023-10-11T15:36:15.796 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CDpu2JlDH/VM2jxJf22QR0fVd2w= 2023-10-11T15:36:15.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CDpu2JlDH/VM2jxJf22QR0fVd2w=', 'Content-Length': '0'} 2023-10-11T15:36:15.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c57c46d7b0ba2e-006526c0ef-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.799 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.799 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.800 DEBUG:boto:Method: HEAD 2023-10-11T15:36:15.800 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:36:15.800 DEBUG:boto:Data: 2023-10-11T15:36:15.801 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.802 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.802 DEBUG:boto:Params: {} 2023-10-11T15:36:15.803 DEBUG:boto:Token: None 2023-10-11T15:36:15.803 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy 2023-10-11T15:36:15.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CDpu2JlDH/VM2jxJf22QR0fVd2w= 2023-10-11T15:36:15.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CDpu2JlDH/VM2jxJf22QR0fVd2w=', 'Content-Length': '0'} 2023-10-11T15:36:15.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebde1c80d6e453c-006526c0ef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:36:15.807 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.807 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.808 DEBUG:boto:Method: GET 2023-10-11T15:36:15.808 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:36:15.809 DEBUG:boto:Data: 2023-10-11T15:36:15.809 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.809 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.810 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.810 DEBUG:boto:Params: {} 2023-10-11T15:36:15.811 DEBUG:boto:Token: None 2023-10-11T15:36:15.811 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy 2023-10-11T15:36:15.812 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AWMO4TbrF67cHAL1UR6VgUPyeMI= 2023-10-11T15:36:15.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AWMO4TbrF67cHAL1UR6VgUPyeMI=', 'Content-Length': '0'} 2023-10-11T15:36:15.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a236eeb70015181-006526c0ef-1f5c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.858 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:36:15.859 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:36:15.860 DEBUG:boto:Method: GET 2023-10-11T15:36:15.860 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:36:15.860 DEBUG:boto:Data: 2023-10-11T15:36:15.861 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.861 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.862 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.862 DEBUG:boto:Params: {} 2023-10-11T15:36:15.863 DEBUG:boto:Token: None 2023-10-11T15:36:15.863 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/dummy 2023-10-11T15:36:15.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AWMO4TbrF67cHAL1UR6VgUPyeMI= 2023-10-11T15:36:15.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AWMO4TbrF67cHAL1UR6VgUPyeMI=', 'Content-Length': '0'} 2023-10-11T15:36:15.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004edd49ad88ac6c24-006526c0ef-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.867 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.867 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.868 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.868 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.869 DEBUG:boto:Method: GET 2023-10-11T15:36:15.869 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.870 DEBUG:boto:Data: 2023-10-11T15:36:15.870 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.870 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:15.871 DEBUG:boto:Port: 8001 2023-10-11T15:36:15.871 DEBUG:boto:Params: {} 2023-10-11T15:36:15.872 DEBUG:boto:Token: None 2023-10-11T15:36:15.872 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.873 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak= 2023-10-11T15:36:15.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak=', 'Content-Length': '0'} 2023-10-11T15:36:15.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045933408f02ef390-006526c0ef-1f5c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.876 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.877 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:15.878 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:15.878 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.879 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.879 DEBUG:boto:Method: GET 2023-10-11T15:36:15.879 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:36:15.880 DEBUG:boto:Data: 2023-10-11T15:36:15.880 DEBUG:boto:Headers: {} 2023-10-11T15:36:15.881 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:15.881 DEBUG:boto:Port: 8000 2023-10-11T15:36:15.882 DEBUG:boto:Params: {} 2023-10-11T15:36:15.882 DEBUG:boto:Token: None 2023-10-11T15:36:15.883 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:36:15 GMT /dghqvd-62/ 2023-10-11T15:36:15.883 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak= 2023-10-11T15:36:15.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:15 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5lLT6RVO6BNHFalLYvDEq2jcvak=', 'Content-Length': '0'} 2023-10-11T15:36:15.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb3ba37583fa247-006526c0ef-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:36:15 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:15.930 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:36:15.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-62 zones={a2, a1} 2023-10-11T15:36:15.932 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:36:15.932 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:36:15.933 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:36:15.932+0000 7ff0f60b8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 175722) UID: 0 2023-10-11T15:36:15.934 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:36:15.932+0000 7ff0fa3217c0 -1 shutting down 2023-10-11T15:36:19.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:36:22.031 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:36:22.032 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:22.032 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:22.033 DEBUG:boto:Method: HEAD 2023-10-11T15:36:22.033 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:22.033 DEBUG:boto:Data: 2023-10-11T15:36:22.034 DEBUG:boto:Headers: {} 2023-10-11T15:36:22.034 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.035 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.035 DEBUG:boto:Params: {} 2023-10-11T15:36:22.035 DEBUG:boto:Token: None 2023-10-11T15:36:22.036 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/ 2023-10-11T15:36:22.036 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI= 2023-10-11T15:36:22.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI=', 'Content-Length': '0'} 2023-10-11T15:36:22.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000156d341313371e1e-006526c0f6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.040 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:36:22.041 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:36:22.041 DEBUG:boto:Method: PUT 2023-10-11T15:36:22.041 DEBUG:boto:Path: /dghqvd-62/obj11 2023-10-11T15:36:22.042 DEBUG:boto:Data: 2023-10-11T15:36:22.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:36:22.043 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.043 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.043 DEBUG:boto:Params: {} 2023-10-11T15:36:22.044 DEBUG:boto:Token: None 2023-10-11T15:36:22.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/obj11 2023-10-11T15:36:22.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1N367VdueF1ExvdVSYrW/gaRWUw= 2023-10-11T15:36:22.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1N367VdueF1ExvdVSYrW/gaRWUw='} 2023-10-11T15:36:22.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012a3352f676502ff-006526c0f6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.053 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:22.053 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:22.054 DEBUG:boto:Method: HEAD 2023-10-11T15:36:22.054 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:22.055 DEBUG:boto:Data: 2023-10-11T15:36:22.055 DEBUG:boto:Headers: {} 2023-10-11T15:36:22.055 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.056 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.056 DEBUG:boto:Params: {} 2023-10-11T15:36:22.057 DEBUG:boto:Token: None 2023-10-11T15:36:22.057 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/ 2023-10-11T15:36:22.058 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI= 2023-10-11T15:36:22.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI=', 'Content-Length': '0'} 2023-10-11T15:36:22.060 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000504a07669730f014-006526c0f6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.061 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:36:22.061 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:36:22.062 DEBUG:boto:Method: PUT 2023-10-11T15:36:22.062 DEBUG:boto:Path: /dghqvd-62/obj22 2023-10-11T15:36:22.063 DEBUG:boto:Data: 2023-10-11T15:36:22.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:36:22.063 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.064 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.064 DEBUG:boto:Params: {} 2023-10-11T15:36:22.065 DEBUG:boto:Token: None 2023-10-11T15:36:22.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/obj22 2023-10-11T15:36:22.066 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lxuY/C234yyARgno9D27E307nCA= 2023-10-11T15:36:22.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lxuY/C234yyARgno9D27E307nCA='} 2023-10-11T15:36:22.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044c91f43cd23e129-006526c0f6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.072 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:22.072 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:22.072 DEBUG:boto:Method: HEAD 2023-10-11T15:36:22.073 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:22.073 DEBUG:boto:Data: 2023-10-11T15:36:22.074 DEBUG:boto:Headers: {} 2023-10-11T15:36:22.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.074 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.075 DEBUG:boto:Params: {} 2023-10-11T15:36:22.075 DEBUG:boto:Token: None 2023-10-11T15:36:22.076 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/ 2023-10-11T15:36:22.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI= 2023-10-11T15:36:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI=', 'Content-Length': '0'} 2023-10-11T15:36:22.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038dc1967109fca64-006526c0f6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.080 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:36:22.080 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:36:22.081 DEBUG:boto:Method: PUT 2023-10-11T15:36:22.081 DEBUG:boto:Path: /dghqvd-62/obj33 2023-10-11T15:36:22.082 DEBUG:boto:Data: 2023-10-11T15:36:22.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:36:22.083 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.083 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.083 DEBUG:boto:Params: {} 2023-10-11T15:36:22.084 DEBUG:boto:Token: None 2023-10-11T15:36:22.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/obj33 2023-10-11T15:36:22.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZaEAXUgGXUyuZqRTmaOqM+I6Hm0= 2023-10-11T15:36:22.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZaEAXUgGXUyuZqRTmaOqM+I6Hm0='} 2023-10-11T15:36:22.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7c0fb8a5e8465cd-006526c0f6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.131 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:22.132 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:22.132 DEBUG:boto:Method: HEAD 2023-10-11T15:36:22.132 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:22.133 DEBUG:boto:Data: 2023-10-11T15:36:22.133 DEBUG:boto:Headers: {} 2023-10-11T15:36:22.133 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.134 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.134 DEBUG:boto:Params: {} 2023-10-11T15:36:22.135 DEBUG:boto:Token: None 2023-10-11T15:36:22.135 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/ 2023-10-11T15:36:22.136 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI= 2023-10-11T15:36:22.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI=', 'Content-Length': '0'} 2023-10-11T15:36:22.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076121205e0740916-006526c0f6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.139 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:36:22.140 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:36:22.140 DEBUG:boto:Method: PUT 2023-10-11T15:36:22.141 DEBUG:boto:Path: /dghqvd-62/obj44 2023-10-11T15:36:22.141 DEBUG:boto:Data: 2023-10-11T15:36:22.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:36:22.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.142 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.143 DEBUG:boto:Params: {} 2023-10-11T15:36:22.143 DEBUG:boto:Token: None 2023-10-11T15:36:22.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/obj44 2023-10-11T15:36:22.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y4PMsvzwJBgLXWrX1A7T5+Oo43M= 2023-10-11T15:36:22.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y4PMsvzwJBgLXWrX1A7T5+Oo43M='} 2023-10-11T15:36:22.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000480fafa9db6dd401-006526c0f6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.152 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:36:22.152 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:36:22.153 DEBUG:boto:Method: HEAD 2023-10-11T15:36:22.153 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:36:22.153 DEBUG:boto:Data: 2023-10-11T15:36:22.154 DEBUG:boto:Headers: {} 2023-10-11T15:36:22.154 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.155 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.155 DEBUG:boto:Params: {} 2023-10-11T15:36:22.156 DEBUG:boto:Token: None 2023-10-11T15:36:22.156 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/ 2023-10-11T15:36:22.156 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI= 2023-10-11T15:36:22.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:V4JqqCe4rkYkZJ9HUGgf+1YoLwI=', 'Content-Length': '0'} 2023-10-11T15:36:22.159 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c17777b80ee10d8-006526c0f6-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.160 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:36:22.161 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:36:22.161 DEBUG:boto:Method: PUT 2023-10-11T15:36:22.161 DEBUG:boto:Path: /dghqvd-62/obj55 2023-10-11T15:36:22.162 DEBUG:boto:Data: 2023-10-11T15:36:22.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:36:22.163 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:36:22.163 DEBUG:boto:Port: 8000 2023-10-11T15:36:22.163 DEBUG:boto:Params: {} 2023-10-11T15:36:22.164 DEBUG:boto:Token: None 2023-10-11T15:36:22.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:36:22 GMT /dghqvd-62/obj55 2023-10-11T15:36:22.165 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gTR3jo8zxou5M84VOO7GHUId+VA= 2023-10-11T15:36:22.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:36:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gTR3jo8zxou5M84VOO7GHUId+VA='} 2023-10-11T15:36:22.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a5a75c0886839e2-006526c0f6-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:36:22.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:36:22.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:36:22.216 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:36:22.274 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:36:22.275 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:36:22.311 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:22.309+0000 7f86bea18780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-11T15:36:22.312 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:22.311+0000 7f86bea18780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-11T15:36:22.335 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:36:22.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:36:22.338 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:36:22.338 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:22.359 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:36:22.359 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:36:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:36:22.362 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:36:23.363 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:36:23.386 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:36:23.386 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:36:23.386 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:36:23.387 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:36:23.388 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-62 2023-10-11T15:36:23.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:36:23.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:36:23.389 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:36:23.483 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:36:23.484 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:36:23.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:23.519+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:23.520 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:23.521 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:25.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:36:28.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:28.520+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:28.522 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:28.522 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:33.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:33.521+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:33.523 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:33.523 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:38.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:38.522+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:38.524 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:38.524 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:43.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:43.524+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:43.525 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:43.526 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:48.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:48.525+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:48.527 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:48.527 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:53.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:53.526+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:53.528 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:53.528 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:36:58.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:36:58.527+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:36:58.529 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:36:58.530 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:03.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:03.528+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:03.531 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:37:03.531 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:08.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:08.529+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:08.532 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:37:08.532 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:13.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:13.533+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:13.536 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:37:13.536 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:18.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:18.534+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:18.537 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:37:18.537 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:23.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:23.535+0000 7fb1fffb7780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:23.538 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , , , ] 2023-10-11T15:37:23.538 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:28.539 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:28.537+0000 7fb1fffb7780 1 bucket sync caught up with source: 2023-10-11T15:37:28.539 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:28.540 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.97.6, , , 00000000001.71.6, 00000000002.118.6, 00000000002.76.6] 2023-10-11T15:37:28.540 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:28.537+0000 7fb1fffb7780 0 bucket checkpoint complete 2023-10-11T15:37:28.557 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-62 2023-10-11T15:37:28.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:37:28.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:37:28.559 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:37:28.622 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:28.622 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:28.659 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:28.657+0000 7efe8e0ff780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:28.659 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.237.6, , , , , ] 2023-10-11T15:37:28.660 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.237.6, , , 00000000001.58.6, 00000000002.303.6, 00000000002.173.6] 2023-10-11T15:37:33.660 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:33.658+0000 7efe8e0ff780 1 waiting for incremental sync to catch up: 2023-10-11T15:37:33.661 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.237.6, , , , , ] 2023-10-11T15:37:33.661 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.237.6, , , 00000000001.58.6, 00000000002.303.6, 00000000002.173.6] 2023-10-11T15:37:38.662 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:38.659+0000 7efe8e0ff780 1 bucket sync caught up with source: 2023-10-11T15:37:38.662 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , 00000000001.237.6, , , 00000000001.58.6, 00000000002.303.6, 00000000002.173.6] 2023-10-11T15:37:38.662 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , 00000000001.237.6, , , 00000000001.58.6, 00000000002.303.6, 00000000002.173.6] 2023-10-11T15:37:38.663 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:37:38.659+0000 7efe8e0ff780 0 bucket checkpoint complete 2023-10-11T15:37:38.679 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-62 zones={a2, a1} 2023-10-11T15:37:38.680 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.681 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.681 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.682 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:37:38.682 DEBUG:boto:Data: 2023-10-11T15:37:38.683 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.683 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.683 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.684 DEBUG:boto:Params: {} 2023-10-11T15:37:38.685 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:37:38.685 DEBUG:boto:Token: None 2023-10-11T15:37:38.686 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.686 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Gp+ojn1AoY3UR5iCsHTOyOMKjU= 2023-10-11T15:37:38.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Gp+ojn1AoY3UR5iCsHTOyOMKjU=', 'Content-Length': '0'} 2023-10-11T15:37:38.693 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db07e291593b63b7-006526c142-1fbc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.693 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.694 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.694 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.695 DEBUG:boto:Path: /dghqvd-62/ 2023-10-11T15:37:38.695 DEBUG:boto:Data: 2023-10-11T15:37:38.696 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.696 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.697 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.697 DEBUG:boto:Params: {} 2023-10-11T15:37:38.698 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:37:38.698 DEBUG:boto:Token: None 2023-10-11T15:37:38.699 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.699 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9Gp+ojn1AoY3UR5iCsHTOyOMKjU= 2023-10-11T15:37:38.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9Gp+ojn1AoY3UR5iCsHTOyOMKjU=', 'Content-Length': '0'} 2023-10-11T15:37:38.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a43922d0b569ac7-006526c142-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:37:38.705 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.705 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.706 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.706 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.709 DEBUG:boto:Method: GET 2023-10-11T15:37:38.709 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.710 DEBUG:boto:Data: 2023-10-11T15:37:38.710 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.710 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.711 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.711 DEBUG:boto:Params: {} 2023-10-11T15:37:38.712 DEBUG:boto:Token: None 2023-10-11T15:37:38.712 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/?versions 2023-10-11T15:37:38.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA= 2023-10-11T15:37:38.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA=', 'Content-Length': '0'} 2023-10-11T15:37:38.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035469f5909f554ab-006526c142-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.717 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.718 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T15:37:38.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:37:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:37:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:37:38.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:37:38.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:37:38.720 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:37:38.721 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.721 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.721 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.722 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.722 DEBUG:boto:Method: GET 2023-10-11T15:37:38.723 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.723 DEBUG:boto:Data: 2023-10-11T15:37:38.723 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.724 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.724 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.725 DEBUG:boto:Params: {} 2023-10-11T15:37:38.725 DEBUG:boto:Token: None 2023-10-11T15:37:38.726 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/?versions 2023-10-11T15:37:38.726 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA= 2023-10-11T15:37:38.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA=', 'Content-Length': '0'} 2023-10-11T15:37:38.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000324f1ef907b915e0-006526c142-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.730 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-10-11T15:37:38.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:37:38.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:37:38.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:37:38.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:37:38.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:37:38.733 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.734 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.734 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.735 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.735 DEBUG:boto:Method: GET 2023-10-11T15:37:38.736 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.736 DEBUG:boto:Data: 2023-10-11T15:37:38.736 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.737 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.737 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.738 DEBUG:boto:Params: {} 2023-10-11T15:37:38.738 DEBUG:boto:Token: None 2023-10-11T15:37:38.739 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/?versions 2023-10-11T15:37:38.739 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA= 2023-10-11T15:37:38.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA=', 'Content-Length': '0'} 2023-10-11T15:37:38.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e19a6b333cad13-006526c142-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.743 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.744 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.744 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.744 DEBUG:boto:path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.745 DEBUG:boto:auth_path=/dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.745 DEBUG:boto:Method: GET 2023-10-11T15:37:38.746 DEBUG:boto:Path: /dghqvd-62/?versions&max-keys=999 2023-10-11T15:37:38.746 DEBUG:boto:Data: 2023-10-11T15:37:38.747 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.747 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.748 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.748 DEBUG:boto:Params: {} 2023-10-11T15:37:38.748 DEBUG:boto:Token: None 2023-10-11T15:37:38.749 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/?versions 2023-10-11T15:37:38.749 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA= 2023-10-11T15:37:38.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PUveOv216R2UAazoB5+KzhyQ/oA=', 'Content-Length': '0'} 2023-10-11T15:37:38.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd44bb7c21e7207-006526c142-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.753 DEBUG:boto:b'dghqvd-62999falsedummynulltrue2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:37:38.754 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.755 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.755 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.756 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.756 DEBUG:boto:Method: GET 2023-10-11T15:37:38.757 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.757 DEBUG:boto:Data: 2023-10-11T15:37:38.757 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.758 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.759 DEBUG:boto:Params: {} 2023-10-11T15:37:38.759 DEBUG:boto:Token: None 2023-10-11T15:37:38.760 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY= 2023-10-11T15:37:38.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY=', 'Content-Length': '0'} 2023-10-11T15:37:38.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f6ea98716e319d-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.765 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.765 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.765 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.766 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.766 DEBUG:boto:Method: GET 2023-10-11T15:37:38.767 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.767 DEBUG:boto:Data: 2023-10-11T15:37:38.768 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.769 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.769 DEBUG:boto:Params: {} 2023-10-11T15:37:38.770 DEBUG:boto:Token: None 2023-10-11T15:37:38.770 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY= 2023-10-11T15:37:38.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY=', 'Content-Length': '0'} 2023-10-11T15:37:38.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080faef11b0e15f66-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.774 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.774 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.774 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.775 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.775 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.776 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.776 DEBUG:boto:Data: 2023-10-11T15:37:38.776 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.777 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.777 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.778 DEBUG:boto:Params: {} 2023-10-11T15:37:38.778 DEBUG:boto:Token: None 2023-10-11T15:37:38.778 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZbgUHjikg5Xmwv2jlUHMDjb21tE= 2023-10-11T15:37:38.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZbgUHjikg5Xmwv2jlUHMDjb21tE=', 'Content-Length': '0'} 2023-10-11T15:37:38.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080acf6cbb44bc8da-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.782 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.782 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.783 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.783 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.784 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.784 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.785 DEBUG:boto:Data: 2023-10-11T15:37:38.785 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.786 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.786 DEBUG:boto:Params: {} 2023-10-11T15:37:38.787 DEBUG:boto:Token: None 2023-10-11T15:37:38.787 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.788 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZbgUHjikg5Xmwv2jlUHMDjb21tE= 2023-10-11T15:37:38.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZbgUHjikg5Xmwv2jlUHMDjb21tE=', 'Content-Length': '0'} 2023-10-11T15:37:38.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d632fde102aaf23d-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:37:38.791 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.792 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.792 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.793 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.793 DEBUG:boto:Method: GET 2023-10-11T15:37:38.794 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.794 DEBUG:boto:Data: 2023-10-11T15:37:38.795 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.795 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.796 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.796 DEBUG:boto:Params: {} 2023-10-11T15:37:38.797 DEBUG:boto:Token: None 2023-10-11T15:37:38.797 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY= 2023-10-11T15:37:38.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY=', 'Content-Length': '0'} 2023-10-11T15:37:38.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007943517580267117-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.843 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.843 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.844 DEBUG:boto:path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.844 DEBUG:boto:auth_path=/dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.844 DEBUG:boto:Method: GET 2023-10-11T15:37:38.845 DEBUG:boto:Path: /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.845 DEBUG:boto:Data: 2023-10-11T15:37:38.846 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.846 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.847 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.847 DEBUG:boto:Params: {} 2023-10-11T15:37:38.848 DEBUG:boto:Token: None 2023-10-11T15:37:38.848 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy?versionId=null 2023-10-11T15:37:38.849 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY= 2023-10-11T15:37:38.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d2OgLNDL5ruwl0hDWsOXBBpfguY=', 'Content-Length': '0'} 2023-10-11T15:37:38.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb2bc512c0aa1c5-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.852 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.853 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.853 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.854 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.854 DEBUG:boto:Method: GET 2023-10-11T15:37:38.854 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.855 DEBUG:boto:Data: 2023-10-11T15:37:38.855 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.856 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.856 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.857 DEBUG:boto:Params: {} 2023-10-11T15:37:38.858 DEBUG:boto:Token: None 2023-10-11T15:37:38.858 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.859 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k= 2023-10-11T15:37:38.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k=', 'Content-Length': '0'} 2023-10-11T15:37:38.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005663b43f7e0ff810-006526c142-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.862 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.863 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.864 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.864 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.865 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.865 DEBUG:boto:Method: GET 2023-10-11T15:37:38.866 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.866 DEBUG:boto:Data: 2023-10-11T15:37:38.867 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.867 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.867 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.868 DEBUG:boto:Params: {} 2023-10-11T15:37:38.868 DEBUG:boto:Token: None 2023-10-11T15:37:38.869 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.869 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k= 2023-10-11T15:37:38.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k=', 'Content-Length': '0'} 2023-10-11T15:37:38.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483291d53f3f7738-006526c142-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.873 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.874 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.874 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.875 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.875 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:37:38.876 DEBUG:boto:Data: 2023-10-11T15:37:38.876 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.877 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.877 DEBUG:boto:Params: {} 2023-10-11T15:37:38.878 DEBUG:boto:Token: None 2023-10-11T15:37:38.878 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy 2023-10-11T15:37:38.879 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8c64Li/RNhJzqjikVvhqOH9hLQ= 2023-10-11T15:37:38.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8c64Li/RNhJzqjikVvhqOH9hLQ=', 'Content-Length': '0'} 2023-10-11T15:37:38.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0592cc93bdd240-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.882 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.883 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.883 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.884 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:37:38.884 DEBUG:boto:Data: 2023-10-11T15:37:38.884 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.885 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.885 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.886 DEBUG:boto:Params: {} 2023-10-11T15:37:38.886 DEBUG:boto:Token: None 2023-10-11T15:37:38.886 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy 2023-10-11T15:37:38.887 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U8c64Li/RNhJzqjikVvhqOH9hLQ= 2023-10-11T15:37:38.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U8c64Li/RNhJzqjikVvhqOH9hLQ=', 'Content-Length': '0'} 2023-10-11T15:37:38.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8108f6c2e4b704-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-10-11T15:37:38.890 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.891 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.891 DEBUG:boto:Method: GET 2023-10-11T15:37:38.892 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:37:38.892 DEBUG:boto:Data: 2023-10-11T15:37:38.893 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.893 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.894 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.894 DEBUG:boto:Params: {} 2023-10-11T15:37:38.895 DEBUG:boto:Token: None 2023-10-11T15:37:38.895 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy 2023-10-11T15:37:38.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VEFpQdjDBITtmiGcpvxHJl2gfhA= 2023-10-11T15:37:38.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VEFpQdjDBITtmiGcpvxHJl2gfhA=', 'Content-Length': '0'} 2023-10-11T15:37:38.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2e3ed62f880730-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.899 DEBUG:boto:path=/dghqvd-62/dummy 2023-10-11T15:37:38.899 DEBUG:boto:auth_path=/dghqvd-62/dummy 2023-10-11T15:37:38.900 DEBUG:boto:Method: GET 2023-10-11T15:37:38.900 DEBUG:boto:Path: /dghqvd-62/dummy 2023-10-11T15:37:38.901 DEBUG:boto:Data: 2023-10-11T15:37:38.901 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.902 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.902 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.903 DEBUG:boto:Params: {} 2023-10-11T15:37:38.903 DEBUG:boto:Token: None 2023-10-11T15:37:38.904 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/dummy 2023-10-11T15:37:38.904 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VEFpQdjDBITtmiGcpvxHJl2gfhA= 2023-10-11T15:37:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VEFpQdjDBITtmiGcpvxHJl2gfhA=', 'Content-Length': '0'} 2023-10-11T15:37:38.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:35:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3911fed0b4d3d11-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.907 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.908 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.908 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.908 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.909 DEBUG:boto:Method: GET 2023-10-11T15:37:38.909 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.910 DEBUG:boto:Data: 2023-10-11T15:37:38.910 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.911 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.911 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.911 DEBUG:boto:Params: {} 2023-10-11T15:37:38.912 DEBUG:boto:Token: None 2023-10-11T15:37:38.912 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k= 2023-10-11T15:37:38.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k=', 'Content-Length': '0'} 2023-10-11T15:37:38.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5bee8291a1c4b9-006526c142-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.916 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.917 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:38.918 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:38.918 DEBUG:boto:path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.918 DEBUG:boto:auth_path=/dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.919 DEBUG:boto:Method: GET 2023-10-11T15:37:38.919 DEBUG:boto:Path: /dghqvd-62/?prefix=dummy 2023-10-11T15:37:38.920 DEBUG:boto:Data: 2023-10-11T15:37:38.920 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.921 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.921 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.921 DEBUG:boto:Params: {} 2023-10-11T15:37:38.922 DEBUG:boto:Token: None 2023-10-11T15:37:38.922 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/ 2023-10-11T15:37:38.923 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k= 2023-10-11T15:37:38.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QzKcgoic2tSwdq5BieWOCWj1E6k=', 'Content-Length': '0'} 2023-10-11T15:37:38.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46d59485c4f52d7-006526c142-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.927 DEBUG:boto:b'dghqvd-62dummy1000falsedummy2023-10-11T15:35:54.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:38.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:37:38.928 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:38.928 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:38.929 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.929 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.929 DEBUG:boto:Method: GET 2023-10-11T15:37:38.930 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.930 DEBUG:boto:Data: 2023-10-11T15:37:38.931 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.931 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.932 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.932 DEBUG:boto:Params: {} 2023-10-11T15:37:38.933 DEBUG:boto:Token: None 2023-10-11T15:37:38.933 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.934 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q0EqIKLBXOcIQSqrYMTEU7dJ+hg= 2023-10-11T15:37:38.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q0EqIKLBXOcIQSqrYMTEU7dJ+hg=', 'Content-Length': '0'} 2023-10-11T15:37:38.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006183353fd42e8f1e-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.982 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:38.983 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:38.984 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.984 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.985 DEBUG:boto:Method: GET 2023-10-11T15:37:38.985 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.986 DEBUG:boto:Data: 2023-10-11T15:37:38.986 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.987 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:38.987 DEBUG:boto:Port: 8000 2023-10-11T15:37:38.988 DEBUG:boto:Params: {} 2023-10-11T15:37:38.988 DEBUG:boto:Token: None 2023-10-11T15:37:38.989 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.989 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q0EqIKLBXOcIQSqrYMTEU7dJ+hg= 2023-10-11T15:37:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q0EqIKLBXOcIQSqrYMTEU7dJ+hg=', 'Content-Length': '0'} 2023-10-11T15:37:38.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003496909c3f869bfa-006526c142-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:38 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:38.993 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:38.993 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:38.994 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.994 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.994 DEBUG:boto:Method: HEAD 2023-10-11T15:37:38.995 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.995 DEBUG:boto:Data: 2023-10-11T15:37:38.996 DEBUG:boto:Headers: {} 2023-10-11T15:37:38.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:38.996 DEBUG:boto:Port: 8001 2023-10-11T15:37:38.997 DEBUG:boto:Params: {} 2023-10-11T15:37:38.997 DEBUG:boto:Token: None 2023-10-11T15:37:38.998 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:38 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:38.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mF+wBDkLlbRXuZ0B+Br9PHOSyko= 2023-10-11T15:37:38.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:38 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mF+wBDkLlbRXuZ0B+Br9PHOSyko=', 'Content-Length': '0'} 2023-10-11T15:37:39.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dad47e5ffe343a3-006526c142-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.002 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.002 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.003 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.003 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.003 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.004 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.004 DEBUG:boto:Data: 2023-10-11T15:37:39.005 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.006 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.006 DEBUG:boto:Params: {} 2023-10-11T15:37:39.007 DEBUG:boto:Token: None 2023-10-11T15:37:39.007 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.009 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wCGqev0savgaunos02/bKBpnMPs= 2023-10-11T15:37:39.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wCGqev0savgaunos02/bKBpnMPs=', 'Content-Length': '0'} 2023-10-11T15:37:39.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5127b0e0c55b79-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:37:39.013 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.013 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.014 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.014 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.015 DEBUG:boto:Method: GET 2023-10-11T15:37:39.015 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.016 DEBUG:boto:Data: 2023-10-11T15:37:39.016 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.017 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.017 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.018 DEBUG:boto:Params: {} 2023-10-11T15:37:39.019 DEBUG:boto:Token: None 2023-10-11T15:37:39.019 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.020 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2mdQZnjynHfUAOwHUuM5HEUtXC0= 2023-10-11T15:37:39.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2mdQZnjynHfUAOwHUuM5HEUtXC0=', 'Content-Length': '0'} 2023-10-11T15:37:39.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000889a093e4f47130-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.024 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.025 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.026 DEBUG:boto:path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.026 DEBUG:boto:auth_path=/dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.027 DEBUG:boto:Method: GET 2023-10-11T15:37:39.028 DEBUG:boto:Path: /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.028 DEBUG:boto:Data: 2023-10-11T15:37:39.029 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.031 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.031 DEBUG:boto:Params: {} 2023-10-11T15:37:39.032 DEBUG:boto:Token: None 2023-10-11T15:37:39.032 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11?versionId=null 2023-10-11T15:37:39.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2mdQZnjynHfUAOwHUuM5HEUtXC0= 2023-10-11T15:37:39.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2mdQZnjynHfUAOwHUuM5HEUtXC0=', 'Content-Length': '0'} 2023-10-11T15:37:39.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f162a49140bdd04b-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.078 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.079 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.080 DEBUG:boto:path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.080 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.080 DEBUG:boto:Method: GET 2023-10-11T15:37:39.081 DEBUG:boto:Path: /dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.081 DEBUG:boto:Data: 2023-10-11T15:37:39.082 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.082 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.083 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.083 DEBUG:boto:Params: {} 2023-10-11T15:37:39.084 DEBUG:boto:Token: None 2023-10-11T15:37:39.084 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.085 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deff8cc8baed0254-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.088 DEBUG:boto:b'dghqvd-62obj111000falseobj112023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.090 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.090 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.091 DEBUG:boto:path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.091 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.091 DEBUG:boto:Method: GET 2023-10-11T15:37:39.092 DEBUG:boto:Path: /dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.092 DEBUG:boto:Data: 2023-10-11T15:37:39.093 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.093 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.094 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.094 DEBUG:boto:Params: {} 2023-10-11T15:37:39.095 DEBUG:boto:Token: None 2023-10-11T15:37:39.095 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f2e5b311c10c062-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.100 DEBUG:boto:b'dghqvd-62obj111000falseobj112023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.101 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.102 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.103 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.103 DEBUG:boto:Path: /dghqvd-62/obj11 2023-10-11T15:37:39.104 DEBUG:boto:Data: 2023-10-11T15:37:39.105 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.105 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.106 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.106 DEBUG:boto:Params: {} 2023-10-11T15:37:39.107 DEBUG:boto:Token: None 2023-10-11T15:37:39.108 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11 2023-10-11T15:37:39.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5S76wFUFSyUn8xu2WcFLPxQBlwY= 2023-10-11T15:37:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5S76wFUFSyUn8xu2WcFLPxQBlwY=', 'Content-Length': '0'} 2023-10-11T15:37:39.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf08be8b5216e28-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.112 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.112 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.112 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.113 DEBUG:boto:Path: /dghqvd-62/obj11 2023-10-11T15:37:39.113 DEBUG:boto:Data: 2023-10-11T15:37:39.114 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.114 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.115 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.115 DEBUG:boto:Params: {} 2023-10-11T15:37:39.116 DEBUG:boto:Token: None 2023-10-11T15:37:39.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11 2023-10-11T15:37:39.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5S76wFUFSyUn8xu2WcFLPxQBlwY= 2023-10-11T15:37:39.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5S76wFUFSyUn8xu2WcFLPxQBlwY=', 'Content-Length': '0'} 2023-10-11T15:37:39.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f8b564bab71a10a-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:37:39.123 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.123 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.124 DEBUG:boto:Method: GET 2023-10-11T15:37:39.124 DEBUG:boto:Path: /dghqvd-62/obj11 2023-10-11T15:37:39.124 DEBUG:boto:Data: 2023-10-11T15:37:39.125 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.126 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.127 DEBUG:boto:Params: {} 2023-10-11T15:37:39.127 DEBUG:boto:Token: None 2023-10-11T15:37:39.128 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11 2023-10-11T15:37:39.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dxZ2aawuJxDGAKvh3aP//XzjvQE= 2023-10-11T15:37:39.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dxZ2aawuJxDGAKvh3aP//XzjvQE=', 'Content-Length': '0'} 2023-10-11T15:37:39.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac20fc90a2c9b104-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.132 DEBUG:boto:path=/dghqvd-62/obj11 2023-10-11T15:37:39.133 DEBUG:boto:auth_path=/dghqvd-62/obj11 2023-10-11T15:37:39.134 DEBUG:boto:Method: GET 2023-10-11T15:37:39.135 DEBUG:boto:Path: /dghqvd-62/obj11 2023-10-11T15:37:39.135 DEBUG:boto:Data: 2023-10-11T15:37:39.136 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.137 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.137 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.138 DEBUG:boto:Params: {} 2023-10-11T15:37:39.139 DEBUG:boto:Token: None 2023-10-11T15:37:39.139 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj11 2023-10-11T15:37:39.140 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:dxZ2aawuJxDGAKvh3aP//XzjvQE= 2023-10-11T15:37:39.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dxZ2aawuJxDGAKvh3aP//XzjvQE=', 'Content-Length': '0'} 2023-10-11T15:37:39.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d944d1f307b197-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.190 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.191 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.191 DEBUG:boto:path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.192 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.192 DEBUG:boto:Method: GET 2023-10-11T15:37:39.193 DEBUG:boto:Path: /dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.193 DEBUG:boto:Data: 2023-10-11T15:37:39.193 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.194 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.195 DEBUG:boto:Params: {} 2023-10-11T15:37:39.195 DEBUG:boto:Token: None 2023-10-11T15:37:39.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040441972ecb63b1c-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.200 DEBUG:boto:b'dghqvd-62obj111000falseobj112023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.201 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.201 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.202 DEBUG:boto:path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.202 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.203 DEBUG:boto:Method: GET 2023-10-11T15:37:39.203 DEBUG:boto:Path: /dghqvd-62/?prefix=obj11 2023-10-11T15:37:39.204 DEBUG:boto:Data: 2023-10-11T15:37:39.204 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.205 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.205 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.206 DEBUG:boto:Params: {} 2023-10-11T15:37:39.206 DEBUG:boto:Token: None 2023-10-11T15:37:39.207 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.207 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343c6b87847e248b-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.211 DEBUG:boto:b'dghqvd-62obj111000falseobj112023-10-11T15:36:22.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:37:39.212 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.213 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.213 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.213 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.214 DEBUG:boto:Method: GET 2023-10-11T15:37:39.215 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.215 DEBUG:boto:Data: 2023-10-11T15:37:39.216 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.216 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.217 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.217 DEBUG:boto:Params: {} 2023-10-11T15:37:39.218 DEBUG:boto:Token: None 2023-10-11T15:37:39.218 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.219 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE= 2023-10-11T15:37:39.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE=', 'Content-Length': '0'} 2023-10-11T15:37:39.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2dd0b5eb0f4d77-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.262 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.263 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.263 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.264 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.264 DEBUG:boto:Method: GET 2023-10-11T15:37:39.265 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.265 DEBUG:boto:Data: 2023-10-11T15:37:39.265 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.266 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.267 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.267 DEBUG:boto:Params: {} 2023-10-11T15:37:39.267 DEBUG:boto:Token: None 2023-10-11T15:37:39.268 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.268 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE= 2023-10-11T15:37:39.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE=', 'Content-Length': '0'} 2023-10-11T15:37:39.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b027c94d47a087d-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.272 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.273 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.273 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.273 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.274 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.274 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.275 DEBUG:boto:Data: 2023-10-11T15:37:39.275 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.276 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.277 DEBUG:boto:Params: {} 2023-10-11T15:37:39.277 DEBUG:boto:Token: None 2023-10-11T15:37:39.278 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BbjnmArXx4x00SFS3EisggVSoa0= 2023-10-11T15:37:39.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BbjnmArXx4x00SFS3EisggVSoa0=', 'Content-Length': '0'} 2023-10-11T15:37:39.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d83d7c99f892eba-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.281 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.282 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.282 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.282 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.283 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.283 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.284 DEBUG:boto:Data: 2023-10-11T15:37:39.284 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.284 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.285 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.285 DEBUG:boto:Params: {} 2023-10-11T15:37:39.286 DEBUG:boto:Token: None 2023-10-11T15:37:39.286 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BbjnmArXx4x00SFS3EisggVSoa0= 2023-10-11T15:37:39.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BbjnmArXx4x00SFS3EisggVSoa0=', 'Content-Length': '0'} 2023-10-11T15:37:39.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e00fbba2d53051-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:37:39.290 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.290 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.291 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.291 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.293 DEBUG:boto:Method: GET 2023-10-11T15:37:39.294 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.294 DEBUG:boto:Data: 2023-10-11T15:37:39.295 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.295 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.296 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.296 DEBUG:boto:Params: {} 2023-10-11T15:37:39.297 DEBUG:boto:Token: None 2023-10-11T15:37:39.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.297 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE= 2023-10-11T15:37:39.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE=', 'Content-Length': '0'} 2023-10-11T15:37:39.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e42b50b79ed6f7b8-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.301 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.301 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.301 DEBUG:boto:path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.302 DEBUG:boto:auth_path=/dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.302 DEBUG:boto:Method: GET 2023-10-11T15:37:39.303 DEBUG:boto:Path: /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.303 DEBUG:boto:Data: 2023-10-11T15:37:39.303 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.304 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.304 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.305 DEBUG:boto:Params: {} 2023-10-11T15:37:39.305 DEBUG:boto:Token: None 2023-10-11T15:37:39.305 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22?versionId=null 2023-10-11T15:37:39.306 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE= 2023-10-11T15:37:39.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+nscAF0P9SBns+KdkIahkKj+JAE=', 'Content-Length': '0'} 2023-10-11T15:37:39.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d817c69e2185f376-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.309 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.309 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.310 DEBUG:boto:path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.310 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.311 DEBUG:boto:Method: GET 2023-10-11T15:37:39.311 DEBUG:boto:Path: /dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.311 DEBUG:boto:Data: 2023-10-11T15:37:39.312 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.313 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.313 DEBUG:boto:Params: {} 2023-10-11T15:37:39.313 DEBUG:boto:Token: None 2023-10-11T15:37:39.314 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029417af0eed78a47-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.318 DEBUG:boto:b'dghqvd-62obj221000falseobj222023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.319 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.319 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.320 DEBUG:boto:path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.320 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.320 DEBUG:boto:Method: GET 2023-10-11T15:37:39.321 DEBUG:boto:Path: /dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.321 DEBUG:boto:Data: 2023-10-11T15:37:39.322 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.322 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.322 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.323 DEBUG:boto:Params: {} 2023-10-11T15:37:39.323 DEBUG:boto:Token: None 2023-10-11T15:37:39.324 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.324 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1da352d5267792-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.328 DEBUG:boto:b'dghqvd-62obj221000falseobj222023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.328 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.329 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.329 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.330 DEBUG:boto:Path: /dghqvd-62/obj22 2023-10-11T15:37:39.330 DEBUG:boto:Data: 2023-10-11T15:37:39.331 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.331 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.331 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.332 DEBUG:boto:Params: {} 2023-10-11T15:37:39.332 DEBUG:boto:Token: None 2023-10-11T15:37:39.333 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22 2023-10-11T15:37:39.333 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FBMvQLo+C3me1X24vABidqjg8A8= 2023-10-11T15:37:39.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FBMvQLo+C3me1X24vABidqjg8A8=', 'Content-Length': '0'} 2023-10-11T15:37:39.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d377aa57dcc5aed6-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.336 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.337 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.337 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.337 DEBUG:boto:Path: /dghqvd-62/obj22 2023-10-11T15:37:39.338 DEBUG:boto:Data: 2023-10-11T15:37:39.338 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.339 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.340 DEBUG:boto:Params: {} 2023-10-11T15:37:39.340 DEBUG:boto:Token: None 2023-10-11T15:37:39.340 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22 2023-10-11T15:37:39.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FBMvQLo+C3me1X24vABidqjg8A8= 2023-10-11T15:37:39.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FBMvQLo+C3me1X24vABidqjg8A8=', 'Content-Length': '0'} 2023-10-11T15:37:39.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cb46dcf207438a-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:37:39.344 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.344 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.345 DEBUG:boto:Method: GET 2023-10-11T15:37:39.345 DEBUG:boto:Path: /dghqvd-62/obj22 2023-10-11T15:37:39.345 DEBUG:boto:Data: 2023-10-11T15:37:39.346 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.346 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.347 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.347 DEBUG:boto:Params: {} 2023-10-11T15:37:39.348 DEBUG:boto:Token: None 2023-10-11T15:37:39.348 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22 2023-10-11T15:37:39.349 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Ow5qUR0E3h82oUWbn1IkH9o66w= 2023-10-11T15:37:39.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Ow5qUR0E3h82oUWbn1IkH9o66w=', 'Content-Length': '0'} 2023-10-11T15:37:39.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcdd290b03f1f137-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.352 DEBUG:boto:path=/dghqvd-62/obj22 2023-10-11T15:37:39.352 DEBUG:boto:auth_path=/dghqvd-62/obj22 2023-10-11T15:37:39.352 DEBUG:boto:Method: GET 2023-10-11T15:37:39.353 DEBUG:boto:Path: /dghqvd-62/obj22 2023-10-11T15:37:39.353 DEBUG:boto:Data: 2023-10-11T15:37:39.354 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.354 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.354 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.355 DEBUG:boto:Params: {} 2023-10-11T15:37:39.355 DEBUG:boto:Token: None 2023-10-11T15:37:39.356 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj22 2023-10-11T15:37:39.356 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3Ow5qUR0E3h82oUWbn1IkH9o66w= 2023-10-11T15:37:39.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3Ow5qUR0E3h82oUWbn1IkH9o66w=', 'Content-Length': '0'} 2023-10-11T15:37:39.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4995b8e8e3118b7-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.402 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.403 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.403 DEBUG:boto:path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.404 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.404 DEBUG:boto:Method: GET 2023-10-11T15:37:39.405 DEBUG:boto:Path: /dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.405 DEBUG:boto:Data: 2023-10-11T15:37:39.405 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.406 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.406 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.407 DEBUG:boto:Params: {} 2023-10-11T15:37:39.407 DEBUG:boto:Token: None 2023-10-11T15:37:39.408 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.408 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eac5f4112947bbc-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.412 DEBUG:boto:b'dghqvd-62obj221000falseobj222023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.413 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.413 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.414 DEBUG:boto:path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.414 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.414 DEBUG:boto:Method: GET 2023-10-11T15:37:39.415 DEBUG:boto:Path: /dghqvd-62/?prefix=obj22 2023-10-11T15:37:39.415 DEBUG:boto:Data: 2023-10-11T15:37:39.416 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.416 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.417 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.417 DEBUG:boto:Params: {} 2023-10-11T15:37:39.417 DEBUG:boto:Token: None 2023-10-11T15:37:39.418 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.418 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8bd4c0764a32f10-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.422 DEBUG:boto:b'dghqvd-62obj221000falseobj222023-10-11T15:36:22.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:37:39.423 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.424 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.424 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.424 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.425 DEBUG:boto:Method: GET 2023-10-11T15:37:39.425 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.426 DEBUG:boto:Data: 2023-10-11T15:37:39.426 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.427 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.427 DEBUG:boto:Params: {} 2023-10-11T15:37:39.428 DEBUG:boto:Token: None 2023-10-11T15:37:39.428 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE= 2023-10-11T15:37:39.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE=', 'Content-Length': '0'} 2023-10-11T15:37:39.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d543c5ff540f042-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.432 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.433 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.433 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.434 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.434 DEBUG:boto:Method: GET 2023-10-11T15:37:39.434 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.435 DEBUG:boto:Data: 2023-10-11T15:37:39.435 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.436 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.436 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.436 DEBUG:boto:Params: {} 2023-10-11T15:37:39.437 DEBUG:boto:Token: None 2023-10-11T15:37:39.437 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.438 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE= 2023-10-11T15:37:39.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE=', 'Content-Length': '0'} 2023-10-11T15:37:39.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077ed2e85095f44b3-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.441 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.442 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.442 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.443 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.443 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.443 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.444 DEBUG:boto:Data: 2023-10-11T15:37:39.444 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.444 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.445 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.445 DEBUG:boto:Params: {} 2023-10-11T15:37:39.446 DEBUG:boto:Token: None 2023-10-11T15:37:39.446 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.446 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tySN5NWsn9i37wnI5wgG1ms+dt0= 2023-10-11T15:37:39.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tySN5NWsn9i37wnI5wgG1ms+dt0=', 'Content-Length': '0'} 2023-10-11T15:37:39.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa492ea62c0929d7-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.449 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.450 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.450 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.450 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.451 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.451 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.451 DEBUG:boto:Data: 2023-10-11T15:37:39.452 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.452 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.453 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.453 DEBUG:boto:Params: {} 2023-10-11T15:37:39.453 DEBUG:boto:Token: None 2023-10-11T15:37:39.454 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.454 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tySN5NWsn9i37wnI5wgG1ms+dt0= 2023-10-11T15:37:39.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tySN5NWsn9i37wnI5wgG1ms+dt0=', 'Content-Length': '0'} 2023-10-11T15:37:39.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df96c41f47f44570-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:37:39.458 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.458 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.459 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.459 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.459 DEBUG:boto:Method: GET 2023-10-11T15:37:39.460 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.460 DEBUG:boto:Data: 2023-10-11T15:37:39.461 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.461 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.462 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.462 DEBUG:boto:Params: {} 2023-10-11T15:37:39.462 DEBUG:boto:Token: None 2023-10-11T15:37:39.463 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.463 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE= 2023-10-11T15:37:39.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE=', 'Content-Length': '0'} 2023-10-11T15:37:39.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000731ad214ff433761-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.510 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.511 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.511 DEBUG:boto:path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.512 DEBUG:boto:auth_path=/dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.512 DEBUG:boto:Method: GET 2023-10-11T15:37:39.513 DEBUG:boto:Path: /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.513 DEBUG:boto:Data: 2023-10-11T15:37:39.513 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.514 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.514 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.515 DEBUG:boto:Params: {} 2023-10-11T15:37:39.515 DEBUG:boto:Token: None 2023-10-11T15:37:39.516 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33?versionId=null 2023-10-11T15:37:39.516 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE= 2023-10-11T15:37:39.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JaMX56bgYjkqhhPHkgFtf4zXzuE=', 'Content-Length': '0'} 2023-10-11T15:37:39.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008349bb6dae3b885c-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.520 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.520 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.521 DEBUG:boto:path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.521 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.521 DEBUG:boto:Method: GET 2023-10-11T15:37:39.522 DEBUG:boto:Path: /dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.522 DEBUG:boto:Data: 2023-10-11T15:37:39.523 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.523 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.523 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.524 DEBUG:boto:Params: {} 2023-10-11T15:37:39.524 DEBUG:boto:Token: None 2023-10-11T15:37:39.525 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d56d2ede90f087-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.529 DEBUG:boto:b'dghqvd-62obj331000falseobj332023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.530 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.530 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.531 DEBUG:boto:path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.531 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.532 DEBUG:boto:Method: GET 2023-10-11T15:37:39.532 DEBUG:boto:Path: /dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.532 DEBUG:boto:Data: 2023-10-11T15:37:39.533 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.533 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.534 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.534 DEBUG:boto:Params: {} 2023-10-11T15:37:39.535 DEBUG:boto:Token: None 2023-10-11T15:37:39.535 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.535 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7781364daa4c184-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.539 DEBUG:boto:b'dghqvd-62obj331000falseobj332023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.540 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.540 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.540 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.541 DEBUG:boto:Path: /dghqvd-62/obj33 2023-10-11T15:37:39.541 DEBUG:boto:Data: 2023-10-11T15:37:39.542 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.542 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.543 DEBUG:boto:Params: {} 2023-10-11T15:37:39.543 DEBUG:boto:Token: None 2023-10-11T15:37:39.544 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33 2023-10-11T15:37:39.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RFUgGbsPVHeU4tFp1aij9HdfAjc= 2023-10-11T15:37:39.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RFUgGbsPVHeU4tFp1aij9HdfAjc=', 'Content-Length': '0'} 2023-10-11T15:37:39.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abac87cb695f209b-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.547 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.547 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.548 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.548 DEBUG:boto:Path: /dghqvd-62/obj33 2023-10-11T15:37:39.548 DEBUG:boto:Data: 2023-10-11T15:37:39.549 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.549 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.549 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.550 DEBUG:boto:Params: {} 2023-10-11T15:37:39.550 DEBUG:boto:Token: None 2023-10-11T15:37:39.551 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33 2023-10-11T15:37:39.551 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RFUgGbsPVHeU4tFp1aij9HdfAjc= 2023-10-11T15:37:39.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RFUgGbsPVHeU4tFp1aij9HdfAjc=', 'Content-Length': '0'} 2023-10-11T15:37:39.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619c3547a66ca988-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:37:39.555 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.555 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.556 DEBUG:boto:Method: GET 2023-10-11T15:37:39.556 DEBUG:boto:Path: /dghqvd-62/obj33 2023-10-11T15:37:39.557 DEBUG:boto:Data: 2023-10-11T15:37:39.557 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.557 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.558 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.558 DEBUG:boto:Params: {} 2023-10-11T15:37:39.559 DEBUG:boto:Token: None 2023-10-11T15:37:39.559 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33 2023-10-11T15:37:39.560 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nq1h9pxr4eWAEz6otJ8gsYXkeBE= 2023-10-11T15:37:39.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nq1h9pxr4eWAEz6otJ8gsYXkeBE=', 'Content-Length': '0'} 2023-10-11T15:37:39.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c641e97f0c51a3-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.563 DEBUG:boto:path=/dghqvd-62/obj33 2023-10-11T15:37:39.564 DEBUG:boto:auth_path=/dghqvd-62/obj33 2023-10-11T15:37:39.564 DEBUG:boto:Method: GET 2023-10-11T15:37:39.565 DEBUG:boto:Path: /dghqvd-62/obj33 2023-10-11T15:37:39.565 DEBUG:boto:Data: 2023-10-11T15:37:39.565 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.566 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.566 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.567 DEBUG:boto:Params: {} 2023-10-11T15:37:39.567 DEBUG:boto:Token: None 2023-10-11T15:37:39.568 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj33 2023-10-11T15:37:39.568 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nq1h9pxr4eWAEz6otJ8gsYXkeBE= 2023-10-11T15:37:39.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nq1h9pxr4eWAEz6otJ8gsYXkeBE=', 'Content-Length': '0'} 2023-10-11T15:37:39.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014426f805a435313-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.614 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.615 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.615 DEBUG:boto:path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.616 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.616 DEBUG:boto:Method: GET 2023-10-11T15:37:39.616 DEBUG:boto:Path: /dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.617 DEBUG:boto:Data: 2023-10-11T15:37:39.617 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.618 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.619 DEBUG:boto:Params: {} 2023-10-11T15:37:39.619 DEBUG:boto:Token: None 2023-10-11T15:37:39.620 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030661173386f992e-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.623 DEBUG:boto:b'dghqvd-62obj331000falseobj332023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.625 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.625 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.625 DEBUG:boto:path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.626 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.626 DEBUG:boto:Method: GET 2023-10-11T15:37:39.627 DEBUG:boto:Path: /dghqvd-62/?prefix=obj33 2023-10-11T15:37:39.627 DEBUG:boto:Data: 2023-10-11T15:37:39.628 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.628 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.628 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.629 DEBUG:boto:Params: {} 2023-10-11T15:37:39.629 DEBUG:boto:Token: None 2023-10-11T15:37:39.630 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.630 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003910e4ef24097990-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.634 DEBUG:boto:b'dghqvd-62obj331000falseobj332023-10-11T15:36:22.087Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:37:39.635 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.635 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.636 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.636 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.637 DEBUG:boto:Method: GET 2023-10-11T15:37:39.637 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.638 DEBUG:boto:Data: 2023-10-11T15:37:39.638 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.638 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.639 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.639 DEBUG:boto:Params: {} 2023-10-11T15:37:39.640 DEBUG:boto:Token: None 2023-10-11T15:37:39.640 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.641 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY= 2023-10-11T15:37:39.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY=', 'Content-Length': '0'} 2023-10-11T15:37:39.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1eab17466b905f-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.644 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.645 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.645 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.645 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.646 DEBUG:boto:Method: GET 2023-10-11T15:37:39.647 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.647 DEBUG:boto:Data: 2023-10-11T15:37:39.648 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.648 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.649 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.650 DEBUG:boto:Params: {} 2023-10-11T15:37:39.650 DEBUG:boto:Token: None 2023-10-11T15:37:39.651 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.651 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY= 2023-10-11T15:37:39.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY=', 'Content-Length': '0'} 2023-10-11T15:37:39.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c143393e43bc9ca5-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.654 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.655 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.655 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.656 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.656 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.657 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.657 DEBUG:boto:Data: 2023-10-11T15:37:39.657 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.658 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.658 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.659 DEBUG:boto:Params: {} 2023-10-11T15:37:39.660 DEBUG:boto:Token: None 2023-10-11T15:37:39.660 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.660 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Et3+eoIr+IymVNxwDDZlfTdPHeI= 2023-10-11T15:37:39.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Et3+eoIr+IymVNxwDDZlfTdPHeI=', 'Content-Length': '0'} 2023-10-11T15:37:39.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb10dc5433d8187-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.664 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.664 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.664 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.665 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.665 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.666 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.666 DEBUG:boto:Data: 2023-10-11T15:37:39.667 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.667 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.668 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.668 DEBUG:boto:Params: {} 2023-10-11T15:37:39.669 DEBUG:boto:Token: None 2023-10-11T15:37:39.670 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.671 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Et3+eoIr+IymVNxwDDZlfTdPHeI= 2023-10-11T15:37:39.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Et3+eoIr+IymVNxwDDZlfTdPHeI=', 'Content-Length': '0'} 2023-10-11T15:37:39.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38e29efc69106ce-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:37:39.675 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.675 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.676 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.676 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.676 DEBUG:boto:Method: GET 2023-10-11T15:37:39.677 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.677 DEBUG:boto:Data: 2023-10-11T15:37:39.678 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.678 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.679 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.679 DEBUG:boto:Params: {} 2023-10-11T15:37:39.680 DEBUG:boto:Token: None 2023-10-11T15:37:39.680 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.681 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY= 2023-10-11T15:37:39.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY=', 'Content-Length': '0'} 2023-10-11T15:37:39.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d01e682cb81c4853-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.726 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.727 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.727 DEBUG:boto:path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.728 DEBUG:boto:auth_path=/dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.728 DEBUG:boto:Method: GET 2023-10-11T15:37:39.729 DEBUG:boto:Path: /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.729 DEBUG:boto:Data: 2023-10-11T15:37:39.730 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.730 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.731 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.731 DEBUG:boto:Params: {} 2023-10-11T15:37:39.731 DEBUG:boto:Token: None 2023-10-11T15:37:39.732 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44?versionId=null 2023-10-11T15:37:39.732 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY= 2023-10-11T15:37:39.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L4ySjg3og7oibkb5rMm8Zf6N8pY=', 'Content-Length': '0'} 2023-10-11T15:37:39.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a2ba2ef2534828-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.736 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.737 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.737 DEBUG:boto:path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.737 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.738 DEBUG:boto:Method: GET 2023-10-11T15:37:39.738 DEBUG:boto:Path: /dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.739 DEBUG:boto:Data: 2023-10-11T15:37:39.739 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.740 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.740 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.740 DEBUG:boto:Params: {} 2023-10-11T15:37:39.741 DEBUG:boto:Token: None 2023-10-11T15:37:39.741 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.742 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df2b3c3ba275e27-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.745 DEBUG:boto:b'dghqvd-62obj441000falseobj442023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.746 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.746 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.747 DEBUG:boto:path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.747 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.748 DEBUG:boto:Method: GET 2023-10-11T15:37:39.748 DEBUG:boto:Path: /dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.749 DEBUG:boto:Data: 2023-10-11T15:37:39.749 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.749 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.750 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.750 DEBUG:boto:Params: {} 2023-10-11T15:37:39.751 DEBUG:boto:Token: None 2023-10-11T15:37:39.751 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b5d3aaffe05547-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.755 DEBUG:boto:b'dghqvd-62obj441000falseobj442023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.755 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.756 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.756 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.757 DEBUG:boto:Path: /dghqvd-62/obj44 2023-10-11T15:37:39.757 DEBUG:boto:Data: 2023-10-11T15:37:39.758 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.758 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.758 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.759 DEBUG:boto:Params: {} 2023-10-11T15:37:39.759 DEBUG:boto:Token: None 2023-10-11T15:37:39.760 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44 2023-10-11T15:37:39.760 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rTimjp+7Zau78b0i0c+UEUXT3Y0= 2023-10-11T15:37:39.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rTimjp+7Zau78b0i0c+UEUXT3Y0=', 'Content-Length': '0'} 2023-10-11T15:37:39.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d8247c4b263f475-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.763 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.764 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.764 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.764 DEBUG:boto:Path: /dghqvd-62/obj44 2023-10-11T15:37:39.765 DEBUG:boto:Data: 2023-10-11T15:37:39.765 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.766 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.766 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.766 DEBUG:boto:Params: {} 2023-10-11T15:37:39.767 DEBUG:boto:Token: None 2023-10-11T15:37:39.767 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44 2023-10-11T15:37:39.768 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rTimjp+7Zau78b0i0c+UEUXT3Y0= 2023-10-11T15:37:39.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rTimjp+7Zau78b0i0c+UEUXT3Y0=', 'Content-Length': '0'} 2023-10-11T15:37:39.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32fe7b42f6cca9b-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:37:39.771 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.771 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.772 DEBUG:boto:Method: GET 2023-10-11T15:37:39.772 DEBUG:boto:Path: /dghqvd-62/obj44 2023-10-11T15:37:39.773 DEBUG:boto:Data: 2023-10-11T15:37:39.773 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.774 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.774 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.774 DEBUG:boto:Params: {} 2023-10-11T15:37:39.775 DEBUG:boto:Token: None 2023-10-11T15:37:39.775 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44 2023-10-11T15:37:39.776 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tq+q7wyJzM2hEFkOk05yTeDfd2E= 2023-10-11T15:37:39.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tq+q7wyJzM2hEFkOk05yTeDfd2E=', 'Content-Length': '0'} 2023-10-11T15:37:39.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c42fa42f16f0c31-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.779 DEBUG:boto:path=/dghqvd-62/obj44 2023-10-11T15:37:39.779 DEBUG:boto:auth_path=/dghqvd-62/obj44 2023-10-11T15:37:39.780 DEBUG:boto:Method: GET 2023-10-11T15:37:39.780 DEBUG:boto:Path: /dghqvd-62/obj44 2023-10-11T15:37:39.780 DEBUG:boto:Data: 2023-10-11T15:37:39.781 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.781 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.782 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.782 DEBUG:boto:Params: {} 2023-10-11T15:37:39.782 DEBUG:boto:Token: None 2023-10-11T15:37:39.783 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj44 2023-10-11T15:37:39.783 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Tq+q7wyJzM2hEFkOk05yTeDfd2E= 2023-10-11T15:37:39.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Tq+q7wyJzM2hEFkOk05yTeDfd2E=', 'Content-Length': '0'} 2023-10-11T15:37:39.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080940edbee68ed60-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.826 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.827 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.828 DEBUG:boto:path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.828 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.828 DEBUG:boto:Method: GET 2023-10-11T15:37:39.829 DEBUG:boto:Path: /dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.829 DEBUG:boto:Data: 2023-10-11T15:37:39.830 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.831 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.831 DEBUG:boto:Params: {} 2023-10-11T15:37:39.832 DEBUG:boto:Token: None 2023-10-11T15:37:39.832 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.833 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096dced75a9626c66-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.836 DEBUG:boto:b'dghqvd-62obj441000falseobj442023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.837 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.838 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.838 DEBUG:boto:path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.838 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.839 DEBUG:boto:Method: GET 2023-10-11T15:37:39.839 DEBUG:boto:Path: /dghqvd-62/?prefix=obj44 2023-10-11T15:37:39.840 DEBUG:boto:Data: 2023-10-11T15:37:39.840 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.841 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.841 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.842 DEBUG:boto:Params: {} 2023-10-11T15:37:39.842 DEBUG:boto:Token: None 2023-10-11T15:37:39.843 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.843 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000803ed71cba4012d3-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.847 DEBUG:boto:b'dghqvd-62obj441000falseobj442023-10-11T15:36:22.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:37:39.848 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.848 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.849 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.849 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.849 DEBUG:boto:Method: GET 2023-10-11T15:37:39.850 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.850 DEBUG:boto:Data: 2023-10-11T15:37:39.851 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.851 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.852 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.852 DEBUG:boto:Params: {} 2023-10-11T15:37:39.852 DEBUG:boto:Token: None 2023-10-11T15:37:39.853 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.853 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c= 2023-10-11T15:37:39.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c=', 'Content-Length': '0'} 2023-10-11T15:37:39.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012645d7facdb99de-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.857 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.857 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.858 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.858 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.858 DEBUG:boto:Method: GET 2023-10-11T15:37:39.859 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.859 DEBUG:boto:Data: 2023-10-11T15:37:39.860 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.860 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.861 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.861 DEBUG:boto:Params: {} 2023-10-11T15:37:39.862 DEBUG:boto:Token: None 2023-10-11T15:37:39.862 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.863 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c= 2023-10-11T15:37:39.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c=', 'Content-Length': '0'} 2023-10-11T15:37:39.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a10e42d8da69038-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.866 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.866 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.867 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.867 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.868 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.868 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.868 DEBUG:boto:Data: 2023-10-11T15:37:39.869 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.869 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.870 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.870 DEBUG:boto:Params: {} 2023-10-11T15:37:39.870 DEBUG:boto:Token: None 2023-10-11T15:37:39.871 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.871 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CDvYlBaArpEJr4F5pe8V+Ol9Lcg= 2023-10-11T15:37:39.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CDvYlBaArpEJr4F5pe8V+Ol9Lcg=', 'Content-Length': '0'} 2023-10-11T15:37:39.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5804f6dbfd8bc6-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.874 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.875 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.875 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.875 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.876 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.876 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.877 DEBUG:boto:Data: 2023-10-11T15:37:39.877 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.877 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.878 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.879 DEBUG:boto:Params: {} 2023-10-11T15:37:39.879 DEBUG:boto:Token: None 2023-10-11T15:37:39.879 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.880 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CDvYlBaArpEJr4F5pe8V+Ol9Lcg= 2023-10-11T15:37:39.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CDvYlBaArpEJr4F5pe8V+Ol9Lcg=', 'Content-Length': '0'} 2023-10-11T15:37:39.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aceb30eb56076522-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:37:39.883 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.884 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.884 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.885 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.885 DEBUG:boto:Method: GET 2023-10-11T15:37:39.885 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.886 DEBUG:boto:Data: 2023-10-11T15:37:39.886 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.887 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.887 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.888 DEBUG:boto:Params: {} 2023-10-11T15:37:39.888 DEBUG:boto:Token: None 2023-10-11T15:37:39.889 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.889 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c= 2023-10-11T15:37:39.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c=', 'Content-Length': '0'} 2023-10-11T15:37:39.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efe8bec9f2ec4c9c-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.892 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.893 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.893 DEBUG:boto:path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.894 DEBUG:boto:auth_path=/dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.894 DEBUG:boto:Method: GET 2023-10-11T15:37:39.895 DEBUG:boto:Path: /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.895 DEBUG:boto:Data: 2023-10-11T15:37:39.896 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.896 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.897 DEBUG:boto:Params: {} 2023-10-11T15:37:39.897 DEBUG:boto:Token: None 2023-10-11T15:37:39.898 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55?versionId=null 2023-10-11T15:37:39.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c= 2023-10-11T15:37:39.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ONkfPXWjHQ6yw6RUXD163THwy8c=', 'Content-Length': '0'} 2023-10-11T15:37:39.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4e5329ed35eb38-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.942 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.943 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.944 DEBUG:boto:path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.944 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.944 DEBUG:boto:Method: GET 2023-10-11T15:37:39.945 DEBUG:boto:Path: /dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.945 DEBUG:boto:Data: 2023-10-11T15:37:39.946 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.947 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.947 DEBUG:boto:Params: {} 2023-10-11T15:37:39.948 DEBUG:boto:Token: None 2023-10-11T15:37:39.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dbf7ca1b62263ff-006526c143-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.952 DEBUG:boto:b'dghqvd-62obj551000falseobj552023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.953 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:39.954 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:39.954 DEBUG:boto:path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.955 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.955 DEBUG:boto:Method: GET 2023-10-11T15:37:39.956 DEBUG:boto:Path: /dghqvd-62/?prefix=obj55 2023-10-11T15:37:39.956 DEBUG:boto:Data: 2023-10-11T15:37:39.957 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.958 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.958 DEBUG:boto:Params: {} 2023-10-11T15:37:39.959 DEBUG:boto:Token: None 2023-10-11T15:37:39.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/ 2023-10-11T15:37:39.960 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg= 2023-10-11T15:37:39.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gg1TT6UNFvM4BuzHplKx+uRq0Hg=', 'Content-Length': '0'} 2023-10-11T15:37:39.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004bbda137ee54aa4-006526c143-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.964 DEBUG:boto:b'dghqvd-62obj551000falseobj552023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:39.965 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.965 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.966 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.966 DEBUG:boto:Path: /dghqvd-62/obj55 2023-10-11T15:37:39.966 DEBUG:boto:Data: 2023-10-11T15:37:39.967 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.967 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.968 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.968 DEBUG:boto:Params: {} 2023-10-11T15:37:39.968 DEBUG:boto:Token: None 2023-10-11T15:37:39.969 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55 2023-10-11T15:37:39.969 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nLHgZmdmQQ/hD4ZzTvxY/Zme91c= 2023-10-11T15:37:39.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nLHgZmdmQQ/hD4ZzTvxY/Zme91c=', 'Content-Length': '0'} 2023-10-11T15:37:39.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007946bb67c05a28b4-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.972 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.973 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.973 DEBUG:boto:Method: HEAD 2023-10-11T15:37:39.974 DEBUG:boto:Path: /dghqvd-62/obj55 2023-10-11T15:37:39.974 DEBUG:boto:Data: 2023-10-11T15:37:39.974 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.975 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.975 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.976 DEBUG:boto:Params: {} 2023-10-11T15:37:39.976 DEBUG:boto:Token: None 2023-10-11T15:37:39.976 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55 2023-10-11T15:37:39.977 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nLHgZmdmQQ/hD4ZzTvxY/Zme91c= 2023-10-11T15:37:39.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nLHgZmdmQQ/hD4ZzTvxY/Zme91c=', 'Content-Length': '0'} 2023-10-11T15:37:39.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf2d7a32e8d7aa7-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:37:39.980 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.981 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.981 DEBUG:boto:Method: GET 2023-10-11T15:37:39.981 DEBUG:boto:Path: /dghqvd-62/obj55 2023-10-11T15:37:39.982 DEBUG:boto:Data: 2023-10-11T15:37:39.982 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:39.983 DEBUG:boto:Port: 8001 2023-10-11T15:37:39.984 DEBUG:boto:Params: {} 2023-10-11T15:37:39.984 DEBUG:boto:Token: None 2023-10-11T15:37:39.984 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55 2023-10-11T15:37:39.985 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iB2ncz2uPCIrbA0xWLhHUfNNlOs= 2023-10-11T15:37:39.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iB2ncz2uPCIrbA0xWLhHUfNNlOs=', 'Content-Length': '0'} 2023-10-11T15:37:39.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-62:daa316e9-869c-4074-a756-a8adfb4a3573.5751.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048e938a4eb682014-006526c143-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:39.988 DEBUG:boto:path=/dghqvd-62/obj55 2023-10-11T15:37:39.988 DEBUG:boto:auth_path=/dghqvd-62/obj55 2023-10-11T15:37:39.989 DEBUG:boto:Method: GET 2023-10-11T15:37:39.989 DEBUG:boto:Path: /dghqvd-62/obj55 2023-10-11T15:37:39.990 DEBUG:boto:Data: 2023-10-11T15:37:39.990 DEBUG:boto:Headers: {} 2023-10-11T15:37:39.991 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:39.991 DEBUG:boto:Port: 8000 2023-10-11T15:37:39.991 DEBUG:boto:Params: {} 2023-10-11T15:37:39.992 DEBUG:boto:Token: None 2023-10-11T15:37:39.992 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:39 GMT /dghqvd-62/obj55 2023-10-11T15:37:39.993 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iB2ncz2uPCIrbA0xWLhHUfNNlOs= 2023-10-11T15:37:39.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:39 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iB2ncz2uPCIrbA0xWLhHUfNNlOs=', 'Content-Length': '0'} 2023-10-11T15:37:39.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:36:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090dd08e27b72d91b-006526c143-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:37:39 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:40.038 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:40.039 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:40.039 DEBUG:boto:path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.040 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.040 DEBUG:boto:Method: GET 2023-10-11T15:37:40.041 DEBUG:boto:Path: /dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.041 DEBUG:boto:Data: 2023-10-11T15:37:40.042 DEBUG:boto:Headers: {} 2023-10-11T15:37:40.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:37:40.043 DEBUG:boto:Port: 8001 2023-10-11T15:37:40.043 DEBUG:boto:Params: {} 2023-10-11T15:37:40.043 DEBUG:boto:Token: None 2023-10-11T15:37:40.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:40 GMT /dghqvd-62/ 2023-10-11T15:37:40.044 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hwZGPhG/OpmD3bEXjebFa/nLNi8= 2023-10-11T15:37:40.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hwZGPhG/OpmD3bEXjebFa/nLNi8=', 'Content-Length': '0'} 2023-10-11T15:37:40.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002861101fd74f7451-006526c144-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:40.048 DEBUG:boto:b'dghqvd-62obj551000falseobj552023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:40.049 DEBUG:boto:path=/dghqvd-62/ 2023-10-11T15:37:40.049 DEBUG:boto:auth_path=/dghqvd-62/ 2023-10-11T15:37:40.050 DEBUG:boto:path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.050 DEBUG:boto:auth_path=/dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.051 DEBUG:boto:Method: GET 2023-10-11T15:37:40.051 DEBUG:boto:Path: /dghqvd-62/?prefix=obj55 2023-10-11T15:37:40.051 DEBUG:boto:Data: 2023-10-11T15:37:40.052 DEBUG:boto:Headers: {} 2023-10-11T15:37:40.052 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:37:40.053 DEBUG:boto:Port: 8000 2023-10-11T15:37:40.053 DEBUG:boto:Params: {} 2023-10-11T15:37:40.054 DEBUG:boto:Token: None 2023-10-11T15:37:40.054 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:37:40 GMT /dghqvd-62/ 2023-10-11T15:37:40.055 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hwZGPhG/OpmD3bEXjebFa/nLNi8= 2023-10-11T15:37:40.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:37:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hwZGPhG/OpmD3bEXjebFa/nLNi8=', 'Content-Length': '0'} 2023-10-11T15:37:40.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d1e0316264861f-006526c144-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:37:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:37:40.058 DEBUG:boto:b'dghqvd-62obj551000falseobj552023-10-11T15:36:22.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:37:40.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-62 zones={a2, a1} 2023-10-11T15:37:40.059 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-10-11T15:37:40.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:40.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:40.061 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:37:40.115 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:40.115 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:40.166 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:37:40.167 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.167 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032631", 2023-10-11T15:37:40.167 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:36:43.123755Z" 2023-10-11T15:37:40.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.168 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.168 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:37:40.168 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:37:40.168 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.168 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.169 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:37:40.169 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:37:40.169 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.169 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.170 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:37:40.170 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:37:40.170 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.170 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:37:40.171 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:40.172 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:37:40.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:40.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:40.173 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:37:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:40.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:40.270 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:37:40.270 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:37:40.270 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:37:40.271 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:37:40.271 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:37:40.271 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 16319284139904412300 2023-10-11T15:37:40.271 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.271 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:37:40.272 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.272 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:37:40.272 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:40.272 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:40.273 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:40.273 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000032489", 2023-10-11T15:37:40.273 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:40.273 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:40.273 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.051057Z" 2023-10-11T15:37:40.274 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.274 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.274 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:37:40.275 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:40.275 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:40.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:40.275 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:37:40.275 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:40.276 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:40.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:37:40.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.277 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.277 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:37:40.277 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:40.277 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:40.277 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:37:40.278 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:40.278 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 53, 2023-10-11T15:37:40.278 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:40.278 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:37:40.279 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.279 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.279 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:40.279 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:37:40.279 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:40.280 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:40.280 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:37:40.280 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:40.280 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:37:40.280 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:40.281 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:37:40.281 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.281 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:40.281 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:37:40.281 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.282 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:37:40.282 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 62, 2023-10-11T15:37:40.282 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 62 2023-10-11T15:37:40.282 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:40.283 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:37:40Z" 2023-10-11T15:37:40.283 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:37:40.284 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16319284139904412300 }, "markers": [ { "key": 0, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000032489", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:36:22.051057Z" } }, { "key": 1, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000023427", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-11T15:36:22.149856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032717", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:36:43.125680Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-11T15:37:40Z" } 2023-10-11T15:37:40.284 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000032489', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.051057Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000023427', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.149856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032717', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.125680Z'}}] 2023-10-11T15:37:40.285 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:40.285 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:40.286 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032631 target=, shard=1 master=00000000000000000000:00000000000000023427 target= 2023-10-11T15:37:45.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:45.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:45.288 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:37:45.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:45.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:45.397 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:37:45.397 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:37:45.397 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:37:45.398 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:37:45.398 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:37:45.398 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 16319284139904412300 2023-10-11T15:37:45.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.398 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:37:45.399 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:45.399 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:37:45.399 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:45.399 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:45.400 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:45.400 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000032489", 2023-10-11T15:37:45.400 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:45.400 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:45.400 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.051057Z" 2023-10-11T15:37:45.401 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:45.401 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.401 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:45.401 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:37:45.401 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:45.402 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:45.402 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:45.402 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:37:45.402 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:45.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:45.403 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:37:45.403 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:45.403 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.403 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:45.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:37:45.404 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:45.404 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:45.404 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:37:45.404 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:45.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 53, 2023-10-11T15:37:45.405 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:45.405 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:37:45.405 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:45.405 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.406 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:45.406 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:37:45.406 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:45.406 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:45.407 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:37:45.407 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:45.407 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:37:45.407 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:45.407 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:37:45.408 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:45.408 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:45.408 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:37:45.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.408 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:37:45.409 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 62, 2023-10-11T15:37:45.409 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 62 2023-10-11T15:37:45.409 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:45.409 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:37:45Z" 2023-10-11T15:37:45.409 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:37:45.411 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16319284139904412300 }, "markers": [ { "key": 0, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000032489", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:36:22.051057Z" } }, { "key": 1, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000023427", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-11T15:36:22.149856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032717", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:36:43.125680Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-11T15:37:45Z" } 2023-10-11T15:37:45.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000032489', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.051057Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000023427', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.149856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032717', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.125680Z'}}] 2023-10-11T15:37:45.412 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:45.412 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:45.412 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032631 target=, shard=1 master=00000000000000000000:00000000000000023427 target= 2023-10-11T15:37:50.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:50.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:50.415 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:37:50.476 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:50.477 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:50.530 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:37:50.531 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:37:50.531 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:37:50.531 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:37:50.532 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:37:50.532 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 16319284139904412300 2023-10-11T15:37:50.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.532 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:37:50.532 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:50.533 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:37:50.533 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:50.533 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:50.533 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:50.533 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000032489", 2023-10-11T15:37:50.534 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:50.534 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:50.534 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.051057Z" 2023-10-11T15:37:50.534 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:50.535 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.535 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:50.535 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:37:50.535 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:50.535 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:50.536 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:50.536 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:37:50.536 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:50.536 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:50.537 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:37:50.537 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:50.537 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.537 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:50.538 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:37:50.538 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:50.538 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:50.538 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:37:50.539 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:50.539 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 53, 2023-10-11T15:37:50.539 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:50.539 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:37:50.539 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:50.540 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.540 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:50.540 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:37:50.540 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:50.540 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:50.541 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:37:50.541 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:50.541 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:37:50.541 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:50.541 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:37:50.542 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:50.542 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:50.542 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:37:50.542 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.542 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:37:50.543 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 62, 2023-10-11T15:37:50.543 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 62 2023-10-11T15:37:50.543 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:50.543 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:37:50Z" 2023-10-11T15:37:50.544 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:37:50.545 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16319284139904412300 }, "markers": [ { "key": 0, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000032489", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:36:22.051057Z" } }, { "key": 1, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000023427", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-11T15:36:22.149856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032717", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:36:43.125680Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-11T15:37:50Z" } 2023-10-11T15:37:50.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000032489', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.051057Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000023427', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.149856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032717', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.125680Z'}}] 2023-10-11T15:37:50.546 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:50.546 DEBUG:rgw_multi.tests:sync_status={0: '', 1: '', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:50.547 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000032631 target=, shard=1 master=00000000000000000000:00000000000000023427 target= 2023-10-11T15:37:55.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:55.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:37:55.550 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:37:55.610 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:37:55.611 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:37:55.658 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:37:55.658 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:37:55.658 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:37:55.659 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:37:55.659 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:37:55.659 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 16319284139904412300 2023-10-11T15:37:55.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.660 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:37:55.660 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:55.660 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:37:55.661 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:55.662 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:55.662 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032631", 2023-10-11T15:37:55.662 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:55.663 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:55.663 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:55.663 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.123755Z" 2023-10-11T15:37:55.664 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:55.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:55.664 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:37:55.664 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:55.665 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "full-sync", 2023-10-11T15:37:55.665 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:37:55.665 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:37:55.665 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:37:55.666 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:55.666 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:37:55.666 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:55.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.667 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:55.667 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:37:55.667 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:55.667 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:55.668 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:37:55.668 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:55.668 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 53, 2023-10-11T15:37:55.668 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:55.668 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:37:55.669 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:55.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.669 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:37:55.669 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:37:55.670 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:37:55.670 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:37:55.670 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:37:55.671 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:37:55.671 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:37:55.672 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:37:55.672 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:37:55.672 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:55.673 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:37:55.673 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:37:55.673 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.673 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:37:55.673 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 62, 2023-10-11T15:37:55.674 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 62 2023-10-11T15:37:55.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:37:55.674 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:37:55Z" 2023-10-11T15:37:55.674 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:37:55.676 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16319284139904412300 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032631", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:36:43.123755Z" } }, { "key": 1, "val": { "status": "full-sync", "marker": "", "next_step_marker": "00000000000000000000:00000000000000023427", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-11T15:36:22.149856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032717", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:36:43.125680Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-11T15:37:55Z" } 2023-10-11T15:37:55.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032631', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.123755Z'}}, {'key': 1, 'val': {'status': 'full-sync', 'marker': '', 'next_step_marker': '00000000000000000000:00000000000000023427', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.149856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032717', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.125680Z'}}] 2023-10-11T15:37:55.677 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:55.677 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032631', 1: '', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:37:55.678 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023427 target= 2023-10-11T15:38:00.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.680 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:00.740 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:00.741 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:00.787 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:00.788 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:00.788 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:00.788 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:00.788 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:00.789 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 16319284139904412300 2023-10-11T15:38:00.789 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.789 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:00.789 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.790 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:00.790 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:00.790 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:00.791 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032631", 2023-10-11T15:38:00.791 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:00.791 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:00.792 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:00.792 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.123755Z" 2023-10-11T15:38:00.792 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.793 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.793 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.793 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:00.794 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:00.794 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:00.794 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000023427", 2023-10-11T15:38:00.794 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:00.795 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:00.795 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:00.795 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:31:57.119178Z" 2023-10-11T15:38:00.795 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.796 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.796 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.796 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:00.796 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:00.796 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:00.797 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000037529", 2023-10-11T15:38:00.797 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:00.797 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 53, 2023-10-11T15:38:00.797 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:00.798 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:22.149856Z" 2023-10-11T15:38:00.798 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.798 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.798 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.799 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:00.799 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:00.799 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:00.799 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032717", 2023-10-11T15:38:00.799 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:00.800 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:38:00.800 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:00.800 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:36:43.125680Z" 2023-10-11T15:38:00.800 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.800 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.801 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:00.801 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.801 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:00.801 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 62, 2023-10-11T15:38:00.802 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 62 2023-10-11T15:38:00.802 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.802 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:00Z" 2023-10-11T15:38:00.802 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:00.804 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16319284139904412300 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032631", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:36:43.123755Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023427", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:31:57.119178Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037529", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-10-11T15:36:22.149856Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032717", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:36:43.125680Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-10-11T15:38:00Z" } 2023-10-11T15:38:00.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032631', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.123755Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023427', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:31:57.119178Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037529', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-10-11T15:36:22.149856Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032717', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:36:43.125680Z'}}] 2023-10-11T15:38:00.805 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:38:00.805 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000032631', 1: '00000000000000000000:00000000000000023427', 2: '00000000000000000000:00000000000000037529', 3: '00000000000000000000:00000000000000032717'} 2023-10-11T15:38:00.806 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:38:00.806 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-10-11T15:38:00.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.808 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:00.858 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:00.859 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:00.908 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:38:00.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.909 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-10-11T15:38:00.909 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:37:28.448167Z" 2023-10-11T15:38:00.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.910 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005586", 2023-10-11T15:38:00.910 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:33:19.029468Z" 2023-10-11T15:38:00.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.910 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.911 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015729", 2023-10-11T15:38:00.911 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:37:26.724956Z" 2023-10-11T15:38:00.911 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:00.911 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:00.912 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-10-11T15:38:00.912 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:37:28.450335Z" 2023-10-11T15:38:00.912 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:00.912 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:38:00.913 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000011678', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015729', 3: '00000000000000000000:00000000000000014604'} 2023-10-11T15:38:00.914 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:38:00.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:00.915 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:00.966 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:00.967 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:01.010 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:01.010 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:01.010 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:01.011 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:01.011 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:01.011 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T15:38:01.011 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.012 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:01.012 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.012 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:01.012 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.012 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-10-11T15:38:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:37:28.448167Z" 2023-10-11T15:38:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000005586", 2023-10-11T15:38:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:33:19.029468Z" 2023-10-11T15:38:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000015729", 2023-10-11T15:38:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:37:26.724956Z" 2023-10-11T15:38:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:38:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-10-11T15:38:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:37:28.450335Z" 2023-10-11T15:38:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T15:38:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T15:38:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:00Z" 2023-10-11T15:38:01.023 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:01.024 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011678", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:37:28.448167Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005586", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:33:19.029468Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015729", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:37:26.724956Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014604", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:37:28.450335Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T15:38:00Z" } 2023-10-11T15:38:01.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011678', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:37:28.448167Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005586', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:33:19.029468Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015729', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:37:26.724956Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014604', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:37:28.450335Z'}}] 2023-10-11T15:38:01.025 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011678', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015729', 3: '00000000000000000000:00000000000000014604'} 2023-10-11T15:38:01.025 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011678', 1: '00000000000000000000:00000000000000005586', 2: '00000000000000000000:00000000000000015729', 3: '00000000000000000000:00000000000000014604'} 2023-10-11T15:38:01.025 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T15:38:01.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-10-11T15:38:01.027 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:01.028 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:01.028 DEBUG:boto:Method: PUT 2023-10-11T15:38:01.029 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:01.029 DEBUG:boto:Data: 2023-10-11T15:38:01.030 DEBUG:boto:Headers: {} 2023-10-11T15:38:01.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:01.030 DEBUG:boto:Port: 8000 2023-10-11T15:38:01.031 DEBUG:boto:Params: {} 2023-10-11T15:38:01.031 DEBUG:boto:Token: None 2023-10-11T15:38:01.032 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:38:01 GMT /dghqvd-63/ 2023-10-11T15:38:01.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:98GGLTNRHLptXpJIMNE9FbzqtGk= 2023-10-11T15:38:01.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:98GGLTNRHLptXpJIMNE9FbzqtGk=', 'Content-Length': '0'} 2023-10-11T15:38:01.048 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000166d13f94cc0cb46-006526c159-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:01.049 DEBUG:rgw_multi.tests:created bucket=dghqvd-63 2023-10-11T15:38:01.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:01.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:01.050 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:01.099 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:01.100 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:01.129 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:38:01.141 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:38:01.142 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.142 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:01.142 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:38:01.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.143 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.143 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:01.143 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:38:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:38:01.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.145 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.145 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038681.046367_553.1", 2023-10-11T15:38:01.145 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:38:01.046367Z" 2023-10-11T15:38:01.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.145 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:38:01.147 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:01.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:38:01.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:01.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:01.149 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:01.197 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:01.198 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:01.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:01.226+0000 7f04d8e2b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:01.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:01.227+0000 7f04d8e2b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:01.240 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:01.240 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:01.241 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:01.241 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:01.241 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:01.241 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:01.242 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:01.242 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.242 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:01.242 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.242 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:01.243 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.243 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:01.243 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:01.243 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.243 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:01.244 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.244 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:01.244 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:01.244 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.244 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.245 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.245 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:01.245 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.245 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:01.246 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:01.246 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.246 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.246 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.246 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:01.247 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:01.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.247 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.247 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:01.248 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.248 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:01.248 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:01.248 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.248 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.249 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.249 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:01.249 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:01.249 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.249 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.250 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:01.250 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:01.250 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:01.250 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:01.250 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:38:01.251 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:01.251 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:01.251 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:01.251 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:38:01.252 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:01.252 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.252 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:01.252 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:01.252 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.253 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:01.253 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:01.253 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:01.253 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:01.253 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:01Z" 2023-10-11T15:38:01.254 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:01.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:01Z" } 2023-10-11T15:38:01.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:38:01.256 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:01.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:38:01.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T15:38:06.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:06.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:06.259 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:06.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:06.316 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:06.346 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:06.345+0000 7f9f29d51780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:06.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:06.346+0000 7f9f29d51780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:06.360 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:06.360 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:06.361 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:06.361 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:06.361 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:06.361 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:06.361 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:06.362 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.362 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:06.362 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:06.362 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:06.363 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:06.363 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:06.363 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:06.363 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:06.364 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:06.364 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:06.364 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:06.364 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:06.364 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:06.365 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.365 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:06.365 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:06.365 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:06.366 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:06.366 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:06.366 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:06.367 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:06.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:06.367 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:06.367 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:06.367 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:06.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.368 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:06.369 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:06.369 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:06.369 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:06.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:06.370 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:06.370 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:06.370 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:06.371 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:06.371 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:06.371 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:06.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:06.372 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:38:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:38:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:06.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:06.374 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:06.374 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:06.374 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.375 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:06.375 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:06.375 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:06.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:06.376 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:06Z" 2023-10-11T15:38:06.376 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:06.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:06Z" } 2023-10-11T15:38:06.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:38:06.378 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:06.379 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:38:06.380 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T15:38:11.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:11.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:11.382 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:11.436 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:11.436 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:11.465 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:11.464+0000 7f83719e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:11.466 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:11.465+0000 7f83719e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:11.479 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:11.479 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:11.479 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:11.480 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:11.480 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:11.480 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:11.480 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:11.481 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:11.482 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:11.483 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:11.484 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:11.485 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:11.486 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:11.486 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:11.486 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:11.486 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:11.487 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:11.487 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:11.487 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:11.487 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:11.488 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:11.488 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:11.488 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:11.488 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:11.489 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.489 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:11.489 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:11.489 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:11.490 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:11.490 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:38:11.490 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:11.490 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:11.490 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:11.491 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:38:11.491 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:11.491 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:11.491 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:11.492 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:11.492 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.492 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:11.492 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:11.492 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:11.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:11.493 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:11Z" 2023-10-11T15:38:11.493 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:11.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:11Z" } 2023-10-11T15:38:11.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:38:11.496 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:11.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:38:11.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T15:38:16.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:16.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:16.499 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:16.553 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:16.554 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:16.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:16.584+0000 7ffa10843780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:16.585 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:16.585+0000 7ffa10843780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:16.599 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:16.599 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:16.600 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:16.600 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:16.600 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:16.600 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:16.600 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:16.601 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.601 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:16.601 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:16.601 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:16.602 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:16.603 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:16.604 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:16.605 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:16.606 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:16.607 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:16.608 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:16.608 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:16.608 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:16.608 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:16.609 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:16.609 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:16.609 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.609 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:16.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:16.610 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:16.610 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:16.610 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:38:16.610 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:16.610 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:16.611 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:16.611 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:38:16.611 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:16.611 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:16.611 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:16.612 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:16.612 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.612 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:16.612 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:16.613 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:16.613 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:16.613 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:16Z" 2023-10-11T15:38:16.613 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:16.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:16Z" } 2023-10-11T15:38:16.615 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:38:16.615 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:16.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:38:16.616 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T15:38:21.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:21.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:21.619 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:21.676 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:21.676 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:21.708 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:21.706+0000 7f30bb2eb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:21.709 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:21.707+0000 7f30bb2eb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:21.722 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:21.722 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:21.723 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:21.723 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:21.723 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:21.723 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:21.724 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:21.724 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.724 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:21.724 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:21.724 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:21.725 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:21.726 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:21.726 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:21.726 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:21.726 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:21.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.727 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:21.727 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:21.727 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:21.728 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:21.728 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:21.728 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:21.728 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:21.728 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:21.729 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:21.729 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:21.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:21.730 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.730 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:21.730 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:21.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:21.731 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:21.731 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:21.731 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:21.731 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:21.732 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:21.732 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:21.732 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.733 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:21.734 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038554.119201_549.1", 2023-10-11T15:38:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:21.735 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T15:38:21.736 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:21.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:21.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:21.736 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:21.737 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.737 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:21.737 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:21.737 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:21.738 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:21.738 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:21Z" 2023-10-11T15:38:21.738 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:21.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038554.119201_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:35:54.119201Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:21Z" } 2023-10-11T15:38:21.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T15:38:21.740 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:21.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T15:38:21.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T15:38:26.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:26.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:38:26.744 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:38:26.800 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:26.800 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:26.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:26.830+0000 7f4c3ef26780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:26.831 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:26.831+0000 7f4c3ef26780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:38:26.845 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:38:26.845 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:38:26.846 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:38:26.846 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:38:26.846 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:38:26.846 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:38:26.847 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:26.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.847 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:38:26.847 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:26.848 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:38:26.848 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:26.848 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:26.848 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:38:26.848 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:26.849 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:38:26.849 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:26.849 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:38:26.849 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:26.849 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:26.850 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.850 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:26.850 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:38:26.850 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:26.850 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:26.851 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:38:26.851 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:26.851 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:26.851 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:26.851 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:38:26.852 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:26.852 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:26.852 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.852 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:26.853 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:38:26.853 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:26.853 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:26.853 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:38:26.853 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:26.854 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:26.854 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:26.854 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:38:26.854 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:26.855 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:26.855 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.855 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:38:26.855 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:38:26.855 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:38:26.856 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:38:26.856 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038681.046367_553.1", 2023-10-11T15:38:26.856 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:38:26.856 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:38:26.857 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:38:26.857 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:38:01.046367Z", 2023-10-11T15:38:26.857 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:38:26.857 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:26.857 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:38:26.858 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:38:26.858 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.858 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:38:26.858 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:38:26.859 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:38:26.859 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:38:26.859 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:38:26Z" 2023-10-11T15:38:26.859 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:38:26.860 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697038681.046367_553.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:38:01.046367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:38:26Z" } 2023-10-11T15:38:26.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038681.046367_553.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:38:01.046367Z', 'realm_epoch': 2}}] 2023-10-11T15:38:26.861 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:26.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T15:38:26.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:38:26.863 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:26.863 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:26.864 DEBUG:boto:Method: HEAD 2023-10-11T15:38:26.864 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:26.864 DEBUG:boto:Data: 2023-10-11T15:38:26.865 DEBUG:boto:Headers: {} 2023-10-11T15:38:26.865 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.865 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.866 DEBUG:boto:Params: {} 2023-10-11T15:38:26.866 DEBUG:boto:Token: None 2023-10-11T15:38:26.867 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/ 2023-10-11T15:38:26.867 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T15:38:26.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T15:38:26.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003600e0738f1cb693-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.871 DEBUG:boto:path=/dghqvd-63/obj11 2023-10-11T15:38:26.872 DEBUG:boto:auth_path=/dghqvd-63/obj11 2023-10-11T15:38:26.872 DEBUG:boto:Method: PUT 2023-10-11T15:38:26.873 DEBUG:boto:Path: /dghqvd-63/obj11 2023-10-11T15:38:26.873 DEBUG:boto:Data: 2023-10-11T15:38:26.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:26.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.874 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.875 DEBUG:boto:Params: {} 2023-10-11T15:38:26.875 DEBUG:boto:Token: None 2023-10-11T15:38:26.876 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/obj11 2023-10-11T15:38:26.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IT9TJnDT9x0eVzK0ZAfrZiTh1iY= 2023-10-11T15:38:26.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IT9TJnDT9x0eVzK0ZAfrZiTh1iY='} 2023-10-11T15:38:26.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af848d6b90b1a4cc-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.883 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:26.884 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:26.884 DEBUG:boto:Method: HEAD 2023-10-11T15:38:26.884 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:26.885 DEBUG:boto:Data: 2023-10-11T15:38:26.885 DEBUG:boto:Headers: {} 2023-10-11T15:38:26.886 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.886 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.887 DEBUG:boto:Params: {} 2023-10-11T15:38:26.887 DEBUG:boto:Token: None 2023-10-11T15:38:26.887 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/ 2023-10-11T15:38:26.888 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T15:38:26.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T15:38:26.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e53ed03168c62ce-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.891 DEBUG:boto:path=/dghqvd-63/obj22 2023-10-11T15:38:26.892 DEBUG:boto:auth_path=/dghqvd-63/obj22 2023-10-11T15:38:26.892 DEBUG:boto:Method: PUT 2023-10-11T15:38:26.892 DEBUG:boto:Path: /dghqvd-63/obj22 2023-10-11T15:38:26.893 DEBUG:boto:Data: 2023-10-11T15:38:26.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:26.894 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.894 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.894 DEBUG:boto:Params: {} 2023-10-11T15:38:26.895 DEBUG:boto:Token: None 2023-10-11T15:38:26.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/obj22 2023-10-11T15:38:26.896 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UH1KqnfCxA9wrSNYKUBQZozqCus= 2023-10-11T15:38:26.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UH1KqnfCxA9wrSNYKUBQZozqCus='} 2023-10-11T15:38:26.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016bc7bb285c87211-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.941 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:26.942 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:26.942 DEBUG:boto:Method: HEAD 2023-10-11T15:38:26.943 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:26.943 DEBUG:boto:Data: 2023-10-11T15:38:26.944 DEBUG:boto:Headers: {} 2023-10-11T15:38:26.944 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.944 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.945 DEBUG:boto:Params: {} 2023-10-11T15:38:26.945 DEBUG:boto:Token: None 2023-10-11T15:38:26.946 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/ 2023-10-11T15:38:26.946 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T15:38:26.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T15:38:26.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dea82871e9517bd4-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.950 DEBUG:boto:path=/dghqvd-63/obj33 2023-10-11T15:38:26.950 DEBUG:boto:auth_path=/dghqvd-63/obj33 2023-10-11T15:38:26.951 DEBUG:boto:Method: PUT 2023-10-11T15:38:26.951 DEBUG:boto:Path: /dghqvd-63/obj33 2023-10-11T15:38:26.952 DEBUG:boto:Data: 2023-10-11T15:38:26.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:26.953 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.953 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.954 DEBUG:boto:Params: {} 2023-10-11T15:38:26.954 DEBUG:boto:Token: None 2023-10-11T15:38:26.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/obj33 2023-10-11T15:38:26.955 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:G1y3yNF0ln7EbL4mB+r6+ZkGaVo= 2023-10-11T15:38:26.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G1y3yNF0ln7EbL4mB+r6+ZkGaVo='} 2023-10-11T15:38:26.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3b8b7f1598833c9-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.962 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:26.962 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:26.963 DEBUG:boto:Method: HEAD 2023-10-11T15:38:26.964 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:26.965 DEBUG:boto:Data: 2023-10-11T15:38:26.966 DEBUG:boto:Headers: {} 2023-10-11T15:38:26.966 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.967 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.968 DEBUG:boto:Params: {} 2023-10-11T15:38:26.968 DEBUG:boto:Token: None 2023-10-11T15:38:26.969 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/ 2023-10-11T15:38:26.970 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T15:38:26.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T15:38:26.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645075452b45e415-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.974 DEBUG:boto:path=/dghqvd-63/obj44 2023-10-11T15:38:26.975 DEBUG:boto:auth_path=/dghqvd-63/obj44 2023-10-11T15:38:26.975 DEBUG:boto:Method: PUT 2023-10-11T15:38:26.976 DEBUG:boto:Path: /dghqvd-63/obj44 2023-10-11T15:38:26.976 DEBUG:boto:Data: 2023-10-11T15:38:26.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:26.977 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.977 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.978 DEBUG:boto:Params: {} 2023-10-11T15:38:26.978 DEBUG:boto:Token: None 2023-10-11T15:38:26.979 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/obj44 2023-10-11T15:38:26.979 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jOwhl5UH1qsdbfVBmPTpUw3KIf8= 2023-10-11T15:38:26.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jOwhl5UH1qsdbfVBmPTpUw3KIf8='} 2023-10-11T15:38:26.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3ec626978b334bd-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.986 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:26.986 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:26.986 DEBUG:boto:Method: HEAD 2023-10-11T15:38:26.987 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:26.987 DEBUG:boto:Data: 2023-10-11T15:38:26.988 DEBUG:boto:Headers: {} 2023-10-11T15:38:26.988 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.988 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.989 DEBUG:boto:Params: {} 2023-10-11T15:38:26.989 DEBUG:boto:Token: None 2023-10-11T15:38:26.990 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/ 2023-10-11T15:38:26.990 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T15:38:26.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T15:38:26.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbbc5c01a78a6c56-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:26.994 DEBUG:boto:path=/dghqvd-63/obj55 2023-10-11T15:38:26.994 DEBUG:boto:auth_path=/dghqvd-63/obj55 2023-10-11T15:38:26.994 DEBUG:boto:Method: PUT 2023-10-11T15:38:26.995 DEBUG:boto:Path: /dghqvd-63/obj55 2023-10-11T15:38:26.995 DEBUG:boto:Data: 2023-10-11T15:38:26.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:26.996 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:26.996 DEBUG:boto:Port: 8000 2023-10-11T15:38:26.997 DEBUG:boto:Params: {} 2023-10-11T15:38:26.997 DEBUG:boto:Token: None 2023-10-11T15:38:26.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:26 GMT /dghqvd-63/obj55 2023-10-11T15:38:26.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ni7ca7b8t1OMiULdccTimySwv3A= 2023-10-11T15:38:26.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ni7ca7b8t1OMiULdccTimySwv3A='} 2023-10-11T15:38:27.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c59af43569dfbce-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.048 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:27.101 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:27.141 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:27.141 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:27.145 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 5 2023-10-11T15:38:27.181 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.181 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.182 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.182 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.183 DEBUG:boto:Data: 2023-10-11T15:38:27.183 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.184 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.184 DEBUG:boto:Params: {} 2023-10-11T15:38:27.185 DEBUG:boto:Token: None 2023-10-11T15:38:27.185 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d393f4f6df229de9-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.189 DEBUG:boto:path=/dghqvd-63/obj17 2023-10-11T15:38:27.189 DEBUG:boto:auth_path=/dghqvd-63/obj17 2023-10-11T15:38:27.190 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.190 DEBUG:boto:Path: /dghqvd-63/obj17 2023-10-11T15:38:27.191 DEBUG:boto:Data: 2023-10-11T15:38:27.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.191 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.192 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.192 DEBUG:boto:Params: {} 2023-10-11T15:38:27.193 DEBUG:boto:Token: None 2023-10-11T15:38:27.193 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj17 2023-10-11T15:38:27.194 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WWHXk42balAFDD4A1KLvpN/Dh5U= 2023-10-11T15:38:27.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WWHXk42balAFDD4A1KLvpN/Dh5U='} 2023-10-11T15:38:27.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000107313ff247f973b-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.200 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.200 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.201 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.201 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.201 DEBUG:boto:Data: 2023-10-11T15:38:27.202 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.203 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.203 DEBUG:boto:Params: {} 2023-10-11T15:38:27.203 DEBUG:boto:Token: None 2023-10-11T15:38:27.204 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.204 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069db2224bbd240df-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.208 DEBUG:boto:path=/dghqvd-63/obj34 2023-10-11T15:38:27.208 DEBUG:boto:auth_path=/dghqvd-63/obj34 2023-10-11T15:38:27.208 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.209 DEBUG:boto:Path: /dghqvd-63/obj34 2023-10-11T15:38:27.209 DEBUG:boto:Data: 2023-10-11T15:38:27.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.210 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.210 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.211 DEBUG:boto:Params: {} 2023-10-11T15:38:27.211 DEBUG:boto:Token: None 2023-10-11T15:38:27.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj34 2023-10-11T15:38:27.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8A5mIETNeYQ72zeX8Jm1ASkBJFQ= 2023-10-11T15:38:27.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8A5mIETNeYQ72zeX8Jm1ASkBJFQ='} 2023-10-11T15:38:27.217 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d664a0757739c79-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.218 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.218 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.218 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.219 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.219 DEBUG:boto:Data: 2023-10-11T15:38:27.220 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.220 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.220 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.221 DEBUG:boto:Params: {} 2023-10-11T15:38:27.221 DEBUG:boto:Token: None 2023-10-11T15:38:27.222 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.222 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030addc0609d960f6-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.226 DEBUG:boto:path=/dghqvd-63/obj51 2023-10-11T15:38:27.226 DEBUG:boto:auth_path=/dghqvd-63/obj51 2023-10-11T15:38:27.226 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.227 DEBUG:boto:Path: /dghqvd-63/obj51 2023-10-11T15:38:27.227 DEBUG:boto:Data: 2023-10-11T15:38:27.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.228 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.228 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.229 DEBUG:boto:Params: {} 2023-10-11T15:38:27.229 DEBUG:boto:Token: None 2023-10-11T15:38:27.230 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj51 2023-10-11T15:38:27.230 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VUAUtMZm1CDdi51Jm6yd2AnbsPM= 2023-10-11T15:38:27.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VUAUtMZm1CDdi51Jm6yd2AnbsPM='} 2023-10-11T15:38:27.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57b358b620758b3-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.275 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.276 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.276 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.277 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.277 DEBUG:boto:Data: 2023-10-11T15:38:27.277 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.278 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.278 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.279 DEBUG:boto:Params: {} 2023-10-11T15:38:27.279 DEBUG:boto:Token: None 2023-10-11T15:38:27.280 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.280 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028703e5120562330-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.284 DEBUG:boto:path=/dghqvd-63/obj68 2023-10-11T15:38:27.284 DEBUG:boto:auth_path=/dghqvd-63/obj68 2023-10-11T15:38:27.285 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.285 DEBUG:boto:Path: /dghqvd-63/obj68 2023-10-11T15:38:27.286 DEBUG:boto:Data: 2023-10-11T15:38:27.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.287 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.287 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.287 DEBUG:boto:Params: {} 2023-10-11T15:38:27.288 DEBUG:boto:Token: None 2023-10-11T15:38:27.288 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj68 2023-10-11T15:38:27.289 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:25JgIO3LgRnTaYuCfPeGVE/+6oo= 2023-10-11T15:38:27.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:25JgIO3LgRnTaYuCfPeGVE/+6oo='} 2023-10-11T15:38:27.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005969c0a876294c84-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.295 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.295 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.296 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.296 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.297 DEBUG:boto:Data: 2023-10-11T15:38:27.297 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.297 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.298 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.298 DEBUG:boto:Params: {} 2023-10-11T15:38:27.299 DEBUG:boto:Token: None 2023-10-11T15:38:27.299 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7a1f571556c7ecc-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.303 DEBUG:boto:path=/dghqvd-63/obj85 2023-10-11T15:38:27.303 DEBUG:boto:auth_path=/dghqvd-63/obj85 2023-10-11T15:38:27.304 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.304 DEBUG:boto:Path: /dghqvd-63/obj85 2023-10-11T15:38:27.304 DEBUG:boto:Data: 2023-10-11T15:38:27.305 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.305 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.306 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.306 DEBUG:boto:Params: {} 2023-10-11T15:38:27.307 DEBUG:boto:Token: None 2023-10-11T15:38:27.307 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj85 2023-10-11T15:38:27.308 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:beK/+Uj9ynvf8gAHkq/WgfxV514= 2023-10-11T15:38:27.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:beK/+Uj9ynvf8gAHkq/WgfxV514='} 2023-10-11T15:38:27.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003070663804cce92b-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.352 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:27.405 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:27.405 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:27.447 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:27.448 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:27.454 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 10 2023-10-11T15:38:27.496 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.497 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.497 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.498 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.498 DEBUG:boto:Data: 2023-10-11T15:38:27.499 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.499 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.499 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.500 DEBUG:boto:Params: {} 2023-10-11T15:38:27.500 DEBUG:boto:Token: None 2023-10-11T15:38:27.501 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.501 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023a48f95830ba992-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.505 DEBUG:boto:path=/dghqvd-63/obj19 2023-10-11T15:38:27.506 DEBUG:boto:auth_path=/dghqvd-63/obj19 2023-10-11T15:38:27.506 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.506 DEBUG:boto:Path: /dghqvd-63/obj19 2023-10-11T15:38:27.507 DEBUG:boto:Data: 2023-10-11T15:38:27.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.508 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.508 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.508 DEBUG:boto:Params: {} 2023-10-11T15:38:27.509 DEBUG:boto:Token: None 2023-10-11T15:38:27.509 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj19 2023-10-11T15:38:27.510 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:hvWIJ1XF54Ic/B6JtjfllyVHu8s= 2023-10-11T15:38:27.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hvWIJ1XF54Ic/B6JtjfllyVHu8s='} 2023-10-11T15:38:27.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd7736b4657790eb-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.516 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.516 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.517 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.517 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.518 DEBUG:boto:Data: 2023-10-11T15:38:27.518 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.518 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.519 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.519 DEBUG:boto:Params: {} 2023-10-11T15:38:27.520 DEBUG:boto:Token: None 2023-10-11T15:38:27.520 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.520 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3596654f737c850-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.524 DEBUG:boto:path=/dghqvd-63/obj38 2023-10-11T15:38:27.525 DEBUG:boto:auth_path=/dghqvd-63/obj38 2023-10-11T15:38:27.525 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.526 DEBUG:boto:Path: /dghqvd-63/obj38 2023-10-11T15:38:27.526 DEBUG:boto:Data: 2023-10-11T15:38:27.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.527 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.527 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.528 DEBUG:boto:Params: {} 2023-10-11T15:38:27.528 DEBUG:boto:Token: None 2023-10-11T15:38:27.529 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj38 2023-10-11T15:38:27.529 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VdczRJrbl7TB1Py8t23PIocqmtQ= 2023-10-11T15:38:27.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VdczRJrbl7TB1Py8t23PIocqmtQ='} 2023-10-11T15:38:27.533 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000088e8e3c6bfe5d6a-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.534 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.534 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.535 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.535 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.536 DEBUG:boto:Data: 2023-10-11T15:38:27.536 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.536 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.537 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.537 DEBUG:boto:Params: {} 2023-10-11T15:38:27.538 DEBUG:boto:Token: None 2023-10-11T15:38:27.538 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068bfe75efae03226-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.542 DEBUG:boto:path=/dghqvd-63/obj57 2023-10-11T15:38:27.543 DEBUG:boto:auth_path=/dghqvd-63/obj57 2023-10-11T15:38:27.543 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.544 DEBUG:boto:Path: /dghqvd-63/obj57 2023-10-11T15:38:27.544 DEBUG:boto:Data: 2023-10-11T15:38:27.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.545 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.545 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.546 DEBUG:boto:Params: {} 2023-10-11T15:38:27.546 DEBUG:boto:Token: None 2023-10-11T15:38:27.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj57 2023-10-11T15:38:27.547 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:J4OK1NdR2vblypj+1486IbaesT4= 2023-10-11T15:38:27.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J4OK1NdR2vblypj+1486IbaesT4='} 2023-10-11T15:38:27.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090ae412bfdbabaaa-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.552 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.552 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.553 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.553 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.553 DEBUG:boto:Data: 2023-10-11T15:38:27.554 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.554 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.555 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.555 DEBUG:boto:Params: {} 2023-10-11T15:38:27.556 DEBUG:boto:Token: None 2023-10-11T15:38:27.556 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.556 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014503f59ef9db480-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.562 DEBUG:boto:path=/dghqvd-63/obj76 2023-10-11T15:38:27.562 DEBUG:boto:auth_path=/dghqvd-63/obj76 2023-10-11T15:38:27.563 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.563 DEBUG:boto:Path: /dghqvd-63/obj76 2023-10-11T15:38:27.563 DEBUG:boto:Data: 2023-10-11T15:38:27.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.565 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.565 DEBUG:boto:Params: {} 2023-10-11T15:38:27.566 DEBUG:boto:Token: None 2023-10-11T15:38:27.566 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj76 2023-10-11T15:38:27.567 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vUmYAW3h6+/LG+v+78O71+vL21o= 2023-10-11T15:38:27.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vUmYAW3h6+/LG+v+78O71+vL21o='} 2023-10-11T15:38:27.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003dd4fbeddc7ce2d-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.615 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.616 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.616 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.617 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.617 DEBUG:boto:Data: 2023-10-11T15:38:27.617 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.618 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.619 DEBUG:boto:Params: {} 2023-10-11T15:38:27.619 DEBUG:boto:Token: None 2023-10-11T15:38:27.620 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007abf1c50c6bd6890-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.624 DEBUG:boto:path=/dghqvd-63/obj95 2023-10-11T15:38:27.625 DEBUG:boto:auth_path=/dghqvd-63/obj95 2023-10-11T15:38:27.625 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.626 DEBUG:boto:Path: /dghqvd-63/obj95 2023-10-11T15:38:27.626 DEBUG:boto:Data: 2023-10-11T15:38:27.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.627 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.627 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.628 DEBUG:boto:Params: {} 2023-10-11T15:38:27.628 DEBUG:boto:Token: None 2023-10-11T15:38:27.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj95 2023-10-11T15:38:27.629 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:So76MOqMB8NJCmtK4rm7ALVs5l0= 2023-10-11T15:38:27.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:So76MOqMB8NJCmtK4rm7ALVs5l0='} 2023-10-11T15:38:27.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001294bc477b5757e-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:27.635 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:27.688 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:27.732 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:27.733 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:27.739 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 15 2023-10-11T15:38:27.787 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.788 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.788 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.789 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.789 DEBUG:boto:Data: 2023-10-11T15:38:27.789 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.790 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.790 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.791 DEBUG:boto:Params: {} 2023-10-11T15:38:27.791 DEBUG:boto:Token: None 2023-10-11T15:38:27.792 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.792 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d08e04b8f593068-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.796 DEBUG:boto:path=/dghqvd-63/obj23 2023-10-11T15:38:27.796 DEBUG:boto:auth_path=/dghqvd-63/obj23 2023-10-11T15:38:27.797 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.797 DEBUG:boto:Path: /dghqvd-63/obj23 2023-10-11T15:38:27.798 DEBUG:boto:Data: 2023-10-11T15:38:27.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.799 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.799 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.799 DEBUG:boto:Params: {} 2023-10-11T15:38:27.800 DEBUG:boto:Token: None 2023-10-11T15:38:27.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj23 2023-10-11T15:38:27.801 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vztRQJ5chdFczeJYeVchMG/gCqQ= 2023-10-11T15:38:27.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vztRQJ5chdFczeJYeVchMG/gCqQ='} 2023-10-11T15:38:27.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a6ed00dd87faacf-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.851 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.852 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.852 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.853 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.853 DEBUG:boto:Data: 2023-10-11T15:38:27.853 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.854 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.855 DEBUG:boto:Params: {} 2023-10-11T15:38:27.855 DEBUG:boto:Token: None 2023-10-11T15:38:27.856 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a279c8934b7f6a-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.860 DEBUG:boto:path=/dghqvd-63/obj46 2023-10-11T15:38:27.860 DEBUG:boto:auth_path=/dghqvd-63/obj46 2023-10-11T15:38:27.861 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.861 DEBUG:boto:Path: /dghqvd-63/obj46 2023-10-11T15:38:27.862 DEBUG:boto:Data: 2023-10-11T15:38:27.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.863 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.863 DEBUG:boto:Params: {} 2023-10-11T15:38:27.864 DEBUG:boto:Token: None 2023-10-11T15:38:27.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj46 2023-10-11T15:38:27.865 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I47fi8iWUxnZivO3wnM7/OdAqgw= 2023-10-11T15:38:27.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I47fi8iWUxnZivO3wnM7/OdAqgw='} 2023-10-11T15:38:27.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50bbbe9b68f847a-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.871 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.871 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.872 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.872 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.873 DEBUG:boto:Data: 2023-10-11T15:38:27.873 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.874 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.874 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.874 DEBUG:boto:Params: {} 2023-10-11T15:38:27.875 DEBUG:boto:Token: None 2023-10-11T15:38:27.875 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.876 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fff960f79ab8c863-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.880 DEBUG:boto:path=/dghqvd-63/obj69 2023-10-11T15:38:27.880 DEBUG:boto:auth_path=/dghqvd-63/obj69 2023-10-11T15:38:27.881 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.881 DEBUG:boto:Path: /dghqvd-63/obj69 2023-10-11T15:38:27.881 DEBUG:boto:Data: 2023-10-11T15:38:27.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.882 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.883 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.883 DEBUG:boto:Params: {} 2023-10-11T15:38:27.884 DEBUG:boto:Token: None 2023-10-11T15:38:27.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj69 2023-10-11T15:38:27.885 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bbRZiH2psJQO2H9csiVQMccRr3s= 2023-10-11T15:38:27.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bbRZiH2psJQO2H9csiVQMccRr3s='} 2023-10-11T15:38:27.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f89fbb719b1b1477-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.932 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.932 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.933 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.933 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.933 DEBUG:boto:Data: 2023-10-11T15:38:27.934 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.934 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.935 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.935 DEBUG:boto:Params: {} 2023-10-11T15:38:27.935 DEBUG:boto:Token: None 2023-10-11T15:38:27.936 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.936 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea2662a7d397fd3a-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.940 DEBUG:boto:path=/dghqvd-63/obj92 2023-10-11T15:38:27.941 DEBUG:boto:auth_path=/dghqvd-63/obj92 2023-10-11T15:38:27.941 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.942 DEBUG:boto:Path: /dghqvd-63/obj92 2023-10-11T15:38:27.942 DEBUG:boto:Data: 2023-10-11T15:38:27.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.943 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.943 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.944 DEBUG:boto:Params: {} 2023-10-11T15:38:27.944 DEBUG:boto:Token: None 2023-10-11T15:38:27.945 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj92 2023-10-11T15:38:27.945 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qtX+GtlKd88MrVGeIwg0FINuGyA= 2023-10-11T15:38:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qtX+GtlKd88MrVGeIwg0FINuGyA='} 2023-10-11T15:38:27.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a45807860d4e747f-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.952 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:27.952 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:27.952 DEBUG:boto:Method: HEAD 2023-10-11T15:38:27.953 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:27.953 DEBUG:boto:Data: 2023-10-11T15:38:27.954 DEBUG:boto:Headers: {} 2023-10-11T15:38:27.954 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.954 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.955 DEBUG:boto:Params: {} 2023-10-11T15:38:27.955 DEBUG:boto:Token: None 2023-10-11T15:38:27.956 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/ 2023-10-11T15:38:27.956 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T15:38:27.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T15:38:27.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ece60d04333717d-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:27.960 DEBUG:boto:path=/dghqvd-63/obj115 2023-10-11T15:38:27.960 DEBUG:boto:auth_path=/dghqvd-63/obj115 2023-10-11T15:38:27.961 DEBUG:boto:Method: PUT 2023-10-11T15:38:27.961 DEBUG:boto:Path: /dghqvd-63/obj115 2023-10-11T15:38:27.962 DEBUG:boto:Data: 2023-10-11T15:38:27.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:27.962 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:27.963 DEBUG:boto:Port: 8000 2023-10-11T15:38:27.963 DEBUG:boto:Params: {} 2023-10-11T15:38:27.964 DEBUG:boto:Token: None 2023-10-11T15:38:27.964 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:27 GMT /dghqvd-63/obj115 2023-10-11T15:38:27.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qEetUc9cN7GMsamiwkL6ri8WB/A= 2023-10-11T15:38:27.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qEetUc9cN7GMsamiwkL6ri8WB/A='} 2023-10-11T15:38:28.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea2177d4ce022337-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.012 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:28.065 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:28.065 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:28.111 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:28.111 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:28.119 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 20 2023-10-11T15:38:28.169 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.169 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.170 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.170 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.171 DEBUG:boto:Data: 2023-10-11T15:38:28.171 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.173 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.174 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.174 DEBUG:boto:Params: {} 2023-10-11T15:38:28.174 DEBUG:boto:Token: None 2023-10-11T15:38:28.175 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.175 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a94b01c45198897e-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.180 DEBUG:boto:path=/dghqvd-63/obj29 2023-10-11T15:38:28.180 DEBUG:boto:auth_path=/dghqvd-63/obj29 2023-10-11T15:38:28.181 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.181 DEBUG:boto:Path: /dghqvd-63/obj29 2023-10-11T15:38:28.182 DEBUG:boto:Data: 2023-10-11T15:38:28.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.182 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.183 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.183 DEBUG:boto:Params: {} 2023-10-11T15:38:28.184 DEBUG:boto:Token: None 2023-10-11T15:38:28.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj29 2023-10-11T15:38:28.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I7DbNYblV8CowsFBOu1OQy9eJ04= 2023-10-11T15:38:28.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I7DbNYblV8CowsFBOu1OQy9eJ04='} 2023-10-11T15:38:28.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f154eca03afed3-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.191 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.191 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.192 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.192 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.193 DEBUG:boto:Data: 2023-10-11T15:38:28.193 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.194 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.194 DEBUG:boto:Params: {} 2023-10-11T15:38:28.195 DEBUG:boto:Token: None 2023-10-11T15:38:28.195 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092c6d2b6ab4eae59-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.200 DEBUG:boto:path=/dghqvd-63/obj58 2023-10-11T15:38:28.200 DEBUG:boto:auth_path=/dghqvd-63/obj58 2023-10-11T15:38:28.200 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.201 DEBUG:boto:Path: /dghqvd-63/obj58 2023-10-11T15:38:28.201 DEBUG:boto:Data: 2023-10-11T15:38:28.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.202 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.203 DEBUG:boto:Params: {} 2023-10-11T15:38:28.203 DEBUG:boto:Token: None 2023-10-11T15:38:28.204 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj58 2023-10-11T15:38:28.204 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/mL3gq6TkR6BYpF76hfMkjC1Nyg= 2023-10-11T15:38:28.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/mL3gq6TkR6BYpF76hfMkjC1Nyg='} 2023-10-11T15:38:28.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c27f40055698aa5-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.248 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.248 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.248 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.249 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.250 DEBUG:boto:Data: 2023-10-11T15:38:28.250 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.251 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.251 DEBUG:boto:Params: {} 2023-10-11T15:38:28.252 DEBUG:boto:Token: None 2023-10-11T15:38:28.252 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0bf01ea5535565d-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.256 DEBUG:boto:path=/dghqvd-63/obj87 2023-10-11T15:38:28.257 DEBUG:boto:auth_path=/dghqvd-63/obj87 2023-10-11T15:38:28.257 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.258 DEBUG:boto:Path: /dghqvd-63/obj87 2023-10-11T15:38:28.258 DEBUG:boto:Data: 2023-10-11T15:38:28.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.260 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.260 DEBUG:boto:Params: {} 2023-10-11T15:38:28.261 DEBUG:boto:Token: None 2023-10-11T15:38:28.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj87 2023-10-11T15:38:28.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LB2vzqNkyi8AGB0ybbPO4vUVp00= 2023-10-11T15:38:28.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LB2vzqNkyi8AGB0ybbPO4vUVp00='} 2023-10-11T15:38:28.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d216aff0569ca3d-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.268 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.268 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.268 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.269 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.269 DEBUG:boto:Data: 2023-10-11T15:38:28.270 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.270 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.270 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.271 DEBUG:boto:Params: {} 2023-10-11T15:38:28.271 DEBUG:boto:Token: None 2023-10-11T15:38:28.272 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.272 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcfe9d56ec20d5ca-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.276 DEBUG:boto:path=/dghqvd-63/obj116 2023-10-11T15:38:28.277 DEBUG:boto:auth_path=/dghqvd-63/obj116 2023-10-11T15:38:28.277 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.277 DEBUG:boto:Path: /dghqvd-63/obj116 2023-10-11T15:38:28.278 DEBUG:boto:Data: 2023-10-11T15:38:28.278 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.279 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.279 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.279 DEBUG:boto:Params: {} 2023-10-11T15:38:28.280 DEBUG:boto:Token: None 2023-10-11T15:38:28.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj116 2023-10-11T15:38:28.281 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n0EX3NuwLwwhA96sgxk+IKgz0Ac= 2023-10-11T15:38:28.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0EX3NuwLwwhA96sgxk+IKgz0Ac='} 2023-10-11T15:38:28.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092cd856f9ab302da-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.327 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.328 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.328 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.329 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.329 DEBUG:boto:Data: 2023-10-11T15:38:28.329 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.330 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.331 DEBUG:boto:Params: {} 2023-10-11T15:38:28.331 DEBUG:boto:Token: None 2023-10-11T15:38:28.332 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afca4652d2778e7f-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.336 DEBUG:boto:path=/dghqvd-63/obj145 2023-10-11T15:38:28.337 DEBUG:boto:auth_path=/dghqvd-63/obj145 2023-10-11T15:38:28.337 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.338 DEBUG:boto:Path: /dghqvd-63/obj145 2023-10-11T15:38:28.338 DEBUG:boto:Data: 2023-10-11T15:38:28.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.339 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.341 DEBUG:boto:Params: {} 2023-10-11T15:38:28.341 DEBUG:boto:Token: None 2023-10-11T15:38:28.341 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj145 2023-10-11T15:38:28.342 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:mo+li2p2QZMOaqHpQi1TyTcx+bM= 2023-10-11T15:38:28.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mo+li2p2QZMOaqHpQi1TyTcx+bM='} 2023-10-11T15:38:28.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e37e930a07d85a60-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.349 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:28.404 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:28.454 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:28.454 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:28.464 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 25 2023-10-11T15:38:28.525 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.526 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.526 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.527 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.527 DEBUG:boto:Data: 2023-10-11T15:38:28.528 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.528 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.528 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.529 DEBUG:boto:Params: {} 2023-10-11T15:38:28.529 DEBUG:boto:Token: None 2023-10-11T15:38:28.530 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.530 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b6d0eb35c0dd0d2-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.535 DEBUG:boto:path=/dghqvd-63/obj31 2023-10-11T15:38:28.535 DEBUG:boto:auth_path=/dghqvd-63/obj31 2023-10-11T15:38:28.536 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.536 DEBUG:boto:Path: /dghqvd-63/obj31 2023-10-11T15:38:28.536 DEBUG:boto:Data: 2023-10-11T15:38:28.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.537 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.538 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.538 DEBUG:boto:Params: {} 2023-10-11T15:38:28.538 DEBUG:boto:Token: None 2023-10-11T15:38:28.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj31 2023-10-11T15:38:28.539 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tXOAnCumLi/n+Y6sm3Ul2Ghsfyk= 2023-10-11T15:38:28.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tXOAnCumLi/n+Y6sm3Ul2Ghsfyk='} 2023-10-11T15:38:28.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b03f6e4767e0190-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.546 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.547 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.547 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.547 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.548 DEBUG:boto:Data: 2023-10-11T15:38:28.548 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.549 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.549 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.550 DEBUG:boto:Params: {} 2023-10-11T15:38:28.551 DEBUG:boto:Token: None 2023-10-11T15:38:28.551 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.552 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002425b26e3d34862b-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.556 DEBUG:boto:path=/dghqvd-63/obj62 2023-10-11T15:38:28.557 DEBUG:boto:auth_path=/dghqvd-63/obj62 2023-10-11T15:38:28.557 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.557 DEBUG:boto:Path: /dghqvd-63/obj62 2023-10-11T15:38:28.558 DEBUG:boto:Data: 2023-10-11T15:38:28.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.559 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.560 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.561 DEBUG:boto:Params: {} 2023-10-11T15:38:28.561 DEBUG:boto:Token: None 2023-10-11T15:38:28.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj62 2023-10-11T15:38:28.562 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ji51MCrWhhD3IqXBmMyTmus1Yac= 2023-10-11T15:38:28.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ji51MCrWhhD3IqXBmMyTmus1Yac='} 2023-10-11T15:38:28.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075884ac137a8613c-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.607 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.608 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.608 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.608 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.609 DEBUG:boto:Data: 2023-10-11T15:38:28.609 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.610 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.610 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.611 DEBUG:boto:Params: {} 2023-10-11T15:38:28.611 DEBUG:boto:Token: None 2023-10-11T15:38:28.612 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.612 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009467bff4fd3a3f96-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.617 DEBUG:boto:path=/dghqvd-63/obj93 2023-10-11T15:38:28.617 DEBUG:boto:auth_path=/dghqvd-63/obj93 2023-10-11T15:38:28.618 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.618 DEBUG:boto:Path: /dghqvd-63/obj93 2023-10-11T15:38:28.619 DEBUG:boto:Data: 2023-10-11T15:38:28.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.619 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.620 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.620 DEBUG:boto:Params: {} 2023-10-11T15:38:28.621 DEBUG:boto:Token: None 2023-10-11T15:38:28.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj93 2023-10-11T15:38:28.622 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8dQNKzUFtNEget2vy5FhQOS/KsA= 2023-10-11T15:38:28.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8dQNKzUFtNEget2vy5FhQOS/KsA='} 2023-10-11T15:38:28.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053697fe018eff2e7-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.629 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.629 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.630 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.631 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.631 DEBUG:boto:Data: 2023-10-11T15:38:28.631 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.632 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.632 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.634 DEBUG:boto:Params: {} 2023-10-11T15:38:28.635 DEBUG:boto:Token: None 2023-10-11T15:38:28.635 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.636 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918ab5480cde9fa4-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.640 DEBUG:boto:path=/dghqvd-63/obj124 2023-10-11T15:38:28.640 DEBUG:boto:auth_path=/dghqvd-63/obj124 2023-10-11T15:38:28.641 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.641 DEBUG:boto:Path: /dghqvd-63/obj124 2023-10-11T15:38:28.642 DEBUG:boto:Data: 2023-10-11T15:38:28.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.642 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.643 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.643 DEBUG:boto:Params: {} 2023-10-11T15:38:28.644 DEBUG:boto:Token: None 2023-10-11T15:38:28.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj124 2023-10-11T15:38:28.645 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4jb/oWx7/bG/Gt+85JCwM5qbn/4= 2023-10-11T15:38:28.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4jb/oWx7/bG/Gt+85JCwM5qbn/4='} 2023-10-11T15:38:28.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c7505711b9f1356-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.691 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.692 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.692 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.693 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.693 DEBUG:boto:Data: 2023-10-11T15:38:28.693 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.694 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.695 DEBUG:boto:Params: {} 2023-10-11T15:38:28.695 DEBUG:boto:Token: None 2023-10-11T15:38:28.696 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.697 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050867af917ce502b-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.701 DEBUG:boto:path=/dghqvd-63/obj155 2023-10-11T15:38:28.702 DEBUG:boto:auth_path=/dghqvd-63/obj155 2023-10-11T15:38:28.702 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.703 DEBUG:boto:Path: /dghqvd-63/obj155 2023-10-11T15:38:28.703 DEBUG:boto:Data: 2023-10-11T15:38:28.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.704 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.705 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.705 DEBUG:boto:Params: {} 2023-10-11T15:38:28.706 DEBUG:boto:Token: None 2023-10-11T15:38:28.707 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj155 2023-10-11T15:38:28.708 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ep/OKTojQ810olyJ7nY+mp5Me+I= 2023-10-11T15:38:28.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ep/OKTojQ810olyJ7nY+mp5Me+I='} 2023-10-11T15:38:28.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000895b49986b0ea804-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:38:28.717 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:38:28.771 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:28.772 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:28.821 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:38:28.822 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-63 2023-10-11T15:38:28.833 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 30 2023-10-11T15:38:28.893 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.894 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.894 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.895 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.895 DEBUG:boto:Data: 2023-10-11T15:38:28.895 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.896 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.896 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.896 DEBUG:boto:Params: {} 2023-10-11T15:38:28.897 DEBUG:boto:Token: None 2023-10-11T15:38:28.897 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.898 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a97f0036ff0d22-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.903 DEBUG:boto:path=/dghqvd-63/obj37 2023-10-11T15:38:28.903 DEBUG:boto:auth_path=/dghqvd-63/obj37 2023-10-11T15:38:28.904 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.904 DEBUG:boto:Path: /dghqvd-63/obj37 2023-10-11T15:38:28.904 DEBUG:boto:Data: 2023-10-11T15:38:28.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.905 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.906 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.906 DEBUG:boto:Params: {} 2023-10-11T15:38:28.907 DEBUG:boto:Token: None 2023-10-11T15:38:28.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj37 2023-10-11T15:38:28.907 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LjrbKb7oNBywgKEedf3de5N2Kcw= 2023-10-11T15:38:28.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LjrbKb7oNBywgKEedf3de5N2Kcw='} 2023-10-11T15:38:28.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022314a689b53a213-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.913 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.914 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.914 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.915 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.915 DEBUG:boto:Data: 2023-10-11T15:38:28.915 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.916 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.916 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.917 DEBUG:boto:Params: {} 2023-10-11T15:38:28.917 DEBUG:boto:Token: None 2023-10-11T15:38:28.918 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.918 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ddefc8c3db311f-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.923 DEBUG:boto:path=/dghqvd-63/obj74 2023-10-11T15:38:28.923 DEBUG:boto:auth_path=/dghqvd-63/obj74 2023-10-11T15:38:28.924 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.924 DEBUG:boto:Path: /dghqvd-63/obj74 2023-10-11T15:38:28.925 DEBUG:boto:Data: 2023-10-11T15:38:28.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.926 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.926 DEBUG:boto:Params: {} 2023-10-11T15:38:28.927 DEBUG:boto:Token: None 2023-10-11T15:38:28.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj74 2023-10-11T15:38:28.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:q8cPyg1X605WmH0eXiHHiQpNN0Y= 2023-10-11T15:38:28.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q8cPyg1X605WmH0eXiHHiQpNN0Y='} 2023-10-11T15:38:28.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c13bbe7aedc1159a-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.935 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.935 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.935 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.936 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.936 DEBUG:boto:Data: 2023-10-11T15:38:28.937 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.937 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.938 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.938 DEBUG:boto:Params: {} 2023-10-11T15:38:28.938 DEBUG:boto:Token: None 2023-10-11T15:38:28.939 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/ 2023-10-11T15:38:28.939 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T15:38:28.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T15:38:28.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0fd753c3c4449aa-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.944 DEBUG:boto:path=/dghqvd-63/obj111 2023-10-11T15:38:28.945 DEBUG:boto:auth_path=/dghqvd-63/obj111 2023-10-11T15:38:28.945 DEBUG:boto:Method: PUT 2023-10-11T15:38:28.945 DEBUG:boto:Path: /dghqvd-63/obj111 2023-10-11T15:38:28.946 DEBUG:boto:Data: 2023-10-11T15:38:28.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:28.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.947 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.947 DEBUG:boto:Params: {} 2023-10-11T15:38:28.948 DEBUG:boto:Token: None 2023-10-11T15:38:28.948 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:28 GMT /dghqvd-63/obj111 2023-10-11T15:38:28.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Ao7a5MA4IUnOTk/y/Q4XnncI3A= 2023-10-11T15:38:28.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Ao7a5MA4IUnOTk/y/Q4XnncI3A='} 2023-10-11T15:38:28.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfad7bc1ce7d7197-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:28.995 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:28.996 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:28.996 DEBUG:boto:Method: HEAD 2023-10-11T15:38:28.997 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:28.997 DEBUG:boto:Data: 2023-10-11T15:38:28.998 DEBUG:boto:Headers: {} 2023-10-11T15:38:28.998 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:28.999 DEBUG:boto:Port: 8000 2023-10-11T15:38:28.999 DEBUG:boto:Params: {} 2023-10-11T15:38:29.000 DEBUG:boto:Token: None 2023-10-11T15:38:29.000 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:29 GMT /dghqvd-63/ 2023-10-11T15:38:29.001 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU= 2023-10-11T15:38:29.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU=', 'Content-Length': '0'} 2023-10-11T15:38:29.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cc5ca6207bf12ad-006526c175-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:29.006 DEBUG:boto:path=/dghqvd-63/obj148 2023-10-11T15:38:29.006 DEBUG:boto:auth_path=/dghqvd-63/obj148 2023-10-11T15:38:29.007 DEBUG:boto:Method: PUT 2023-10-11T15:38:29.007 DEBUG:boto:Path: /dghqvd-63/obj148 2023-10-11T15:38:29.008 DEBUG:boto:Data: 2023-10-11T15:38:29.008 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:29.009 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:29.009 DEBUG:boto:Port: 8000 2023-10-11T15:38:29.009 DEBUG:boto:Params: {} 2023-10-11T15:38:29.010 DEBUG:boto:Token: None 2023-10-11T15:38:29.010 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:29 GMT /dghqvd-63/obj148 2023-10-11T15:38:29.011 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tQnQwbiH4cC0Iv8QIjB25UhSWxo= 2023-10-11T15:38:29.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tQnQwbiH4cC0Iv8QIjB25UhSWxo='} 2023-10-11T15:38:29.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d057ec3e37516a1d-006526c175-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:29.017 DEBUG:boto:path=/dghqvd-63/ 2023-10-11T15:38:29.017 DEBUG:boto:auth_path=/dghqvd-63/ 2023-10-11T15:38:29.018 DEBUG:boto:Method: HEAD 2023-10-11T15:38:29.018 DEBUG:boto:Path: /dghqvd-63/ 2023-10-11T15:38:29.019 DEBUG:boto:Data: 2023-10-11T15:38:29.019 DEBUG:boto:Headers: {} 2023-10-11T15:38:29.020 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:29.020 DEBUG:boto:Port: 8000 2023-10-11T15:38:29.020 DEBUG:boto:Params: {} 2023-10-11T15:38:29.021 DEBUG:boto:Token: None 2023-10-11T15:38:29.021 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:38:29 GMT /dghqvd-63/ 2023-10-11T15:38:29.022 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU= 2023-10-11T15:38:29.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU=', 'Content-Length': '0'} 2023-10-11T15:38:29.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095003e3331e8b9c4-006526c175-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:29.026 DEBUG:boto:path=/dghqvd-63/obj185 2023-10-11T15:38:29.027 DEBUG:boto:auth_path=/dghqvd-63/obj185 2023-10-11T15:38:29.027 DEBUG:boto:Method: PUT 2023-10-11T15:38:29.027 DEBUG:boto:Path: /dghqvd-63/obj185 2023-10-11T15:38:29.028 DEBUG:boto:Data: 2023-10-11T15:38:29.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:38:29.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:38:29.029 DEBUG:boto:Port: 8000 2023-10-11T15:38:29.029 DEBUG:boto:Params: {} 2023-10-11T15:38:29.030 DEBUG:boto:Token: None 2023-10-11T15:38:29.030 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:38:29 GMT /dghqvd-63/obj185 2023-10-11T15:38:29.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a+P7XsISuZYa6fXgAoy/sImlwfY= 2023-10-11T15:38:29.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a+P7XsISuZYa6fXgAoy/sImlwfY='} 2023-10-11T15:38:29.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc1e45979e605836-006526c175-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:38:29.037 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-63 2023-10-11T15:38:29.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:38:29.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:38:29.039 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:38:29.097 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:29.097 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:29.133 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:29.132+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T15:38:34.134 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:34.132+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T15:38:39.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:39.133+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T15:38:44.135 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:44.133+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T15:38:49.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:49.136+0000 7fdfb357e780 1 bucket sync caught up with source: 2023-10-11T15:38:49.139 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , 00000000002.97.6, , , , , , 00000000002.64.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.108.6, , , , 00000000001.72.6] 2023-10-11T15:38:49.139 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , 00000000002.97.6, , , , , , 00000000002.64.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.108.6, , , , 00000000001.72.6] 2023-10-11T15:38:49.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:49.136+0000 7fdfb357e780 0 bucket checkpoint complete 2023-10-11T15:38:49.154 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-63 2023-10-11T15:38:49.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:38:49.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:38:49.156 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:38:49.215 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:38:49.216 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:38:49.252 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:49.250+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:38:54.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:54.251+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:38:59.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:38:59.252+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:04.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:04.253+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:09.255 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:09.253+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:14.256 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:14.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:19.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:19.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:24.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:24.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:29.257 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:29.256+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:34.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:34.257+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:39.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:39.257+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:44.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:44.258+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:49.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:49.258+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:54.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:54.259+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:39:59.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:39:59.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:04.261 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:04.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:09.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:09.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:14.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:14.261+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:19.263 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:19.261+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:24.264 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:24.262+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:29.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:29.263+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:34.265 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:34.263+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:39.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:39.264+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:44.266 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:44.265+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:49.266+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:54.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:54.267+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:40:59.268 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:40:59.267+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:04.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:04.268+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:09.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:09.268+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:14.270 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:14.269+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:19.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:19.270+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:24.271 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:24.270+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:29.272 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:29.271+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:34.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:34.272+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:39.273 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:39.271+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:44.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:44.272+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:49.275 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:49.273+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:54.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:54.274+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:41:59.276 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:41:59.274+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:04.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:04.275+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:09.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:09.276+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:14.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:14.276+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:19.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:19.277+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:24.279 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:24.278+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:29.280 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:29.279+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:34.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:34.280+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:39.281 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:39.280+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:44.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:44.281+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:49.283 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:49.282+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:54.284 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:54.282+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:42:59.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:42:59.283+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:04.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:04.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:09.285 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:09.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:14.286 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:14.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:19.287 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:19.285+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:24.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:24.285+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:29.286+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:34.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:34.286+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:39.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:39.287+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T15:43:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.288+0000 7f864fab3780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-11T15:43:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.288+0000 7f864fab3780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T15:43:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.288+0000 7f864fab3780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T15:43:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-11T15:43:44.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-10-11T15:43:44.311 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.312 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.312 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.313 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.313 DEBUG:boto:Data: 2023-10-11T15:43:44.313 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.314 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.314 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.315 DEBUG:boto:Params: {} 2023-10-11T15:43:44.316 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:43:44.316 DEBUG:boto:Token: None 2023-10-11T15:43:44.317 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.317 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:55kHjM3BYzBk9RvwPPmXoJC74/4= 2023-10-11T15:43:44.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:55kHjM3BYzBk9RvwPPmXoJC74/4=', 'Content-Length': '0'} 2023-10-11T15:43:44.338 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a961adc8f22cd208-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.339 DEBUG:rgw_multi.tests:created bucket=dghqvd-64 2023-10-11T15:43:44.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:43:44.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:43:44.340 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:43:44.394 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:43:44.395 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:43:44.428 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:43:44.443 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:43:44.444 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.444 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:43:44.444 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:43:44.444 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.445 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.445 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:43:44.445 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:43:44.445 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.446 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.446 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:43:44.446 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:43:44.447 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.447 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.447 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039024.336393_729.1", 2023-10-11T15:43:44.448 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:43:44.336393Z" 2023-10-11T15:43:44.448 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.448 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:43:44.449 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T15:43:44.450 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:43:44.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:43:44.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:43:44.451 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:43:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:43:44.503 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:43:44.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.529+0000 7f969b7ff780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:43:44.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.530+0000 7f969b7ff780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:43:44.544 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:43:44.545 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:43:44.545 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:43:44.545 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:43:44.545 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:43:44.546 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:43:44.546 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:43:44.546 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.546 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:43:44.547 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.547 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:43:44.547 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:43:44.547 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:43:44.547 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:43:44.548 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:43:44.548 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:43:44.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:43:44.548 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:43:44.548 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:43:44.549 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:43:44.550 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:43:44.550 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:43:44.550 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:43:44.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:43:44.550 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:43:44.551 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:43:44.551 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.551 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.551 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.551 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:43:44.552 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:43:44.552 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:43:44.552 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:43:44.552 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:43:44.552 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:43:44.553 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:43:44.553 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:43:44.553 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:43:44.553 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.554 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:43:44.554 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:43:44.554 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:43:44.554 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:43:44.555 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039024.336393_729.1", 2023-10-11T15:43:44.555 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:43:44.555 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:43:44.555 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:43:44.555 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:43:44.336393Z", 2023-10-11T15:43:44.556 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:43:44.556 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.556 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:43:44.556 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:43:44.556 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.557 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:43:44.557 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:43:44.557 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:43:44.557 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:43:44.557 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:43:44Z" 2023-10-11T15:43:44.558 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:43:44.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039024.336393_729.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:43:44.336393Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:43:44Z" } 2023-10-11T15:43:44.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039024.336393_729.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:43:44.336393Z', 'realm_epoch': 2}}] 2023-10-11T15:43:44.560 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T15:43:44.560 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T15:43:44.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:43:44.561 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.561 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.562 DEBUG:boto:Method: HEAD 2023-10-11T15:43:44.562 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.562 DEBUG:boto:Data: 2023-10-11T15:43:44.563 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.563 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.564 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.564 DEBUG:boto:Params: {} 2023-10-11T15:43:44.565 DEBUG:boto:Token: None 2023-10-11T15:43:44.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T15:43:44.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T15:43:44.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a681a4ea78e06344-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.569 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:43:44.570 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:43:44.570 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.570 DEBUG:boto:Path: /dghqvd-64/obj11 2023-10-11T15:43:44.571 DEBUG:boto:Data: 2023-10-11T15:43:44.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:43:44.572 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.572 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.572 DEBUG:boto:Params: {} 2023-10-11T15:43:44.573 DEBUG:boto:Token: None 2023-10-11T15:43:44.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/obj11 2023-10-11T15:43:44.574 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ol4YF+Z5QFZsj6b0Dwvoo/Tz/2A= 2023-10-11T15:43:44.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ol4YF+Z5QFZsj6b0Dwvoo/Tz/2A='} 2023-10-11T15:43:44.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000273591553edb2d15-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.581 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.581 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.581 DEBUG:boto:Method: HEAD 2023-10-11T15:43:44.582 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.582 DEBUG:boto:Data: 2023-10-11T15:43:44.582 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.583 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.583 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.584 DEBUG:boto:Params: {} 2023-10-11T15:43:44.584 DEBUG:boto:Token: None 2023-10-11T15:43:44.585 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.585 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T15:43:44.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T15:43:44.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000761f6cddb9ccd394-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.588 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:43:44.588 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:43:44.589 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.589 DEBUG:boto:Path: /dghqvd-64/obj22 2023-10-11T15:43:44.590 DEBUG:boto:Data: 2023-10-11T15:43:44.590 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:43:44.590 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.591 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.591 DEBUG:boto:Params: {} 2023-10-11T15:43:44.592 DEBUG:boto:Token: None 2023-10-11T15:43:44.592 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/obj22 2023-10-11T15:43:44.593 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KZ3Ny9USaQuo4rmHN8YXVbdM5ec= 2023-10-11T15:43:44.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KZ3Ny9USaQuo4rmHN8YXVbdM5ec='} 2023-10-11T15:43:44.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009430e88add24bcba-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.597 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.598 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.598 DEBUG:boto:Method: HEAD 2023-10-11T15:43:44.599 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.599 DEBUG:boto:Data: 2023-10-11T15:43:44.599 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.600 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.600 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.600 DEBUG:boto:Params: {} 2023-10-11T15:43:44.601 DEBUG:boto:Token: None 2023-10-11T15:43:44.601 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.602 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T15:43:44.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T15:43:44.604 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd5e6043c88e1336-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.605 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:43:44.605 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:43:44.606 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.606 DEBUG:boto:Path: /dghqvd-64/obj33 2023-10-11T15:43:44.606 DEBUG:boto:Data: 2023-10-11T15:43:44.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:43:44.607 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.608 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.608 DEBUG:boto:Params: {} 2023-10-11T15:43:44.609 DEBUG:boto:Token: None 2023-10-11T15:43:44.609 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/obj33 2023-10-11T15:43:44.610 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:JEl6kuCtYlNBj81mH7TQBIUwymU= 2023-10-11T15:43:44.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JEl6kuCtYlNBj81mH7TQBIUwymU='} 2023-10-11T15:43:44.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf3985b3560ee56f-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.615 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.616 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.616 DEBUG:boto:Method: HEAD 2023-10-11T15:43:44.617 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.617 DEBUG:boto:Data: 2023-10-11T15:43:44.617 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.618 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.618 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.619 DEBUG:boto:Params: {} 2023-10-11T15:43:44.619 DEBUG:boto:Token: None 2023-10-11T15:43:44.619 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T15:43:44.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T15:43:44.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072d154304e98e589-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.623 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:43:44.623 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:43:44.623 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.624 DEBUG:boto:Path: /dghqvd-64/obj44 2023-10-11T15:43:44.624 DEBUG:boto:Data: 2023-10-11T15:43:44.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:43:44.625 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.625 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.626 DEBUG:boto:Params: {} 2023-10-11T15:43:44.626 DEBUG:boto:Token: None 2023-10-11T15:43:44.627 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/obj44 2023-10-11T15:43:44.627 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SaZwDoKWjmJD85iCHGLxKhJkFBY= 2023-10-11T15:43:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SaZwDoKWjmJD85iCHGLxKhJkFBY='} 2023-10-11T15:43:44.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044cfa94aa7a95c36-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.671 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:44.671 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:44.672 DEBUG:boto:Method: HEAD 2023-10-11T15:43:44.672 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:44.672 DEBUG:boto:Data: 2023-10-11T15:43:44.673 DEBUG:boto:Headers: {} 2023-10-11T15:43:44.673 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.674 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.674 DEBUG:boto:Params: {} 2023-10-11T15:43:44.675 DEBUG:boto:Token: None 2023-10-11T15:43:44.675 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/ 2023-10-11T15:43:44.675 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T15:43:44.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T15:43:44.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a388212bee61182-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.679 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:43:44.679 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:43:44.679 DEBUG:boto:Method: PUT 2023-10-11T15:43:44.680 DEBUG:boto:Path: /dghqvd-64/obj55 2023-10-11T15:43:44.680 DEBUG:boto:Data: 2023-10-11T15:43:44.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:43:44.681 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:44.681 DEBUG:boto:Port: 8000 2023-10-11T15:43:44.682 DEBUG:boto:Params: {} 2023-10-11T15:43:44.682 DEBUG:boto:Token: None 2023-10-11T15:43:44.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:43:44 GMT /dghqvd-64/obj55 2023-10-11T15:43:44.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:h4xRouq9jpYZSGoheUwtMSIPtB8= 2023-10-11T15:43:44.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h4xRouq9jpYZSGoheUwtMSIPtB8='} 2023-10-11T15:43:44.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6738e0bdf19fbce-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:44.688 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-64 2023-10-11T15:43:44.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:43:44.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:43:44.689 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:43:44.747 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:43:44.748 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:43:44.782 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:44.780+0000 7f6062859780 1 waiting to reach incremental sync.. 2023-10-11T15:43:49.783 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:49.782+0000 7f6062859780 1 bucket sync caught up with source: 2023-10-11T15:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.111.6, 00000000002.102.6, 00000000002.118.6] 2023-10-11T15:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.111.6, 00000000002.102.6, 00000000002.118.6] 2023-10-11T15:43:49.784 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:49.782+0000 7f6062859780 0 bucket checkpoint complete 2023-10-11T15:43:49.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dghqvd-64 2023-10-11T15:43:49.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:43:49.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:43:49.803 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:43:49.865 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:43:49.866 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:43:49.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:49.901+0000 7f57bca48780 1 waiting to reach incremental sync.. 2023-10-11T15:43:54.902 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:54.901+0000 7f57bca48780 1 waiting to reach incremental sync.. 2023-10-11T15:43:59.904 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:59.903+0000 7f57bca48780 1 bucket sync caught up with source: 2023-10-11T15:43:59.905 INFO:teuthology.orchestra.run.smithi158.stderr: local status: [, , , , , , , , 00000000001.318.6, 00000000002.199.6, 00000000002.118.6] 2023-10-11T15:43:59.905 INFO:teuthology.orchestra.run.smithi158.stderr: remote markers: [, , , , , , , , 00000000001.318.6, 00000000002.199.6, 00000000002.118.6] 2023-10-11T15:43:59.905 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:43:59.903+0000 7f57bca48780 0 bucket checkpoint complete 2023-10-11T15:43:59.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dghqvd-64 zones={a2, a1} 2023-10-11T15:43:59.923 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.923 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.924 DEBUG:boto:Method: HEAD 2023-10-11T15:43:59.924 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:59.924 DEBUG:boto:Data: 2023-10-11T15:43:59.925 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.925 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:43:59.926 DEBUG:boto:Port: 8001 2023-10-11T15:43:59.926 DEBUG:boto:Params: {} 2023-10-11T15:43:59.927 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:43:59.927 DEBUG:boto:Token: None 2023-10-11T15:43:59.928 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/ 2023-10-11T15:43:59.928 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8= 2023-10-11T15:43:59.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8=', 'Content-Length': '0'} 2023-10-11T15:43:59.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e4b0abd88306396-006526c2bf-1fbc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.933 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.934 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.934 DEBUG:boto:Method: HEAD 2023-10-11T15:43:59.935 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:43:59.935 DEBUG:boto:Data: 2023-10-11T15:43:59.935 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.936 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:59.936 DEBUG:boto:Port: 8000 2023-10-11T15:43:59.937 DEBUG:boto:Params: {} 2023-10-11T15:43:59.937 DEBUG:boto:Token: None 2023-10-11T15:43:59.938 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/ 2023-10-11T15:43:59.938 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8= 2023-10-11T15:43:59.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8=', 'Content-Length': '0'} 2023-10-11T15:43:59.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e66f49af8ee3d9bb-006526c2bf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-10-11T15:43:59.942 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.943 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.943 DEBUG:boto:path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.944 DEBUG:boto:auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.944 DEBUG:boto:Method: GET 2023-10-11T15:43:59.944 DEBUG:boto:Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.945 DEBUG:boto:Data: 2023-10-11T15:43:59.945 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.946 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:43:59.946 DEBUG:boto:Port: 8001 2023-10-11T15:43:59.947 DEBUG:boto:Params: {} 2023-10-11T15:43:59.947 DEBUG:boto:Token: None 2023-10-11T15:43:59.948 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/?versions 2023-10-11T15:43:59.948 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T15:43:59.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T15:43:59.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c943f737f65372-006526c2bf-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.951 DEBUG:boto:b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:43:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:43:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:43:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:43:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:43:59.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:43:59.954 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-10-11T15:43:59.955 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.955 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.956 DEBUG:boto:path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.956 DEBUG:boto:auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.956 DEBUG:boto:Method: GET 2023-10-11T15:43:59.957 DEBUG:boto:Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.957 DEBUG:boto:Data: 2023-10-11T15:43:59.958 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.958 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:59.959 DEBUG:boto:Port: 8000 2023-10-11T15:43:59.959 DEBUG:boto:Params: {} 2023-10-11T15:43:59.959 DEBUG:boto:Token: None 2023-10-11T15:43:59.960 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/?versions 2023-10-11T15:43:59.961 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T15:43:59.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T15:43:59.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d205eef3bf1c852-006526c2bf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.964 DEBUG:boto:b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:43:59.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-10-11T15:43:59.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-10-11T15:43:59.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-10-11T15:43:59.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-10-11T15:43:59.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-10-11T15:43:59.967 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.968 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.968 DEBUG:boto:path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.969 DEBUG:boto:auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.969 DEBUG:boto:Method: GET 2023-10-11T15:43:59.969 DEBUG:boto:Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.970 DEBUG:boto:Data: 2023-10-11T15:43:59.970 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:43:59.972 DEBUG:boto:Port: 8001 2023-10-11T15:43:59.973 DEBUG:boto:Params: {} 2023-10-11T15:43:59.973 DEBUG:boto:Token: None 2023-10-11T15:43:59.974 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/?versions 2023-10-11T15:43:59.974 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T15:43:59.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T15:43:59.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa96647f64ea1e87-006526c2bf-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.978 DEBUG:boto:b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:43:59.979 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:43:59.979 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:43:59.979 DEBUG:boto:path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.980 DEBUG:boto:auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.981 DEBUG:boto:Method: GET 2023-10-11T15:43:59.982 DEBUG:boto:Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T15:43:59.982 DEBUG:boto:Data: 2023-10-11T15:43:59.983 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:43:59.984 DEBUG:boto:Port: 8000 2023-10-11T15:43:59.984 DEBUG:boto:Params: {} 2023-10-11T15:43:59.984 DEBUG:boto:Token: None 2023-10-11T15:43:59.985 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/?versions 2023-10-11T15:43:59.986 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T15:43:59.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T15:43:59.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000837bdaf262d9725c-006526c2bf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:43:59.989 DEBUG:boto:b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:43:59.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:43:59.991 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:43:59.992 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:43:59.993 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:43:59.993 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:43:59.994 DEBUG:boto:Method: GET 2023-10-11T15:43:59.994 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:43:59.995 DEBUG:boto:Data: 2023-10-11T15:43:59.995 DEBUG:boto:Headers: {} 2023-10-11T15:43:59.995 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:43:59.996 DEBUG:boto:Port: 8001 2023-10-11T15:43:59.996 DEBUG:boto:Params: {} 2023-10-11T15:43:59.997 DEBUG:boto:Token: None 2023-10-11T15:43:59.997 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:43:59 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:43:59.998 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4Ekld866laImbjdWfLjt8OlUybM= 2023-10-11T15:43:59.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Ekld866laImbjdWfLjt8OlUybM=', 'Content-Length': '0'} 2023-10-11T15:44:00.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e109e4911d1034-006526c2bf-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.002 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.002 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.003 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.003 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.004 DEBUG:boto:Method: GET 2023-10-11T15:44:00.004 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.005 DEBUG:boto:Data: 2023-10-11T15:44:00.005 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.005 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.006 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.006 DEBUG:boto:Params: {} 2023-10-11T15:44:00.007 DEBUG:boto:Token: None 2023-10-11T15:44:00.007 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.008 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T15:44:00.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T15:44:00.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a8718138c60b60f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.011 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.011 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.012 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.012 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.013 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.013 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.014 DEBUG:boto:Data: 2023-10-11T15:44:00.014 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.015 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.015 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.015 DEBUG:boto:Params: {} 2023-10-11T15:44:00.016 DEBUG:boto:Token: None 2023-10-11T15:44:00.016 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.017 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw= 2023-10-11T15:44:00.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw=', 'Content-Length': '0'} 2023-10-11T15:44:00.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000198660d72466a0a-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.020 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.020 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.021 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.021 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.021 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.022 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.022 DEBUG:boto:Data: 2023-10-11T15:44:00.023 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.024 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.024 DEBUG:boto:Params: {} 2023-10-11T15:44:00.025 DEBUG:boto:Token: None 2023-10-11T15:44:00.025 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw= 2023-10-11T15:44:00.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw=', 'Content-Length': '0'} 2023-10-11T15:44:00.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee28d6fe3dd5e5d8-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:44:00.029 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.030 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.030 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.031 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.031 DEBUG:boto:Method: GET 2023-10-11T15:44:00.031 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.032 DEBUG:boto:Data: 2023-10-11T15:44:00.032 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.033 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.034 DEBUG:boto:Params: {} 2023-10-11T15:44:00.034 DEBUG:boto:Token: None 2023-10-11T15:44:00.035 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T15:44:00.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T15:44:00.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851a717181b43361-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.039 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.039 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.039 DEBUG:boto:path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.040 DEBUG:boto:auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.040 DEBUG:boto:Method: GET 2023-10-11T15:44:00.041 DEBUG:boto:Path: /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.041 DEBUG:boto:Data: 2023-10-11T15:44:00.042 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.042 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.043 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.043 DEBUG:boto:Params: {} 2023-10-11T15:44:00.044 DEBUG:boto:Token: None 2023-10-11T15:44:00.044 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11?versionId=null 2023-10-11T15:44:00.045 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T15:44:00.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T15:44:00.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438e7d2d713f81e5-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.090 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.091 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.091 DEBUG:boto:path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.092 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.092 DEBUG:boto:Method: GET 2023-10-11T15:44:00.093 DEBUG:boto:Path: /dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.093 DEBUG:boto:Data: 2023-10-11T15:44:00.094 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.094 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.094 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.095 DEBUG:boto:Params: {} 2023-10-11T15:44:00.095 DEBUG:boto:Token: None 2023-10-11T15:44:00.096 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.096 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f2c576ab3c8210-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.101 DEBUG:boto:b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.102 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.102 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.103 DEBUG:boto:path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.103 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.104 DEBUG:boto:Method: GET 2023-10-11T15:44:00.104 DEBUG:boto:Path: /dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.104 DEBUG:boto:Data: 2023-10-11T15:44:00.105 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.105 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.106 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.106 DEBUG:boto:Params: {} 2023-10-11T15:44:00.107 DEBUG:boto:Token: None 2023-10-11T15:44:00.107 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1e27914efab94d-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.111 DEBUG:boto:b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.112 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.112 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.113 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.113 DEBUG:boto:Path: /dghqvd-64/obj11 2023-10-11T15:44:00.114 DEBUG:boto:Data: 2023-10-11T15:44:00.114 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.115 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.115 DEBUG:boto:Params: {} 2023-10-11T15:44:00.116 DEBUG:boto:Token: None 2023-10-11T15:44:00.116 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11 2023-10-11T15:44:00.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc= 2023-10-11T15:44:00.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc=', 'Content-Length': '0'} 2023-10-11T15:44:00.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a1173d2377454b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.120 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.120 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.121 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.121 DEBUG:boto:Path: /dghqvd-64/obj11 2023-10-11T15:44:00.121 DEBUG:boto:Data: 2023-10-11T15:44:00.122 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.122 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.123 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.123 DEBUG:boto:Params: {} 2023-10-11T15:44:00.123 DEBUG:boto:Token: None 2023-10-11T15:44:00.124 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11 2023-10-11T15:44:00.124 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc= 2023-10-11T15:44:00.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc=', 'Content-Length': '0'} 2023-10-11T15:44:00.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fad7491aacc3369-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-10-11T15:44:00.127 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.128 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.128 DEBUG:boto:Method: GET 2023-10-11T15:44:00.129 DEBUG:boto:Path: /dghqvd-64/obj11 2023-10-11T15:44:00.129 DEBUG:boto:Data: 2023-10-11T15:44:00.130 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.130 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.131 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.131 DEBUG:boto:Params: {} 2023-10-11T15:44:00.131 DEBUG:boto:Token: None 2023-10-11T15:44:00.132 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11 2023-10-11T15:44:00.132 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8= 2023-10-11T15:44:00.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8=', 'Content-Length': '0'} 2023-10-11T15:44:00.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53123cf258aa197-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.182 DEBUG:boto:path=/dghqvd-64/obj11 2023-10-11T15:44:00.183 DEBUG:boto:auth_path=/dghqvd-64/obj11 2023-10-11T15:44:00.183 DEBUG:boto:Method: GET 2023-10-11T15:44:00.184 DEBUG:boto:Path: /dghqvd-64/obj11 2023-10-11T15:44:00.184 DEBUG:boto:Data: 2023-10-11T15:44:00.185 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.185 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.186 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.186 DEBUG:boto:Params: {} 2023-10-11T15:44:00.186 DEBUG:boto:Token: None 2023-10-11T15:44:00.187 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj11 2023-10-11T15:44:00.187 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8= 2023-10-11T15:44:00.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8=', 'Content-Length': '0'} 2023-10-11T15:44:00.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e023432cd59902c-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.191 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.191 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.192 DEBUG:boto:path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.192 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.192 DEBUG:boto:Method: GET 2023-10-11T15:44:00.193 DEBUG:boto:Path: /dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.193 DEBUG:boto:Data: 2023-10-11T15:44:00.193 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.194 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.195 DEBUG:boto:Params: {} 2023-10-11T15:44:00.195 DEBUG:boto:Token: None 2023-10-11T15:44:00.196 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8f887088500722-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.199 DEBUG:boto:b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.200 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.201 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.201 DEBUG:boto:path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.201 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.202 DEBUG:boto:Method: GET 2023-10-11T15:44:00.202 DEBUG:boto:Path: /dghqvd-64/?prefix=obj11 2023-10-11T15:44:00.203 DEBUG:boto:Data: 2023-10-11T15:44:00.203 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.204 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.204 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.204 DEBUG:boto:Params: {} 2023-10-11T15:44:00.205 DEBUG:boto:Token: None 2023-10-11T15:44:00.205 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.206 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000320d1c132d2a9a5-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.254 DEBUG:boto:b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:44:00.255 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.256 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.256 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.256 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.257 DEBUG:boto:Method: GET 2023-10-11T15:44:00.257 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.258 DEBUG:boto:Data: 2023-10-11T15:44:00.258 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.259 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.259 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.259 DEBUG:boto:Params: {} 2023-10-11T15:44:00.260 DEBUG:boto:Token: None 2023-10-11T15:44:00.260 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.261 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T15:44:00.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T15:44:00.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f512fe5c0faa681c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.264 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.264 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.265 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.265 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.266 DEBUG:boto:Method: GET 2023-10-11T15:44:00.266 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.267 DEBUG:boto:Data: 2023-10-11T15:44:00.267 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.267 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.268 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.268 DEBUG:boto:Params: {} 2023-10-11T15:44:00.269 DEBUG:boto:Token: None 2023-10-11T15:44:00.269 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T15:44:00.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T15:44:00.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c4f58e769ab1d8-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.273 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.274 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.274 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.274 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.275 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.275 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.276 DEBUG:boto:Data: 2023-10-11T15:44:00.276 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.276 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.277 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.277 DEBUG:boto:Params: {} 2023-10-11T15:44:00.278 DEBUG:boto:Token: None 2023-10-11T15:44:00.278 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM= 2023-10-11T15:44:00.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM=', 'Content-Length': '0'} 2023-10-11T15:44:00.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e149957ec6c8ab49-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.281 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.282 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.282 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.283 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.283 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.284 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.284 DEBUG:boto:Data: 2023-10-11T15:44:00.284 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.285 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.286 DEBUG:boto:Params: {} 2023-10-11T15:44:00.286 DEBUG:boto:Token: None 2023-10-11T15:44:00.287 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM= 2023-10-11T15:44:00.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM=', 'Content-Length': '0'} 2023-10-11T15:44:00.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098318bf4d8f33b79-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:44:00.291 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.291 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.291 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.292 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.292 DEBUG:boto:Method: GET 2023-10-11T15:44:00.293 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.293 DEBUG:boto:Data: 2023-10-11T15:44:00.293 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.294 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.295 DEBUG:boto:Params: {} 2023-10-11T15:44:00.295 DEBUG:boto:Token: None 2023-10-11T15:44:00.296 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.296 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T15:44:00.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T15:44:00.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4a4717efdda9db-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.299 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.299 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.300 DEBUG:boto:path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.300 DEBUG:boto:auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.301 DEBUG:boto:Method: GET 2023-10-11T15:44:00.301 DEBUG:boto:Path: /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.301 DEBUG:boto:Data: 2023-10-11T15:44:00.302 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.302 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.303 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.303 DEBUG:boto:Params: {} 2023-10-11T15:44:00.304 DEBUG:boto:Token: None 2023-10-11T15:44:00.304 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22?versionId=null 2023-10-11T15:44:00.305 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T15:44:00.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T15:44:00.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e034a66bc2d3fb-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.308 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.309 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.309 DEBUG:boto:path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.309 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.310 DEBUG:boto:Method: GET 2023-10-11T15:44:00.310 DEBUG:boto:Path: /dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.311 DEBUG:boto:Data: 2023-10-11T15:44:00.311 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.312 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.312 DEBUG:boto:Params: {} 2023-10-11T15:44:00.313 DEBUG:boto:Token: None 2023-10-11T15:44:00.313 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30a722a79747ace-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.362 DEBUG:boto:b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.363 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.364 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.364 DEBUG:boto:path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.365 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.365 DEBUG:boto:Method: GET 2023-10-11T15:44:00.365 DEBUG:boto:Path: /dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.366 DEBUG:boto:Data: 2023-10-11T15:44:00.366 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.367 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.367 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.368 DEBUG:boto:Params: {} 2023-10-11T15:44:00.368 DEBUG:boto:Token: None 2023-10-11T15:44:00.369 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.369 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002101cf78f7b1477b-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.373 DEBUG:boto:b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.373 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.374 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.374 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.375 DEBUG:boto:Path: /dghqvd-64/obj22 2023-10-11T15:44:00.375 DEBUG:boto:Data: 2023-10-11T15:44:00.376 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.376 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.376 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.377 DEBUG:boto:Params: {} 2023-10-11T15:44:00.377 DEBUG:boto:Token: None 2023-10-11T15:44:00.378 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22 2023-10-11T15:44:00.378 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA= 2023-10-11T15:44:00.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA=', 'Content-Length': '0'} 2023-10-11T15:44:00.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2790ede29d211b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.381 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.382 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.382 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.382 DEBUG:boto:Path: /dghqvd-64/obj22 2023-10-11T15:44:00.383 DEBUG:boto:Data: 2023-10-11T15:44:00.383 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.384 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.384 DEBUG:boto:Params: {} 2023-10-11T15:44:00.385 DEBUG:boto:Token: None 2023-10-11T15:44:00.385 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22 2023-10-11T15:44:00.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA= 2023-10-11T15:44:00.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA=', 'Content-Length': '0'} 2023-10-11T15:44:00.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006074bb9a9dbff557-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-10-11T15:44:00.389 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.390 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.390 DEBUG:boto:Method: GET 2023-10-11T15:44:00.390 DEBUG:boto:Path: /dghqvd-64/obj22 2023-10-11T15:44:00.391 DEBUG:boto:Data: 2023-10-11T15:44:00.391 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.392 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.393 DEBUG:boto:Params: {} 2023-10-11T15:44:00.393 DEBUG:boto:Token: None 2023-10-11T15:44:00.393 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22 2023-10-11T15:44:00.394 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4= 2023-10-11T15:44:00.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4=', 'Content-Length': '0'} 2023-10-11T15:44:00.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0cc94804b4b725-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.397 DEBUG:boto:path=/dghqvd-64/obj22 2023-10-11T15:44:00.398 DEBUG:boto:auth_path=/dghqvd-64/obj22 2023-10-11T15:44:00.398 DEBUG:boto:Method: GET 2023-10-11T15:44:00.399 DEBUG:boto:Path: /dghqvd-64/obj22 2023-10-11T15:44:00.399 DEBUG:boto:Data: 2023-10-11T15:44:00.399 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.400 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.400 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.401 DEBUG:boto:Params: {} 2023-10-11T15:44:00.401 DEBUG:boto:Token: None 2023-10-11T15:44:00.402 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj22 2023-10-11T15:44:00.402 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4= 2023-10-11T15:44:00.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4=', 'Content-Length': '0'} 2023-10-11T15:44:00.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4c3221e9b148f3-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.405 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.406 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.406 DEBUG:boto:path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.407 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.407 DEBUG:boto:Method: GET 2023-10-11T15:44:00.408 DEBUG:boto:Path: /dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.408 DEBUG:boto:Data: 2023-10-11T15:44:00.409 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.409 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.409 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.410 DEBUG:boto:Params: {} 2023-10-11T15:44:00.410 DEBUG:boto:Token: None 2023-10-11T15:44:00.411 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.411 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be27341306fd96ba-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.415 DEBUG:boto:b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.416 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.416 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.416 DEBUG:boto:path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.417 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.417 DEBUG:boto:Method: GET 2023-10-11T15:44:00.418 DEBUG:boto:Path: /dghqvd-64/?prefix=obj22 2023-10-11T15:44:00.418 DEBUG:boto:Data: 2023-10-11T15:44:00.418 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.419 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.420 DEBUG:boto:Params: {} 2023-10-11T15:44:00.420 DEBUG:boto:Token: None 2023-10-11T15:44:00.421 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d4c59906a018e5-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.425 DEBUG:boto:b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:44:00.426 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.426 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.427 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.427 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.427 DEBUG:boto:Method: GET 2023-10-11T15:44:00.428 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.428 DEBUG:boto:Data: 2023-10-11T15:44:00.429 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.430 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.430 DEBUG:boto:Params: {} 2023-10-11T15:44:00.431 DEBUG:boto:Token: None 2023-10-11T15:44:00.431 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.431 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T15:44:00.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T15:44:00.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fda38c380222dfc-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.435 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.435 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.436 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.436 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.437 DEBUG:boto:Method: GET 2023-10-11T15:44:00.437 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.437 DEBUG:boto:Data: 2023-10-11T15:44:00.438 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.438 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.439 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.439 DEBUG:boto:Params: {} 2023-10-11T15:44:00.440 DEBUG:boto:Token: None 2023-10-11T15:44:00.440 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T15:44:00.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T15:44:00.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd82730528a7af0-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.444 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.444 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.445 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.445 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.445 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.446 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.446 DEBUG:boto:Data: 2023-10-11T15:44:00.447 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.447 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.448 DEBUG:boto:Params: {} 2023-10-11T15:44:00.448 DEBUG:boto:Token: None 2023-10-11T15:44:00.449 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM= 2023-10-11T15:44:00.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM=', 'Content-Length': '0'} 2023-10-11T15:44:00.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644c8beb29bb620d-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.452 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.453 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.453 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.454 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.454 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.454 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.455 DEBUG:boto:Data: 2023-10-11T15:44:00.455 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.456 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.456 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.457 DEBUG:boto:Params: {} 2023-10-11T15:44:00.457 DEBUG:boto:Token: None 2023-10-11T15:44:00.457 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.458 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM= 2023-10-11T15:44:00.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM=', 'Content-Length': '0'} 2023-10-11T15:44:00.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6891f602eb4dff1-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:44:00.462 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.463 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.463 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.463 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.464 DEBUG:boto:Method: GET 2023-10-11T15:44:00.464 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.465 DEBUG:boto:Data: 2023-10-11T15:44:00.465 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.466 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.466 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.467 DEBUG:boto:Params: {} 2023-10-11T15:44:00.467 DEBUG:boto:Token: None 2023-10-11T15:44:00.468 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.468 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T15:44:00.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T15:44:00.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674915dec152e000-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.518 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.519 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.520 DEBUG:boto:path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.520 DEBUG:boto:auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.520 DEBUG:boto:Method: GET 2023-10-11T15:44:00.521 DEBUG:boto:Path: /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.521 DEBUG:boto:Data: 2023-10-11T15:44:00.522 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.522 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.523 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.523 DEBUG:boto:Params: {} 2023-10-11T15:44:00.524 DEBUG:boto:Token: None 2023-10-11T15:44:00.524 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33?versionId=null 2023-10-11T15:44:00.525 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T15:44:00.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T15:44:00.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82a52bb6494a6cb-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.528 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.529 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.529 DEBUG:boto:path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.530 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.530 DEBUG:boto:Method: GET 2023-10-11T15:44:00.531 DEBUG:boto:Path: /dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.531 DEBUG:boto:Data: 2023-10-11T15:44:00.531 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.532 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.532 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.533 DEBUG:boto:Params: {} 2023-10-11T15:44:00.533 DEBUG:boto:Token: None 2023-10-11T15:44:00.534 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.534 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043e4925f16fa38b2-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.538 DEBUG:boto:b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.539 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.539 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.539 DEBUG:boto:path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.540 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.540 DEBUG:boto:Method: GET 2023-10-11T15:44:00.541 DEBUG:boto:Path: /dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.541 DEBUG:boto:Data: 2023-10-11T15:44:00.542 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.542 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.543 DEBUG:boto:Params: {} 2023-10-11T15:44:00.543 DEBUG:boto:Token: None 2023-10-11T15:44:00.544 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.544 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586ac7b6b311c130-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.590 DEBUG:boto:b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.591 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.592 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.593 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.593 DEBUG:boto:Path: /dghqvd-64/obj33 2023-10-11T15:44:00.594 DEBUG:boto:Data: 2023-10-11T15:44:00.594 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.595 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.595 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.596 DEBUG:boto:Params: {} 2023-10-11T15:44:00.596 DEBUG:boto:Token: None 2023-10-11T15:44:00.597 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33 2023-10-11T15:44:00.597 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw= 2023-10-11T15:44:00.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw=', 'Content-Length': '0'} 2023-10-11T15:44:00.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4c3a1585c58e34-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.600 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.601 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.601 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.602 DEBUG:boto:Path: /dghqvd-64/obj33 2023-10-11T15:44:00.602 DEBUG:boto:Data: 2023-10-11T15:44:00.603 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.603 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.603 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.604 DEBUG:boto:Params: {} 2023-10-11T15:44:00.604 DEBUG:boto:Token: None 2023-10-11T15:44:00.605 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33 2023-10-11T15:44:00.605 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw= 2023-10-11T15:44:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw=', 'Content-Length': '0'} 2023-10-11T15:44:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd4f1605714f74d-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-10-11T15:44:00.609 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.610 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.610 DEBUG:boto:Method: GET 2023-10-11T15:44:00.611 DEBUG:boto:Path: /dghqvd-64/obj33 2023-10-11T15:44:00.611 DEBUG:boto:Data: 2023-10-11T15:44:00.611 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.612 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.612 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.613 DEBUG:boto:Params: {} 2023-10-11T15:44:00.613 DEBUG:boto:Token: None 2023-10-11T15:44:00.614 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33 2023-10-11T15:44:00.614 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE= 2023-10-11T15:44:00.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE=', 'Content-Length': '0'} 2023-10-11T15:44:00.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002607db9b03a6761c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.618 DEBUG:boto:path=/dghqvd-64/obj33 2023-10-11T15:44:00.618 DEBUG:boto:auth_path=/dghqvd-64/obj33 2023-10-11T15:44:00.619 DEBUG:boto:Method: GET 2023-10-11T15:44:00.619 DEBUG:boto:Path: /dghqvd-64/obj33 2023-10-11T15:44:00.620 DEBUG:boto:Data: 2023-10-11T15:44:00.620 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.621 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.621 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.621 DEBUG:boto:Params: {} 2023-10-11T15:44:00.622 DEBUG:boto:Token: None 2023-10-11T15:44:00.622 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj33 2023-10-11T15:44:00.623 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE= 2023-10-11T15:44:00.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE=', 'Content-Length': '0'} 2023-10-11T15:44:00.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090519a4e798f25c4-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.626 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.627 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.627 DEBUG:boto:path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.628 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.628 DEBUG:boto:Method: GET 2023-10-11T15:44:00.628 DEBUG:boto:Path: /dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.629 DEBUG:boto:Data: 2023-10-11T15:44:00.629 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.630 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.630 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.631 DEBUG:boto:Params: {} 2023-10-11T15:44:00.631 DEBUG:boto:Token: None 2023-10-11T15:44:00.632 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.632 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73be8447f2ff09f-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.635 DEBUG:boto:b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.636 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.637 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.637 DEBUG:boto:path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.637 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.638 DEBUG:boto:Method: GET 2023-10-11T15:44:00.638 DEBUG:boto:Path: /dghqvd-64/?prefix=obj33 2023-10-11T15:44:00.639 DEBUG:boto:Data: 2023-10-11T15:44:00.639 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.640 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.640 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.641 DEBUG:boto:Params: {} 2023-10-11T15:44:00.641 DEBUG:boto:Token: None 2023-10-11T15:44:00.642 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.643 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbef885706054fd2-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.647 DEBUG:boto:b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:44:00.648 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.648 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.648 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.649 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.649 DEBUG:boto:Method: GET 2023-10-11T15:44:00.650 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.650 DEBUG:boto:Data: 2023-10-11T15:44:00.651 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.651 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.651 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.652 DEBUG:boto:Params: {} 2023-10-11T15:44:00.652 DEBUG:boto:Token: None 2023-10-11T15:44:00.653 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.653 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T15:44:00.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T15:44:00.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007692888ec23e6834-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.657 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.657 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.657 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.658 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.658 DEBUG:boto:Method: GET 2023-10-11T15:44:00.659 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.659 DEBUG:boto:Data: 2023-10-11T15:44:00.660 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.660 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.661 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.661 DEBUG:boto:Params: {} 2023-10-11T15:44:00.662 DEBUG:boto:Token: None 2023-10-11T15:44:00.662 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T15:44:00.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T15:44:00.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f882ace4b467a5f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.667 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.667 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.668 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.668 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.669 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.669 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.670 DEBUG:boto:Data: 2023-10-11T15:44:00.670 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.671 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.672 DEBUG:boto:Params: {} 2023-10-11T15:44:00.672 DEBUG:boto:Token: None 2023-10-11T15:44:00.673 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.674 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM= 2023-10-11T15:44:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM=', 'Content-Length': '0'} 2023-10-11T15:44:00.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047413763f8337e97-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.677 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.677 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.678 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.678 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.679 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.679 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.679 DEBUG:boto:Data: 2023-10-11T15:44:00.680 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.681 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.681 DEBUG:boto:Params: {} 2023-10-11T15:44:00.681 DEBUG:boto:Token: None 2023-10-11T15:44:00.682 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM= 2023-10-11T15:44:00.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM=', 'Content-Length': '0'} 2023-10-11T15:44:00.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ab39dfc21f7459-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:44:00.686 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.687 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.687 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.687 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.688 DEBUG:boto:Method: GET 2023-10-11T15:44:00.688 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.689 DEBUG:boto:Data: 2023-10-11T15:44:00.689 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.689 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.690 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.690 DEBUG:boto:Params: {} 2023-10-11T15:44:00.691 DEBUG:boto:Token: None 2023-10-11T15:44:00.691 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.692 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T15:44:00.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T15:44:00.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a41f38c56b3b5e2-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.695 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.695 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.696 DEBUG:boto:path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.696 DEBUG:boto:auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.697 DEBUG:boto:Method: GET 2023-10-11T15:44:00.697 DEBUG:boto:Path: /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.697 DEBUG:boto:Data: 2023-10-11T15:44:00.698 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.698 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.699 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.699 DEBUG:boto:Params: {} 2023-10-11T15:44:00.700 DEBUG:boto:Token: None 2023-10-11T15:44:00.700 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44?versionId=null 2023-10-11T15:44:00.700 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T15:44:00.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T15:44:00.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478b0c7c55965fd2-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.704 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.704 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.705 DEBUG:boto:path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.705 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.706 DEBUG:boto:Method: GET 2023-10-11T15:44:00.706 DEBUG:boto:Path: /dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.706 DEBUG:boto:Data: 2023-10-11T15:44:00.707 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.707 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.708 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.708 DEBUG:boto:Params: {} 2023-10-11T15:44:00.708 DEBUG:boto:Token: None 2023-10-11T15:44:00.709 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.709 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d68a265eb5a797b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.754 DEBUG:boto:b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.755 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.756 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.756 DEBUG:boto:path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.756 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.757 DEBUG:boto:Method: GET 2023-10-11T15:44:00.757 DEBUG:boto:Path: /dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.758 DEBUG:boto:Data: 2023-10-11T15:44:00.758 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.759 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.760 DEBUG:boto:Params: {} 2023-10-11T15:44:00.760 DEBUG:boto:Token: None 2023-10-11T15:44:00.761 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbac6441772653b8-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.765 DEBUG:boto:b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.766 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.766 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.767 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.767 DEBUG:boto:Path: /dghqvd-64/obj44 2023-10-11T15:44:00.768 DEBUG:boto:Data: 2023-10-11T15:44:00.768 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.768 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.769 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.769 DEBUG:boto:Params: {} 2023-10-11T15:44:00.770 DEBUG:boto:Token: None 2023-10-11T15:44:00.770 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44 2023-10-11T15:44:00.771 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c= 2023-10-11T15:44:00.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c=', 'Content-Length': '0'} 2023-10-11T15:44:00.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb42e7113dbefa4-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.774 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.774 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.775 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.775 DEBUG:boto:Path: /dghqvd-64/obj44 2023-10-11T15:44:00.776 DEBUG:boto:Data: 2023-10-11T15:44:00.776 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.776 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.777 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.777 DEBUG:boto:Params: {} 2023-10-11T15:44:00.778 DEBUG:boto:Token: None 2023-10-11T15:44:00.778 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44 2023-10-11T15:44:00.779 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c= 2023-10-11T15:44:00.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c=', 'Content-Length': '0'} 2023-10-11T15:44:00.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9029923af47a72-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-10-11T15:44:00.782 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.782 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.783 DEBUG:boto:Method: GET 2023-10-11T15:44:00.783 DEBUG:boto:Path: /dghqvd-64/obj44 2023-10-11T15:44:00.784 DEBUG:boto:Data: 2023-10-11T15:44:00.784 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.785 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.785 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.786 DEBUG:boto:Params: {} 2023-10-11T15:44:00.786 DEBUG:boto:Token: None 2023-10-11T15:44:00.787 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44 2023-10-11T15:44:00.787 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0= 2023-10-11T15:44:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0=', 'Content-Length': '0'} 2023-10-11T15:44:00.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000372bfb5ff45985a3-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.790 DEBUG:boto:path=/dghqvd-64/obj44 2023-10-11T15:44:00.790 DEBUG:boto:auth_path=/dghqvd-64/obj44 2023-10-11T15:44:00.791 DEBUG:boto:Method: GET 2023-10-11T15:44:00.791 DEBUG:boto:Path: /dghqvd-64/obj44 2023-10-11T15:44:00.792 DEBUG:boto:Data: 2023-10-11T15:44:00.792 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.793 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.793 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.793 DEBUG:boto:Params: {} 2023-10-11T15:44:00.794 DEBUG:boto:Token: None 2023-10-11T15:44:00.794 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj44 2023-10-11T15:44:00.795 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0= 2023-10-11T15:44:00.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0=', 'Content-Length': '0'} 2023-10-11T15:44:00.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae546aed97213128-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.798 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.798 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.799 DEBUG:boto:path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.799 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.800 DEBUG:boto:Method: GET 2023-10-11T15:44:00.800 DEBUG:boto:Path: /dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.800 DEBUG:boto:Data: 2023-10-11T15:44:00.801 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.801 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.802 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.802 DEBUG:boto:Params: {} 2023-10-11T15:44:00.803 DEBUG:boto:Token: None 2023-10-11T15:44:00.803 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.803 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047692bbf75caf3b3-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.807 DEBUG:boto:b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.808 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.809 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.809 DEBUG:boto:path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.809 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.810 DEBUG:boto:Method: GET 2023-10-11T15:44:00.810 DEBUG:boto:Path: /dghqvd-64/?prefix=obj44 2023-10-11T15:44:00.811 DEBUG:boto:Data: 2023-10-11T15:44:00.811 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.812 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.812 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.813 DEBUG:boto:Params: {} 2023-10-11T15:44:00.813 DEBUG:boto:Token: None 2023-10-11T15:44:00.814 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.814 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc4e7913d6d8677-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.818 DEBUG:boto:b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:44:00.819 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.820 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.820 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.821 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.821 DEBUG:boto:Method: GET 2023-10-11T15:44:00.821 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.822 DEBUG:boto:Data: 2023-10-11T15:44:00.822 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.823 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.823 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.824 DEBUG:boto:Params: {} 2023-10-11T15:44:00.824 DEBUG:boto:Token: None 2023-10-11T15:44:00.824 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.825 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T15:44:00.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T15:44:00.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0cfdf3ebb69a29-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.828 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.829 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.829 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.829 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.830 DEBUG:boto:Method: GET 2023-10-11T15:44:00.830 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.831 DEBUG:boto:Data: 2023-10-11T15:44:00.831 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.832 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.832 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.833 DEBUG:boto:Params: {} 2023-10-11T15:44:00.833 DEBUG:boto:Token: None 2023-10-11T15:44:00.834 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.834 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T15:44:00.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T15:44:00.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c18207e979bd72f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.837 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.838 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.838 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.839 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.839 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.840 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.841 DEBUG:boto:Data: 2023-10-11T15:44:00.841 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.842 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.843 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.844 DEBUG:boto:Params: {} 2023-10-11T15:44:00.844 DEBUG:boto:Token: None 2023-10-11T15:44:00.845 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.846 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE= 2023-10-11T15:44:00.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE=', 'Content-Length': '0'} 2023-10-11T15:44:00.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178f2ae69256b3f1-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.849 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.850 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.850 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.851 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.851 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.852 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.852 DEBUG:boto:Data: 2023-10-11T15:44:00.852 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.853 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.853 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.854 DEBUG:boto:Params: {} 2023-10-11T15:44:00.854 DEBUG:boto:Token: None 2023-10-11T15:44:00.855 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.855 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE= 2023-10-11T15:44:00.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE=', 'Content-Length': '0'} 2023-10-11T15:44:00.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66cbec0c10f63a2-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:44:00.859 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.859 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.859 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.860 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.860 DEBUG:boto:Method: GET 2023-10-11T15:44:00.861 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.861 DEBUG:boto:Data: 2023-10-11T15:44:00.861 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.862 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.862 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.863 DEBUG:boto:Params: {} 2023-10-11T15:44:00.863 DEBUG:boto:Token: None 2023-10-11T15:44:00.864 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.864 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T15:44:00.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T15:44:00.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66579cb26ab1f9b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.867 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.867 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.868 DEBUG:boto:path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.868 DEBUG:boto:auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.869 DEBUG:boto:Method: GET 2023-10-11T15:44:00.869 DEBUG:boto:Path: /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.870 DEBUG:boto:Data: 2023-10-11T15:44:00.871 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.871 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.872 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.872 DEBUG:boto:Params: {} 2023-10-11T15:44:00.873 DEBUG:boto:Token: None 2023-10-11T15:44:00.873 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55?versionId=null 2023-10-11T15:44:00.874 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T15:44:00.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T15:44:00.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f44bbabb5e7a571-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.876 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.877 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.877 DEBUG:boto:path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.878 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.878 DEBUG:boto:Method: GET 2023-10-11T15:44:00.879 DEBUG:boto:Path: /dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.879 DEBUG:boto:Data: 2023-10-11T15:44:00.880 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.880 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.880 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.881 DEBUG:boto:Params: {} 2023-10-11T15:44:00.881 DEBUG:boto:Token: None 2023-10-11T15:44:00.882 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.882 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011df784e8529462c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.926 DEBUG:boto:b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.927 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:00.927 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:00.928 DEBUG:boto:path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.928 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.929 DEBUG:boto:Method: GET 2023-10-11T15:44:00.929 DEBUG:boto:Path: /dghqvd-64/?prefix=obj55 2023-10-11T15:44:00.929 DEBUG:boto:Data: 2023-10-11T15:44:00.930 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.930 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.931 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.931 DEBUG:boto:Params: {} 2023-10-11T15:44:00.932 DEBUG:boto:Token: None 2023-10-11T15:44:00.932 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/ 2023-10-11T15:44:00.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T15:44:00.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T15:44:00.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab6674b8f96e1f7-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.936 DEBUG:boto:b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:00.937 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.937 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.938 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.938 DEBUG:boto:Path: /dghqvd-64/obj55 2023-10-11T15:44:00.939 DEBUG:boto:Data: 2023-10-11T15:44:00.939 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.940 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.941 DEBUG:boto:Params: {} 2023-10-11T15:44:00.941 DEBUG:boto:Token: None 2023-10-11T15:44:00.941 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55 2023-10-11T15:44:00.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ= 2023-10-11T15:44:00.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ=', 'Content-Length': '0'} 2023-10-11T15:44:00.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d419868adf3d6f3a-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.945 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.946 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.947 DEBUG:boto:Method: HEAD 2023-10-11T15:44:00.947 DEBUG:boto:Path: /dghqvd-64/obj55 2023-10-11T15:44:00.947 DEBUG:boto:Data: 2023-10-11T15:44:00.948 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.948 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.949 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.949 DEBUG:boto:Params: {} 2023-10-11T15:44:00.950 DEBUG:boto:Token: None 2023-10-11T15:44:00.950 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55 2023-10-11T15:44:00.951 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ= 2023-10-11T15:44:00.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ=', 'Content-Length': '0'} 2023-10-11T15:44:00.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef728494d0f659d1-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-10-11T15:44:00.954 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.954 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.955 DEBUG:boto:Method: GET 2023-10-11T15:44:00.955 DEBUG:boto:Path: /dghqvd-64/obj55 2023-10-11T15:44:00.956 DEBUG:boto:Data: 2023-10-11T15:44:00.956 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.957 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:00.957 DEBUG:boto:Port: 8001 2023-10-11T15:44:00.958 DEBUG:boto:Params: {} 2023-10-11T15:44:00.958 DEBUG:boto:Token: None 2023-10-11T15:44:00.959 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55 2023-10-11T15:44:00.959 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o= 2023-10-11T15:44:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o=', 'Content-Length': '0'} 2023-10-11T15:44:00.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71bd7fbad963273-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:00.962 DEBUG:boto:path=/dghqvd-64/obj55 2023-10-11T15:44:00.963 DEBUG:boto:auth_path=/dghqvd-64/obj55 2023-10-11T15:44:00.963 DEBUG:boto:Method: GET 2023-10-11T15:44:00.964 DEBUG:boto:Path: /dghqvd-64/obj55 2023-10-11T15:44:00.964 DEBUG:boto:Data: 2023-10-11T15:44:00.965 DEBUG:boto:Headers: {} 2023-10-11T15:44:00.965 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:00.966 DEBUG:boto:Port: 8000 2023-10-11T15:44:00.966 DEBUG:boto:Params: {} 2023-10-11T15:44:00.967 DEBUG:boto:Token: None 2023-10-11T15:44:00.967 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:00 GMT /dghqvd-64/obj55 2023-10-11T15:44:00.968 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o= 2023-10-11T15:44:00.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o=', 'Content-Length': '0'} 2023-10-11T15:44:00.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3b229035ca5c17-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:01.014 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:01.015 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:01.016 DEBUG:boto:path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.017 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.017 DEBUG:boto:Method: GET 2023-10-11T15:44:01.018 DEBUG:boto:Path: /dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.018 DEBUG:boto:Data: 2023-10-11T15:44:01.019 DEBUG:boto:Headers: {} 2023-10-11T15:44:01.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:01.020 DEBUG:boto:Port: 8001 2023-10-11T15:44:01.020 DEBUG:boto:Params: {} 2023-10-11T15:44:01.021 DEBUG:boto:Token: None 2023-10-11T15:44:01.021 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:01 GMT /dghqvd-64/ 2023-10-11T15:44:01.022 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0= 2023-10-11T15:44:01.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0=', 'Content-Length': '0'} 2023-10-11T15:44:01.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8e7b8c6a68c5b71-006526c2c1-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:01.026 DEBUG:boto:b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:01.028 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:01.028 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:01.029 DEBUG:boto:path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.029 DEBUG:boto:auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.029 DEBUG:boto:Method: GET 2023-10-11T15:44:01.030 DEBUG:boto:Path: /dghqvd-64/?prefix=obj55 2023-10-11T15:44:01.030 DEBUG:boto:Data: 2023-10-11T15:44:01.031 DEBUG:boto:Headers: {} 2023-10-11T15:44:01.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:01.032 DEBUG:boto:Port: 8000 2023-10-11T15:44:01.032 DEBUG:boto:Params: {} 2023-10-11T15:44:01.032 DEBUG:boto:Token: None 2023-10-11T15:44:01.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:44:01 GMT /dghqvd-64/ 2023-10-11T15:44:01.033 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0= 2023-10-11T15:44:01.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0=', 'Content-Length': '0'} 2023-10-11T15:44:01.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d20cd5794907780-006526c2c1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:01.037 DEBUG:boto:b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T15:44:01.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dghqvd-64 zones={a2, a1} 2023-10-11T15:44:01.039 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T15:44:01.039 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T15:44:01.040 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:44:01.039+0000 7fa8713ba700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 177660) UID: 0 2023-10-11T15:44:01.040 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:44:01.039+0000 7fa8756237c0 -1 shutting down 2023-10-11T15:44:01.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T15:44:07.138 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T15:44:07.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.140 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:07.200 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:07.200 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:07.247 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:07.248 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:07.251 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 5 2023-10-11T15:44:07.289 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.289 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.290 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.290 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.290 DEBUG:boto:Data: 2023-10-11T15:44:07.291 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.292 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.292 DEBUG:boto:Params: {} 2023-10-11T15:44:07.293 DEBUG:boto:Token: None 2023-10-11T15:44:07.293 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017b84873285865de-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.298 DEBUG:boto:path=/dghqvd-64/obj17 2023-10-11T15:44:07.298 DEBUG:boto:auth_path=/dghqvd-64/obj17 2023-10-11T15:44:07.299 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.299 DEBUG:boto:Path: /dghqvd-64/obj17 2023-10-11T15:44:07.299 DEBUG:boto:Data: 2023-10-11T15:44:07.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.300 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.301 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.301 DEBUG:boto:Params: {} 2023-10-11T15:44:07.302 DEBUG:boto:Token: None 2023-10-11T15:44:07.302 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj17 2023-10-11T15:44:07.303 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xzcHUgxn9YdjxsxVPlx4271Amc8= 2023-10-11T15:44:07.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xzcHUgxn9YdjxsxVPlx4271Amc8='} 2023-10-11T15:44:07.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f43ca72fa332906-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.309 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.309 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.310 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.310 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.310 DEBUG:boto:Data: 2023-10-11T15:44:07.311 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.311 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.312 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.312 DEBUG:boto:Params: {} 2023-10-11T15:44:07.312 DEBUG:boto:Token: None 2023-10-11T15:44:07.313 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.313 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d364f39943d96032-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.317 DEBUG:boto:path=/dghqvd-64/obj34 2023-10-11T15:44:07.317 DEBUG:boto:auth_path=/dghqvd-64/obj34 2023-10-11T15:44:07.318 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.318 DEBUG:boto:Path: /dghqvd-64/obj34 2023-10-11T15:44:07.319 DEBUG:boto:Data: 2023-10-11T15:44:07.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.319 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.320 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.320 DEBUG:boto:Params: {} 2023-10-11T15:44:07.321 DEBUG:boto:Token: None 2023-10-11T15:44:07.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj34 2023-10-11T15:44:07.322 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0/I7ZaT18bLnkdRXl19NQkxJkgI= 2023-10-11T15:44:07.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0/I7ZaT18bLnkdRXl19NQkxJkgI='} 2023-10-11T15:44:07.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb9e3483309e4cbe-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.328 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.328 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.329 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.329 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.329 DEBUG:boto:Data: 2023-10-11T15:44:07.330 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.331 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.331 DEBUG:boto:Params: {} 2023-10-11T15:44:07.331 DEBUG:boto:Token: None 2023-10-11T15:44:07.332 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d930c82ebe958e9-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.336 DEBUG:boto:path=/dghqvd-64/obj51 2023-10-11T15:44:07.336 DEBUG:boto:auth_path=/dghqvd-64/obj51 2023-10-11T15:44:07.337 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.337 DEBUG:boto:Path: /dghqvd-64/obj51 2023-10-11T15:44:07.338 DEBUG:boto:Data: 2023-10-11T15:44:07.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.339 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.339 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.339 DEBUG:boto:Params: {} 2023-10-11T15:44:07.340 DEBUG:boto:Token: None 2023-10-11T15:44:07.340 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj51 2023-10-11T15:44:07.341 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UTccSc6+L5BfbQHyi8w7Cxkjr/U= 2023-10-11T15:44:07.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UTccSc6+L5BfbQHyi8w7Cxkjr/U='} 2023-10-11T15:44:07.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba6cdfbf99fab9ea-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.387 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.388 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.388 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.388 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.389 DEBUG:boto:Data: 2023-10-11T15:44:07.389 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.390 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.390 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.390 DEBUG:boto:Params: {} 2023-10-11T15:44:07.391 DEBUG:boto:Token: None 2023-10-11T15:44:07.391 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.392 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bce69b7d1c3cf9bb-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.395 DEBUG:boto:path=/dghqvd-64/obj68 2023-10-11T15:44:07.396 DEBUG:boto:auth_path=/dghqvd-64/obj68 2023-10-11T15:44:07.396 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.396 DEBUG:boto:Path: /dghqvd-64/obj68 2023-10-11T15:44:07.397 DEBUG:boto:Data: 2023-10-11T15:44:07.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.398 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.398 DEBUG:boto:Params: {} 2023-10-11T15:44:07.399 DEBUG:boto:Token: None 2023-10-11T15:44:07.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj68 2023-10-11T15:44:07.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/aT/IY0IA2IQqUYg50GTK/rGTFs= 2023-10-11T15:44:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/aT/IY0IA2IQqUYg50GTK/rGTFs='} 2023-10-11T15:44:07.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035fd752bce3a1d65-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.406 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.406 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.407 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.407 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.408 DEBUG:boto:Data: 2023-10-11T15:44:07.408 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.408 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.409 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.409 DEBUG:boto:Params: {} 2023-10-11T15:44:07.410 DEBUG:boto:Token: None 2023-10-11T15:44:07.410 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.410 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000847bac2994db53ee-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.414 DEBUG:boto:path=/dghqvd-64/obj85 2023-10-11T15:44:07.414 DEBUG:boto:auth_path=/dghqvd-64/obj85 2023-10-11T15:44:07.415 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.415 DEBUG:boto:Path: /dghqvd-64/obj85 2023-10-11T15:44:07.416 DEBUG:boto:Data: 2023-10-11T15:44:07.416 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.417 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.417 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.418 DEBUG:boto:Params: {} 2023-10-11T15:44:07.418 DEBUG:boto:Token: None 2023-10-11T15:44:07.418 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj85 2023-10-11T15:44:07.419 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:C4wqv8z2XqNweWzNH4ugGDGA5zc= 2023-10-11T15:44:07.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C4wqv8z2XqNweWzNH4ugGDGA5zc='} 2023-10-11T15:44:07.467 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e32246fcb23f759a-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.469 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:07.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T15:44:07.522 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:07.522 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:07.566 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:07.566 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:07.571 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 10 2023-10-11T15:44:07.615 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.615 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.616 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.616 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.617 DEBUG:boto:Data: 2023-10-11T15:44:07.617 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.617 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.618 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.618 DEBUG:boto:Params: {} 2023-10-11T15:44:07.619 DEBUG:boto:Token: None 2023-10-11T15:44:07.619 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.620 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcb23927bec7d2e6-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.624 DEBUG:boto:path=/dghqvd-64/obj19 2023-10-11T15:44:07.624 DEBUG:boto:auth_path=/dghqvd-64/obj19 2023-10-11T15:44:07.625 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.625 DEBUG:boto:Path: /dghqvd-64/obj19 2023-10-11T15:44:07.626 DEBUG:boto:Data: 2023-10-11T15:44:07.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.626 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.627 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.627 DEBUG:boto:Params: {} 2023-10-11T15:44:07.628 DEBUG:boto:Token: None 2023-10-11T15:44:07.628 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj19 2023-10-11T15:44:07.629 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:xDY2irvpVjBpPJ/59QgB1wdaXNw= 2023-10-11T15:44:07.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDY2irvpVjBpPJ/59QgB1wdaXNw='} 2023-10-11T15:44:07.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf71e23bbdb33cf8-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.635 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.635 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.636 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.636 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.636 DEBUG:boto:Data: 2023-10-11T15:44:07.637 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.637 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.638 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.638 DEBUG:boto:Params: {} 2023-10-11T15:44:07.638 DEBUG:boto:Token: None 2023-10-11T15:44:07.639 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.639 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a023bfdd95867ffa-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.643 DEBUG:boto:path=/dghqvd-64/obj38 2023-10-11T15:44:07.643 DEBUG:boto:auth_path=/dghqvd-64/obj38 2023-10-11T15:44:07.644 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.644 DEBUG:boto:Path: /dghqvd-64/obj38 2023-10-11T15:44:07.644 DEBUG:boto:Data: 2023-10-11T15:44:07.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.645 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.646 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.646 DEBUG:boto:Params: {} 2023-10-11T15:44:07.647 DEBUG:boto:Token: None 2023-10-11T15:44:07.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj38 2023-10-11T15:44:07.647 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BiOSnukwVsxoOClFY031cIRxX/U= 2023-10-11T15:44:07.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BiOSnukwVsxoOClFY031cIRxX/U='} 2023-10-11T15:44:07.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad72778d364674db-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.691 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.692 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.692 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.692 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.693 DEBUG:boto:Data: 2023-10-11T15:44:07.693 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.694 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.694 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.694 DEBUG:boto:Params: {} 2023-10-11T15:44:07.695 DEBUG:boto:Token: None 2023-10-11T15:44:07.695 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.696 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082c9fad529bfd244-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.699 DEBUG:boto:path=/dghqvd-64/obj57 2023-10-11T15:44:07.700 DEBUG:boto:auth_path=/dghqvd-64/obj57 2023-10-11T15:44:07.700 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.701 DEBUG:boto:Path: /dghqvd-64/obj57 2023-10-11T15:44:07.701 DEBUG:boto:Data: 2023-10-11T15:44:07.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.702 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.702 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.703 DEBUG:boto:Params: {} 2023-10-11T15:44:07.703 DEBUG:boto:Token: None 2023-10-11T15:44:07.703 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj57 2023-10-11T15:44:07.704 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:SfpVt1qs+Qby8DJj77kBAi8GN5g= 2023-10-11T15:44:07.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SfpVt1qs+Qby8DJj77kBAi8GN5g='} 2023-10-11T15:44:07.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6651f45f83fab1b-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.709 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.709 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.710 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.710 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.710 DEBUG:boto:Data: 2023-10-11T15:44:07.711 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.711 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.712 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.712 DEBUG:boto:Params: {} 2023-10-11T15:44:07.712 DEBUG:boto:Token: None 2023-10-11T15:44:07.713 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.713 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000314e22a13c522f3-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.717 DEBUG:boto:path=/dghqvd-64/obj76 2023-10-11T15:44:07.717 DEBUG:boto:auth_path=/dghqvd-64/obj76 2023-10-11T15:44:07.718 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.718 DEBUG:boto:Path: /dghqvd-64/obj76 2023-10-11T15:44:07.718 DEBUG:boto:Data: 2023-10-11T15:44:07.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.719 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.720 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.720 DEBUG:boto:Params: {} 2023-10-11T15:44:07.720 DEBUG:boto:Token: None 2023-10-11T15:44:07.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj76 2023-10-11T15:44:07.721 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:L8NPBDo51NLZZgT3+ETDgaEWX60= 2023-10-11T15:44:07.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L8NPBDo51NLZZgT3+ETDgaEWX60='} 2023-10-11T15:44:07.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8f9222554808ecc-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.767 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.768 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.768 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.768 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.769 DEBUG:boto:Data: 2023-10-11T15:44:07.769 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.770 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.770 DEBUG:boto:Params: {} 2023-10-11T15:44:07.771 DEBUG:boto:Token: None 2023-10-11T15:44:07.771 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004695ccec0f2b4f37-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.775 DEBUG:boto:path=/dghqvd-64/obj95 2023-10-11T15:44:07.776 DEBUG:boto:auth_path=/dghqvd-64/obj95 2023-10-11T15:44:07.776 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.776 DEBUG:boto:Path: /dghqvd-64/obj95 2023-10-11T15:44:07.777 DEBUG:boto:Data: 2023-10-11T15:44:07.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.778 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.778 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.779 DEBUG:boto:Params: {} 2023-10-11T15:44:07.779 DEBUG:boto:Token: None 2023-10-11T15:44:07.779 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj95 2023-10-11T15:44:07.780 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wJklscFLXwOh+1oltP6D2Mx96YE= 2023-10-11T15:44:07.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wJklscFLXwOh+1oltP6D2Mx96YE='} 2023-10-11T15:44:07.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0150af8aab51722-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:07.786 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:07.838 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:07.839 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:07.883 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:07.883 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:07.890 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 15 2023-10-11T15:44:07.938 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:07.939 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:07.939 DEBUG:boto:Method: HEAD 2023-10-11T15:44:07.939 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:07.940 DEBUG:boto:Data: 2023-10-11T15:44:07.940 DEBUG:boto:Headers: {} 2023-10-11T15:44:07.940 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.941 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.941 DEBUG:boto:Params: {} 2023-10-11T15:44:07.942 DEBUG:boto:Token: None 2023-10-11T15:44:07.942 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/ 2023-10-11T15:44:07.943 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T15:44:07.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T15:44:07.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adbd91bfba63df41-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.947 DEBUG:boto:path=/dghqvd-64/obj23 2023-10-11T15:44:07.947 DEBUG:boto:auth_path=/dghqvd-64/obj23 2023-10-11T15:44:07.948 DEBUG:boto:Method: PUT 2023-10-11T15:44:07.948 DEBUG:boto:Path: /dghqvd-64/obj23 2023-10-11T15:44:07.949 DEBUG:boto:Data: 2023-10-11T15:44:07.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:07.949 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:07.950 DEBUG:boto:Port: 8000 2023-10-11T15:44:07.950 DEBUG:boto:Params: {} 2023-10-11T15:44:07.951 DEBUG:boto:Token: None 2023-10-11T15:44:07.951 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:07 GMT /dghqvd-64/obj23 2023-10-11T15:44:07.952 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jUbqLGqASHIF0pWZyooI9hylkVY= 2023-10-11T15:44:07.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jUbqLGqASHIF0pWZyooI9hylkVY='} 2023-10-11T15:44:07.999 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f613fa9f5b21fa1-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:07.999 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.000 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.000 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.000 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.001 DEBUG:boto:Data: 2023-10-11T15:44:08.001 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.002 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.002 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.002 DEBUG:boto:Params: {} 2023-10-11T15:44:08.003 DEBUG:boto:Token: None 2023-10-11T15:44:08.003 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.004 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021a5f09d24188a31-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.008 DEBUG:boto:path=/dghqvd-64/obj46 2023-10-11T15:44:08.008 DEBUG:boto:auth_path=/dghqvd-64/obj46 2023-10-11T15:44:08.009 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.009 DEBUG:boto:Path: /dghqvd-64/obj46 2023-10-11T15:44:08.010 DEBUG:boto:Data: 2023-10-11T15:44:08.010 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.011 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.011 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.011 DEBUG:boto:Params: {} 2023-10-11T15:44:08.012 DEBUG:boto:Token: None 2023-10-11T15:44:08.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj46 2023-10-11T15:44:08.013 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ks9DRLNx/ZC8LKu93PYaHt7mYUc= 2023-10-11T15:44:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ks9DRLNx/ZC8LKu93PYaHt7mYUc='} 2023-10-11T15:44:08.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007997910d350b9b5e-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.019 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.019 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.020 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.020 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.021 DEBUG:boto:Data: 2023-10-11T15:44:08.021 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.022 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.022 DEBUG:boto:Params: {} 2023-10-11T15:44:08.023 DEBUG:boto:Token: None 2023-10-11T15:44:08.023 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.024 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003501574f8211a525-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.027 DEBUG:boto:path=/dghqvd-64/obj69 2023-10-11T15:44:08.028 DEBUG:boto:auth_path=/dghqvd-64/obj69 2023-10-11T15:44:08.028 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.029 DEBUG:boto:Path: /dghqvd-64/obj69 2023-10-11T15:44:08.029 DEBUG:boto:Data: 2023-10-11T15:44:08.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.030 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.030 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.031 DEBUG:boto:Params: {} 2023-10-11T15:44:08.031 DEBUG:boto:Token: None 2023-10-11T15:44:08.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj69 2023-10-11T15:44:08.032 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Rkyq4POxmIeR9ewPiAD/bmZGjDI= 2023-10-11T15:44:08.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rkyq4POxmIeR9ewPiAD/bmZGjDI='} 2023-10-11T15:44:08.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023f19cf187279a79-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.075 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.076 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.076 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.076 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.077 DEBUG:boto:Data: 2023-10-11T15:44:08.077 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.078 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.078 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.078 DEBUG:boto:Params: {} 2023-10-11T15:44:08.079 DEBUG:boto:Token: None 2023-10-11T15:44:08.079 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.080 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8e794c21413a4a4-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.086 DEBUG:boto:path=/dghqvd-64/obj92 2023-10-11T15:44:08.086 DEBUG:boto:auth_path=/dghqvd-64/obj92 2023-10-11T15:44:08.087 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.087 DEBUG:boto:Path: /dghqvd-64/obj92 2023-10-11T15:44:08.088 DEBUG:boto:Data: 2023-10-11T15:44:08.088 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.089 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.089 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.089 DEBUG:boto:Params: {} 2023-10-11T15:44:08.090 DEBUG:boto:Token: None 2023-10-11T15:44:08.090 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj92 2023-10-11T15:44:08.091 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sj/F5CrY4WFQjwS42isQxV9Lqxc= 2023-10-11T15:44:08.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sj/F5CrY4WFQjwS42isQxV9Lqxc='} 2023-10-11T15:44:08.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028fffae5d762a430-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.097 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.097 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.098 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.098 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.099 DEBUG:boto:Data: 2023-10-11T15:44:08.099 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.099 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.100 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.100 DEBUG:boto:Params: {} 2023-10-11T15:44:08.101 DEBUG:boto:Token: None 2023-10-11T15:44:08.101 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.101 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85c5a070cc76be9-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.105 DEBUG:boto:path=/dghqvd-64/obj115 2023-10-11T15:44:08.105 DEBUG:boto:auth_path=/dghqvd-64/obj115 2023-10-11T15:44:08.106 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.106 DEBUG:boto:Path: /dghqvd-64/obj115 2023-10-11T15:44:08.107 DEBUG:boto:Data: 2023-10-11T15:44:08.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.107 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.108 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.108 DEBUG:boto:Params: {} 2023-10-11T15:44:08.109 DEBUG:boto:Token: None 2023-10-11T15:44:08.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj115 2023-10-11T15:44:08.110 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:0RQ94yxt/NVAJhUPz17J0tjob/Y= 2023-10-11T15:44:08.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0RQ94yxt/NVAJhUPz17J0tjob/Y='} 2023-10-11T15:44:08.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069d986a168c74400-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.156 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:08.209 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:08.209 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:08.257 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:08.258 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:08.265 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 20 2023-10-11T15:44:08.318 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.319 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.319 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.320 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.320 DEBUG:boto:Data: 2023-10-11T15:44:08.320 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.321 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.321 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.322 DEBUG:boto:Params: {} 2023-10-11T15:44:08.322 DEBUG:boto:Token: None 2023-10-11T15:44:08.323 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7434fe44b5a8341-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.327 DEBUG:boto:path=/dghqvd-64/obj29 2023-10-11T15:44:08.328 DEBUG:boto:auth_path=/dghqvd-64/obj29 2023-10-11T15:44:08.328 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.329 DEBUG:boto:Path: /dghqvd-64/obj29 2023-10-11T15:44:08.329 DEBUG:boto:Data: 2023-10-11T15:44:08.329 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.330 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.330 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.331 DEBUG:boto:Params: {} 2023-10-11T15:44:08.331 DEBUG:boto:Token: None 2023-10-11T15:44:08.332 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj29 2023-10-11T15:44:08.332 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Fd88ITtd5i26KvcmHdGc/uUsIwU= 2023-10-11T15:44:08.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Fd88ITtd5i26KvcmHdGc/uUsIwU='} 2023-10-11T15:44:08.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001be3a08a0af36425-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.339 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.339 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.339 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.340 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.340 DEBUG:boto:Data: 2023-10-11T15:44:08.341 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.341 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.341 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.342 DEBUG:boto:Params: {} 2023-10-11T15:44:08.342 DEBUG:boto:Token: None 2023-10-11T15:44:08.343 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.343 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de765108d9c50d5-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.347 DEBUG:boto:path=/dghqvd-64/obj58 2023-10-11T15:44:08.348 DEBUG:boto:auth_path=/dghqvd-64/obj58 2023-10-11T15:44:08.348 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.349 DEBUG:boto:Path: /dghqvd-64/obj58 2023-10-11T15:44:08.349 DEBUG:boto:Data: 2023-10-11T15:44:08.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.350 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.350 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.351 DEBUG:boto:Params: {} 2023-10-11T15:44:08.351 DEBUG:boto:Token: None 2023-10-11T15:44:08.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj58 2023-10-11T15:44:08.352 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9q7MUbxxTuVjeQkvLR5v3odc/wk= 2023-10-11T15:44:08.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9q7MUbxxTuVjeQkvLR5v3odc/wk='} 2023-10-11T15:44:08.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033f3ba0abca820c4-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.395 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.396 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.396 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.396 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.397 DEBUG:boto:Data: 2023-10-11T15:44:08.397 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.398 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.398 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.398 DEBUG:boto:Params: {} 2023-10-11T15:44:08.399 DEBUG:boto:Token: None 2023-10-11T15:44:08.399 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.400 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008be039eb2c7659f2-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.404 DEBUG:boto:path=/dghqvd-64/obj87 2023-10-11T15:44:08.405 DEBUG:boto:auth_path=/dghqvd-64/obj87 2023-10-11T15:44:08.405 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.405 DEBUG:boto:Path: /dghqvd-64/obj87 2023-10-11T15:44:08.406 DEBUG:boto:Data: 2023-10-11T15:44:08.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.407 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.408 DEBUG:boto:Params: {} 2023-10-11T15:44:08.408 DEBUG:boto:Token: None 2023-10-11T15:44:08.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj87 2023-10-11T15:44:08.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KBvwYDkRmYXM76VoJq59VY5SKvs= 2023-10-11T15:44:08.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KBvwYDkRmYXM76VoJq59VY5SKvs='} 2023-10-11T15:44:08.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000332dcbeeba98f7e1-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.415 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.416 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.416 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.417 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.417 DEBUG:boto:Data: 2023-10-11T15:44:08.417 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.418 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.418 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.419 DEBUG:boto:Params: {} 2023-10-11T15:44:08.419 DEBUG:boto:Token: None 2023-10-11T15:44:08.419 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.420 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b301f4813eb1b6bd-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.424 DEBUG:boto:path=/dghqvd-64/obj116 2023-10-11T15:44:08.425 DEBUG:boto:auth_path=/dghqvd-64/obj116 2023-10-11T15:44:08.425 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.426 DEBUG:boto:Path: /dghqvd-64/obj116 2023-10-11T15:44:08.426 DEBUG:boto:Data: 2023-10-11T15:44:08.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.427 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.427 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.428 DEBUG:boto:Params: {} 2023-10-11T15:44:08.428 DEBUG:boto:Token: None 2023-10-11T15:44:08.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj116 2023-10-11T15:44:08.429 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y3ew7+SM57YQ8FB/vRVLCW2NcfM= 2023-10-11T15:44:08.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y3ew7+SM57YQ8FB/vRVLCW2NcfM='} 2023-10-11T15:44:08.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013a25839401342e9-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.475 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.476 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.476 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.476 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.477 DEBUG:boto:Data: 2023-10-11T15:44:08.477 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.478 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.478 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.478 DEBUG:boto:Params: {} 2023-10-11T15:44:08.479 DEBUG:boto:Token: None 2023-10-11T15:44:08.479 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.480 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f66c436421cb75ce-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.484 DEBUG:boto:path=/dghqvd-64/obj145 2023-10-11T15:44:08.485 DEBUG:boto:auth_path=/dghqvd-64/obj145 2023-10-11T15:44:08.486 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.486 DEBUG:boto:Path: /dghqvd-64/obj145 2023-10-11T15:44:08.486 DEBUG:boto:Data: 2023-10-11T15:44:08.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.487 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.488 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.488 DEBUG:boto:Params: {} 2023-10-11T15:44:08.488 DEBUG:boto:Token: None 2023-10-11T15:44:08.489 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj145 2023-10-11T15:44:08.489 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3/orUfb6W8J5qutfSmxPLfWM6Nk= 2023-10-11T15:44:08.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3/orUfb6W8J5qutfSmxPLfWM6Nk='} 2023-10-11T15:44:08.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a540907fcf400d-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.496 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:08.550 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:08.550 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:08.601 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:08.601 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:08.611 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 25 2023-10-11T15:44:08.668 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.669 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.669 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.670 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.670 DEBUG:boto:Data: 2023-10-11T15:44:08.670 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.671 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.671 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.672 DEBUG:boto:Params: {} 2023-10-11T15:44:08.672 DEBUG:boto:Token: None 2023-10-11T15:44:08.673 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.673 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988e9c5f1340387c-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.678 DEBUG:boto:path=/dghqvd-64/obj31 2023-10-11T15:44:08.678 DEBUG:boto:auth_path=/dghqvd-64/obj31 2023-10-11T15:44:08.679 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.679 DEBUG:boto:Path: /dghqvd-64/obj31 2023-10-11T15:44:08.680 DEBUG:boto:Data: 2023-10-11T15:44:08.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.680 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.681 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.681 DEBUG:boto:Params: {} 2023-10-11T15:44:08.682 DEBUG:boto:Token: None 2023-10-11T15:44:08.682 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj31 2023-10-11T15:44:08.683 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FQZh3MA03NdPY7ZNFpX3069JerQ= 2023-10-11T15:44:08.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FQZh3MA03NdPY7ZNFpX3069JerQ='} 2023-10-11T15:44:08.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa2bb7852f902fd6-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.689 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.690 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.690 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.690 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.691 DEBUG:boto:Data: 2023-10-11T15:44:08.691 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.692 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.692 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.693 DEBUG:boto:Params: {} 2023-10-11T15:44:08.693 DEBUG:boto:Token: None 2023-10-11T15:44:08.694 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.694 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee4e963c199be7eb-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.699 DEBUG:boto:path=/dghqvd-64/obj62 2023-10-11T15:44:08.699 DEBUG:boto:auth_path=/dghqvd-64/obj62 2023-10-11T15:44:08.699 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.700 DEBUG:boto:Path: /dghqvd-64/obj62 2023-10-11T15:44:08.700 DEBUG:boto:Data: 2023-10-11T15:44:08.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.701 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.701 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.702 DEBUG:boto:Params: {} 2023-10-11T15:44:08.702 DEBUG:boto:Token: None 2023-10-11T15:44:08.703 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj62 2023-10-11T15:44:08.703 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2uHshzFVmfNcFv/N0B3Y4pfSEjo= 2023-10-11T15:44:08.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2uHshzFVmfNcFv/N0B3Y4pfSEjo='} 2023-10-11T15:44:08.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005aa0aa2ce9879b90-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.747 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.748 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.748 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.749 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.749 DEBUG:boto:Data: 2023-10-11T15:44:08.749 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.750 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.750 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.751 DEBUG:boto:Params: {} 2023-10-11T15:44:08.751 DEBUG:boto:Token: None 2023-10-11T15:44:08.752 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.752 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51be8e2424051ff-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.756 DEBUG:boto:path=/dghqvd-64/obj93 2023-10-11T15:44:08.757 DEBUG:boto:auth_path=/dghqvd-64/obj93 2023-10-11T15:44:08.757 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.757 DEBUG:boto:Path: /dghqvd-64/obj93 2023-10-11T15:44:08.758 DEBUG:boto:Data: 2023-10-11T15:44:08.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.759 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.759 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.760 DEBUG:boto:Params: {} 2023-10-11T15:44:08.760 DEBUG:boto:Token: None 2023-10-11T15:44:08.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj93 2023-10-11T15:44:08.761 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nItTpqznpQ/xmkCXdEHy9ETyKjY= 2023-10-11T15:44:08.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nItTpqznpQ/xmkCXdEHy9ETyKjY='} 2023-10-11T15:44:08.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072f462ca4933ec31-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.768 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.768 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.768 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.769 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.769 DEBUG:boto:Data: 2023-10-11T15:44:08.770 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.770 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.771 DEBUG:boto:Params: {} 2023-10-11T15:44:08.771 DEBUG:boto:Token: None 2023-10-11T15:44:08.772 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.772 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b039be01fceeb487-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.777 DEBUG:boto:path=/dghqvd-64/obj124 2023-10-11T15:44:08.777 DEBUG:boto:auth_path=/dghqvd-64/obj124 2023-10-11T15:44:08.778 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.778 DEBUG:boto:Path: /dghqvd-64/obj124 2023-10-11T15:44:08.779 DEBUG:boto:Data: 2023-10-11T15:44:08.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.779 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.780 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.780 DEBUG:boto:Params: {} 2023-10-11T15:44:08.781 DEBUG:boto:Token: None 2023-10-11T15:44:08.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj124 2023-10-11T15:44:08.782 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Kz56knOUqYm9Kn0Nv98WWwwvZNk= 2023-10-11T15:44:08.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kz56knOUqYm9Kn0Nv98WWwwvZNk='} 2023-10-11T15:44:08.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b46c68dba1183ae-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.827 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:08.828 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:08.828 DEBUG:boto:Method: HEAD 2023-10-11T15:44:08.828 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:08.829 DEBUG:boto:Data: 2023-10-11T15:44:08.829 DEBUG:boto:Headers: {} 2023-10-11T15:44:08.830 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.830 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.831 DEBUG:boto:Params: {} 2023-10-11T15:44:08.831 DEBUG:boto:Token: None 2023-10-11T15:44:08.832 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/ 2023-10-11T15:44:08.832 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T15:44:08.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T15:44:08.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004562046336953b61-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.837 DEBUG:boto:path=/dghqvd-64/obj155 2023-10-11T15:44:08.837 DEBUG:boto:auth_path=/dghqvd-64/obj155 2023-10-11T15:44:08.837 DEBUG:boto:Method: PUT 2023-10-11T15:44:08.838 DEBUG:boto:Path: /dghqvd-64/obj155 2023-10-11T15:44:08.838 DEBUG:boto:Data: 2023-10-11T15:44:08.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:08.839 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:08.840 DEBUG:boto:Port: 8000 2023-10-11T15:44:08.840 DEBUG:boto:Params: {} 2023-10-11T15:44:08.840 DEBUG:boto:Token: None 2023-10-11T15:44:08.841 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:08 GMT /dghqvd-64/obj155 2023-10-11T15:44:08.841 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BLKRE922d5bITrWXXnSzD7E9fZA= 2023-10-11T15:44:08.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLKRE922d5bITrWXXnSzD7E9fZA='} 2023-10-11T15:44:08.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aefea83d65b6daf7-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:08.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:08.849 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:08.901 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:08.901 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:08.952 INFO:teuthology.orchestra.run.smithi158.stdout:tenant: 2023-10-11T15:44:08.952 INFO:teuthology.orchestra.run.smithi158.stdout:bucket name: dghqvd-64 2023-10-11T15:44:08.962 INFO:teuthology.orchestra.run.smithi158.stdout:total entries: 30 2023-10-11T15:44:09.024 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:09.025 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:09.025 DEBUG:boto:Method: HEAD 2023-10-11T15:44:09.026 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:09.026 DEBUG:boto:Data: 2023-10-11T15:44:09.026 DEBUG:boto:Headers: {} 2023-10-11T15:44:09.027 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.027 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.028 DEBUG:boto:Params: {} 2023-10-11T15:44:09.028 DEBUG:boto:Token: None 2023-10-11T15:44:09.029 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/ 2023-10-11T15:44:09.029 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T15:44:09.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T15:44:09.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c2576370e5d957-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.035 DEBUG:boto:path=/dghqvd-64/obj37 2023-10-11T15:44:09.035 DEBUG:boto:auth_path=/dghqvd-64/obj37 2023-10-11T15:44:09.035 DEBUG:boto:Method: PUT 2023-10-11T15:44:09.036 DEBUG:boto:Path: /dghqvd-64/obj37 2023-10-11T15:44:09.036 DEBUG:boto:Data: 2023-10-11T15:44:09.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:09.037 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.038 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.038 DEBUG:boto:Params: {} 2023-10-11T15:44:09.039 DEBUG:boto:Token: None 2023-10-11T15:44:09.039 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/obj37 2023-10-11T15:44:09.039 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iI8Mv8BrHRNR8zndcTT7EyHEcXU= 2023-10-11T15:44:09.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iI8Mv8BrHRNR8zndcTT7EyHEcXU='} 2023-10-11T15:44:09.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009592dfd765025fc2-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.046 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:09.046 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:09.047 DEBUG:boto:Method: HEAD 2023-10-11T15:44:09.047 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:09.048 DEBUG:boto:Data: 2023-10-11T15:44:09.048 DEBUG:boto:Headers: {} 2023-10-11T15:44:09.048 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.049 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.049 DEBUG:boto:Params: {} 2023-10-11T15:44:09.050 DEBUG:boto:Token: None 2023-10-11T15:44:09.050 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/ 2023-10-11T15:44:09.051 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T15:44:09.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T15:44:09.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef94b821db3a67b6-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.055 DEBUG:boto:path=/dghqvd-64/obj74 2023-10-11T15:44:09.056 DEBUG:boto:auth_path=/dghqvd-64/obj74 2023-10-11T15:44:09.056 DEBUG:boto:Method: PUT 2023-10-11T15:44:09.057 DEBUG:boto:Path: /dghqvd-64/obj74 2023-10-11T15:44:09.057 DEBUG:boto:Data: 2023-10-11T15:44:09.057 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:09.058 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.058 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.059 DEBUG:boto:Params: {} 2023-10-11T15:44:09.059 DEBUG:boto:Token: None 2023-10-11T15:44:09.059 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/obj74 2023-10-11T15:44:09.060 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/nsZIBohCzBYRF3MAaqfSsEoMzs= 2023-10-11T15:44:09.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/nsZIBohCzBYRF3MAaqfSsEoMzs='} 2023-10-11T15:44:09.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d33c200753c881d4-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.103 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:09.104 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:09.104 DEBUG:boto:Method: HEAD 2023-10-11T15:44:09.104 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:09.105 DEBUG:boto:Data: 2023-10-11T15:44:09.105 DEBUG:boto:Headers: {} 2023-10-11T15:44:09.106 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.106 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.106 DEBUG:boto:Params: {} 2023-10-11T15:44:09.107 DEBUG:boto:Token: None 2023-10-11T15:44:09.107 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/ 2023-10-11T15:44:09.108 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T15:44:09.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T15:44:09.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eabe3be6b3a8070c-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.113 DEBUG:boto:path=/dghqvd-64/obj111 2023-10-11T15:44:09.113 DEBUG:boto:auth_path=/dghqvd-64/obj111 2023-10-11T15:44:09.113 DEBUG:boto:Method: PUT 2023-10-11T15:44:09.114 DEBUG:boto:Path: /dghqvd-64/obj111 2023-10-11T15:44:09.114 DEBUG:boto:Data: 2023-10-11T15:44:09.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:09.115 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.116 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.116 DEBUG:boto:Params: {} 2023-10-11T15:44:09.116 DEBUG:boto:Token: None 2023-10-11T15:44:09.117 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/obj111 2023-10-11T15:44:09.117 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2xSlVctOxefxZEBs56CuB2kIM58= 2023-10-11T15:44:09.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2xSlVctOxefxZEBs56CuB2kIM58='} 2023-10-11T15:44:09.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aae6300131353ab-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.123 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:09.124 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:09.124 DEBUG:boto:Method: HEAD 2023-10-11T15:44:09.125 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:09.125 DEBUG:boto:Data: 2023-10-11T15:44:09.125 DEBUG:boto:Headers: {} 2023-10-11T15:44:09.126 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.126 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.127 DEBUG:boto:Params: {} 2023-10-11T15:44:09.127 DEBUG:boto:Token: None 2023-10-11T15:44:09.127 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/ 2023-10-11T15:44:09.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T15:44:09.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T15:44:09.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089f8d8bab8d86e79-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.132 DEBUG:boto:path=/dghqvd-64/obj148 2023-10-11T15:44:09.133 DEBUG:boto:auth_path=/dghqvd-64/obj148 2023-10-11T15:44:09.133 DEBUG:boto:Method: PUT 2023-10-11T15:44:09.134 DEBUG:boto:Path: /dghqvd-64/obj148 2023-10-11T15:44:09.134 DEBUG:boto:Data: 2023-10-11T15:44:09.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:09.135 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.135 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.136 DEBUG:boto:Params: {} 2023-10-11T15:44:09.136 DEBUG:boto:Token: None 2023-10-11T15:44:09.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/obj148 2023-10-11T15:44:09.137 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gR0zFmjb60AwRkuBSM2qElU98HA= 2023-10-11T15:44:09.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gR0zFmjb60AwRkuBSM2qElU98HA='} 2023-10-11T15:44:09.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd3d43df4d747087-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.187 DEBUG:boto:path=/dghqvd-64/ 2023-10-11T15:44:09.188 DEBUG:boto:auth_path=/dghqvd-64/ 2023-10-11T15:44:09.188 DEBUG:boto:Method: HEAD 2023-10-11T15:44:09.188 DEBUG:boto:Path: /dghqvd-64/ 2023-10-11T15:44:09.189 DEBUG:boto:Data: 2023-10-11T15:44:09.189 DEBUG:boto:Headers: {} 2023-10-11T15:44:09.190 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.190 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.191 DEBUG:boto:Params: {} 2023-10-11T15:44:09.191 DEBUG:boto:Token: None 2023-10-11T15:44:09.191 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/ 2023-10-11T15:44:09.192 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T15:44:09.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T15:44:09.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d386a66b0c18456-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.196 DEBUG:boto:path=/dghqvd-64/obj185 2023-10-11T15:44:09.197 DEBUG:boto:auth_path=/dghqvd-64/obj185 2023-10-11T15:44:09.197 DEBUG:boto:Method: PUT 2023-10-11T15:44:09.198 DEBUG:boto:Path: /dghqvd-64/obj185 2023-10-11T15:44:09.198 DEBUG:boto:Data: 2023-10-11T15:44:09.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T15:44:09.199 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:44:09.199 DEBUG:boto:Port: 8000 2023-10-11T15:44:09.200 DEBUG:boto:Params: {} 2023-10-11T15:44:09.201 DEBUG:boto:Token: None 2023-10-11T15:44:09.201 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 11 Oct 2023 15:44:09 GMT /dghqvd-64/obj185 2023-10-11T15:44:09.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MTSDNRnp7yz0nikjZBHwl4pVGmI= 2023-10-11T15:44:09.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSDNRnp7yz0nikjZBHwl4pVGmI='} 2023-10-11T15:44:09.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003daaa8562779af0e-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:44:09.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:09.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:44:09.208 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:44:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:09.259 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:09.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:09.289+0000 7f0df07b6780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-11T15:44:09.290 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:09.290+0000 7f0df07b6780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-11T15:44:09.312 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-10-11T15:44:09.312 DEBUG:teuthology.orchestra.run.smithi158:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T15:44:09.315 INFO:tasks.rgw.c1.client.1:Started 2023-10-11T15:44:09.315 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:09.336 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:44:09.336 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:44:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-11T15:44:09.338 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T15:44:10.341 DEBUG:teuthology.orchestra.run.smithi158:> curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T15:44:10.364 INFO:teuthology.orchestra.run.smithi158.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T15:44:10.364 INFO:teuthology.orchestra.run.smithi158.stderr: Dload Upload Total Spent Left Speed 2023-10-11T15:44:10.365 INFO:teuthology.orchestra.run.smithi158.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T15:44:10.366 INFO:teuthology.orchestra.run.smithi158.stdout:anonymous 2023-10-11T15:44:10.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dghqvd-64 2023-10-11T15:44:10.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:44:10.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T15:44:10.369 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T15:44:10.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:44:10.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:44:10.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:10.501+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:15.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:15.502+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:20.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:20.502+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:25.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:25.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:30.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:30.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:35.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:35.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:40.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:40.504+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:45.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:45.504+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:50.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:50.505+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:44:55.507 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:44:55.506+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:00.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:00.506+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:05.508 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:05.507+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:10.509 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:10.508+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:15.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:15.509+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:20.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:20.509+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:25.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:25.510+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:30.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:30.510+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:35.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:35.511+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:40.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:40.511+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:45.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:45.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:50.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:50.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:45:55.514 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:45:55.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:46:00.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:00.513+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:46:05.515 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:05.513+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:46:10.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:10.514+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T15:46:15.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:15.514+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:20.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:20.515+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:25.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:25.516+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:30.518 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:30.517+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:35.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:35.518+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:40.519 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:40.518+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:45.520 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:45.519+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:50.521 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:50.519+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:46:55.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:46:55.520+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:00.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:00.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:05.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:05.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:10.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:10.522+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:15.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:15.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:20.524 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:20.522+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:25.525 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:25.523+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:30.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:30.523+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:35.526 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:35.524+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:40.527 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:40.525+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:45.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:45.526+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:50.528 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:50.527+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:47:55.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:47:55.527+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:00.529 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:00.528+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:05.530 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:05.528+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:10.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:10.529+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:15.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:15.530+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:20.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:20.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:25.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:25.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:30.533 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:30.532+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:35.534 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:35.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:40.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:40.532+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:45.535 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:45.533+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:50.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:50.533+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:48:55.536 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:48:55.534+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:49:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:00.535+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T15:49:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:05.536+0000 7f6d581d5780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-11T15:49:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:05.536+0000 7f6d581d5780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T15:49:05.538 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:05.536+0000 7f6d581d5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T15:49:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-10-11T15:49:05.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-10-11T15:49:05.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:05.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:05.563 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:05.618 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:05.619 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:05.651 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:49:05.665 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:49:05.665 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.666 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:05.666 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:49:05.666 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.666 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:05.667 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:49:05.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.667 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:05.668 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:49:05.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.668 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.668 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:05.668 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:08.970763Z" 2023-10-11T15:49:05.669 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.669 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:49:05.670 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:05.671 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:49:05.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:05.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:05.672 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:05.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:05.723 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:05.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:05.749+0000 7f64225c6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:05.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:05.750+0000 7f64225c6780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:05.765 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:05.765 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:05.765 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:05.765 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:05.766 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:05.766 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:05.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:05.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.767 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:05.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:05.767 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:05.767 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:05.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:05.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:05.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:05.768 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:05.768 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:05.769 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:05.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:05.770 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:05.770 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:05.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:05.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:05.771 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:05.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:05.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:05.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:05.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:05.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:05.772 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:05.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:05.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:05.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:05.773 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:05.774 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:05.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:05.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.774 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:05.775 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:05.775 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:05.775 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:05.775 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:05.775 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:05.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:05.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:05.776 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:05.776 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:05.776 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:05.777 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:05.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.777 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:05.777 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:05.778 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:05.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:05.778 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:05Z" 2023-10-11T15:49:05.778 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:05.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:05Z" } 2023-10-11T15:49:05.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:05.781 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:05.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:05.781 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:49:05.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.783 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:05.828 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:05.829 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:05.872 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T15:49:05.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.874 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:05.923 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:05.923 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:05.970 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T15:49:05.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:05.971 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.023 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.070 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:06.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.072 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.120 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.164 INFO:teuthology.orchestra.run.smithi158.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-11T15:49:06.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.165 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.258 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:06.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.260 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.361 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:06.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.363 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.419 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.419 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.464 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:06.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.466 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.520 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.520 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.565 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:49:06.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.567 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.615 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.616 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.659 INFO:teuthology.orchestra.run.smithi158.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-10-11T15:49:06.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:06.661 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:06.710 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:06.710 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:06.763 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:06.763 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:06.764 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 13, 2023-10-11T15:49:06.764 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:49:06.764 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:49:06.765 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:49:06.765 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:06.765 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:49:06.765 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.766 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:49:06.766 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:49:06.766 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:49:06.766 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:49:06.767 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.767 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:49:06.767 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.767 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:49:06.768 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:49:06.768 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:49:06.768 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:49:06.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.769 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:49:06.769 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:49:06.769 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.770 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:49:06.770 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.770 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:49:06.770 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:49:06.771 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:49:06.771 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:49:06.771 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:49:06.772 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:49:06.772 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:49:06.772 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:49:06.772 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:49:06.773 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:49:06.773 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.773 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.774 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.774 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:49:06.774 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:49:06.775 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.775 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:49:06.775 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.775 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:49:06.776 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:49:06.776 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:49:06.776 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:49:06.776 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:49:06.777 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:49:06.777 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:49:06.777 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:49:06.777 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:49:06.778 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:49:06.778 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.779 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.779 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.779 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.779 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:49:06.780 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.780 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:49:06.780 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:49:06.781 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:49:06.781 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:49:06.781 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.781 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.781 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.782 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:49:06.782 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:49:06.782 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:49:06.782 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T15:49:06.782 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.783 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T15:49:06.783 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T15:49:06.783 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T15:49:06.783 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.783 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T15:49:06.784 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:49:06.784 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:49:06.784 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:49:06.784 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.785 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.785 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.785 INFO:teuthology.orchestra.run.smithi158.stdout: "directional": [ 2023-10-11T15:49:06.785 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.786 INFO:teuthology.orchestra.run.smithi158.stdout: "source_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:49:06.786 INFO:teuthology.orchestra.run.smithi158.stdout: "dest_zone": "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T15:49:06.786 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.786 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.786 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.787 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T15:49:06.787 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.787 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T15:49:06.787 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:49:06.788 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:49:06.788 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:49:06.788 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:49:06.788 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:49:06.788 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.789 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.789 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T15:49:06.789 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:49:06.789 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:49:06.789 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:49:06.790 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:49:06.790 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.790 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.791 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T15:49:06.791 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:49:06.791 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T15:49:06.792 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T15:49:06.792 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.792 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.792 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T15:49:06.793 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T15:49:06.793 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T15:49:06.793 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T15:49:06.793 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.794 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.794 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.794 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T15:49:06.794 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.795 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.795 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.795 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:49:06.795 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.796 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.797 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.797 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.797 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:49:06.798 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:49:06.798 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:49:06.798 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:49:06.798 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.799 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:49:06.799 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.799 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:49:06.799 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:49:06.800 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:49:06.800 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:49:06.800 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.800 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:49:06.801 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:49:06.801 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.801 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:49:06.801 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.802 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:49:06.802 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:49:06.802 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:49:06.802 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:49:06.803 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:49:06.803 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:49:06.803 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:49:06.803 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:49:06.804 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:49:06.804 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:49:06.804 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.804 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.805 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.805 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.805 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:49:06.805 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:49:06.806 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:49:06.806 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:49:06.806 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.807 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:49:06.807 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:49:06.807 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:49:06.807 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:49:06.808 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:49:06.808 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:49:06.808 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:49:06.808 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:49:06.809 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:49:06.809 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:49:06.809 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.809 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.809 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.810 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.810 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:49:06.810 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.810 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:49:06.810 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:49:06.811 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:49:06.811 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:49:06.811 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.811 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.811 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.812 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:49:06.812 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:49:06.812 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:49:06.812 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:49:06.812 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.813 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:49:06.814 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:49:06.814 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.814 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.815 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:49:06.815 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:49:06.815 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.815 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:49:06.815 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:49:06.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.816 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.816 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:49:06.816 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:49:06.817 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.817 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.817 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:49:06.817 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:49:06.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.818 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:06.818 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:49:06.818 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:49:06.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.819 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:06.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.819 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:49:06.819 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:49:06.819 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:49:06.820 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:49:06.820 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:49:06.820 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:49:06.820 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:49:06.821 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:49:06.821 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:49:06.821 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.822 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:49:06.822 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:49:06.822 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:49:06.822 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:49:06.822 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:49:06.823 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:49:06.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.823 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:49:06.823 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:49:06.824 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:49:06.824 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:49:06.824 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:49:06.824 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:49:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:49:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:49:06.825 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:49:06.826 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:49:06.826 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:49:06.826 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:49:06.827 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.827 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:49:06.827 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:49:06.827 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:49:06.828 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:49:06.828 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:49:06.829 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:49:06.829 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:06.829 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:06.829 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:49:06.830 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:49:06.830 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:06.830 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:06.831 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-65 2023-10-11T15:49:06.832 DEBUG:boto:path=/dghqvd-65/ 2023-10-11T15:49:06.832 DEBUG:boto:auth_path=/dghqvd-65/ 2023-10-11T15:49:06.833 DEBUG:boto:Method: PUT 2023-10-11T15:49:06.833 DEBUG:boto:Path: /dghqvd-65/ 2023-10-11T15:49:06.834 DEBUG:boto:Data: 2023-10-11T15:49:06.834 DEBUG:boto:Headers: {} 2023-10-11T15:49:06.835 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:49:06.835 DEBUG:boto:Port: 8000 2023-10-11T15:49:06.836 DEBUG:boto:Params: {} 2023-10-11T15:49:06.836 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:49:06.837 DEBUG:boto:Token: None 2023-10-11T15:49:06.838 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:49:06 GMT /dghqvd-65/ 2023-10-11T15:49:06.838 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4zX16/m+LCqNrwxOlOxTSzpqL8s= 2023-10-11T15:49:06.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:49:06 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4zX16/m+LCqNrwxOlOxTSzpqL8s=', 'Content-Length': '0'} 2023-10-11T15:49:07.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f00f3d5ab0d0d408-006526c3f3-2156-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:49:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:49:07.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.112 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.186 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.186 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.291 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T15:49:07.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.293 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.370 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.371 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.434 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:07.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.436 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.544 INFO:teuthology.orchestra.run.smithi158.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-10-11T15:49:07.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.546 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.594 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.640 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:07.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.642 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.698 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.698 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.746 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:07.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.748 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.804 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.853 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:49:07.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.855 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:07.909 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:07.909 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:07.957 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T15:49:07.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:49:07.959 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:49:08.009 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:08.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:08.056 INFO:teuthology.orchestra.run.smithi158.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-10-11T15:49:08.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:08.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:08.058 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:08.108 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:08.139 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:49:08.152 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:49:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.152 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:49:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.153 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:49:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:49:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:49:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:49:07.942802Z" 2023-10-11T15:49:08.156 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.157 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:49:08.158 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:08.159 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:49:08.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:08.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:08.160 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:08.209 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:08.210 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:08.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:08.238+0000 7fe8555fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:08.243 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:08.239+0000 7fe8555fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:08.255 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:08.255 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:08.255 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:08.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:08.256 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:08.256 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:08.256 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:08.256 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.257 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:08.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.257 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:08.257 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:08.258 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:08.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:08.258 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:08.258 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:08.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:08.259 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:08.259 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:08.259 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.259 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.260 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:08.260 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:08.260 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:08.260 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:08.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:08.261 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:08.261 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:08.261 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:08.262 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:08.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.262 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.262 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.262 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:08.263 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:08.263 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:08.263 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:08.263 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:08.264 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:08.264 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:08.264 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:08.264 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:08.264 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.265 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:08.265 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:08.265 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:08.265 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:08.266 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:08.266 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:08.266 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:08.266 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:08.267 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:08.267 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:08.267 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.267 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:08.267 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:08.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.268 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:08.268 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:08.268 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:08.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:08.269 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:08Z" 2023-10-11T15:49:08.269 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:08.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:08Z" } 2023-10-11T15:49:08.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:08.271 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:08.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:08.272 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:13.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:13.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:13.274 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:13.332 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:13.333 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:13.363+0000 7f8969ee2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:13.365 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:13.364+0000 7f8969ee2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:13.379 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:13.379 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:13.380 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:13.380 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:13.380 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:13.380 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:13.381 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:13.381 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.381 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:13.381 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:13.381 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:13.382 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:13.382 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:13.382 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:13.382 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:13.383 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:13.383 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:13.383 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:13.383 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:13.383 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:13.384 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.384 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:13.384 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:13.384 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:13.385 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:13.385 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:13.385 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:13.385 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:13.385 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:13.386 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:13.386 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:13.386 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:13.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.387 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:13.387 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:13.387 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:13.387 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:13.387 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:13.388 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:13.388 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:13.388 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:13.388 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:13.388 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:13.389 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:13.389 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.389 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:13.389 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:13.390 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:13.390 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:13.390 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:13.390 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:13.390 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:13.391 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:13.391 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:13.391 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:13.391 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:13.391 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:13.392 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:13.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.392 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:13.392 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:13.392 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:13.393 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:13.393 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:13Z" 2023-10-11T15:49:13.393 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:13.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:13Z" } 2023-10-11T15:49:13.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:13.395 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:13.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:13.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:18.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:18.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:18.398 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:18.455 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:18.456 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:18.492 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:18.491+0000 7fb6150bf780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:18.493 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:18.492+0000 7fb6150bf780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:18.508 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:18.508 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:18.508 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:18.509 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:18.509 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:18.509 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:18.509 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:18.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.510 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:18.510 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:18.510 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:18.510 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:18.511 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:18.511 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:18.511 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:18.511 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:18.511 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:18.512 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:18.512 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:18.512 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:18.512 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.512 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:18.513 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:18.513 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:18.513 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:18.513 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:18.514 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:18.514 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:18.514 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:18.514 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:18.514 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:18.515 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:18.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:18.515 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:18.516 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:18.516 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:18.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:18.516 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:18.516 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:18.517 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:18.517 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:18.517 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:18.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:18.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.518 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:18.518 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:18.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:18.519 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:18.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:18.519 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:18.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:18.520 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:18.520 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:18.520 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:18.520 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:18.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:18.521 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:18.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.521 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:18.521 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:18.522 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:18.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:18.522 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:18Z" 2023-10-11T15:49:18.522 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:18.524 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:18Z" } 2023-10-11T15:49:18.524 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:18.525 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:18.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:18.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:23.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:23.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:23.529 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:23.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:23.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:23.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:23.622+0000 7f79669b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:23.626 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:23.624+0000 7f79669b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:23.640 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:23.640 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:23.640 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:23.641 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:23.641 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:23.641 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:23.641 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:23.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.642 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:23.642 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:23.642 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:23.642 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:23.643 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:23.643 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:23.643 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:23.643 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:23.643 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:23.644 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:23.644 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:23.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:23.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.645 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:23.645 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:23.645 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:23.645 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:23.645 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:23.646 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:23.646 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:23.646 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:23.646 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:23.647 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:23.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:23.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:23.647 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:23.648 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:23.648 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:23.648 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:23.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:23.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:23.649 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:23.649 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:23.649 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:23.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:23.650 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.650 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:23.650 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:23.650 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:23.650 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:23.651 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:23.651 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:23.651 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:23.651 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:23.652 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:23.652 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:23.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:23.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:23.652 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:23.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.653 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:23.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:23.653 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:23.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:23.654 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:23Z" 2023-10-11T15:49:23.654 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:23.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:23Z" } 2023-10-11T15:49:23.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:23.656 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:23.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:23.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:28.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:28.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:28.659 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:28.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:28.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:28.748+0000 7ff1f8c51780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:28.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:28.749+0000 7ff1f8c51780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:28.764 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:28.764 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:28.764 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:28.764 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:28.765 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:28.765 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:28.765 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:28.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.766 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:28.766 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:28.766 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:28.766 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:28.766 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:28.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:28.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:28.768 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:28.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:28.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:28.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:28.769 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:28.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:28.769 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:28.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:28.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:28.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:28.770 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:28.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:28.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.771 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:28.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:28.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:28.772 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:28.772 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:28.772 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:28.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:28.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:28.773 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:28.773 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:28.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:28.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.773 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:28.774 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:28.774 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:28.774 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:28.774 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:28.774 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:28.775 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:28.775 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:28.775 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:28.775 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:28.775 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:28.776 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:28.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:28.777 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:28Z" 2023-10-11T15:49:28.777 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:28.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:28Z" } 2023-10-11T15:49:28.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:28.779 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:28.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:28.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:33.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:33.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:33.783 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:33.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:33.842 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:33.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:33.873+0000 7fe653f2c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:33.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:33.874+0000 7fe653f2c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:33.889 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:33.889 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.890 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:33.892 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:33.893 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:33.894 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:33.895 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:33.896 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:33.897 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:33.898 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:33.899 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:33.900 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:33.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:33.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:33.901 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:33.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.901 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:33.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:33.902 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:33.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:33.902 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:33Z" 2023-10-11T15:49:33.902 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:33.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:33Z" } 2023-10-11T15:49:33.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:33.905 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:33.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:33.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:38.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:38.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:38.908 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:38.965 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:38.966 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:38.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:38.996+0000 7fab93d91780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:38.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:38.998+0000 7fab93d91780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:39.012 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:39.012 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:39.013 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:39.013 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:39.013 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:39.013 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:39.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:39.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.014 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:39.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:39.014 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:39.015 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:39.015 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:39.015 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:39.015 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:39.015 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:39.016 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:39.016 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:39.016 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:39.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:39.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.017 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:39.017 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:39.017 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:39.017 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:39.018 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:39.018 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:39.018 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:39.018 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:39.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:39.019 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:39.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:39.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:39.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:39.020 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:39.020 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:39.020 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:39.020 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:39.020 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:39.021 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:39.021 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:39.021 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:39.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:39.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.022 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:39.022 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:39.022 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:39.022 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:39.022 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:39.023 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:39.023 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:39.023 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:39.023 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:39.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:39.024 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:39.024 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:39.024 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:39.024 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.024 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:39.025 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:39.025 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:39.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:39.025 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:38Z" 2023-10-11T15:49:39.025 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:39.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:38Z" } 2023-10-11T15:49:39.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:39.028 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:39.028 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:39.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:44.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:44.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:44.031 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:44.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:44.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:44.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:44.121+0000 7fd1b2a15780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:44.123 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:44.122+0000 7fd1b2a15780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:44.138 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:44.138 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:44.138 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:44.139 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:44.139 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:44.139 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:44.139 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:44.140 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.140 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:44.140 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:44.140 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:44.140 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:44.141 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:44.141 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:44.141 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:44.141 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:44.141 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:44.142 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:44.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:44.142 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:44.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.143 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:44.143 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:44.143 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:44.143 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:44.144 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:44.144 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:44.144 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:44.144 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:44.144 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:44.145 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:44.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:44.145 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.145 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:44.145 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:44.146 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:44.146 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:44.146 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:44.146 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:44.147 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:44.147 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:44.147 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:44.147 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:44.147 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:44.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:44.148 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:44.148 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:44.149 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:44.149 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:44.149 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:44.149 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:44.149 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:44.150 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:44.150 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:44.150 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:44.150 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:44.151 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:44.151 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.151 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:44.151 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:44.151 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:44.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:44.152 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:44Z" 2023-10-11T15:49:44.152 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:44.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:44Z" } 2023-10-11T15:49:44.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:44.154 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:44.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:44.155 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:49.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:49.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:49.158 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:49.216 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:49.217 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:49.247+0000 7fdccf2b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:49.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:49.248+0000 7fdccf2b8780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:49.263 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:49.267 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:49.267 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:49.267 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:49.267 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:49.267 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.268 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:49.268 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:49.268 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:49.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:49.269 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:49.269 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:49.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:49.269 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:49.270 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:49.270 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:49.270 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:49.270 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.271 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:49.271 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:49.271 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:49.271 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:49.272 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:49.272 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:49.272 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:49.272 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:49.272 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:49.273 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:49.274 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:49.275 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:49.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:49.277 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:49Z" 2023-10-11T15:49:49.278 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:49.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:49Z" } 2023-10-11T15:49:49.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:49.280 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:49.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:49.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:54.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:54.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:54.284 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:54.343 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:54.344 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:54.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:54.374+0000 7f4b2fdac780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:54.376 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:54.375+0000 7f4b2fdac780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:54.390 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:54.399 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:54.399 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:54.399 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:54.399 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:54.399 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:54.400 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:54Z" 2023-10-11T15:49:54.404 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:54.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:54Z" } 2023-10-11T15:49:54.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:54.406 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:54.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:54.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:49:59.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:59.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:49:59.411 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:49:59.469 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:49:59.469 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:49:59.502 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:59.499+0000 7efe67a67780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:59.503 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:49:59.500+0000 7efe67a67780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:49:59.516 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:49:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:49:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:49:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:49:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:49:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:49:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:49:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:49:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:49:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:49:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:49:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:49:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:49:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:49:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:59.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:49:59.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:59.525 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:59.525 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:49:59.525 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:49:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:49:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:49:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:49:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:49:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:49:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:49:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:49:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:49:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:49:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:49:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:49:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:49:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:49:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:49:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:49:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:49:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:49:59Z" 2023-10-11T15:49:59.532 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:49:59.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:49:59Z" } 2023-10-11T15:49:59.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:49:59.535 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:49:59.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:49:59.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:04.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:04.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:04.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:04.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:04.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:04.627+0000 7fe26c21f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:04.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:04.628+0000 7fe26c21f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:04.644 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:04.650 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:04.650 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:04Z" 2023-10-11T15:50:04.659 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:04.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:04Z" } 2023-10-11T15:50:04.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:04.661 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:04.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:04.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:09.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:09.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:09.664 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:09.721 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:09.722 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:09.752+0000 7fb3c4013780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:09.755 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:09.753+0000 7fb3c4013780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:09.770 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:09Z" 2023-10-11T15:50:09.785 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:09.786 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:09Z" } 2023-10-11T15:50:09.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:09.787 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:09.787 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:09.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:14.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:14.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:14.790 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:14.845 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:14.846 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:14.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:14.876+0000 7fef5f8df780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:14.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:14.877+0000 7fef5f8df780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:14.892 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:14Z" 2023-10-11T15:50:14.906 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:14.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:14Z" } 2023-10-11T15:50:14.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:14.908 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:14.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:14.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:19.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:19.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:19.912 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:19.968 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:19.969 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:20.000 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:19.998+0000 7f9e2d39c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:20.001 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:19.999+0000 7f9e2d39c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:20.015 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:20.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:20.037 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:20.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.039 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:20.040 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:20Z" 2023-10-11T15:50:20.041 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:20.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:20Z" } 2023-10-11T15:50:20.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:20.043 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:20.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:20.044 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:25.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:25.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:25.046 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:25.104 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:25.138 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:25.136+0000 7fb7cf02d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:25.139 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:25.137+0000 7fb7cf02d780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:25.153 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:25.160 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:25.160 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:25.160 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:25.160 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.161 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:25.161 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:25.161 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:25.161 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:25.161 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:25.162 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:25.162 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:25.162 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:25.162 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:25.163 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:25.163 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:25.163 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.163 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:25.163 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:25.164 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:25.164 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:25.164 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:25.164 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:25.164 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:25.165 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:25.166 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:25.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:25.167 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:25Z" 2023-10-11T15:50:25.167 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:25.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:25Z" } 2023-10-11T15:50:25.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:25.169 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:25.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:25.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:30.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:30.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:30.172 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:30.230 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:30.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:30.266+0000 7f145120f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:30.269 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:30.267+0000 7f145120f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:30.284 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:30.289 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:30.289 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:30.289 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:30.289 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:30.290 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:30.290 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:30.290 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:30.290 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:30.290 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:30.291 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.291 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:30.291 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:30.291 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:30.291 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:30.292 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:30.293 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:30.293 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.293 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:30.293 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:30.294 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:30.294 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:30.294 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:30.294 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:30.294 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:30.295 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:30.295 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:30.295 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:30.295 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:30.295 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:30.296 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:30.296 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.296 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:30.296 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:30.296 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:30.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:30.297 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:30Z" 2023-10-11T15:50:30.297 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:30.298 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:30Z" } 2023-10-11T15:50:30.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:30.299 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:30.299 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:30.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:35.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:35.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:35.303 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:35.361 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:35.362 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:35.395 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:35.393+0000 7fb98c46f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:35.396 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:35.394+0000 7fb98c46f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:35.412 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:35.413 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:35.413 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:35.413 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:35.413 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.413 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:35.414 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:35.414 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:35.414 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:35.415 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:35.415 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:35.415 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:35.416 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:35.416 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:35.416 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:35.416 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:35.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:35.417 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.417 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:35.417 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:35.417 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:35.418 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:35.418 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:35.418 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:35.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:35.419 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:35.419 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:35.419 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:35.419 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:35.419 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.420 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:35.420 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:35.420 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:35.420 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:35.420 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:35.421 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:35.421 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:35.421 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:35.421 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:35.421 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:35.422 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:35.422 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.422 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:35.422 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:35.422 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:35.423 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:35.423 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:35.423 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:35.423 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:35.423 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:35.424 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:35.424 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:35.424 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:35.424 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:35.424 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:35.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.425 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:35.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:35.425 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:35.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:35.426 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:35Z" 2023-10-11T15:50:35.426 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:35.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:35Z" } 2023-10-11T15:50:35.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:35.428 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:35.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:35.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:40.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:40.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:40.431 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:40.489 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:40.490 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:40.522 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:40.520+0000 7fb9a9fa2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:40.523 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:40.521+0000 7fb9a9fa2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:40.536 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:40.541 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:40.541 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.541 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:40.541 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:40.542 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:40.542 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:40.542 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:40.542 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:40.543 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:40.543 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:40.543 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:40.543 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:40.543 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:40.544 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.544 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:40.544 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:40.544 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:40.544 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:40.545 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:40.545 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:40.545 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:40.545 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:40.546 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:40.546 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:40.546 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:40.546 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.546 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:40.547 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:40.547 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:40.547 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:40.547 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:40.548 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:40.548 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:40.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:40.548 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:40.548 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:40.549 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:40.549 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:40.549 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:40.549 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.549 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:40.550 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:40.550 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:40.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:40.550 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:40Z" 2023-10-11T15:50:40.551 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:40.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:40Z" } 2023-10-11T15:50:40.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:40.553 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:40.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:40.553 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:45.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:45.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:45.556 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:45.613 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:45.613 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:45.646 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:45.644+0000 7f960f43e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:45.647 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:45.646+0000 7f960f43e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:45.662 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:45.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.669 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:45.669 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:45.669 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:45.669 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:45.670 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:45.670 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:45.670 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:45.670 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:45.671 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:45.671 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:45.671 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:45.671 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.671 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:45.672 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:45.672 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:45.672 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:45.672 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039048.970763_766.1", 2023-10-11T15:50:45.672 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:45.673 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:45.673 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:45.673 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:08.970763Z", 2023-10-11T15:50:45.673 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:45.673 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:45.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:45.674 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:45.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.674 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:45.675 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:45.675 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:45.675 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:45.675 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:45Z" 2023-10-11T15:50:45.675 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:45.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039048.970763_766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:08.970763Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:45Z" } 2023-10-11T15:50:45.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039048.970763_766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:08.970763Z', 'realm_epoch': 2}}] 2023-10-11T15:50:45.677 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:45.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039048.970763_766.1'} 2023-10-11T15:50:45.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039048.970763_766.1 2023-10-11T15:50:50.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:50.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:50.681 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:50.738 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:50.739 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:50.771 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:50.770+0000 7f367bcd7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:50.772 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:50.771+0000 7f367bcd7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:50.786 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:50.796 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:50.796 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:50.796 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.796 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:50.796 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:50.797 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:50.797 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:50.797 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.728689_878.1", 2023-10-11T15:50:50.797 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:50.797 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:50.798 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:50.798 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.728689Z", 2023-10-11T15:50:50.798 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:50.798 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:50.799 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:50.800 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:50.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:50.800 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:50Z" 2023-10-11T15:50:50.800 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:50.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.728689_878.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.728689Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:50Z" } 2023-10-11T15:50:50.802 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.728689_878.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.728689Z', 'realm_epoch': 2}}] 2023-10-11T15:50:50.802 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:50.803 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.728689_878.1'} 2023-10-11T15:50:50.803 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039347.728689_878.1 2023-10-11T15:50:55.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:55.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:50:55.806 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:50:55.866 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:50:55.866 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:50:55.897 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:55.896+0000 7f5f97050780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:55.899 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:50:55.897+0000 7f5f97050780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:50:55.912 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:50:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:50:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:50:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:50:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:50:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:50:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:50:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:50:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:50:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:50:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:50:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:50:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:50:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:50:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:50:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:50:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:50:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:50:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.728689_878.1", 2023-10-11T15:50:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:50:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:50:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:50:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.728689Z", 2023-10-11T15:50:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:50:55.924 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:55.924 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:50:55.924 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:50:55.924 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.924 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:50:55.925 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:50:55.925 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:50:55.925 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:50:55.925 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:50:55Z" 2023-10-11T15:50:55.925 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:50:55.926 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.728689_878.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.728689Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:50:55Z" } 2023-10-11T15:50:55.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.728689_878.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.728689Z', 'realm_epoch': 2}}] 2023-10-11T15:50:55.927 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:50:55.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.728689_878.1'} 2023-10-11T15:50:55.928 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039347.942802_883.1 target=1_1697039347.728689_878.1 2023-10-11T15:51:00.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:00.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:00.931 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:00.988 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:00.988 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.021 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:01.019+0000 7f4cc9aa2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:01.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:01.021+0000 7f4cc9aa2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:01.036 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:51:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.942802Z", 2023-10-11T15:51:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:01Z" 2023-10-11T15:51:01.048 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:01.049 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.942802_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.942802Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:01Z" } 2023-10-11T15:51:01.049 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.942802_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.942802Z', 'realm_epoch': 2}}] 2023-10-11T15:51:01.050 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:01.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:01.051 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:51:01.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.052 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:01.104 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.105 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.154 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:01.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.156 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.212 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.263 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:01.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.265 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:01.324 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.324 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.373 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:01.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.375 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:01.424 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.425 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.474 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T15:51:01.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dghqvd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:01.475 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dghqvd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:01.522 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.523 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.568 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T15:51:01.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:01.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:01.569 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:01.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.617 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.646 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:51:01.662 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:51:01.662 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.662 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:01.662 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:51:01.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.663 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.663 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:01.663 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:51:01.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.664 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:01.664 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:51:01.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.665 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:01.665 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:51:01.459205Z" 2023-10-11T15:51:01.665 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.665 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:51:01.666 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:01.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:51:01.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:01.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:01.667 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:01.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:01.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:01.746 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:01.744+0000 7f097465f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:01.747 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:01.745+0000 7f097465f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:01.760 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:01.760 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:01.760 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:01.760 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:01.761 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:01.761 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:01.761 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.761 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.761 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:01.762 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.762 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:01.762 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.762 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.762 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:01.763 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:01.763 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.763 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:01.763 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.764 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:01.764 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.765 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.767 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:01.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:01.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:51:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:01.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.942802Z", 2023-10-11T15:51:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:01.772 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:01.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:01.773 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:01Z" 2023-10-11T15:51:01.773 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:01.774 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.942802_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.942802Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:01Z" } 2023-10-11T15:51:01.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.942802_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.942802Z', 'realm_epoch': 2}}] 2023-10-11T15:51:01.775 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:01.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:01.776 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039461.459205_891.1 target=1_1697039347.942802_883.1 2023-10-11T15:51:06.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:06.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:06.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:06.835 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:06.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:06.867+0000 7fdfb8c8e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:06.868+0000 7fdfb8c8e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:06.883 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:06.884 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:06.884 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:06.884 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:06.884 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:06.885 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:06.885 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:06.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.885 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:06.886 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:06.886 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:06.886 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:06.886 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:06.887 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:06.887 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:06.887 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:06.887 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:06.887 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:06.888 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:06.888 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:06.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.888 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:06.889 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:06.889 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:06.889 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:06.889 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:06.889 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:06.890 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:06.890 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:06.890 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:06.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:06.890 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:06.891 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.891 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:06.891 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:06.891 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:06.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:06.892 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:06.892 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:06.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:06.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:06.893 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:06.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:06.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:06.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.893 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:06.894 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:06.894 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:06.894 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:06.894 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:51:06.895 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:06.895 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:06.895 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:06.895 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.942802Z", 2023-10-11T15:51:06.895 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:06.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:06.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:06.896 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:06.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.896 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:06.897 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:06.897 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:06.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:06.897 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:06Z" 2023-10-11T15:51:06.898 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:06.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.942802_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.942802Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:06Z" } 2023-10-11T15:51:06.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.942802_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.942802Z', 'realm_epoch': 2}}] 2023-10-11T15:51:06.900 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:06.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:06.900 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039461.459205_891.1 target=1_1697039347.942802_883.1 2023-10-11T15:51:11.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:11.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:11.903 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:11.960 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:11.961 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:11.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:11.992+0000 7f5fd8ae4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:11.994 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:11.993+0000 7f5fd8ae4780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:12.008 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:12.008 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:12.009 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:12.009 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:12.009 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:12.009 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:12.010 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:12.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.010 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:12.010 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:12.011 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:12.011 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:12.011 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:12.011 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:12.012 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:12.012 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:12.012 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:12.012 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:12.013 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:12.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:12.013 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.013 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:12.013 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:12.014 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:12.014 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:12.014 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:12.014 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:12.015 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:12.015 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:12.015 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:12.015 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:12.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:12.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:12.016 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:12.016 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:12.017 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:12.017 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:12.017 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:12.017 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:12.018 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:12.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:12.018 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:12.018 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:12.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:12.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:12.019 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:12.019 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:12.020 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:51:12.020 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:12.020 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:12.020 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:12.021 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.942802Z", 2023-10-11T15:51:12.021 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:12.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:12.021 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:12.021 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:12.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.022 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:12.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:12.022 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:12.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:12.023 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:11Z" 2023-10-11T15:51:12.023 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:12.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.942802_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.942802Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:11Z" } 2023-10-11T15:51:12.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.942802_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.942802Z', 'realm_epoch': 2}}] 2023-10-11T15:51:12.025 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:12.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:12.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039461.459205_891.1 target=1_1697039347.942802_883.1 2023-10-11T15:51:17.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:17.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:17.029 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:17.085 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:17.085 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:17.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:17.116+0000 7fb00b9d3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:17.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:17.117+0000 7fb00b9d3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:17.132 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:17.132 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:17.133 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:17.134 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:17.135 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:17.137 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:17.138 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:17.139 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:17.140 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:17.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:17.142 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039347.942802_883.1", 2023-10-11T15:51:17.143 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:17.143 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:17.143 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:17.143 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:49:07.942802Z", 2023-10-11T15:51:17.143 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:17.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:17.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:17.144 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:17.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.144 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:17.145 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:17.145 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:17.145 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:17.145 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:17Z" 2023-10-11T15:51:17.145 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:17.146 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039347.942802_883.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:49:07.942802Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:17Z" } 2023-10-11T15:51:17.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039347.942802_883.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:49:07.942802Z', 'realm_epoch': 2}}] 2023-10-11T15:51:17.147 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:17.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039347.942802_883.1'} 2023-10-11T15:51:17.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039461.459205_891.1 target=1_1697039347.942802_883.1 2023-10-11T15:51:22.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:22.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:22.151 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:22.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.211 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.248 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:22.246+0000 7fdfea2e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:22.249 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:22.247+0000 7fdfea2e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:22.263 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:22.263 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:22.264 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:22.264 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:22.264 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:22.264 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:22.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.265 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:22.265 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.265 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:22.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:22.266 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:22.266 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:22.266 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:22.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:22.267 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:22.267 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:22.267 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.268 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.268 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:22.269 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:22.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:22.269 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:22.269 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:22.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:22.270 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:22.270 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:22.270 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.270 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.271 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.271 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.271 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:22.272 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:22.272 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:22.272 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:22.272 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:22.272 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:22.273 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:22.273 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:22.273 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.273 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.274 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.274 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:22.274 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:22.275 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:22.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:22.275 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:22.275 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:22.275 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:22.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:22.276 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.277 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.277 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:22.277 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:22.277 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:22.278 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.278 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:22Z" 2023-10-11T15:51:22.278 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:22.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:22Z" } 2023-10-11T15:51:22.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:22.280 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:22.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:22.281 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:51:22.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.282 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.336 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.383 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:22.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.385 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.441 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.441 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.487 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:22.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.488 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.543 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.543 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.588 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T15:51:22.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.590 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.639 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.639 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.685 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T15:51:22.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.687 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.732 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.733 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.776 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T15:51:22.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:22.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:22.826 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.827 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:22.877 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:22.878 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:22.878 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 14, 2023-10-11T15:51:22.878 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:51:22.878 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:51:22.879 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:51:22.879 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:22.879 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:51:22.879 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.879 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:51:22.880 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:51:22.880 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:51:22.880 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:51:22.880 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.881 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:51:22.881 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.881 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:51:22.881 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:51:22.881 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:22.882 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:22.882 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.882 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:22.882 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:51:22.882 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.883 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:51:22.883 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.883 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:22.883 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:22.883 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.886 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.886 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:22.886 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:51:22.886 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.886 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:22.887 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:22.888 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:22.888 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:22.888 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:22.888 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:22.888 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:22.889 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.889 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.889 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.889 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.889 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:51:22.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.890 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:51:22.890 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:51:22.890 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:51:22.890 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:51:22.891 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.891 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.891 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.891 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:51:22.891 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:22.892 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:51:22.892 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:51:22.892 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.892 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:51:22.892 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:51:22.893 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:51:22.894 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:51:22.894 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.894 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:51:22.894 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.895 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:22.896 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:51:22.896 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.896 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:51:22.896 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.896 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:22.897 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:22.897 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:22.897 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:22.897 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:22.897 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:22.898 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:22.898 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:22.898 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:22.898 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:22.899 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.899 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.900 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:51:22.900 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:51:22.900 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:22.901 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:22.902 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:22.902 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:22.902 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:22.902 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:22.902 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:22.903 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:22.903 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.903 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.903 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.904 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:51:22.904 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.904 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:51:22.904 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:51:22.904 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:51:22.905 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:22.906 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:51:22.906 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:51:22.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.906 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:51:22.906 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:22.907 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.907 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.907 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:22.907 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:51:22.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:22.908 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:51:22.908 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:22.909 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:51:22.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.909 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:51:22.909 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:51:22.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.910 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:22.910 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:22.910 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:51:22.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:51:22.911 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:51:22.912 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:51:22.912 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:51:22.912 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:51:22.912 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:51:22.912 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:51:22.913 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.913 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:51:22.913 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:51:22.913 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:51:22.913 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:51:22.914 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:51:22.914 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:51:22.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.914 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:51:22.914 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.915 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:51:22.916 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:22.916 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:22.916 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:22.916 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:22.916 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:22.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.917 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:51:22.917 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:22.917 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:22.917 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:22.918 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:22.918 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:22.918 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:22.918 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:22.918 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:22.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:51:22.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:22.919 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:22.920 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-10-11T15:51:22.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:22.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:22.922 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:22.978 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:22.979 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:51:23.063 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:51:23.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.064 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:23.064 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:51:23.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.065 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:23.065 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:51:23.065 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.066 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:23.066 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:51:23.066 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.066 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.067 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:23.067 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:51:01.459205Z" 2023-10-11T15:51:23.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.067 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:51:23.068 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:23.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:51:23.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:23.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:23.070 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:23.136 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.136 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.227 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:23.224+0000 7f7fba8e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:23.228 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:23.225+0000 7f7fba8e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:23.264 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:23.265 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:23.265 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:23.265 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:23.265 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:23.265 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:23.266 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:23.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.266 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:23.266 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.267 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:23.267 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:23.267 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:23.267 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:23.267 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:23.268 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:23.268 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:23.268 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:23.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:23.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.269 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.269 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:23.269 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:23.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:23.270 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:23.270 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:23.270 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:23.270 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:23.270 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:23.271 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:23.271 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.271 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.271 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.271 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:23.272 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:23.272 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:23.272 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:23.272 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:23.272 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:23.273 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:23.273 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:23.273 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:23.273 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.274 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.274 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:23.274 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:23.274 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:23.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:23.275 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:23.275 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:23.275 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:23.275 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:23.276 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:23.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.276 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:23.276 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:23.276 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.277 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:23.277 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:23.277 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:23.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.277 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:23Z" 2023-10-11T15:51:23.278 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:23.279 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:23Z" } 2023-10-11T15:51:23.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:23.280 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:23.280 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:23.281 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:51:23.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.282 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.352 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.352 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.447 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T15:51:23.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.448 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.514 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.514 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.587 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T15:51:23.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.589 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.647 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.647 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.693 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T15:51:23.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.695 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.747 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.748 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.793 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:51:23.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.794 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.844 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.889 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:51:23.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:23.891 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:23.939 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:23.940 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:23.990 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:23.990 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:23.990 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 15, 2023-10-11T15:51:23.990 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:51:23.991 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:51:23.991 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:51:23.991 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:23.991 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:51:23.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.992 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:51:23.992 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:51:23.992 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:51:23.992 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:51:23.993 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:23.993 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:51:23.993 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:23.993 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:51:23.993 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:51:23.994 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:23.994 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:23.994 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.994 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:23.995 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:51:23.995 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:23.995 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:51:23.995 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:23.995 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:23.996 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:23.996 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:23.996 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:23.996 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:23.997 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:23.997 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:23.997 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:23.997 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:23.997 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:23.998 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:23.998 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:23.998 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:23.998 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:23.999 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:23.999 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:51:23.999 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:23.999 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:51:23.999 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.000 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:24.000 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:24.000 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:24.000 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:24.001 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.002 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.003 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:51:24.003 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.003 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:51:24.003 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:51:24.003 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:51:24.004 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:51:24.004 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.004 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.004 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:51:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:51:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T15:51:24.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.006 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T15:51:24.006 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T15:51:24.006 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T15:51:24.006 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.007 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T15:51:24.007 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:24.007 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:24.007 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:51:24.007 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.008 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.008 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.008 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T15:51:24.008 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.009 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T15:51:24.009 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:51:24.009 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:51:24.009 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:24.009 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:24.010 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:51:24.010 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.010 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T15:51:24.011 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:51:24.011 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:24.011 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:24.011 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:51:24.011 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.012 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T15:51:24.012 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:51:24.012 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T15:51:24.012 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T15:51:24.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.013 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.013 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T15:51:24.013 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T15:51:24.013 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T15:51:24.014 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T15:51:24.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.014 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.014 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T15:51:24.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.015 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.015 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.015 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:51:24.015 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:24.016 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.016 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:51:24.017 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:51:24.017 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:51:24.017 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:51:24.017 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:24.017 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:51:24.018 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.018 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:51:24.018 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:51:24.018 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:24.018 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:51:24.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.019 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:24.019 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:51:24.019 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:24.019 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:51:24.020 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.020 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:24.020 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:24.020 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:24.021 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:24.021 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:24.021 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:24.021 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:24.021 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:24.022 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:24.022 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:24.022 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:24.022 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.023 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:51:24.023 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:51:24.023 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:51:24.023 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:51:24.024 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.024 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:51:24.024 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:51:24.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:51:24.024 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:51:24.025 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:51:24.025 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:51:24.025 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:51:24.025 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:51:24.025 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:51:24.026 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:51:24.026 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:24.026 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.026 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.027 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:51:24.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.027 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:51:24.027 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:51:24.027 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:51:24.028 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:51:24.028 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.028 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.028 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.029 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:51:24.029 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:24.029 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:51:24.029 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:51:24.030 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.030 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:51:24.030 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:51:24.030 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.031 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:51:24.031 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:51:24.031 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.031 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:51:24.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:51:24.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:51:24.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:51:24.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:51:24.034 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:51:24.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.035 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.035 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:24.035 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:51:24.035 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.036 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.036 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:51:24.036 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:51:24.037 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:51:24.037 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:51:24.037 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:51:24.037 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:51:24.037 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:51:24.038 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:51:24.038 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:51:24.038 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.038 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:51:24.039 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:51:24.039 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:51:24.039 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:51:24.039 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:51:24.039 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:51:24.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.040 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:51:24.040 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:24.040 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:24.040 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:24.041 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:24.041 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:24.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.041 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:51:24.041 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:24.042 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:24.042 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:24.042 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:24.042 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:24.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.043 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:51:24.043 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:51:24.043 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:51:24.043 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:51:24.044 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:51:24.044 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:51:24.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.044 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:51:24.045 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:51:24.045 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.045 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:24.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:24.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:51:24.047 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:51:24.099 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:24.099 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:24.189 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:51:24.190 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-66 2023-10-11T15:51:24.191 DEBUG:boto:path=/dghqvd-66/ 2023-10-11T15:51:24.191 DEBUG:boto:auth_path=/dghqvd-66/ 2023-10-11T15:51:24.191 DEBUG:boto:Method: PUT 2023-10-11T15:51:24.192 DEBUG:boto:Path: /dghqvd-66/ 2023-10-11T15:51:24.192 DEBUG:boto:Data: 2023-10-11T15:51:24.193 DEBUG:boto:Headers: {} 2023-10-11T15:51:24.193 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:51:24.194 DEBUG:boto:Port: 8000 2023-10-11T15:51:24.194 DEBUG:boto:Params: {} 2023-10-11T15:51:24.195 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:51:24.195 DEBUG:boto:Token: None 2023-10-11T15:51:24.196 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-66/ 2023-10-11T15:51:24.196 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:MrlYqEGWcmlT/QSs0ZxsMEnHER4= 2023-10-11T15:51:24.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MrlYqEGWcmlT/QSs0ZxsMEnHER4=', 'Content-Length': '0'} 2023-10-11T15:51:24.223 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009a8f8d9b22f7eb88-006526c47c-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.224 DEBUG:boto:path=/dghqvd-66/ 2023-10-11T15:51:24.224 DEBUG:boto:auth_path=/dghqvd-66/ 2023-10-11T15:51:24.225 DEBUG:boto:Method: HEAD 2023-10-11T15:51:24.225 DEBUG:boto:Path: /dghqvd-66/ 2023-10-11T15:51:24.226 DEBUG:boto:Data: 2023-10-11T15:51:24.226 DEBUG:boto:Headers: {} 2023-10-11T15:51:24.226 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:51:24.227 DEBUG:boto:Port: 8000 2023-10-11T15:51:24.227 DEBUG:boto:Params: {} 2023-10-11T15:51:24.228 DEBUG:boto:Token: None 2023-10-11T15:51:24.228 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-66/ 2023-10-11T15:51:24.228 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eJfKpF3FQUY94cCqkoJljnfeCm8= 2023-10-11T15:51:24.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eJfKpF3FQUY94cCqkoJljnfeCm8=', 'Content-Length': '0'} 2023-10-11T15:51:24.231 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef1ef82a77f54332-006526c47c-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.232 DEBUG:boto:path=/dghqvd-66/obj1 2023-10-11T15:51:24.233 DEBUG:boto:auth_path=/dghqvd-66/obj1 2023-10-11T15:51:24.233 DEBUG:boto:Method: PUT 2023-10-11T15:51:24.233 DEBUG:boto:Path: /dghqvd-66/obj1 2023-10-11T15:51:24.234 DEBUG:boto:Data: 2023-10-11T15:51:24.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:51:24.235 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:51:24.235 DEBUG:boto:Port: 8000 2023-10-11T15:51:24.236 DEBUG:boto:Params: {} 2023-10-11T15:51:24.236 DEBUG:boto:Token: None 2023-10-11T15:51:24.236 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-66/obj1 2023-10-11T15:51:24.237 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gXgszBL+Y1ZxJ0JduLBZHkNEGsw= 2023-10-11T15:51:24.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gXgszBL+Y1ZxJ0JduLBZHkNEGsw='} 2023-10-11T15:51:24.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000242bba2554e56617-006526c47c-2322-a1'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.248 INFO:rgw_multi.tests:create bucket zone=a2 name=dghqvd-67 2023-10-11T15:51:24.249 DEBUG:boto:path=/dghqvd-67/ 2023-10-11T15:51:24.249 DEBUG:boto:auth_path=/dghqvd-67/ 2023-10-11T15:51:24.249 DEBUG:boto:Method: PUT 2023-10-11T15:51:24.250 DEBUG:boto:Path: /dghqvd-67/ 2023-10-11T15:51:24.250 DEBUG:boto:Data: 2023-10-11T15:51:24.251 DEBUG:boto:Headers: {} 2023-10-11T15:51:24.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:51:24.251 DEBUG:boto:Port: 8001 2023-10-11T15:51:24.252 DEBUG:boto:Params: {} 2023-10-11T15:51:24.252 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:51:24.253 DEBUG:boto:Token: None 2023-10-11T15:51:24.253 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-67/ 2023-10-11T15:51:24.254 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ls2E1+6bFNKp5ApB/ghf95gOUUQ= 2023-10-11T15:51:24.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ls2E1+6bFNKp5ApB/ghf95gOUUQ=', 'Content-Length': '0'} 2023-10-11T15:51:24.353 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1da0f351ed3dba6-006526c47c-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.354 DEBUG:boto:path=/dghqvd-67/ 2023-10-11T15:51:24.354 DEBUG:boto:auth_path=/dghqvd-67/ 2023-10-11T15:51:24.354 DEBUG:boto:Method: HEAD 2023-10-11T15:51:24.355 DEBUG:boto:Path: /dghqvd-67/ 2023-10-11T15:51:24.355 DEBUG:boto:Data: 2023-10-11T15:51:24.356 DEBUG:boto:Headers: {} 2023-10-11T15:51:24.356 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:51:24.356 DEBUG:boto:Port: 8001 2023-10-11T15:51:24.357 DEBUG:boto:Params: {} 2023-10-11T15:51:24.357 DEBUG:boto:Token: None 2023-10-11T15:51:24.358 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-67/ 2023-10-11T15:51:24.358 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oRUX9PH+5CruoElarAQJAiCUHE4= 2023-10-11T15:51:24.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oRUX9PH+5CruoElarAQJAiCUHE4=', 'Content-Length': '0'} 2023-10-11T15:51:24.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083a5251a8fbb9bbc-006526c47c-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.363 DEBUG:boto:path=/dghqvd-67/obj2 2023-10-11T15:51:24.363 DEBUG:boto:auth_path=/dghqvd-67/obj2 2023-10-11T15:51:24.364 DEBUG:boto:Method: PUT 2023-10-11T15:51:24.364 DEBUG:boto:Path: /dghqvd-67/obj2 2023-10-11T15:51:24.365 DEBUG:boto:Data: 2023-10-11T15:51:24.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:51:24.365 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:51:24.366 DEBUG:boto:Port: 8001 2023-10-11T15:51:24.366 DEBUG:boto:Params: {} 2023-10-11T15:51:24.367 DEBUG:boto:Token: None 2023-10-11T15:51:24.367 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:51:24 GMT /dghqvd-67/obj2 2023-10-11T15:51:24.368 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Gwwsl4LTc2dl8hwQ7xbqydUCpt4= 2023-10-11T15:51:24.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gwwsl4LTc2dl8hwQ7xbqydUCpt4='} 2023-10-11T15:51:24.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddc549f40fe2f225-006526c47c-232c-a2'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:51:24.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:24.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:24.381 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:24.448 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:24.449 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:24.499 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:51:24.519 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:51:24.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:24.520 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:51:24.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.520 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:24.521 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:51:24.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.521 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:24.521 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:51:24.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.522 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039484.295655_913.1", 2023-10-11T15:51:24.522 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:51:24.295655Z" 2023-10-11T15:51:24.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.523 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:51:24.524 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:24.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:51:24.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:24.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:24.526 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:24.590 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:24.591 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:24.633 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:24.630+0000 7f9ae7a6e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:24.634 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:24.631+0000 7f9ae7a6e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:24.649 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:24.649 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:24.650 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:24.650 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:24.650 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:24.650 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:24.651 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.651 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.651 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:24.651 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.651 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:24Z" 2023-10-11T15:51:24.664 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:24.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:24Z" } 2023-10-11T15:51:24.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:24.666 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:24.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:24.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:29.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:29.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:29.670 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:29.727 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:29.727 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:29.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:29.757+0000 7f3fa2e0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:29.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:29.758+0000 7f3fa2e0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:29.774 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:29.774 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.775 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:29Z" 2023-10-11T15:51:29.787 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:29.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:29Z" } 2023-10-11T15:51:29.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:29.789 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:29.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:29.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:34.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:34.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:34.792 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:34.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:34.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:34.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:34.881+0000 7f8033a04780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:34.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:34.882+0000 7f8033a04780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:34.898 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:34.898 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:34.899 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:34.899 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:34.899 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:34.899 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:34.900 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:34.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.900 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:34.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:34.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:34.901 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:34.901 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:34.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:34.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:34.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:34.902 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:34.902 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:34.902 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:34.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:34.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:34Z" 2023-10-11T15:51:34.911 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:34.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:34Z" } 2023-10-11T15:51:34.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:34.913 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:34.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:34.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:39.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:39.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:39.916 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:39.975 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:40.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:40.004+0000 7f8f5e3f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:40.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:40.005+0000 7f8f5e3f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:40.021 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:40.022 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:40.022 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:40.022 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:40.023 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:40.023 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:40.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:40.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.023 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:40.024 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:40.024 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:40.024 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:40.024 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:40.025 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:40.025 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:40.025 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:40.025 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:40.025 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:40.026 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:40.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:40.026 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.026 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:40.027 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:40.027 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:40.027 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:40.027 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:40.027 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:40.028 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:40.028 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:40.028 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:40.028 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:40.029 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:40.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:40.029 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:40.030 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:40.030 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:40.030 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:40.030 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:40.030 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:40.031 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:40.031 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:40.031 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:40.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:40.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:40.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:40.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:40.032 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:40.033 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:40.033 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:40.033 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:40.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:40.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:40.034 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:40.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:40.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:40.034 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:40.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.035 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:40.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:40.035 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:40.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:40.036 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:40Z" 2023-10-11T15:51:40.036 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:40.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:40Z" } 2023-10-11T15:51:40.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:40.038 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:40.039 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:40.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:45.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:45.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:45.041 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:45.098 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:45.099 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:45.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:45.129+0000 7f7d2f77a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:45.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:45.130+0000 7f7d2f77a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:45.146 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:45.146 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:45.146 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:45.147 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:45.147 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:45.148 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:45.148 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:45.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.149 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:45.149 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:45.150 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:45.150 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:45.150 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:45.151 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:45.151 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:45.151 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:45.152 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:45.152 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:45.152 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:45.152 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:45.153 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:45.154 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:45.154 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:45.154 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:45.154 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:45.155 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:45.155 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:45.155 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:45.155 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:45.155 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:45.156 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:45.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:45.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:45.156 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:45.157 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:45.157 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:45.157 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:45.157 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:45.157 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:45.158 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:45.158 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:45.158 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:45.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.159 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:45.159 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:45.159 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:45.159 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:45.159 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:45.160 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:45.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:45.161 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:45.161 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.161 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:45.161 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:45.162 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:45.162 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:45.162 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:45Z" 2023-10-11T15:51:45.162 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:45.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:45Z" } 2023-10-11T15:51:45.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:45.164 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:45.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:45.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:50.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:50.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:50.168 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:50.225 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:50.226 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:50.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:50.256+0000 7efd7c11c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:50.260 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:50.257+0000 7efd7c11c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:50.273 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:50.274 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:50.274 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:50.274 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:50.274 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:50.275 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:50.275 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:50.275 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.275 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:50.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:50.276 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:50.276 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:50.276 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:50.276 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:50.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:50.277 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:50.277 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:50.277 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:50.277 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:50.278 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:50.278 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.278 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:50.278 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:50.278 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:50.279 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:50.279 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:50.279 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:50.279 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:50.280 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:50.280 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:50.280 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:50.280 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:50.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.281 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:50.281 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:50.281 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:50.281 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:50.281 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:50.282 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:50.282 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:50.282 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:50.282 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:50.282 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:50.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:50.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.283 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:50.283 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:50.284 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:50.284 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:50.284 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:50.284 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:50.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:50.285 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:50.285 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:50.285 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:50.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:50.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:50.286 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:50.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.286 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:50.286 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:50.287 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:50.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:50.287 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:50Z" 2023-10-11T15:51:50.287 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:50.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:50Z" } 2023-10-11T15:51:50.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:50.290 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:50.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:50.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:51:55.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:55.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:51:55.292 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:51:55.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:51:55.349 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:51:55.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:55.376+0000 7fdf6f99a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:55.381 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:51:55.378+0000 7fdf6f99a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:51:55.394 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:51:55.394 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:51:55.394 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:51:55.394 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:51:55.395 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:51:55.395 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:51:55.395 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:55.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.396 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:51:55.396 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:55.396 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:51:55.396 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:55.396 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:55.397 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:51:55.397 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:55.397 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:51:55.397 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:51:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:55.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:51:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:55.399 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:51:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:55.400 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:51:55.401 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:55.401 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:55.401 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.401 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:55.402 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:51:55.402 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:55.402 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:55.402 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:51:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:51:55.403 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:55.404 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:55.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:51:55.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:51:55.404 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:51:55.405 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:51:55.405 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:51:55.405 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:51:55.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:51:55.405 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:51:55.406 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:51:55.406 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:51:55.406 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:55.406 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:51:55.406 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:51:55.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.407 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:51:55.407 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:51:55.407 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:51:55.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:51:55.408 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:51:55Z" 2023-10-11T15:51:55.408 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:51:55.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:51:55Z" } 2023-10-11T15:51:55.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:51:55.410 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:51:55.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:51:55.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:00.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:00.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:00.413 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:00.471 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:00.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:00.504+0000 7fcd08439780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:00.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:00.505+0000 7fcd08439780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:00.519 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:00.520 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:00.520 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:00.520 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:00.520 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:00.521 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:00.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:00.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.521 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:00.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:00.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:00.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:00.522 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:00.522 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:00.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:00.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:00.523 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:00.524 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:00.524 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:00.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:00.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:00.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:00.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:00.525 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:00.525 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:00.526 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:00.526 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:00.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:00.526 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:00.527 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:00.527 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:00.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.527 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:00.527 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:00.528 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:00.528 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:00.528 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:00.528 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:00.529 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:00.529 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:00.529 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:00.530 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:00.530 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:00.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.530 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:00.531 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:00.531 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:00.531 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:00.531 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:00.531 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:00.532 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:00.532 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:00.532 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:00.532 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:00.532 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:00.533 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:00.534 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:00.534 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:00.534 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:00Z" 2023-10-11T15:52:00.534 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:00.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:00Z" } 2023-10-11T15:52:00.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:00.536 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:00.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:00.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:05.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:05.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:05.540 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:05.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:05.597 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:05.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:05.628+0000 7f3235ac7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:05.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:05.629+0000 7f3235ac7780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:05.644 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:05.644 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:05.644 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:05.645 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:05.645 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:05.645 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:05.645 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:05.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.646 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:05.646 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:05.646 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:05.646 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:05.647 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:05.647 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:05.647 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:05.647 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:05.647 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:05.648 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:05.648 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:05.648 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:05.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.649 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:05.649 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:05.649 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:05.649 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:05.649 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:05.650 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:05.650 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:05.650 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:05.650 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:05.651 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:05.651 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:05.651 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.651 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:05.651 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:05.652 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:05.652 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:05.652 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:05.652 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:05.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:05.653 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:05.653 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:05.653 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:05.653 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:05.654 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:05.655 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:05.655 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:05.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:05.655 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:05.655 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:05.656 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:05.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:05.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:05.656 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:05.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.657 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:05.657 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:05.657 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:05.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:05.658 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:05Z" 2023-10-11T15:52:05.658 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:05.659 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:05Z" } 2023-10-11T15:52:05.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:05.660 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:05.660 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:05.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:10.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:10.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:10.663 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:10.720 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:10.721 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:10.753 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:10.752+0000 7f30bd8d5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:10.754 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:10.753+0000 7f30bd8d5780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:10.768 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:10.769 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:10.769 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:10.769 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:10.769 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:10.769 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:10.770 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:10.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.770 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:10.770 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:10.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:10.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:10.771 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:10.771 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:10.771 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:10.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:10.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:10.772 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:10.772 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:10.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:10.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.773 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:10.773 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:10.773 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:10.774 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:10.774 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:10.774 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:10.774 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:10.774 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:10.775 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:10.775 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:10.775 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:10.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.775 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:10.776 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:10.776 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:10.776 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:10.776 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:10.776 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:10.777 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:10.777 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:10.777 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:10.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:10.778 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:10.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.778 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:10.778 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:10.778 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:10.779 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:10.779 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:10.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:10.779 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:10.779 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:10.780 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:10.780 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:10.781 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:10.782 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:10.782 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:10.783 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.783 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:10.783 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:10.783 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:10.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:10.784 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:10Z" 2023-10-11T15:52:10.784 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:10.785 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:10Z" } 2023-10-11T15:52:10.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:10.786 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:10.787 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:10.787 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:15.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:15.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:15.789 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:15.846 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:15.847 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:15.878 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:15.877+0000 7f472a7ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:15.879 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:15.878+0000 7f472a7ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:15.893 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:15.893 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:15.893 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:15.894 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:15.894 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:15.894 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:15.894 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:15.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.895 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:15.895 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:15.895 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:15.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:15.896 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:15.896 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:15.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:15.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:15.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:15.897 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:15.897 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:15.897 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:15.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.897 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:15.898 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:15.898 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:15.898 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:15.898 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:15.899 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:15.899 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:15.899 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:15.899 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:15.899 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:15.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:15.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:15.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:15.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:15.901 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:15.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:15.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:15.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:15.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:15.902 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:15.902 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:15.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:15.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:15.903 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:15.903 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:15.903 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:15.903 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:15.904 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:15.904 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:15.904 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:15.904 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:15.904 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:15.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:15.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:15.905 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:15.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.905 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:15.906 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:15.906 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:15.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:15.906 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:15Z" 2023-10-11T15:52:15.907 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:15.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:15Z" } 2023-10-11T15:52:15.908 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:15.909 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:15.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:15.910 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:20.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:20.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:20.913 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:20.970 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:21.002 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:21.001+0000 7f69aa35a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:21.003 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:21.002+0000 7f69aa35a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:21.019 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:21.019 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:21.019 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:21.019 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:21.020 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:21.020 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:21.020 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:21.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.020 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:21.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:21.021 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:21.021 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:21.021 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:21.022 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:21.022 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:21.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:21.022 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:21.022 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:21.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:21.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:21.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:21.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:21.024 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:21.024 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:21.024 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:21.024 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:21.024 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:21.025 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:21.025 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:21.025 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:21.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:21.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.026 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:21.026 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:21.026 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:21.026 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:21.026 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:21.027 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:21.027 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:21.027 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:21.027 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:21.027 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:21.028 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:21.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.028 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:21.028 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:21.028 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:21.029 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:21.029 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:21.029 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:21.029 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:21.029 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:21.030 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:21.030 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:21.030 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:21.030 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:21.030 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:21.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.031 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:21.031 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:21.031 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:21.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:21.032 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:21Z" 2023-10-11T15:52:21.032 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:21.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:21Z" } 2023-10-11T15:52:21.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:21.034 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:21.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:21.035 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:26.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:26.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:26.037 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:26.094 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:26.095 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:26.127 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:26.126+0000 7f5397e48780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:26.130 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:26.127+0000 7f5397e48780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:26.142 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:26.142 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:26.142 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:26.143 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:26.143 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:26.143 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:26.143 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:26.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.144 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:26.144 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:26.144 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:26.144 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:26.145 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:26.145 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:26.145 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:26.145 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:26.146 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:26.146 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:26.146 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:26.146 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:26.146 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.147 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:26.147 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:26.147 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:26.147 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:26.148 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:26.148 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:26.148 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:26.148 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:26.149 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:26.149 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:26.149 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:26.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.149 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:26.150 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:26.150 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:26.150 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:26.150 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:26.151 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:26.151 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:26.151 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:26.151 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:26.152 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:26.152 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:26.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.152 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:26.153 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:26.153 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:26.153 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:26.153 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:26.154 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:26.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.156 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:26.156 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:26.156 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:26.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:26.157 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:26Z" 2023-10-11T15:52:26.157 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:26.158 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:26Z" } 2023-10-11T15:52:26.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:26.159 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:26.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:26.160 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:31.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:31.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:31.163 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:31.220 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:31.221 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:31.253 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:31.251+0000 7effad2e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:31.254 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:31.252+0000 7effad2e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:31.267 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:31.268 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:31.269 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:31.269 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:31.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.269 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:31.270 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:31.270 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:31.270 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:31.270 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:31.270 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:31.271 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:31.271 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:31.271 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:31.271 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:31.271 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:31.272 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:31.272 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.272 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:31.272 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:31.272 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:31.273 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:31.273 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:31.273 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:31.273 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:31.273 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:31.274 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:31.274 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:31.274 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:31.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:31.275 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:31.275 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:31.275 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:31.275 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:31.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:31.276 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:31.276 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:31.276 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:31.276 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:31.277 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:31.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.277 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:31.277 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:31.277 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:31.278 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:31.278 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:31.278 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:31.278 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:31.278 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:31.279 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:31.279 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:31.279 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:31.279 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:31.279 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:31.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.280 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:31.280 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:31.280 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:31.281 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:31.281 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:31Z" 2023-10-11T15:52:31.281 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:31.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:31Z" } 2023-10-11T15:52:31.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:31.283 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:31.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:31.284 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:36.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:36.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:36.287 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:36.346 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:36.346 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:36.378 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:36.377+0000 7f95bce7e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:36.379 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:36.378+0000 7f95bce7e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:36.393 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:36.393 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:36.394 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:36.394 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:36.394 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:36.394 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:36.395 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:36.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.395 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:36.395 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:36.395 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:36.396 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:36.396 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:36.396 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:36.396 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:36.397 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:36.397 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:36.397 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:36.397 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:36.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:36.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.398 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:36.398 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:36.398 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:36.399 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:36.399 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:36.399 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:36.399 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:36.399 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:36.400 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:36.400 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:36.400 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:36.400 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.400 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:36.401 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:36.401 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:36.401 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:36.401 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:36.402 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:36.402 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:36.402 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:36.402 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:36.402 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:36.403 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:36.403 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.403 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:36.403 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:36.403 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:36.404 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:36.404 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:36.404 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:36.404 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:36.404 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:36.405 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:36.405 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:36.405 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:36.405 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:36.406 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:36.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.406 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:36.406 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:36.406 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:36.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:36.407 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:36Z" 2023-10-11T15:52:36.407 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:36.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:36Z" } 2023-10-11T15:52:36.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:36.409 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:36.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:36.410 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:41.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:41.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:41.412 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:41.470 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:41.470 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:41.504 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:41.502+0000 7fbc6cfef780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:41.505 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:41.503+0000 7fbc6cfef780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:41.518 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:41.519 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:41.519 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:41.519 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:41.520 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:41.520 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:41.520 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:41.520 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.520 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:41.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:41.521 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:41.521 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:41.521 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:41.521 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:41.522 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:41.522 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:41.522 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:41.522 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:41.523 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:41.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:41.523 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.523 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:41.523 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:41.524 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:41.524 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:41.524 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:41.524 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:41.524 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:41.525 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:41.525 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:41.525 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:41.525 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:41.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.526 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:41.526 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:41.526 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:41.526 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:41.527 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:41.527 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:41.527 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:41.527 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:41.527 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:41.528 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:41.528 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:41.528 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.528 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:41.529 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:41.529 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:41.529 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:41.529 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:41.529 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:41.530 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:41.530 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:41.530 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:41.530 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:41.530 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:41.531 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:41.531 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:41.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.531 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:41.532 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:41.532 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:41.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:41.532 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:41Z" 2023-10-11T15:52:41.532 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:41.534 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:41Z" } 2023-10-11T15:52:41.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:41.535 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:41.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:41.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:46.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:46.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:46.538 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:46.595 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:46.625+0000 7fc0d7f5c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:46.628 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:46.626+0000 7fc0d7f5c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:46.642 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:46.642 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:46.642 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:46.643 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:46.643 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:46.643 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:46.643 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:46.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.644 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:46.644 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:46.644 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:46.645 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:46.646 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:46.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:46.647 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:46.647 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:46.648 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:46.648 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:46.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:46.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:46.648 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:46.649 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:46.649 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:46.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:46.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.650 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:46.650 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:46.650 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:46.650 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:46.650 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:46.651 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:46.651 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:46.651 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:46.651 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:46.652 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:46.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:46.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:46.652 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:46.653 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:46.653 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:46.653 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:46.653 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:46.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:46.654 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:46.654 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:46.654 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:46.654 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:46.655 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:46.655 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:46.655 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.655 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:46.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:46.656 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:46.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:46.656 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:46Z" 2023-10-11T15:52:46.656 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:46.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:46Z" } 2023-10-11T15:52:46.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:46.658 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:46.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:46.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:51.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:51.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:51.662 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:51.717 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:51.718 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:51.749 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:51.746+0000 7f4c6bc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:51.750 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:51.747+0000 7f4c6bc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:51.763 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:51.763 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:51.764 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:51.764 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:51.764 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:51.764 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:51.765 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:51.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.765 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:51.765 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:51.765 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:51.766 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:51.766 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:51.766 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:51.766 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:51.766 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:51.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:51.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:51.767 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:51.767 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:51.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:51.768 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:51.768 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:51.768 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:51.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:51.769 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:51.769 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:51.769 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:51.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:51.770 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:51.770 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:51.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.770 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:51.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:51.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:51.771 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:51.771 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:51.771 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:51.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:51.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:51.772 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:51.772 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:51.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:51.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.773 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:51.773 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:51.773 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:51.774 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:51.774 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:51.774 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:51.774 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:51.775 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:51.775 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:51.775 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:51.775 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:51.775 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:51.776 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:51.776 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.776 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:51.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:51.777 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:51.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:51.777 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:51Z" 2023-10-11T15:52:51.777 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:51.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:51Z" } 2023-10-11T15:52:51.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:51.780 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:51.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:51.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:52:55.285 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T15:52:55.282+0000 7fb229538700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-11T15:52:56.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:56.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:52:56.783 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:52:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:52:56.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:52:56.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:56.873+0000 7f8e89b4b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:56.876 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:52:56.874+0000 7f8e89b4b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:52:56.890 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:52:56.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:52:56.891 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:52:56.891 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:52:56.892 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:52:56.892 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:52:56.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:56.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.894 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:52:56.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:56.894 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:52:56.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:56.895 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:56.896 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:52:56.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:56.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:52:56.897 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:56.897 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:52:56.898 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:56.898 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:56.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:56.899 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:52:56.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:56.900 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:56.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:52:56.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:56.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:56.902 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:56.902 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:52:56.903 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:56.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:56.904 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.904 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:56.904 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:52:56.905 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:56.905 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:56.906 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:52:56.906 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:56.906 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:56.907 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:56.907 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:52:56.908 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:56.908 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:56.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:52:56.909 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:52:56.910 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:52:56.910 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:52:56.911 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:52:56.911 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:52:56.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:52:56.912 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:52:56.912 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:52:56.913 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:52:56.913 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:56.914 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:52:56.914 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:52:56.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.915 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:52:56.915 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:52:56.916 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:52:56.916 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:52:56.916 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:52:56Z" 2023-10-11T15:52:56.917 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:52:56.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:52:56Z" } 2023-10-11T15:52:56.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:52:56.920 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:52:56.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:52:56.922 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:53:01.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:01.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:01.924 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:01.983 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:01.983 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:02.016 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:02.014+0000 7f8c9cd0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:02.017 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:02.015+0000 7f8c9cd0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:02.031 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:02.031 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:02.032 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:02.033 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:02.034 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:53:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:53:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:02.035 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:02.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.036 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:02.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:02.036 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:02.036 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:02.037 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:02.037 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:02.037 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:02.037 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:02.037 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:53:02.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:02.038 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:02.038 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.038 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:02.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:02.039 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:02.039 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:02.039 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:02.039 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:02.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:02.040 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:02.040 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:53:02.040 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:02.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:02.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.041 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:02.041 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:02.042 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:02.042 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:02.042 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039461.459205_891.1", 2023-10-11T15:53:02.042 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:02.042 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:02.043 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:02.043 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T15:53:02.043 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:02.043 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:02.043 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:02.044 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:02.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.044 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:02.044 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:53:02.045 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:53:02.045 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:02.045 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:02Z" 2023-10-11T15:53:02.045 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:02.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039461.459205_891.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:01.459205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:53:02Z" } 2023-10-11T15:53:02.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T15:53:02.047 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:02.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T15:53:02.048 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T15:53:07.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.051 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.108 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:07.137+0000 7f8819985780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:07.140 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:07.138+0000 7f8819985780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:07.154 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:07.155 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:07.155 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:07.155 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:07.155 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:07.156 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:07.156 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.156 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:07.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.157 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:07.157 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.157 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.157 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:07.157 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.158 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:53:07.158 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.158 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:53:07.158 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.159 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.159 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.159 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.159 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:07.159 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.160 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.160 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:07.160 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.160 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.160 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.161 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:53:07.161 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.161 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.161 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.162 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:07.162 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.162 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.162 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:07.162 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.163 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.163 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.163 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:53:07.163 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.163 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.164 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.164 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.164 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:07.164 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.165 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.165 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039484.295655_913.1", 2023-10-11T15:53:07.165 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.165 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.166 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.166 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T15:53:07.166 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.166 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.167 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.167 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:07.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.167 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:07.168 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:53:07.168 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:53:07.168 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.168 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T15:53:07.168 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:07.169 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039484.295655_913.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:24.295655Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:53:07Z" } 2023-10-11T15:53:07.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039484.295655_913.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.295655Z', 'realm_epoch': 2}}] 2023-10-11T15:53:07.171 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:07.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:07.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:53:07.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.173 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.223 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.271 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:53:07.271 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.272 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:07.272 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:07.272 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.272 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.273 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:07.273 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:07.273 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.273 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.273 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:07.274 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:07.274 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.274 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.274 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:07.274 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:07.275 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.275 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:53:07.276 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:07.276 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:53:07.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.278 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.330 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.331 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.374 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:07.374 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:07.374 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:07.374 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:07.375 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:07.375 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:07.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.375 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:07.376 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.376 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:07.376 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.376 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:07.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:07.377 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.377 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:07.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:07.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.379 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:07.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:07.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.379 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.380 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:07.380 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.380 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.380 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.381 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.382 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:07.383 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:07.384 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:07.385 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.386 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T15:53:07.386 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:07.387 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:07Z" } 2023-10-11T15:53:07.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:07.388 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:07.389 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:07.389 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:53:07.390 DEBUG:boto:path=/dghqvd-66/ 2023-10-11T15:53:07.390 DEBUG:boto:auth_path=/dghqvd-66/ 2023-10-11T15:53:07.391 DEBUG:boto:Method: HEAD 2023-10-11T15:53:07.391 DEBUG:boto:Path: /dghqvd-66/ 2023-10-11T15:53:07.391 DEBUG:boto:Data: 2023-10-11T15:53:07.392 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.392 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:53:07.392 DEBUG:boto:Port: 8001 2023-10-11T15:53:07.393 DEBUG:boto:Params: {} 2023-10-11T15:53:07.394 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:53:07.394 DEBUG:boto:Token: None 2023-10-11T15:53:07.395 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-66/ 2023-10-11T15:53:07.395 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8BjPqGGqepAMgr9Qdvhu+Ty6Btg= 2023-10-11T15:53:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8BjPqGGqepAMgr9Qdvhu+Ty6Btg=', 'Content-Length': '0'} 2023-10-11T15:53:07.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7df2aba59629334-006526c4e3-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.401 DEBUG:boto:path=/dghqvd-66/obj1 2023-10-11T15:53:07.401 DEBUG:boto:auth_path=/dghqvd-66/obj1 2023-10-11T15:53:07.402 DEBUG:boto:Method: HEAD 2023-10-11T15:53:07.402 DEBUG:boto:Path: /dghqvd-66/obj1 2023-10-11T15:53:07.403 DEBUG:boto:Data: 2023-10-11T15:53:07.403 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.403 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:53:07.404 DEBUG:boto:Port: 8001 2023-10-11T15:53:07.404 DEBUG:boto:Params: {} 2023-10-11T15:53:07.405 DEBUG:boto:Token: None 2023-10-11T15:53:07.405 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-66/obj1 2023-10-11T15:53:07.405 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:++2ygImC4Ugc4Gm130iMHETmHok= 2023-10-11T15:53:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:++2ygImC4Ugc4Gm130iMHETmHok=', 'Content-Length': '0'} 2023-10-11T15:53:07.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-66:daa316e9-869c-4074-a756-a8adfb4a3573.8996.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002acbc670a5739d36-006526c4e3-232c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.409 DEBUG:boto:path=/dghqvd-66/obj1 2023-10-11T15:53:07.409 DEBUG:boto:auth_path=/dghqvd-66/obj1 2023-10-11T15:53:07.410 DEBUG:boto:Method: GET 2023-10-11T15:53:07.410 DEBUG:boto:Path: /dghqvd-66/obj1 2023-10-11T15:53:07.410 DEBUG:boto:Data: 2023-10-11T15:53:07.411 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.411 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:53:07.412 DEBUG:boto:Port: 8001 2023-10-11T15:53:07.412 DEBUG:boto:Params: {} 2023-10-11T15:53:07.413 DEBUG:boto:Token: None 2023-10-11T15:53:07.413 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-66/obj1 2023-10-11T15:53:07.414 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:U6nUSieF2ZZjN7mFM0Ep++VefzU= 2023-10-11T15:53:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U6nUSieF2ZZjN7mFM0Ep++VefzU=', 'Content-Length': '0'} 2023-10-11T15:53:07.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-66:daa316e9-869c-4074-a756-a8adfb4a3573.8996.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca19dff8a91a471f-006526c4e3-232c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.416 DEBUG:boto:path=/dghqvd-67/ 2023-10-11T15:53:07.417 DEBUG:boto:auth_path=/dghqvd-67/ 2023-10-11T15:53:07.417 DEBUG:boto:Method: HEAD 2023-10-11T15:53:07.418 DEBUG:boto:Path: /dghqvd-67/ 2023-10-11T15:53:07.418 DEBUG:boto:Data: 2023-10-11T15:53:07.419 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.419 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:07.419 DEBUG:boto:Port: 8000 2023-10-11T15:53:07.420 DEBUG:boto:Params: {} 2023-10-11T15:53:07.420 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:53:07.421 DEBUG:boto:Token: None 2023-10-11T15:53:07.421 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-67/ 2023-10-11T15:53:07.421 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:A+W7uV6JC2e4/T6fyh5Nq9pi498= 2023-10-11T15:53:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A+W7uV6JC2e4/T6fyh5Nq9pi498=', 'Content-Length': '0'} 2023-10-11T15:53:07.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abbc392af5ac44c6-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.426 DEBUG:boto:path=/dghqvd-67/obj2 2023-10-11T15:53:07.427 DEBUG:boto:auth_path=/dghqvd-67/obj2 2023-10-11T15:53:07.427 DEBUG:boto:Method: HEAD 2023-10-11T15:53:07.427 DEBUG:boto:Path: /dghqvd-67/obj2 2023-10-11T15:53:07.428 DEBUG:boto:Data: 2023-10-11T15:53:07.428 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.429 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:07.429 DEBUG:boto:Port: 8000 2023-10-11T15:53:07.429 DEBUG:boto:Params: {} 2023-10-11T15:53:07.430 DEBUG:boto:Token: None 2023-10-11T15:53:07.430 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-67/obj2 2023-10-11T15:53:07.431 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9U9+zeOXd50UbJ2nzGY2Ypu0fug= 2023-10-11T15:53:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9U9+zeOXd50UbJ2nzGY2Ypu0fug=', 'Content-Length': '0'} 2023-10-11T15:53:07.433 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000005875d73ddf36534-006526c4e3-2322-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.435 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2023-10-11T15:53:07.436 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-11T15:53:07.437 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-10-11T15:53:07.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.439 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.487 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.487 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.517 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:53:07.529 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:53:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:07.530 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:53:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:53:07.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:53:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.532 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039484.295655_913.1", 2023-10-11T15:53:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:51:24.295655Z" 2023-10-11T15:53:07.533 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.533 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:53:07.534 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:07.535 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:53:07.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.536 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.586 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:07.614+0000 7f035d347780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:07.617 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:07.615+0000 7f035d347780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:07.631 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:53:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:53:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:53:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.638 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.639 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:53:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.640 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039484.295655_913.1", 2023-10-11T15:53:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:07.641 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T15:53:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.642 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:53:07.643 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:53:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.644 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T15:53:07.644 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:07.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039484.295655_913.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:51:24.295655Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:53:07Z" } 2023-10-11T15:53:07.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039484.295655_913.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.295655Z', 'realm_epoch': 2}}] 2023-10-11T15:53:07.646 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:07.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T15:53:07.647 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:53:07.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:07.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:07.648 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:07.694 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.694 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.738 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:53:07.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:07.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:07.740 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:07.785 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.785 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.829 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:53:07.830 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-68 2023-10-11T15:53:07.831 DEBUG:boto:path=/dghqvd-68/ 2023-10-11T15:53:07.831 DEBUG:boto:auth_path=/dghqvd-68/ 2023-10-11T15:53:07.831 DEBUG:boto:Method: PUT 2023-10-11T15:53:07.832 DEBUG:boto:Path: /dghqvd-68/ 2023-10-11T15:53:07.832 DEBUG:boto:Data: 2023-10-11T15:53:07.833 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.833 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:07.833 DEBUG:boto:Port: 8000 2023-10-11T15:53:07.834 DEBUG:boto:Params: {} 2023-10-11T15:53:07.834 DEBUG:boto:Token: None 2023-10-11T15:53:07.835 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-68/ 2023-10-11T15:53:07.835 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HeImfQnPRmIaV/RebVFVUNHqiGk= 2023-10-11T15:53:07.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HeImfQnPRmIaV/RebVFVUNHqiGk=', 'Content-Length': '0'} 2023-10-11T15:53:07.851 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006fa3fdfcb0615b3e-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.851 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-69 2023-10-11T15:53:07.852 DEBUG:boto:path=/dghqvd-69/ 2023-10-11T15:53:07.852 DEBUG:boto:auth_path=/dghqvd-69/ 2023-10-11T15:53:07.853 DEBUG:boto:Method: PUT 2023-10-11T15:53:07.853 DEBUG:boto:Path: /dghqvd-69/ 2023-10-11T15:53:07.853 DEBUG:boto:Data: 2023-10-11T15:53:07.854 DEBUG:boto:Headers: {} 2023-10-11T15:53:07.854 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:07.855 DEBUG:boto:Port: 8000 2023-10-11T15:53:07.855 DEBUG:boto:Params: {} 2023-10-11T15:53:07.855 DEBUG:boto:Token: None 2023-10-11T15:53:07.856 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:53:07 GMT /dghqvd-69/ 2023-10-11T15:53:07.856 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jH02q8Ci+aAXbxTSBxBG64rd5EU= 2023-10-11T15:53:07.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jH02q8Ci+aAXbxTSBxBG64rd5EU=', 'Content-Length': '0'} 2023-10-11T15:53:07.871 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ead90a13f2e8e8cf-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:07.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.873 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:07.921 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:07.922 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:07.951 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:53:07.964 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:53:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:53:07.965 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:53:07.966 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:53:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:07.967 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:53:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:07.869026Z" 2023-10-11T15:53:07.968 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:07.968 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:53:07.969 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:07.970 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:53:07.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:07.971 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:08.025 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.026 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.061 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:08.059+0000 7fe695668780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:08.062 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:08.060+0000 7fe695668780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:08.074 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:08.075 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:08.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:08.077 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:53:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:53:08.078 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:08.079 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:08.080 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:53:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.081 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:08.082 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:53:08.083 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:08.084 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:53:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:08.085 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T15:53:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.086 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.087 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:53:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:53:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.088 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:08Z" 2023-10-11T15:53:08.088 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:08.089 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039587.869026_925.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:07.869026Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:53:08Z" } 2023-10-11T15:53:08.090 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039587.869026_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:07.869026Z', 'realm_epoch': 2}}] 2023-10-11T15:53:08.090 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:08.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:08.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:53:08.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.093 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.142 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.142 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.187 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:53:08.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.189 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.243 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.244 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.289 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:53:08.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dghqvd-68', '--dest-bucket=dghqvd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dghqvd-68', '--dest-bucket=dghqvd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.290 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=dghqvd-68 --dest-bucket=dghqvd-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.348 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.394 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:53:08.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.395 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.446 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.446 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.492 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:53:08.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.493 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.541 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.542 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.587 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:53:08.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.588 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.638 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.639 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.690 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:08.691 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:08.691 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 16, 2023-10-11T15:53:08.691 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:53:08.691 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:53:08.691 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:53:08.692 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:08.692 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:53:08.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.692 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:53:08.692 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:53:08.693 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:53:08.693 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:53:08.693 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.693 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:53:08.694 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.694 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:53:08.694 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:53:08.694 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:53:08.694 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:53:08.695 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.695 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:53:08.695 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:53:08.695 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.695 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:53:08.696 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.696 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:53:08.696 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:53:08.696 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:53:08.696 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:53:08.697 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:53:08.697 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:53:08.697 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:53:08.697 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:53:08.697 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:53:08.698 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:53:08.698 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.698 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.698 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.698 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.699 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:53:08.699 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:53:08.699 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.699 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:53:08.699 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.700 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:53:08.700 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:53:08.700 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:53:08.700 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:53:08.701 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:53:08.702 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.702 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.702 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.702 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.703 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:53:08.703 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.703 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:53:08.703 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:53:08.703 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:53:08.704 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:53:08.704 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.704 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.704 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.705 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:53:08.705 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:53:08.705 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:53:08.705 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T15:53:08.706 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.706 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T15:53:08.706 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T15:53:08.706 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T15:53:08.706 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.707 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T15:53:08.707 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:53:08.707 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:53:08.707 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:53:08.707 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T15:53:08.708 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.709 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T15:53:08.709 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:53:08.709 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-68", 2023-10-11T15:53:08.709 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:53:08.709 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:53:08.710 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:53:08.710 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.710 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.710 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T15:53:08.710 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-68", 2023-10-11T15:53:08.711 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:53:08.711 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:53:08.711 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:53:08.711 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.711 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T15:53:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:53:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T15:53:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T15:53:08.712 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.713 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.713 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T15:53:08.713 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T15:53:08.713 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T15:53:08.713 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T15:53:08.714 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.714 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.714 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.714 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T15:53:08.715 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.715 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.715 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.715 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:53:08.716 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.716 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.716 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.716 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.716 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:53:08.717 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:53:08.717 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:53:08.717 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:53:08.717 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.718 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:53:08.718 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.718 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:53:08.718 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:53:08.718 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:53:08.719 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:53:08.719 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.719 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:53:08.719 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:53:08.719 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.720 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:53:08.720 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.720 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:53:08.720 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:53:08.720 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:53:08.721 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:53:08.721 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:53:08.721 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:53:08.721 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:53:08.721 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:53:08.722 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:53:08.722 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:53:08.722 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.722 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.723 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.723 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.723 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:53:08.723 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:53:08.723 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:53:08.724 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:53:08.724 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.724 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:53:08.724 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:53:08.724 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:53:08.725 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:53:08.725 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:53:08.725 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:53:08.725 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:53:08.726 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:53:08.726 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:53:08.726 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:53:08.726 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.726 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:53:08.727 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:53:08.728 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:53:08.728 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:53:08.728 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.728 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.728 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.729 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:53:08.730 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:53:08.730 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.730 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:53:08.730 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:53:08.731 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.731 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:53:08.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:53:08.731 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.731 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.732 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:53:08.733 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:53:08.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.733 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.733 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:53:08.734 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:53:08.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.734 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:08.734 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:53:08.735 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:53:08.735 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.735 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:08.735 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.735 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:53:08.736 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:53:08.736 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:53:08.736 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:53:08.736 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:53:08.736 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:53:08.737 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:53:08.737 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:53:08.737 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:53:08.737 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.738 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:53:08.738 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:53:08.738 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:53:08.738 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:53:08.738 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:53:08.739 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:53:08.739 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.739 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:53:08.739 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:53:08.739 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:53:08.740 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:53:08.740 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:53:08.740 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:53:08.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.740 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:53:08.741 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:53:08.741 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:53:08.741 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:53:08.741 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:53:08.741 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:53:08.742 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.742 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:53:08.742 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:53:08.742 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:53:08.742 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:53:08.743 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:53:08.743 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:53:08.743 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:08.743 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:08.743 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:53:08.744 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:53:08.744 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:08.744 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:08.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:53:08.746 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:53:08.799 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:08.800 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:08.910 INFO:teuthology.orchestra.run.smithi158.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"dghqvd-68"},"dest":{"zone":"a1","bucket":"dghqvd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"dghqvd-68"},"dest":{"zone":"a2","bucket":"dghqvd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-10-11T15:53:08.911 DEBUG:boto:path=/dghqvd-68/ 2023-10-11T15:53:08.911 DEBUG:boto:auth_path=/dghqvd-68/ 2023-10-11T15:53:08.911 DEBUG:boto:Method: HEAD 2023-10-11T15:53:08.912 DEBUG:boto:Path: /dghqvd-68/ 2023-10-11T15:53:08.912 DEBUG:boto:Data: 2023-10-11T15:53:08.913 DEBUG:boto:Headers: {} 2023-10-11T15:53:08.913 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:08.913 DEBUG:boto:Port: 8000 2023-10-11T15:53:08.914 DEBUG:boto:Params: {} 2023-10-11T15:53:08.914 DEBUG:boto:Token: None 2023-10-11T15:53:08.915 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:08 GMT /dghqvd-68/ 2023-10-11T15:53:08.915 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OyH5GR5cQtap/5hQ/+o24n+X7SA= 2023-10-11T15:53:08.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OyH5GR5cQtap/5hQ/+o24n+X7SA=', 'Content-Length': '0'} 2023-10-11T15:53:08.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea7eb2c99e208d69-006526c4e4-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:08.981 DEBUG:boto:path=/dghqvd-68/obj1 2023-10-11T15:53:08.981 DEBUG:boto:auth_path=/dghqvd-68/obj1 2023-10-11T15:53:08.982 DEBUG:boto:Method: PUT 2023-10-11T15:53:08.982 DEBUG:boto:Path: /dghqvd-68/obj1 2023-10-11T15:53:08.983 DEBUG:boto:Data: 2023-10-11T15:53:08.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:53:08.983 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:08.984 DEBUG:boto:Port: 8000 2023-10-11T15:53:08.984 DEBUG:boto:Params: {} 2023-10-11T15:53:08.985 DEBUG:boto:Token: None 2023-10-11T15:53:08.985 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:53:08 GMT /dghqvd-68/obj1 2023-10-11T15:53:08.986 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a0oSY9OW79IP8iBPaw05xnxj2kY= 2023-10-11T15:53:08.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a0oSY9OW79IP8iBPaw05xnxj2kY='} 2023-10-11T15:53:09.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056e9a796b04e1665-006526c4e4-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.005 DEBUG:boto:path=/dghqvd-68/ 2023-10-11T15:53:09.005 DEBUG:boto:auth_path=/dghqvd-68/ 2023-10-11T15:53:09.006 DEBUG:boto:Method: HEAD 2023-10-11T15:53:09.006 DEBUG:boto:Path: /dghqvd-68/ 2023-10-11T15:53:09.007 DEBUG:boto:Data: 2023-10-11T15:53:09.007 DEBUG:boto:Headers: {} 2023-10-11T15:53:09.008 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.008 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.008 DEBUG:boto:Params: {} 2023-10-11T15:53:09.009 DEBUG:boto:Token: None 2023-10-11T15:53:09.009 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-68/ 2023-10-11T15:53:09.010 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg= 2023-10-11T15:53:09.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg=', 'Content-Length': '0'} 2023-10-11T15:53:09.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abe56cdd17094c64-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.016 DEBUG:boto:path=/dghqvd-68/obj2 2023-10-11T15:53:09.017 DEBUG:boto:auth_path=/dghqvd-68/obj2 2023-10-11T15:53:09.017 DEBUG:boto:Method: PUT 2023-10-11T15:53:09.017 DEBUG:boto:Path: /dghqvd-68/obj2 2023-10-11T15:53:09.018 DEBUG:boto:Data: 2023-10-11T15:53:09.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:53:09.019 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.019 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.020 DEBUG:boto:Params: {} 2023-10-11T15:53:09.020 DEBUG:boto:Token: None 2023-10-11T15:53:09.020 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-68/obj2 2023-10-11T15:53:09.021 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:z+69En56i8KJ+/pFU3EX1DDNUCk= 2023-10-11T15:53:09.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z+69En56i8KJ+/pFU3EX1DDNUCk='} 2023-10-11T15:53:09.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d488bed76932db49-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.071 DEBUG:boto:path=/dghqvd-68/ 2023-10-11T15:53:09.072 DEBUG:boto:auth_path=/dghqvd-68/ 2023-10-11T15:53:09.072 DEBUG:boto:Method: HEAD 2023-10-11T15:53:09.072 DEBUG:boto:Path: /dghqvd-68/ 2023-10-11T15:53:09.073 DEBUG:boto:Data: 2023-10-11T15:53:09.073 DEBUG:boto:Headers: {} 2023-10-11T15:53:09.074 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.074 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.075 DEBUG:boto:Params: {} 2023-10-11T15:53:09.075 DEBUG:boto:Token: None 2023-10-11T15:53:09.075 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-68/ 2023-10-11T15:53:09.076 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg= 2023-10-11T15:53:09.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg=', 'Content-Length': '0'} 2023-10-11T15:53:09.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046a0beb3ac0e496d-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.082 DEBUG:boto:path=/dghqvd-68/obj3 2023-10-11T15:53:09.083 DEBUG:boto:auth_path=/dghqvd-68/obj3 2023-10-11T15:53:09.083 DEBUG:boto:Method: PUT 2023-10-11T15:53:09.083 DEBUG:boto:Path: /dghqvd-68/obj3 2023-10-11T15:53:09.084 DEBUG:boto:Data: 2023-10-11T15:53:09.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:53:09.085 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.085 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.086 DEBUG:boto:Params: {} 2023-10-11T15:53:09.086 DEBUG:boto:Token: None 2023-10-11T15:53:09.087 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-68/obj3 2023-10-11T15:53:09.087 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a0RYF3WomOAgOOjDjBlPWwM863A= 2023-10-11T15:53:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a0RYF3WomOAgOOjDjBlPWwM863A='} 2023-10-11T15:53:09.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000829fa6e6f49083a7-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.098 DEBUG:boto:path=/dghqvd-69/ 2023-10-11T15:53:09.099 DEBUG:boto:auth_path=/dghqvd-69/ 2023-10-11T15:53:09.099 DEBUG:boto:Method: HEAD 2023-10-11T15:53:09.099 DEBUG:boto:Path: /dghqvd-69/ 2023-10-11T15:53:09.100 DEBUG:boto:Data: 2023-10-11T15:53:09.100 DEBUG:boto:Headers: {} 2023-10-11T15:53:09.101 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.101 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.101 DEBUG:boto:Params: {} 2023-10-11T15:53:09.102 DEBUG:boto:Token: None 2023-10-11T15:53:09.102 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-69/ 2023-10-11T15:53:09.103 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:t491/9/tz6qHYMvBT/5avjxLi3U= 2023-10-11T15:53:09.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t491/9/tz6qHYMvBT/5avjxLi3U=', 'Content-Length': '0'} 2023-10-11T15:53:09.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016d157d2b874b50c-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.110 DEBUG:boto:path=/dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T15:53:09.110 DEBUG:boto:auth_path=/dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T15:53:09.111 DEBUG:boto:Method: PUT 2023-10-11T15:53:09.111 DEBUG:boto:Path: /dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T15:53:09.112 DEBUG:boto:Data: 2023-10-11T15:53:09.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:53:09.112 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:53:09.113 DEBUG:boto:Port: 8000 2023-10-11T15:53:09.113 DEBUG:boto:Params: {} 2023-10-11T15:53:09.114 DEBUG:boto:Token: None 2023-10-11T15:53:09.114 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:53:09 GMT /dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T15:53:09.115 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:smoOnp+pM6Z0eWZko8vxUCFll2o= 2023-10-11T15:53:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smoOnp+pM6Z0eWZko8vxUCFll2o='} 2023-10-11T15:53:09.165 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079fc455dda6cec8a-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:53:09.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.167 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:09.237 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:09.237 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:09.309 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:53:09.330 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:53:09.331 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.331 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:09.331 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:53:09.332 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.332 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.332 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:09.332 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:53:09.333 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.333 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.333 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:09.333 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:53:09.334 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.334 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.334 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:53:09.334 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:07.869026Z" 2023-10-11T15:53:09.334 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.335 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:53:09.336 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:09.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:53:09.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.338 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:09.401 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:09.439 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:09.437+0000 7febbcb49780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:09.441 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:53:09.438+0000 7febbcb49780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:53:09.456 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:09.456 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:09.456 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:09.456 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:09.457 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:09.457 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:53:09.457 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:09.457 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.458 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:09.458 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.458 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:09.458 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.459 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:09.459 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:53:09.459 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.459 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:53:09.459 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.460 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:53:09.460 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:09.460 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.461 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.461 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:09.461 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.461 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:09.461 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:53:09.462 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.462 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:09.462 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.462 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:53:09.463 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:09.463 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.463 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.463 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.464 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:09.464 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.464 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:09.465 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:53:09.465 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.465 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:09.465 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.466 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:53:09.466 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:09.466 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.466 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.466 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.467 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:09.467 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.467 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:53:09.467 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:53:09.468 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.468 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:09.468 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.468 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T15:53:09.469 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:53:09.469 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.469 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.469 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:09.469 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.470 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:09.470 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:53:09.470 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:53:09.470 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.471 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T15:53:09.471 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:09.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039587.869026_925.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:07.869026Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:53:09Z" } 2023-10-11T15:53:09.473 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039587.869026_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:07.869026Z', 'realm_epoch': 2}}] 2023-10-11T15:53:09.473 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:09.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:53:09.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:53:09.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.476 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:09.524 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:09.525 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:09.574 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:53:09.574 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.574 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:53:09.575 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:53:09.575 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.575 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.575 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:09.576 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:09.576 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.576 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.576 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:53:09.576 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:53:09.577 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.577 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.577 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:53:09.577 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:53:09.578 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.578 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:53:09.579 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:09.579 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:53:09.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:09.581 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:09.632 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:09.632 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:09.676 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:09.677 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:09.677 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:09.677 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:09.677 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:09.678 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:09.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.678 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:09.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.679 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:09.679 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.679 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:09.679 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:09.679 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.680 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:09.680 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.680 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:09.680 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.681 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.681 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.681 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:09.681 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.681 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:09.682 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:09.682 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.682 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:09.682 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.683 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:09.683 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.683 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.683 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.683 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:09.684 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.684 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:09.684 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:09.684 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.684 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:09.685 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.685 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:09.685 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.685 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.685 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:09.686 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:09.686 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:09.686 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:09.686 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:09.687 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:09.687 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:09.687 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:09.687 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:09.687 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.688 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:09.688 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:09.688 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.688 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:09.689 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:09.689 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:09.689 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:09.689 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T15:53:09.689 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:09.691 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:09Z" } 2023-10-11T15:53:09.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:09.692 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:09.692 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:09.692 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:14.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:14.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:14.695 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:14.754 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:14.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:14.802 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:14.803 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:14.803 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:14.803 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:14.803 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:14.804 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:14.804 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.804 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:14.804 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:14.804 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:14.805 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:14.805 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:14.805 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:14.805 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:14.806 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:14.806 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:14.806 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:14.806 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:14.806 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.807 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:14.807 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:14.807 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:14.807 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:14.807 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:14.808 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:14.808 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:14.808 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:14.808 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:14.809 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:14.809 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.809 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:14.809 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:14.809 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:14.810 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:14.810 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:14.810 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:14.810 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:14.810 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:14.811 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:14.811 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:14.811 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.811 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:14.811 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:14.812 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:14.812 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:14.812 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:14.812 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:14.813 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:14.813 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:14.813 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:14.813 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:14.813 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:14.814 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:14.814 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.814 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:14.814 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:14.814 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:14.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:14.815 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:14Z" 2023-10-11T15:53:14.815 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:14.816 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:14Z" } 2023-10-11T15:53:14.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:14.817 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:14.818 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:14.818 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:19.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:19.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:19.821 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:19.880 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:19.881 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:19.930 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:19.931 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:19.931 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:19.931 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:19.931 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:19.931 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:19.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.932 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:19.932 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:19.932 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:19.933 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:19.933 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:19.933 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:19.933 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:19.933 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:19.934 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:19.934 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:19.934 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:19.934 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.935 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:19.935 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:19.935 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:19.935 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:19.935 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:19.936 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:19.936 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:19.936 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:19.936 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:19.937 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:19.937 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.937 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:19.937 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:19.938 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:19.938 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:19.938 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:19.938 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:19.939 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:19.939 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:19.939 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:19.939 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:19.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.940 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:19.940 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:19.940 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:19.940 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:19.940 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:19.941 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:19.941 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:19.941 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:19.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:19.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:19.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:19.942 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:19.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.942 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:19.943 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:19.943 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:19.943 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:19.943 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:19Z" 2023-10-11T15:53:19.943 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:19.944 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:19Z" } 2023-10-11T15:53:19.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:19.945 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:19.946 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:19.946 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:24.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:24.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:24.949 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:25.008 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:25.008 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:25.056 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:25.057 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:25.057 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:25.057 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:25.057 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:25.058 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:25.058 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.058 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:25.058 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:25.059 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:25.059 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:25.059 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:25.060 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:25.060 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:25.060 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:25.060 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:25.060 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:25.061 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:25.061 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.061 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:25.061 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:25.062 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:25.062 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:25.062 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:25.062 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:25.063 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:25.063 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:25.063 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:25.063 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:25.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.064 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:25.064 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:25.064 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:25.064 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:25.065 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:25.065 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:25.065 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:25.065 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:25.066 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:25.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:25.066 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.066 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:25.067 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:25.067 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:25.067 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:25.067 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:25.068 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:25.068 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:25.068 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:25.068 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:25.068 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:25.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:25.069 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:25.069 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.069 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:25.070 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:25.070 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:25.070 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:25.070 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:25Z" 2023-10-11T15:53:25.070 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:25.071 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:25Z" } 2023-10-11T15:53:25.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:25.073 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:25.073 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:25.073 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:30.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:30.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:30.076 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:30.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:30.136 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:30.184 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:30.184 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:30.184 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:30.185 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:30.185 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:30.185 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:30.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.185 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:30.186 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:30.186 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:30.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:30.186 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:30.187 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:30.187 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:30.187 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:30.187 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:30.188 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:30.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:30.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.189 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:30.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:30.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:30.189 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:30.189 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:30.190 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:30.190 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:30.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:30.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:30.190 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:30.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.191 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:30.191 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:30.191 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:30.191 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:30.192 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:30.192 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:30.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:30.192 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:30.193 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:30.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:30.193 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.193 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:30.193 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:30.194 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:30.194 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:30.194 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:30.194 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:30.194 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:30.195 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:30.195 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:30.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:30.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:30.195 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:30.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.196 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:30.196 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:30.196 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:30.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:30.197 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:30Z" 2023-10-11T15:53:30.197 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:30.198 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:30Z" } 2023-10-11T15:53:30.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:30.199 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:30.199 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:30.200 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:35.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:35.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:35.202 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:35.261 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:35.262 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:35.310 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:35.310 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:35.310 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:35.310 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:35.311 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:35.311 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:35.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.311 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:35.312 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:35.312 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:35.312 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:35.312 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:35.312 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:35.313 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:35.313 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:35.313 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:35.313 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:35.314 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:35.314 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.314 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:35.314 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:35.314 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:35.315 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:35.315 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:35.315 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:35.315 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:35.315 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:35.316 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:35.316 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:35.316 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.316 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:35.317 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:35.317 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:35.317 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:35.317 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:35.317 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:35.318 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:35.318 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:35.318 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:35.318 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:35.318 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.319 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:35.319 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:35.319 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:35.319 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:35.320 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:35.320 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:35.320 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:35.320 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:35.320 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:35.321 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:35.321 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:35.321 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:35.321 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.321 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:35.322 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:35.322 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:35.322 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:35.322 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:35Z" 2023-10-11T15:53:35.323 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:35.324 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:35Z" } 2023-10-11T15:53:35.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:35.325 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:35.325 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:35.325 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:40.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:40.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:40.328 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:40.387 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:40.387 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:40.436 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:40.436 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:40.436 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:40.436 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:40.437 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:40.437 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:40.437 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.438 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:40.438 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:40.438 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:40.438 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:40.439 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:40.439 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:40.439 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:40.439 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:40.440 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:40.440 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:40.440 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:40.440 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.440 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:40.441 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:40.441 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:40.441 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:40.441 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:40.441 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:40.442 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:40.442 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:40.442 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:40.442 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:40.443 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.443 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:40.443 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:40.443 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:40.443 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:40.444 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:40.444 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:40.444 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:40.444 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:40.445 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:40.445 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:40.445 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.445 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:40.446 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:40.446 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:40.446 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:40.446 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:40.446 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:40.447 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:40.447 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:40.447 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:40.447 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:40.448 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:40.448 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:40.448 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.448 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:40.449 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:40.449 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:40.449 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:40.449 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:40Z" 2023-10-11T15:53:40.449 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:40.451 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:40Z" } 2023-10-11T15:53:40.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:40.452 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:40.452 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:40.452 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:45.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:45.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:45.455 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:45.514 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:45.515 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:45.563 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:45.564 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:45.564 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:45.564 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:45.564 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:45.565 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:45.565 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.565 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:45.565 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:45.566 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:45.566 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:45.566 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:45.566 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:45.566 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:45.567 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:45.567 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:45.567 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:45.567 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:45.567 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.568 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:45.568 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:45.568 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:45.568 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:45.569 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:45.570 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:45.570 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.570 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:45.570 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:45.571 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:45.571 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:45.571 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:45.571 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:45.571 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:45.572 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:45.572 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:45.572 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:45.572 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.572 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:45.573 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:45.573 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:45.573 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:45.573 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:45.573 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:45.574 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:45.574 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:45.574 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:45.574 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:45.575 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:45.575 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:45.575 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.575 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:45.575 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:45.576 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:45.576 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:45.576 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:45Z" 2023-10-11T15:53:45.576 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:45.577 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:45Z" } 2023-10-11T15:53:45.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:45.578 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:45.579 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:45.579 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:50.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:50.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:50.582 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:50.642 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:50.643 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:50.690 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:50.690 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:50.690 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:50.691 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:50.691 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:50.691 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:50.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.692 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:50.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:50.692 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:50.692 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:50.692 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:50.693 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:50.693 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:50.693 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:50.693 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:50.693 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:50.694 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:50.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.694 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:50.694 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:50.695 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:50.695 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:50.695 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:50.695 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:50.695 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:50.696 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:50.696 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:50.696 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:50.696 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.696 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:50.697 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:50.697 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:50.697 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:50.697 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:50.697 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:50.698 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:50.698 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:50.698 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:50.698 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:50.699 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.699 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:50.699 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:50.699 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:50.699 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:50.700 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:50.700 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:50.700 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:50.700 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:50.700 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:50.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:50.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:50.701 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:50.701 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.701 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:50.702 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:50.702 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:50.702 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:50.702 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:50Z" 2023-10-11T15:53:50.702 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:50.703 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:50Z" } 2023-10-11T15:53:50.704 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:50.704 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:50.705 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:50.705 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:53:55.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:55.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:53:55.708 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:53:55.767 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:53:55.768 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:53:55.817 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:53:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:53:55.817 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:53:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:53:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:53:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:53:55.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:53:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:53:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:55.819 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:53:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:55.820 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:53:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:53:55.821 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:55.822 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:55.822 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:53:55.822 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:55.822 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:53:55.823 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:55.823 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:53:55.823 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:55.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.823 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:55.824 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:53:55.824 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:55.824 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:55.824 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:53:55.824 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:55.825 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:53:55.825 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:55.825 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:53:55.825 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:55.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.826 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:53:55.826 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:53:55.826 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:53:55.826 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:53:55.826 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:53:55.827 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:53:55.827 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:53:55.827 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:53:55.827 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:53:55.828 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:55.828 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:53:55.828 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:53:55.828 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.828 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:53:55.829 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:53:55.829 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:53:55.829 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:53:55.829 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:53:55Z" 2023-10-11T15:53:55.829 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:53:55.830 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:53:55Z" } 2023-10-11T15:53:55.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:53:55.832 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:53:55.832 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:53:55.832 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:00.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:00.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:00.835 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:00.894 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:00.894 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:00.941 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:00.942 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:00.942 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:00.942 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:00.943 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:00.943 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:00.943 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.943 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:00.943 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:00.944 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:00.944 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:00.944 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:00.944 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:00.944 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:00.945 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:00.945 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:00.945 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:00.945 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:00.946 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.946 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:00.946 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:00.946 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:00.946 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:00.947 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:00.947 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:00.947 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:00.947 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:00.947 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:00.948 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:00.948 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.948 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:00.948 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:00.949 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:00.949 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:00.949 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:00.949 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:00.950 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:00.950 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:00.950 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:00.950 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:00.950 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.951 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:00.951 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:00.951 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:00.951 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:00.951 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:00.952 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:00.952 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:00.952 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:00.952 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:00.952 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:00.953 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:00.953 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:00.953 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.953 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:00.954 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:00.954 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:00.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:00.954 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:00Z" 2023-10-11T15:54:00.954 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:00.956 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:00Z" } 2023-10-11T15:54:00.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:00.957 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:00.957 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:00.957 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:05.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:05.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:05.960 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:06.020 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:06.021 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:06.070 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:06.070 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:06.071 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:06.071 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:06.071 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:06.071 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:06.072 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.072 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:06.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:06.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:06.073 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:06.073 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:06.073 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:06.074 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:06.074 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:06.074 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:06.074 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:06.075 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:06.075 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.075 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:06.076 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:06.076 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:06.076 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:06.076 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:06.076 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:06.077 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:06.077 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:06.077 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:06.077 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:06.078 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.078 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:06.078 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:06.078 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:06.079 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:06.079 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:06.079 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:06.079 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:06.080 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:06.080 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:06.080 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:06.080 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.080 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:06.081 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:06.081 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:06.081 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:06.081 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:06.082 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:06.082 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:06.082 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:06.082 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:06.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:06.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:06.083 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:06.083 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.083 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:06.084 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:06.084 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:06.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:06.084 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:06Z" 2023-10-11T15:54:06.085 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:06.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:06Z" } 2023-10-11T15:54:06.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:06.087 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:06.087 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:06.088 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:11.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:11.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:11.094 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:11.153 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:11.153 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:11.202 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:11.202 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:11.203 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:11.203 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:11.203 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:11.203 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:11.203 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.204 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:11.204 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:11.204 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:11.204 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:11.204 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:11.205 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:11.205 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:11.205 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:11.205 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:11.205 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:11.206 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:11.206 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.206 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:11.206 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:11.207 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:11.207 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:11.207 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:11.207 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:11.207 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:11.208 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:11.208 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:11.208 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:11.208 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.209 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:11.209 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:11.209 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:11.209 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:11.209 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:11.210 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:11.210 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:11.210 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:11.210 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:11.210 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:11.211 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.211 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:11.211 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:11.211 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:11.211 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:11.212 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:11.212 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:11.212 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:11.212 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:11.213 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:11.213 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:11.213 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:11.213 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:11.213 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.214 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:11.214 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:11.214 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:11.214 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:11.214 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:11Z" 2023-10-11T15:54:11.215 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:11.216 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:11Z" } 2023-10-11T15:54:11.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:11.217 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:11.217 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:11.217 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:16.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:16.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:16.220 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:16.281 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:16.281 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:16.328 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:16.329 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:16.329 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:16.329 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:16.329 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:16.330 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:16.330 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.330 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:16.330 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:16.331 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:16.331 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:16.331 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:16.331 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:16.331 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:16.332 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:16.332 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:16.332 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:16.332 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:16.333 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.333 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:16.333 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:16.333 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:16.333 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:16.334 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:16.334 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:16.334 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:16.334 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:16.335 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:16.335 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:16.335 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.335 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:16.335 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:16.336 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:16.336 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:16.336 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:16.336 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:16.336 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:16.337 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:16.337 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:16.337 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:16.337 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.337 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:16.338 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:16.338 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:16.338 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:16.338 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:16.339 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:16.340 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:16.340 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:16.340 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.340 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:16.341 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:16.341 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:16.341 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:16.341 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:16Z" 2023-10-11T15:54:16.341 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:16.342 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:16Z" } 2023-10-11T15:54:16.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:16.343 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:16.344 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:16.344 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:21.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:21.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:21.347 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:21.406 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:21.406 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:21.454 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:21.455 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:21.455 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:21.455 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:21.455 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:21.456 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:21.456 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.456 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:21.456 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:21.456 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:21.457 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:21.457 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:21.457 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:21.457 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:21.458 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:21.458 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:21.458 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:21.458 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:21.458 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.459 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:21.459 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:21.459 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:21.459 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:21.460 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:21.460 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:21.460 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:21.460 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:21.460 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:21.461 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:21.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.461 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:21.461 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:21.462 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:21.462 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:21.462 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:21.462 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:21.463 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:21.463 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:21.463 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:21.463 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:21.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.464 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:21.464 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:21.464 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:21.464 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:21.465 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:21.465 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:21.465 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:21.465 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:21.465 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:21.466 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:21.466 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:21.466 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:21.466 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.467 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:21.467 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:21.467 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:21.467 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:21.467 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:21Z" 2023-10-11T15:54:21.468 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:21.469 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:21Z" } 2023-10-11T15:54:21.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:21.470 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:21.470 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:21.471 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:26.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:26.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:26.473 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:26.534 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:26.534 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:26.583 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:26.583 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:26.584 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:26.584 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:26.584 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:26.584 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:26.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.585 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:26.585 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:26.585 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:26.585 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:26.586 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:26.586 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:26.586 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:26.586 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:26.587 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:26.587 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:26.587 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:26.587 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.587 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:26.588 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:26.588 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:26.588 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:26.589 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:26.589 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:26.589 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:26.589 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:26.589 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:26.590 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:26.590 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.590 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:26.590 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:26.591 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:26.591 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:26.591 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:26.591 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:26.591 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:26.592 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:26.592 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:26.592 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:26.593 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.593 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:26.593 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:26.594 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:26.594 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:26.595 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:26.595 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:26.595 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:26.595 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:26.596 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:26.596 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:26.596 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:26.597 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:26.597 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.597 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:26.598 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:26.598 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:26.598 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:26.599 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:26Z" 2023-10-11T15:54:26.599 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:26.600 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:26Z" } 2023-10-11T15:54:26.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:26.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:26.602 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:26.602 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:31.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:31.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:31.605 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:31.665 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:31.666 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:31.714 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:31.715 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:31.715 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:31.715 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:31.715 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:31.716 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:31.716 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.716 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:31.716 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:31.717 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:31.718 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:31.718 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:31.718 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:31.718 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:31.719 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.719 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:31.719 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:31.719 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:31.720 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:31.720 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:31.720 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:31.720 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:31.720 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:31.721 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:31.721 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:31.721 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.721 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:31.722 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:31.722 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:31.722 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:31.722 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:31.722 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:31.723 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.724 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:31.724 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:31.724 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:31.724 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:31.725 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:31.725 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:31.725 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:31.725 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:31.725 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:31.726 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:31.726 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:31.726 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:31.726 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.726 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:31.727 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:31.727 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:31.727 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:31.727 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:31Z" 2023-10-11T15:54:31.727 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:31.729 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:31Z" } 2023-10-11T15:54:31.729 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:31.730 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:31.730 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:31.731 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:36.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:36.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:36.734 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:36.793 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:36.794 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:36.842 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:36.843 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:36.843 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:36.843 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:36.843 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:36.844 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:36.844 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.844 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:36.844 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:36.845 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:36.846 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:36.847 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:36.848 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:36.849 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:36.849 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.849 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:36.849 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:36.850 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:36.850 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:36.850 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:36.850 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:36.851 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:36.851 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:36.851 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:36.851 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:36.851 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.852 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:36.852 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:36.852 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:36.852 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:36.853 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:36.853 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:36.853 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:36.853 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:36.854 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:36.854 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:36.854 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:36.854 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:36.854 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.855 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:36.855 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:36.855 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:36.855 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:36.855 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:36Z" 2023-10-11T15:54:36.856 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:36.857 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:36Z" } 2023-10-11T15:54:36.858 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:36.858 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:36.859 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:36.859 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:41.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:41.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:41.862 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:41.921 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:41.921 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:41.971 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:41.971 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:41.971 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:41.972 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:41.972 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:41.972 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:41.972 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.972 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:41.973 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:41.973 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:41.973 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:41.973 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:41.973 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:41.974 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:41.974 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:41.974 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:41.974 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:41.974 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:41.975 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.975 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:41.975 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:41.975 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:41.976 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:41.976 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:41.976 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:41.976 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:41.976 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:41.977 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:41.978 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:41.978 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:41.978 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:41.978 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:41.979 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:41.979 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:41.979 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:41.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.979 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:41.980 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:41.980 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:41.980 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:41.980 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:41.980 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:41.981 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:41.981 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:41.981 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:41.981 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:41.981 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:41.982 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:41.982 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.982 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:41.982 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:41.983 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:41Z" 2023-10-11T15:54:41.983 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:41.984 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:41Z" } 2023-10-11T15:54:41.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:41.985 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:41.985 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:41.986 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:46.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:46.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:46.989 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:47.048 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:47.048 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:47.098 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:47.098 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:47.099 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:47.099 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:47.099 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:47.099 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:47.099 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.100 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:47.100 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:47.100 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:47.100 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:47.100 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:47.101 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:47.101 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:47.101 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:47.101 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:47.102 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:47.102 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:47.102 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.102 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:47.102 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:47.103 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:47.103 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:47.103 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:47.103 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:47.103 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:47.104 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:47.104 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:47.104 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:47.104 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.104 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:47.105 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:47.105 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:47.105 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:47.105 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:47.105 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:47.106 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:47.106 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:47.106 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:47.106 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:47.107 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:47.107 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:47.107 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:47.107 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:47.108 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:47.108 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:47.108 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:47.108 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:47.109 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:47.109 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:47.109 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:47.109 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:47.110 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.110 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:47.111 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:47.111 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:47.111 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:47.112 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:47Z" 2023-10-11T15:54:47.112 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:47.114 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:47Z" } 2023-10-11T15:54:47.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:47.116 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:47.117 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:47.117 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:52.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:52.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:52.120 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:52.179 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:52.227 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:52.227 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:52.228 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.229 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:52.229 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:52.229 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:52.229 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:52.230 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:52.230 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:52.230 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:52.230 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:52.230 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:52.231 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:52.231 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:52.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.231 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:52.231 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:52.232 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:52.232 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:52.232 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:52.232 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:52.232 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:52.233 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:52.233 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:52.233 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:52.233 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.234 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:52.234 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:52.234 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:52.234 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:52.234 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:52.235 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:52.235 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:52.235 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:52.236 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:52.236 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:52.236 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.236 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:52.236 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:52.237 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:52.238 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:52.239 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:52.240 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:52Z" 2023-10-11T15:54:52.240 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:52.241 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:52Z" } 2023-10-11T15:54:52.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:52.242 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:52.243 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:52.243 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:54:57.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:57.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:54:57.246 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:54:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:54:57.306 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:54:57.355 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:54:57.356 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:54:57.356 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:54:57.357 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:54:57.357 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:54:57.357 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:54:57.358 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.358 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:54:57.358 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:57.359 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:54:57.359 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:57.360 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:57.360 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:54:57.360 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:57.361 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:57.361 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:57.362 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:54:57.362 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:57.362 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.363 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:57.363 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:54:57.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:57.364 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:57.364 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:54:57.365 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:57.365 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:54:57.366 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:57.366 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:54:57.367 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:57.367 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.367 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:57.368 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:54:57.368 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:57.369 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:57.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:54:57.369 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:57.370 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:54:57.370 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:57.371 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:54:57.371 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:57.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:54:57.372 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:54:57.373 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:54:57.373 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:54:57.373 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:54:57.374 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:54:57.374 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:54:57.375 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:54:57.375 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:54:57.376 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:57.376 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:54:57.376 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:54:57.377 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.377 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:54:57.377 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:54:57.378 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:54:57.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:54:57.378 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:54:57Z" 2023-10-11T15:54:57.379 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:54:57.381 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:54:57Z" } 2023-10-11T15:54:57.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:54:57.382 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:54:57.383 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:54:57.383 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:55:02.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:02.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:02.386 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:02.446 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:02.447 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:02.495 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:02.495 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:02.496 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:02.496 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:02.496 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:02.497 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:02.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.497 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:02.497 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:02.497 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:02.498 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:02.498 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:02.498 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:55:02.498 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:02.499 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:02.499 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:02.499 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:55:02.499 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:02.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.500 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:02.500 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:02.500 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:02.500 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:02.500 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:02.501 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:02.501 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:02.501 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:02.501 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:02.501 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:02.502 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.502 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:02.502 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:02.502 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:02.503 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:02.503 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:55:02.503 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:02.503 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:02.503 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:02.504 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:55:02.504 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:02.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.504 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:02.504 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:02.505 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:02.505 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:02.505 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:55:02.505 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:02.506 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:02.506 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:02.506 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:55:02.506 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:02.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:02.507 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:02.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.507 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:02.508 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:02.508 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:02.508 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:02.508 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:02Z" 2023-10-11T15:55:02.509 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:02.510 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:02Z" } 2023-10-11T15:55:02.510 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:55:02.511 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:02.511 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:55:02.511 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:55:07.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:07.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:07.514 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:07.573 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:07.574 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:07.623 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:07.623 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:07.624 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:07.625 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T15:55:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:07.626 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T15:55:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:07.627 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:07.628 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:07.629 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:07.630 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:07.631 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T15:55:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:07.632 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T15:55:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:07.633 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T15:55:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:07.634 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T15:55:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:07.635 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:07.636 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:07.637 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:07Z" 2023-10-11T15:55:07.637 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:07.639 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045376", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:09.122137Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050970", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:51:24.245911Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044032", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:44:09.044541Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:07Z" } 2023-10-11T15:55:07.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T15:55:07.640 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:07.640 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T15:55:07.641 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T15:55:12.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.643 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:12.702 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:12.702 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:12.751 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:12.752 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:12.752 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:12.752 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:12.752 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:12.752 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:12.753 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.753 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:12.753 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.753 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:12.753 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:12.754 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:12.754 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:12.754 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:12.754 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:12.754 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:12.755 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:12.755 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.755 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.755 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.756 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:12.756 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:12.756 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:12.756 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:12.756 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:12.757 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:12.757 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:12.757 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:12.757 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.757 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.758 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.758 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:12.758 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:12.758 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:12.758 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:12.759 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.760 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.760 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:12.760 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:12.760 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:12.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:12.761 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:12.761 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:12.761 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:12.761 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:12.762 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.762 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.762 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:12.762 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.762 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:12.763 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:12.763 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:12.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.763 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:12Z" 2023-10-11T15:55:12.764 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:12.765 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:12Z" } 2023-10-11T15:55:12.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:12.766 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:12.766 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:12.767 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:55:12.767 DEBUG:boto:path=/dghqvd-68/ 2023-10-11T15:55:12.768 DEBUG:boto:auth_path=/dghqvd-68/ 2023-10-11T15:55:12.768 DEBUG:boto:Method: HEAD 2023-10-11T15:55:12.769 DEBUG:boto:Path: /dghqvd-68/ 2023-10-11T15:55:12.769 DEBUG:boto:Data: 2023-10-11T15:55:12.770 DEBUG:boto:Headers: {} 2023-10-11T15:55:12.770 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:55:12.771 DEBUG:boto:Port: 8001 2023-10-11T15:55:12.771 DEBUG:boto:Params: {} 2023-10-11T15:55:12.772 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:55:12.773 DEBUG:boto:Token: None 2023-10-11T15:55:12.773 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:55:12 GMT /dghqvd-68/ 2023-10-11T15:55:12.774 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wMO2+Kun5BYCNS7h1/tbxdbspC0= 2023-10-11T15:55:12.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wMO2+Kun5BYCNS7h1/tbxdbspC0=', 'Content-Length': '0'} 2023-10-11T15:55:12.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000055de4899916b366-006526c560-2451-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:12.781 DEBUG:boto:path=/dghqvd-68/obj1 2023-10-11T15:55:12.782 DEBUG:boto:auth_path=/dghqvd-68/obj1 2023-10-11T15:55:12.782 DEBUG:boto:Method: HEAD 2023-10-11T15:55:12.783 DEBUG:boto:Path: /dghqvd-68/obj1 2023-10-11T15:55:12.783 DEBUG:boto:Data: 2023-10-11T15:55:12.783 DEBUG:boto:Headers: {} 2023-10-11T15:55:12.784 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:55:12.784 DEBUG:boto:Port: 8001 2023-10-11T15:55:12.784 DEBUG:boto:Params: {} 2023-10-11T15:55:12.785 DEBUG:boto:Token: None 2023-10-11T15:55:12.785 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:55:12 GMT /dghqvd-68/obj1 2023-10-11T15:55:12.786 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T+KtNqibTadTT+JtBwfp6+CxnHk= 2023-10-11T15:55:12.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T+KtNqibTadTT+JtBwfp6+CxnHk=', 'Content-Length': '0'} 2023-10-11T15:55:12.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-68:daa316e9-869c-4074-a756-a8adfb4a3573.8996.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c624b81c10e1ad0-006526c560-2451-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:12.789 DEBUG:boto:path=/dghqvd-68/obj1 2023-10-11T15:55:12.790 DEBUG:boto:auth_path=/dghqvd-68/obj1 2023-10-11T15:55:12.790 DEBUG:boto:Method: GET 2023-10-11T15:55:12.795 DEBUG:boto:Path: /dghqvd-68/obj1 2023-10-11T15:55:12.795 DEBUG:boto:Data: 2023-10-11T15:55:12.796 DEBUG:boto:Headers: {} 2023-10-11T15:55:12.796 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:55:12.796 DEBUG:boto:Port: 8001 2023-10-11T15:55:12.797 DEBUG:boto:Params: {} 2023-10-11T15:55:12.797 DEBUG:boto:Token: None 2023-10-11T15:55:12.798 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:55:12 GMT /dghqvd-68/obj1 2023-10-11T15:55:12.798 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:+fhUKPIHQ6GkKTwksK49YyTRFbg= 2023-10-11T15:55:12.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+fhUKPIHQ6GkKTwksK49YyTRFbg=', 'Content-Length': '0'} 2023-10-11T15:55:12.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-68:daa316e9-869c-4074-a756-a8adfb4a3573.8996.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc7378ec3049b81-006526c560-2451-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:12.802 DEBUG:boto:path=/dghqvd-68/obj2 2023-10-11T15:55:12.802 DEBUG:boto:auth_path=/dghqvd-68/obj2 2023-10-11T15:55:12.802 DEBUG:boto:Method: HEAD 2023-10-11T15:55:12.803 DEBUG:boto:Path: /dghqvd-68/obj2 2023-10-11T15:55:12.803 DEBUG:boto:Data: 2023-10-11T15:55:12.804 DEBUG:boto:Headers: {} 2023-10-11T15:55:12.804 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:55:12.804 DEBUG:boto:Port: 8001 2023-10-11T15:55:12.805 DEBUG:boto:Params: {} 2023-10-11T15:55:12.805 DEBUG:boto:Token: None 2023-10-11T15:55:12.806 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:55:12 GMT /dghqvd-68/obj2 2023-10-11T15:55:12.806 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6iyxDcB8fFjs8V9QR8kIXYPgcW0= 2023-10-11T15:55:12.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6iyxDcB8fFjs8V9QR8kIXYPgcW0=', 'Content-Length': '0'} 2023-10-11T15:55:12.808 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000074f64392d3c581b5-006526c560-2451-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:12.811 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2023-10-11T15:55:12.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.813 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:12.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:12.863 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:12.892 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:55:12.905 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:55:12.905 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.905 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:12.906 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:55:12.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.906 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.906 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:12.907 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:55:12.907 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.907 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:12.907 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:55:12.908 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:12.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:12.908 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:55:12.908 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:07.869026Z" 2023-10-11T15:55:12.909 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:12.909 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:55:12.910 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:55:12.911 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:55:12.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:12.912 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:12.963 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:12.963 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:12.991 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:12.989+0000 7feb907d0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:12.992 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:12.990+0000 7feb907d0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:13.008 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:13.009 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:13.009 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:13.009 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:13.009 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:13.010 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:55:13.010 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.010 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.010 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:13.010 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.011 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:13.011 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.011 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.011 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:13.012 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.012 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:55:13.012 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.012 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:55:13.012 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.013 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.013 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.013 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.013 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:13.013 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.014 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.014 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:13.014 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.014 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.014 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.015 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:55:13.015 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.015 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.015 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.015 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.016 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:13.016 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.016 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.016 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:13.016 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.017 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.018 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.018 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:13.018 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.018 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.018 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039587.869026_925.1", 2023-10-11T15:55:13.019 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.019 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.019 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.019 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T15:55:13.020 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.020 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.021 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:13.021 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:55:13.021 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:55:13.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.021 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:12Z" 2023-10-11T15:55:13.022 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:13.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039587.869026_925.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:07.869026Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:55:12Z" } 2023-10-11T15:55:13.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039587.869026_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:07.869026Z', 'realm_epoch': 2}}] 2023-10-11T15:55:13.024 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:55:13.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T15:55:13.024 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:55:13.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.026 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.071 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.072 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.115 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:55:13.116 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-70 2023-10-11T15:55:13.116 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:55:13.117 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:55:13.117 DEBUG:boto:Method: PUT 2023-10-11T15:55:13.118 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:55:13.118 DEBUG:boto:Data: 2023-10-11T15:55:13.118 DEBUG:boto:Headers: {} 2023-10-11T15:55:13.119 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:13.119 DEBUG:boto:Port: 8000 2023-10-11T15:55:13.120 DEBUG:boto:Params: {} 2023-10-11T15:55:13.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:55:13.121 DEBUG:boto:Token: None 2023-10-11T15:55:13.121 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:55:13 GMT /dghqvd-70/ 2023-10-11T15:55:13.122 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sTJtqzQ45KRGGhbdJqx1QqldC44= 2023-10-11T15:55:13.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sTJtqzQ45KRGGhbdJqx1QqldC44=', 'Content-Length': '0'} 2023-10-11T15:55:13.138 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2e84819940ca892-006526c561-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:13.139 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-71 2023-10-11T15:55:13.139 DEBUG:boto:path=/dghqvd-71/ 2023-10-11T15:55:13.140 DEBUG:boto:auth_path=/dghqvd-71/ 2023-10-11T15:55:13.140 DEBUG:boto:Method: PUT 2023-10-11T15:55:13.140 DEBUG:boto:Path: /dghqvd-71/ 2023-10-11T15:55:13.141 DEBUG:boto:Data: 2023-10-11T15:55:13.141 DEBUG:boto:Headers: {} 2023-10-11T15:55:13.142 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:13.142 DEBUG:boto:Port: 8000 2023-10-11T15:55:13.142 DEBUG:boto:Params: {} 2023-10-11T15:55:13.143 DEBUG:boto:Token: None 2023-10-11T15:55:13.143 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 15:55:13 GMT /dghqvd-71/ 2023-10-11T15:55:13.144 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:4P0/ySWpSDsw2+vzHjz1vjIK2Pc= 2023-10-11T15:55:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4P0/ySWpSDsw2+vzHjz1vjIK2Pc=', 'Content-Length': '0'} 2023-10-11T15:55:13.159 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd8fa6d10de6e0d0-006526c561-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:13.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:13.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:13.160 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:13.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.209 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.241 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:55:13.255 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:55:13.255 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.255 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:13.255 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:55:13.256 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.256 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.256 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:13.256 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:55:13.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.257 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.257 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:13.257 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:55:13.257 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.258 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.258 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:55:13.258 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:55:13.156327Z" 2023-10-11T15:55:13.258 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.259 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:55:13.260 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:13.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:55:13.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:13.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:13.262 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.315 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.343 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:13.341+0000 7fb61e279780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:13.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:13.342+0000 7fb61e279780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:13.356 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:13.357 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:13.357 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:13.357 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:13.357 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:55:13.358 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.358 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.358 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:13.358 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.359 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:13.359 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.359 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.359 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:13.359 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.360 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:55:13.360 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.360 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:55:13.360 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.360 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.361 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.361 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.361 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:13.361 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.361 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.362 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:13.362 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.362 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.362 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.363 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:55:13.363 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.363 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.363 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.364 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:13.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.364 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.364 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:13.364 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.365 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.365 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.365 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:55:13.365 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.365 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.366 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.366 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:13.366 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:13.366 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:13.367 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:55:13.367 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:13.367 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:13.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:13.367 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:55:13.368 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.368 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.368 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.368 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.369 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:13.369 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:55:13.369 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:55:13.369 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.369 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:13Z" 2023-10-11T15:55:13.370 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:13.371 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:55:13Z" } 2023-10-11T15:55:13.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:55:13.372 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:13.372 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:13.372 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:55:13.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.374 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.424 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.424 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.470 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:55:13.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.472 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.525 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.526 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.571 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:55:13.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-70', '--dest-bucket=dghqvd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-70', '--dest-bucket=dghqvd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.573 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-70 --dest-bucket=dghqvd-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.628 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.629 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.674 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-70","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:55:13.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.676 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.726 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.726 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.772 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-70","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:55:13.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.774 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.822 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.823 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:13.873 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:13.874 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:55:13.874 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 17, 2023-10-11T15:55:13.874 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:55:13.875 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:55:13.875 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:55:13.875 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:55:13.875 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:55:13.876 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.876 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:55:13.876 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:55:13.877 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:55:13.877 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:55:13.877 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.877 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:55:13.877 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.878 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:55:13.878 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:55:13.878 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:55:13.878 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:55:13.878 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.879 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:55:13.879 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:55:13.879 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.879 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:55:13.880 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.880 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:55:13.880 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:55:13.880 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:55:13.880 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:55:13.881 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:55:13.881 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:55:13.881 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:55:13.881 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:55:13.881 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:55:13.882 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:55:13.882 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.882 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.882 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.883 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.883 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:55:13.883 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:55:13.884 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.884 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:55:13.884 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.884 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:55:13.885 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:55:13.885 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:55:13.885 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:55:13.885 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:55:13.885 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:55:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:55:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:55:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:55:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:55:13.886 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.887 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.887 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.887 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.887 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:55:13.888 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.888 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:55:13.888 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:55:13.888 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:55:13.889 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:55:13.889 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.889 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.889 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:55:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:55:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:55:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T15:55:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T15:55:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T15:55:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T15:55:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T15:55:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:55:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:55:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:55:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T15:55:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T15:55:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:55:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-70", 2023-10-11T15:55:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:55:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:55:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:55:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T15:55:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-71", 2023-10-11T15:55:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:55:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:55:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:55:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T15:55:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:55:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T15:55:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T15:55:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T15:55:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T15:55:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T15:55:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T15:55:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T15:55:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:55:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:55:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:55:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:55:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:55:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.904 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:55:13.904 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.904 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:55:13.904 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:55:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:55:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:55:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:55:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:55:13.906 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.906 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:55:13.906 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.906 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:55:13.907 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:55:13.907 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:55:13.907 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:55:13.907 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:55:13.908 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:55:13.908 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:55:13.908 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:55:13.908 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:55:13.908 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:55:13.909 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.909 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.910 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:55:13.910 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:55:13.910 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:55:13.910 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:55:13.910 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.911 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:55:13.911 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:55:13.911 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:55:13.911 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:55:13.911 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:55:13.912 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:55:13.912 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:55:13.912 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:55:13.912 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:55:13.912 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:55:13.913 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.913 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.913 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.913 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.914 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:55:13.914 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.915 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:55:13.915 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:55:13.915 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:55:13.915 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:55:13.915 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.916 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.916 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.916 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:55:13.916 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:55:13.917 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:55:13.917 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:55:13.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.917 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:55:13.918 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:55:13.918 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.918 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.918 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:55:13.919 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:55:13.919 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.919 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:55:13.919 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:55:13.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.920 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.920 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:55:13.921 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:55:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:55:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:55:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:55:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:55:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:55:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:55:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:55:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:55:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:55:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:55:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:55:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:55:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:55:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:55:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:55:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:55:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:55:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:55:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:55:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:55:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:55:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:55:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:55:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:55:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:55:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:55:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:55:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:55:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:55:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:55:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:55:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:55:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:55:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:55:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:55:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:55:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:55:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:55:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:55:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:13.934 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:13.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:55:13.935 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:55:13.981 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:13.982 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:14.030 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-70","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:55:14.030 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:55:14.031 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:55:14.031 DEBUG:boto:Method: HEAD 2023-10-11T15:55:14.031 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:55:14.032 DEBUG:boto:Data: 2023-10-11T15:55:14.032 DEBUG:boto:Headers: {} 2023-10-11T15:55:14.033 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:14.033 DEBUG:boto:Port: 8000 2023-10-11T15:55:14.033 DEBUG:boto:Params: {} 2023-10-11T15:55:14.034 DEBUG:boto:Token: None 2023-10-11T15:55:14.034 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:55:14 GMT /dghqvd-70/ 2023-10-11T15:55:14.035 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hk2+TfHs2J8TSrwWtFIgrtDMBAE= 2023-10-11T15:55:14.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hk2+TfHs2J8TSrwWtFIgrtDMBAE=', 'Content-Length': '0'} 2023-10-11T15:55:14.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087ab3510d3a71ef8-006526c562-2576-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:14.125 DEBUG:boto:path=/dghqvd-70/obj1 2023-10-11T15:55:14.126 DEBUG:boto:auth_path=/dghqvd-70/obj1 2023-10-11T15:55:14.126 DEBUG:boto:Method: PUT 2023-10-11T15:55:14.127 DEBUG:boto:Path: /dghqvd-70/obj1 2023-10-11T15:55:14.127 DEBUG:boto:Data: 2023-10-11T15:55:14.127 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:55:14.128 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:14.128 DEBUG:boto:Port: 8000 2023-10-11T15:55:14.129 DEBUG:boto:Params: {} 2023-10-11T15:55:14.129 DEBUG:boto:Token: None 2023-10-11T15:55:14.130 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:55:14 GMT /dghqvd-70/obj1 2023-10-11T15:55:14.130 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ej/BRK89UOu6egK8qzsxE1edojs= 2023-10-11T15:55:14.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:55:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ej/BRK89UOu6egK8qzsxE1edojs='} 2023-10-11T15:55:14.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a333a42d19f33099-006526c562-2576-a1'), ('Date', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:14.145 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:55:14.145 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:55:14.146 DEBUG:boto:Method: HEAD 2023-10-11T15:55:14.146 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:55:14.146 DEBUG:boto:Data: 2023-10-11T15:55:14.147 DEBUG:boto:Headers: {} 2023-10-11T15:55:14.147 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:14.148 DEBUG:boto:Port: 8000 2023-10-11T15:55:14.148 DEBUG:boto:Params: {} 2023-10-11T15:55:14.149 DEBUG:boto:Token: None 2023-10-11T15:55:14.149 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:55:14 GMT /dghqvd-70/ 2023-10-11T15:55:14.150 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hk2+TfHs2J8TSrwWtFIgrtDMBAE= 2023-10-11T15:55:14.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hk2+TfHs2J8TSrwWtFIgrtDMBAE=', 'Content-Length': '0'} 2023-10-11T15:55:14.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000290237a2fd1d2b7a-006526c562-2576-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:14.154 DEBUG:boto:path=/dghqvd-70/obj2 2023-10-11T15:55:14.154 DEBUG:boto:auth_path=/dghqvd-70/obj2 2023-10-11T15:55:14.155 DEBUG:boto:Method: PUT 2023-10-11T15:55:14.155 DEBUG:boto:Path: /dghqvd-70/obj2 2023-10-11T15:55:14.155 DEBUG:boto:Data: 2023-10-11T15:55:14.156 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:55:14.156 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:55:14.157 DEBUG:boto:Port: 8000 2023-10-11T15:55:14.157 DEBUG:boto:Params: {} 2023-10-11T15:55:14.157 DEBUG:boto:Token: None 2023-10-11T15:55:14.158 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:55:14 GMT /dghqvd-70/obj2 2023-10-11T15:55:14.158 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:uy9sqv4XANUzVdE4WXJPgL03wyU= 2023-10-11T15:55:14.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:55:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uy9sqv4XANUzVdE4WXJPgL03wyU='} 2023-10-11T15:55:14.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000662443e01538efc3-006526c562-2576-a1'), ('Date', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:55:14.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.208 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:14.283 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:14.283 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:14.380 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:55:14.424 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:55:14.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:14.425 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:55:14.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.425 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.425 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:14.426 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:55:14.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.426 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:14.427 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:55:14.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.427 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:55:14.427 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:55:13.156327Z" 2023-10-11T15:55:14.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.429 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:55:14.430 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:14.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:55:14.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.432 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:14.507 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:14.508 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:14.577 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:14.575+0000 7fe07bd48780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:14.579 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:55:14.578+0000 7fe07bd48780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:55:14.603 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:14.603 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:14.604 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:14.604 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:14.604 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:14.604 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:55:14.605 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:14.605 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.605 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:14.605 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.606 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:14.606 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.606 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:14.606 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:55:14.607 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.607 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:55:14.607 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.607 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:55:14.608 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:14.608 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.608 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.609 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:14.609 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.609 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:14.610 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:55:14.610 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.610 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:14.610 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.610 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:55:14.611 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:14.611 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.611 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.611 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.612 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:14.612 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.612 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:14.612 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:55:14.612 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.613 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:14.613 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.613 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:55:14.613 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:14.613 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.614 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.614 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.614 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:14.614 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.614 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:55:14.615 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:55:14.615 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.615 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:14.615 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:55:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:55:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.616 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:14.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.617 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:14.617 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:55:14.617 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:55:14.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.618 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:14Z" 2023-10-11T15:55:14.618 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:14.619 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:55:14Z" } 2023-10-11T15:55:14.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:55:14.620 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:14.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:55:14.621 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:55:14.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.623 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:14.685 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:14.686 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:14.758 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:55:14.758 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.759 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:55:14.759 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:55:14.759 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.759 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.759 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:14.760 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:14.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.760 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.760 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:55:14.760 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:55:14.761 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.761 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:14.761 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:14.762 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.762 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:55:14.763 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:14.763 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:55:14.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:14.765 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:14.817 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:14.861 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:14.861 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:14.862 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:14.862 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:14.862 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:14.862 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:14.863 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.863 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:14.863 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.863 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:14.863 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.864 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:14.864 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:14.864 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.864 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:14.865 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.865 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:14.865 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.865 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.866 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:14.866 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.866 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:14.867 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:14.867 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.867 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:14.867 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.867 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.870 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:14.870 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.870 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.870 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:14.870 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:14.871 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:14.871 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:14.871 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:14.871 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:14.872 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:14.872 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:14.872 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:14.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:14.873 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:14.873 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.873 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:14.873 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:14.873 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:14.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:14.874 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:14Z" 2023-10-11T15:55:14.874 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:14.875 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:14Z" } 2023-10-11T15:55:14.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:14.876 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:14.877 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:14.877 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:19.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:19.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:19.880 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:19.941 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:19.942 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:19.989 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:19.989 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:19.989 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:19.990 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:19.990 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:19.990 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:19.991 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:19.991 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:19.991 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:19.991 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:19.991 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:19.992 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:19.992 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:19.992 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:19.992 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:19.993 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:19.993 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:19.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:19.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:19.994 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:19.994 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:19.994 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:19.994 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:19.995 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:19.995 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:19.995 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:19.995 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:19.996 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:19.996 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:19.996 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:19.996 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:19.996 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:19.997 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:19.997 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:19.997 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:19.997 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:19.998 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:19.998 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:19.998 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:19.999 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:19.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:19.999 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:19.999 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:20.000 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:20.000 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:20.000 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:20.001 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:20.001 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:20.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:20.002 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:20.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:20.002 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:20.002 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:20.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:20.003 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:20.003 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:20.003 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:20.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:20.004 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:19Z" 2023-10-11T15:55:20.004 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:20.005 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:19Z" } 2023-10-11T15:55:20.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:20.006 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:20.007 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:20.007 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:25.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:25.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:25.009 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:25.068 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:25.116 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:25.116 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:25.117 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:25.117 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:25.117 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:25.117 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:25.118 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.118 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:25.118 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:25.119 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:25.119 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:25.119 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:25.119 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:25.119 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:25.120 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:25.120 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:25.120 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:25.120 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:25.120 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:25.121 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:25.121 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:25.121 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:25.121 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:25.122 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:25.122 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:25.122 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:25.122 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:25.122 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:25.123 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.123 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:25.123 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:25.123 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:25.123 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:25.124 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:25.125 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.125 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:25.125 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:25.125 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:25.125 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:25.126 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:25.126 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:25.126 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:25.126 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:25.126 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:25.127 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:25.127 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:25.127 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:25.127 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.127 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:25.128 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:25.128 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:25.128 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:25.128 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:25Z" 2023-10-11T15:55:25.128 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:25.129 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:25Z" } 2023-10-11T15:55:25.130 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:25.130 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:25.131 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:25.131 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:30.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:30.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:30.133 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:30.193 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:30.246 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:30.246 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:30.247 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:30.247 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:30.247 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:30.247 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:30.248 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.248 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:30.248 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:30.248 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:30.249 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:30.249 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:30.249 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:30.249 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:30.249 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:30.250 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:30.250 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:30.250 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:30.250 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.251 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:30.251 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:30.251 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:30.251 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:30.251 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:30.252 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:30.252 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:30.252 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:30.252 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:30.252 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:30.253 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.253 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:30.253 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:30.253 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:30.253 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:30.254 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:30.254 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:30.254 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:30.254 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:30.254 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:30.255 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:30.255 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.255 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:30.255 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:30.255 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:30.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:30.256 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:30.256 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:30.256 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:30.257 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:30.257 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:30.257 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:30.257 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:30.257 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:30.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.258 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:30.258 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:30.258 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:30.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:30.259 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:30Z" 2023-10-11T15:55:30.259 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:30.259 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:30Z" } 2023-10-11T15:55:30.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:30.260 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:30.261 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:30.261 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:35.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:35.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:35.264 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:35.322 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:35.323 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:35.371 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:35.371 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:35.372 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:35.372 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:35.372 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:35.372 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:35.373 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.373 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:35.373 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:35.373 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:35.374 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:35.374 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:35.374 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:35.374 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:35.374 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:35.375 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:35.375 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:35.375 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:35.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:35.376 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:35.376 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:35.376 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:35.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:35.376 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:35.377 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:35.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:35.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:35.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:35.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:35.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:35.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:35.378 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:35.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:35.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:35.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:35.379 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:35.379 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:35.380 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:35.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.380 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:35.380 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:35.380 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:35.381 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:35.381 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:35.381 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:35.381 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:35.381 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:35.382 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:35.382 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:35.382 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:35.382 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:35.382 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.383 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:35.383 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:35.383 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:35.383 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:35.384 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:35Z" 2023-10-11T15:55:35.384 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:35.385 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:35Z" } 2023-10-11T15:55:35.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:35.386 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:35.386 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:35.387 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:40.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:40.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:40.389 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:40.451 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:40.451 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:40.501 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:40.502 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:40.502 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:40.503 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:40.503 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:40.503 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:40.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.504 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:40.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:40.505 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:40.505 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:40.506 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:40.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:40.507 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:40.507 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:40.508 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:40.508 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:40.508 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:40.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:40.510 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:40.510 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:40.510 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:40.511 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:40.511 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:40.512 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:40.512 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:40.512 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:40.513 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:40.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.513 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:40.514 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:40.514 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:40.515 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:40.515 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:40.515 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:40.516 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:40.516 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:40.517 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:40.517 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:40.517 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.518 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:40.518 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:40.519 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:40.525 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:40Z" 2023-10-11T15:55:40.525 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:40.527 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:40Z" } 2023-10-11T15:55:40.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:40.528 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:40.529 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:40.529 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:45.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:45.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:45.531 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:45.591 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:45.592 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:45.640 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:45.641 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:45.641 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:45.641 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:45.641 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:45.642 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:45.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.642 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:45.642 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:45.642 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:45Z" 2023-10-11T15:55:45.654 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:45.655 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:45Z" } 2023-10-11T15:55:45.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:45.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:45.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:45.656 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:50.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:50.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:50.659 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:50.719 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:50.719 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:50.768 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:50.779 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:50.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:50.779 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:50.779 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:50.780 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:50.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:50.780 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.781 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:50.781 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:50Z" 2023-10-11T15:55:50.785 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:50.787 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:50Z" } 2023-10-11T15:55:50.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:50.788 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:50.788 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:50.789 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:55:55.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:55.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:55:55.792 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:55:55.852 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:55:55.853 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:55:55.901 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:55:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:55:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:55:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:55:55.903 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:55:55.903 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:55:55.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.903 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:55:55.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:55.904 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:55:55.904 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:55.904 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:55.904 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:55:55.905 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:55.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:55.905 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:55.905 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:55:55.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:55.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.906 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:55.906 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:55:55.906 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:55.907 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:55.907 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:55:55.907 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:55.907 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:55:55.907 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:55.908 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:55:55.908 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:55.908 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:55.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:55:55.909 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:55.909 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:55.909 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:55:55.909 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:55.910 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:55:55.910 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:55.910 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:55:55.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:55.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:55:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:55:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:55:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:55:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:55:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:55:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:55:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:55:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:55:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:55:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:55:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:55:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:55:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:55:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:55:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:55:55Z" 2023-10-11T15:55:55.915 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:55:55.916 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:55:55Z" } 2023-10-11T15:55:55.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:55:55.917 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:55.917 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:55:55.918 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:00.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:00.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:00.920 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:00.980 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:00.980 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:01.027 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:01.028 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:01.028 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:01.028 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:01.028 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:01.029 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:01.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.029 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:01.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:01.029 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:01.030 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:01.030 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:01.030 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:01.030 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:01.030 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:01.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:01.032 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:01.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:01.034 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:01.035 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:01.036 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:01Z" 2023-10-11T15:56:01.040 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:01.041 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:01Z" } 2023-10-11T15:56:01.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:01.042 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:01.042 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:01.043 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:06.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:06.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:06.046 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:06.105 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:06.154 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:06.155 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:06.155 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:06.155 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:06.156 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:06.156 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:06.156 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.157 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:06.157 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:06.157 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:06.158 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:06.158 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:06.159 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:06.159 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:06.159 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:06.160 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:06.160 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:06.161 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:06.161 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.161 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:06.161 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:06.162 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:06.162 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:06.162 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:06.162 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:06.163 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:06.163 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:06.163 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:06.163 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:06.163 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.164 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:06.164 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:06.164 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:06.164 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:06.164 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:06.165 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:06.165 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:06.166 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:06.166 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:06.166 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:06.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.167 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:06.167 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:06.167 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:06.167 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:06.168 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:06.168 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:06.168 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:06.168 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:06.168 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:06.169 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:06.169 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:06.169 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:06.169 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.169 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:06Z" 2023-10-11T15:56:06.170 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:06.171 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:06Z" } 2023-10-11T15:56:06.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:06.173 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:06.173 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:06.173 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:11.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:11.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:11.176 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:11.236 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:11.284 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:11.285 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:11.285 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:11.285 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:11.285 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:11.286 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:11.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.286 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:11.286 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:11.287 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:11.287 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:11.287 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:11.287 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:11.287 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:11.288 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:11.288 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:11.288 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:11.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:11.288 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.289 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:11.289 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:11.289 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:11.289 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:11.289 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:11.290 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:11.290 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:11.290 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:11.290 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:11.291 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:11.292 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:11.293 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:11.294 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:11Z" 2023-10-11T15:56:11.300 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:11.301 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:11Z" } 2023-10-11T15:56:11.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:11.302 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:11.303 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:11.303 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:16.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:16.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:16.306 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:16.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:16.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:16.419 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:16.419 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:16.420 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:16.420 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:16.420 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:16.420 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:16.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.421 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:16.421 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:16.421 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:16.421 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:16.422 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:16.422 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:16.422 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:16.422 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:16.422 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:16.423 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:16.423 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:16.423 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.423 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:16.423 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:16.424 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:16.424 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:16.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:16.424 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:16Z" 2023-10-11T15:56:16.432 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:16.433 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:16Z" } 2023-10-11T15:56:16.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:16.434 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:16.434 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:16.435 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:21.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:21.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:21.437 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:21.498 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:21.498 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:21.546 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:21.547 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:21.547 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:21.547 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:21.547 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:21.548 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:21.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.548 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:21.548 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:21.548 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:21.549 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:21.549 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:21.549 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:21.549 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:21.550 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:21.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:21.550 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:21.550 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:21.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.551 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:21.551 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:21.551 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:21.551 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:21.551 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:21.552 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:21.552 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:21.552 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:21.552 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:21.552 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:21.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.553 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:21.553 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:21.553 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:21.553 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:21Z" 2023-10-11T15:56:21.559 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:21.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:21Z" } 2023-10-11T15:56:21.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:21.562 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:21.562 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:21.563 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:26.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:26.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:26.565 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:26.626 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:26.627 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:26.677 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:26.677 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:26.677 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:26.678 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:26.678 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:26.678 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:26.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.679 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:26.679 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:26.679 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:26.679 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:26.679 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:26.680 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:26.680 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:26.680 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:26.680 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:26.680 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:26Z" 2023-10-11T15:56:26.691 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:26.692 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:26Z" } 2023-10-11T15:56:26.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:26.693 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:26.694 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:26.695 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:31.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:31.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:31.697 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:31.757 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:31.806 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:31.807 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:31.807 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:31.807 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:31.807 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:31.808 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:31.808 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.808 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:31.808 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:31.809 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:31.809 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:31.809 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:31.809 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:31.809 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:31.810 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:31.810 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:31.810 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:31.810 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:31Z" 2023-10-11T15:56:31.819 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:31.821 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:31Z" } 2023-10-11T15:56:31.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:31.822 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:31.822 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:31.822 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:36.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:36.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:36.825 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:36.883 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:36.884 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:36.931 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:36.931 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:36.932 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:36.932 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:36.932 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:36.932 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:36.933 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.933 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:36.933 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:36.933 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:36.934 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:36.934 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:36.934 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:36.934 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:36.934 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:36.935 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:36.935 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:36.935 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:36.935 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.935 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:36.936 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:36.936 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:36.936 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:36.936 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:36.936 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:36.938 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:36Z" 2023-10-11T15:56:36.944 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:36.945 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:36Z" } 2023-10-11T15:56:36.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:36.947 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:36.947 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:36.947 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:41.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:41.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:41.950 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:42.009 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:42.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:42.056 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:42.056 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:42.057 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:42.057 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:42.057 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:42.057 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:42.058 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.058 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:42.058 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:42.058 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:42.059 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:42.059 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:42.059 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:42.059 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:42.059 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:42.060 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:42.060 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:42.060 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:42.060 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.060 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:42.061 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:42.061 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:42.061 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:42.061 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:42.062 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:42Z" 2023-10-11T15:56:42.069 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:42.071 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:42Z" } 2023-10-11T15:56:42.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:42.072 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:42.072 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:42.072 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:47.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:47.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:47.075 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:47.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:47.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:47.182 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:47.183 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:47.183 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:47.183 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:47.183 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:47.184 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:47.184 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.184 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:47.184 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:47.184 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:47.185 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:47.185 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:47.185 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:47.185 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:47.186 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:47.186 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:47.186 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:47.186 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:47.186 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.187 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:47.187 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:47.187 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:47.187 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:47.187 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:47.188 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:47.188 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:47.188 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:47.188 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:47.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:47.189 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.189 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:47.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:47.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:47.189 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:47.195 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:47Z" 2023-10-11T15:56:47.195 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:47.196 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:47Z" } 2023-10-11T15:56:47.196 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:47.197 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:47.197 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:47.198 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:52.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:52.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:52.201 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:52.262 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:52.314 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:52.314 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:52.314 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:52.314 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:52.315 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:52.315 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:52.315 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.315 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:52.315 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:52.316 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:52.316 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:52.316 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:52.316 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:52.316 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:52.317 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:52.317 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:52.317 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:52.317 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:52.317 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.318 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:52.318 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:52.318 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:52.318 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:52.319 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:52.319 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:52.319 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:52.319 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:52.319 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:52.320 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:52.320 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.320 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:52.320 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:52.320 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:52.321 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:52.321 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:52.321 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:52.321 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:52.321 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:52.322 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:52.323 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:52.324 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:52.325 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:52.326 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:52Z" 2023-10-11T15:56:52.326 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:52.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:52Z" } 2023-10-11T15:56:52.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:52.328 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:52.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:52.329 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:56:57.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:57.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:56:57.331 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:56:57.391 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:56:57.391 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:56:57.440 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:56:57.440 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:56:57.440 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:56:57.440 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:56:57.441 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:56:57.441 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:56:57.441 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.441 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:56:57.442 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:57.442 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:56:57.442 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:57.442 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:57.442 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:56:57.443 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:57.443 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:57.443 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:57.443 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:56:57.443 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:57.444 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.444 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:57.444 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:56:57.444 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:57.444 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:57.445 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:56:57.445 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:57.445 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:56:57.445 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:57.445 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:56:57.446 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:57.446 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.446 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:57.447 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:56:57.447 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:57.447 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:57.447 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:56:57.447 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:57.448 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:56:57.448 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:57.448 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:56:57.448 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:57.449 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.449 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:56:57.449 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:56:57.449 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:56:57.450 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:56:57.450 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:56:57.450 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:56:57.450 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:56:57.450 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:56:57.451 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:56:57.451 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:57.451 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:56:57.451 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:56:57.451 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.452 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:56:57.452 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:56:57.452 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:56:57.452 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:56:57.452 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:56:57Z" 2023-10-11T15:56:57.453 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:56:57.454 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:56:57Z" } 2023-10-11T15:56:57.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:56:57.455 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:57.455 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:56:57.456 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:57:02.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:02.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:02.458 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:02.519 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:02.520 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:02.569 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:02.570 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:02.570 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:02.570 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:02.570 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:02.571 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:57:02.571 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.571 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:02.571 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:02.571 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:02.572 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:02.572 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:02.572 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:57:02.572 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:02.573 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:02.573 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:02.573 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:57:02.573 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:02.573 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.574 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:02.574 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:02.574 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:02.574 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:02.575 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:57:02.575 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:02.575 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:02.575 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:02.575 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:57:02.576 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:02.576 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.576 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:02.576 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:02.577 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:02.577 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:02.577 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:57:02.577 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:02.578 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:57:02.578 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:02.578 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:57:02.578 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:02.579 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.579 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:02.579 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:02.579 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:02.580 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:02.580 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:57:02.580 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:02.580 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:57:02.580 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:57:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:02.581 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.582 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:02.582 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:57:02.582 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:57:02.582 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:02.583 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:02Z" 2023-10-11T15:57:02.583 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:02.584 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:57:02Z" } 2023-10-11T15:57:02.584 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:57:02.585 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:02.585 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:02.586 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:57:07.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:07.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:07.588 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:07.649 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:07.649 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:07.699 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:07.699 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:07.700 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:57:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:07.701 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:57:07.702 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:07.703 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:07.704 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:57:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:07.705 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:57:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:07.706 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:57:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:07.707 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:57:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:57:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.708 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:57:07.709 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:57:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:57:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:07.710 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:07.711 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:57:07.712 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:57:07.712 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:07.712 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:07Z" 2023-10-11T15:57:07.712 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:07.713 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:57:07Z" } 2023-10-11T15:57:07.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:57:07.714 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:07.715 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:07.715 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:57:12.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:12.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:12.718 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:12.778 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:12.778 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:12.827 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:12.827 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:12.827 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:12.827 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:12.827 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:12.828 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:57:12.828 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.828 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:12.828 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:12.828 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:12.829 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:12.829 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:12.829 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T15:57:12.829 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:12.829 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:12.830 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:12.830 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T15:57:12.830 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:12.830 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.831 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:12.831 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:12.831 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:12.831 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:12.831 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:57:12.832 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:12.832 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:12.832 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:12.832 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:57:12.832 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:12.833 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.833 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:12.833 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:12.833 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:12.833 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:12.834 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T15:57:12.834 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:12.834 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:57:12.834 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:12.834 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T15:57:12.835 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:12.835 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.835 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:12.835 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:12.835 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:12.836 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:12.836 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:57:12.836 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:12.836 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:57:12.836 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:12.837 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:57:12.837 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:12.837 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:12.837 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:12.837 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.838 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:12.838 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:57:12.838 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:57:12.838 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:12.838 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:12Z" 2023-10-11T15:57:12.839 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:12.839 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:53:09.033937Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051250", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:53:09.095029Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:57:12Z" } 2023-10-11T15:57:12.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:57:12.840 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:12.841 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:12.841 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517, shard=2 master=00000000000000000000:00000000000000051390 target=00000000000000000000:00000000000000051250 2023-10-11T15:57:17.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:17.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:17.844 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:17.903 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:17.904 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:17.953 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:17.953 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:17.954 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:17.954 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:17.954 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:17.954 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:57:17.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.955 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:17.955 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:17.955 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:17.955 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:17.956 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:17.956 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:57:17.956 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:17.956 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:17.956 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:17.957 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:57:17.957 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:17.957 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.957 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:17.957 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:17.958 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:17.958 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:17.958 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:57:17.958 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:17.959 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:17.959 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:17.959 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:57:17.959 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:17.959 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.960 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:17.960 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:17.960 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:17.960 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:17.960 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:57:17.961 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:17.961 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:57:17.961 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:17.961 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:57:17.962 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:17.962 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.962 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:17.962 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:17.962 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:17.963 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:57:17.963 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:57:17.963 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:17.963 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:57:17.963 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:17.964 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:57:17.964 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:17.964 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:17.964 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:17.964 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.965 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:17.965 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:57:17.965 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:57:17.965 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:17.965 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:17Z" 2023-10-11T15:57:17.966 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:17.967 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:14.168373Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051390", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:55:14.141173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:57:17Z" } 2023-10-11T15:57:17.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.168373Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051390', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.141173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:57:17.968 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:17.968 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:57:17.969 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:57:17.969 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:57:17.969 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:57:17.970 DEBUG:boto:Method: HEAD 2023-10-11T15:57:17.970 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:57:17.971 DEBUG:boto:Data: 2023-10-11T15:57:17.971 DEBUG:boto:Headers: {} 2023-10-11T15:57:17.972 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:17.972 DEBUG:boto:Port: 8001 2023-10-11T15:57:17.972 DEBUG:boto:Params: {} 2023-10-11T15:57:17.973 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:57:17.974 DEBUG:boto:Token: None 2023-10-11T15:57:17.974 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:17 GMT /dghqvd-70/ 2023-10-11T15:57:17.975 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Hsy670NsgQaSDQ/qnKduwEcr3rM= 2023-10-11T15:57:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Hsy670NsgQaSDQ/qnKduwEcr3rM=', 'Content-Length': '0'} 2023-10-11T15:57:17.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afc2b2436f82d37c-006526c5dd-257a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:57:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:17.982 DEBUG:boto:path=/dghqvd-70/obj1 2023-10-11T15:57:17.983 DEBUG:boto:auth_path=/dghqvd-70/obj1 2023-10-11T15:57:17.983 DEBUG:boto:Method: HEAD 2023-10-11T15:57:17.984 DEBUG:boto:Path: /dghqvd-70/obj1 2023-10-11T15:57:17.984 DEBUG:boto:Data: 2023-10-11T15:57:17.985 DEBUG:boto:Headers: {} 2023-10-11T15:57:17.985 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:17.985 DEBUG:boto:Port: 8001 2023-10-11T15:57:17.986 DEBUG:boto:Params: {} 2023-10-11T15:57:17.986 DEBUG:boto:Token: None 2023-10-11T15:57:17.987 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:17 GMT /dghqvd-70/obj1 2023-10-11T15:57:17.987 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ciqYwgpB0quybfEe6nQN8e+TdHY= 2023-10-11T15:57:17.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ciqYwgpB0quybfEe6nQN8e+TdHY=', 'Content-Length': '0'} 2023-10-11T15:57:17.989 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005d7e63e7b3a9ff8f-006526c5dd-257a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:57:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:17.990 DEBUG:boto:path=/dghqvd-70/obj2 2023-10-11T15:57:17.990 DEBUG:boto:auth_path=/dghqvd-70/obj2 2023-10-11T15:57:17.991 DEBUG:boto:Method: HEAD 2023-10-11T15:57:17.991 DEBUG:boto:Path: /dghqvd-70/obj2 2023-10-11T15:57:17.992 DEBUG:boto:Data: 2023-10-11T15:57:17.992 DEBUG:boto:Headers: {} 2023-10-11T15:57:17.992 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:17.993 DEBUG:boto:Port: 8001 2023-10-11T15:57:17.993 DEBUG:boto:Params: {} 2023-10-11T15:57:17.994 DEBUG:boto:Token: None 2023-10-11T15:57:17.994 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:17 GMT /dghqvd-70/obj2 2023-10-11T15:57:17.995 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:oijZPc+Vc15rzS8dG78eXUgayx4= 2023-10-11T15:57:17.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:17 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oijZPc+Vc15rzS8dG78eXUgayx4=', 'Content-Length': '0'} 2023-10-11T15:57:17.997 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000034c34d63b10f4453-006526c5dd-257a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:57:17 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:17.997 DEBUG:boto:path=/dghqvd-71/ 2023-10-11T15:57:17.998 DEBUG:boto:auth_path=/dghqvd-71/ 2023-10-11T15:57:17.998 DEBUG:boto:Method: HEAD 2023-10-11T15:57:17.998 DEBUG:boto:Path: /dghqvd-71/ 2023-10-11T15:57:17.999 DEBUG:boto:Data: 2023-10-11T15:57:17.999 DEBUG:boto:Headers: {} 2023-10-11T15:57:18.000 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:18.000 DEBUG:boto:Port: 8001 2023-10-11T15:57:18.000 DEBUG:boto:Params: {} 2023-10-11T15:57:18.001 DEBUG:boto:Token: None 2023-10-11T15:57:18.001 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:18 GMT /dghqvd-71/ 2023-10-11T15:57:18.002 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NpoPn/Jj6ncTrlt/rSQQR/jCVG8= 2023-10-11T15:57:18.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NpoPn/Jj6ncTrlt/rSQQR/jCVG8=', 'Content-Length': '0'} 2023-10-11T15:57:18.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000858e1af0d2df2f8d-006526c5de-257a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:57:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:18.005 DEBUG:boto:path=/dghqvd-71/obj1 2023-10-11T15:57:18.005 DEBUG:boto:auth_path=/dghqvd-71/obj1 2023-10-11T15:57:18.006 DEBUG:boto:Method: HEAD 2023-10-11T15:57:18.006 DEBUG:boto:Path: /dghqvd-71/obj1 2023-10-11T15:57:18.007 DEBUG:boto:Data: 2023-10-11T15:57:18.007 DEBUG:boto:Headers: {} 2023-10-11T15:57:18.007 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:18.008 DEBUG:boto:Port: 8001 2023-10-11T15:57:18.008 DEBUG:boto:Params: {} 2023-10-11T15:57:18.009 DEBUG:boto:Token: None 2023-10-11T15:57:18.009 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:18 GMT /dghqvd-71/obj1 2023-10-11T15:57:18.010 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jTY7JSW9uNYc2FuAYzuzTio5KlQ= 2023-10-11T15:57:18.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jTY7JSW9uNYc2FuAYzuzTio5KlQ=', 'Content-Length': '0'} 2023-10-11T15:57:18.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abeac8e0d6abf192-006526c5de-257a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:57:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:18.013 DEBUG:boto:path=/dghqvd-71/obj1 2023-10-11T15:57:18.013 DEBUG:boto:auth_path=/dghqvd-71/obj1 2023-10-11T15:57:18.014 DEBUG:boto:Method: GET 2023-10-11T15:57:18.014 DEBUG:boto:Path: /dghqvd-71/obj1 2023-10-11T15:57:18.015 DEBUG:boto:Data: 2023-10-11T15:57:18.015 DEBUG:boto:Headers: {} 2023-10-11T15:57:18.016 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:18.016 DEBUG:boto:Port: 8001 2023-10-11T15:57:18.016 DEBUG:boto:Params: {} 2023-10-11T15:57:18.017 DEBUG:boto:Token: None 2023-10-11T15:57:18.017 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:57:18 GMT /dghqvd-71/obj1 2023-10-11T15:57:18.018 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:I0pQNbUibVolQw6bnMIWkg9Ay1Q= 2023-10-11T15:57:18.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I0pQNbUibVolQw6bnMIWkg9Ay1Q=', 'Content-Length': '0'} 2023-10-11T15:57:18.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b093edf2bd9ff7e-006526c5de-257a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:57:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:18.021 DEBUG:boto:path=/dghqvd-71/obj2 2023-10-11T15:57:18.021 DEBUG:boto:auth_path=/dghqvd-71/obj2 2023-10-11T15:57:18.022 DEBUG:boto:Method: HEAD 2023-10-11T15:57:18.022 DEBUG:boto:Path: /dghqvd-71/obj2 2023-10-11T15:57:18.023 DEBUG:boto:Data: 2023-10-11T15:57:18.023 DEBUG:boto:Headers: {} 2023-10-11T15:57:18.023 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:18.024 DEBUG:boto:Port: 8001 2023-10-11T15:57:18.024 DEBUG:boto:Params: {} 2023-10-11T15:57:18.025 DEBUG:boto:Token: None 2023-10-11T15:57:18.025 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:57:18 GMT /dghqvd-71/obj2 2023-10-11T15:57:18.026 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:T+JjybSWMC1VLIAW9b2do4yuJsY= 2023-10-11T15:57:18.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T+JjybSWMC1VLIAW9b2do4yuJsY=', 'Content-Length': '0'} 2023-10-11T15:57:18.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038ed8549ffde2314-006526c5de-257a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:57:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:18.029 DEBUG:boto:path=/dghqvd-71/obj2 2023-10-11T15:57:18.029 DEBUG:boto:auth_path=/dghqvd-71/obj2 2023-10-11T15:57:18.029 DEBUG:boto:Method: GET 2023-10-11T15:57:18.030 DEBUG:boto:Path: /dghqvd-71/obj2 2023-10-11T15:57:18.030 DEBUG:boto:Data: 2023-10-11T15:57:18.031 DEBUG:boto:Headers: {} 2023-10-11T15:57:18.031 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:57:18.032 DEBUG:boto:Port: 8001 2023-10-11T15:57:18.032 DEBUG:boto:Params: {} 2023-10-11T15:57:18.033 DEBUG:boto:Token: None 2023-10-11T15:57:18.033 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:57:18 GMT /dghqvd-71/obj2 2023-10-11T15:57:18.034 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KcAJvXs92nUQPlsEoRmC2zjGVlI= 2023-10-11T15:57:18.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:57:18 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KcAJvXs92nUQPlsEoRmC2zjGVlI=', 'Content-Length': '0'} 2023-10-11T15:57:18.036 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015e7f1aa85b672e7-006526c5de-257a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:57:18 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:57:18.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.038 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.139 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T15:57:18.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.140 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.192 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.193 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.238 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T15:57:18.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.240 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.289 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.290 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.336 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:57:18.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.337 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.385 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.386 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.437 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:18.437 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:18.438 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 18, 2023-10-11T15:57:18.438 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T15:57:18.438 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T15:57:18.438 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T15:57:18.439 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:18.439 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T15:57:18.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.439 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T15:57:18.439 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T15:57:18.440 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:57:18.440 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T15:57:18.440 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.440 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:57:18.441 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.441 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:57:18.441 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:57:18.441 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:57:18.441 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:57:18.442 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.442 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:57:18.442 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T15:57:18.442 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.443 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T15:57:18.443 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.443 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:57:18.443 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:57:18.443 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:57:18.444 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:57:18.444 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:57:18.444 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:57:18.444 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:57:18.445 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:57:18.445 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:57:18.445 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:57:18.445 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.446 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.446 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.446 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.446 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:57:18.447 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T15:57:18.447 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.447 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T15:57:18.447 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.448 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:57:18.448 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:57:18.448 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:57:18.448 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:57:18.448 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:57:18.449 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:57:18.449 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:57:18.449 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:57:18.449 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:57:18.449 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:57:18.450 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.450 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.450 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.450 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.451 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:57:18.451 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.451 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:57:18.451 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:57:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:57:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:57:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.452 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.453 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:57:18.453 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:57:18.453 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:57:18.453 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T15:57:18.453 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.454 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T15:57:18.454 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T15:57:18.454 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T15:57:18.454 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T15:57:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:57:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:57:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:57:18.455 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.456 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T15:57:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T15:57:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:57:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:57:18.457 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:57:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:57:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:57:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.458 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T15:57:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T15:57:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:57:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:57:18.459 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T15:57:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T15:57:18.460 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T15:57:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T15:57:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T15:57:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.461 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T15:57:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T15:57:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T15:57:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T15:57:18.462 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "allowed" 2023-10-11T15:57:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.463 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:57:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.464 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T15:57:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T15:57:18.465 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T15:57:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T15:57:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:57:18.466 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T15:57:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T15:57:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:57:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T15:57:18.467 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:57:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T15:57:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T15:57:18.468 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:57:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:57:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:57:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:57:18.469 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:57:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:57:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:57:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:57:18.470 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:57:18.471 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:57:18.471 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.471 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.471 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.471 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.472 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:57:18.472 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T15:57:18.472 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T15:57:18.472 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T15:57:18.472 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T15:57:18.473 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T15:57:18.474 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T15:57:18.474 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T15:57:18.474 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T15:57:18.474 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T15:57:18.475 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.475 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.475 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.475 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.476 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T15:57:18.476 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.476 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T15:57:18.476 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T15:57:18.476 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T15:57:18.477 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T15:57:18.477 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.477 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.477 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.477 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T15:57:18.478 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:57:18.478 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T15:57:18.478 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T15:57:18.478 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T15:57:18.479 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T15:57:18.480 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.480 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T15:57:18.480 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T15:57:18.480 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.481 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T15:57:18.481 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T15:57:18.482 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.483 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.483 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T15:57:18.483 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T15:57:18.483 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.484 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:18.484 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:57:18.484 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T15:57:18.484 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.484 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:18.485 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.485 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T15:57:18.485 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T15:57:18.485 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T15:57:18.485 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T15:57:18.486 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:57:18.486 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:57:18.486 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:57:18.486 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:57:18.487 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:57:18.487 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.487 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T15:57:18.487 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T15:57:18.487 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T15:57:18.488 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T15:57:18.488 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T15:57:18.488 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T15:57:18.488 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.488 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T15:57:18.489 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:57:18.489 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:57:18.489 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:57:18.489 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:57:18.489 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:57:18.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.490 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T15:57:18.490 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:57:18.490 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:57:18.491 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:57:18.491 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:57:18.491 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:57:18.491 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.492 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T15:57:18.492 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T15:57:18.492 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T15:57:18.492 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T15:57:18.492 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T15:57:18.493 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T15:57:18.493 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:18.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:18.493 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T15:57:18.494 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T15:57:18.494 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:18.494 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:18.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.496 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.554 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.554 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.662 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T15:57:18.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.664 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.735 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.736 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.828 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T15:57:18.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.830 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:18.914 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:18.915 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:18.997 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T15:57:18.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-70', '--source-bucket=*', '--dest-bucket=dghqvd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-70', '--source-bucket=*', '--dest-bucket=dghqvd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:18.999 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dghqvd-70 '--source-bucket=*' --dest-bucket=dghqvd-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:19.132 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T15:57:19.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:19.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:19.134 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:19.186 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:19.236 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T15:57:19.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:19.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:57:19.238 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:57:19.285 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:19.286 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:19.331 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T15:57:19.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:19.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:19.332 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:19.380 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:19.381 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:19.411 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:57:19.424 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:57:19.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:19.425 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:57:19.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.425 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.425 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:19.426 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:57:19.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.426 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:19.426 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:57:19.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.427 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:57:19.427 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:57:19.220443Z" 2023-10-11T15:57:19.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.428 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:57:19.429 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:19.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:57:19.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:19.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:19.431 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:19.480 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:19.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:19.507+0000 7fba36500780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:19.510 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:19.508+0000 7fba36500780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:19.523 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:19.523 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:19.523 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:19.524 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:19.524 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:19.524 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:19.524 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:19.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.525 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:19.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:19.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:19.526 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:19.526 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:19.526 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:19.526 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:19.527 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:19.527 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:19.527 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:19.527 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.528 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.528 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:19.528 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:19.528 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:19.528 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:19.529 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:19.529 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:19.529 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:19.529 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:19.530 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:19.530 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.530 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.531 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:19.534 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:19.534 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:19.535 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:19.535 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:19.535 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:19.535 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:19.536 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:19.536 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:19.536 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.536 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:19.537 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:19.537 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:19.537 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:19.537 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:19.538 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:19.538 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:19.538 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:19.538 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:19.538 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:19.539 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.539 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:19.539 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:19.539 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.540 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:19.540 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:19.540 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:19.540 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:19.540 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:19Z" 2023-10-11T15:57:19.541 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:19.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:19Z" } 2023-10-11T15:57:19.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:19.543 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:19.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:19.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:24.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:24.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:24.546 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:24.604 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:24.604 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:24.637 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:24.634+0000 7f864e96e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:24.638 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:24.635+0000 7f864e96e780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:24.652 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:24.652 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:24.653 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:24.654 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:24.655 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:24.656 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:24.657 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:24.658 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:24.659 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:24.660 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:24.661 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:24.662 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:24.663 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:24.664 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:24.665 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:24.665 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:24.665 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:24Z" 2023-10-11T15:57:24.665 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:24.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:24Z" } 2023-10-11T15:57:24.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:24.667 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:24.668 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:24.668 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:29.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:29.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:29.671 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:29.728 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:29.729 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:29.761 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:29.758+0000 7fdbb109f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:29.762 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:29.759+0000 7fdbb109f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:29.776 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:29.776 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.777 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:29.778 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:29.779 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:29.780 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:29.781 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.782 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:29.783 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:29.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:29.785 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:29.786 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:29.787 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:29.788 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.788 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:29.788 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:29.788 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:29.789 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:29.789 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:29Z" 2023-10-11T15:57:29.789 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:29.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:29Z" } 2023-10-11T15:57:29.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:29.791 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:29.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:29.792 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:34.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:34.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:34.795 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:34.852 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:34.853 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:34.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:34.883+0000 7f88d7311780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:34.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:34.884+0000 7f88d7311780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:34.903 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:34.903 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:34.904 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:34.905 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.906 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:34.907 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:34.908 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:34.909 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:34.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:34.911 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:34.912 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:34.912 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:34.912 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:34.913 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:34.913 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:34.913 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:34.913 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:34.914 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:34.914 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:34.914 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.914 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:34.915 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:34.915 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:34.915 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:34.915 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:34.915 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:34.916 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:34.916 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:34.916 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:34.916 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:34.917 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:34.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.917 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:34.917 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:34.918 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:34.918 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:34.918 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:34.918 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:34.919 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:34.919 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:34.919 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:34.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:34.919 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:34.920 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:34.920 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:34.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.920 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:34.921 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:34.921 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:34.921 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:34.921 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:34Z" 2023-10-11T15:57:34.922 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:34.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:34Z" } 2023-10-11T15:57:34.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:34.923 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:34.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:34.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:39.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:39.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:39.927 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:39.986 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:39.987 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:40.022 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:40.020+0000 7f1a4f436780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:40.023 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:40.021+0000 7f1a4f436780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:40.037 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:40.037 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:40.037 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:40.038 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:40.038 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:40.038 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:40.038 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:40.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.039 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:40.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:40.039 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:40.040 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:40.040 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:40.040 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:40.040 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:40.040 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:40.041 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:40.041 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:40.041 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:40.041 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:40.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.042 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:40.042 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:40.042 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:40.043 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:40.043 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:40.043 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:40.043 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:40.043 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:40.044 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:40.044 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:40.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:40.044 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.045 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:40.045 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:40.045 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:40.045 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:40.045 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:40.046 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:40.046 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:40.046 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:40.046 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:40.046 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:40.047 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:40.047 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.047 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:40.047 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:40.048 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:40.048 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:40.048 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:40.048 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:40.048 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:40.049 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:40.049 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:40.049 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:40.049 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:40.049 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:40.050 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:40.050 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.050 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:40.050 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:40.050 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:40.051 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:40.051 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:40Z" 2023-10-11T15:57:40.051 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:40.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:40Z" } 2023-10-11T15:57:40.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:40.053 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:40.054 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:40.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:45.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:45.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:45.057 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:45.117 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:45.118 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:45.150 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:45.148+0000 7fbefee04780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:45.151 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:45.150+0000 7fbefee04780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:45.164 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:45.165 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:45.165 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:45.165 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:45.165 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:45.166 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:45.166 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:45.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.166 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:45.166 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:45.167 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:45.167 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:45.167 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:45.167 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:45.167 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:45.168 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:45.168 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:45.168 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:45.168 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:45.169 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:45.169 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.169 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:45.169 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:45.169 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:45.170 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:45.170 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:45.170 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:45.170 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:45.171 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:45.171 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:45.171 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:45.171 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:45.171 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.172 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:45.172 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:45.172 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:45.172 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:45.172 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:45.173 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:45.173 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:45.173 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:45.173 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:45.173 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:45.174 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:45.174 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.174 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:45.174 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:45.175 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:45.175 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:45.175 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:45.175 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:45.175 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:45.176 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:45.176 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:45.176 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:45.176 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:45.177 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:45.178 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:45.178 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:45.178 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:45Z" 2023-10-11T15:57:45.178 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:45.179 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:45Z" } 2023-10-11T15:57:45.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:45.180 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:45.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:45.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:50.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:50.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:50.184 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:50.242 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:50.242 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:50.277 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:50.276+0000 7f7f4bacd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:50.278 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:50.277+0000 7f7f4bacd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:50.292 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:50.292 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:50.293 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:50.293 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:50.293 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:50.293 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:50.294 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:50.294 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.294 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:50.294 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:50.295 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:50.295 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:50.295 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:50.295 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:50.295 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:50.296 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:50.296 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:50.296 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:50.296 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:50.297 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:50.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.297 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:50.297 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:50.297 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:50.298 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:50.298 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:50.298 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:50.298 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:50.298 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:50.299 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:50.299 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:50.299 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:50.299 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.300 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:50.300 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:50.300 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:50.300 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:50.301 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:50.301 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:50.301 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:50.301 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:50.302 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:50.302 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:50.302 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:50.302 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.302 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:50.303 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:50.303 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:50.303 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:50.303 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:50.303 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:50.304 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:50.304 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:50.304 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:50.305 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:50.305 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:50.305 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:50.305 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:50.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:50.307 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:50Z" 2023-10-11T15:57:50.307 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:50.308 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:50Z" } 2023-10-11T15:57:50.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:50.309 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:50.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:50.310 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:57:55.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:55.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:57:55.313 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:57:55.370 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:57:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:57:55.404 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:55.402+0000 7fb65352b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:55.405 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:57:55.403+0000 7fb65352b780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:57:55.419 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:57:55.419 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:57:55.419 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:57:55.420 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:57:55.420 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:57:55.420 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:57:55.420 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:55.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.421 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:57:55.421 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:55.421 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:57:55.422 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:55.422 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:55.422 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:57:55.422 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:55.423 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:57:55.423 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:55.423 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:57:55.423 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:55.424 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:55.424 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:55.424 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:57:55.425 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:55.425 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:55.425 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:57:55.425 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:55.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:55.426 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:55.426 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:57:55.426 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:55.426 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:55.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.427 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:55.427 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:57:55.427 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:55.428 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:55.428 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:57:55.428 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:55.428 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:55.428 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:55.429 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:57:55.429 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:55.429 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:55.429 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.430 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:57:55.430 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:57:55.430 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:57:55.430 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:57:55.430 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:57:55.431 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:57:55.431 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:57:55.431 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:57:55.431 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:57:55.432 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:57:55.432 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:55.432 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:57:55.432 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:57:55.432 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.433 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:57:55.433 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:57:55.433 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:57:55.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:57:55.434 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:57:55Z" 2023-10-11T15:57:55.434 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:57:55.435 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:57:55Z" } 2023-10-11T15:57:55.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:57:55.436 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:57:55.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:57:55.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:00.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:00.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:00.440 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:00.498 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:00.531 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:00.529+0000 7f401a5cd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:00.532 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:00.530+0000 7f401a5cd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:00.545 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:00.546 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:00.546 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:00.546 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:00.547 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:00.547 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:00.547 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:00.547 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.547 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:00.548 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:00.548 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:00.548 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:00.548 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:00.548 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:00.549 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:00.549 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:00.549 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:00.549 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:00.550 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:00.550 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:00.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.550 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:00.550 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:00.551 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:00.551 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:00.551 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:00.551 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:00.551 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:00.552 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:00.552 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:00.552 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:00.552 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:00.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.553 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:00.553 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:00.553 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:00.553 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:00.554 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:00.554 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:00.554 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:00.554 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:00.554 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:00.555 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:00.555 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:00.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.555 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:00.556 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:00.556 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:00.556 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:00.556 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:00.556 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:00.557 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:00.557 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:00.557 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:00.557 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:00.557 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:00.558 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:00.559 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:00.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:00.559 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:00Z" 2023-10-11T15:58:00.559 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:00.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:00Z" } 2023-10-11T15:58:00.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:00.561 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:00.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:00.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:05.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:05.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:05.566 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:05.623 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:05.624 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:05.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:05.653+0000 7fe20409a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:05.656 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:05.655+0000 7fe20409a780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:05.670 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:05.671 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:05.671 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:05.671 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:05.672 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:05.672 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:05.672 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:05.673 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.673 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:05.673 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:05.674 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:05.674 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:05.675 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:05.675 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:05.675 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:05.676 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:05.676 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:05.676 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:05.677 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:05.677 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:05.677 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:05.678 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:05.678 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:05.679 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:05.679 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:05.680 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:05.680 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:05.680 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:05.680 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:05.681 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:05.681 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:05.681 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.681 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:05.681 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:05.682 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:05.682 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:05.682 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:05.682 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:05.682 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:05.683 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:05.683 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:05.683 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:05.683 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:05.684 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:05.684 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:05.684 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:05.684 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:05.685 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:05.685 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:05.685 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:05.685 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:05.685 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:05.686 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:05.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:05.688 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:05Z" 2023-10-11T15:58:05.688 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:05.689 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:05Z" } 2023-10-11T15:58:05.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:05.691 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:05.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:05.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:10.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:10.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:10.694 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:10.754 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:10.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:10.787 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:10.785+0000 7fd47a7aa780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:10.788 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:10.786+0000 7fd47a7aa780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:10.802 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:10.802 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:10.802 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:10.802 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:10.803 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:10.803 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:10.803 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:10.803 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.804 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:10.804 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:10.804 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:10.804 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:10.804 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:10.805 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:10.805 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:10.805 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:10.805 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:10.805 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:10.806 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:10.806 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:10.806 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.807 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:10.807 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:10.807 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:10.807 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:10.807 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:10.808 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:10.808 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:10.808 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:10.808 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:10.809 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:10.809 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:10.809 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.810 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:10.810 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:10.810 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:10.811 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:10.811 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:10.811 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:10.811 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:10.811 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:10.812 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:10.812 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:10.812 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:10.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:10.813 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:10.813 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:10.813 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:10.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:10.814 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:10.814 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:10.815 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:10.815 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:10.815 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:10.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:10.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:10.816 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:10.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.816 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:10.816 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:10.816 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:10.817 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:10.817 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:10Z" 2023-10-11T15:58:10.817 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:10.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:10Z" } 2023-10-11T15:58:10.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:10.819 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:10.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:10.820 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:15.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:15.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:15.823 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:15.882 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:15.882 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:15.914 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:15.913+0000 7f3a31ee9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:15.915 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:15.914+0000 7f3a31ee9780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:15.929 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:15.930 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:15.930 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:15.930 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:15.930 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:15.931 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:15.931 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:15.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.931 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:15.931 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:15.932 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:15.932 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:15.932 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:15.932 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:15.933 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:15.933 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:15.933 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:15.933 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:15.933 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:15.934 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:15.934 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.934 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:15.934 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:15.934 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:15.935 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:15.935 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:15.935 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:15.935 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:15.935 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:15.936 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:15.936 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:15.936 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:15.936 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.936 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:15.937 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:15.937 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:15.937 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:15.937 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:15.938 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:15.938 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:15.938 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:15.938 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:15.938 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:15.939 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:15.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.939 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:15.939 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:15.939 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:15.940 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:15.940 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:15.940 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:15.940 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:15.940 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:15.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:15.941 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:15.941 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:15.941 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:15.941 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:15.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.942 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:15.942 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:15.942 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:15.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:15.943 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:15Z" 2023-10-11T15:58:15.943 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:15.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:15Z" } 2023-10-11T15:58:15.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:15.945 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:15.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:15.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:20.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:20.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:20.949 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:21.006 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:21.006 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:21.039 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:21.038+0000 7fb9d98ca780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:21.040 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:21.039+0000 7fb9d98ca780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:21.054 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:21.055 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:21.055 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:21.055 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:21.056 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:21.056 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:21.056 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:21.056 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.057 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:21.057 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:21.057 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:21.057 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:21.058 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:21.058 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:21.058 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:21.058 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:21.059 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:21.059 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:21.059 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:21.059 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:21.060 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.060 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:21.060 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:21.060 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:21.060 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:21.061 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:21.061 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:21.061 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:21.061 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:21.061 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:21.062 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:21.062 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:21.062 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.062 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:21.062 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:21.063 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:21.063 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:21.063 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:21.063 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:21.064 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:21.064 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:21.064 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:21.064 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:21.064 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:21.065 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:21.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:21.065 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:21.066 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:21.066 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:21.066 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:21.066 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:21.066 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:21.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:21.068 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:21.069 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:21.069 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:21Z" 2023-10-11T15:58:21.069 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:21.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:21Z" } 2023-10-11T15:58:21.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:21.071 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:21.072 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:21.072 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:26.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:26.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:26.074 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:26.131 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:26.131 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:26.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:26.162+0000 7efd571bd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:26.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:26.163+0000 7efd571bd780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:26.178 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:26.178 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:26.178 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:26.179 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:26.179 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:26.179 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:26.179 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:26.180 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.180 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:26.180 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:26.180 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:26.180 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:26.181 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:26.181 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:26.181 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:26.181 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:26.182 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:26.182 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:26.182 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:26.182 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:26.182 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.183 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:26.183 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:26.183 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:26.183 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:26.184 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:26.184 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:26.184 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:26.184 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:26.184 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:26.185 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:26.185 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:26.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.185 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:26.185 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:26.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:26.186 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:26.187 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:26.187 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:26.187 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:26.188 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:26.188 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:26.188 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:26.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:26.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.189 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:26.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:26.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:26.189 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:26.189 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:26.190 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:26.190 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:26.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:26.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:26.191 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:26.191 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:26.191 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:26.191 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:26.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.192 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:26.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:26.192 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:26.192 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:26.192 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:26Z" 2023-10-11T15:58:26.193 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:26.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:26Z" } 2023-10-11T15:58:26.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:26.195 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:26.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:26.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:31.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:31.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:31.199 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:31.259 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:31.259 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:31.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:31.288+0000 7fbba1d7f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:31.291 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:31.289+0000 7fbba1d7f780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:31.306 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:31.306 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:31.306 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:31.307 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:31.307 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:31.307 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:31.307 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:31.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.308 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:31.308 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:31.308 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:31.309 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:31.309 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:31.309 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:31.309 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:31.309 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:31.310 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:31.310 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:31.310 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:31.310 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:31.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.311 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:31.311 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:31.311 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:31.311 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:31.312 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:31.312 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:31.312 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:31.312 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:31.313 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:31.313 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:31.313 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:31.313 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.314 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:31.314 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:31.314 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:31.315 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:31.315 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:31.315 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:31.315 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:31.315 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:31.316 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:31.316 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:31.316 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:31.316 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.317 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:31.317 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:31.317 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:31.317 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:31.317 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:31.318 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:31.318 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:31.318 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:31.318 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:31.318 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:31.319 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:31.319 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:31.319 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:31.319 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.320 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:31.320 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:31.320 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:31.320 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:31.320 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:31Z" 2023-10-11T15:58:31.321 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:31.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:31Z" } 2023-10-11T15:58:31.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:31.323 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:31.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:31.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:36.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:36.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:36.326 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:36.384 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:36.384 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:36.417 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:36.414+0000 7f87e79a3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:36.415+0000 7f87e79a3780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:36.431 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:36.432 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:36.432 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:36.432 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:36.433 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:36.433 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:36.433 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:36.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.433 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:36.434 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:36.434 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:36.434 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:36.434 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:36.435 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:36.435 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:36.435 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:36.435 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:36.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:36.436 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:36.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:36.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.436 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:36.437 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:36.437 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:36.437 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:36.437 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:36.437 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:36.438 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:36.438 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:36.438 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:36.438 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:36.439 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:36.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:36.439 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:36.439 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:36.440 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:36.440 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:36.440 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:36.440 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:36.441 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:36.441 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:36.441 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:36.441 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:36.442 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.442 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:36.442 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:36.442 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:36.443 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:36.443 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:36.443 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:36.443 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:36.444 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:36.444 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:36.444 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:36.445 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:36.445 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:36.445 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:36.445 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.446 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:36.446 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:36.446 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:36.447 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:36.447 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:36Z" 2023-10-11T15:58:36.447 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:36.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:36Z" } 2023-10-11T15:58:36.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:36.450 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:36.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:36.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:41.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:41.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:41.453 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:41.511 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:41.511 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:41.543 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:41.541+0000 7fbaf0f61780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:41.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:41.542+0000 7fbaf0f61780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:41.558 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:41.558 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:41.558 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:41.559 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:41.559 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:41.559 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:41.559 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:41.560 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.560 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:41.560 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:41.560 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:41.560 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:41.561 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:41.561 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:41.561 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:41.561 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:41.562 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:41.562 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:41.562 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:41.562 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:41.563 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.563 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:41.563 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:41.563 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:41.563 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:41.564 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:41.565 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:41.565 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:41.565 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.565 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:41.566 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:41.566 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:41.566 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:41.566 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:41.566 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:41.567 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:41.567 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:41.567 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:41.567 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:41.568 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:41.568 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.568 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:41.568 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:41.568 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:41.569 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:41.569 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:41.569 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:41.569 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:41.569 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:41.570 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:41.570 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:41.570 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:41.570 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:41.571 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:41.571 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.571 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:41.571 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:41.571 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:41.572 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:41.572 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:41Z" 2023-10-11T15:58:41.572 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:41.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:41Z" } 2023-10-11T15:58:41.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:41.576 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:41.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:41.576 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:46.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:46.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:46.579 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:46.637 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:46.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:46.666+0000 7fc2162f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:46.670 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:46.667+0000 7fc2162f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:46.684 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:46.697 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:46.697 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:46.697 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.697 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:46.697 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:46.698 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:46.698 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:46.698 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:46Z" 2023-10-11T15:58:46.698 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:46.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:46Z" } 2023-10-11T15:58:46.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:46.701 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:46.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:46.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:51.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:51.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:51.704 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:51.762 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:51.762 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:51.793 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:51.791+0000 7fc46a5e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:51.792+0000 7fc46a5e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:51.808 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:51.808 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:51.809 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:51.809 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:51.809 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:51.809 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:51.809 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:51.810 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.810 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:51.810 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:51.810 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:51.811 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:51.811 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:51.811 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:51.811 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:51.811 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:51.812 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:51.812 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:51.812 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:51.812 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:51.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:51.813 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:51.813 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:51.813 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:51.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:51.814 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:51.814 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:51.814 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:51.815 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:51.815 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:51.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:51.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.815 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:51.816 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:51.816 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:51.816 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:51.816 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:51.817 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:51.817 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:51.817 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:51.817 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:51.818 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:51.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:51.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.818 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:51.819 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:51.819 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:51.819 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:51.819 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:51.819 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:51.820 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:51.820 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:51.820 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:51.820 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:51.821 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:51.821 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:51.821 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:51.821 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.821 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:51.822 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:51.822 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:51.822 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:51.822 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:51Z" 2023-10-11T15:58:51.823 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:51.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:51Z" } 2023-10-11T15:58:51.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:51.827 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:51.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:51.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:58:56.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:56.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:58:56.831 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:58:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:58:56.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:58:56.920 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:56.917+0000 7f0c172fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:56.921 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:58:56.918+0000 7f0c172fb780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:58:56.934 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:58:56.935 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:58:56.935 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:58:56.935 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:58:56.935 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:58:56.936 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:58:56.936 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:56.936 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.936 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:58:56.937 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:56.937 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:58:56.937 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:56.937 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:56.938 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:58:56.938 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:56.938 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:58:56.938 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:56.938 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:58:56.939 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:56.939 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:56.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.939 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:56.940 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:58:56.940 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:56.940 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:56.940 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:58:56.940 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:56.941 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:56.941 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:56.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:58:56.941 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:56.941 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:56.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.942 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:56.942 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:58:56.942 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:56.942 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:56.943 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:58:56.943 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:56.943 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:56.943 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:56.944 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:58:56.944 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:56.944 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:56.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.944 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:58:56.945 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:58:56.945 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:58:56.945 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:58:56.945 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:58:56.946 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:58:56.946 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:58:56.946 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:58:56.946 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:58:56.946 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:58:56.947 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:56.947 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:58:56.947 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:58:56.947 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:58:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:58:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:58:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:58:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:58:56Z" 2023-10-11T15:58:56.949 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:58:56.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:58:56Z" } 2023-10-11T15:58:56.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:58:56.951 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:58:56.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:58:56.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:59:01.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:01.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:01.954 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:02.011 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:02.012 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:02.043 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:02.041+0000 7fd69f6e2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:02.044 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:02.042+0000 7fd69f6e2780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:02.058 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:02.058 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:02.058 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:02.059 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:02.059 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:02.059 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:02.059 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:02.059 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.060 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:02.060 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:02.060 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:02.060 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:02.060 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:02.061 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:02.062 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:02.062 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.062 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:02.062 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:02.063 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:02.063 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:02.063 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:02.063 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:02.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:02.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:02.065 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:02.065 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:02.066 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:02.066 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:02.066 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:02.066 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:02.066 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:02.067 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:02.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:02.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.067 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:02.067 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:02.068 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:02.068 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:02.068 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:59:02.068 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:02.069 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:02.069 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:02.069 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:59:02.069 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:02.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:02.070 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:02.070 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:02.070 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.070 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:02.070 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:02.071 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:02.071 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:02.071 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:02Z" 2023-10-11T15:59:02.071 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:02.072 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:02Z" } 2023-10-11T15:59:02.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:59:02.073 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:02.074 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:59:02.074 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:59:07.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:07.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:07.077 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:07.134 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:07.135 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:07.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:07.164+0000 7fef5cf99780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:07.167 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:07.165+0000 7fef5cf99780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:07.183 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:07.183 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:07.184 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:07.184 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:07.184 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:07.184 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:07.185 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:07.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.185 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:07.186 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:07.186 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:07.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:07.186 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:07.186 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:07.187 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:07.187 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:07.187 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:07.187 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:07.188 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:07.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:07.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.188 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:07.188 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:07.189 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:07.190 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.191 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:07.191 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:07.191 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:07.191 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:07.192 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:07.192 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:07.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:07.192 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:07.192 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:07.193 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:07.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:07.193 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.193 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:07.193 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:07.194 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:07.194 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:07.194 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:59:07.194 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:07.195 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:07.195 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:07.195 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:59:07.195 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:07.195 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:07.196 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:07.196 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:07.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.196 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:07.197 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:07.197 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:07.197 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:07.197 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:07Z" 2023-10-11T15:59:07.197 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:07.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:07Z" } 2023-10-11T15:59:07.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:59:07.199 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:07.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:59:07.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:59:12.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:12.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:12.203 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:12.260 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:12.261 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:12.294 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:12.293+0000 7f2db7d10780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:12.295 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:12.294+0000 7f2db7d10780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:12.309 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:12.310 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:12.310 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:12.310 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:12.310 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:12.311 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:12.311 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:12.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.311 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:12.312 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:12.312 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:12.312 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:12.312 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:12.312 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:12.313 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:12.313 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:12.313 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:12.313 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:12.314 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:12.314 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:12.314 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.314 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:12.314 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:12.315 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:12.315 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:12.315 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:12.315 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:12.316 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:12.316 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:12.316 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:12.316 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:12.316 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:12.317 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.317 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:12.317 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:12.317 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:12.317 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:12.318 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:12.318 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:12.318 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:12.318 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:12.318 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:12.319 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:12.319 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:12.319 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.319 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:12.320 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:12.320 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:12.320 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:12.320 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:59:12.320 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:12.321 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:12.321 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:12.321 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:59:12.321 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:12.321 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:12.322 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:12.322 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:12.322 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.322 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:12.322 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:12.323 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:12.323 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:12.323 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:12Z" 2023-10-11T15:59:12.323 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:12.324 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:12Z" } 2023-10-11T15:59:12.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:59:12.326 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:12.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:59:12.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:59:17.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:17.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:17.329 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:17.385 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:17.386 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:17.418 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:17.416+0000 7f8b7c4ea780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:17.419 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:17.417+0000 7f8b7c4ea780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:17.432 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:17.432 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:17.433 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:17.433 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:17.433 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:17.433 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:17.433 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:17.434 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.434 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:17.434 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:17.434 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:17.434 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:17.435 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:17.435 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:17.435 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:17.435 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:17.435 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:17.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:17.436 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:17.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:17.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.437 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:17.437 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:17.437 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:17.437 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:17.437 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:17.438 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:17.438 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:17.438 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:17.438 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:17.438 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:17.439 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:17.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:17.439 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:17.440 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:17.440 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:17.440 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:17.440 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:17.440 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:17.441 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:17.441 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:17.441 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:17.441 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:17.441 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.442 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:17.442 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:17.442 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:17.442 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:17.443 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039713.156327_942.1", 2023-10-11T15:59:17.443 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:17.443 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:17.443 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:17.443 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:13.156327Z", 2023-10-11T15:59:17.444 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:17.444 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:17.444 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:17.444 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:17.444 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.445 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:17.445 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:17.445 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:17.445 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:17.446 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:17Z" 2023-10-11T15:59:17.446 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:17.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039713.156327_942.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:13.156327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:17Z" } 2023-10-11T15:59:17.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039713.156327_942.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:13.156327Z', 'realm_epoch': 2}}] 2023-10-11T15:59:17.448 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:17.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039713.156327_942.1'} 2023-10-11T15:59:17.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039839.220443_968.1 target=1_1697039713.156327_942.1 2023-10-11T15:59:22.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.451 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:22.511 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:22.512 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:22.544 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:22.543+0000 7fd5a25ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:22.545 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:22.544+0000 7fd5a25ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:22.558 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:22.559 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:22.559 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:22.559 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:22.559 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:22.560 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:22.560 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.560 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.560 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:22.561 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.561 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:22.561 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.561 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.562 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:22.562 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.562 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:22.562 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.563 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:22.563 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.563 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.563 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.563 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.564 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:22.564 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.564 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.564 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:22.564 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.565 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.565 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.565 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:22.565 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.566 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.566 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.566 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.566 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:22.567 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.567 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.567 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:22.567 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.567 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.568 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.569 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:22.569 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.569 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.569 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:22.569 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.570 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.570 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.570 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:22.570 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.571 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.571 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.571 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:22.571 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.571 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:22.572 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:22.572 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:22.572 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.572 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:22Z" 2023-10-11T15:59:22.572 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:22.573 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:22Z" } 2023-10-11T15:59:22.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:22.574 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:22.575 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:22.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:59:22.576 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:59:22.576 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:59:22.577 DEBUG:boto:Method: HEAD 2023-10-11T15:59:22.577 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:59:22.578 DEBUG:boto:Data: 2023-10-11T15:59:22.578 DEBUG:boto:Headers: {} 2023-10-11T15:59:22.579 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:59:22.579 DEBUG:boto:Port: 8000 2023-10-11T15:59:22.580 DEBUG:boto:Params: {} 2023-10-11T15:59:22.580 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T15:59:22.581 DEBUG:boto:Token: None 2023-10-11T15:59:22.582 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:22 GMT /dghqvd-70/ 2023-10-11T15:59:22.582 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/n+a5q6rE2Pub38jqV5oe7nelrY= 2023-10-11T15:59:22.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/n+a5q6rE2Pub38jqV5oe7nelrY=', 'Content-Length': '0'} 2023-10-11T15:59:22.589 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020201221328cc4c9-006526c65a-2688-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:22.590 DEBUG:boto:path=/dghqvd-70/obj5 2023-10-11T15:59:22.590 DEBUG:boto:auth_path=/dghqvd-70/obj5 2023-10-11T15:59:22.590 DEBUG:boto:Method: PUT 2023-10-11T15:59:22.591 DEBUG:boto:Path: /dghqvd-70/obj5 2023-10-11T15:59:22.591 DEBUG:boto:Data: 2023-10-11T15:59:22.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:59:22.592 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:59:22.592 DEBUG:boto:Port: 8000 2023-10-11T15:59:22.593 DEBUG:boto:Params: {} 2023-10-11T15:59:22.593 DEBUG:boto:Token: None 2023-10-11T15:59:22.594 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:59:22 GMT /dghqvd-70/obj5 2023-10-11T15:59:22.594 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zrc2Ye5hytvelYlQannjL8zkXes= 2023-10-11T15:59:22.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:59:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zrc2Ye5hytvelYlQannjL8zkXes='} 2023-10-11T15:59:22.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000139d071f8609835f-006526c65a-2688-a1'), ('Date', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:22.601 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:59:22.602 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:59:22.602 DEBUG:boto:Method: HEAD 2023-10-11T15:59:22.603 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:59:22.603 DEBUG:boto:Data: 2023-10-11T15:59:22.603 DEBUG:boto:Headers: {} 2023-10-11T15:59:22.604 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:59:22.604 DEBUG:boto:Port: 8000 2023-10-11T15:59:22.604 DEBUG:boto:Params: {} 2023-10-11T15:59:22.605 DEBUG:boto:Token: None 2023-10-11T15:59:22.605 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:22 GMT /dghqvd-70/ 2023-10-11T15:59:22.606 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/n+a5q6rE2Pub38jqV5oe7nelrY= 2023-10-11T15:59:22.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/n+a5q6rE2Pub38jqV5oe7nelrY=', 'Content-Length': '0'} 2023-10-11T15:59:22.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096bf95009ca2b9d0-006526c65a-2688-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:22.610 DEBUG:boto:path=/dghqvd-70/obj6 2023-10-11T15:59:22.610 DEBUG:boto:auth_path=/dghqvd-70/obj6 2023-10-11T15:59:22.611 DEBUG:boto:Method: PUT 2023-10-11T15:59:22.611 DEBUG:boto:Path: /dghqvd-70/obj6 2023-10-11T15:59:22.612 DEBUG:boto:Data: 2023-10-11T15:59:22.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T15:59:22.613 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T15:59:22.613 DEBUG:boto:Port: 8000 2023-10-11T15:59:22.614 DEBUG:boto:Params: {} 2023-10-11T15:59:22.614 DEBUG:boto:Token: None 2023-10-11T15:59:22.615 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 15:59:22 GMT /dghqvd-70/obj6 2023-10-11T15:59:22.615 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gSzKwbiL4lsMaEmSbr0VNDpiEMw= 2023-10-11T15:59:22.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:59:22 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gSzKwbiL4lsMaEmSbr0VNDpiEMw='} 2023-10-11T15:59:22.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae9ae952be2f69a2-006526c65a-2688-a1'), ('Date', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:22.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.623 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:22.673 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:22.673 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:22.704 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:59:22.716 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:59:22.717 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.717 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:22.717 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:59:22.717 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.717 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.718 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:22.718 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:59:22.718 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.718 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.719 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:22.719 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:59:22.719 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.719 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.719 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:22.720 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:57:19.220443Z" 2023-10-11T15:59:22.720 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.720 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:59:22.721 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:22.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:59:22.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.723 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:22.772 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:22.772 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:22.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:22.800+0000 7fc56ac0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:22.802 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:22.801+0000 7fc56ac0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:22.815 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:22.815 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:22.816 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:22.816 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:22.816 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:22.816 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:22.817 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.817 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.817 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:22.817 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.818 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:22.818 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.818 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.818 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:22.819 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.819 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:22.819 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.819 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:22.819 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.820 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.820 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.820 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.820 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:22.820 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.821 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.821 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:22.821 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.821 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.821 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.822 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:22.822 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.822 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.822 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.823 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.823 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:22.823 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.823 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.824 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:22.824 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.824 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.824 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.824 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:22.825 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.825 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.825 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.825 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:22.826 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:22.826 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:22.826 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:22.826 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:22.826 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:22.827 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:22.827 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:22.827 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:22.827 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.828 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.828 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:22.828 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.828 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:22.828 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:22.829 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:22.829 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.829 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:22Z" 2023-10-11T15:59:22.829 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:22.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:22Z" } 2023-10-11T15:59:22.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:22.831 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:22.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:22.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T15:59:22.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.833 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:22.882 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:22.882 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:22.930 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:59:22.931 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.931 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T15:59:22.931 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:59:22.619973Z" 2023-10-11T15:59:22.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.932 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.932 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:22.932 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:22.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.933 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.933 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T15:59:22.933 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:59:22.599357Z" 2023-10-11T15:59:22.933 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:22.934 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:22.934 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:22.934 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:22.934 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:22.934 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:59:22.935 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:22.936 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:59:22.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:22.937 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:22.988 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:22.989 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:23.033 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:23.033 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:23.034 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:23.034 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:23.034 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:23.034 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:59:23.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.035 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:23.035 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:23.035 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:23.035 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:23.036 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:23.036 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:59:23.036 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:23.036 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:23.037 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:23.037 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:59:23.037 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:23.037 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.037 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:23.038 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:23.038 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:23.038 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:23.038 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:23.038 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:23.039 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:23.039 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:23.039 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:23.039 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:23.040 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.040 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:23.040 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:23.040 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:23.041 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:23.041 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:59:23.041 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:23.041 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:59:23.041 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:23.042 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:59:23.042 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:23.042 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.042 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:23.042 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:23.043 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:23.043 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:23.043 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:23.043 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:23.044 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:59:23.044 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:23.044 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:23.044 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:23.045 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:23.045 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:23.045 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.045 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:23.045 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:59:23.046 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:59:23.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:23.046 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:23Z" 2023-10-11T15:59:23.046 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:23.047 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:14.168373Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051390", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:55:14.141173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:59:23Z" } 2023-10-11T15:59:23.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.168373Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051390', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.141173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:59:23.048 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:23.049 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:23.049 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657, shard=2 master=00000000000000000000:00000000000000051530 target=00000000000000000000:00000000000000051390 2023-10-11T15:59:28.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:28.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:28.052 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:28.111 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:28.112 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:28.161 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:28.161 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:28.161 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:28.161 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:28.162 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:28.162 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:59:28.162 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.162 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:28.163 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:28.163 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:28.163 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:28.163 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:28.163 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:59:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:59:28.164 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:28.165 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.165 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:28.165 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:28.165 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:28.166 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:28.166 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:28.166 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:28.166 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:28.166 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:28.167 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:28.167 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:28.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.167 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:28.167 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:28.168 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:28.168 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:28.168 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:59:28.168 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:28.169 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:59:28.169 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:28.169 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:59:28.169 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:28.169 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.170 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:28.170 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:28.170 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:28.170 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:28.170 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:28.171 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:28.171 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:59:28.171 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:28.171 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:28.171 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:28.172 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:28.172 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:28.172 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.172 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:28.172 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:59:28.173 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:59:28.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:28.173 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:28Z" 2023-10-11T15:59:28.173 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:28.174 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:14.168373Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051390", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:55:14.141173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:59:28Z" } 2023-10-11T15:59:28.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.168373Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051390', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.141173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:59:28.175 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:28.176 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:28.176 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657, shard=2 master=00000000000000000000:00000000000000051530 target=00000000000000000000:00000000000000051390 2023-10-11T15:59:33.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:33.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:33.178 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:33.238 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:33.285 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:33.286 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:33.286 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:33.286 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:33.286 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:33.287 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:59:33.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.287 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:33.287 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:33.287 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:33.288 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:33.288 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:33.288 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:59:33.288 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:33.288 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:33.289 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:33.290 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:33.290 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:33.290 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:33.290 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.291 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:33.292 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:33.292 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:33.292 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:33.292 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:59:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:59:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:59:33.293 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:33.294 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.294 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:33.294 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:33.294 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:33.295 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:33.295 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:33.295 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:33.295 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:59:33.296 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:33.296 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:33.296 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:33.296 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:33.296 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:33.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.297 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:33.297 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:59:33.297 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:59:33.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:33.298 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:33Z" 2023-10-11T15:59:33.298 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:33.299 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:14.168373Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051390", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:55:14.141173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:59:33Z" } 2023-10-11T15:59:33.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.168373Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051390', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.141173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:59:33.300 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:33.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:33.301 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657, shard=2 master=00000000000000000000:00000000000000051530 target=00000000000000000000:00000000000000051390 2023-10-11T15:59:38.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:38.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:38.304 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:38.365 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:38.365 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:38.414 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:38.414 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:38.414 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:38.415 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:38.415 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:38.415 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:59:38.415 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.416 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:38.416 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:38.416 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:38.416 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:38.417 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:38.417 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-10-11T15:59:38.417 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:38.417 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:38.418 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:38.418 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.168373Z" 2023-10-11T15:59:38.418 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:38.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.419 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:38.419 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:38.419 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:38.419 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:38.419 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:38.420 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:38.420 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:38.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:38.420 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:38.421 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:38.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.421 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:38.421 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:38.422 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:38.422 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:38.422 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051390", 2023-10-11T15:59:38.422 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:38.423 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:59:38.423 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:38.423 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:55:14.141173Z" 2023-10-11T15:59:38.423 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:38.424 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.424 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:38.424 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:38.424 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:38.425 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:38.425 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:38.425 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:38.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:59:38.426 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:38.426 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:38.426 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:38.426 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:38.427 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:38.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.427 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:38.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:59:38.428 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:59:38.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:38.428 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:38Z" 2023-10-11T15:59:38.428 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:38.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:55:14.168373Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051390", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:55:14.141173Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:59:38Z" } 2023-10-11T15:59:38.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.168373Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051390', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:55:14.141173Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:59:38.431 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:38.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045657', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051390', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:38.432 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657, shard=2 master=00000000000000000000:00000000000000051530 target=00000000000000000000:00000000000000051390 2023-10-11T15:59:43.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.434 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:43.494 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:43.495 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:43.544 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:43.544 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:43.544 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:43.544 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:43.545 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:43.545 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T15:59:43.545 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.545 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:43.546 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.546 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:43.546 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.546 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:43.546 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T15:59:43.547 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.547 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:43.547 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.547 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T15:59:43.547 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.548 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.548 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:43.548 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.549 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:43.549 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T15:59:43.549 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.549 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:43.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.550 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T15:59:43.550 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.550 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.550 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.551 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:43.551 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.551 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:43.551 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T15:59:43.552 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.552 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T15:59:43.552 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.552 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T15:59:43.552 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.553 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.553 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:43.553 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.553 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T15:59:43.554 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T15:59:43.554 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.554 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T15:59:43.554 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.554 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T15:59:43.555 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.555 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.555 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:43.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.556 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:43.556 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T15:59:43.556 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T15:59:43.556 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.556 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:43Z" 2023-10-11T15:59:43.557 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:43.558 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032203", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:44:25.137463Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044174", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T15:53:09.128491Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T15:59:43Z" } 2023-10-11T15:59:43.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T15:59:43.559 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:43.559 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T15:59:43.560 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T15:59:43.560 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T15:59:43.560 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T15:59:43.561 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.561 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T15:59:43.561 DEBUG:boto:Data: 2023-10-11T15:59:43.562 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.562 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.563 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.563 DEBUG:boto:Params: {} 2023-10-11T15:59:43.564 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T15:59:43.564 DEBUG:boto:Token: None 2023-10-11T15:59:43.565 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-70/ 2023-10-11T15:59:43.565 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:OdE7m73QqVpDrLodlpdw7XO11N4= 2023-10-11T15:59:43.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OdE7m73QqVpDrLodlpdw7XO11N4=', 'Content-Length': '0'} 2023-10-11T15:59:43.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a754dda0cbf3eab8-006526c66f-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.572 DEBUG:boto:path=/dghqvd-70/obj5 2023-10-11T15:59:43.573 DEBUG:boto:auth_path=/dghqvd-70/obj5 2023-10-11T15:59:43.573 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.574 DEBUG:boto:Path: /dghqvd-70/obj5 2023-10-11T15:59:43.574 DEBUG:boto:Data: 2023-10-11T15:59:43.575 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.575 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.575 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.576 DEBUG:boto:Params: {} 2023-10-11T15:59:43.576 DEBUG:boto:Token: None 2023-10-11T15:59:43.577 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-70/obj5 2023-10-11T15:59:43.577 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:vOh4IdGkqfkg5VHRM2aQaR0Vxag= 2023-10-11T15:59:43.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vOh4IdGkqfkg5VHRM2aQaR0Vxag=', 'Content-Length': '0'} 2023-10-11T15:59:43.579 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000015a9f54e3ba86c10-006526c66f-267e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.580 DEBUG:boto:path=/dghqvd-70/obj6 2023-10-11T15:59:43.580 DEBUG:boto:auth_path=/dghqvd-70/obj6 2023-10-11T15:59:43.581 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.581 DEBUG:boto:Path: /dghqvd-70/obj6 2023-10-11T15:59:43.581 DEBUG:boto:Data: 2023-10-11T15:59:43.582 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.582 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.583 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.583 DEBUG:boto:Params: {} 2023-10-11T15:59:43.583 DEBUG:boto:Token: None 2023-10-11T15:59:43.584 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-70/obj6 2023-10-11T15:59:43.584 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:387K9avdvt4ZhPm/22wQoBW08Gk= 2023-10-11T15:59:43.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:387K9avdvt4ZhPm/22wQoBW08Gk=', 'Content-Length': '0'} 2023-10-11T15:59:43.586 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f6e3f4ad841d5208-006526c66f-267e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.587 DEBUG:boto:path=/dghqvd-71/ 2023-10-11T15:59:43.587 DEBUG:boto:auth_path=/dghqvd-71/ 2023-10-11T15:59:43.588 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.588 DEBUG:boto:Path: /dghqvd-71/ 2023-10-11T15:59:43.589 DEBUG:boto:Data: 2023-10-11T15:59:43.589 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.589 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.590 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.590 DEBUG:boto:Params: {} 2023-10-11T15:59:43.591 DEBUG:boto:Token: None 2023-10-11T15:59:43.591 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-71/ 2023-10-11T15:59:43.591 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Y1NrKqp0k2j5+kklPXtzPOOIS+w= 2023-10-11T15:59:43.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y1NrKqp0k2j5+kklPXtzPOOIS+w=', 'Content-Length': '0'} 2023-10-11T15:59:43.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aa26406ec212e0d-006526c66f-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.595 DEBUG:boto:path=/dghqvd-71/obj5 2023-10-11T15:59:43.595 DEBUG:boto:auth_path=/dghqvd-71/obj5 2023-10-11T15:59:43.595 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.596 DEBUG:boto:Path: /dghqvd-71/obj5 2023-10-11T15:59:43.596 DEBUG:boto:Data: 2023-10-11T15:59:43.597 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.597 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.598 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.598 DEBUG:boto:Params: {} 2023-10-11T15:59:43.598 DEBUG:boto:Token: None 2023-10-11T15:59:43.599 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-71/obj5 2023-10-11T15:59:43.599 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1zvSb26q3PWw8HaCrMzPwsSScFM= 2023-10-11T15:59:43.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1zvSb26q3PWw8HaCrMzPwsSScFM=', 'Content-Length': '0'} 2023-10-11T15:59:43.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f61ab3e74dcb45fe-006526c66f-267e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.602 DEBUG:boto:path=/dghqvd-71/obj5 2023-10-11T15:59:43.603 DEBUG:boto:auth_path=/dghqvd-71/obj5 2023-10-11T15:59:43.603 DEBUG:boto:Method: GET 2023-10-11T15:59:43.604 DEBUG:boto:Path: /dghqvd-71/obj5 2023-10-11T15:59:43.604 DEBUG:boto:Data: 2023-10-11T15:59:43.604 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.605 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.605 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.606 DEBUG:boto:Params: {} 2023-10-11T15:59:43.606 DEBUG:boto:Token: None 2023-10-11T15:59:43.607 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-71/obj5 2023-10-11T15:59:43.607 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e2woXEm18EpSQKJPyCPOdrYmKR4= 2023-10-11T15:59:43.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e2woXEm18EpSQKJPyCPOdrYmKR4=', 'Content-Length': '0'} 2023-10-11T15:59:43.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a71319599180d8ef-006526c66f-267e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.650 DEBUG:boto:path=/dghqvd-71/obj6 2023-10-11T15:59:43.651 DEBUG:boto:auth_path=/dghqvd-71/obj6 2023-10-11T15:59:43.651 DEBUG:boto:Method: HEAD 2023-10-11T15:59:43.652 DEBUG:boto:Path: /dghqvd-71/obj6 2023-10-11T15:59:43.652 DEBUG:boto:Data: 2023-10-11T15:59:43.653 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.653 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.654 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.654 DEBUG:boto:Params: {} 2023-10-11T15:59:43.654 DEBUG:boto:Token: None 2023-10-11T15:59:43.655 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-71/obj6 2023-10-11T15:59:43.655 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PhV2WnTdfLzKYuhjf4UX1yK5Z9Q= 2023-10-11T15:59:43.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PhV2WnTdfLzKYuhjf4UX1yK5Z9Q=', 'Content-Length': '0'} 2023-10-11T15:59:43.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000135e1f79c350dcc8-006526c66f-267e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.658 DEBUG:boto:path=/dghqvd-71/obj6 2023-10-11T15:59:43.659 DEBUG:boto:auth_path=/dghqvd-71/obj6 2023-10-11T15:59:43.659 DEBUG:boto:Method: GET 2023-10-11T15:59:43.660 DEBUG:boto:Path: /dghqvd-71/obj6 2023-10-11T15:59:43.660 DEBUG:boto:Data: 2023-10-11T15:59:43.660 DEBUG:boto:Headers: {} 2023-10-11T15:59:43.661 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T15:59:43.661 DEBUG:boto:Port: 8001 2023-10-11T15:59:43.662 DEBUG:boto:Params: {} 2023-10-11T15:59:43.662 DEBUG:boto:Token: None 2023-10-11T15:59:43.663 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 15:59:43 GMT /dghqvd-71/obj6 2023-10-11T15:59:43.663 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:2QH8ZQd1nczUVeH5iCehFa5BdOI= 2023-10-11T15:59:43.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:59:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2QH8ZQd1nczUVeH5iCehFa5BdOI=', 'Content-Length': '0'} 2023-10-11T15:59:43.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:59:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-70:daa316e9-869c-4074-a756-a8adfb4a3573.9311.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c621b2e8844e0c6d-006526c66f-267e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:59:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T15:59:43.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:59:43.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T15:59:43.667 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T15:59:43.718 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:43.718 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:43.771 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T15:59:43.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.772 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:43.821 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:43.821 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:43.851 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T15:59:43.864 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T15:59:43.864 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.864 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:43.865 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T15:59:43.865 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.865 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.865 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:43.865 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T15:59:43.866 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.866 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:43.866 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T15:59:43.866 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.867 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.867 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T15:59:43.867 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:59:43.751700Z" 2023-10-11T15:59:43.867 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.867 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T15:59:43.869 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T15:59:43.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T15:59:43.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:43.870 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:43.920 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:43.920 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:43.949 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:43.947+0000 7f71dbbab780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:43.950 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:43.948+0000 7f71dbbab780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:43.963 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:43.963 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:43.963 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:43.963 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:43.963 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:43.964 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:43.964 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:43.964 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.964 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:43.965 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.965 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:43.965 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.965 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:43.965 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:43.966 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.966 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:43.966 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.966 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:43.966 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:43.967 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.967 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.967 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.967 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:43.968 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.968 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:43.968 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:43.968 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.968 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:43.969 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.969 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:43.969 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:43.969 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.969 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.970 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.970 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:43.970 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.970 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:43.971 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:43.971 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.971 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:43.971 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.971 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:43.972 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:43.972 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.972 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.972 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:43.972 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:43.973 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:43.973 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:43.973 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:43.973 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:43.973 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:43.974 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:43.974 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:43.974 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:43.974 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.974 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:43.975 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:43.975 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.975 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:43.975 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:43.975 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:43.976 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:43.976 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:43Z" 2023-10-11T15:59:43.976 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:43.977 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:43Z" } 2023-10-11T15:59:43.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:43.978 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T15:59:43.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:43.979 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039983.751700_970.1 target=1_1697039839.220443_968.1 2023-10-11T15:59:48.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:48.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:48.981 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:49.038 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:49.039 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:49.071 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:49.069+0000 7f36e7c98780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:49.072 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:49.071+0000 7f36e7c98780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:49.086 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:49.086 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:49.086 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:49.087 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:49.087 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:49.087 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:49.087 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:49.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.088 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:49.088 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:49.088 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:49.088 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:49.089 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:49.089 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:49.089 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:49.089 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:49.089 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:49.090 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:49.090 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:49.090 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:49.090 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:49.091 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:49.092 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:49.092 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:49.092 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:49.092 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:49.092 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:49.093 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.093 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:49.093 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:49.093 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:49.094 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:49.094 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:49.094 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:49.094 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:49.094 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:49.095 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:49.095 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:49.095 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:49.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.095 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:49.096 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:49.096 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:49.096 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:49.096 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:49.097 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:49.097 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:49.097 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:49.097 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:49.097 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:49.098 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:49.098 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:49.098 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:49.098 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.099 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:49.099 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:49.099 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:49.099 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:49.099 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:49Z" 2023-10-11T15:59:49.100 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:49.101 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:49Z" } 2023-10-11T15:59:49.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:49.102 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T15:59:49.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:49.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039983.751700_970.1 target=1_1697039839.220443_968.1 2023-10-11T15:59:54.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:54.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:54.105 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:54.162 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:54.195 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:54.194+0000 7fc524df1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:54.196 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:54.195+0000 7fc524df1780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:54.210 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:54.210 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:54.210 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:54.211 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:54.211 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:54.211 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:54.211 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:54.212 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.212 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:54.212 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:54.212 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:54.212 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:54.213 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:54.213 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:54.213 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:54.213 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:54.214 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:54.214 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:54.214 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:54.214 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:54.214 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.215 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:54.215 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:54.215 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:54.215 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:54.216 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:54.216 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:54.216 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:54.216 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:54.216 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:54.217 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:54.217 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:54.217 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.217 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:54.217 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:54.218 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:54.218 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:54.218 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:54.218 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:54.219 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:54.219 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:54.219 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:54.219 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:54.219 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:54.220 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:54.221 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:54.222 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:54.223 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:54Z" 2023-10-11T15:59:54.224 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:54.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:54Z" } 2023-10-11T15:59:54.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:54.226 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T15:59:54.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:54.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039983.751700_970.1 target=1_1697039839.220443_968.1 2023-10-11T15:59:59.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:59.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T15:59:59.229 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T15:59:59.288 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T15:59:59.288 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T15:59:59.321 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:59.319+0000 7f922ee14780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:59.322 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T15:59:59.320+0000 7f922ee14780 20 RGW-SYNC:meta: read sync status 2023-10-11T15:59:59.336 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T15:59:59.336 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T15:59:59.336 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T15:59:59.336 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T15:59:59.337 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T15:59:59.337 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T15:59:59.337 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:59.337 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.338 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T15:59:59.338 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:59.338 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T15:59:59.338 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:59.338 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:59.339 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T15:59:59.339 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:59.339 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T15:59:59.339 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:59.339 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T15:59:59.340 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:59.340 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:59.340 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.340 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:59.340 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T15:59:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T15:59:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:59.342 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:59.342 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T15:59:59.342 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:59.342 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:59.342 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.343 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:59.343 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T15:59:59.343 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:59.343 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:59.344 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T15:59:59.344 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:59.344 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:59.344 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:59.344 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T15:59:59.345 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:59.345 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:59.345 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.345 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T15:59:59.345 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T15:59:59.346 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T15:59:59.346 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T15:59:59.346 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039839.220443_968.1", 2023-10-11T15:59:59.346 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T15:59:59.346 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T15:59:59.347 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T15:59:59.347 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:57:19.220443Z", 2023-10-11T15:59:59.347 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T15:59:59.347 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:59.347 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T15:59:59.348 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T15:59:59.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.348 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T15:59:59.348 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T15:59:59.348 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T15:59:59.349 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T15:59:59.349 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T15:59:59Z" 2023-10-11T15:59:59.349 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T15:59:59.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039839.220443_968.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:57:19.220443Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T15:59:59Z" } 2023-10-11T15:59:59.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039839.220443_968.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:57:19.220443Z', 'realm_epoch': 2}}] 2023-10-11T15:59:59.351 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T15:59:59.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039839.220443_968.1'} 2023-10-11T15:59:59.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697039983.751700_970.1 target=1_1697039839.220443_968.1 2023-10-11T16:00:04.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:04.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:04.354 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:04.412 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.412 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.445 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:04.442+0000 7fe1e6d82780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:04.446 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:04.443+0000 7fe1e6d82780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:04.460 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:04.460 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:04.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:04.462 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:04.463 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.464 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:04.465 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:04.466 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:04.467 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:04.467 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:04.467 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:04.467 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:04.467 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.468 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:04.468 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:04.468 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:04.469 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:04.469 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:04.469 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:04.469 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:04.470 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:04.470 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:04.470 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:04.470 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:04.471 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.471 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:04.471 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:04.471 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:04.472 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:04.472 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T16:00:04.472 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:04.472 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:04.473 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:04.473 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:43.751700Z", 2023-10-11T16:00:04.473 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:04.473 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:04.474 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:04.474 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:04.474 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.474 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:04.475 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:04.475 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:04.475 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:04.475 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:04Z" 2023-10-11T16:00:04.475 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:04.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039983.751700_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:43.751700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:04Z" } 2023-10-11T16:00:04.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039983.751700_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:43.751700Z', 'realm_epoch': 2}}] 2023-10-11T16:00:04.478 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:04.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:04.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:00:04.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.481 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:04.532 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.532 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.577 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T16:00:04.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.579 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:04.630 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.631 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.680 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:00:04.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.681 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:04.737 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.737 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.785 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:00:04.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-70', '--source-bucket=dghqvd-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dghqvd-70', '--source-bucket=dghqvd-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.786 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dghqvd-70 --source-bucket=dghqvd-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:04.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.842 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.895 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T16:00:04.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.896 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:04.948 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:04.948 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:04.997 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T16:00:04.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:04.999 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:05.046 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:05.047 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:05.092 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T16:00:05.092 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T16:00:05.093 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T16:00:05.093 DEBUG:boto:Method: HEAD 2023-10-11T16:00:05.094 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T16:00:05.094 DEBUG:boto:Data: 2023-10-11T16:00:05.095 DEBUG:boto:Headers: {} 2023-10-11T16:00:05.095 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.095 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.096 DEBUG:boto:Params: {} 2023-10-11T16:00:05.096 DEBUG:boto:Token: None 2023-10-11T16:00:05.097 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-70/ 2023-10-11T16:00:05.098 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wj+mmUkV6EgHWweS3HMJ+c0GGYI= 2023-10-11T16:00:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wj+mmUkV6EgHWweS3HMJ+c0GGYI=', 'Content-Length': '0'} 2023-10-11T16:00:05.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a955958ebe47e192-006526c685-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.101 DEBUG:boto:path=/dghqvd-70/obj7 2023-10-11T16:00:05.102 DEBUG:boto:auth_path=/dghqvd-70/obj7 2023-10-11T16:00:05.102 DEBUG:boto:Method: PUT 2023-10-11T16:00:05.103 DEBUG:boto:Path: /dghqvd-70/obj7 2023-10-11T16:00:05.103 DEBUG:boto:Data: 2023-10-11T16:00:05.104 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:05.104 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.104 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.105 DEBUG:boto:Params: {} 2023-10-11T16:00:05.105 DEBUG:boto:Token: None 2023-10-11T16:00:05.106 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-70/obj7 2023-10-11T16:00:05.106 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:gJq1hyk1Vk5M3ckhXgxgQWsbTQQ= 2023-10-11T16:00:05.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gJq1hyk1Vk5M3ckhXgxgQWsbTQQ='} 2023-10-11T16:00:05.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe393ee647bff908-006526c685-267e-a2'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.113 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T16:00:05.114 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T16:00:05.114 DEBUG:boto:Method: HEAD 2023-10-11T16:00:05.115 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T16:00:05.115 DEBUG:boto:Data: 2023-10-11T16:00:05.115 DEBUG:boto:Headers: {} 2023-10-11T16:00:05.116 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.116 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.117 DEBUG:boto:Params: {} 2023-10-11T16:00:05.117 DEBUG:boto:Token: None 2023-10-11T16:00:05.118 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-70/ 2023-10-11T16:00:05.118 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wj+mmUkV6EgHWweS3HMJ+c0GGYI= 2023-10-11T16:00:05.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wj+mmUkV6EgHWweS3HMJ+c0GGYI=', 'Content-Length': '0'} 2023-10-11T16:00:05.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd762e203dc6c66c-006526c685-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.122 DEBUG:boto:path=/dghqvd-70/obj8 2023-10-11T16:00:05.122 DEBUG:boto:auth_path=/dghqvd-70/obj8 2023-10-11T16:00:05.123 DEBUG:boto:Method: PUT 2023-10-11T16:00:05.123 DEBUG:boto:Path: /dghqvd-70/obj8 2023-10-11T16:00:05.124 DEBUG:boto:Data: 2023-10-11T16:00:05.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:05.125 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.125 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.126 DEBUG:boto:Params: {} 2023-10-11T16:00:05.126 DEBUG:boto:Token: None 2023-10-11T16:00:05.127 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-70/obj8 2023-10-11T16:00:05.128 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ANoxXpnkGWH+5Oalcj1Lf7x9Vbs= 2023-10-11T16:00:05.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ANoxXpnkGWH+5Oalcj1Lf7x9Vbs='} 2023-10-11T16:00:05.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023e19bbd51e963bb-006526c685-267e-a2'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.134 DEBUG:boto:path=/dghqvd-71/ 2023-10-11T16:00:05.134 DEBUG:boto:auth_path=/dghqvd-71/ 2023-10-11T16:00:05.135 DEBUG:boto:Method: HEAD 2023-10-11T16:00:05.135 DEBUG:boto:Path: /dghqvd-71/ 2023-10-11T16:00:05.136 DEBUG:boto:Data: 2023-10-11T16:00:05.136 DEBUG:boto:Headers: {} 2023-10-11T16:00:05.136 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.137 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.137 DEBUG:boto:Params: {} 2023-10-11T16:00:05.138 DEBUG:boto:Token: None 2023-10-11T16:00:05.138 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-71/ 2023-10-11T16:00:05.139 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CsNRgvYJ/Qt6v8cYftPjKp20hLU= 2023-10-11T16:00:05.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CsNRgvYJ/Qt6v8cYftPjKp20hLU=', 'Content-Length': '0'} 2023-10-11T16:00:05.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f46a5cc71455d68-006526c685-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.142 DEBUG:boto:path=/dghqvd-71/obj9 2023-10-11T16:00:05.142 DEBUG:boto:auth_path=/dghqvd-71/obj9 2023-10-11T16:00:05.143 DEBUG:boto:Method: PUT 2023-10-11T16:00:05.143 DEBUG:boto:Path: /dghqvd-71/obj9 2023-10-11T16:00:05.144 DEBUG:boto:Data: 2023-10-11T16:00:05.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:05.145 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.145 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.145 DEBUG:boto:Params: {} 2023-10-11T16:00:05.146 DEBUG:boto:Token: None 2023-10-11T16:00:05.146 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-71/obj9 2023-10-11T16:00:05.147 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:eCkI8vQVHgZGafphLiWzcboInak= 2023-10-11T16:00:05.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eCkI8vQVHgZGafphLiWzcboInak='} 2023-10-11T16:00:05.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075835e10b11da4b2-006526c685-267e-a2'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.191 DEBUG:boto:path=/dghqvd-71/ 2023-10-11T16:00:05.192 DEBUG:boto:auth_path=/dghqvd-71/ 2023-10-11T16:00:05.192 DEBUG:boto:Method: HEAD 2023-10-11T16:00:05.192 DEBUG:boto:Path: /dghqvd-71/ 2023-10-11T16:00:05.193 DEBUG:boto:Data: 2023-10-11T16:00:05.194 DEBUG:boto:Headers: {} 2023-10-11T16:00:05.194 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.195 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.195 DEBUG:boto:Params: {} 2023-10-11T16:00:05.196 DEBUG:boto:Token: None 2023-10-11T16:00:05.196 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-71/ 2023-10-11T16:00:05.197 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:CsNRgvYJ/Qt6v8cYftPjKp20hLU= 2023-10-11T16:00:05.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:CsNRgvYJ/Qt6v8cYftPjKp20hLU=', 'Content-Length': '0'} 2023-10-11T16:00:05.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eedb7e475d16683-006526c685-267e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.200 DEBUG:boto:path=/dghqvd-71/obj10 2023-10-11T16:00:05.200 DEBUG:boto:auth_path=/dghqvd-71/obj10 2023-10-11T16:00:05.201 DEBUG:boto:Method: PUT 2023-10-11T16:00:05.201 DEBUG:boto:Path: /dghqvd-71/obj10 2023-10-11T16:00:05.201 DEBUG:boto:Data: 2023-10-11T16:00:05.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:05.202 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:00:05.203 DEBUG:boto:Port: 8001 2023-10-11T16:00:05.203 DEBUG:boto:Params: {} 2023-10-11T16:00:05.204 DEBUG:boto:Token: None 2023-10-11T16:00:05.204 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:05 GMT /dghqvd-71/obj10 2023-10-11T16:00:05.205 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Wc+NhUnpG+MbtpPxrOMYXcxazT0= 2023-10-11T16:00:05.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:05 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Wc+NhUnpG+MbtpPxrOMYXcxazT0='} 2023-10-11T16:00:05.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a08ecf357d5d7b6-006526c685-267e-a2'), ('Date', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:05.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:05.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:05.212 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:05.261 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:05.291 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:00:05.305 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:05.305 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.305 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:05.306 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:00:05.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.306 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.306 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:05.307 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:00:05.307 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.307 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:00:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040004.982172_978.1", 2023-10-11T16:00:05.308 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:04.982172Z" 2023-10-11T16:00:05.309 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.309 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:05.310 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:05.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:00:05.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:05.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:05.312 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:05.362 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:05.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:05.389+0000 7feadc6d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:05.392 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:05.390+0000 7feadc6d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:05.405 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:05.405 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:05.406 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:05.406 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:05.406 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:05.406 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:05.406 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:05.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.407 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:05.407 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.407 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:05.408 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:05.408 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:05.408 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:05.408 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:05.408 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:05.409 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:05.409 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:05.409 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:05.409 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.410 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.410 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.410 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:05.410 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:05.410 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:05.411 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:05.411 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:05.411 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:05.411 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:05.411 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:05.412 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:05.412 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.412 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.412 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.412 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:05.413 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:05.413 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:05.413 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:05.413 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:05.413 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:05.414 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:05.414 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:05.414 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:05.414 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.415 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.415 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:05.415 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:05.415 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:05.415 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:05.416 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T16:00:05.416 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:05.416 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:05.416 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:05.417 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:43.751700Z", 2023-10-11T16:00:05.417 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:05.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.417 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:05.417 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:05.418 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:05.419 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:05Z" 2023-10-11T16:00:05.419 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:05.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039983.751700_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:43.751700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:05Z" } 2023-10-11T16:00:05.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039983.751700_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:43.751700Z', 'realm_epoch': 2}}] 2023-10-11T16:00:05.421 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:05.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:05.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040004.982172_978.1 target=1_1697039983.751700_970.1 2023-10-11T16:00:10.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:10.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:10.424 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:10.481 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:10.482 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:10.516 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:10.513+0000 7f49c38d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:10.517 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:10.515+0000 7f49c38d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:10.531 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:10.531 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:10.531 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:10.532 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:10.532 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:10.532 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:10.532 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:10.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.533 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:10.533 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:10.533 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:10.533 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:10.534 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:10.534 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:10.534 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:10.534 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:10.534 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:10.535 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:10.535 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:10.535 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:10.535 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.535 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:10.536 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:10.536 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:10.536 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:10.536 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:10.537 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:10.537 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:10.537 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:10.537 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:10.537 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:10.538 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:10.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.538 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:10.538 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:10.538 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:10.539 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:10.539 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:10.539 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:10.539 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:10.539 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:10.540 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:10.540 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:10.540 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:10.540 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.540 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:10.541 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:10.541 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:10.541 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:10.541 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T16:00:10.542 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:10.542 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:10.542 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:10.542 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:43.751700Z", 2023-10-11T16:00:10.542 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:10.543 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:10.543 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:10.543 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:10.543 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.543 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:10.544 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:10.544 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:10.544 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:10.544 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:10Z" 2023-10-11T16:00:10.544 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:10.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039983.751700_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:43.751700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:10Z" } 2023-10-11T16:00:10.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039983.751700_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:43.751700Z', 'realm_epoch': 2}}] 2023-10-11T16:00:10.547 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:10.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:10.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040004.982172_978.1 target=1_1697039983.751700_970.1 2023-10-11T16:00:15.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:15.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:15.551 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:15.607 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:15.608 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:15.640 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:15.637+0000 7f0611069780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:15.641 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:15.638+0000 7f0611069780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:15.654 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:15.655 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:15.655 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:15.655 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:15.655 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:15.656 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:15.657 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:15.657 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:15.657 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:15.657 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:15.658 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:15.658 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:15.658 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:15.658 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:15.658 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:15.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.659 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:15.659 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:15.659 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:15.659 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:15.660 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:15.660 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:15.660 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:15.660 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:15.661 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:15.661 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:15.661 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:15.661 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.661 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:15.662 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:15.662 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:15.662 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:15.662 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:15.662 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:15.663 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:15.663 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:15.663 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:15.663 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:15.663 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:15.664 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T16:00:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:15.665 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:43.751700Z", 2023-10-11T16:00:15.666 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:15.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:15Z" 2023-10-11T16:00:15.670 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:15.671 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039983.751700_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:43.751700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:15Z" } 2023-10-11T16:00:15.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039983.751700_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:43.751700Z', 'realm_epoch': 2}}] 2023-10-11T16:00:15.672 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:15.673 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:15.673 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040004.982172_978.1 target=1_1697039983.751700_970.1 2023-10-11T16:00:20.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:20.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:20.675 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:20.732 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:20.732 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:20.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:20.761+0000 7ffaf3a90780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:20.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:20.762+0000 7ffaf3a90780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:20.778 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:20.778 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:20.779 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:20.779 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:20.779 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:20.779 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:20.779 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:20.780 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.780 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:20.780 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:20.780 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:20.780 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:20.781 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:20.781 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:20.781 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:20.781 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:20.781 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:20.782 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:20.782 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:20.782 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:20.782 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.783 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:20.783 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:20.783 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:20.783 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:20.783 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:20.784 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:20.784 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:20.784 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:20.784 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:20.785 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:20.786 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:20.787 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:20.787 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:20.787 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:20.787 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:20.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:20.788 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697039983.751700_970.1", 2023-10-11T16:00:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:20.789 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:43.751700Z", 2023-10-11T16:00:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:20.790 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:20.791 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:20.792 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:20Z" 2023-10-11T16:00:20.792 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:20.793 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697039983.751700_970.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:43.751700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:20Z" } 2023-10-11T16:00:20.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039983.751700_970.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:43.751700Z', 'realm_epoch': 2}}] 2023-10-11T16:00:20.794 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:20.794 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039983.751700_970.1'} 2023-10-11T16:00:20.795 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040004.982172_978.1 target=1_1697039983.751700_970.1 2023-10-11T16:00:25.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:25.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:25.798 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:25.855 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:25.856 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:25.890 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:25.887+0000 7f9629562780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:25.891 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:25.888+0000 7f9629562780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:25.905 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:25.905 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:25.906 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:25.906 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:25.906 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:25.906 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:25.907 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:25.907 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.907 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:25.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:25.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:25.908 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:25.908 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:25.908 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:25.909 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:25.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:25.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:25.909 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:25.909 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:25.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:25.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.910 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:25.910 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:25.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:25.912 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.913 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:25.913 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:25.913 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:25.913 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:25.914 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:25.914 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:25.914 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:25.914 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:25.914 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:25.915 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:25.915 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:25.915 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.915 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:25.916 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:25.916 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:25.916 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:25.916 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040004.982172_978.1", 2023-10-11T16:00:25.917 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:25.917 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:25.917 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:25.917 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:04.982172Z", 2023-10-11T16:00:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:25Z" 2023-10-11T16:00:25.921 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:25.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040004.982172_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:04.982172Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:25Z" } 2023-10-11T16:00:25.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040004.982172_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:04.982172Z', 'realm_epoch': 2}}] 2023-10-11T16:00:25.923 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:25.923 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:25.924 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:00:25.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:25.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:25.925 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:25.975 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:25.975 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.026 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:26.026 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.026 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000013798", 2023-10-11T16:00:26.027 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:59:39.856771Z" 2023-10-11T16:00:26.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.027 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006572", 2023-10-11T16:00:26.028 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:05.208654Z" 2023-10-11T16:00:26.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.028 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.028 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000017839", 2023-10-11T16:00:26.028 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:05.111317Z" 2023-10-11T16:00:26.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.029 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000016719", 2023-10-11T16:00:26.029 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:05.151723Z" 2023-10-11T16:00:26.029 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.030 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:26.031 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000013798', 1: '00000000000000000000:00000000000000006572', 2: '00000000000000000000:00000000000000017839', 3: '00000000000000000000:00000000000000016719'} 2023-10-11T16:00:26.031 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T16:00:26.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.033 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:26.084 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:26.085 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.129 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:26.129 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:26.129 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:26.129 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:26.130 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:26.130 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 1069743287075291108 2023-10-11T16:00:26.130 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.130 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:26.130 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.131 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:26.131 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.131 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:26.131 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000013798", 2023-10-11T16:00:26.132 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.132 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.132 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.132 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:39.856771Z" 2023-10-11T16:00:26.132 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.133 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.133 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:26.134 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.134 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:26.134 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000006572", 2023-10-11T16:00:26.134 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.135 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.135 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.135 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:05.208654Z" 2023-10-11T16:00:26.135 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.136 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:26.136 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.136 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:26.137 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000017839", 2023-10-11T16:00:26.137 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.137 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.138 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.138 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:05.111317Z" 2023-10-11T16:00:26.138 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.138 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.138 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.139 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:26.139 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.139 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:26.139 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000016719", 2023-10-11T16:00:26.139 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.140 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.140 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.140 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:05.151723Z" 2023-10-11T16:00:26.140 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.141 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:26.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.141 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:26.141 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 0, 2023-10-11T16:00:26.142 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 0 2023-10-11T16:00:26.142 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.142 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:26Z" 2023-10-11T16:00:26.142 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:26.143 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1069743287075291108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013798", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:39.856771Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006572", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:05.208654Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017839", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:05.111317Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016719", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:05.151723Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-10-11T16:00:26Z" } 2023-10-11T16:00:26.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013798', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:39.856771Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006572', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:05.208654Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017839', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:05.111317Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016719', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:05.151723Z'}}] 2023-10-11T16:00:26.145 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013798', 1: '00000000000000000000:00000000000000006572', 2: '00000000000000000000:00000000000000017839', 3: '00000000000000000000:00000000000000016719'} 2023-10-11T16:00:26.145 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013798', 1: '00000000000000000000:00000000000000006572', 2: '00000000000000000000:00000000000000017839', 3: '00000000000000000000:00000000000000016719'} 2023-10-11T16:00:26.146 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-10-11T16:00:26.146 DEBUG:boto:path=/dghqvd-70/ 2023-10-11T16:00:26.146 DEBUG:boto:auth_path=/dghqvd-70/ 2023-10-11T16:00:26.147 DEBUG:boto:Method: HEAD 2023-10-11T16:00:26.147 DEBUG:boto:Path: /dghqvd-70/ 2023-10-11T16:00:26.148 DEBUG:boto:Data: 2023-10-11T16:00:26.148 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.149 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.149 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.149 DEBUG:boto:Params: {} 2023-10-11T16:00:26.150 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:00:26.151 DEBUG:boto:Token: None 2023-10-11T16:00:26.151 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/ 2023-10-11T16:00:26.152 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:BXLI3eXTo6lqcQbj0UCM2AP6u9Q= 2023-10-11T16:00:26.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BXLI3eXTo6lqcQbj0UCM2AP6u9Q=', 'Content-Length': '0'} 2023-10-11T16:00:26.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000774c633112d881c6-006526c69a-2688-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.157 DEBUG:boto:path=/dghqvd-70/obj7 2023-10-11T16:00:26.157 DEBUG:boto:auth_path=/dghqvd-70/obj7 2023-10-11T16:00:26.158 DEBUG:boto:Method: HEAD 2023-10-11T16:00:26.158 DEBUG:boto:Path: /dghqvd-70/obj7 2023-10-11T16:00:26.159 DEBUG:boto:Data: 2023-10-11T16:00:26.159 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.160 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.160 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.160 DEBUG:boto:Params: {} 2023-10-11T16:00:26.161 DEBUG:boto:Token: None 2023-10-11T16:00:26.161 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj7 2023-10-11T16:00:26.162 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:poKQMiuMayKq1SRbWWRoFjgudsQ= 2023-10-11T16:00:26.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:poKQMiuMayKq1SRbWWRoFjgudsQ=', 'Content-Length': '0'} 2023-10-11T16:00:26.164 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000023c853ea5ad9a7a3-006526c69a-2688-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.165 DEBUG:boto:path=/dghqvd-70/obj8 2023-10-11T16:00:26.165 DEBUG:boto:auth_path=/dghqvd-70/obj8 2023-10-11T16:00:26.165 DEBUG:boto:Method: HEAD 2023-10-11T16:00:26.166 DEBUG:boto:Path: /dghqvd-70/obj8 2023-10-11T16:00:26.166 DEBUG:boto:Data: 2023-10-11T16:00:26.167 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.167 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.168 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.168 DEBUG:boto:Params: {} 2023-10-11T16:00:26.169 DEBUG:boto:Token: None 2023-10-11T16:00:26.169 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj8 2023-10-11T16:00:26.169 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iaFTrSQkg12Nt+bGel17wCXerKQ= 2023-10-11T16:00:26.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iaFTrSQkg12Nt+bGel17wCXerKQ=', 'Content-Length': '0'} 2023-10-11T16:00:26.172 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008249285b18ce8c7c-006526c69a-2688-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.172 DEBUG:boto:path=/dghqvd-70/obj9 2023-10-11T16:00:26.173 DEBUG:boto:auth_path=/dghqvd-70/obj9 2023-10-11T16:00:26.173 DEBUG:boto:Method: HEAD 2023-10-11T16:00:26.174 DEBUG:boto:Path: /dghqvd-70/obj9 2023-10-11T16:00:26.174 DEBUG:boto:Data: 2023-10-11T16:00:26.174 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.175 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.175 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.175 DEBUG:boto:Params: {} 2023-10-11T16:00:26.176 DEBUG:boto:Token: None 2023-10-11T16:00:26.176 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj9 2023-10-11T16:00:26.177 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:VcqtgW9UtlmBnW7H3q2O3zrH5ZI= 2023-10-11T16:00:26.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VcqtgW9UtlmBnW7H3q2O3zrH5ZI=', 'Content-Length': '0'} 2023-10-11T16:00:26.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-71:daa316e9-869c-4074-a756-a8adfb4a3573.9311.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2ed22d8b1643483-006526c69a-2688-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.180 DEBUG:boto:path=/dghqvd-70/obj9 2023-10-11T16:00:26.180 DEBUG:boto:auth_path=/dghqvd-70/obj9 2023-10-11T16:00:26.181 DEBUG:boto:Method: GET 2023-10-11T16:00:26.182 DEBUG:boto:Path: /dghqvd-70/obj9 2023-10-11T16:00:26.182 DEBUG:boto:Data: 2023-10-11T16:00:26.183 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.183 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.183 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.184 DEBUG:boto:Params: {} 2023-10-11T16:00:26.184 DEBUG:boto:Token: None 2023-10-11T16:00:26.185 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj9 2023-10-11T16:00:26.185 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:phv8HOj41zTDu6BeTGPqB8rxO0Y= 2023-10-11T16:00:26.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:phv8HOj41zTDu6BeTGPqB8rxO0Y=', 'Content-Length': '0'} 2023-10-11T16:00:26.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-71:daa316e9-869c-4074-a756-a8adfb4a3573.9311.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a65ed88ea047363-006526c69a-2688-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.188 DEBUG:boto:path=/dghqvd-70/obj10 2023-10-11T16:00:26.189 DEBUG:boto:auth_path=/dghqvd-70/obj10 2023-10-11T16:00:26.189 DEBUG:boto:Method: HEAD 2023-10-11T16:00:26.190 DEBUG:boto:Path: /dghqvd-70/obj10 2023-10-11T16:00:26.190 DEBUG:boto:Data: 2023-10-11T16:00:26.190 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.191 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.191 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.192 DEBUG:boto:Params: {} 2023-10-11T16:00:26.192 DEBUG:boto:Token: None 2023-10-11T16:00:26.193 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj10 2023-10-11T16:00:26.193 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:WCg3H+RZCmw/+PMzXVfhCpGiIbc= 2023-10-11T16:00:26.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WCg3H+RZCmw/+PMzXVfhCpGiIbc=', 'Content-Length': '0'} 2023-10-11T16:00:26.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-71:daa316e9-869c-4074-a756-a8adfb4a3573.9311.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000774b9978ab650a96-006526c69a-2688-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.196 DEBUG:boto:path=/dghqvd-70/obj10 2023-10-11T16:00:26.196 DEBUG:boto:auth_path=/dghqvd-70/obj10 2023-10-11T16:00:26.197 DEBUG:boto:Method: GET 2023-10-11T16:00:26.197 DEBUG:boto:Path: /dghqvd-70/obj10 2023-10-11T16:00:26.198 DEBUG:boto:Data: 2023-10-11T16:00:26.198 DEBUG:boto:Headers: {} 2023-10-11T16:00:26.198 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:26.199 DEBUG:boto:Port: 8000 2023-10-11T16:00:26.199 DEBUG:boto:Params: {} 2023-10-11T16:00:26.200 DEBUG:boto:Token: None 2023-10-11T16:00:26.200 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:00:26 GMT /dghqvd-70/obj10 2023-10-11T16:00:26.201 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Aas4SNDwy66K2GWzKZwj/lij0Bk= 2023-10-11T16:00:26.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Aas4SNDwy66K2GWzKZwj/lij0Bk=', 'Content-Length': '0'} 2023-10-11T16:00:26.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '37b32c30-76c0-479d-b458-e4824d081f13:dghqvd-71:daa316e9-869c-4074-a756-a8adfb4a3573.9311.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e57271a2633e7c50-006526c69a-2688-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:26.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:26.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:26.205 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dghqvd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:26.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:26.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.310 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:00:26.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:26.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:26.313 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:26.363 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:26.364 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.410 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:00:26.411 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-10-11T16:00:26.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.414 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:26.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:26.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.493 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:00:26.506 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:26.507 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.507 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:26.507 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:00:26.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.508 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:26.508 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:00:26.508 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.509 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:26.509 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:00:26.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.509 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040026.290696_980.1", 2023-10-11T16:00:26.510 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:26.290696Z" 2023-10-11T16:00:26.510 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.510 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:26.515 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:26.516 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:00:26.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:26.517 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:26.567 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:26.568 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:26.596 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:26.594+0000 7f730427a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:26.597 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:26.594+0000 7f730427a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:26.609 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:26.610 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:26.610 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:26.610 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:26.610 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:26.611 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:26.611 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:26.611 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.611 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:26.612 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.612 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:26.612 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.612 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:26.613 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:26.613 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.613 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:26.613 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.613 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:26.614 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:26.614 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.614 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.615 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.615 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:26.615 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.615 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:26.615 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:26.616 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.616 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.616 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.616 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:26.617 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:26.617 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.617 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.618 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:26.618 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.618 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:26.618 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:26.619 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:26.620 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.620 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:26.620 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:26.620 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:26.621 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:26.621 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040004.982172_978.1", 2023-10-11T16:00:26.621 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:26.621 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:26.621 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:26.622 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:04.982172Z", 2023-10-11T16:00:26.622 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:26.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:26.623 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:26.624 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:26.624 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:26.624 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:26Z" 2023-10-11T16:00:26.624 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:26.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040004.982172_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:04.982172Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:26Z" } 2023-10-11T16:00:26.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040004.982172_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:04.982172Z', 'realm_epoch': 2}}] 2023-10-11T16:00:26.627 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:26.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:26.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040026.290696_980.1 target=1_1697040004.982172_978.1 2023-10-11T16:00:31.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:31.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:31.630 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:31.691 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:31.692 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:31.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:31.721+0000 7fd43c50a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:31.724 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:31.722+0000 7fd43c50a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:31.738 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:31.738 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:31.739 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:31.740 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:31.741 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:31.742 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:31.743 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:31.744 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:31.745 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:31.746 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:31.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:31.748 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040004.982172_978.1", 2023-10-11T16:00:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:31.749 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:04.982172Z", 2023-10-11T16:00:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:31.750 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:31.751 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:31.752 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:31Z" 2023-10-11T16:00:31.752 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:31.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040004.982172_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:04.982172Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:31Z" } 2023-10-11T16:00:31.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040004.982172_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:04.982172Z', 'realm_epoch': 2}}] 2023-10-11T16:00:31.754 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:31.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:31.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040026.290696_980.1 target=1_1697040004.982172_978.1 2023-10-11T16:00:36.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:36.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:36.760 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:36.818 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:36.850 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:36.847+0000 7f0679c2a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:36.851 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:36.848+0000 7f0679c2a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:36.864 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:36.865 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.866 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:36.867 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:36.868 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:36.869 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:36.870 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:36.871 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:36.872 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:36.873 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:36.874 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.875 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:36.875 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:36.875 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:36.875 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:36.876 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040004.982172_978.1", 2023-10-11T16:00:36.876 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:36.876 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:36.876 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:36.876 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:04.982172Z", 2023-10-11T16:00:36.877 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:36.877 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:36.877 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:36.877 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:36.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.878 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:36.878 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:36.878 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:36.878 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:36.879 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:36Z" 2023-10-11T16:00:36.879 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:36.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040004.982172_978.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:04.982172Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:36Z" } 2023-10-11T16:00:36.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040004.982172_978.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:04.982172Z', 'realm_epoch': 2}}] 2023-10-11T16:00:36.881 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:36.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040004.982172_978.1'} 2023-10-11T16:00:36.882 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040026.290696_980.1 target=1_1697040004.982172_978.1 2023-10-11T16:00:41.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:41.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:41.885 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:41.942 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:41.973 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:41.972+0000 7f5c0cfd6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:41.975 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:41.973+0000 7f5c0cfd6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:41.988 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:41.989 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:41.990 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:41.991 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:41.992 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:41.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:41.994 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:41.995 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:41.996 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:41.997 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:41.997 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:41.997 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:41.997 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:41.997 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:41.998 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:41.998 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:41.998 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:41.998 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:41.999 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:41.999 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:41.999 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:41.999 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040026.290696_980.1", 2023-10-11T16:00:41.999 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:42.000 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:42.000 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:42.000 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:26.290696Z", 2023-10-11T16:00:42.000 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.001 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.001 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.001 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:42.002 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:42.002 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:42.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.002 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:41Z" 2023-10-11T16:00:42.002 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:42.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040026.290696_980.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:26.290696Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:41Z" } 2023-10-11T16:00:42.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040026.290696_980.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:26.290696Z', 'realm_epoch': 2}}] 2023-10-11T16:00:42.004 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:42.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040026.290696_980.1'} 2023-10-11T16:00:42.005 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:00:42.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.007 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.054 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.055 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.099 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T16:00:42.099 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-72 2023-10-11T16:00:42.100 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:00:42.100 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:00:42.101 DEBUG:boto:Method: PUT 2023-10-11T16:00:42.101 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:00:42.101 DEBUG:boto:Data: 2023-10-11T16:00:42.102 DEBUG:boto:Headers: {} 2023-10-11T16:00:42.102 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:42.103 DEBUG:boto:Port: 8000 2023-10-11T16:00:42.103 DEBUG:boto:Params: {} 2023-10-11T16:00:42.104 DEBUG:boto:Token: None 2023-10-11T16:00:42.104 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 16:00:42 GMT /dghqvd-72/ 2023-10-11T16:00:42.105 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kTPZbhIvWGLcVibSMpl9pPJHHHU= 2023-10-11T16:00:42.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kTPZbhIvWGLcVibSMpl9pPJHHHU=', 'Content-Length': '0'} 2023-10-11T16:00:42.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a93de64b98c030c-006526c6aa-2688-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:42.121 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-73 2023-10-11T16:00:42.122 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:00:42.122 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:00:42.122 DEBUG:boto:Method: PUT 2023-10-11T16:00:42.123 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:00:42.123 DEBUG:boto:Data: 2023-10-11T16:00:42.124 DEBUG:boto:Headers: {} 2023-10-11T16:00:42.124 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:42.124 DEBUG:boto:Port: 8000 2023-10-11T16:00:42.125 DEBUG:boto:Params: {} 2023-10-11T16:00:42.125 DEBUG:boto:Token: None 2023-10-11T16:00:42.126 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 16:00:42 GMT /dghqvd-73/ 2023-10-11T16:00:42.126 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3nwbxAVZ6Hqpyy/QQ7RT83RF3JI= 2023-10-11T16:00:42.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:42 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3nwbxAVZ6Hqpyy/QQ7RT83RF3JI=', 'Content-Length': '0'} 2023-10-11T16:00:42.141 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d06275c89ebf0bb1-006526c6aa-2688-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:42 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:42.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:42.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:42.143 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:42.191 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.192 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.228 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:00:42.241 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:42.241 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.241 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:42.242 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:00:42.242 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.242 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.243 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:42.243 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:00:42.243 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.243 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.244 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:42.244 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:00:42.244 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.244 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.244 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:00:42.245 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:42.138525Z" 2023-10-11T16:00:42.245 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.245 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:42.246 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:42.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:00:42.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:42.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:42.248 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:42.298 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.299 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.328 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:42.327+0000 7f1dfc0d6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:42.329 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:42.328+0000 7f1dfc0d6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:42.344 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:42.345 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:42.345 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:42.345 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:42.345 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:42.346 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:42.346 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.346 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.346 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:42.347 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.347 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:42.347 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:42.347 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:42.347 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:42.348 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:42.348 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:42.348 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:42.348 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:42.349 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.349 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.349 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.349 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.349 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:42.350 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:42.350 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:42.350 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:42.350 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:42.351 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:42.351 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:42.351 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:42.351 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.352 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.352 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.352 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.352 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:42.352 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:42.353 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:42.353 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:42.353 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:42.353 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:42.353 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:42.354 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:42.354 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.354 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.354 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:42.355 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:00:42.356 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:42.356 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:42.356 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:42.356 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:00:42.357 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:42.357 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.357 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.357 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.357 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.358 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:42.358 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:42.358 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:42.358 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.359 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:42Z" 2023-10-11T16:00:42.359 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:42.360 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:42Z" } 2023-10-11T16:00:42.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:00:42.361 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:42.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:42.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:00:42.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.363 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.412 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.412 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.459 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:00:42.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.460 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.516 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.516 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.565 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:00:42.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-72', '--dest-bucket=dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-72', '--dest-bucket=dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.567 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dghqvd-72 --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-72 --dest-bucket=dghqvd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.621 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.666 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:00:42.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-73', '--dest-bucket=dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-73', '--dest-bucket=dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.668 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dghqvd-73 --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-73 --dest-bucket=dghqvd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.723 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.723 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.769 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:00:42.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.771 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.822 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.822 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.869 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:00:42.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:42.871 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:42.919 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:42.920 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:42.970 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:42.970 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:42.970 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 19, 2023-10-11T16:00:42.971 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:00:42.971 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T16:00:42.971 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T16:00:42.971 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:42.972 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T16:00:42.972 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.972 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T16:00:42.972 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T16:00:42.972 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:00:42.973 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T16:00:42.973 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:42.973 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:00:42.973 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:42.974 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:00:42.974 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:00:42.974 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:00:42.974 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:42.975 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.975 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:00:42.975 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T16:00:42.975 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:42.976 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:00:42.976 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:42.976 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:00:42.976 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:00:42.977 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:00:42.977 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:00:42.977 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:00:42.977 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:00:42.977 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:00:42.978 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:00:42.978 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:00:42.978 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:00:42.978 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:42.979 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.979 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.979 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:00:42.979 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T16:00:42.980 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:42.980 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:00:42.980 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:42.980 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:00:42.980 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:00:42.981 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:00:42.981 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:00:42.981 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:00:42.981 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:00:42.982 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:00:42.982 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:00:42.982 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:00:42.982 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:00:42.982 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:42.983 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.983 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.983 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:42.983 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:00:42.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.984 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:00:42.984 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:00:42.984 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:00:42.985 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:00:42.985 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.985 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.985 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:42.986 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:00:42.986 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:00:42.986 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:00:42.986 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T16:00:42.986 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.987 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T16:00:42.987 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T16:00:42.987 INFO:teuthology.orchestra.run.smithi158.stdout: "directional": [ 2023-10-11T16:00:42.987 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.987 INFO:teuthology.orchestra.run.smithi158.stdout: "source_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:00:42.988 INFO:teuthology.orchestra.run.smithi158.stdout: "dest_zone": "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:00:42.988 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.988 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.988 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.988 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T16:00:42.989 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.989 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-72", 2023-10-11T16:00:42.989 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:00:42.989 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-72", 2023-10-11T16:00:42.990 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:42.990 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:00:42.990 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.990 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.990 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:00:42.991 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-73", 2023-10-11T16:00:42.991 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:42.991 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:00:42.991 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.992 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:00:42.992 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:00:42.992 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:00:42.992 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:00:42.993 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.993 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.993 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:00:42.993 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:00:42.994 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:00:42.994 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:00:42.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.994 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.995 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:42.995 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-73", 2023-10-11T16:00:42.995 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:00:42.995 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-73", 2023-10-11T16:00:42.995 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:42.996 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:00:42.996 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.996 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.996 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:00:42.997 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-73", 2023-10-11T16:00:42.997 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:42.997 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:00:42.997 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:42.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.998 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:00:42.998 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:00:42.998 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:00:42.998 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:00:42.998 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:42.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:42.999 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:00:42.999 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:00:42.999 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:00:43.000 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:00:43.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.000 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.000 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.001 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T16:00:43.001 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.001 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.001 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.001 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:00:43.002 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:43.002 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.002 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.003 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T16:00:43.003 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T16:00:43.003 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:00:43.003 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T16:00:43.003 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:43.004 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:00:43.004 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.004 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:00:43.004 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:00:43.004 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:00:43.005 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:00:43.005 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.005 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:00:43.005 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T16:00:43.005 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:43.006 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:00:43.006 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.006 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:00:43.006 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:00:43.007 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:00:43.007 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:00:43.007 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:00:43.007 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:00:43.007 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:00:43.008 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:00:43.008 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:00:43.008 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:00:43.008 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:43.009 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.009 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.009 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.009 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:00:43.009 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T16:00:43.010 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:00:43.010 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:00:43.010 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.010 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:00:43.011 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:00:43.011 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:00:43.011 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:00:43.011 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:00:43.011 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:00:43.012 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:00:43.012 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:00:43.013 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:00:43.014 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:00:43.014 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:43.014 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.015 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.015 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:00:43.015 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.015 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:00:43.015 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:00:43.016 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:00:43.016 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:00:43.016 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.017 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.017 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:00:43.017 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:00:43.017 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:00:43.017 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T16:00:43.018 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.018 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:00:43.018 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:00:43.018 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.019 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:00:43.019 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T16:00:43.019 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:00:43.020 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T16:00:43.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.020 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:00:43.020 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T16:00:43.021 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.021 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:00:43.021 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T16:00:43.022 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.022 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.022 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:00:43.022 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T16:00:43.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.023 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.023 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T16:00:43.023 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:00:43.024 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T16:00:43.024 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T16:00:43.024 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:00:43.024 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:00:43.025 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:00:43.025 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:00:43.025 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:00:43.025 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.025 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T16:00:43.026 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:00:43.026 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:00:43.026 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:00:43.026 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:00:43.026 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:00:43.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.027 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T16:00:43.027 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:00:43.027 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:00:43.028 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:00:43.028 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:00:43.028 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:00:43.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.029 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T16:00:43.029 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:00:43.029 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:00:43.029 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:00:43.029 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:00:43.030 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:00:43.030 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.030 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T16:00:43.030 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:00:43.031 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:00:43.031 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:00:43.031 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:00:43.031 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:00:43.032 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.032 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:00:43.032 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T16:00:43.032 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.033 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:43.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:43.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:00:43.034 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:00:43.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:43.091 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:43.216 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:00:43.216 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:00:43.217 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:00:43.217 DEBUG:boto:Method: HEAD 2023-10-11T16:00:43.218 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:00:43.218 DEBUG:boto:Data: 2023-10-11T16:00:43.218 DEBUG:boto:Headers: {} 2023-10-11T16:00:43.219 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.219 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.220 DEBUG:boto:Params: {} 2023-10-11T16:00:43.220 DEBUG:boto:Token: None 2023-10-11T16:00:43.221 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-72/ 2023-10-11T16:00:43.221 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tF5gYLw2xQmltu9RvFqCWSOixt8= 2023-10-11T16:00:43.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tF5gYLw2xQmltu9RvFqCWSOixt8=', 'Content-Length': '0'} 2023-10-11T16:00:43.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f7e07a798747d9-006526c6ab-28ae-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.231 DEBUG:boto:path=/dghqvd-72/obj1 2023-10-11T16:00:43.231 DEBUG:boto:auth_path=/dghqvd-72/obj1 2023-10-11T16:00:43.232 DEBUG:boto:Method: PUT 2023-10-11T16:00:43.232 DEBUG:boto:Path: /dghqvd-72/obj1 2023-10-11T16:00:43.233 DEBUG:boto:Data: 2023-10-11T16:00:43.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:43.233 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.234 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.234 DEBUG:boto:Params: {} 2023-10-11T16:00:43.235 DEBUG:boto:Token: None 2023-10-11T16:00:43.235 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-72/obj1 2023-10-11T16:00:43.235 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5BuvL1zIDunYADaRW8vWenSrr0A= 2023-10-11T16:00:43.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5BuvL1zIDunYADaRW8vWenSrr0A='} 2023-10-11T16:00:43.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e24fbdf6395f6bd-006526c6ab-28ae-a1'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.279 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:00:43.279 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:00:43.280 DEBUG:boto:Method: HEAD 2023-10-11T16:00:43.280 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:00:43.281 DEBUG:boto:Data: 2023-10-11T16:00:43.281 DEBUG:boto:Headers: {} 2023-10-11T16:00:43.281 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.282 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.282 DEBUG:boto:Params: {} 2023-10-11T16:00:43.283 DEBUG:boto:Token: None 2023-10-11T16:00:43.283 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-72/ 2023-10-11T16:00:43.284 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:tF5gYLw2xQmltu9RvFqCWSOixt8= 2023-10-11T16:00:43.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tF5gYLw2xQmltu9RvFqCWSOixt8=', 'Content-Length': '0'} 2023-10-11T16:00:43.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000404073378d97fd0b-006526c6ab-28ae-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.289 DEBUG:boto:path=/dghqvd-72/obj2 2023-10-11T16:00:43.289 DEBUG:boto:auth_path=/dghqvd-72/obj2 2023-10-11T16:00:43.290 DEBUG:boto:Method: PUT 2023-10-11T16:00:43.290 DEBUG:boto:Path: /dghqvd-72/obj2 2023-10-11T16:00:43.291 DEBUG:boto:Data: 2023-10-11T16:00:43.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:43.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.292 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.292 DEBUG:boto:Params: {} 2023-10-11T16:00:43.293 DEBUG:boto:Token: None 2023-10-11T16:00:43.293 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-72/obj2 2023-10-11T16:00:43.294 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:jC3Kuj7YpyPNOx4+auUOzg1nCFo= 2023-10-11T16:00:43.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jC3Kuj7YpyPNOx4+auUOzg1nCFo='} 2023-10-11T16:00:43.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001efd8636b0f75256-006526c6ab-28ae-a1'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.309 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:00:43.309 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:00:43.310 DEBUG:boto:Method: HEAD 2023-10-11T16:00:43.310 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:00:43.311 DEBUG:boto:Data: 2023-10-11T16:00:43.311 DEBUG:boto:Headers: {} 2023-10-11T16:00:43.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.312 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.312 DEBUG:boto:Params: {} 2023-10-11T16:00:43.313 DEBUG:boto:Token: None 2023-10-11T16:00:43.313 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-73/ 2023-10-11T16:00:43.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wJd2hF7/L+SwKzTlZitMVf/ZdqY= 2023-10-11T16:00:43.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wJd2hF7/L+SwKzTlZitMVf/ZdqY=', 'Content-Length': '0'} 2023-10-11T16:00:43.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000527f0e36dd019c1d-006526c6ab-28ae-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.321 DEBUG:boto:path=/dghqvd-73/obj3 2023-10-11T16:00:43.321 DEBUG:boto:auth_path=/dghqvd-73/obj3 2023-10-11T16:00:43.322 DEBUG:boto:Method: PUT 2023-10-11T16:00:43.322 DEBUG:boto:Path: /dghqvd-73/obj3 2023-10-11T16:00:43.323 DEBUG:boto:Data: 2023-10-11T16:00:43.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:43.324 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.324 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.324 DEBUG:boto:Params: {} 2023-10-11T16:00:43.325 DEBUG:boto:Token: None 2023-10-11T16:00:43.325 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-73/obj3 2023-10-11T16:00:43.326 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iB05BiEpg+MGdwbPaNLItUEe7ko= 2023-10-11T16:00:43.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iB05BiEpg+MGdwbPaNLItUEe7ko='} 2023-10-11T16:00:43.371 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9c43bb99d4c3d81-006526c6ab-28ae-a1'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.371 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:00:43.371 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:00:43.372 DEBUG:boto:Method: HEAD 2023-10-11T16:00:43.372 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:00:43.373 DEBUG:boto:Data: 2023-10-11T16:00:43.373 DEBUG:boto:Headers: {} 2023-10-11T16:00:43.374 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.374 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.375 DEBUG:boto:Params: {} 2023-10-11T16:00:43.375 DEBUG:boto:Token: None 2023-10-11T16:00:43.376 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-73/ 2023-10-11T16:00:43.376 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wJd2hF7/L+SwKzTlZitMVf/ZdqY= 2023-10-11T16:00:43.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wJd2hF7/L+SwKzTlZitMVf/ZdqY=', 'Content-Length': '0'} 2023-10-11T16:00:43.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d34173dd0f38fe10-006526c6ab-28ae-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.381 DEBUG:boto:path=/dghqvd-73/obj4 2023-10-11T16:00:43.382 DEBUG:boto:auth_path=/dghqvd-73/obj4 2023-10-11T16:00:43.382 DEBUG:boto:Method: PUT 2023-10-11T16:00:43.382 DEBUG:boto:Path: /dghqvd-73/obj4 2023-10-11T16:00:43.383 DEBUG:boto:Data: 2023-10-11T16:00:43.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:00:43.384 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:00:43.384 DEBUG:boto:Port: 8000 2023-10-11T16:00:43.385 DEBUG:boto:Params: {} 2023-10-11T16:00:43.385 DEBUG:boto:Token: None 2023-10-11T16:00:43.386 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:00:43 GMT /dghqvd-73/obj4 2023-10-11T16:00:43.386 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:IyMr5VSY3Y1lUkoLN9gsGppMAd8= 2023-10-11T16:00:43.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:00:43 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IyMr5VSY3Y1lUkoLN9gsGppMAd8='} 2023-10-11T16:00:43.405 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000566d08dd470f47c6-006526c6ab-28ae-a1'), ('Date', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:00:43.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.406 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:43.484 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:43.485 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:00:43.582 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:43.583 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.583 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:43.583 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:00:43.583 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.584 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.584 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:43.584 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:00:43.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.584 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.585 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:43.585 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:00:43.585 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.585 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.585 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:00:43.586 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:42.138525Z" 2023-10-11T16:00:43.586 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.586 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:43.587 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:43.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:00:43.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.589 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:43.655 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:43.656 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:43.700 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:43.698+0000 7fc67cc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:43.702 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:00:43.701+0000 7fc67cc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:00:43.721 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:43.721 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:43.721 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:43.721 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:43.722 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:43.722 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:00:43.722 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.722 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.723 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:43.723 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.723 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:43.723 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.723 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:00:43.724 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.725 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.725 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.725 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.725 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:43.725 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.726 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:43.726 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:00:43.726 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.726 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:43.730 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.730 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:00:43.730 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.730 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.731 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.731 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.731 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:43.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.732 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:43.732 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:00:43.732 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.732 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:43.732 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.733 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:00:43.733 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.733 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.733 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.733 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.734 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:43.734 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.734 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:00:43.734 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:00:43.735 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.735 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:43.735 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.735 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:00:43.735 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:00:43.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.736 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.736 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.736 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.736 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:43.737 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:00:43.737 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:00:43.737 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.737 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:43Z" 2023-10-11T16:00:43.738 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:43.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:00:43Z" } 2023-10-11T16:00:43.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:00:43.740 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:43.740 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:00:43.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:00:43.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.742 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:43.790 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:43.791 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:43.839 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:00:43.839 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.839 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:00:43.840 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:00:43.840 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.840 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.840 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:00:43.840 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:00:43.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.841 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.841 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:00:43.841 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:00:43.841 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.842 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.842 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:00:43.842 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:00:43.842 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.843 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:00:43.844 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:43.844 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:00:43.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:43.845 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:43.896 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:43.897 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:43.940 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:43.940 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:43.941 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:43.941 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:43.941 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:43.941 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:00:43.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.942 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:43.942 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.942 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:43.943 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.943 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:43.943 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:00:43.943 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.943 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:43.944 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.944 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:00:43.944 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.945 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.945 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:43.945 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.945 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:43.945 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:00:43.946 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.946 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:43.946 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.946 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:00:43.946 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.947 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.947 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.947 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:43.947 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.947 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:43.948 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:00:43.948 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.948 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:00:43.948 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.949 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:00:43.949 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.949 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.949 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:43.949 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:43.950 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:43.950 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:43.950 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:00:43.950 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:43.951 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:00:43.951 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:43.951 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:00:43.951 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.952 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:43.952 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:43.952 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.952 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:43.952 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:00:43.953 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:00:43.953 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:43.953 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:43Z" 2023-10-11T16:00:43.953 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:43.954 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:00:43Z" } 2023-10-11T16:00:43.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:00:43.955 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:43.956 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:43.958 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:00:48.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:49.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:49.145 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:49.205 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:49.205 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:49.253 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:49.254 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:49.254 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:49.254 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:49.255 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:49.255 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:00:49.255 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.255 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:49.256 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:49.256 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:49.256 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:49.256 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:49.256 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:00:49.257 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:49.257 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:49.257 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:49.257 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:00:49.257 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:49.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.258 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:49.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:49.258 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:49.259 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:49.259 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:00:49.259 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:49.259 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:49.259 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:49.260 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:00:49.260 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:49.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.260 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:49.260 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:49.261 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:49.261 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:49.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:00:49.261 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:49.261 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:00:49.262 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:49.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:00:49.262 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:49.262 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:49.263 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:00:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:00:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:00:49.264 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:49.265 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:00:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:00:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:49.266 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:49Z" 2023-10-11T16:00:49.266 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:49.270 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:00:49Z" } 2023-10-11T16:00:49.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:00:49.271 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:49.271 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:49.271 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:00:54.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:54.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:54.274 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:54.333 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:54.334 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:54.384 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:54.384 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:54.384 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:54.385 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:54.385 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:54.385 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:00:54.385 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.386 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:54.386 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:54.387 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:54.387 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:54.387 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:54.387 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:00:54.387 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:54.388 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:54.388 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:54.388 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:00:54.388 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:54.389 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:00:54.390 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:54.390 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:54.390 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:54.390 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:00:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:54.391 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:00:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:00:54.392 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:00:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:54.393 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:54.394 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:00:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:00:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:54.395 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:00:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:54.396 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:00:54.397 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:00:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:54.398 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:54Z" 2023-10-11T16:00:54.398 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:54.399 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:00:54Z" } 2023-10-11T16:00:54.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:00:54.400 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:54.400 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:54.401 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:00:59.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:59.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:00:59.403 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:00:59.462 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:00:59.463 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:00:59.512 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:00:59.512 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:00:59.513 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:00:59.513 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:00:59.513 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:00:59.513 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:00:59.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.514 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:00:59.514 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:59.514 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:00:59.514 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:59.515 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:59.515 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:00:59.515 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:59.515 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:59.515 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:59.516 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:00:59.516 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:59.516 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.516 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:59.516 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:00:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:00:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:59.517 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:00:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:00:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:59.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:00:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:59.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:00:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:00:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:00:59.520 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:00:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:00:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:00:59.521 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:00:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:00:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:00:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:00:59.522 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:00:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:00:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:00:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:00:59.523 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:00:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:00:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:00:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:00:59.524 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:00:59Z" 2023-10-11T16:00:59.525 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:00:59.526 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:00:59Z" } 2023-10-11T16:00:59.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:00:59.527 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:59.527 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:00:59.528 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:04.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:04.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:04.530 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:04.589 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:04.589 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:04.637 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:04.637 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:04.637 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:04.637 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:04.638 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:04.638 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:04.638 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.638 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:04.639 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:04.639 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:04.639 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:04.639 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:04.639 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:04.640 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:04.640 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:04.640 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:04.640 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:04.640 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:04.641 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.641 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:04.641 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:04.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:04.641 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:04.642 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:04.642 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:04.642 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:04.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:04.643 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:04.643 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:04.643 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.643 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:04.643 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:04.644 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:04.644 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:04.644 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:04.644 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:04.644 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.645 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:04.646 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:04.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:04.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:04.649 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:04Z" 2023-10-11T16:01:04.649 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:04.651 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:04Z" } 2023-10-11T16:01:04.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:04.652 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:04.652 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:04.652 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:09.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:09.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:09.655 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:09.714 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:09.715 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:09.762 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:09.763 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:09.763 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:09.763 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:09.763 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:09.764 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:09.764 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.764 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:09.764 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:09.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:09.765 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:09.765 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:09.765 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:09.765 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:09.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:09.766 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:09.766 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:09.766 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:09.766 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.767 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:09.767 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:09.767 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:09.767 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:09.767 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:09.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.769 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:09.769 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:09.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:09.769 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:09.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:09.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:09.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:09.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:09Z" 2023-10-11T16:01:09.775 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:09.776 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:09Z" } 2023-10-11T16:01:09.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:09.777 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:09.777 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:09.777 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:14.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:14.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:14.779 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:14.839 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:14.839 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:14.890 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:14.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:14.891 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:14.891 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:14.891 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:14.891 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:14.892 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.892 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:14.892 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:14.892 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:14.893 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:14.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:14.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:14.897 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:14Z" 2023-10-11T16:01:14.903 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:14.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:14Z" } 2023-10-11T16:01:14.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:14.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:14.905 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:14.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:19.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:19.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:19.908 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:19.964 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:19.965 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:20.014 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:20.014 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:20.015 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:20.016 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:20.017 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.018 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:20.019 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.020 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:19Z" 2023-10-11T16:01:20.028 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:20.028 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:19Z" } 2023-10-11T16:01:20.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:20.030 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:20.031 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:20.031 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:25.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:25.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:25.035 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:25.094 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:25.094 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:25.142 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:25.143 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:25.143 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:25.143 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:25.143 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:25.144 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:25.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.144 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:25.144 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:25.144 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:25.145 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:25.145 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:25.145 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:25.145 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:25.145 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:25Z" 2023-10-11T16:01:25.155 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:25.156 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:25Z" } 2023-10-11T16:01:25.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:25.157 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:25.158 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:25.158 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:30.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:30.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:30.161 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:30.222 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:30.222 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:30.273 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:30.273 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:30Z" 2023-10-11T16:01:30.286 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:30.287 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:30Z" } 2023-10-11T16:01:30.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:30.288 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:30.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:30.289 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:35.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:35.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:35.292 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:35.351 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:35.399 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:35.399 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:35Z" 2023-10-11T16:01:35.413 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:35.414 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:35Z" } 2023-10-11T16:01:35.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:35.415 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:35.415 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:35.415 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:40.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:40.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:40.418 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:40.478 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:40.478 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:40.527 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:40.535 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:40.535 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:40.535 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:40.535 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:40.536 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:40.536 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:40.536 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.536 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:40.536 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:40.537 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:40.538 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:40.539 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:40.540 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:40Z" 2023-10-11T16:01:40.540 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:40.541 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:40Z" } 2023-10-11T16:01:40.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:40.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:40.543 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:40.543 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:45.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:45.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:45.546 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:45.606 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:45.607 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:45.656 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:45.657 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:45.657 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:45.657 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:45.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.657 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:45.658 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:45.658 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:45.658 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:45.658 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:45.659 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:45.659 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:45.659 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:45.659 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:45.659 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:45.661 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:45.661 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:45.661 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:45.661 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:45.661 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:45.662 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:45.663 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:45.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:45.665 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:45.666 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:45.667 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:45.668 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:45Z" 2023-10-11T16:01:45.668 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:45.669 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:45Z" } 2023-10-11T16:01:45.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:45.671 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:45.671 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:45.671 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:50.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:50.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:50.673 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:50.732 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:50.733 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:50.782 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:50.782 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.783 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:50.784 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:50.785 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:50.786 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:50.786 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.786 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:50.786 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:50.786 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:50.787 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:50.788 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:50.789 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:50.790 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:50.791 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:50.792 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:50.793 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:50.794 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:50.795 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:50Z" 2023-10-11T16:01:50.795 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:50.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:50Z" } 2023-10-11T16:01:50.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:50.797 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:50.797 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:50.798 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:01:55.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:55.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:01:55.800 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:01:55.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:01:55.862 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:01:55.910 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:01:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:01:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:01:55.911 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:01:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:01:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:01:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:01:55.912 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:01:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:01:55.913 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:01:55.914 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:01:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:55.915 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:01:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:01:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:55.916 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:01:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:55.917 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:01:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:01:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:55.918 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:01:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:01:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.919 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:01:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:01:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:01:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:01:55.920 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:01:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:01:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:01:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:01:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:01:55.921 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:01:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:01:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:01:55.922 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:01:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:01:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:01:55.923 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:01:55Z" 2023-10-11T16:01:55.923 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:01:55.924 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:01:55Z" } 2023-10-11T16:01:55.925 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:01:55.925 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:55.926 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:01:55.926 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:00.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:00.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:00.928 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:00.987 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:00.987 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:01.037 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:01.037 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:01.038 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:01.039 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:01.040 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:01.041 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:01.042 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:01.043 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:01.044 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.045 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:01.046 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:01.047 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:01.048 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:01.049 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:01.049 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:01.049 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:01.050 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:01.050 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:01.050 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:01.051 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:01.051 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:01.052 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:01.052 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:01.053 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.053 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:01.053 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:01.054 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:01.054 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:01.054 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:01Z" 2023-10-11T16:02:01.055 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:01.057 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:01Z" } 2023-10-11T16:02:01.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:01.058 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:01.059 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:01.059 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:06.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:06.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:06.061 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:06.169 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:06.170 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:06.171 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:06.171 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:06.171 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.171 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:06.171 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:06.172 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:06.172 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:06.172 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:06.172 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:06.172 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:06.173 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:06.173 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:06.173 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:06.173 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:06.174 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.174 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:06.174 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:06.174 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:06.174 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:06.175 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:06.175 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:06.175 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:06.175 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:06.176 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:06.176 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:06.176 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.176 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:06.176 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:06.177 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:06.177 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:06.177 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:06.177 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:06.177 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:06.178 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:06.178 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:06.178 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:06.178 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.179 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:06.179 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:06.179 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:06.179 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:06.179 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:06.180 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:06.180 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:06.180 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:06.180 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:06.180 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:06.181 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:06.181 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:06.181 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.181 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:06.181 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:06.182 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:06.182 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:06.182 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:06Z" 2023-10-11T16:02:06.182 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:06.183 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:06Z" } 2023-10-11T16:02:06.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:06.184 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:06.185 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:06.185 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:11.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:11.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:11.187 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:11.246 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:11.247 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:11.296 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:11.296 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.297 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:11.298 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:11.299 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:11.300 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:11.301 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.302 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:11.303 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:11.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.305 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:11.305 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:11.305 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:11.305 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:11.305 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:11.306 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:11.306 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:11.306 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:11.306 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:11.307 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:11.308 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:11.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:11.308 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:11Z" 2023-10-11T16:02:11.308 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:11.309 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:11Z" } 2023-10-11T16:02:11.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:11.310 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:11.311 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:11.311 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:16.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:16.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:16.314 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:16.424 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:16.424 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:16.425 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:16.426 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:16.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.428 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:16.429 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:16.430 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:16.431 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:16.432 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:16.433 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:16.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:16.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.433 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:16.434 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:16.434 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:16.434 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:16.434 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:16.435 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:16.435 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:16.435 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:16.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:16.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:16.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:16.436 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:16.437 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.437 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:16.437 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:16.437 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:16.437 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:16.438 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:16Z" 2023-10-11T16:02:16.438 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:16.439 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:16Z" } 2023-10-11T16:02:16.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:16.440 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:16.441 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:16.441 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:21.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:21.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:21.444 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:21.504 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:21.505 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:21.553 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:21.554 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:21.555 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:21.556 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.557 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:21.558 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:21.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.560 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:21.560 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:21.560 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:21.560 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:21.560 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:21.561 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:21.561 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:21.561 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:21.561 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:21.562 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:21.562 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.562 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:21.562 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:21.562 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:21.563 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:21.563 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:21.563 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:21.563 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:21.563 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:21.564 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:21.564 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:21.564 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:21.564 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:21.564 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.565 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:21.565 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:21.565 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:21.565 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:21.565 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:21Z" 2023-10-11T16:02:21.566 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:21.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:21Z" } 2023-10-11T16:02:21.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:21.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:21.568 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:21.568 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:26.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:26.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:26.571 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:26.632 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:26.633 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:26.681 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:26.681 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:26.682 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:26.683 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:26.684 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:26.685 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:26.686 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.687 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:26.688 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:26.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:26.690 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:26.691 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:26.692 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:26.692 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:26.692 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:26.692 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.692 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:26.693 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:26.693 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:26.693 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:26.693 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:26Z" 2023-10-11T16:02:26.694 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:26.695 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:26Z" } 2023-10-11T16:02:26.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:26.696 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:26.696 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:26.696 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:31.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:31.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:31.699 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:31.758 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:31.759 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:31.811 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:31.811 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.812 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:31.813 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:31.814 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:31.815 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:31.816 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:31.817 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:31.818 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:31.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:31.820 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:31.821 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.822 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:31.823 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:31Z" 2023-10-11T16:02:31.823 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:31.824 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:31Z" } 2023-10-11T16:02:31.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:31.825 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:31.826 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:31.826 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:36.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:36.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:36.829 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:36.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:36.888 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:36.936 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:36.937 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:36.938 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.938 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:36.938 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:36.938 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:36.939 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:36.940 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:36.941 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:36.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:36.943 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:36.944 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:36.945 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:36.945 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.945 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:36.945 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:36.945 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:36.946 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:36.947 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:36.948 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:36.949 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:36Z" 2023-10-11T16:02:36.949 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:36.950 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:36Z" } 2023-10-11T16:02:36.951 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:36.951 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:36.951 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:36.952 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:41.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:41.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:41.955 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:42.013 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:42.014 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:42.063 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:42.063 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.064 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:42.065 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-10-11T16:02:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.619973Z" 2023-10-11T16:02:42.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:42.067 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032344", 2023-10-11T16:02:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:42.068 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842163Z" 2023-10-11T16:02:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:42.069 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051530", 2023-10-11T16:02:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:42.070 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:59:22.599357Z" 2023-10-11T16:02:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:42.071 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:42.072 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:42.073 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:42.074 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:42.074 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:42.074 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.074 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:42.075 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:42Z" 2023-10-11T16:02:42.075 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:42.076 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:59:22.619973Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032344", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:19.842163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051530", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T15:59:22.599357Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:42Z" } 2023-10-11T16:02:42.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.619973Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032344', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051530', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:59:22.599357Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:42.078 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:42.078 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045797', 1: '00000000000000000000:00000000000000032344', 2: '00000000000000000000:00000000000000051530', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:42.078 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797, shard=1 master=00000000000000000000:00000000000000032484 target=00000000000000000000:00000000000000032344, shard=2 master=00000000000000000000:00000000000000051810 target=00000000000000000000:00000000000000051530 2023-10-11T16:02:47.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:47.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:47.081 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:47.140 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.141 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.189 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:47.190 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:02:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:47.191 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:02:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:47.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:02:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.193 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:47.194 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:02:47.195 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:47.195 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:47.195 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:47.196 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:02:47.196 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:47.196 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.196 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:47.196 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:47.197 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:47.197 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:47.197 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:02:47.197 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:02:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:02:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:47.198 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:02:47.199 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:02:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:02:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:47.200 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:02:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.201 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:02:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:02:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:47.202 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:47Z" 2023-10-11T16:02:47.203 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:47.204 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.306165Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.400452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:00:43.337900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:02:47Z" } 2023-10-11T16:02:47.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.306165Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.400452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.337900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:02:47.205 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:47.205 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:02:47.205 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:02:47.206 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:02:47.206 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:02:47.207 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.207 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:02:47.208 DEBUG:boto:Data: 2023-10-11T16:02:47.208 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.209 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.209 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.209 DEBUG:boto:Params: {} 2023-10-11T16:02:47.210 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:02:47.211 DEBUG:boto:Token: None 2023-10-11T16:02:47.211 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-72/ 2023-10-11T16:02:47.212 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ajC+R5zSIjEuwjEU2VRzUHD2kqk= 2023-10-11T16:02:47.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ajC+R5zSIjEuwjEU2VRzUHD2kqk=', 'Content-Length': '0'} 2023-10-11T16:02:47.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003231d38081a29b6d-006526c727-28b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.220 DEBUG:boto:path=/dghqvd-72/obj1 2023-10-11T16:02:47.220 DEBUG:boto:auth_path=/dghqvd-72/obj1 2023-10-11T16:02:47.220 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.221 DEBUG:boto:Path: /dghqvd-72/obj1 2023-10-11T16:02:47.221 DEBUG:boto:Data: 2023-10-11T16:02:47.222 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.222 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.223 DEBUG:boto:Params: {} 2023-10-11T16:02:47.223 DEBUG:boto:Token: None 2023-10-11T16:02:47.224 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-72/obj1 2023-10-11T16:02:47.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:iC+t5w5ix1/X2mRb1gYzrHx3aBQ= 2023-10-11T16:02:47.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iC+t5w5ix1/X2mRb1gYzrHx3aBQ=', 'Content-Length': '0'} 2023-10-11T16:02:47.227 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b1904e15abae5609-006526c727-28b2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.227 DEBUG:boto:path=/dghqvd-72/obj2 2023-10-11T16:02:47.228 DEBUG:boto:auth_path=/dghqvd-72/obj2 2023-10-11T16:02:47.228 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.228 DEBUG:boto:Path: /dghqvd-72/obj2 2023-10-11T16:02:47.229 DEBUG:boto:Data: 2023-10-11T16:02:47.229 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.230 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.230 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.231 DEBUG:boto:Params: {} 2023-10-11T16:02:47.231 DEBUG:boto:Token: None 2023-10-11T16:02:47.232 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-72/obj2 2023-10-11T16:02:47.232 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:HuVpQRqgODKBL5G297keuqkuadY= 2023-10-11T16:02:47.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HuVpQRqgODKBL5G297keuqkuadY=', 'Content-Length': '0'} 2023-10-11T16:02:47.234 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004672c88219c9ca6f-006526c727-28b2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.235 DEBUG:boto:path=/dghqvd-72/obj3 2023-10-11T16:02:47.235 DEBUG:boto:auth_path=/dghqvd-72/obj3 2023-10-11T16:02:47.235 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.236 DEBUG:boto:Path: /dghqvd-72/obj3 2023-10-11T16:02:47.236 DEBUG:boto:Data: 2023-10-11T16:02:47.237 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.237 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.238 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.238 DEBUG:boto:Params: {} 2023-10-11T16:02:47.239 DEBUG:boto:Token: None 2023-10-11T16:02:47.239 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-72/obj3 2023-10-11T16:02:47.239 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PLCXtv9DvmQ/qDBrDm+tqB75Vhg= 2023-10-11T16:02:47.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PLCXtv9DvmQ/qDBrDm+tqB75Vhg=', 'Content-Length': '0'} 2023-10-11T16:02:47.241 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002a62bc52c6e9561e-006526c727-28b2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.242 DEBUG:boto:path=/dghqvd-72/obj4 2023-10-11T16:02:47.242 DEBUG:boto:auth_path=/dghqvd-72/obj4 2023-10-11T16:02:47.242 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.243 DEBUG:boto:Path: /dghqvd-72/obj4 2023-10-11T16:02:47.243 DEBUG:boto:Data: 2023-10-11T16:02:47.244 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.244 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.244 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.245 DEBUG:boto:Params: {} 2023-10-11T16:02:47.245 DEBUG:boto:Token: None 2023-10-11T16:02:47.246 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-72/obj4 2023-10-11T16:02:47.246 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:zoA8Ag1qS0h9UP+Uj42Cz1AzHwk= 2023-10-11T16:02:47.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zoA8Ag1qS0h9UP+Uj42Cz1AzHwk=', 'Content-Length': '0'} 2023-10-11T16:02:47.248 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000f59ffe07df429ca-006526c727-28b2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.249 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:02:47.249 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:02:47.250 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.250 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:02:47.251 DEBUG:boto:Data: 2023-10-11T16:02:47.251 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.251 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.252 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.252 DEBUG:boto:Params: {} 2023-10-11T16:02:47.253 DEBUG:boto:Token: None 2023-10-11T16:02:47.253 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/ 2023-10-11T16:02:47.253 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:e8Wh7hayyqKEQRJLBzKwy1BTplk= 2023-10-11T16:02:47.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e8Wh7hayyqKEQRJLBzKwy1BTplk=', 'Content-Length': '0'} 2023-10-11T16:02:47.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e085e5caeb4ec478-006526c727-28b2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.257 DEBUG:boto:path=/dghqvd-73/obj1 2023-10-11T16:02:47.257 DEBUG:boto:auth_path=/dghqvd-73/obj1 2023-10-11T16:02:47.258 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.258 DEBUG:boto:Path: /dghqvd-73/obj1 2023-10-11T16:02:47.259 DEBUG:boto:Data: 2023-10-11T16:02:47.259 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.260 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.260 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.261 DEBUG:boto:Params: {} 2023-10-11T16:02:47.261 DEBUG:boto:Token: None 2023-10-11T16:02:47.261 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj1 2023-10-11T16:02:47.262 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:d+PBD4f2ii8KFh9PbqVK8+BwGsQ= 2023-10-11T16:02:47.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:d+PBD4f2ii8KFh9PbqVK8+BwGsQ=', 'Content-Length': '0'} 2023-10-11T16:02:47.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-72:daa316e9-869c-4074-a756-a8adfb4a3573.9868.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5cf9cb6cb37b994-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.265 DEBUG:boto:path=/dghqvd-73/obj1 2023-10-11T16:02:47.265 DEBUG:boto:auth_path=/dghqvd-73/obj1 2023-10-11T16:02:47.266 DEBUG:boto:Method: GET 2023-10-11T16:02:47.266 DEBUG:boto:Path: /dghqvd-73/obj1 2023-10-11T16:02:47.267 DEBUG:boto:Data: 2023-10-11T16:02:47.267 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.268 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.269 DEBUG:boto:Params: {} 2023-10-11T16:02:47.269 DEBUG:boto:Token: None 2023-10-11T16:02:47.270 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj1 2023-10-11T16:02:47.270 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:pv94ieu9R/uhTFcYlyRhP065u6s= 2023-10-11T16:02:47.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pv94ieu9R/uhTFcYlyRhP065u6s=', 'Content-Length': '0'} 2023-10-11T16:02:47.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-72:daa316e9-869c-4074-a756-a8adfb4a3573.9868.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072ca44ee2fbf74c8-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.273 DEBUG:boto:path=/dghqvd-73/obj2 2023-10-11T16:02:47.273 DEBUG:boto:auth_path=/dghqvd-73/obj2 2023-10-11T16:02:47.274 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.274 DEBUG:boto:Path: /dghqvd-73/obj2 2023-10-11T16:02:47.275 DEBUG:boto:Data: 2023-10-11T16:02:47.275 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.275 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.276 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.276 DEBUG:boto:Params: {} 2023-10-11T16:02:47.277 DEBUG:boto:Token: None 2023-10-11T16:02:47.277 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj2 2023-10-11T16:02:47.278 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bljS6nCT8s9/fBZur0HMHjeRDQg= 2023-10-11T16:02:47.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bljS6nCT8s9/fBZur0HMHjeRDQg=', 'Content-Length': '0'} 2023-10-11T16:02:47.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-72:daa316e9-869c-4074-a756-a8adfb4a3573.9868.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dd74f02adad7b41-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.280 DEBUG:boto:path=/dghqvd-73/obj2 2023-10-11T16:02:47.281 DEBUG:boto:auth_path=/dghqvd-73/obj2 2023-10-11T16:02:47.281 DEBUG:boto:Method: GET 2023-10-11T16:02:47.281 DEBUG:boto:Path: /dghqvd-73/obj2 2023-10-11T16:02:47.282 DEBUG:boto:Data: 2023-10-11T16:02:47.282 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.283 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.283 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.284 DEBUG:boto:Params: {} 2023-10-11T16:02:47.284 DEBUG:boto:Token: None 2023-10-11T16:02:47.285 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj2 2023-10-11T16:02:47.285 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:bqm8UzLqxe/XcO0XKeoljiFcHv0= 2023-10-11T16:02:47.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bqm8UzLqxe/XcO0XKeoljiFcHv0=', 'Content-Length': '0'} 2023-10-11T16:02:47.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-72:daa316e9-869c-4074-a756-a8adfb4a3573.9868.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000618b8c299b116ccb-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.288 DEBUG:boto:path=/dghqvd-73/obj3 2023-10-11T16:02:47.289 DEBUG:boto:auth_path=/dghqvd-73/obj3 2023-10-11T16:02:47.289 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.289 DEBUG:boto:Path: /dghqvd-73/obj3 2023-10-11T16:02:47.290 DEBUG:boto:Data: 2023-10-11T16:02:47.290 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.291 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.291 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.291 DEBUG:boto:Params: {} 2023-10-11T16:02:47.292 DEBUG:boto:Token: None 2023-10-11T16:02:47.292 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj3 2023-10-11T16:02:47.293 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:UZLDup5fR4DUU3W/OqwIUCh2lzw= 2023-10-11T16:02:47.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UZLDup5fR4DUU3W/OqwIUCh2lzw=', 'Content-Length': '0'} 2023-10-11T16:02:47.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e35b3f1369d0622-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.295 DEBUG:boto:path=/dghqvd-73/obj3 2023-10-11T16:02:47.296 DEBUG:boto:auth_path=/dghqvd-73/obj3 2023-10-11T16:02:47.296 DEBUG:boto:Method: GET 2023-10-11T16:02:47.297 DEBUG:boto:Path: /dghqvd-73/obj3 2023-10-11T16:02:47.297 DEBUG:boto:Data: 2023-10-11T16:02:47.298 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.299 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.299 DEBUG:boto:Params: {} 2023-10-11T16:02:47.300 DEBUG:boto:Token: None 2023-10-11T16:02:47.300 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj3 2023-10-11T16:02:47.300 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PN/aGh9dJf9ayv2zA5TnEbFvBuI= 2023-10-11T16:02:47.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PN/aGh9dJf9ayv2zA5TnEbFvBuI=', 'Content-Length': '0'} 2023-10-11T16:02:47.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e44453387259b41c-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.350 DEBUG:boto:path=/dghqvd-73/obj4 2023-10-11T16:02:47.351 DEBUG:boto:auth_path=/dghqvd-73/obj4 2023-10-11T16:02:47.351 DEBUG:boto:Method: HEAD 2023-10-11T16:02:47.352 DEBUG:boto:Path: /dghqvd-73/obj4 2023-10-11T16:02:47.352 DEBUG:boto:Data: 2023-10-11T16:02:47.353 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.353 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.353 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.354 DEBUG:boto:Params: {} 2023-10-11T16:02:47.354 DEBUG:boto:Token: None 2023-10-11T16:02:47.355 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj4 2023-10-11T16:02:47.355 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ji5wmvh6qW2kZI8JvU+BVijmKE0= 2023-10-11T16:02:47.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ji5wmvh6qW2kZI8JvU+BVijmKE0=', 'Content-Length': '0'} 2023-10-11T16:02:47.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acae2acc0a418148-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.358 DEBUG:boto:path=/dghqvd-73/obj4 2023-10-11T16:02:47.359 DEBUG:boto:auth_path=/dghqvd-73/obj4 2023-10-11T16:02:47.359 DEBUG:boto:Method: GET 2023-10-11T16:02:47.360 DEBUG:boto:Path: /dghqvd-73/obj4 2023-10-11T16:02:47.360 DEBUG:boto:Data: 2023-10-11T16:02:47.361 DEBUG:boto:Headers: {} 2023-10-11T16:02:47.361 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:02:47.361 DEBUG:boto:Port: 8001 2023-10-11T16:02:47.362 DEBUG:boto:Params: {} 2023-10-11T16:02:47.362 DEBUG:boto:Token: None 2023-10-11T16:02:47.363 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:02:47 GMT /dghqvd-73/obj4 2023-10-11T16:02:47.363 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:lawftG9ic4OU9DJImBFtadWctSw= 2023-10-11T16:02:47.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:02:47 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lawftG9ic4OU9DJImBFtadWctSw=', 'Content-Length': '0'} 2023-10-11T16:02:47.366 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:00:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009641d2f51dfcbf7c-006526c727-28b2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:02:47.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.367 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dghqvd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.419 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.419 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.467 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"dghqvd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:02:47.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.469 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dghqvd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.519 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.519 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.564 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-10-11T16:02:47.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.566 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.620 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.621 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.668 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-10-11T16:02:47.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.669 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.725 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.772 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-10-11T16:02:47.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.774 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.827 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.827 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.871 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:02:47.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.873 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:47.922 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:47.923 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:47.967 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:02:47.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:47.969 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.017 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.018 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.067 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:48.068 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:02:48.068 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 20, 2023-10-11T16:02:48.068 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:02:48.068 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T16:02:48.069 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T16:02:48.069 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:02:48.069 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T16:02:48.069 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.069 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T16:02:48.070 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T16:02:48.070 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:02:48.070 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T16:02:48.070 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.071 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:02:48.071 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.071 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:02:48.071 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:02:48.071 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:02:48.072 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:02:48.072 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.072 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:02:48.072 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T16:02:48.072 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.073 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:02:48.073 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.073 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:02:48.073 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:02:48.074 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:02:48.074 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:02:48.074 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:02:48.074 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:02:48.075 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:02:48.075 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:02:48.075 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:02:48.075 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:02:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.076 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:02:48.077 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T16:02:48.077 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.077 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:02:48.078 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.078 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:02:48.078 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:02:48.078 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:02:48.079 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:02:48.079 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:02:48.079 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:02:48.079 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:02:48.079 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:02:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:02:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:02:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.080 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.081 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.081 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:02:48.081 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.081 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:02:48.082 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:02:48.082 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:02:48.082 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:02:48.082 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.083 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.083 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.083 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:02:48.083 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:02:48.083 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:02:48.084 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T16:02:48.084 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.084 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T16:02:48.084 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T16:02:48.085 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T16:02:48.085 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.085 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T16:02:48.085 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:02:48.085 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:02:48.086 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:02:48.086 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.086 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.086 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.087 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.087 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T16:02:48.087 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.087 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T16:02:48.088 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:02:48.088 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T16:02:48.088 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:02:48.088 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:02:48.089 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:02:48.089 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.089 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.089 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:02:48.089 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T16:02:48.090 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:02:48.090 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:02:48.090 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:02:48.090 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.091 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.091 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:02:48.091 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:02:48.091 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:02:48.091 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:02:48.092 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.092 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.092 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:02:48.092 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:02:48.092 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:02:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:02:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.093 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "allowed" 2023-10-11T16:02:48.094 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.094 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.094 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.094 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:02:48.095 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.095 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.095 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.095 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T16:02:48.096 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T16:02:48.096 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:02:48.096 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T16:02:48.096 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.096 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:02:48.097 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.097 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:02:48.097 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:02:48.097 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:02:48.098 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:02:48.098 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.098 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:02:48.098 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T16:02:48.099 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.099 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:02:48.099 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.099 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:02:48.100 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:02:48.100 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:02:48.100 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:02:48.100 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:02:48.100 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:02:48.101 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:02:48.101 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:02:48.101 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:02:48.101 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:02:48.101 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.102 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.102 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.102 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.102 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:02:48.103 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T16:02:48.103 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:02:48.103 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:02:48.103 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.103 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:02:48.104 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:02:48.104 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:02:48.104 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:02:48.104 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:02:48.104 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:02:48.105 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:02:48.105 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:02:48.105 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:02:48.105 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:02:48.105 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.106 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.106 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.106 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.106 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:02:48.107 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.107 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:02:48.107 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:02:48.107 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:02:48.107 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:02:48.108 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.108 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.108 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.109 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:02:48.109 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:02:48.109 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:02:48.109 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T16:02:48.109 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.110 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:02:48.110 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:02:48.110 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.110 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.111 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:02:48.111 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T16:02:48.111 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.111 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:02:48.111 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T16:02:48.112 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.112 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.112 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:02:48.112 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T16:02:48.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.114 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.114 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:02:48.114 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T16:02:48.114 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.115 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:48.115 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:02:48.115 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T16:02:48.115 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.115 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:48.116 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.116 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T16:02:48.116 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:02:48.116 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T16:02:48.116 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T16:02:48.117 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:02:48.117 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:02:48.117 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:02:48.117 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:02:48.118 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:02:48.118 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.118 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T16:02:48.118 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:02:48.119 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:02:48.119 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:02:48.119 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:02:48.120 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:02:48.120 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.121 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T16:02:48.121 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:02:48.121 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:02:48.121 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:02:48.122 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:02:48.122 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:02:48.122 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.122 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T16:02:48.122 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:02:48.123 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:02:48.123 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:02:48.123 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:02:48.123 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:02:48.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.124 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T16:02:48.124 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:02:48.124 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:02:48.125 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:02:48.125 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:02:48.125 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:02:48.125 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:48.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:48.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:02:48.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T16:02:48.126 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:48.126 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:48.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.128 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.174 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.175 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.234 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:02:48.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.236 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dghqvd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.309 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.309 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.408 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:02:48.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.409 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.485 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.485 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.580 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:02:48.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-72', '--source-bucket=dghqvd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-72', '--source-bucket=dghqvd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.582 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dghqvd-72 --source-zones=a1 --dest-zones=a2 --bucket=dghqvd-72 --source-bucket=dghqvd-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.663 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.664 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.752 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T16:02:48.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-72', '--source-bucket=dghqvd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-72', '--source-bucket=dghqvd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.754 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dghqvd-73 --source-zones=a1 --dest-zones=a2 --bucket=dghqvd-72 --source-bucket=dghqvd-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.821 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.822 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.888 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T16:02:48.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.889 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dghqvd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:48.941 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:48.941 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:48.990 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-72","source":{"bucket":"dghqvd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dghqvd-73","source":{"bucket":"dghqvd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T16:02:48.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:02:48.992 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:02:49.037 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:49.038 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:49.082 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:02:49.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:49.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:49.084 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:49.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:49.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:49.163 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:02:49.175 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:02:49.176 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.176 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:02:49.176 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:02:49.176 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.177 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.177 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:02:49.177 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:02:49.177 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.178 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.178 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:02:49.178 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:02:49.178 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.179 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.179 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:02:49.179 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:02:48.974927Z" 2023-10-11T16:02:49.179 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.179 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:02:49.180 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:02:49.181 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:02:49.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:49.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:49.183 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:49.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:49.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:49.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:49.260+0000 7efc69c1c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:49.262 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:49.261+0000 7efc69c1c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:49.278 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:49.279 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:02:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.280 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:49.281 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:02:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:02:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:02:49.282 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.283 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:02:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:49.284 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:02:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.285 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:49.286 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:02:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:49.287 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:02:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:49.288 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:02:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:49.289 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:02:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:49.290 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:49.291 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:02:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:02:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:49.292 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:49Z" 2023-10-11T16:02:49.292 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:49.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:02:49Z" } 2023-10-11T16:02:49.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:02:49.294 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:02:49.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:02:49.295 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:02:54.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:54.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:54.297 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:54.354 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:54.355 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:54.386 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:54.384+0000 7f8bb4e08780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:54.387 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:54.385+0000 7f8bb4e08780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:54.400 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:54.401 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:02:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.402 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:54.403 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:02:54.404 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:54.404 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:02:54.404 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:54.404 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:02:54.404 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:54.405 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:54.405 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.405 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:54.405 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:54.405 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:54.406 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:54.406 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:02:54.406 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:54.406 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:54.407 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:54.407 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:02:54.407 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:54.407 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:54.407 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.408 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:54.408 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:54.408 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:54.408 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:54.408 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:02:54.409 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:54.409 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:54.409 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:54.409 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:02:54.410 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:54.410 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:54.410 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.410 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:54.410 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:54.411 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:54.411 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:54.411 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:02:54.411 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:54.411 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:54.412 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:54.412 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:02:54.412 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:54.412 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:54.413 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:54.413 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:54.413 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.413 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:54.413 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:02:54.414 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:02:54.414 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:54.414 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:54Z" 2023-10-11T16:02:54.414 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:54.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:02:54Z" } 2023-10-11T16:02:54.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:02:54.417 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:02:54.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:02:54.417 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:02:59.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:59.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:02:59.420 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:02:59.477 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:02:59.478 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:02:59.511 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:59.509+0000 7fb83f087780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:59.512 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:02:59.510+0000 7fb83f087780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:02:59.526 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:02:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:02:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:02:59.526 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:02:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:02:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:02:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:59.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:02:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:02:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:59.528 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:02:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:02:59.529 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:02:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.530 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:59.531 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:02:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:59.532 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.533 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:59.533 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:02:59.533 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:59.533 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:59.533 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:02:59.534 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:59.534 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:59.534 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:59.534 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:02:59.534 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:59.535 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:59.535 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.535 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:02:59.535 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:02:59.535 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:02:59.536 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:02:59.536 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:02:59.536 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:02:59.536 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:02:59.537 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:02:59.537 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:02:59.537 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:02:59.537 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:59.537 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:02:59.538 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:02:59.538 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.538 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:02:59.538 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:02:59.538 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:02:59.539 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:02:59.539 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:02:59Z" 2023-10-11T16:02:59.539 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:02:59.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:02:59Z" } 2023-10-11T16:02:59.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:02:59.541 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:02:59.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:02:59.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:04.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:04.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:04.545 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:04.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:04.603 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:04.635 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:04.634+0000 7ffb5cbea780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:04.636 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:04.635+0000 7ffb5cbea780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:04.650 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:04.650 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:04.651 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:04.652 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:04.653 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:04.654 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:04.655 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:04.656 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.657 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:04.658 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:04.659 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:04.660 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.660 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:04.660 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:04.660 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:04.660 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:04.661 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:04.661 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:04.661 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:04.661 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:04.662 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:04.662 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:04.662 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:04.662 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:04.662 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:04.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.663 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:04.663 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:04.663 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:04.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:04.664 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:04Z" 2023-10-11T16:03:04.664 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:04.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:04Z" } 2023-10-11T16:03:04.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:04.666 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:04.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:04.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:09.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:09.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:09.669 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:09.726 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:09.727 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:09.759 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:09.757+0000 7ff6f0f43780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:09.760 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:09.758+0000 7ff6f0f43780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:09.773 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:09.773 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:09.774 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.775 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:09.776 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:09.777 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:09.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:09.779 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:09.780 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:09.781 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:09.782 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:09.783 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:09.784 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:09.785 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:09.785 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:09.785 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:09.785 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:09.785 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:09.786 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:09.786 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:09.787 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:09.787 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:09.787 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:09.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.787 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:09.788 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:09.788 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:09.788 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:09.788 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:09Z" 2023-10-11T16:03:09.788 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:09.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:09Z" } 2023-10-11T16:03:09.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:09.791 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:09.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:09.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:14.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:14.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:14.795 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:14.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:14.851 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:14.883 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:14.880+0000 7f6c2f682780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:14.884 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:14.881+0000 7f6c2f682780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:14.897 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:14.898 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:14.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:14.900 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:14.901 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.902 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:14.903 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:14.904 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:14.905 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:14.906 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:14.907 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:14.907 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:14.907 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:14.907 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:14.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:14.908 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:14.908 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:14.908 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:14.908 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:14.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:14.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:14.909 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:14.909 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:14.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:14.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:14.910 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:14.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.910 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:14.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:14.911 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:14.911 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:14.911 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:14Z" 2023-10-11T16:03:14.911 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:14.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:14Z" } 2023-10-11T16:03:14.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:14.914 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:14.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:14.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:19.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:19.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:19.917 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:19.973 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:19.974 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:20.006 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:20.003+0000 7fd577476780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:20.007 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:20.004+0000 7fd577476780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:20.021 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:20.021 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:20.022 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:20.023 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:20.024 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:20.025 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:20.026 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:20.027 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:20.028 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:20.029 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:20.030 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:20.031 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:20.032 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:20.033 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.034 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:20.035 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:20Z" 2023-10-11T16:03:20.035 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:20.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:20Z" } 2023-10-11T16:03:20.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:20.038 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:20.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:20.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:25.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:25.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:25.041 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:25.098 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:25.099 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:25.131 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:25.128+0000 7f7fd00a0780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:25.132 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:25.129+0000 7f7fd00a0780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:25.146 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:25.146 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:25.147 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:25.148 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:25.149 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:25.150 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:25.151 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:25.152 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:25.153 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:25.154 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:25.155 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:25.156 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:25.157 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:25.158 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:25.159 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:25Z" 2023-10-11T16:03:25.159 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:25.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:25Z" } 2023-10-11T16:03:25.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:25.161 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:25.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:25.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:30.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:30.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:30.164 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:30.222 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:30.223 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:30.258 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:30.255+0000 7fbe9b518780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:30.259 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:30.256+0000 7fbe9b518780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:30.273 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:30.273 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:30.274 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:30.275 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:30.276 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:30.277 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:30.278 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:30.279 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:30.280 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:30.281 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:30.282 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:30.283 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:30.284 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:30.285 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:30.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:30.287 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:30.288 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:30Z" 2023-10-11T16:03:30.288 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:30.289 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:30Z" } 2023-10-11T16:03:30.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:30.290 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:30.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:30.291 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:35.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:35.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:35.293 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:35.352 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:35.353 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:35.384 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:35.382+0000 7ff150480780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:35.385 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:35.383+0000 7ff150480780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:35.399 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:35.399 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:35.399 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:35.400 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:35.401 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:35.402 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:35.403 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:35.404 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:35.405 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:35.406 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:35.407 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:35.408 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:35.409 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:35.410 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.411 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:35.412 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:35Z" 2023-10-11T16:03:35.412 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:35.413 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:35Z" } 2023-10-11T16:03:35.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:35.415 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:35.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:35.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:40.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:40.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:40.418 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:40.474 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:40.474 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:40.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:40.503+0000 7f93b26fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:40.506 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:40.504+0000 7f93b26fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:40.520 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:40.520 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:40.521 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:40.522 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:40.523 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:40.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.525 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:40.525 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:40.525 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:40.525 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:40.526 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:40.526 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:40.526 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:40.526 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:40.526 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:40.527 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:40.527 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:40.527 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.527 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:40.528 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:40.529 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:40.530 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:40.531 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:40.532 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:40.533 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:40.534 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:40Z" 2023-10-11T16:03:40.534 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:40.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:40Z" } 2023-10-11T16:03:40.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:40.536 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:40.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:40.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:45.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:45.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:45.540 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:45.596 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:45.597 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:45.629 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:45.626+0000 7f494750c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:45.630 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:45.627+0000 7f494750c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:45.643 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:45.643 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:45.644 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:45.645 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:45.646 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.647 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:45.648 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:45.649 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:45.650 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:45.651 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:45.653 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:45.654 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:45.654 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:45.654 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:45.654 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:45.654 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:45.655 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:45.655 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:45.655 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.655 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:45.656 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:45Z" 2023-10-11T16:03:45.656 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:45.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:45Z" } 2023-10-11T16:03:45.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:45.659 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:45.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:45.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:50.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:50.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:50.661 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:50.718 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:50.719 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:50.751 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:50.749+0000 7f3ae52ac780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:50.752 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:50.750+0000 7f3ae52ac780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:50.765 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:50.766 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:50.766 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:50.766 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:50.766 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:50.767 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:50.767 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:50.767 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.767 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:50.768 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:50.769 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:50.770 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:50.771 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:50.772 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:50.773 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:50.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:50.775 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:50.776 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:50.777 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:50.778 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:50Z" 2023-10-11T16:03:50.779 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:50.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:50Z" } 2023-10-11T16:03:50.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:50.781 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:50.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:50.781 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:03:55.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:55.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:03:55.784 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:03:55.841 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:03:55.842 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:03:55.874 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:55.873+0000 7f477d485780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:55.875 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:03:55.874+0000 7f477d485780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:03:55.889 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:03:55.889 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:03:55.890 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:03:55.890 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:03:55.890 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:03:55.890 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:03:55.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:55.891 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.891 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:03:55.891 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:55.891 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:03:55.891 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:55.892 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:55.892 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:03:55.892 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:55.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:03:55.893 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:55.893 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:03:55.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:55.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:55.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:55.894 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:03:55.894 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:55.894 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:55.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:03:55.895 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:55.895 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:55.895 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:55.895 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:03:55.896 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:55.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:55.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.896 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:55.896 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:03:55.897 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:55.897 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:55.897 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:03:55.897 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:55.897 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:55.898 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:55.898 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:03:55.898 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:55.898 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:55.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.899 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:03:55.899 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:03:55.899 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:03:55.899 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:03:55.900 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:03:55.900 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:03:55.900 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:03:55.900 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:03:55.900 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:03:55.901 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:03:55.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:55.901 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:03:55.901 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:03:55.901 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:03:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:03:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:03:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:03:55.902 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:03:55Z" 2023-10-11T16:03:55.903 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:03:55.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:03:55Z" } 2023-10-11T16:03:55.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:03:55.904 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:03:55.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:03:55.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:00.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:00.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:00.908 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:00.964 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:00.965 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:00.998 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:00.996+0000 7f226c27b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:00.999 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:00.997+0000 7f226c27b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:01.012 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:01.013 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.014 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:01.015 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:01.016 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.017 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:01.018 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:01.019 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:01.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:01.021 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:01.022 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:01.023 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:01.023 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:01.023 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:01.023 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.023 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:01.024 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:01.024 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:01.024 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:01.024 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:01.025 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:01.025 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:01.025 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:01.025 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:01.026 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:01.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:01.026 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:01.026 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:01.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.027 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:01.027 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:01.027 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:01.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:01.028 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:00Z" 2023-10-11T16:04:01.028 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:01.029 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:00Z" } 2023-10-11T16:04:01.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:01.031 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:01.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:01.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:06.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:06.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:06.035 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:06.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:06.090 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:06.121 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:06.120+0000 7f666b559780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:06.122 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:06.121+0000 7f666b559780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:06.136 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:06.136 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:06.136 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:06.136 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:06.137 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:06.137 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:06.137 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:06.137 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.137 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:06.138 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:06.138 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:06.138 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:06.138 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:06.139 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:06.139 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:06.139 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:06.139 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:06.139 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:06.140 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:06.140 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:06.140 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.140 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:06.140 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:06.141 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:06.141 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:06.141 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:06.141 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:06.142 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:06.142 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:06.142 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:06.142 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:06.142 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:06.143 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.143 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:06.143 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:06.143 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:06.143 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:06.144 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:06.144 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:06.144 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:06.144 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:06.145 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:06.145 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:06.145 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:06.146 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.146 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:06.146 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:06.147 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:06.147 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:06.147 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:06.147 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:06.147 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:06.148 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:06.148 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:06.148 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:06.148 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:06.148 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:06.149 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:06.149 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.149 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:06.149 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:06.150 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:06.150 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:06.150 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:06Z" 2023-10-11T16:04:06.150 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:06.152 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:06Z" } 2023-10-11T16:04:06.152 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:06.153 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:06.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:06.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:11.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:11.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:11.156 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:11.213 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:11.213 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:11.245 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:11.243+0000 7f0e215d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:11.246 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:11.244+0000 7f0e215d7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:11.259 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:11.259 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:11.260 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:11.260 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:11.260 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:11.260 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:11.260 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:11.261 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.261 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:11.261 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:11.261 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:11.261 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:11.262 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:11.262 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:11.262 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:11.262 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:11.262 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:11.263 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:11.263 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:11.263 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:11.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:11.264 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:11.264 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:11.264 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:11.264 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:11.264 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:11.265 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:11.265 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:11.265 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:11.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:11.265 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:11.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:11.267 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:11.267 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:11.267 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:11.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:11.268 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:11.268 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:11.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:11.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:11.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.269 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:11.269 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:11.269 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:11.269 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:11.269 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:11.270 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:11.270 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:11.270 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:11.270 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:11.271 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:11.271 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:11.271 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:11.271 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:11.271 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.272 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:11.272 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:11.272 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:11.272 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:11.272 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:11Z" 2023-10-11T16:04:11.273 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:11.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:11Z" } 2023-10-11T16:04:11.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:11.275 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:11.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:11.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:16.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:16.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:16.278 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:16.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:16.335 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:16.369+0000 7efdf92fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:16.370+0000 7efdf92fd780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:16.385 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:16.386 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:16.386 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:16.386 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:16.386 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:16.387 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:16.387 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:16.387 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.387 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:16.388 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:16.388 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:16.388 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:16.388 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:16.388 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:16.389 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:16.389 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:16.389 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:16.389 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:16.390 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:16.391 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:16.392 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:16.392 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:16.392 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:16.392 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:16.393 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.393 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:16.393 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:16.393 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:16.393 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:16.394 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:16.394 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:16.394 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:16.394 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:16.394 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:16.395 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:16.395 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:16.395 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.395 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:16.396 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:16.396 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:16.396 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:16.396 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:16.396 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:16.397 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:16.397 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:16.397 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:16.397 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:16.397 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:16.398 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:16.398 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:16.398 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.398 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:16.398 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:16.399 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:16.399 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:16.399 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:16Z" 2023-10-11T16:04:16.399 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:16.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:16Z" } 2023-10-11T16:04:16.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:16.401 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:16.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:16.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:21.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:21.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:21.405 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:21.464 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:21.464 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:21.496 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:21.495+0000 7f5a432e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:21.497 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:21.496+0000 7f5a432e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:21.511 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:21.511 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:21.512 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:21.512 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:21.512 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:21.512 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:21.512 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:21.513 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.513 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:21.513 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:21.513 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:21.514 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:21.514 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:21.514 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:21.514 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:21.514 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:21.515 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:21.515 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:21.515 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:21.515 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:21.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.516 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:21.516 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:21.516 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:21.516 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:21.517 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:21.517 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:21.517 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:21.517 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:21.517 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:21.518 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:21.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:21.518 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.518 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:21.519 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:21.519 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:21.519 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:21.519 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:21.519 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:21.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:21.520 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:21.520 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:21.520 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:21.521 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:21.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.521 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:21.521 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:21.521 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:21.522 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:21.522 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:21.522 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:21.522 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:21.523 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:21.523 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:21.523 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:21.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:21.523 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:21.524 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:21.524 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.524 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:21.524 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:21.525 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:21.525 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:21.525 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:21Z" 2023-10-11T16:04:21.525 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:21.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:21Z" } 2023-10-11T16:04:21.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:21.527 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:21.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:21.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:26.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:26.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:26.531 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:26.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:26.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:26.618 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:26.617+0000 7f189d6db780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:26.619 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:26.618+0000 7f189d6db780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:26.633 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:26.633 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:26.633 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:26.633 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:26.634 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:26.634 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:26.634 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:26.634 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.635 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:26.635 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:26.635 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:26.635 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:26.635 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:26.636 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:26.636 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:26.636 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:26.636 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:26.636 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:26.637 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:26.637 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:26.637 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.637 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:26.637 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:26.638 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:26.638 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:26.638 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:26.638 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:26.638 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:26.639 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:26.639 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:26.639 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:26.639 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:26.639 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:26.640 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:26.640 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:26.640 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:26.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:26.641 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:26.641 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:26.641 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:26.641 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:26.642 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:26.642 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:26.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.642 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:26.642 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:26.643 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:26.643 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:26.643 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:26.643 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:26.643 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:26.644 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:26.644 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:26.644 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:26.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:26.645 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:26.645 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:26.645 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.645 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:26.645 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:26.646 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:26.646 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:26.646 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:26Z" 2023-10-11T16:04:26.646 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:26.647 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:26Z" } 2023-10-11T16:04:26.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:26.648 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:26.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:26.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:31.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:31.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:31.652 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:31.711 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:31.711 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:31.743 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:31.742+0000 7f7a125d8780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:31.744 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:31.743+0000 7f7a125d8780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:31.758 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:31.758 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:31.759 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:31.759 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:31.759 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:31.759 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:31.760 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:31.760 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.760 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:31.760 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:31.760 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:31.761 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:31.761 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:31.761 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:31.762 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:31.762 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:31.762 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:31.762 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:31.762 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:31.763 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:31.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.763 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:31.763 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:31.764 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:31.764 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:31.764 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:31.764 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:31.764 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:31.765 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:31.765 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:31.765 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:31.765 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:31.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.766 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:31.766 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:31.766 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:31.766 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:31.766 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:31.767 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:31.767 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:31.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:31.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:31.768 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:31.768 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:31.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:31.768 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:31.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:31.769 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:31.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:31.769 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:31.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:31.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:31.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:31.770 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:31.770 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:31.771 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:31.771 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:31.771 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.771 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:31.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:31.772 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:31.772 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:31.772 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:31Z" 2023-10-11T16:04:31.772 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:31.773 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:31Z" } 2023-10-11T16:04:31.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:31.775 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:31.775 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:31.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:36.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:36.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:36.778 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:36.836 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:36.837 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:36.869 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:36.867+0000 7fa56ac60780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:36.870 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:36.868+0000 7fa56ac60780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:36.883 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:36.884 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:36.884 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:36.884 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:36.885 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:36.885 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:36.885 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:36.885 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.885 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:36.886 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:36.886 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:36.886 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:36.886 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:36.886 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:36.887 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:36.887 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:36.887 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:36.887 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:36.888 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:36.888 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:36.888 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.888 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:36.888 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:36.889 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:36.889 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:36.889 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:36.889 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:36.889 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:36.890 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:36.890 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:36.890 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:36.890 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:36.890 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.891 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:36.891 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:36.891 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:36.891 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:36.892 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:36.892 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:36.892 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:36.892 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:36.893 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:36.893 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:36.893 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:36.893 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:36.894 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:36.894 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:36.894 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:36.894 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:36.895 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:36.895 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:36.895 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:36.895 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:36.895 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:36.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:36.896 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:36.896 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:36.896 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.896 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:36.897 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:36.897 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:36.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:36.897 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:36Z" 2023-10-11T16:04:36.897 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:36.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:36Z" } 2023-10-11T16:04:36.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:36.900 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:36.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:36.900 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:41.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:41.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:41.903 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:41.959 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:41.960 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:41.990+0000 7f495004a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:41.993 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:41.991+0000 7f495004a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:42.007 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:42.007 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:42.007 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:42.008 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:42.008 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:42.008 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:42.008 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:42.008 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.009 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:42.009 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:42.009 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:42.009 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:42.010 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:42.010 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:42.010 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:42.011 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:42.011 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:42.011 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:42.011 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:42.011 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:42.012 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.012 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:42.012 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:42.012 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:42.012 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:42.013 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:42.013 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:42.013 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:42.013 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:42.013 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:42.014 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:42.014 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:42.014 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.014 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:42.014 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:42.015 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:42.016 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:42.016 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:42.016 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:42.016 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.016 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:42.017 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:42.017 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:42.017 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:42.017 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:42.018 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:42.018 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:42.018 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:42.018 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:42.018 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:42.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:42.019 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:42.019 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:42.019 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.019 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:42.020 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:42.020 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:42.020 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:42.020 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:41Z" 2023-10-11T16:04:42.020 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:42.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:41Z" } 2023-10-11T16:04:42.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:42.022 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:42.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:42.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:47.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:47.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:47.025 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:47.083 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:47.083 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:47.117 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:47.114+0000 7f6ab8e8f780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:47.118 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:47.115+0000 7f6ab8e8f780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:47.131 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:47.132 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:47.132 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:47.132 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:47.133 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:47.133 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:47.133 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:47.133 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.133 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:47.134 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:47.134 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:47.134 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:47.134 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:47.135 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:47.135 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:47.135 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:47.135 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:47.135 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:47.136 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:47.136 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:47.136 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.136 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:47.137 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:47.137 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:47.137 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:47.137 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:47.137 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:47.138 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:47.138 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:47.138 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:47.138 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:47.138 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:47.139 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.139 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:47.139 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:47.139 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:47.139 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:47.140 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:47.140 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:47.140 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:47.140 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:47.140 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:47.141 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:47.141 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:47.141 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.141 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:47.142 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:47.142 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:47.142 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:47.142 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040042.138525_988.1", 2023-10-11T16:04:47.142 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:47.143 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:47.143 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:47.143 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:42.138525Z", 2023-10-11T16:04:47.143 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:47.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:47.144 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:47.144 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:47.144 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.144 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:47.145 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:47.145 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:47.145 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:47.145 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:47Z" 2023-10-11T16:04:47.145 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:47.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040042.138525_988.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:42.138525Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:47Z" } 2023-10-11T16:04:47.147 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040042.138525_988.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:42.138525Z', 'realm_epoch': 2}}] 2023-10-11T16:04:47.148 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:47.148 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040042.138525_988.1'} 2023-10-11T16:04:47.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040168.974927_1016.1 target=1_1697040042.138525_988.1 2023-10-11T16:04:52.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.151 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:52.208 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:52.209 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:52.241 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:52.239+0000 7f7286ff8780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:52.242 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:52.240+0000 7f7286ff8780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:52.256 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:52.256 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:52.256 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:52.257 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:52.257 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:52.257 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:52.257 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.258 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.258 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:52.258 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.258 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:52.258 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.259 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.259 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:52.259 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.259 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:52.260 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.260 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:52.260 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.260 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.260 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.261 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.261 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:52.261 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.261 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.261 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:52.262 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.262 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.262 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.262 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:52.263 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.263 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.263 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.263 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.263 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:52.264 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.264 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.264 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:52.264 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.264 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.265 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.265 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:52.265 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.265 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.266 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.266 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.266 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:52.266 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.266 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.267 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:04:52.267 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.267 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.267 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.267 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:04:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:52.268 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:52.269 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.270 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:52Z" 2023-10-11T16:04:52.270 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:52.272 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:52Z" } 2023-10-11T16:04:52.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:04:52.274 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:52.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:52.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:04:52.276 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:04:52.277 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:04:52.277 DEBUG:boto:Method: HEAD 2023-10-11T16:04:52.278 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:04:52.279 DEBUG:boto:Data: 2023-10-11T16:04:52.280 DEBUG:boto:Headers: {} 2023-10-11T16:04:52.280 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.281 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.282 DEBUG:boto:Params: {} 2023-10-11T16:04:52.283 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:04:52.284 DEBUG:boto:Token: None 2023-10-11T16:04:52.285 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-72/ 2023-10-11T16:04:52.286 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTh+CJheGPSRSK6n1666Mmhev7Y= 2023-10-11T16:04:52.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTh+CJheGPSRSK6n1666Mmhev7Y=', 'Content-Length': '0'} 2023-10-11T16:04:52.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dcdb2d4dffa1762-006526c7a4-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.295 DEBUG:boto:path=/dghqvd-72/obj5 2023-10-11T16:04:52.295 DEBUG:boto:auth_path=/dghqvd-72/obj5 2023-10-11T16:04:52.296 DEBUG:boto:Method: PUT 2023-10-11T16:04:52.297 DEBUG:boto:Path: /dghqvd-72/obj5 2023-10-11T16:04:52.297 DEBUG:boto:Data: 2023-10-11T16:04:52.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:04:52.298 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.299 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.299 DEBUG:boto:Params: {} 2023-10-11T16:04:52.300 DEBUG:boto:Token: None 2023-10-11T16:04:52.300 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-72/obj5 2023-10-11T16:04:52.301 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:yQiAzsQi80BzE4Rk5YpXPv6+K1A= 2023-10-11T16:04:52.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yQiAzsQi80BzE4Rk5YpXPv6+K1A='} 2023-10-11T16:04:52.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c26193089e7531d4-006526c7a4-29dc-a1'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.310 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:04:52.310 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:04:52.310 DEBUG:boto:Method: HEAD 2023-10-11T16:04:52.311 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:04:52.311 DEBUG:boto:Data: 2023-10-11T16:04:52.311 DEBUG:boto:Headers: {} 2023-10-11T16:04:52.312 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.312 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.313 DEBUG:boto:Params: {} 2023-10-11T16:04:52.313 DEBUG:boto:Token: None 2023-10-11T16:04:52.313 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-72/ 2023-10-11T16:04:52.314 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KTh+CJheGPSRSK6n1666Mmhev7Y= 2023-10-11T16:04:52.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KTh+CJheGPSRSK6n1666Mmhev7Y=', 'Content-Length': '0'} 2023-10-11T16:04:52.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006070e8f827dbeada-006526c7a4-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.317 DEBUG:boto:path=/dghqvd-72/obj6 2023-10-11T16:04:52.318 DEBUG:boto:auth_path=/dghqvd-72/obj6 2023-10-11T16:04:52.318 DEBUG:boto:Method: PUT 2023-10-11T16:04:52.318 DEBUG:boto:Path: /dghqvd-72/obj6 2023-10-11T16:04:52.319 DEBUG:boto:Data: 2023-10-11T16:04:52.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:04:52.320 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.320 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.320 DEBUG:boto:Params: {} 2023-10-11T16:04:52.322 DEBUG:boto:Token: None 2023-10-11T16:04:52.322 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-72/obj6 2023-10-11T16:04:52.323 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sKPTYI9gGfj9IP6m/oPKqQXZPLc= 2023-10-11T16:04:52.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sKPTYI9gGfj9IP6m/oPKqQXZPLc='} 2023-10-11T16:04:52.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f8e07478cc8b74d-006526c7a4-29dc-a1'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.331 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:04:52.332 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:04:52.332 DEBUG:boto:Method: HEAD 2023-10-11T16:04:52.333 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:04:52.333 DEBUG:boto:Data: 2023-10-11T16:04:52.333 DEBUG:boto:Headers: {} 2023-10-11T16:04:52.334 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.334 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.335 DEBUG:boto:Params: {} 2023-10-11T16:04:52.336 DEBUG:boto:Token: None 2023-10-11T16:04:52.336 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-73/ 2023-10-11T16:04:52.337 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H13D9g2kt3mcTBgQKhlieXh81Ck= 2023-10-11T16:04:52.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H13D9g2kt3mcTBgQKhlieXh81Ck=', 'Content-Length': '0'} 2023-10-11T16:04:52.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052010dfb3e7717e9-006526c7a4-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.341 DEBUG:boto:path=/dghqvd-73/obj7 2023-10-11T16:04:52.341 DEBUG:boto:auth_path=/dghqvd-73/obj7 2023-10-11T16:04:52.342 DEBUG:boto:Method: PUT 2023-10-11T16:04:52.342 DEBUG:boto:Path: /dghqvd-73/obj7 2023-10-11T16:04:52.342 DEBUG:boto:Data: 2023-10-11T16:04:52.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:04:52.343 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.344 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.344 DEBUG:boto:Params: {} 2023-10-11T16:04:52.344 DEBUG:boto:Token: None 2023-10-11T16:04:52.345 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-73/obj7 2023-10-11T16:04:52.345 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:KU4fCqfF724PiC089jIZ4wrqkLY= 2023-10-11T16:04:52.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KU4fCqfF724PiC089jIZ4wrqkLY='} 2023-10-11T16:04:52.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000043e40fb953fc133-006526c7a4-29dc-a1'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.353 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:04:52.353 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:04:52.354 DEBUG:boto:Method: HEAD 2023-10-11T16:04:52.354 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:04:52.354 DEBUG:boto:Data: 2023-10-11T16:04:52.355 DEBUG:boto:Headers: {} 2023-10-11T16:04:52.355 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.356 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.356 DEBUG:boto:Params: {} 2023-10-11T16:04:52.356 DEBUG:boto:Token: None 2023-10-11T16:04:52.357 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-73/ 2023-10-11T16:04:52.357 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H13D9g2kt3mcTBgQKhlieXh81Ck= 2023-10-11T16:04:52.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H13D9g2kt3mcTBgQKhlieXh81Ck=', 'Content-Length': '0'} 2023-10-11T16:04:52.360 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d03745d7f8bb1e80-006526c7a4-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.361 DEBUG:boto:path=/dghqvd-73/obj8 2023-10-11T16:04:52.361 DEBUG:boto:auth_path=/dghqvd-73/obj8 2023-10-11T16:04:52.362 DEBUG:boto:Method: PUT 2023-10-11T16:04:52.362 DEBUG:boto:Path: /dghqvd-73/obj8 2023-10-11T16:04:52.363 DEBUG:boto:Data: 2023-10-11T16:04:52.363 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:04:52.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:04:52.364 DEBUG:boto:Port: 8000 2023-10-11T16:04:52.364 DEBUG:boto:Params: {} 2023-10-11T16:04:52.365 DEBUG:boto:Token: None 2023-10-11T16:04:52.365 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:04:52 GMT /dghqvd-73/obj8 2023-10-11T16:04:52.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sBzDHr40Y0zvrCrZs3kK8A/0YSI= 2023-10-11T16:04:52.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:04:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sBzDHr40Y0zvrCrZs3kK8A/0YSI='} 2023-10-11T16:04:52.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000161e16a04a47043b-006526c7a4-29dc-a1'), ('Date', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:04:52.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.416 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:52.466 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:52.466 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:52.496 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:04:52.508 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:04:52.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.509 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:52.509 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:04:52.509 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.509 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.509 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:52.510 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:04:52.510 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.510 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.510 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:52.511 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:04:52.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.511 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.511 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:04:52.511 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:02:48.974927Z" 2023-10-11T16:04:52.512 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.512 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:04:52.513 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:52.513 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:04:52.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.515 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:52.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:52.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:52.594 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:52.591+0000 7fd0b4ac5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:52.595 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:04:52.592+0000 7fd0b4ac5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:04:52.607 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:52.608 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:52.608 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:52.608 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:52.608 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:52.609 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:04:52.609 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.609 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.609 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:52.610 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.610 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:52.610 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.611 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.611 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:04:52.611 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.611 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:04:52.611 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.612 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:04:52.612 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.612 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.612 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.612 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.613 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.614 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.614 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:04:52.614 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.614 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.614 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.615 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.615 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:52.615 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.615 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.615 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:04:52.616 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.616 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.616 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.616 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:04:52.616 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.617 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:04:52.618 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:04:52.618 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.618 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.618 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:04:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:04:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.619 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:52.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.620 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:52.620 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:04:52.620 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:04:52.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.621 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:52Z" 2023-10-11T16:04:52.621 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:52.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:04:52Z" } 2023-10-11T16:04:52.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:04:52.623 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:52.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:04:52.624 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:04:52.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.625 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:52.673 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:52.674 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:52.721 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:04:52.721 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.722 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:04:52.722 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:04:52.722 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.722 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.723 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:04:52.723 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:04:52.723 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.723 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.723 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:04:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:04:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:04:52.724 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:04:52.725 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.725 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:04:52.726 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:04:52.727 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:04:52.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:52.728 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:52.779 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:52.780 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:52.824 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:52.824 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:52.824 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:52.824 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:52.825 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:52.825 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:04:52.825 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.825 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:52.826 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.826 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:52.826 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.826 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:52.827 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:04:52.827 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.827 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.828 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.828 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:04:52.828 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.828 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.829 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.829 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:52.829 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.829 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:52.830 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:04:52.830 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.830 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:52.831 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.831 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:04:52.831 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.831 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.832 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.832 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:52.832 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.832 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:52.832 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:04:52.833 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.833 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:04:52.833 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.833 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:04:52.834 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.834 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.834 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:52.834 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:52.834 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:52.835 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:52.835 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:04:52.835 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:52.835 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:04:52.836 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:52.836 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:04:52.836 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.837 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:52.837 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:52.837 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.837 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:52.837 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:04:52.838 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:04:52.838 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:52.838 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:52Z" 2023-10-11T16:04:52.838 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:52.839 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.306165Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.400452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:00:43.337900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:04:52Z" } 2023-10-11T16:04:52.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.306165Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.400452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.337900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:04:52.841 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:04:52.841 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:04:52.841 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=1 master=00000000000000000000:00000000000000032624 target=00000000000000000000:00000000000000032484, shard=2 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051810 2023-10-11T16:04:57.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:57.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:04:57.844 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:04:57.903 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:04:57.904 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:04:57.952 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:04:57.953 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:04:57.953 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:04:57.953 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:04:57.953 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:04:57.954 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:04:57.954 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.954 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:04:57.954 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:57.955 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:57.956 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:57.956 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:04:57.956 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:57.956 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.956 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:57.957 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:04:57.957 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:57.957 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:57.957 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:04:57.958 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:57.958 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:04:57.958 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:57.958 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:04:57.958 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:57.959 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.959 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:57.959 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:04:57.959 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:57.959 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:57.960 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:04:57.960 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:57.960 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:04:57.960 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:57.961 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:04:57.961 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:57.961 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.961 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:04:57.961 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:04:57.962 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:04:57.962 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:04:57.962 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:04:57.962 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:04:57.962 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:04:57.963 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:04:57.963 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:04:57.963 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:57.963 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:04:57.964 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:04:57.964 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.964 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:04:57.964 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:04:57.964 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:04:57.965 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:04:57.965 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:04:57Z" 2023-10-11T16:04:57.965 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:04:57.966 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.306165Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.400452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:00:43.337900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:04:57Z" } 2023-10-11T16:04:57.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.306165Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.400452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.337900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:04:57.967 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:04:57.967 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:04:57.968 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=1 master=00000000000000000000:00000000000000032624 target=00000000000000000000:00000000000000032484, shard=2 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051810 2023-10-11T16:05:02.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:02.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:02.971 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:03.030 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:03.030 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:03.082 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:03.082 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:03.083 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:03.084 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:05:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:03.085 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:05:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:03.086 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:05:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:05:03.088 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:03.089 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.089 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:03.089 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:03.089 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:03.089 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:03.090 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:05:03.090 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:03.090 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:03.090 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:03.091 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:05:03.091 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:03.091 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.091 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:03.091 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:03.092 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:03.092 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:03.092 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:03.092 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:03.092 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:03.093 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:03.093 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:03.093 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:03.093 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:03.093 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:03.094 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.094 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:03.094 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:03.094 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:03.095 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:03.095 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:03Z" 2023-10-11T16:05:03.095 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:03.096 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.306165Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.400452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:00:43.337900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:03Z" } 2023-10-11T16:05:03.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.306165Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.400452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.337900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:03.097 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:03.097 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:03.098 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=1 master=00000000000000000000:00000000000000032624 target=00000000000000000000:00000000000000032484, shard=2 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051810 2023-10-11T16:05:08.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:08.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:08.100 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:08.159 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:08.159 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:08.215 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:08.215 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:08.216 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:08.217 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.217 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:08.217 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:08.218 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:08.218 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:08.218 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:08.218 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-10-11T16:05:08.218 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:08.219 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:08.219 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:08.219 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.306165Z" 2023-10-11T16:05:08.219 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:08.220 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.220 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:08.220 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:08.220 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:08.220 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032484", 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.400452Z" 2023-10-11T16:05:08.221 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:08.222 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.222 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:08.222 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:08.222 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:08.223 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:08.223 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000051810", 2023-10-11T16:05:08.223 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:08.223 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:08.223 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:08.224 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:43.337900Z" 2023-10-11T16:05:08.224 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:08.224 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.224 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:08.225 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:08.225 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:08.225 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:08.225 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:08.225 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:08.226 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:08.228 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:08Z" 2023-10-11T16:05:08.229 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:08.230 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.306165Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:00:43.400452Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051810", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:00:43.337900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:08Z" } 2023-10-11T16:05:08.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.306165Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.400452Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051810', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:00:43.337900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:08.231 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:08.231 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045937', 1: '00000000000000000000:00000000000000032484', 2: '00000000000000000000:00000000000000051810', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:08.232 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=1 master=00000000000000000000:00000000000000032624 target=00000000000000000000:00000000000000032484, shard=2 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051810 2023-10-11T16:05:13.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.234 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:13.294 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:13.295 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:13.343 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:13.344 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:13.345 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:13.346 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:05:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:05:13.347 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:13.348 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.349 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.350 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:13.351 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:05:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:05:13.352 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:13.353 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.354 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:13.354 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:13.354 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.354 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:13.354 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.355 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:13.355 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.355 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.355 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:13.356 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.357 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:13Z" 2023-10-11T16:05:13.357 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:13.358 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:13Z" } 2023-10-11T16:05:13.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:13.359 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:13.359 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:13.360 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:05:13.360 DEBUG:boto:path=/dghqvd-73/ 2023-10-11T16:05:13.361 DEBUG:boto:auth_path=/dghqvd-73/ 2023-10-11T16:05:13.361 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.361 DEBUG:boto:Path: /dghqvd-73/ 2023-10-11T16:05:13.362 DEBUG:boto:Data: 2023-10-11T16:05:13.362 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.363 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.363 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.363 DEBUG:boto:Params: {} 2023-10-11T16:05:13.364 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:05:13.365 DEBUG:boto:Token: None 2023-10-11T16:05:13.365 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-73/ 2023-10-11T16:05:13.366 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:df/eUP3zb+B8NFqILdGkq9Y2Gyw= 2023-10-11T16:05:13.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:df/eUP3zb+B8NFqILdGkq9Y2Gyw=', 'Content-Length': '0'} 2023-10-11T16:05:13.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000982f104f12f58b7a-006526c7b9-29e2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.372 DEBUG:boto:path=/dghqvd-73/obj5 2023-10-11T16:05:13.373 DEBUG:boto:auth_path=/dghqvd-73/obj5 2023-10-11T16:05:13.373 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.373 DEBUG:boto:Path: /dghqvd-73/obj5 2023-10-11T16:05:13.374 DEBUG:boto:Data: 2023-10-11T16:05:13.374 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.375 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.375 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.375 DEBUG:boto:Params: {} 2023-10-11T16:05:13.376 DEBUG:boto:Token: None 2023-10-11T16:05:13.376 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-73/obj5 2023-10-11T16:05:13.377 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:PuVa8bEhGCWSOuQwEASVFJ9NTYA= 2023-10-11T16:05:13.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PuVa8bEhGCWSOuQwEASVFJ9NTYA=', 'Content-Length': '0'} 2023-10-11T16:05:13.379 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006230dc4562d8a7e7-006526c7b9-29e2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.379 DEBUG:boto:path=/dghqvd-73/obj6 2023-10-11T16:05:13.380 DEBUG:boto:auth_path=/dghqvd-73/obj6 2023-10-11T16:05:13.380 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.381 DEBUG:boto:Path: /dghqvd-73/obj6 2023-10-11T16:05:13.381 DEBUG:boto:Data: 2023-10-11T16:05:13.381 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.382 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.382 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.382 DEBUG:boto:Params: {} 2023-10-11T16:05:13.383 DEBUG:boto:Token: None 2023-10-11T16:05:13.383 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-73/obj6 2023-10-11T16:05:13.384 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l4ViJ2O9L8Chknf7A27weiWI4Uk= 2023-10-11T16:05:13.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l4ViJ2O9L8Chknf7A27weiWI4Uk=', 'Content-Length': '0'} 2023-10-11T16:05:13.386 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c26fd725ef6fb944-006526c7b9-29e2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.386 DEBUG:boto:path=/dghqvd-73/obj7 2023-10-11T16:05:13.387 DEBUG:boto:auth_path=/dghqvd-73/obj7 2023-10-11T16:05:13.387 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.388 DEBUG:boto:Path: /dghqvd-73/obj7 2023-10-11T16:05:13.388 DEBUG:boto:Data: 2023-10-11T16:05:13.389 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.389 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.390 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.390 DEBUG:boto:Params: {} 2023-10-11T16:05:13.390 DEBUG:boto:Token: None 2023-10-11T16:05:13.391 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-73/obj7 2023-10-11T16:05:13.391 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FbZVxaXX1uXkhune55zAdjuETr0= 2023-10-11T16:05:13.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FbZVxaXX1uXkhune55zAdjuETr0=', 'Content-Length': '0'} 2023-10-11T16:05:13.393 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fae1a9efff34c2d9-006526c7b9-29e2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.394 DEBUG:boto:path=/dghqvd-73/obj8 2023-10-11T16:05:13.397 DEBUG:boto:auth_path=/dghqvd-73/obj8 2023-10-11T16:05:13.397 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.398 DEBUG:boto:Path: /dghqvd-73/obj8 2023-10-11T16:05:13.398 DEBUG:boto:Data: 2023-10-11T16:05:13.399 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.399 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.400 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.400 DEBUG:boto:Params: {} 2023-10-11T16:05:13.400 DEBUG:boto:Token: None 2023-10-11T16:05:13.401 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-73/obj8 2023-10-11T16:05:13.401 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:3PqBaXzLcrmz7q6PJonrMYBal64= 2023-10-11T16:05:13.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3PqBaXzLcrmz7q6PJonrMYBal64=', 'Content-Length': '0'} 2023-10-11T16:05:13.404 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b934ce180ebca619-006526c7b9-29e2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.404 DEBUG:boto:path=/dghqvd-72/ 2023-10-11T16:05:13.404 DEBUG:boto:auth_path=/dghqvd-72/ 2023-10-11T16:05:13.405 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.405 DEBUG:boto:Path: /dghqvd-72/ 2023-10-11T16:05:13.406 DEBUG:boto:Data: 2023-10-11T16:05:13.406 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.407 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.407 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.408 DEBUG:boto:Params: {} 2023-10-11T16:05:13.408 DEBUG:boto:Token: None 2023-10-11T16:05:13.409 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/ 2023-10-11T16:05:13.409 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TIccI2P4WE2Mq8yH7gjfbAjLU28= 2023-10-11T16:05:13.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TIccI2P4WE2Mq8yH7gjfbAjLU28=', 'Content-Length': '0'} 2023-10-11T16:05:13.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f624ee880cc059ed-006526c7b9-29e2-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.412 DEBUG:boto:path=/dghqvd-72/obj7 2023-10-11T16:05:13.413 DEBUG:boto:auth_path=/dghqvd-72/obj7 2023-10-11T16:05:13.413 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.413 DEBUG:boto:Path: /dghqvd-72/obj7 2023-10-11T16:05:13.414 DEBUG:boto:Data: 2023-10-11T16:05:13.414 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.415 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.415 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.416 DEBUG:boto:Params: {} 2023-10-11T16:05:13.416 DEBUG:boto:Token: None 2023-10-11T16:05:13.416 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj7 2023-10-11T16:05:13.417 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:USSH6g+82kFStzgfRyucmdxkFF8= 2023-10-11T16:05:13.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:USSH6g+82kFStzgfRyucmdxkFF8=', 'Content-Length': '0'} 2023-10-11T16:05:13.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060cb2388ede8ca7b-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.420 DEBUG:boto:path=/dghqvd-72/obj7 2023-10-11T16:05:13.421 DEBUG:boto:auth_path=/dghqvd-72/obj7 2023-10-11T16:05:13.421 DEBUG:boto:Method: GET 2023-10-11T16:05:13.422 DEBUG:boto:Path: /dghqvd-72/obj7 2023-10-11T16:05:13.422 DEBUG:boto:Data: 2023-10-11T16:05:13.423 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.423 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.423 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.424 DEBUG:boto:Params: {} 2023-10-11T16:05:13.424 DEBUG:boto:Token: None 2023-10-11T16:05:13.425 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj7 2023-10-11T16:05:13.425 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5giQ0gLQkC4IMZQV0c2Ku0PeKiM= 2023-10-11T16:05:13.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5giQ0gLQkC4IMZQV0c2Ku0PeKiM=', 'Content-Length': '0'} 2023-10-11T16:05:13.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e390ff16fa99e25-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.428 DEBUG:boto:path=/dghqvd-72/obj8 2023-10-11T16:05:13.429 DEBUG:boto:auth_path=/dghqvd-72/obj8 2023-10-11T16:05:13.429 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.430 DEBUG:boto:Path: /dghqvd-72/obj8 2023-10-11T16:05:13.430 DEBUG:boto:Data: 2023-10-11T16:05:13.431 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.431 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.432 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.432 DEBUG:boto:Params: {} 2023-10-11T16:05:13.432 DEBUG:boto:Token: None 2023-10-11T16:05:13.433 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj8 2023-10-11T16:05:13.433 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nez9mglCPRhUvKIykdoBHkLAcEo= 2023-10-11T16:05:13.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nez9mglCPRhUvKIykdoBHkLAcEo=', 'Content-Length': '0'} 2023-10-11T16:05:13.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab4af40ede90f63a-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.436 DEBUG:boto:path=/dghqvd-72/obj8 2023-10-11T16:05:13.437 DEBUG:boto:auth_path=/dghqvd-72/obj8 2023-10-11T16:05:13.437 DEBUG:boto:Method: GET 2023-10-11T16:05:13.437 DEBUG:boto:Path: /dghqvd-72/obj8 2023-10-11T16:05:13.438 DEBUG:boto:Data: 2023-10-11T16:05:13.438 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.439 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.439 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.440 DEBUG:boto:Params: {} 2023-10-11T16:05:13.440 DEBUG:boto:Token: None 2023-10-11T16:05:13.441 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj8 2023-10-11T16:05:13.441 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1uyTg47DONWtawcP60Qrd0NJ6Gs= 2023-10-11T16:05:13.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1uyTg47DONWtawcP60Qrd0NJ6Gs=', 'Content-Length': '0'} 2023-10-11T16:05:13.444 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000972f66d1755353ad-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.444 DEBUG:boto:path=/dghqvd-72/obj7 2023-10-11T16:05:13.445 DEBUG:boto:auth_path=/dghqvd-72/obj7 2023-10-11T16:05:13.445 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.446 DEBUG:boto:Path: /dghqvd-72/obj7 2023-10-11T16:05:13.446 DEBUG:boto:Data: 2023-10-11T16:05:13.446 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.447 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.447 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.448 DEBUG:boto:Params: {} 2023-10-11T16:05:13.448 DEBUG:boto:Token: None 2023-10-11T16:05:13.448 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj7 2023-10-11T16:05:13.449 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:USSH6g+82kFStzgfRyucmdxkFF8= 2023-10-11T16:05:13.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:USSH6g+82kFStzgfRyucmdxkFF8=', 'Content-Length': '0'} 2023-10-11T16:05:13.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e127896d50c43997-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.452 DEBUG:boto:path=/dghqvd-72/obj7 2023-10-11T16:05:13.452 DEBUG:boto:auth_path=/dghqvd-72/obj7 2023-10-11T16:05:13.453 DEBUG:boto:Method: GET 2023-10-11T16:05:13.453 DEBUG:boto:Path: /dghqvd-72/obj7 2023-10-11T16:05:13.453 DEBUG:boto:Data: 2023-10-11T16:05:13.454 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.454 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.455 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.455 DEBUG:boto:Params: {} 2023-10-11T16:05:13.456 DEBUG:boto:Token: None 2023-10-11T16:05:13.456 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj7 2023-10-11T16:05:13.457 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:5giQ0gLQkC4IMZQV0c2Ku0PeKiM= 2023-10-11T16:05:13.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5giQ0gLQkC4IMZQV0c2Ku0PeKiM=', 'Content-Length': '0'} 2023-10-11T16:05:13.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092491342adfcfa50-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.507 DEBUG:boto:path=/dghqvd-72/obj8 2023-10-11T16:05:13.507 DEBUG:boto:auth_path=/dghqvd-72/obj8 2023-10-11T16:05:13.508 DEBUG:boto:Method: HEAD 2023-10-11T16:05:13.508 DEBUG:boto:Path: /dghqvd-72/obj8 2023-10-11T16:05:13.509 DEBUG:boto:Data: 2023-10-11T16:05:13.509 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.509 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.510 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.510 DEBUG:boto:Params: {} 2023-10-11T16:05:13.511 DEBUG:boto:Token: None 2023-10-11T16:05:13.511 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj8 2023-10-11T16:05:13.512 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:nez9mglCPRhUvKIykdoBHkLAcEo= 2023-10-11T16:05:13.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nez9mglCPRhUvKIykdoBHkLAcEo=', 'Content-Length': '0'} 2023-10-11T16:05:13.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c59cd7df61d0dea5-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.515 DEBUG:boto:path=/dghqvd-72/obj8 2023-10-11T16:05:13.515 DEBUG:boto:auth_path=/dghqvd-72/obj8 2023-10-11T16:05:13.515 DEBUG:boto:Method: GET 2023-10-11T16:05:13.516 DEBUG:boto:Path: /dghqvd-72/obj8 2023-10-11T16:05:13.516 DEBUG:boto:Data: 2023-10-11T16:05:13.517 DEBUG:boto:Headers: {} 2023-10-11T16:05:13.517 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:05:13.517 DEBUG:boto:Port: 8001 2023-10-11T16:05:13.518 DEBUG:boto:Params: {} 2023-10-11T16:05:13.518 DEBUG:boto:Token: None 2023-10-11T16:05:13.519 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:05:13 GMT /dghqvd-72/obj8 2023-10-11T16:05:13.519 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:1uyTg47DONWtawcP60Qrd0NJ6Gs= 2023-10-11T16:05:13.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1uyTg47DONWtawcP60Qrd0NJ6Gs=', 'Content-Length': '0'} 2023-10-11T16:05:13.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:04:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-73:daa316e9-869c-4074-a756-a8adfb4a3573.9868.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040c35a90afd1684b-006526c7b9-29e2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:13.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:13.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:13.523 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dghqvd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:13.574 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:13.574 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:13.631 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:05:13.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:13.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:13.632 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:13.679 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:13.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:13.725 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:05:13.726 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2023-10-11T16:05:13.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.729 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:13.778 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:13.779 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:13.810 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:05:13.822 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:05:13.822 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.822 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:13.822 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:05:13.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.823 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.823 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:13.823 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:05:13.823 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.824 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.824 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:13.824 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:05:13.824 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.825 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.825 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040313.610444_1018.1", 2023-10-11T16:05:13.825 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:13.610444Z" 2023-10-11T16:05:13.825 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.825 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:05:13.827 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:13.827 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:05:13.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:13.828 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:13.879 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:13.879 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:13.905+0000 7f2b40e4a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:13.909 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:13.906+0000 7f2b40e4a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:13.921 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:13.922 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:13.923 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:13.924 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:13.925 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:13.926 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:13.927 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:13.928 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.929 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:13.930 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:13.931 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:13.932 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:05:13.933 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:05:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:13.934 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.935 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:13.935 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:13.935 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.935 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:13.936 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:13.936 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:13.936 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:13.936 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:13Z" 2023-10-11T16:05:13.936 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:13.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:13Z" } 2023-10-11T16:05:13.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:05:13.939 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:13.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:05:13.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040313.610444_1018.1 target=1_1697040168.974927_1016.1 2023-10-11T16:05:18.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:18.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:18.942 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:18.996 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:18.997 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:19.028 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:19.026+0000 7f667593c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:19.029 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:19.027+0000 7f667593c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:19.044 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:19.044 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:19.044 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:19.044 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:19.045 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:19.045 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:19.045 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:19.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.046 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:19.047 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:19.047 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:19.047 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:19.048 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:19.048 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:19.049 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:19.049 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:19.049 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:19.050 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:19.050 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:19.051 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:19.051 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.051 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:19.052 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:19.052 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:19.052 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:19.053 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:19.053 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:19.054 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:19.054 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:19.054 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:19.055 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:19.055 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:19.056 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.056 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:19.056 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:19.057 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:19.057 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:19.058 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:19.058 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:19.059 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:19.059 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:19.059 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:19.060 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:19.060 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:19.061 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.061 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:19.062 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:19.062 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:19.063 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:19.064 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:05:19.065 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:19.065 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:19.065 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:19.065 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:05:19.066 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:19.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:19.066 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:19.066 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:19.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.067 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:19.067 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:19.067 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:19.068 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:19.068 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:19Z" 2023-10-11T16:05:19.068 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:19.069 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:19Z" } 2023-10-11T16:05:19.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:05:19.070 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:19.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:05:19.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040313.610444_1018.1 target=1_1697040168.974927_1016.1 2023-10-11T16:05:24.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:24.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:24.073 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:24.131 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:24.132 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:24.164 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:24.162+0000 7f27fdc4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:24.165 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:24.163+0000 7f27fdc4e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:24.178 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:24.179 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:24.179 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:24.179 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:24.179 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:24.180 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:24.180 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:24.180 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.180 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:24.180 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:24.181 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:24.181 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:24.181 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:24.181 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:24.181 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:24.182 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:24.182 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:24.182 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:24.182 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:24.183 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:24.184 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.185 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:24.186 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:24.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:24.186 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:24.186 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:24.187 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:24.187 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:24.187 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:24.187 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:24.188 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:24.188 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:24.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.188 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:24.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:24.189 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:24.189 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:24.189 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:05:24.189 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:24.190 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:24.190 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:24.190 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:05:24.190 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:24.190 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:24.191 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:24.191 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:24.191 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.191 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:24.192 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:24.192 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:24.192 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:24.192 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:24Z" 2023-10-11T16:05:24.192 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:24.193 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:24Z" } 2023-10-11T16:05:24.194 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:05:24.195 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:24.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:05:24.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040313.610444_1018.1 target=1_1697040168.974927_1016.1 2023-10-11T16:05:29.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:29.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:29.198 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:29.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:29.256 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:29.288 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:29.287+0000 7f4c9fedb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:29.289 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:29.288+0000 7f4c9fedb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:29.303 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:29.303 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:29.305 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:29.306 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:29.308 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:29.309 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:29.310 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:29.311 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.312 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:29.313 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:29.313 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:29.313 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:29.313 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040168.974927_1016.1", 2023-10-11T16:05:29.313 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:29.314 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:29.314 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:29.314 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:02:48.974927Z", 2023-10-11T16:05:29.314 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:29.314 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:29.315 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:29.315 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:29.315 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.315 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:29.315 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:29.316 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:29.316 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:29.316 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:29Z" 2023-10-11T16:05:29.316 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:29.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040168.974927_1016.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:02:48.974927Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:29Z" } 2023-10-11T16:05:29.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040168.974927_1016.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:02:48.974927Z', 'realm_epoch': 2}}] 2023-10-11T16:05:29.319 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:29.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040168.974927_1016.1'} 2023-10-11T16:05:29.319 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040313.610444_1018.1 target=1_1697040168.974927_1016.1 2023-10-11T16:05:34.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.322 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:34.380 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:34.380 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:34.412 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:34.410+0000 7f5795e3b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:34.413 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:34.412+0000 7f5795e3b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:34.427 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:34.427 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:34.427 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:34.427 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:34.428 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:34.428 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:34.428 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.428 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.429 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:34.429 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.429 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:34.429 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.429 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.430 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:34.430 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.430 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:34.430 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.431 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:34.431 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.431 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.431 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.431 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.432 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:34.432 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.432 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.432 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:34.432 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.433 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.433 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.433 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:34.433 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.433 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.434 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.434 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.434 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:34.434 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.435 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.435 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:34.435 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.435 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.435 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:34.436 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.437 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.437 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:34.437 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.437 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.437 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040313.610444_1018.1", 2023-10-11T16:05:34.438 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.438 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.438 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.438 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:13.610444Z", 2023-10-11T16:05:34.439 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.439 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.439 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.439 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:34.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.440 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:34.440 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:34.440 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:34.440 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.440 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:34Z" 2023-10-11T16:05:34.441 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:34.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040313.610444_1018.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:13.610444Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:34Z" } 2023-10-11T16:05:34.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040313.610444_1018.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:13.610444Z', 'realm_epoch': 2}}] 2023-10-11T16:05:34.443 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:34.443 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040313.610444_1018.1'} 2023-10-11T16:05:34.444 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:05:34.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:34.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:34.445 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:34.491 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:34.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:34.539 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[]} 2023-10-11T16:05:34.539 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-74 2023-10-11T16:05:34.540 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:05:34.540 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:05:34.541 DEBUG:boto:Method: PUT 2023-10-11T16:05:34.541 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:05:34.542 DEBUG:boto:Data: 2023-10-11T16:05:34.542 DEBUG:boto:Headers: {} 2023-10-11T16:05:34.542 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:34.543 DEBUG:boto:Port: 8000 2023-10-11T16:05:34.543 DEBUG:boto:Params: {} 2023-10-11T16:05:34.544 DEBUG:boto:Token: None 2023-10-11T16:05:34.544 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 16:05:34 GMT /dghqvd-74/ 2023-10-11T16:05:34.545 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:kNHJ547zq/SSLVjUL+flLlE9oAU= 2023-10-11T16:05:34.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:kNHJ547zq/SSLVjUL+flLlE9oAU=', 'Content-Length': '0'} 2023-10-11T16:05:34.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5c5762d36908b5b-006526c7ce-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:34.561 INFO:rgw_multi.tests:create bucket zone=a1 name=dghqvd-75 2023-10-11T16:05:34.561 DEBUG:boto:path=/dghqvd-75/ 2023-10-11T16:05:34.562 DEBUG:boto:auth_path=/dghqvd-75/ 2023-10-11T16:05:34.562 DEBUG:boto:Method: PUT 2023-10-11T16:05:34.562 DEBUG:boto:Path: /dghqvd-75/ 2023-10-11T16:05:34.563 DEBUG:boto:Data: 2023-10-11T16:05:34.563 DEBUG:boto:Headers: {} 2023-10-11T16:05:34.564 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:34.564 DEBUG:boto:Port: 8000 2023-10-11T16:05:34.565 DEBUG:boto:Params: {} 2023-10-11T16:05:34.565 DEBUG:boto:Token: None 2023-10-11T16:05:34.565 DEBUG:boto:StringToSign: PUT Wed, 11 Oct 2023 16:05:34 GMT /dghqvd-75/ 2023-10-11T16:05:34.566 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:H/PiWx/65wlITi02ontYl1bAcaE= 2023-10-11T16:05:34.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:H/PiWx/65wlITi02ontYl1bAcaE=', 'Content-Length': '0'} 2023-10-11T16:05:34.582 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000030eefea2f163b99b-006526c7ce-29dc-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:34.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.584 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:34.632 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:34.632 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:34.663 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:05:34.675 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:05:34.676 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.676 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:34.676 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:05:34.676 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.677 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.677 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:34.677 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:05:34.677 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.677 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.678 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:34.678 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:05:34.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.678 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:05:34.679 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:34.579755Z" 2023-10-11T16:05:34.679 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.679 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:05:34.680 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:34.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:05:34.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:34.682 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:34.731 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:34.731 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:34.763 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:34.762+0000 7f904a587780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:34.764 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:34.763+0000 7f904a587780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:34.777 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:34.777 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:34.777 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:34.778 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:34.778 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:34.778 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:34.778 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.779 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.779 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:34.779 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.779 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:34.779 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.780 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.780 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:34.780 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.780 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:34.780 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.781 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:34.781 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.781 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.781 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.782 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.782 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:34.782 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.782 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.782 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:34.783 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.783 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.783 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.783 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:34.784 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.784 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.784 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.784 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.784 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:34.785 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.785 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.785 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:34.785 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.785 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.786 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.786 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:34.786 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.786 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.787 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.787 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:34.787 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:34.787 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:34.787 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:34.788 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.552169_1019.1", 2023-10-11T16:05:34.788 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:34.788 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:34.788 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:34.788 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.552169Z", 2023-10-11T16:05:34.789 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:34.789 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.789 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:34.789 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:34.789 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.790 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:34.790 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:34.790 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:34.790 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:34.791 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:34Z" 2023-10-11T16:05:34.791 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:34.792 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.552169_1019.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.552169Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:34Z" } 2023-10-11T16:05:34.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.552169_1019.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.552169Z', 'realm_epoch': 2}}] 2023-10-11T16:05:34.793 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:34.793 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.552169_1019.1'} 2023-10-11T16:05:34.793 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040334.579755_1026.1 target=1_1697040334.552169_1019.1 2023-10-11T16:05:39.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:39.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:39.796 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:39.854 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:39.886 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:39.885+0000 7faaf340f780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:39.887 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:39.886+0000 7faaf340f780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:39.901 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:39.902 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:39.902 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:39.902 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:39.902 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:39.903 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:39.903 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:39.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.903 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:39.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:39.904 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:39.904 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:39.904 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:39.905 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:39.905 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:39.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:39.905 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:39.906 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:39.906 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:39.906 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:39.906 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.907 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:39.907 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:39.907 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:39.907 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:39.907 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:39.908 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:39.908 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:39.908 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:39.908 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:39.908 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:39.909 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:39.909 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.909 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:39.909 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:39.910 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:39.910 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:39.910 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:39.910 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:39.910 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:39.911 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:39.911 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:39.911 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:39.911 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:39.912 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.912 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:39.913 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:39.913 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:39.914 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:39.914 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:05:39.915 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:39.915 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:39.915 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:39.915 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:05:39.916 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:39.916 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:39.916 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:39.916 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:39.917 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.917 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:39.917 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:39.917 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:39.918 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:39.918 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:39Z" 2023-10-11T16:05:39.918 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:39.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:39Z" } 2023-10-11T16:05:39.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:05:39.920 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:39.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:39.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:05:39.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:39.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:39.923 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:39.974 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.020 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:05:40.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.023 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.076 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.077 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.122 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:05:40.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.123 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dghqvd-74 --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-74 --dest-bucket=dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.177 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.178 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.224 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:05:40.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dghqvd-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.226 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dghqvd-75 --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-74 --dest-bucket=dghqvd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.284 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.285 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.332 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:05:40.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dghqvd-74', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.333 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dghqvd-74 --dest-bucket=dghqvd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.390 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.390 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.436 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:05:40.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.437 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.487 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.487 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.535 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:05:40.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.536 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.586 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.587 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.638 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:40.638 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:40.638 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 21, 2023-10-11T16:05:40.639 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:05:40.639 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T16:05:40.639 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T16:05:40.639 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:40.640 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T16:05:40.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.641 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T16:05:40.642 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T16:05:40.642 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:05:40.642 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T16:05:40.643 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.643 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:05:40.643 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.643 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:05:40.643 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:05:40.644 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:05:40.644 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.644 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.644 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:05:40.645 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T16:05:40.645 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.645 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:05:40.645 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.645 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:05:40.646 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:05:40.646 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:05:40.646 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:05:40.646 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:05:40.646 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:05:40.647 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:05:40.647 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:05:40.647 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:05:40.647 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:05:40.647 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.648 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.648 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.648 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:05:40.648 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T16:05:40.649 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.649 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:05:40.649 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.649 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:05:40.649 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:05:40.650 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:05:40.650 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:05:40.650 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:05:40.650 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:05:40.650 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:05:40.651 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:05:40.651 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:05:40.651 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:05:40.651 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.651 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.652 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.652 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.652 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:05:40.652 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.653 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:05:40.653 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:05:40.653 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:05:40.653 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:05:40.654 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.654 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.654 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.654 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:05:40.655 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:05:40.655 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:05:40.655 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T16:05:40.655 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.655 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T16:05:40.656 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T16:05:40.656 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T16:05:40.656 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.656 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T16:05:40.656 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.657 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:05:40.657 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:05:40.657 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.657 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.657 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.658 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.658 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T16:05:40.658 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.658 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-74", 2023-10-11T16:05:40.658 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.659 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:05:40.660 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:05:40.660 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.660 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:05:40.660 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.660 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.661 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:05:40.661 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.661 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:05:40.661 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:05:40.661 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.662 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.662 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:05:40.662 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:05:40.662 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:05:40.662 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:05:40.663 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.663 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.663 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.663 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-75", 2023-10-11T16:05:40.663 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.664 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:05:40.664 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.664 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:05:40.664 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.664 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.665 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:05:40.665 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-75", 2023-10-11T16:05:40.665 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.665 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:05:40.665 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.666 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.666 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:05:40.666 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.666 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:05:40.666 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:05:40.667 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.667 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:05:40.667 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:05:40.667 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:05:40.668 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:05:40.668 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.668 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.668 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.668 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T16:05:40.669 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.669 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:05:40.669 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.669 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:05:40.669 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.670 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.670 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:05:40.670 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-75", 2023-10-11T16:05:40.670 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.670 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:05:40.671 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.671 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.671 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:05:40.671 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:05:40.671 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:05:40.672 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:05:40.672 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.672 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.672 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:05:40.672 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:05:40.673 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:05:40.673 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:05:40.673 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.673 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.673 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "enabled" 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:05:40.674 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.675 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.675 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.675 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.675 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T16:05:40.675 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T16:05:40.676 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:05:40.676 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T16:05:40.676 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.676 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:05:40.676 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.677 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:05:40.677 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:05:40.677 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:05:40.677 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:05:40.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.679 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:05:40.679 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T16:05:40.679 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.679 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:05:40.679 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.680 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:05:40.680 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:05:40.680 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:05:40.680 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:05:40.680 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:05:40.681 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:05:40.681 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:05:40.681 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:05:40.681 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:05:40.682 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:05:40.682 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.682 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.682 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.682 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.683 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:05:40.683 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T16:05:40.683 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:05:40.683 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:05:40.683 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.684 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:05:40.684 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:05:40.684 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:05:40.684 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:05:40.684 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:05:40.685 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:05:40.685 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:05:40.685 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:05:40.685 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:05:40.685 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:05:40.686 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.686 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.686 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.686 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.686 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:05:40.687 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.687 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:05:40.687 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:05:40.687 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:05:40.687 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:05:40.688 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.688 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.688 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.688 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:05:40.688 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:05:40.689 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.690 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.690 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:05:40.690 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T16:05:40.690 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.690 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:05:40.691 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T16:05:40.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.691 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.691 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:05:40.691 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T16:05:40.692 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.692 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:05:40.692 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T16:05:40.692 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.693 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:40.693 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:05:40.693 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T16:05:40.693 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.693 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:40.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.694 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T16:05:40.694 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:05:40.694 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T16:05:40.694 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T16:05:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:05:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:05:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:05:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:05:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:05:40.696 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.696 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T16:05:40.696 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:05:40.696 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:05:40.696 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:05:40.697 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:05:40.697 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:05:40.697 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.697 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T16:05:40.697 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.698 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T16:05:40.699 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:05:40.699 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:05:40.699 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:05:40.699 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:05:40.699 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:05:40.700 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:05:40.701 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:05:40.701 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:40.701 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:40.701 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:05:40.701 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T16:05:40.702 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:40.702 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:40.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:05:40.703 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:05:40.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:40.755 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:40.864 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:05:40.865 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:05:40.865 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:05:40.866 DEBUG:boto:Method: HEAD 2023-10-11T16:05:40.866 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:05:40.866 DEBUG:boto:Data: 2023-10-11T16:05:40.867 DEBUG:boto:Headers: {} 2023-10-11T16:05:40.867 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:40.867 DEBUG:boto:Port: 8000 2023-10-11T16:05:40.868 DEBUG:boto:Params: {} 2023-10-11T16:05:40.868 DEBUG:boto:Token: None 2023-10-11T16:05:40.869 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:40 GMT /dghqvd-74/ 2023-10-11T16:05:40.869 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m0MPW5LoKHJBaAGYUcW95nS1a4Y= 2023-10-11T16:05:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m0MPW5LoKHJBaAGYUcW95nS1a4Y=', 'Content-Length': '0'} 2023-10-11T16:05:40.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010a68ae70938ae80-006526c7d4-2b96-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:40.944 DEBUG:boto:path=/dghqvd-74/obj1 2023-10-11T16:05:40.945 DEBUG:boto:auth_path=/dghqvd-74/obj1 2023-10-11T16:05:40.945 DEBUG:boto:Method: PUT 2023-10-11T16:05:40.945 DEBUG:boto:Path: /dghqvd-74/obj1 2023-10-11T16:05:40.946 DEBUG:boto:Data: 2023-10-11T16:05:40.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:05:40.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:40.947 DEBUG:boto:Port: 8000 2023-10-11T16:05:40.947 DEBUG:boto:Params: {} 2023-10-11T16:05:40.948 DEBUG:boto:Token: None 2023-10-11T16:05:40.948 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:05:40 GMT /dghqvd-74/obj1 2023-10-11T16:05:40.949 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:FeoO9MEaeByFaq9N6SCj1Rn/+vk= 2023-10-11T16:05:40.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:05:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FeoO9MEaeByFaq9N6SCj1Rn/+vk='} 2023-10-11T16:05:40.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003eba20c46092c9f0-006526c7d4-2b96-a1'), ('Date', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:40.977 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:05:40.977 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:05:40.978 DEBUG:boto:Method: HEAD 2023-10-11T16:05:40.978 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:05:40.978 DEBUG:boto:Data: 2023-10-11T16:05:40.979 DEBUG:boto:Headers: {} 2023-10-11T16:05:40.979 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:40.980 DEBUG:boto:Port: 8000 2023-10-11T16:05:40.980 DEBUG:boto:Params: {} 2023-10-11T16:05:40.980 DEBUG:boto:Token: None 2023-10-11T16:05:40.981 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:05:40 GMT /dghqvd-74/ 2023-10-11T16:05:40.981 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:m0MPW5LoKHJBaAGYUcW95nS1a4Y= 2023-10-11T16:05:40.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:05:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:m0MPW5LoKHJBaAGYUcW95nS1a4Y=', 'Content-Length': '0'} 2023-10-11T16:05:40.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000277b2b84f0e04f3d-006526c7d4-2b96-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:40.986 DEBUG:boto:path=/dghqvd-74/obj2 2023-10-11T16:05:40.986 DEBUG:boto:auth_path=/dghqvd-74/obj2 2023-10-11T16:05:40.987 DEBUG:boto:Method: PUT 2023-10-11T16:05:40.987 DEBUG:boto:Path: /dghqvd-74/obj2 2023-10-11T16:05:40.988 DEBUG:boto:Data: 2023-10-11T16:05:40.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:05:40.989 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:05:40.989 DEBUG:boto:Port: 8000 2023-10-11T16:05:40.989 DEBUG:boto:Params: {} 2023-10-11T16:05:40.990 DEBUG:boto:Token: None 2023-10-11T16:05:40.990 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:05:40 GMT /dghqvd-74/obj2 2023-10-11T16:05:40.991 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:D4koRpf/b0xXpa9wtgvNUwrtGxY= 2023-10-11T16:05:40.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:05:40 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D4koRpf/b0xXpa9wtgvNUwrtGxY='} 2023-10-11T16:05:41.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006f08859974d7f23-006526c7d4-2b96-a1'), ('Date', 'Wed, 11 Oct 2023 16:05:41 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:05:41.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.036 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:41.109 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:41.110 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:41.181 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:05:41.214 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:05:41.215 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.215 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:41.215 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:05:41.215 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.215 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.216 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:41.216 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:05:41.216 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.216 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.217 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:41.217 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:05:41.217 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.217 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.217 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:05:41.218 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:34.579755Z" 2023-10-11T16:05:41.218 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.218 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:05:41.219 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:41.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:05:41.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.221 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:41.293 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:41.293 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:41.344 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:41.342+0000 7f29178cb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:41.347 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:05:41.345+0000 7f29178cb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:05:41.366 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:41.366 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:41.366 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:41.367 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:41.367 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:41.367 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:05:41.367 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:41.367 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.368 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:41.368 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.368 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:41.368 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.368 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:41.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:05:41.369 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.369 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:05:41.369 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.369 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:05:41.370 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:41.370 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.370 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.370 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.370 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:41.371 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.371 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:41.371 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:05:41.371 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.371 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.372 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.373 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:41.373 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.373 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:41.373 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:05:41.373 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.374 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:41.374 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.374 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:05:41.374 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:41.375 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.375 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.375 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:41.376 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.376 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:05:41.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:05:41.376 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.376 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:41.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.378 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:41.378 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:05:41.378 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:05:41.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.379 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:41Z" 2023-10-11T16:05:41.379 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:41.380 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:05:41Z" } 2023-10-11T16:05:41.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:05:41.381 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:41.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:05:41.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:05:41.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.383 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:41.439 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:41.439 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:41.494 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:05:41.494 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.494 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:05:41.494 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:05:41.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.495 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.495 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:41.495 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:41.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.496 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:05:41.496 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:05:41.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.497 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:41.497 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:41.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.497 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:05:41.498 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:41.498 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:05:41.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:41.500 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:41.551 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:41.596 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:41.597 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:41.597 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:41.597 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:41.597 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:41.597 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:41.598 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.598 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:41.598 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.598 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:41.599 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.599 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:41.599 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:05:41.599 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.600 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:41.600 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.600 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:05:41.600 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.600 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.601 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.601 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:41.601 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.601 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:41.601 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:41.602 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.602 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:41.602 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.602 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:41.602 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.603 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.603 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.603 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:41.603 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.603 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:41.604 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:05:41.604 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.604 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:41.604 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.605 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:05:41.605 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.605 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.605 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:41.605 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:41.606 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:41.606 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:41.606 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:41.606 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:41.606 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:41.607 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:41.607 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:41.607 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.607 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:41.608 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:41.608 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.608 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:41.608 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:41.608 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:41.609 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:41.609 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:41Z" 2023-10-11T16:05:41.609 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:41.610 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:41Z" } 2023-10-11T16:05:41.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:41.611 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:41.611 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:41.612 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:05:46.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:46.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:46.615 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:46.673 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:46.674 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:46.722 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:46.723 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:46.723 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:46.723 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:46.723 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:46.723 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:46.724 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.724 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:46.724 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:46.724 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:46.724 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:46.725 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:46.725 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:05:46.725 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:46.725 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:46.726 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:46.726 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:05:46.726 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:46.726 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.726 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:46.727 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:46.727 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:46.727 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:46.727 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:46.727 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:46.728 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:46.728 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:46.728 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:46.728 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:46.728 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.729 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:46.729 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:46.729 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:46.729 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:46.729 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:05:46.730 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:46.730 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:46.730 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:46.730 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:05:46.731 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:46.731 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.731 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:46.731 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:46.731 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:46.732 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:46.732 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:46.732 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:46.732 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:46.732 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:46.733 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:46.733 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:46.733 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:46.733 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:46.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.734 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:46.734 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:46.734 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:46.734 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:46.735 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:46Z" 2023-10-11T16:05:46.735 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:46.736 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:46Z" } 2023-10-11T16:05:46.736 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:46.737 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:46.737 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:46.738 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:05:51.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:51.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:51.741 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:51.799 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:51.800 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:51.848 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:51.849 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:51.849 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:51.850 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:51.850 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:51.850 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:51.851 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.851 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:51.852 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:51.852 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:51.852 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:51.852 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:51.852 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:05:51.853 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:51.853 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:51.853 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:51.853 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:05:51.854 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:51.854 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.854 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:51.854 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:51.854 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:51.855 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:51.855 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:51.855 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:51.855 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:51.855 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:51.856 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:51.856 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:51.856 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.856 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:51.857 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:51.857 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:51.857 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:51.857 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:05:51.857 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:51.858 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:51.858 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:51.858 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:05:51.858 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:51.858 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.859 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:51.859 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:51.859 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:51.859 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:51.859 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:51.860 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:51.860 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:51.860 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:51.860 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:51.861 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:51.861 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:51.861 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:51.861 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.861 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:51.862 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:51.862 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:51.862 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:51.862 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:51Z" 2023-10-11T16:05:51.862 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:51.863 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:51Z" } 2023-10-11T16:05:51.864 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:51.864 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:51.865 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:51.865 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:05:56.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:56.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:05:56.867 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:05:56.926 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:05:56.926 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:05:56.975 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:05:56.976 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:05:56.976 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:05:56.976 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:05:56.976 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:05:56.977 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:05:56.977 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.977 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:05:56.977 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:56.977 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:05:56.978 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:56.978 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:56.978 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:05:56.978 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:56.979 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:56.979 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:56.979 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:05:56.979 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:56.979 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.980 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:56.980 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:05:56.980 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:56.980 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:56.980 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:05:56.981 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:56.981 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:05:56.981 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:56.981 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:05:56.981 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:56.982 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.982 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:56.982 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:05:56.982 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:56.983 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:56.983 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:05:56.983 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:56.983 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:05:56.983 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:56.984 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:05:56.984 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:56.984 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.984 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:05:56.984 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:05:56.985 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:05:56.985 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:05:56.985 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:05:56.985 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:05:56.985 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:05:56.986 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:05:56.986 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:05:56.986 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:56.986 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:05:56.987 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:05:56.987 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.987 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:05:56.987 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:05:56.987 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:05:56.988 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:05:56.988 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:05:56Z" 2023-10-11T16:05:56.988 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:05:56.989 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:05:56Z" } 2023-10-11T16:05:56.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:05:56.990 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:56.991 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:05:56.991 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:01.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:01.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:01.994 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:02.052 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:02.052 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:02.101 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:02.101 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:02.102 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:02.102 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:02.102 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:02.102 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:02.103 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.103 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:02.103 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:02.103 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:02.104 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:02.104 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:02.104 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:02.104 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:02.104 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:02.105 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:02.105 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:02.105 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:02.105 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.106 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:02.106 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:02.106 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:02.106 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:02.107 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:02.107 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:02.107 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:02.107 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:02.108 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:02.108 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:02.108 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.108 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:02.108 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:02.109 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:02.109 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:02.109 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:02.109 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:02.109 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:02.110 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:02.110 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:02.110 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:02.110 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.111 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:02.111 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:02.111 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:02.111 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:02.111 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:02.112 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:02.112 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:02.112 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:02.112 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:02.112 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:02.113 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:02.113 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:02.113 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.113 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:02.113 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:02.114 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:02.114 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:02.114 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:02Z" 2023-10-11T16:06:02.114 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:02.115 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:02Z" } 2023-10-11T16:06:02.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:02.116 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:02.117 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:02.117 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:07.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:07.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:07.119 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:07.180 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:07.181 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:07.229 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:07.230 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:07.230 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:07.230 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:07.230 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:07.230 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:07.231 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.231 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:07.231 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:07.231 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:07.231 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:07.232 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:07.232 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:07.232 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:07.232 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:07.232 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:07.233 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:07.233 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:07.233 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.233 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:07.234 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:07.234 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:07.234 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:07.234 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:07.234 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.236 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:07.236 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:07.236 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:07.236 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:07.237 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:07.237 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:07.237 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:07.237 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:07.237 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:07.238 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:07.238 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.238 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:07.238 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:07.238 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:07.239 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:07.239 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:07.239 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:07.239 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:07.240 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:07.240 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:07.240 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:07.240 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:07.240 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:07.241 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.241 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:07.241 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:07.241 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:07.241 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:07.242 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:07Z" 2023-10-11T16:06:07.242 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:07.243 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:07Z" } 2023-10-11T16:06:07.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:07.244 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:07.244 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:07.245 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:12.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:12.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:12.248 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:12.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:12.310 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:12.359 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:12.359 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:12.360 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:12.360 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:12.360 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:12.360 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:12.361 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.361 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:12.361 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:12.361 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:12.362 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:12.362 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:12.362 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:12.362 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:12.363 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:12.363 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:12.363 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:12.363 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:12.363 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.364 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:12.364 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:12.364 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:12.364 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:12.364 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:12.365 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:12.366 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.366 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:12.366 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:12.366 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:12.367 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:12.367 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:12.367 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:12.367 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:12.367 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:12.368 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:12.368 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:12.368 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.368 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:12.368 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:12.369 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:12.369 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:12.369 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:12.369 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:12.370 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:12.370 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:12.370 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:12.370 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:12.370 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:12.371 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:12.371 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.371 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:12.371 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:12.372 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:12.372 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:12.372 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:12Z" 2023-10-11T16:06:12.372 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:12.373 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:12Z" } 2023-10-11T16:06:12.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:12.374 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:12.375 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:12.375 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:17.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:17.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:17.378 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:17.437 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:17.437 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:17.486 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:17.487 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:17.487 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:17.487 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:17.487 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:17.488 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:17.488 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.488 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:17.488 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:17.489 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:17.489 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:17.489 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:17.489 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:17.489 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:17.490 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:17.490 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:17.490 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:17.490 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:17.490 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.491 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:17.491 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:17.491 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:17.491 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:17.491 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:17.492 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:17.492 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:17.492 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:17.492 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:17.493 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:17.493 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.493 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:17.493 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:17.493 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:17.494 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:17.494 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:17.494 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:17.494 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:17.494 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:17.495 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:17.495 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:17.495 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.495 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:17.495 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:17.496 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:17.496 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:17.496 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:17.496 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:17.497 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:17.497 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:17.497 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:17.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:17.497 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:17.498 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:17.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.498 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:17.498 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:17.498 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:17.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:17.499 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:17Z" 2023-10-11T16:06:17.499 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:17.500 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:17Z" } 2023-10-11T16:06:17.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:17.501 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:17.502 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:17.502 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:22.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:22.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:22.505 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:22.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:22.565 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:22.614 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:22.614 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:22.614 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:22.614 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:22.615 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:22.615 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:22.615 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.615 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:22.616 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:22.616 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:22.616 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:22.616 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:22.616 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:22.617 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:22.617 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:22.617 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:22.617 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:22.617 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:22.618 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.618 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:22.618 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:22.618 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:22.619 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:22.619 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:22.619 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:22.619 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:22.619 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:22.620 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:22.621 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:22.621 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:22.621 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:22.621 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:22.621 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:22.622 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:22.622 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:22.622 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.622 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:22.622 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:22.623 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:22.623 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:22.623 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:22.623 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:22.623 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:22.624 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:22.624 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:22.624 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:22.624 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:22.624 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:22.625 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.625 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:22.625 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:22.625 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:22.626 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:22.626 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:22Z" 2023-10-11T16:06:22.626 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:22.627 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:22Z" } 2023-10-11T16:06:22.628 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:22.628 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:22.628 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:22.629 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:27.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:27.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:27.631 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:27.690 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:27.691 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:27.738 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:27.739 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:27.739 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:27.739 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:27.739 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:27.739 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:27.740 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.740 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:27.740 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:27.740 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:27.741 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:27.741 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:27.741 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:27.741 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:27.741 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:27.742 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:27.742 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:27.742 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:27.742 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.742 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:27.743 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:27.743 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:27.743 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:27.743 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:27.743 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:27.744 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:27.744 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:27.744 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:27.744 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:27.744 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.745 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:27.745 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:27.745 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:27.745 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:27.745 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:27.746 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:27.746 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:27.746 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:27.746 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:27.747 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:27.747 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.747 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:27.747 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:27.747 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:27.748 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:27.748 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:27.748 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:27.748 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:27.748 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.749 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:27.750 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:27.750 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:27.750 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:27.750 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:27Z" 2023-10-11T16:06:27.751 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:27.751 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:27Z" } 2023-10-11T16:06:27.752 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:27.753 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:27.753 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:27.753 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:32.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:32.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:32.756 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:32.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:32.816 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:32.864 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:32.864 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:32.864 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:32.865 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:32.865 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:32.865 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:32.865 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.866 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:32.866 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:32.866 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:32.866 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:32.866 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:32.867 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:32.867 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:32.867 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:32.867 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:32.868 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:32.868 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:32.868 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.868 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:32.869 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:32.869 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:32.869 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:32.869 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:32.869 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:32.870 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:32.870 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:32.870 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:32.870 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:32.870 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.871 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:32.871 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:32.871 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:32.871 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:32.871 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:32.872 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:32.872 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:32.872 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:32.872 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:32.872 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:32.873 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.873 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:32.873 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:32.873 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:32.874 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:32.874 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:32.874 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:32.874 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:32.875 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:32.875 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:32.875 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:32.875 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:32.875 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:32.876 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.876 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:32.876 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:32.876 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:32.877 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:32.877 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:32Z" 2023-10-11T16:06:32.877 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:32.878 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:32Z" } 2023-10-11T16:06:32.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:32.879 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:32.879 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:32.880 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:37.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:37.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:37.883 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:37.942 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:37.943 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:37.990 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:37.991 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:37.991 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:37.991 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:37.992 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:37.993 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:37.994 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:37.995 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:37.996 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:37.997 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:37.998 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:37.999 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:38.000 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:38.001 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:38.002 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:38.003 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:37Z" 2023-10-11T16:06:38.003 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:38.004 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:37Z" } 2023-10-11T16:06:38.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:38.005 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:38.005 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:38.006 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:43.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:43.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:43.008 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:43.067 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:43.067 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:43.117 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:43.118 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:43.118 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:43.118 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:43.118 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:43.119 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:43.119 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.119 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:43.119 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:43.120 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:43.121 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:43.121 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:43.121 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:43.121 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.121 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:43.122 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:43.122 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:43.122 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:43.122 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:43.123 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:43.123 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:43.123 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:43.123 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:43.123 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:43.124 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.124 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:43.124 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:43.124 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:43.124 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:43.125 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:43.125 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:43.125 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:43.125 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:43.125 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:43.126 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:43.126 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.126 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:43.126 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:43.126 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:43.127 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:43.127 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:43.127 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:43.127 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:43.127 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:43.128 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:43.128 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:43.128 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:43.128 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:43.128 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.129 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:43.129 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:43.129 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:43.129 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:43.129 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:43Z" 2023-10-11T16:06:43.130 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:43.133 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:43Z" } 2023-10-11T16:06:43.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:43.134 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:43.135 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:43.135 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:48.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:48.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:48.137 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:48.196 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:48.196 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:48.243 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:48.244 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:48.244 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:48.244 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:48.244 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:48.245 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:48.245 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.245 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:48.245 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:48.245 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:48.246 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:48.246 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:48.246 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:48.246 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:48.247 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:48.247 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:48.247 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:48.247 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:48.247 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.248 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:48.248 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:48.248 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:48.248 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:48.248 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:48.249 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.250 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:48.250 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:48.250 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:48.250 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:48.250 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:48.251 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:48.251 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:48.251 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:48.251 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:48.251 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:48.252 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.252 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:48.252 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:48.252 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:48.252 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:48.253 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:48.253 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:48.253 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:48.253 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:48.253 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:48.254 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:48.254 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:48.254 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:48.254 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.254 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:48.255 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:48.255 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:48.255 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:48.255 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:48Z" 2023-10-11T16:06:48.255 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:48.256 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:48Z" } 2023-10-11T16:06:48.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:48.257 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:48.258 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:48.258 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:53.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:53.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:53.261 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:53.321 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:53.321 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:53.373 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:53.374 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:53.374 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:53.374 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:53.375 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:53.375 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:53.375 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.375 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:53.376 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:53.376 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:53.376 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:53.376 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:53.376 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:53.377 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:53.377 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:53.377 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:53.377 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:53.377 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:53.378 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.378 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:53.378 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:53.378 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:53.379 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:53.379 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:53.379 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:53.379 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:53.380 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:53.380 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:53.380 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:53.380 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.381 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:53.381 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:53.381 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:53.382 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:53.382 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:53.382 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:53.382 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:53.382 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:53.383 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:53.383 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:53.383 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.383 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:53.383 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:53.384 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:53.384 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:53.384 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:53.384 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:53.385 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:53.385 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:53.385 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:53.385 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:53.385 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:53.386 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:53.386 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.386 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:53.387 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:53.387 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:53.387 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:53.387 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:53Z" 2023-10-11T16:06:53.387 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:53.388 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:53Z" } 2023-10-11T16:06:53.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:53.390 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:53.390 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:53.391 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:06:58.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:58.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:06:58.393 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:06:58.454 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:06:58.454 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:06:58.503 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:06:58.504 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:06:58.504 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:06:58.504 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:06:58.504 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:06:58.505 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:06:58.505 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.505 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:06:58.505 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:58.505 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:06:58.506 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:58.506 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:58.506 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:06:58.506 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:58.507 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:58.507 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:58.507 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:06:58.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:58.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.508 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:58.508 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:06:58.508 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:58.508 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:58.509 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:06:58.509 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:58.510 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:06:58.510 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:58.510 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:06:58.511 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:58.511 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.511 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:58.512 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:06:58.512 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:58.512 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:58.513 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:06:58.513 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:58.513 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:06:58.513 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:58.514 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:06:58.514 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:58.515 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.515 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:06:58.515 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:06:58.516 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:06:58.516 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:06:58.516 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:06:58.517 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:06:58.517 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:06:58.518 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:06:58.518 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:06:58.518 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:58.519 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:06:58.519 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:06:58.519 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.520 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:06:58.520 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:06:58.521 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:06:58.521 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:06:58.522 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:06:58Z" 2023-10-11T16:06:58.522 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:06:58.523 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:06:58Z" } 2023-10-11T16:06:58.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:06:58.524 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:58.524 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:06:58.525 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:07:03.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:03.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:03.527 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:03.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:03.588 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:03.636 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:03.636 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:03.636 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:03.636 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.637 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:03.638 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:03.638 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:03.638 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:03.638 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:03.639 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:07:03.639 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:03.639 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:03.639 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:03.639 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:07:03.640 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:03.640 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.640 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:03.640 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:03.641 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:03.641 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:03.641 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:03.641 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:03.642 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:03.642 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:03.642 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:03.642 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:03.642 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.643 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:03.643 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:03.643 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:03.643 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:03.643 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:07:03.644 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:03.644 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:03.644 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:03.644 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:07:03.644 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:03.645 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.645 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:03.645 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:03.645 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:03.646 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:03.646 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:03.646 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:03.646 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:03.646 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:03.647 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:03.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:03.647 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:03.647 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:03.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.648 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:03.648 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:03.648 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:03.648 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:03.649 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:03Z" 2023-10-11T16:07:03.649 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:03.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:03Z" } 2023-10-11T16:07:03.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:03.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:03.651 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:03.652 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:07:08.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:08.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:08.655 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:08.713 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:08.714 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:08.761 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:08.762 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:08.762 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:08.762 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:08.762 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:08.763 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:08.763 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.763 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:08.763 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:08.764 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:08.764 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:08.764 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:08.764 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-10-11T16:07:08.764 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:08.765 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:08.765 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:08.765 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.329482Z" 2023-10-11T16:07:08.765 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:08.765 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.766 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:08.766 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:08.766 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:08.766 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:08.767 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:08.767 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:08.767 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:08.767 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:08.767 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:08.768 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:08.768 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.768 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:08.768 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:08.769 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:08.769 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:08.769 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-10-11T16:07:08.769 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:08.770 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:08.770 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:08.770 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.350900Z" 2023-10-11T16:07:08.770 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:08.770 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.771 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:08.771 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:08.771 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:08.771 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:08.772 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:08.772 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:08.772 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:08.772 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:08.772 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.773 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:08.774 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:08.774 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:08.774 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:08.774 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:08Z" 2023-10-11T16:07:08.774 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:08.775 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.329482Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:04:52.350900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:08Z" } 2023-10-11T16:07:08.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.329482Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.350900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:08.776 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:08.777 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046077', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052090', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:08.777 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000046217 target=00000000000000000000:00000000000000046077, shard=2 master=00000000000000000000:00000000000000052230 target=00000000000000000000:00000000000000052090 2023-10-11T16:07:13.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:13.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:13.780 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:13.840 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:13.840 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:13.890 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:13.890 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:13.891 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:13.892 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:07:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:13.893 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:07:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.894 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:13.895 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:13.896 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:13.897 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:07:13.898 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:07:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:13.899 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:13.900 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:13.901 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:13.902 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:13.903 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:13Z" 2023-10-11T16:07:13.904 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:13.905 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:13Z" } 2023-10-11T16:07:13.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:13.906 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:13.906 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:13.907 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:07:13.907 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:07:13.908 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:07:13.908 DEBUG:boto:Method: HEAD 2023-10-11T16:07:13.909 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:07:13.909 DEBUG:boto:Data: 2023-10-11T16:07:13.910 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.910 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.910 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.911 DEBUG:boto:Params: {} 2023-10-11T16:07:13.912 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:07:13.912 DEBUG:boto:Token: None 2023-10-11T16:07:13.913 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-74/ 2023-10-11T16:07:13.913 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8tldzYhFRFNPdphwH30vlTNVbXA= 2023-10-11T16:07:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8tldzYhFRFNPdphwH30vlTNVbXA=', 'Content-Length': '0'} 2023-10-11T16:07:13.920 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004704fda7153f16b7-006526c831-2b8c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.920 DEBUG:boto:path=/dghqvd-74/obj1 2023-10-11T16:07:13.921 DEBUG:boto:auth_path=/dghqvd-74/obj1 2023-10-11T16:07:13.921 DEBUG:boto:Method: HEAD 2023-10-11T16:07:13.922 DEBUG:boto:Path: /dghqvd-74/obj1 2023-10-11T16:07:13.922 DEBUG:boto:Data: 2023-10-11T16:07:13.922 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.923 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.923 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.924 DEBUG:boto:Params: {} 2023-10-11T16:07:13.924 DEBUG:boto:Token: None 2023-10-11T16:07:13.925 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-74/obj1 2023-10-11T16:07:13.925 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:l/5YPM36TFJ3eHwb16skUcCPfrg= 2023-10-11T16:07:13.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:l/5YPM36TFJ3eHwb16skUcCPfrg=', 'Content-Length': '0'} 2023-10-11T16:07:13.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa07aa3d1257283a-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.928 DEBUG:boto:path=/dghqvd-74/obj1 2023-10-11T16:07:13.929 DEBUG:boto:auth_path=/dghqvd-74/obj1 2023-10-11T16:07:13.929 DEBUG:boto:Method: GET 2023-10-11T16:07:13.930 DEBUG:boto:Path: /dghqvd-74/obj1 2023-10-11T16:07:13.930 DEBUG:boto:Data: 2023-10-11T16:07:13.931 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.931 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.931 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.932 DEBUG:boto:Params: {} 2023-10-11T16:07:13.932 DEBUG:boto:Token: None 2023-10-11T16:07:13.933 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-74/obj1 2023-10-11T16:07:13.933 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:rd2NkNcJ6VcAdt9/ing+DIeDOMk= 2023-10-11T16:07:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rd2NkNcJ6VcAdt9/ing+DIeDOMk=', 'Content-Length': '0'} 2023-10-11T16:07:13.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe47ebe8ce864350-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.936 DEBUG:boto:path=/dghqvd-74/obj2 2023-10-11T16:07:13.937 DEBUG:boto:auth_path=/dghqvd-74/obj2 2023-10-11T16:07:13.937 DEBUG:boto:Method: HEAD 2023-10-11T16:07:13.937 DEBUG:boto:Path: /dghqvd-74/obj2 2023-10-11T16:07:13.938 DEBUG:boto:Data: 2023-10-11T16:07:13.938 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.939 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.939 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.939 DEBUG:boto:Params: {} 2023-10-11T16:07:13.940 DEBUG:boto:Token: None 2023-10-11T16:07:13.940 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-74/obj2 2023-10-11T16:07:13.942 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:6buRutytxKYZuoGLwZwbT1qLTuE= 2023-10-11T16:07:13.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6buRutytxKYZuoGLwZwbT1qLTuE=', 'Content-Length': '0'} 2023-10-11T16:07:13.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b1231db82cf444-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.945 DEBUG:boto:path=/dghqvd-74/obj2 2023-10-11T16:07:13.945 DEBUG:boto:auth_path=/dghqvd-74/obj2 2023-10-11T16:07:13.945 DEBUG:boto:Method: GET 2023-10-11T16:07:13.946 DEBUG:boto:Path: /dghqvd-74/obj2 2023-10-11T16:07:13.946 DEBUG:boto:Data: 2023-10-11T16:07:13.947 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.947 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.948 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.948 DEBUG:boto:Params: {} 2023-10-11T16:07:13.949 DEBUG:boto:Token: None 2023-10-11T16:07:13.949 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-74/obj2 2023-10-11T16:07:13.950 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:qOhiqAGgzy6w/vB28hfRbiAG7OE= 2023-10-11T16:07:13.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qOhiqAGgzy6w/vB28hfRbiAG7OE=', 'Content-Length': '0'} 2023-10-11T16:07:13.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020e87aa779f78e52-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.952 DEBUG:boto:path=/dghqvd-75/ 2023-10-11T16:07:13.953 DEBUG:boto:auth_path=/dghqvd-75/ 2023-10-11T16:07:13.953 DEBUG:boto:Method: HEAD 2023-10-11T16:07:13.953 DEBUG:boto:Path: /dghqvd-75/ 2023-10-11T16:07:13.954 DEBUG:boto:Data: 2023-10-11T16:07:13.954 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.955 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.955 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.955 DEBUG:boto:Params: {} 2023-10-11T16:07:13.956 DEBUG:boto:Token: None 2023-10-11T16:07:13.956 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-75/ 2023-10-11T16:07:13.957 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:8Kjn1Oyk/BdE1QLbHiuaNZaPaRc= 2023-10-11T16:07:13.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8Kjn1Oyk/BdE1QLbHiuaNZaPaRc=', 'Content-Length': '0'} 2023-10-11T16:07:13.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a6f0b1636c77dd5-006526c831-2b8c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.960 DEBUG:boto:path=/dghqvd-75/obj1 2023-10-11T16:07:13.960 DEBUG:boto:auth_path=/dghqvd-75/obj1 2023-10-11T16:07:13.961 DEBUG:boto:Method: HEAD 2023-10-11T16:07:13.961 DEBUG:boto:Path: /dghqvd-75/obj1 2023-10-11T16:07:13.962 DEBUG:boto:Data: 2023-10-11T16:07:13.962 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.963 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.963 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.963 DEBUG:boto:Params: {} 2023-10-11T16:07:13.964 DEBUG:boto:Token: None 2023-10-11T16:07:13.964 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-75/obj1 2023-10-11T16:07:13.965 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:Ao6Pvyz7hmgamGtKDwWSx+1av8s= 2023-10-11T16:07:13.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ao6Pvyz7hmgamGtKDwWSx+1av8s=', 'Content-Length': '0'} 2023-10-11T16:07:13.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050c22b6bb459e901-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:13.968 DEBUG:boto:path=/dghqvd-75/obj1 2023-10-11T16:07:13.968 DEBUG:boto:auth_path=/dghqvd-75/obj1 2023-10-11T16:07:13.969 DEBUG:boto:Method: GET 2023-10-11T16:07:13.969 DEBUG:boto:Path: /dghqvd-75/obj1 2023-10-11T16:07:13.969 DEBUG:boto:Data: 2023-10-11T16:07:13.970 DEBUG:boto:Headers: {} 2023-10-11T16:07:13.970 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:13.971 DEBUG:boto:Port: 8001 2023-10-11T16:07:13.971 DEBUG:boto:Params: {} 2023-10-11T16:07:13.971 DEBUG:boto:Token: None 2023-10-11T16:07:13.972 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:07:13 GMT /dghqvd-75/obj1 2023-10-11T16:07:13.972 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NyqBPZ/7kZHVmTU9X22fNgdgm+k= 2023-10-11T16:07:13.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:13 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NyqBPZ/7kZHVmTU9X22fNgdgm+k=', 'Content-Length': '0'} 2023-10-11T16:07:13.975 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014631e921d4be2c3-006526c831-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:13 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:14.018 DEBUG:boto:path=/dghqvd-75/obj2 2023-10-11T16:07:14.019 DEBUG:boto:auth_path=/dghqvd-75/obj2 2023-10-11T16:07:14.019 DEBUG:boto:Method: HEAD 2023-10-11T16:07:14.020 DEBUG:boto:Path: /dghqvd-75/obj2 2023-10-11T16:07:14.020 DEBUG:boto:Data: 2023-10-11T16:07:14.021 DEBUG:boto:Headers: {} 2023-10-11T16:07:14.021 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:14.021 DEBUG:boto:Port: 8001 2023-10-11T16:07:14.022 DEBUG:boto:Params: {} 2023-10-11T16:07:14.022 DEBUG:boto:Token: None 2023-10-11T16:07:14.023 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:14 GMT /dghqvd-75/obj2 2023-10-11T16:07:14.023 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fqQFntcJb5V4fN4HinyBBd8aMuo= 2023-10-11T16:07:14.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fqQFntcJb5V4fN4HinyBBd8aMuo=', 'Content-Length': '0'} 2023-10-11T16:07:14.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2a31f3bdc2623e4-006526c832-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:14.026 DEBUG:boto:path=/dghqvd-75/obj2 2023-10-11T16:07:14.027 DEBUG:boto:auth_path=/dghqvd-75/obj2 2023-10-11T16:07:14.027 DEBUG:boto:Method: GET 2023-10-11T16:07:14.028 DEBUG:boto:Path: /dghqvd-75/obj2 2023-10-11T16:07:14.028 DEBUG:boto:Data: 2023-10-11T16:07:14.028 DEBUG:boto:Headers: {} 2023-10-11T16:07:14.029 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:07:14.029 DEBUG:boto:Port: 8001 2023-10-11T16:07:14.030 DEBUG:boto:Params: {} 2023-10-11T16:07:14.030 DEBUG:boto:Token: None 2023-10-11T16:07:14.031 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:07:14 GMT /dghqvd-75/obj2 2023-10-11T16:07:14.031 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:NMQngfaNww+QP2PR09w5JnArhps= 2023-10-11T16:07:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:14 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NMQngfaNww+QP2PR09w5JnArhps=', 'Content-Length': '0'} 2023-10-11T16:07:14.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:05:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbb600f0051a76d1-006526c832-2b8c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:07:14 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:14.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.036 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.088 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.089 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.133 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:07:14.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.135 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.189 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.189 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:WARNING: cannot find source zone id for name=* 2023-10-11T16:07:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:WARNING: cannot find source zone id for name=* 2023-10-11T16:07:14.235 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:07:14.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.237 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.288 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.289 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.335 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:07:14.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.337 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.387 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.387 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.438 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:14.438 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:14.439 INFO:teuthology.orchestra.run.smithi158.stdout: "epoch": 22, 2023-10-11T16:07:14.439 INFO:teuthology.orchestra.run.smithi158.stdout: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:07:14.439 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": [], 2023-10-11T16:07:14.439 INFO:teuthology.orchestra.run.smithi158.stdout: "period_map": { 2023-10-11T16:07:14.440 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:14.440 INFO:teuthology.orchestra.run.smithi158.stdout: "zonegroups": [ 2023-10-11T16:07:14.440 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.440 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "a", 2023-10-11T16:07:14.440 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a", 2023-10-11T16:07:14.441 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:07:14.441 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": true, 2023-10-11T16:07:14.441 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.442 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:07:14.442 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.442 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:07:14.442 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:07:14.442 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:07:14.443 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.443 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.443 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:07:14.443 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a2", 2023-10-11T16:07:14.443 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.444 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:07:14.444 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.444 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:07:14.444 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:07:14.445 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:07:14.445 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:07:14.445 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:07:14.445 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:07:14.445 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:07:14.446 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:07:14.446 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:07:14.446 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:07:14.446 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.447 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.447 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.447 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.447 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:07:14.447 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "a1", 2023-10-11T16:07:14.448 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.448 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:07:14.448 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.448 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:07:14.449 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:07:14.449 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:07:14.449 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:07:14.449 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:07:14.450 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:07:14.450 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:07:14.450 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:07:14.450 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:07:14.451 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:07:14.451 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.451 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.451 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.452 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.452 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:07:14.452 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.452 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:07:14.453 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:07:14.453 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:07:14.453 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:07:14.453 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.453 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.454 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.454 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:07:14.454 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:07:14.454 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:07:14.455 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [ 2023-10-11T16:07:14.455 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.455 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-group", 2023-10-11T16:07:14.455 INFO:teuthology.orchestra.run.smithi158.stdout: "data_flow": { 2023-10-11T16:07:14.456 INFO:teuthology.orchestra.run.smithi158.stdout: "symmetrical": [ 2023-10-11T16:07:14.456 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.456 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-flow1", 2023-10-11T16:07:14.456 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.457 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:07:14.457 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:07:14.457 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.457 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.458 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.458 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.458 INFO:teuthology.orchestra.run.smithi158.stdout: "pipes": [ 2023-10-11T16:07:14.458 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.458 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-74", 2023-10-11T16:07:14.459 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.459 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:07:14.459 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.459 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:07:14.460 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.460 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.460 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:07:14.460 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:07:14.460 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.461 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:07:14.461 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.461 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.461 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:07:14.461 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.462 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:07:14.462 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:07:14.462 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.462 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.463 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:07:14.463 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:07:14.463 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:07:14.463 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:07:14.463 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.464 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.464 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.464 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe-dghqvd-75", 2023-10-11T16:07:14.464 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.465 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-74", 2023-10-11T16:07:14.465 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.465 INFO:teuthology.orchestra.run.smithi158.stdout: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:07:14.465 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.465 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.466 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:07:14.466 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "dghqvd-75", 2023-10-11T16:07:14.466 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.466 INFO:teuthology.orchestra.run.smithi158.stdout: "37b32c30-76c0-479d-b458-e4824d081f13" 2023-10-11T16:07:14.466 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.467 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.467 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:07:14.467 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.467 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:07:14.468 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:07:14.468 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.469 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.469 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:07:14.470 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:07:14.470 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:07:14.470 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:07:14.470 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.470 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.471 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.471 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "sync-pipe", 2023-10-11T16:07:14.471 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.471 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T16:07:14.472 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.472 INFO:teuthology.orchestra.run.smithi158.stdout: "*" 2023-10-11T16:07:14.472 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.472 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.472 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": { 2023-10-11T16:07:14.473 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket": "*", 2023-10-11T16:07:14.473 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.473 INFO:teuthology.orchestra.run.smithi158.stdout: "*" 2023-10-11T16:07:14.473 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.473 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.474 INFO:teuthology.orchestra.run.smithi158.stdout: "params": { 2023-10-11T16:07:14.474 INFO:teuthology.orchestra.run.smithi158.stdout: "source": { 2023-10-11T16:07:14.474 INFO:teuthology.orchestra.run.smithi158.stdout: "filter": { 2023-10-11T16:07:14.474 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [] 2023-10-11T16:07:14.475 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.475 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.475 INFO:teuthology.orchestra.run.smithi158.stdout: "dest": {}, 2023-10-11T16:07:14.475 INFO:teuthology.orchestra.run.smithi158.stdout: "priority": 0, 2023-10-11T16:07:14.476 INFO:teuthology.orchestra.run.smithi158.stdout: "mode": "system", 2023-10-11T16:07:14.476 INFO:teuthology.orchestra.run.smithi158.stdout: "user": "" 2023-10-11T16:07:14.476 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.476 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.476 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.477 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "allowed" 2023-10-11T16:07:14.477 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.477 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.477 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.477 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:07:14.478 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.478 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.478 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.478 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.479 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "b", 2023-10-11T16:07:14.479 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b", 2023-10-11T16:07:14.479 INFO:teuthology.orchestra.run.smithi158.stdout: "api_name": "", 2023-10-11T16:07:14.479 INFO:teuthology.orchestra.run.smithi158.stdout: "is_master": false, 2023-10-11T16:07:14.479 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.480 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:07:14.480 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.480 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames": [], 2023-10-11T16:07:14.480 INFO:teuthology.orchestra.run.smithi158.stdout: "hostnames_s3website": [], 2023-10-11T16:07:14.480 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:07:14.481 INFO:teuthology.orchestra.run.smithi158.stdout: "zones": [ 2023-10-11T16:07:14.481 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.481 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:07:14.482 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b1", 2023-10-11T16:07:14.483 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.483 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:07:14.483 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.484 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:07:14.484 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:07:14.484 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:07:14.484 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:07:14.484 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:07:14.485 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:07:14.485 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:07:14.485 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:07:14.485 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:07:14.486 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:07:14.486 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.486 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.486 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.486 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.487 INFO:teuthology.orchestra.run.smithi158.stdout: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:07:14.487 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "b2", 2023-10-11T16:07:14.487 INFO:teuthology.orchestra.run.smithi158.stdout: "endpoints": [ 2023-10-11T16:07:14.487 INFO:teuthology.orchestra.run.smithi158.stdout: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:07:14.488 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.488 INFO:teuthology.orchestra.run.smithi158.stdout: "log_meta": false, 2023-10-11T16:07:14.488 INFO:teuthology.orchestra.run.smithi158.stdout: "log_data": true, 2023-10-11T16:07:14.488 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_index_max_shards": 11, 2023-10-11T16:07:14.488 INFO:teuthology.orchestra.run.smithi158.stdout: "read_only": false, 2023-10-11T16:07:14.489 INFO:teuthology.orchestra.run.smithi158.stdout: "tier_type": "", 2023-10-11T16:07:14.489 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from_all": true, 2023-10-11T16:07:14.489 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_from": [], 2023-10-11T16:07:14.489 INFO:teuthology.orchestra.run.smithi158.stdout: "redirect_zone": "", 2023-10-11T16:07:14.489 INFO:teuthology.orchestra.run.smithi158.stdout: "supported_features": [ 2023-10-11T16:07:14.490 INFO:teuthology.orchestra.run.smithi158.stdout: "compress-encrypted", 2023-10-11T16:07:14.490 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.490 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.490 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: "placement_targets": [ 2023-10-11T16:07:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: "name": "default-placement", 2023-10-11T16:07:14.492 INFO:teuthology.orchestra.run.smithi158.stdout: "tags": [], 2023-10-11T16:07:14.492 INFO:teuthology.orchestra.run.smithi158.stdout: "storage_classes": [ 2023-10-11T16:07:14.492 INFO:teuthology.orchestra.run.smithi158.stdout: "STANDARD" 2023-10-11T16:07:14.492 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.492 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.493 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.493 INFO:teuthology.orchestra.run.smithi158.stdout: "default_placement": "default-placement", 2023-10-11T16:07:14.493 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:07:14.493 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_policy": { 2023-10-11T16:07:14.493 INFO:teuthology.orchestra.run.smithi158.stdout: "groups": [] 2023-10-11T16:07:14.494 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.494 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled_features": [ 2023-10-11T16:07:14.494 INFO:teuthology.orchestra.run.smithi158.stdout: "resharding" 2023-10-11T16:07:14.494 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.495 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.495 INFO:teuthology.orchestra.run.smithi158.stdout: ], 2023-10-11T16:07:14.495 INFO:teuthology.orchestra.run.smithi158.stdout: "short_zone_ids": [ 2023-10-11T16:07:14.495 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.495 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:07:14.496 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2025758522 2023-10-11T16:07:14.496 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.496 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.496 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:07:14.496 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 3270403816 2023-10-11T16:07:14.497 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.497 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.497 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:07:14.497 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 178712359 2023-10-11T16:07:14.498 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.498 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:14.498 INFO:teuthology.orchestra.run.smithi158.stdout: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:07:14.498 INFO:teuthology.orchestra.run.smithi158.stdout: "val": 2663201345 2023-10-11T16:07:14.498 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.499 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:14.499 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.499 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zonegroup": "a", 2023-10-11T16:07:14.499 INFO:teuthology.orchestra.run.smithi158.stdout: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:07:14.500 INFO:teuthology.orchestra.run.smithi158.stdout: "period_config": { 2023-10-11T16:07:14.500 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_quota": { 2023-10-11T16:07:14.500 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:07:14.500 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:07:14.500 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:07:14.501 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:07:14.501 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:07:14.501 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.501 INFO:teuthology.orchestra.run.smithi158.stdout: "user_quota": { 2023-10-11T16:07:14.502 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false, 2023-10-11T16:07:14.502 INFO:teuthology.orchestra.run.smithi158.stdout: "check_on_raw": false, 2023-10-11T16:07:14.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size": -1, 2023-10-11T16:07:14.502 INFO:teuthology.orchestra.run.smithi158.stdout: "max_size_kb": 0, 2023-10-11T16:07:14.503 INFO:teuthology.orchestra.run.smithi158.stdout: "max_objects": -1 2023-10-11T16:07:14.503 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.503 INFO:teuthology.orchestra.run.smithi158.stdout: "user_ratelimit": { 2023-10-11T16:07:14.503 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:07:14.503 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:07:14.504 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:07:14.504 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:07:14.504 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:07:14.504 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.504 INFO:teuthology.orchestra.run.smithi158.stdout: "bucket_ratelimit": { 2023-10-11T16:07:14.505 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:07:14.505 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:07:14.505 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:07:14.505 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:07:14.505 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:07:14.506 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.506 INFO:teuthology.orchestra.run.smithi158.stdout: "anonymous_ratelimit": { 2023-10-11T16:07:14.506 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_ops": 0, 2023-10-11T16:07:14.506 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_ops": 0, 2023-10-11T16:07:14.507 INFO:teuthology.orchestra.run.smithi158.stdout: "max_read_bytes": 0, 2023-10-11T16:07:14.507 INFO:teuthology.orchestra.run.smithi158.stdout: "max_write_bytes": 0, 2023-10-11T16:07:14.507 INFO:teuthology.orchestra.run.smithi158.stdout: "enabled": false 2023-10-11T16:07:14.507 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:14.507 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:14.508 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:07:14.508 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_name": "test-realm", 2023-10-11T16:07:14.508 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:14.508 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:14.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.510 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.557 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.557 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.609 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:07:14.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.611 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.680 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.817 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:07:14.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.819 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:14.900 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:14.901 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:14.991 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:07:14.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-74', '--source-bucket=*', '--dest-bucket=dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-74', '--source-bucket=*', '--dest-bucket=dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:14.992 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dghqvd-74 --source-zones=a1 --dest-zones=a2 --bucket=dghqvd-74 '--source-bucket=*' --dest-bucket=dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:15.066 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.066 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.150 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T16:07:15.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-74', '--source-bucket=*', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dghqvd-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dghqvd-74', '--source-bucket=*', '--dest-bucket=dghqvd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.152 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dghqvd-75 --source-zones=a1 --dest-zones=a2 --bucket=dghqvd-74 '--source-bucket=*' --dest-bucket=dghqvd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:15.217 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.218 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.272 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-10-11T16:07:15.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.274 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:15.327 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.327 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.381 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-10-11T16:07:15.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:07:15.382 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:07:15.428 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.428 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.473 INFO:teuthology.orchestra.run.smithi158.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dghqvd-74","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dghqvd-75","source":{"bucket":"dghqvd-74","zones":["a1"]},"dest":{"bucket":"dghqvd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:07:15.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:15.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:15.476 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:15.524 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.525 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.554 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:07:15.567 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:07:15.568 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.568 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:15.568 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:07:15.568 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.569 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.569 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:15.569 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:07:15.569 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.569 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.570 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:15.570 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:07:15.570 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.570 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.570 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040435.366237_1054.1", 2023-10-11T16:07:15.571 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:07:15.366237Z" 2023-10-11T16:07:15.571 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.571 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:07:15.572 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:15.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:07:15.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:15.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:15.574 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:15.623 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:15.624 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:15.653 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:15.652+0000 7f85d1659780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:15.655 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:15.653+0000 7f85d1659780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:15.668 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:15.668 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:15.669 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.670 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:15.670 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.670 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:15.671 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:15.672 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:15.673 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:15.674 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.675 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:15.676 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:15.677 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.678 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:15.679 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:15.679 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:15.679 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:15.679 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:15.680 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:15.680 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:15.680 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:15.680 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:15.680 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.681 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:15.682 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:15.682 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:15.682 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:15.682 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:15Z" 2023-10-11T16:07:15.683 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:15.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:15Z" } 2023-10-11T16:07:15.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:15.685 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:15.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:15.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:20.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:20.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:20.688 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:20.745 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:20.746 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:20.778 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:20.777+0000 7f5348d3e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:20.779 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:20.778+0000 7f5348d3e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:20.793 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:20.793 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.794 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:20.795 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:20.796 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:20.797 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:20.798 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:20.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:20.800 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:20.801 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.802 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:20.803 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:20.804 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:20.805 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:20.806 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:20Z" 2023-10-11T16:07:20.806 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:20.807 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:20Z" } 2023-10-11T16:07:20.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:20.808 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:20.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:20.809 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:25.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:25.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:25.811 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:25.868 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:25.868 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:25.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:25.901+0000 7f7adec82780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:25.903 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:25.902+0000 7f7adec82780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:25.918 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:25.918 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:25.919 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:25.920 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:25.921 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:25.922 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:25.922 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:25.922 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.922 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:25.922 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:25.923 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:25.923 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:25.923 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:25.923 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:25.924 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:25.924 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:25.924 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:25.924 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:25.924 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:25.925 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.925 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:25.925 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:25.925 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:25.925 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:25.926 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:25.926 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:25.926 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:25.926 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:25.927 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:25.927 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:25.927 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:25.928 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.928 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:25.928 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:25.928 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:25.928 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:25.929 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:25.929 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:25.929 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:25.929 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:25.930 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:25.930 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:25.930 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:25.930 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:25.930 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:25.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.931 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:25.931 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:25.931 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:25.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:25.932 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:25Z" 2023-10-11T16:07:25.932 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:25.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:25Z" } 2023-10-11T16:07:25.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:25.934 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:25.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:25.935 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:30.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:30.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:30.937 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:30.993 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:30.993 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:31.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:31.023+0000 7feb4e373780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:31.025 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:31.024+0000 7feb4e373780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:31.039 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:31.039 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:31.040 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:31.041 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:31.042 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:31.043 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:31.044 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:31.045 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:31.046 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:31.047 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.048 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:31.049 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:31.050 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.051 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:31.052 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:31Z" 2023-10-11T16:07:31.052 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:31.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:31Z" } 2023-10-11T16:07:31.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:31.055 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:31.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:31.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:36.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:36.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:36.058 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:36.115 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:36.147 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:36.146+0000 7f7a11615780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:36.148 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:36.147+0000 7f7a11615780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:36.162 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:36.163 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:36.163 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:36.163 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:36.164 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:36.164 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:36.164 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:36.164 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.164 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:36.165 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:36.165 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:36.165 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:36.165 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:36.165 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:36.166 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:36.166 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:36.166 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:36.166 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:36.167 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:36.167 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:36.167 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.167 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:36.168 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:36.168 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:36.168 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:36.168 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:36.169 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:36.169 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:36.169 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:36.169 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:36.170 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:36.173 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:36.173 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.174 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:36.174 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:36.174 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:36.174 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:36.174 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:36.175 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:36.175 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:36.175 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:36.175 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:36.176 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:36.176 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:36.176 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.176 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:36.176 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:36.177 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:36.177 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:36.177 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:36.177 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:36.177 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:36.178 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:36.178 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:36.178 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:36.178 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:36.178 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:36.179 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:36.179 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.179 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:36.179 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:36.179 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:36.180 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:36.180 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:36Z" 2023-10-11T16:07:36.180 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:36.181 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:36Z" } 2023-10-11T16:07:36.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:36.182 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:36.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:36.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:41.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:41.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:41.185 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:41.241 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:41.242 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:41.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:41.272+0000 7febb5769780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:41.274 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:41.273+0000 7febb5769780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:41.288 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:41.288 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:41.288 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:41.289 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:41.289 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:41.289 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:41.289 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:41.289 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.290 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:41.290 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:41.290 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:41.290 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:41.291 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:41.291 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:41.291 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:41.291 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:41.291 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:41.292 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:41.292 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:41.292 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:41.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.292 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:41.293 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:41.293 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:41.293 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:41.293 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:41.294 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:41.294 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:41.294 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:41.294 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:41.294 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:41.295 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:41.295 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.295 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:41.295 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:41.296 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:41.296 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:41.296 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:41.296 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:41.296 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:41.297 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:41.297 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:41.297 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:41.297 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:41.297 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.298 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:41.298 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:41.298 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:41.298 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:41.298 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040334.579755_1026.1", 2023-10-11T16:07:41.299 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:41.299 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:41.299 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:41.299 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:34.579755Z", 2023-10-11T16:07:41.299 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:41.300 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:41.300 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:41.300 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:41.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.301 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:41.301 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:41.301 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:41.301 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:41.301 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:41Z" 2023-10-11T16:07:41.302 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:41.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040334.579755_1026.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:34.579755Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:41Z" } 2023-10-11T16:07:41.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040334.579755_1026.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:34.579755Z', 'realm_epoch': 2}}] 2023-10-11T16:07:41.304 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:41.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040334.579755_1026.1'} 2023-10-11T16:07:41.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1697040435.366237_1054.1 target=1_1697040334.579755_1026.1 2023-10-11T16:07:46.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.308 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:46.365 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:46.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:46.401 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:46.399+0000 7f6007f45780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:46.402 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:46.400+0000 7f6007f45780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:46.415 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:46.415 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:46.415 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:46.416 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:46.416 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:46.416 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:46.416 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.417 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.417 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:46.417 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.417 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:46.417 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.418 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.418 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:46.418 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:46.419 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.419 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:46.419 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.419 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.419 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.420 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.420 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:46.420 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.420 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.420 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:46.421 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.421 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.421 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.421 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:46.421 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.422 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.423 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:46.425 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.425 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.426 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.426 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:46.426 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.426 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.427 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040435.366237_1054.1", 2023-10-11T16:07:46.427 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.427 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.427 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.427 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:07:15.366237Z", 2023-10-11T16:07:46.428 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.428 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.428 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:46.429 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.429 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:46.429 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:46.429 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:46.429 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.430 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:46Z" 2023-10-11T16:07:46.430 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:46.431 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040435.366237_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:07:15.366237Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:46Z" } 2023-10-11T16:07:46.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040435.366237_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:07:15.366237Z', 'realm_epoch': 2}}] 2023-10-11T16:07:46.432 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:46.432 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:46.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:07:46.433 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:07:46.434 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:07:46.435 DEBUG:boto:Method: HEAD 2023-10-11T16:07:46.435 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:07:46.435 DEBUG:boto:Data: 2023-10-11T16:07:46.436 DEBUG:boto:Headers: {} 2023-10-11T16:07:46.437 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:07:46.438 DEBUG:boto:Port: 8000 2023-10-11T16:07:46.438 DEBUG:boto:Params: {} 2023-10-11T16:07:46.440 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:07:46.441 DEBUG:boto:Token: None 2023-10-11T16:07:46.442 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:46 GMT /dghqvd-74/ 2023-10-11T16:07:46.442 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a+4sRoOJufmO+0XwPU8EbfwTPlE= 2023-10-11T16:07:46.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a+4sRoOJufmO+0XwPU8EbfwTPlE=', 'Content-Length': '0'} 2023-10-11T16:07:46.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adef15d99f4d5cf8-006526c852-2c5b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:46.450 DEBUG:boto:path=/dghqvd-74/obj3 2023-10-11T16:07:46.451 DEBUG:boto:auth_path=/dghqvd-74/obj3 2023-10-11T16:07:46.451 DEBUG:boto:Method: PUT 2023-10-11T16:07:46.452 DEBUG:boto:Path: /dghqvd-74/obj3 2023-10-11T16:07:46.452 DEBUG:boto:Data: 2023-10-11T16:07:46.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:07:46.453 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:07:46.453 DEBUG:boto:Port: 8000 2023-10-11T16:07:46.454 DEBUG:boto:Params: {} 2023-10-11T16:07:46.454 DEBUG:boto:Token: None 2023-10-11T16:07:46.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:07:46 GMT /dghqvd-74/obj3 2023-10-11T16:07:46.455 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:15TVxjiTmKRsFtxEPzIZMvw+9rI= 2023-10-11T16:07:46.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:07:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:15TVxjiTmKRsFtxEPzIZMvw+9rI='} 2023-10-11T16:07:46.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c0fca4e31180b38-006526c852-2c5b-a1'), ('Date', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:46.466 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:07:46.466 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:07:46.467 DEBUG:boto:Method: HEAD 2023-10-11T16:07:46.467 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:07:46.468 DEBUG:boto:Data: 2023-10-11T16:07:46.468 DEBUG:boto:Headers: {} 2023-10-11T16:07:46.469 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:07:46.469 DEBUG:boto:Port: 8000 2023-10-11T16:07:46.470 DEBUG:boto:Params: {} 2023-10-11T16:07:46.470 DEBUG:boto:Token: None 2023-10-11T16:07:46.471 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:07:46 GMT /dghqvd-74/ 2023-10-11T16:07:46.471 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:a+4sRoOJufmO+0XwPU8EbfwTPlE= 2023-10-11T16:07:46.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:07:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a+4sRoOJufmO+0XwPU8EbfwTPlE=', 'Content-Length': '0'} 2023-10-11T16:07:46.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b88183a99bf9d22-006526c852-2c5b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:46.475 DEBUG:boto:path=/dghqvd-74/obj4 2023-10-11T16:07:46.475 DEBUG:boto:auth_path=/dghqvd-74/obj4 2023-10-11T16:07:46.475 DEBUG:boto:Method: PUT 2023-10-11T16:07:46.476 DEBUG:boto:Path: /dghqvd-74/obj4 2023-10-11T16:07:46.476 DEBUG:boto:Data: 2023-10-11T16:07:46.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:07:46.477 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:07:46.477 DEBUG:boto:Port: 8000 2023-10-11T16:07:46.478 DEBUG:boto:Params: {} 2023-10-11T16:07:46.478 DEBUG:boto:Token: None 2023-10-11T16:07:46.479 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 11 Oct 2023 16:07:46 GMT /dghqvd-74/obj4 2023-10-11T16:07:46.479 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:sgV8SBjjPFX9Uiux0Ow6e2cRK3I= 2023-10-11T16:07:46.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 16:07:46 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sgV8SBjjPFX9Uiux0Ow6e2cRK3I='} 2023-10-11T16:07:46.488 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088a2a000b6e4d956-006526c852-2c5b-a1'), ('Date', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:07:46.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.490 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:46.539 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:46.540 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:46.570 INFO:teuthology.orchestra.run.smithi158.stderr:No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:07:46.583 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:07:46.584 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.584 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:46.584 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:07:46.584 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.585 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.585 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:46.585 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "0.000000" 2023-10-11T16:07:46.585 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.586 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.586 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:46.586 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:07:46.586 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.586 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.587 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040435.366237_1054.1", 2023-10-11T16:07:46.587 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:07:15.366237Z" 2023-10-11T16:07:46.587 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.587 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:07:46.588 DEBUG:rgw_multi.tests:master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:46.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-10-11T16:07:46.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.590 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:46.641 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:46.642 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:46.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:46.667+0000 7f1d81420780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:46.669 INFO:teuthology.orchestra.run.smithi158.stderr:2023-10-11T16:07:46.668+0000 7f1d81420780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:07:46.682 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:46.682 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:46.683 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:46.683 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:46.683 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:46.683 INFO:teuthology.orchestra.run.smithi158.stdout: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:07:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:46.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:07:46.685 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 1, 2023-10-11T16:07:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:07:46.686 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.687 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "", 2023-10-11T16:07:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.688 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "0.000000", 2023-10-11T16:07:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.689 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.690 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:07:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.691 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:07:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.692 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "state": 1, 2023-10-11T16:07:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "1_1697040435.366237_1054.1", 2023-10-11T16:07:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.693 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:07:15.366237Z", 2023-10-11T16:07:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: "realm_epoch": 2 2023-10-11T16:07:46.694 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.695 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 1, 2023-10-11T16:07:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 1 2023-10-11T16:07:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.696 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:46Z" 2023-10-11T16:07:46.696 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:46.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "064106cf-4523-427d-9a69-628884ec70af", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1697034656.410656_41.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-10-11T14:30:56.410656Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1697038064.387448_379.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T15:27:44.387448Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1697040435.366237_1054.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:07:15.366237Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-10-11T16:07:46Z" } 2023-10-11T16:07:46.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697040435.366237_1054.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:07:15.366237Z', 'realm_epoch': 2}}] 2023-10-11T16:07:46.699 DEBUG:rgw_multi.tests:log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:46.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697040435.366237_1054.1'} 2023-10-11T16:07:46.700 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-10-11T16:07:46.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.701 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:46.749 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:46.797 INFO:teuthology.orchestra.run.smithi158.stdout:[ 2023-10-11T16:07:46.797 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:07:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:07:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.798 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.799 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032764", 2023-10-11T16:07:46.799 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:07:46.486519Z" 2023-10-11T16:07:46.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.799 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.800 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052371", 2023-10-11T16:07:46.800 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:07:46.464084Z" 2023-10-11T16:07:46.800 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.800 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.800 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:46.801 INFO:teuthology.orchestra.run.smithi158.stdout: "last_update": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:46.801 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.801 INFO:teuthology.orchestra.run.smithi158.stdout:] 2023-10-11T16:07:46.802 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:46.803 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:07:46.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:46.804 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:46.855 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:46.856 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:46.898 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:46.899 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:46.899 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:46.899 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:46.900 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:46.900 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:46.900 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.900 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:46.900 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.901 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:46.901 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.901 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:46.901 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:07:46.902 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.902 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.902 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.902 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:07:46.903 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.903 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.903 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.903 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:46.904 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.904 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:46.904 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:46.904 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.905 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:46.905 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.905 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:46.905 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.905 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.906 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.906 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:46.906 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.906 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:46.906 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:07:46.907 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.907 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:46.907 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.907 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:07:46.908 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.908 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.908 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:46.908 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:46.908 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:46.909 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:46.909 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:46.909 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:46.909 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:46.909 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:46.910 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:46.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.910 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:46.910 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:46.910 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.911 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:46.911 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:46.911 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:46.911 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:46.912 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:46Z" 2023-10-11T16:07:46.912 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:46.913 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:46Z" } 2023-10-11T16:07:46.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:46.914 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:46.914 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:46.915 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:07:51.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:51.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:51.917 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:51.976 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:51.977 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:52.025 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:52.026 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:52.026 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:52.026 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:52.026 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:52.027 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:52.027 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.027 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:52.027 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:52.027 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:52.028 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:52.028 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:52.028 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:07:52.028 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:52.029 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:52.029 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:52.029 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:07:52.029 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:52.029 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.030 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:52.030 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:52.030 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:52.030 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:52.031 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:52.031 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:52.031 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:52.031 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:52.031 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:52.032 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:52.032 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.032 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:52.032 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:52.032 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:52.033 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:52.033 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:07:52.033 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:52.033 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:52.034 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:52.034 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:07:52.034 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:52.034 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.034 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:52.035 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:52.035 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:52.035 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:52.035 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:52.035 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:52.036 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:52.037 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:52.037 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.037 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:52.037 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:52.037 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:52.038 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:52.038 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:52Z" 2023-10-11T16:07:52.038 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:52.039 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:52Z" } 2023-10-11T16:07:52.040 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:52.040 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:52.041 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:52.041 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:07:57.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:57.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:07:57.043 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:07:57.103 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:07:57.103 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:07:57.153 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:07:57.154 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:07:57.154 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:07:57.154 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:07:57.155 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:07:57.155 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:07:57.155 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.155 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:07:57.156 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:57.156 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:07:57.156 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:57.156 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:57.156 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:07:57.157 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:57.157 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:57.157 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:57.157 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:07:57.157 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:57.158 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.158 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:57.158 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:07:57.158 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:57.159 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:57.159 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:07:57.159 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:57.159 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:07:57.159 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:57.160 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:07:57.160 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:57.160 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.160 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:57.160 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:07:57.161 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:57.161 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:57.161 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:07:57.161 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:57.161 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:07:57.162 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:57.162 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:07:57.162 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:57.162 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.163 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:07:57.163 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:07:57.163 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:07:57.163 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:07:57.164 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:07:57.164 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:07:57.164 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:07:57.164 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:07:57.165 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:07:57.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:57.165 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:07:57.165 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:07:57.165 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.166 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:07:57.166 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:07:57.166 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:07:57.166 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:07:57.166 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:07:57Z" 2023-10-11T16:07:57.167 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:07:57.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:07:57Z" } 2023-10-11T16:07:57.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:07:57.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:57.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:07:57.170 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:02.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:02.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:02.173 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:02.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:02.233 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:02.285 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:02.285 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:02.285 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:02.286 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:02.286 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:02.286 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:02.286 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.287 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:02.287 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:02.287 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:02.287 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:02.288 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:02.288 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:02.288 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:02.288 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:02.289 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:02.289 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:02.289 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:02.289 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.290 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:02.290 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:02.290 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:02.290 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:02.290 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:02.291 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:02.291 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:02.291 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:02.291 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:02.292 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:02.292 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.292 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:02.292 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:02.292 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:02.293 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:02.293 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:02.293 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:02.293 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:02.294 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:02.294 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:02.294 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:02.294 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.295 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:02.295 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:02.295 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:02.296 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:02.296 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:02.296 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:02.297 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:02.297 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:02.298 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:02.298 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:02.298 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:02.299 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:02.299 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.299 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:02.300 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:02.300 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:02.300 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:02.301 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:02Z" 2023-10-11T16:08:02.301 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:02.302 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:02Z" } 2023-10-11T16:08:02.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:02.303 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:02.304 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:02.304 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:07.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:07.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:07.307 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:07.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:07.366 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:07.414 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:07.415 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:07.416 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:07.417 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:07.418 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:07.419 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:07.420 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:07.421 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:07.422 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.423 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:07.424 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:07.425 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:07.426 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:07.427 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:07Z" 2023-10-11T16:08:07.427 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:07.428 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:07Z" } 2023-10-11T16:08:07.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:07.429 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:07.429 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:07.430 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:12.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:12.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:12.432 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:12.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:12.492 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:12.542 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:12.542 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:12.542 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:12.542 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:12.543 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:12.543 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:12.543 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.543 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:12.543 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:12.544 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:12.544 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:12.544 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:12.545 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:12.545 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:12.545 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:12.545 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:12.546 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:12.546 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:12.546 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.546 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:12.547 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:12.547 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:12.547 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:12.547 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:12.547 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:12.548 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:12.548 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:12.548 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:12.548 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:12.548 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.549 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:12.549 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:12.549 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:12.549 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:12.549 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:12.550 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:12.550 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:12.550 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:12.550 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:12.550 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:12.551 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.551 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:12.551 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:12.551 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:12.551 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:12.552 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:12.552 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:12.552 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:12.552 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:12.552 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:12.553 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:12.553 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:12.553 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:12.553 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.553 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:12.554 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:12.554 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:12.554 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:12.554 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:12Z" 2023-10-11T16:08:12.555 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:12.555 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:12Z" } 2023-10-11T16:08:12.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:12.556 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:12.557 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:12.557 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:17.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:17.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:17.559 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:17.618 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:17.619 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:17.666 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:17.666 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:17.666 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:17.666 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:17.667 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:17.667 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:17.667 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.667 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:17.668 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:17.668 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:17.668 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:17.668 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:17.669 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:17.669 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:17.669 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:17.669 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:17.670 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:17.670 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:17.670 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.670 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:17.671 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:17.671 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:17.671 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:17.671 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:17.671 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:17.672 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:17.672 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:17.672 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:17.672 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:17.672 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.673 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:17.673 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:17.673 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:17.674 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:17.674 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:17.674 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:17.674 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:17.675 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:17.675 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:17.675 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:17.675 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.676 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:17.676 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:17.676 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:17.677 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:17.677 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:17.677 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:17.677 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:17.677 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:17.678 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:17.678 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:17.678 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:17.678 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:17.678 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.679 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:17.679 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:17.679 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:17.679 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:17.679 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:17Z" 2023-10-11T16:08:17.680 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:17.681 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:17Z" } 2023-10-11T16:08:17.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:17.682 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:17.682 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:17.683 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:22.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:22.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:22.685 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:22.746 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:22.747 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:22.794 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:22.795 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:22.795 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:22.795 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:22.796 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:22.796 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:22.796 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.796 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:22.796 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:22.797 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:22.797 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:22.797 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:22.797 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:22.797 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:22.798 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:22.798 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:22.798 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:22.798 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:22.799 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.799 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:22.799 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:22.799 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:22.799 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:22.800 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:22.800 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:22.800 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:22.800 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:22.800 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:22.801 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:22.801 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.801 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:22.801 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:22.801 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:22.802 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:22.802 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:22.802 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:22.802 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:22.802 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:22.803 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:22.803 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:22.803 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.803 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:22.803 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:22.804 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:22.804 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:22.804 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:22.804 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:22.804 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:22.806 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.806 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:22.806 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:22.806 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:22.807 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:22.807 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:22Z" 2023-10-11T16:08:22.807 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:22.808 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:22Z" } 2023-10-11T16:08:22.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:22.809 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:22.809 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:22.810 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:27.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:27.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:27.813 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:27.872 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:27.873 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:27.922 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:27.923 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:27.923 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:27.923 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:27.923 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:27.924 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:27.924 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.924 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:27.924 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:27.924 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:27.925 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:27.925 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:27.925 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:27.925 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:27.926 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:27.926 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:27.926 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:27.926 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:27.926 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.927 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:27.927 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:27.927 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:27.927 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:27.927 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:27.928 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:27.928 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:27.928 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:27.928 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:27.929 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:27.929 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.929 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:27.929 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:27.929 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:27.930 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:27.930 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:27.930 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:27.930 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:27.930 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:27.931 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:27.931 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:27.931 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.931 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:27.932 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:27.933 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:27.933 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:27.933 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:27.933 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:27.933 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:27.934 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.934 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:27.934 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:27.934 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:27.934 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:27.935 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:27Z" 2023-10-11T16:08:27.935 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:27.936 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:27Z" } 2023-10-11T16:08:27.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:27.937 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:27.937 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:27.937 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:32.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:32.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:32.940 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:33.001 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:33.049 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:33.050 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:33.050 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:33.050 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:33.050 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:33.050 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:33.051 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.051 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:33.051 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:33.051 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:33.051 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:33.052 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:33.052 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:33.052 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:33.052 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:33.053 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:33.053 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:33.053 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:33.053 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.053 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:33.054 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:33.054 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:33.054 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:33.054 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:33.055 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:33.055 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:33.055 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:33.055 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:33.055 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:33.056 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.056 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:33.056 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:33.056 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:33.056 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:33.057 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:33.057 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:33.057 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:33.057 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:33.057 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:33.058 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:33.058 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.058 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:33.058 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:33.059 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:33.059 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:33.059 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:33.059 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:33.059 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:33.060 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:33.060 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:33.061 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:33.061 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:33.061 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:33.062 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.062 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:33.062 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:33.062 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:33.063 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:33.063 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:33Z" 2023-10-11T16:08:33.063 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:33.064 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:33Z" } 2023-10-11T16:08:33.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:33.065 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:33.066 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:33.066 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:38.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:38.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:38.069 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:38.128 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:38.128 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:38.177 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:38.177 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:38.177 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:38.177 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:38.178 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:38.178 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:38.178 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.178 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:38.179 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:38.179 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:38.179 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:38.179 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:38.179 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:38.180 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:38.180 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:38.180 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:38.180 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:38.180 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:38.181 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.181 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:38.181 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:38.181 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:38.181 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:38.182 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:38.182 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:38.182 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:38.182 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:38.182 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:38.183 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:38.183 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.183 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:38.183 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:38.183 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:38.184 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:38.184 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:38.184 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:38.184 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:38.184 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:38.185 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:38.185 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:38.185 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.185 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:38.185 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:38.186 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:38.186 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:38.186 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:38.186 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:38.186 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:38.187 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:38.187 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:38.187 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:38.187 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:38.188 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:38.188 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.188 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:38.188 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:38.188 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:38.189 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:38.189 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:38Z" 2023-10-11T16:08:38.189 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:38.190 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:38Z" } 2023-10-11T16:08:38.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:38.191 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:38.191 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:38.192 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:43.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:43.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:43.195 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:43.254 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:43.254 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:43.302 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:43.302 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:43.303 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:43.303 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:43.303 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:43.303 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:43.304 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.304 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:43.304 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:43.304 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:43.305 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:43.305 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:43.305 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:43.305 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:43.305 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:43.306 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:43.306 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:43.306 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:43.306 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:43.307 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:43.308 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:43.308 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:43.308 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:43.308 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:43.308 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.309 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:43.309 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:43.309 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:43.309 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:43.310 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:43.310 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:43.310 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:43.310 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:43.311 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:43.311 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:43.311 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.311 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:43.311 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:43.312 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:43.312 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:43.312 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:43.312 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:43.312 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:43.313 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:43.313 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:43.313 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:43.313 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:43.314 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:43.314 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.314 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:43.314 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:43.314 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:43.315 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:43.315 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:43Z" 2023-10-11T16:08:43.315 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:43.316 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:43Z" } 2023-10-11T16:08:43.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:43.317 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:43.317 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:43.318 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:48.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:48.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:48.320 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:48.382 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:48.431 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:48.432 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:48.432 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:48.432 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:48.432 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:48.433 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:48.433 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.433 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:48.434 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:48.434 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:48.434 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:48.434 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:48.435 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:48.435 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:48.435 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:48.435 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:48.436 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:48.436 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:48.436 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.436 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:48.436 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:48.437 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:48.437 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:48.437 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:48.437 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:48.437 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:48.438 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:48.438 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:48.439 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:48.439 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.439 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:48.439 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:48.439 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:48.440 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:48.440 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:48.440 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:48.440 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:48.440 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:48.441 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:48.441 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:48.441 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.441 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:48.442 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:48.442 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:48.442 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:48.442 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:48.443 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:48.443 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:48.443 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:48.443 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:48.443 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:48.444 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:48.444 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:48.444 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.444 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:48.444 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:48.445 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:48.445 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:48.445 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:48Z" 2023-10-11T16:08:48.445 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:48.446 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:48Z" } 2023-10-11T16:08:48.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:48.447 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:48.448 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:48.448 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:53.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:53.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:53.451 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:53.508 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:53.509 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:53.555 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:53.555 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:53.555 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:53.556 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:53.556 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:53.556 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:53.556 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.556 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:53.557 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:53.557 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:53.557 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:53.557 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:53.558 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:53.558 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:53.558 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:53.558 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:53.558 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:53.559 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:53.559 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.559 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:53.559 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:53.560 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:53.560 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:53.560 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:53.560 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:53.560 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:53.561 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:53.561 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:53.561 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:53.561 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.562 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:53.562 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:53.562 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:53.562 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:53.563 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:53.563 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:53.563 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:53.563 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:53.563 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:53.564 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:53.564 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.564 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:53.564 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:53.564 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:53.565 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:53.565 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:53.565 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:53.565 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:53.566 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:53.566 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:53.566 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:53.566 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:53.566 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:53.567 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.567 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:53.567 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:53.567 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:53.568 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:53.568 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:53Z" 2023-10-11T16:08:53.568 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:53.569 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:53Z" } 2023-10-11T16:08:53.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:53.570 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:53.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:53.571 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:08:58.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:58.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:08:58.574 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:08:58.633 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:08:58.634 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:08:58.682 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:08:58.682 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:08:58.682 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:08:58.683 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:08:58.683 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:08:58.683 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:08:58.683 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.684 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:08:58.684 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:58.684 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:08:58.684 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:58.684 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:58.685 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:08:58.685 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:58.685 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:58.685 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:58.686 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:08:58.686 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:58.686 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.686 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:58.686 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:08:58.687 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:58.687 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:58.687 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:08:58.687 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:58.688 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:08:58.688 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:58.688 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:08:58.688 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:58.688 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.689 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:58.689 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:08:58.689 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:58.689 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:58.690 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:08:58.690 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:58.690 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:08:58.690 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:08:58.691 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:08:58.692 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:08:58.692 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:08:58.692 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:08:58.692 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:08:58.693 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:08:58.693 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:08:58.693 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:58.693 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:08:58.694 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:08:58.694 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.694 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:08:58.694 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:08:58.694 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:08:58.695 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:08:58.695 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:08:58Z" 2023-10-11T16:08:58.695 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:08:58.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:08:58Z" } 2023-10-11T16:08:58.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:08:58.697 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:58.697 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:08:58.698 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:09:03.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:03.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:03.701 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:03.759 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:03.807 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:09:03.807 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:09:03.807 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:09:03.808 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:09:03.808 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:09:03.808 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:09:03.808 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.808 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:09:03.809 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:03.809 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:09:03.809 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:03.809 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:03.810 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:09:03.810 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:03.810 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:03.810 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:03.810 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:09:03.811 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:03.811 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.811 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:03.811 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:09:03.812 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:03.812 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:03.812 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:09:03.812 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:03.812 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:03.813 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:03.813 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:09:03.813 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:03.813 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.814 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:03.814 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:09:03.814 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:03.814 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:03.814 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:09:03.815 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:03.815 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:09:03.815 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:03.815 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:09:03.815 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:03.816 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.816 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:03.816 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:09:03.816 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:03.816 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:03.817 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:09:03.817 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:03.817 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:09:03.817 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:03.817 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:09:03.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:03.818 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:03.818 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:09:03.818 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.818 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:09:03.819 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:09:03.819 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:09:03.819 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:03.819 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:09:03Z" 2023-10-11T16:09:03.820 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:09:03.820 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:09:03Z" } 2023-10-11T16:09:03.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:09:03.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:03.822 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:03.822 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:09:08.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:08.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:08.824 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:08.883 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:08.884 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:08.933 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:09:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:09:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:09:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:09:08.934 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:09:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:09:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:09:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:08.935 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:09:08.936 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:08.936 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:08.936 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:09:08.936 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:08.936 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:09:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.937 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:09:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:09:08.938 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:09:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:08.939 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:09:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:08.940 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:09:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:09:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:08.941 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:09:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:08.942 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:09:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:09:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:08.943 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:09:08.944 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.945 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:09:08.945 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:09:08.945 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:09:08.945 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:08.945 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:09:08Z" 2023-10-11T16:09:08.946 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:09:08.947 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:09:08Z" } 2023-10-11T16:09:08.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:09:08.948 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:08.948 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:08.948 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:09:13.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:13.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:13.951 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:14.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:14.010 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:14.061 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:09:14.061 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:09:14.061 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:09:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:09:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:09:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:09:14.062 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:09:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:09:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:14.063 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:09:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:14.064 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:09:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:14.065 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:09:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032624", 2023-10-11T16:09:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:14.066 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:04:52.370537Z" 2023-10-11T16:09:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:14.067 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.068 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:14.068 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:09:14.068 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:14.068 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:14.068 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052230", 2023-10-11T16:09:14.069 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:14.069 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:09:14.069 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:14.069 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:40.973202Z" 2023-10-11T16:09:14.069 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:14.070 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.070 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:14.070 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:09:14.070 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:14.071 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:14.071 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:09:14.071 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:14.071 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:09:14.071 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:14.072 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:09:14.072 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:14.072 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:14.072 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:09:14.072 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.073 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:09:14.073 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:09:14.073 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:09:14.073 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:14.074 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:09:14Z" 2023-10-11T16:09:14.074 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:09:14.075 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:04:52.370537Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052230", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:05:40.973202Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:09:14Z" } 2023-10-11T16:09:14.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:04:52.370537Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052230', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:05:40.973202Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:09:14.076 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:14.076 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000052230', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:14.077 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032764 target=00000000000000000000:00000000000000032624, shard=2 master=00000000000000000000:00000000000000052371 target=00000000000000000000:00000000000000052230 2023-10-11T16:09:19.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.080 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.139 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.187 INFO:teuthology.orchestra.run.smithi158.stdout:{ 2023-10-11T16:09:19.187 INFO:teuthology.orchestra.run.smithi158.stdout: "sync_status": { 2023-10-11T16:09:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "info": { 2023-10-11T16:09:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "sync", 2023-10-11T16:09:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "num_shards": 4, 2023-10-11T16:09:19.188 INFO:teuthology.orchestra.run.smithi158.stdout: "instance_id": 10532025945498841711 2023-10-11T16:09:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: "markers": [ 2023-10-11T16:09:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:19.189 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 0, 2023-10-11T16:09:19.190 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:19.190 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:19.190 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000046217", 2023-10-11T16:09:19.191 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:19.191 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:19.191 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:19.191 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:05:41.005285Z" 2023-10-11T16:09:19.192 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:19.192 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.192 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:19.193 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 1, 2023-10-11T16:09:19.193 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:19.193 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:19.193 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000032764", 2023-10-11T16:09:19.194 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:19.194 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 0, 2023-10-11T16:09:19.194 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:19.194 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:07:46.486519Z" 2023-10-11T16:09:19.194 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:19.195 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.195 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:19.195 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 2, 2023-10-11T16:09:19.195 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:19.196 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:19.196 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000052371", 2023-10-11T16:09:19.196 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:19.196 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 55, 2023-10-11T16:09:19.196 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:19.197 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:07:46.464084Z" 2023-10-11T16:09:19.197 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:19.197 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.197 INFO:teuthology.orchestra.run.smithi158.stdout: { 2023-10-11T16:09:19.197 INFO:teuthology.orchestra.run.smithi158.stdout: "key": 3, 2023-10-11T16:09:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: "val": { 2023-10-11T16:09:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: "status": "incremental-sync", 2023-10-11T16:09:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: "marker": "00000000000000000000:00000000000000044314", 2023-10-11T16:09:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: "next_step_marker": "", 2023-10-11T16:09:19.198 INFO:teuthology.orchestra.run.smithi158.stdout: "total_entries": 9, 2023-10-11T16:09:19.199 INFO:teuthology.orchestra.run.smithi158.stdout: "pos": 0, 2023-10-11T16:09:19.199 INFO:teuthology.orchestra.run.smithi158.stdout: "timestamp": "2023-10-11T16:00:19.842125Z" 2023-10-11T16:09:19.199 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:19.199 INFO:teuthology.orchestra.run.smithi158.stdout: } 2023-10-11T16:09:19.199 INFO:teuthology.orchestra.run.smithi158.stdout: ] 2023-10-11T16:09:19.200 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.200 INFO:teuthology.orchestra.run.smithi158.stdout: "full_sync": { 2023-10-11T16:09:19.200 INFO:teuthology.orchestra.run.smithi158.stdout: "total": 64, 2023-10-11T16:09:19.201 INFO:teuthology.orchestra.run.smithi158.stdout: "complete": 64 2023-10-11T16:09:19.201 INFO:teuthology.orchestra.run.smithi158.stdout: }, 2023-10-11T16:09:19.201 INFO:teuthology.orchestra.run.smithi158.stdout: "current_time": "2023-10-11T16:09:19Z" 2023-10-11T16:09:19.201 INFO:teuthology.orchestra.run.smithi158.stdout:} 2023-10-11T16:09:19.203 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10532025945498841711 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:05:41.005285Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-10-11T16:07:46.486519Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052371", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-10-11T16:07:46.464084Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044314", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-10-11T16:00:19.842125Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-10-11T16:09:19Z" } 2023-10-11T16:09:19.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:05:41.005285Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T16:07:46.486519Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052371', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T16:07:46.464084Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044314', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T16:00:19.842125Z'}}] 2023-10-11T16:09:19.204 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:19.204 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000046217', 1: '00000000000000000000:00000000000000032764', 2: '00000000000000000000:00000000000000052371', 3: '00000000000000000000:00000000000000044314'} 2023-10-11T16:09:19.204 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:09:19.205 DEBUG:boto:path=/dghqvd-74/ 2023-10-11T16:09:19.205 DEBUG:boto:auth_path=/dghqvd-74/ 2023-10-11T16:09:19.206 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.206 DEBUG:boto:Path: /dghqvd-74/ 2023-10-11T16:09:19.207 DEBUG:boto:Data: 2023-10-11T16:09:19.207 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.208 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.208 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.208 DEBUG:boto:Params: {} 2023-10-11T16:09:19.209 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:19.210 DEBUG:boto:Token: None 2023-10-11T16:09:19.211 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-74/ 2023-10-11T16:09:19.211 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:ELjX08jjUN5/1THPDHNIn/9+2/0= 2023-10-11T16:09:19.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ELjX08jjUN5/1THPDHNIn/9+2/0=', 'Content-Length': '0'} 2023-10-11T16:09:19.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b71c518bebecff7b-006526c8af-2c61-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.219 DEBUG:boto:path=/dghqvd-74/obj3 2023-10-11T16:09:19.219 DEBUG:boto:auth_path=/dghqvd-74/obj3 2023-10-11T16:09:19.220 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.220 DEBUG:boto:Path: /dghqvd-74/obj3 2023-10-11T16:09:19.221 DEBUG:boto:Data: 2023-10-11T16:09:19.221 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.222 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.222 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.223 DEBUG:boto:Params: {} 2023-10-11T16:09:19.223 DEBUG:boto:Token: None 2023-10-11T16:09:19.223 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-74/obj3 2023-10-11T16:09:19.224 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YvoTPMZ64j74C4Rs9lx/BEZhino= 2023-10-11T16:09:19.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YvoTPMZ64j74C4Rs9lx/BEZhino=', 'Content-Length': '0'} 2023-10-11T16:09:19.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000148465242e0740bf-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.228 DEBUG:boto:path=/dghqvd-74/obj3 2023-10-11T16:09:19.228 DEBUG:boto:auth_path=/dghqvd-74/obj3 2023-10-11T16:09:19.229 DEBUG:boto:Method: GET 2023-10-11T16:09:19.229 DEBUG:boto:Path: /dghqvd-74/obj3 2023-10-11T16:09:19.230 DEBUG:boto:Data: 2023-10-11T16:09:19.230 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.231 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.231 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.232 DEBUG:boto:Params: {} 2023-10-11T16:09:19.232 DEBUG:boto:Token: None 2023-10-11T16:09:19.233 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-74/obj3 2023-10-11T16:09:19.233 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:TRN1u5ehOj888LJ9a85RqX9aWI8= 2023-10-11T16:09:19.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TRN1u5ehOj888LJ9a85RqX9aWI8=', 'Content-Length': '0'} 2023-10-11T16:09:19.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000616de741e71d099c-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.237 DEBUG:boto:path=/dghqvd-74/obj4 2023-10-11T16:09:19.237 DEBUG:boto:auth_path=/dghqvd-74/obj4 2023-10-11T16:09:19.238 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.238 DEBUG:boto:Path: /dghqvd-74/obj4 2023-10-11T16:09:19.239 DEBUG:boto:Data: 2023-10-11T16:09:19.239 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.240 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.240 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.241 DEBUG:boto:Params: {} 2023-10-11T16:09:19.241 DEBUG:boto:Token: None 2023-10-11T16:09:19.241 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-74/obj4 2023-10-11T16:09:19.242 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:069sia//mPSfJctPoP48h4N1GwE= 2023-10-11T16:09:19.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:069sia//mPSfJctPoP48h4N1GwE=', 'Content-Length': '0'} 2023-10-11T16:09:19.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a25ff1a8086a9a1-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.246 DEBUG:boto:path=/dghqvd-74/obj4 2023-10-11T16:09:19.246 DEBUG:boto:auth_path=/dghqvd-74/obj4 2023-10-11T16:09:19.246 DEBUG:boto:Method: GET 2023-10-11T16:09:19.247 DEBUG:boto:Path: /dghqvd-74/obj4 2023-10-11T16:09:19.247 DEBUG:boto:Data: 2023-10-11T16:09:19.248 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.248 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.249 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.249 DEBUG:boto:Params: {} 2023-10-11T16:09:19.250 DEBUG:boto:Token: None 2023-10-11T16:09:19.250 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-74/obj4 2023-10-11T16:09:19.251 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:LOt4Oc6kTNQGs6xxwRFZ50OwM4U= 2023-10-11T16:09:19.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LOt4Oc6kTNQGs6xxwRFZ50OwM4U=', 'Content-Length': '0'} 2023-10-11T16:09:19.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007226e4c666517236-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.254 DEBUG:boto:path=/dghqvd-75/ 2023-10-11T16:09:19.254 DEBUG:boto:auth_path=/dghqvd-75/ 2023-10-11T16:09:19.254 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.255 DEBUG:boto:Path: /dghqvd-75/ 2023-10-11T16:09:19.256 DEBUG:boto:Data: 2023-10-11T16:09:19.256 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.257 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.257 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.258 DEBUG:boto:Params: {} 2023-10-11T16:09:19.259 DEBUG:boto:Token: None 2023-10-11T16:09:19.260 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-75/ 2023-10-11T16:09:19.260 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:YBwcJ1u/Tkpg40+4VyQDNc1QXEg= 2023-10-11T16:09:19.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YBwcJ1u/Tkpg40+4VyQDNc1QXEg=', 'Content-Length': '0'} 2023-10-11T16:09:19.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aa101c85674dce4-006526c8af-2c61-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.264 DEBUG:boto:path=/dghqvd-75/obj3 2023-10-11T16:09:19.265 DEBUG:boto:auth_path=/dghqvd-75/obj3 2023-10-11T16:09:19.266 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.266 DEBUG:boto:Path: /dghqvd-75/obj3 2023-10-11T16:09:19.267 DEBUG:boto:Data: 2023-10-11T16:09:19.268 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.268 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.269 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.269 DEBUG:boto:Params: {} 2023-10-11T16:09:19.270 DEBUG:boto:Token: None 2023-10-11T16:09:19.271 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-75/obj3 2023-10-11T16:09:19.271 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:wdhtLH1w0ME/ITKyS8+xGiFHimM= 2023-10-11T16:09:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wdhtLH1w0ME/ITKyS8+xGiFHimM=', 'Content-Length': '0'} 2023-10-11T16:09:19.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f3ff70ac3e86be2-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.274 DEBUG:boto:path=/dghqvd-75/obj3 2023-10-11T16:09:19.275 DEBUG:boto:auth_path=/dghqvd-75/obj3 2023-10-11T16:09:19.275 DEBUG:boto:Method: GET 2023-10-11T16:09:19.276 DEBUG:boto:Path: /dghqvd-75/obj3 2023-10-11T16:09:19.276 DEBUG:boto:Data: 2023-10-11T16:09:19.277 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.277 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.277 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.278 DEBUG:boto:Params: {} 2023-10-11T16:09:19.278 DEBUG:boto:Token: None 2023-10-11T16:09:19.279 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-75/obj3 2023-10-11T16:09:19.279 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:AonDDMLAACffcvJUQxB3LFYh5+c= 2023-10-11T16:09:19.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AonDDMLAACffcvJUQxB3LFYh5+c=', 'Content-Length': '0'} 2023-10-11T16:09:19.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2d260fcd8754b57-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.282 DEBUG:boto:path=/dghqvd-75/obj4 2023-10-11T16:09:19.283 DEBUG:boto:auth_path=/dghqvd-75/obj4 2023-10-11T16:09:19.283 DEBUG:boto:Method: HEAD 2023-10-11T16:09:19.284 DEBUG:boto:Path: /dghqvd-75/obj4 2023-10-11T16:09:19.284 DEBUG:boto:Data: 2023-10-11T16:09:19.285 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.285 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.285 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.286 DEBUG:boto:Params: {} 2023-10-11T16:09:19.286 DEBUG:boto:Token: None 2023-10-11T16:09:19.287 DEBUG:boto:StringToSign: HEAD Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-75/obj4 2023-10-11T16:09:19.287 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:fbk9mcMhcRmbe6TWBSUbCVeObRU= 2023-10-11T16:09:19.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fbk9mcMhcRmbe6TWBSUbCVeObRU=', 'Content-Length': '0'} 2023-10-11T16:09:19.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000287aa46375931c14-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.290 DEBUG:boto:path=/dghqvd-75/obj4 2023-10-11T16:09:19.291 DEBUG:boto:auth_path=/dghqvd-75/obj4 2023-10-11T16:09:19.291 DEBUG:boto:Method: GET 2023-10-11T16:09:19.292 DEBUG:boto:Path: /dghqvd-75/obj4 2023-10-11T16:09:19.293 DEBUG:boto:Data: 2023-10-11T16:09:19.293 DEBUG:boto:Headers: {} 2023-10-11T16:09:19.294 DEBUG:boto:Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.295 DEBUG:boto:Port: 8001 2023-10-11T16:09:19.295 DEBUG:boto:Params: {} 2023-10-11T16:09:19.296 DEBUG:boto:Token: None 2023-10-11T16:09:19.297 DEBUG:boto:StringToSign: GET Wed, 11 Oct 2023 16:09:19 GMT /dghqvd-75/obj4 2023-10-11T16:09:19.298 DEBUG:boto:Signature: AWS MWUFUL61ESHN8BNC6ISZ:/Zbo2VBWrJnNijFZZEqQLXjO1UI= 2023-10-11T16:09:19.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 16:09:19 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Zbo2VBWrJnNijFZZEqQLXjO1UI=', 'Content-Length': '0'} 2023-10-11T16:09:19.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 16:07:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-74:daa316e9-869c-4074-a756-a8adfb4a3573.10720.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074d8de514075b828-006526c8af-2c61-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 16:09:19 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:19.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dghqvd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:19.302 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dghqvd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:19.355 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.356 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.410 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:09:19.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:19.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:19.411 DEBUG:teuthology.orchestra.run.smithi158:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:19.458 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.459 INFO:teuthology.orchestra.run.smithi158.stderr:ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.504 INFO:teuthology.orchestra.run.smithi158.stdout:{"sync_policy":{"groups":[]}} 2023-10-11T16:09:19.504 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-10-11T16:09:19.506 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.507 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-11T16:09:19.508 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-10-11T16:09:19.508 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:19.511 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-11T16:09:19.511 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-11T16:09:19.512 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-11T16:09:19.512 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-10-11T16:09:19.513 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-10-11T16:09:19.514 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-11T16:09:19.514 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-11T16:09:19.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dghqvd-55 2023-10-11T16:09:19.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.518 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:19.518 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.519 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.519 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.519 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.520 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HzuaHPoN25AuhbsebIRtxjG23TA= 2023-10-11T16:09:19.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HzuaHPoN25AuhbsebIRtxjG23TA=', 'Content-Length': '0'} 2023-10-11T16:09:19.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000000b6e88c4ddf1ed9-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.524 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.525 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.525 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.525 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.525 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.526 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T16:09:19.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T16:09:19.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0dc4a2639a299c4-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:19.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:19.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:19.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.531 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:19.531 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:19.531 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:19.532 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.532 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:19.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.533 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2rrqrUWT6/iwwUcyh47FRD5AHcU= 2023-10-11T16:09:19.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2rrqrUWT6/iwwUcyh47FRD5AHcU='} 2023-10-11T16:09:19.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a52b94000b577e9c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.537 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.537 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.538 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.538 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.538 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.539 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T16:09:19.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T16:09:19.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006275e8abcc26b14f-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:19.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:19.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:19.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.543 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:19.544 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:19.544 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:19.545 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.545 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:19.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.546 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RVRbOvHnC0eBOC4X0dtRwhHg7yM= 2023-10-11T16:09:19.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RVRbOvHnC0eBOC4X0dtRwhHg7yM='} 2023-10-11T16:09:19.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da5aa175fed57e92-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.550 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.551 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.551 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.551 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.552 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.552 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T16:09:19.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T16:09:19.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d784476efce0575-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:19.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:19.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:19.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.557 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:19.557 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:19.557 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:19.557 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.558 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:19.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.558 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:VJJcyCzXNDJ1zcSOrTpJsuNz/zQ= 2023-10-11T16:09:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VJJcyCzXNDJ1zcSOrTpJsuNz/zQ='} 2023-10-11T16:09:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acd9ed497b6df23c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.563 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.563 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.563 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.564 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.564 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.564 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw= 2023-10-11T16:09:19.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rqHaShlWJEu/6BuVpOotmmJvzUw=', 'Content-Length': '0'} 2023-10-11T16:09:19.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924d71486b5d3a11-006526b958-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:19.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:19.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:19.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:19.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.569 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:19.570 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:19.570 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:19.570 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:03:52 GMT 2023-10-11T16:09:19.571 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:19.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.571 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:q0yOkRW3pAD+hvj/oh4lcrQmhrU= 2023-10-11T16:09:19.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:03:52 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q0yOkRW3pAD+hvj/oh4lcrQmhrU='} 2023-10-11T16:09:19.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc1fe989b5bdee0c-006526b958-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:19.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:19.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:19.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:19.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:19.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:19.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:19.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034974.544814_114.1", 2023-10-11T16:09:19.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:36:14.544814Z" 2023-10-11T16:09:19.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697036632.758331_393.1", 2023-10-11T16:09:19.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:03:52.758331Z" 2023-10-11T16:09:19.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:19.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T16:09:19.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:19.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:19.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:19.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:03:53.088+0000 7f03602de780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:19.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:03:53.089+0000 7f03602de780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:19.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:19.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:19.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:19.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:19.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:19.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:19.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:19.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:19.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:19.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:19.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:19.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:19.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:19.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:19.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:19.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:19.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:19.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:19.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:19.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:19.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:19.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:19.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:19.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034974.544814_114.1", 2023-10-11T16:09:19.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:19.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:19.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:19.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T16:09:19.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:19.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:19.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:19.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:19.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:19.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697036632.758331_393.1", 2023-10-11T16:09:19.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:19.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:19.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:19.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:03:52.758331Z", 2023-10-11T16:09:19.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:19.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:19.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:19.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:19.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:19.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:03:53Z" 2023-10-11T16:09:19.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:19.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:19.609 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:19.609 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:19.610 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:19.610 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:19.610 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:19.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:19.612 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.612 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:19.612 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:19.613 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:19.613 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:19.614 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:19.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:19.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:19.616 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:19.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:19.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:19.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:19.618 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:19.618 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.619 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:19.619 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:19.619 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:19.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:19.620 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:19.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:19.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:19.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:19.621 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:19.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:19.622 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:19.622 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.622 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:19.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:19.623 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:19.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:19.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034974.544814_114.1", 2023-10-11T16:09:19.624 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:19.624 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:19.624 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:19.625 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:36:14.544814Z", 2023-10-11T16:09:19.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:19.625 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:19.626 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.626 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:19.626 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:19.627 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:19.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:19.627 INFO:tasks.rgw_multisite_tests: "marker": "1_1697036632.758331_393.1", 2023-10-11T16:09:19.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:19.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:19.628 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:19.628 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:03:52.758331Z", 2023-10-11T16:09:19.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:19.629 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:19.630 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:19.630 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:19.630 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.631 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:19.631 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:19.631 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:19.632 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:19.632 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:03:53Z" 2023-10-11T16:09:19.632 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:19.633 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697034974.544814_114.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T14:36:14.544814Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697036632.758331_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:03:52.758331Z', 'realm_epoch': 2}}] 2023-10-11T16:09:19.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T16:09:19.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697034974.544814_114.1', 3: '1_1697036632.758331_393.1'} 2023-10-11T16:09:19.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:19.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T16:09:19.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:19.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:19.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:19.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:03:53.210+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:03:58.211+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:03.211+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:08.212+0000 7fe22de49780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:13.213+0000 7fe22de49780 1 bucket sync caught up with source: 2023-10-11T16:09:19.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000001.16.6, , , , 00000000001.23.6, , , 00000000001.55.6, , 00000000001.24.6] 2023-10-11T16:09:19.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000001.16.6, , , , 00000000001.23.6, , , 00000000001.55.6, , 00000000001.24.6] 2023-10-11T16:09:19.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:13.213+0000 7fe22de49780 0 bucket checkpoint complete 2023-10-11T16:09:19.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T16:09:19.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:19.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:19.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:19.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:19.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:19.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:13.326+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:18.326+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:23.327+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:28.327+0000 7f43b40db780 1 waiting to reach incremental sync.. 2023-10-11T16:09:19.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:33.329+0000 7f43b40db780 1 bucket sync caught up with source: 2023-10-11T16:09:19.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000001.45.6, , , , 00000000001.261.6, , , 00000000001.270.6, , 00000000001.130.6] 2023-10-11T16:09:19.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000001.45.6, , , , 00000000001.261.6, , , 00000000001.270.6, , 00000000001.130.6] 2023-10-11T16:09:19.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:33.329+0000 7f43b40db780 0 bucket checkpoint complete 2023-10-11T16:09:19.645 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:19.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.649 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.650 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.650 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.651 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.651 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.652 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE= 2023-10-11T16:09:19.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE=', 'Content-Length': '0'} 2023-10-11T16:09:19.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835b372aa59b0410-006526b981-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:19.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.657 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.657 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.657 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.658 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.658 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.659 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE= 2023-10-11T16:09:19.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7ZAAlxAxwiTpVvSxxcqqsJUx3fE=', 'Content-Length': '0'} 2023-10-11T16:09:19.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbbcb7ea56acf70b-006526b981-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.660 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-11T16:09:19.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.664 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.664 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.665 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.665 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.665 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:19.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.666 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T16:09:19.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T16:09:19.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d487caf8eff413-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.667 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:19.668 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:19.668 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:19.669 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:19.669 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-11T16:09:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.674 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.675 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.675 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.675 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.675 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:19.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.676 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T16:09:19.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T16:09:19.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d77e95f411561ec-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.678 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:19.678 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:19.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:19.680 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:19.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.687 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.688 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.688 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:19.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.689 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T16:09:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T16:09:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edf3cbbc09bdde8e-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:19.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.694 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.694 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.695 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.695 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.695 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:19.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.696 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8= 2023-10-11T16:09:19.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Q4HKsJcRcAY8lLSqFukaHat3ES8=', 'Content-Length': '0'} 2023-10-11T16:09:19.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003130f5064589dd22-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.697 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:19.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.701 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.702 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.702 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.702 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.703 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.703 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T16:09:19.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T16:09:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72d44ccdd4e3776-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.709 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.710 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.710 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.711 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.711 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.712 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T16:09:19.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T16:09:19.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b95c900df2c781-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.719 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.720 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.720 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.720 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.720 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.721 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o= 2023-10-11T16:09:19.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o=', 'Content-Length': '0'} 2023-10-11T16:09:19.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b837400205bc50b9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.726 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.726 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.727 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.727 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.727 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.728 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o= 2023-10-11T16:09:19.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:QJaCSub4r7Q7OyyTYIYG8plsm8o=', 'Content-Length': '0'} 2023-10-11T16:09:19.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002931de8b6d4c6460-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.729 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:19.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.733 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.733 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.734 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.734 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.734 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.735 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T16:09:19.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T16:09:19.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee377aabbfd2fe8e-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.740 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.741 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.741 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.742 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.743 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:19.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.744 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0= 2023-10-11T16:09:19.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oOMJtzBQCRdIqx37Sn2EdukdLm0=', 'Content-Length': '0'} 2023-10-11T16:09:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c646328abc2ad766-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:19.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.751 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.751 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.751 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.752 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.752 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.753 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e02b48ae72b7c45e-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:19.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.761 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.761 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.762 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.762 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.763 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.763 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3601ed9e9b2280-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:19.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.768 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.768 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.769 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.769 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.770 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:19.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.771 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE= 2023-10-11T16:09:19.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE=', 'Content-Length': '0'} 2023-10-11T16:09:19.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000297c518b9baaa2ac-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.775 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.775 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.776 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.776 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.776 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.777 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE= 2023-10-11T16:09:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:86yfI5IQpiNafcpOJVXxuLXIcuE=', 'Content-Length': '0'} 2023-10-11T16:09:19.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e845db68e48c80-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.778 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:19.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:19.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.782 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.782 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.782 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.783 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.783 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:19.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.783 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090= 2023-10-11T16:09:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090=', 'Content-Length': '0'} 2023-10-11T16:09:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5775789c680172-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:19.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.788 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.788 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.788 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.788 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.789 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:19.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.789 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090= 2023-10-11T16:09:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/D3jeRGHJMeWj4HzPMao9fPb090=', 'Content-Length': '0'} 2023-10-11T16:09:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb07fa91458a01a8-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:19.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.794 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.795 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.795 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.795 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.795 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.796 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000500430adc62074f8-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:19.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.801 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.801 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.801 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.802 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.802 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.803 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c75861b9d9ae000-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.804 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:19.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.808 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.808 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.808 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.809 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.809 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.810 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T16:09:19.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T16:09:19.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882c925d2bca3d1a-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.814 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.815 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.815 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.815 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.815 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.816 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T16:09:19.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T16:09:19.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697fe8ef732d9830-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.821 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.821 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.821 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.822 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.822 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.823 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU= 2023-10-11T16:09:19.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU=', 'Content-Length': '0'} 2023-10-11T16:09:19.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d85401c5854d652-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.827 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.828 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.828 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.828 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.829 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.829 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU= 2023-10-11T16:09:19.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:niYzODriEqIrb3Us/8hzMV2WAiU=', 'Content-Length': '0'} 2023-10-11T16:09:19.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a12d389abb32d79-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.830 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:19.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.834 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.835 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.835 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.835 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.835 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.837 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T16:09:19.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T16:09:19.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078075b43f5d5f104-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:19.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.842 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.843 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.843 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.843 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.843 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:19.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.844 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY= 2023-10-11T16:09:19.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dzgW/BQ2UlcKNjC3LY4UudTo5lY=', 'Content-Length': '0'} 2023-10-11T16:09:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e24a6d6d7f88f4e-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:19.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.849 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.849 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.849 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.850 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.850 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.851 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5cff76339b8d911-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:19.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.856 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.856 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.856 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.857 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.857 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.858 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d95e11a07839a958-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:19.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.863 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.863 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.863 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.864 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:19.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.864 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs= 2023-10-11T16:09:19.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs=', 'Content-Length': '0'} 2023-10-11T16:09:19.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f743fab14309295-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:19.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.869 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.869 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.870 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.870 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.871 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:19.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.871 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs= 2023-10-11T16:09:19.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IDf55Ysp0kgeTEG03IIOTVFIybs=', 'Content-Length': '0'} 2023-10-11T16:09:19.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bdcf273fdf1cc89-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.872 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:19.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.876 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.876 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.877 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.877 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.877 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:19.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.878 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0= 2023-10-11T16:09:19.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0=', 'Content-Length': '0'} 2023-10-11T16:09:19.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c169e58519b3c9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:19.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:19.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.882 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.883 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.883 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.883 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.884 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:19.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.884 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0= 2023-10-11T16:09:19.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Jq41S3KjO0fk3a+/9I+0sdiK5M0=', 'Content-Length': '0'} 2023-10-11T16:09:19.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f84583db586461-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.890 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.891 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.892 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.892 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.893 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.894 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070f404ee8511348b-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:19.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.903 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.903 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.904 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.904 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.905 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.906 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd5e0f809f22f1d8-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.909 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:19.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.915 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.916 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.916 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.916 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.917 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.917 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T16:09:19.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T16:09:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8666d7af7a5050-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.922 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.923 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.924 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.925 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T16:09:19.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T16:09:19.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c28849b3f94011c-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.932 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.932 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.933 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.933 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.933 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.934 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas= 2023-10-11T16:09:19.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas=', 'Content-Length': '0'} 2023-10-11T16:09:19.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075852bd51deead0-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.940 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.940 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.940 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.941 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.941 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.942 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas= 2023-10-11T16:09:19.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zyF3XYaXAavEmOMpf0SdT5Xwbas=', 'Content-Length': '0'} 2023-10-11T16:09:19.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc5249bf8b8a24ce-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.943 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:19.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.947 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.947 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.948 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.948 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.948 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.949 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T16:09:19.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T16:09:19.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004fe2a720130e1b9-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.954 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.955 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.955 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.955 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.955 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:19.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.956 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84= 2023-10-11T16:09:19.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TvinxVoL85+BhdhpMlFVdxaob84=', 'Content-Length': '0'} 2023-10-11T16:09:19.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9141478267e5d9-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:19.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:19.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.964 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.965 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.965 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.966 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.966 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.967 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ccbddc5056c45e8-006526b981-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:19.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:19.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:19.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:19.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.973 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.973 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.974 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.974 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.975 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:19.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.976 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90= 2023-10-11T16:09:19.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GoJhBg8sS7Sh9BE29xbHpObaF90=', 'Content-Length': '0'} 2023-10-11T16:09:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006213eac78ae7a103-006526b981-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:19.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.981 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.981 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.981 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.982 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.982 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:19.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.983 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og= 2023-10-11T16:09:19.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og=', 'Content-Length': '0'} 2023-10-11T16:09:19.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd638f435135c78-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:19.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.987 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:19.987 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.987 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.988 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.988 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:19.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.989 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og= 2023-10-11T16:09:19.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+K9jsi2AXRF5o1M/G+sn74XN+Og=', 'Content-Length': '0'} 2023-10-11T16:09:19.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea061f48c7cae6a3-006526b981-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.990 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.993 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.993 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.994 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:19.994 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:33 GMT 2023-10-11T16:09:19.994 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:19.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:19.995 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dYi3sVztg4wx2IZ9PUtX6sWq8Z4= 2023-10-11T16:09:19.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dYi3sVztg4wx2IZ9PUtX6sWq8Z4=', 'Content-Length': '0'} 2023-10-11T16:09:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3809ad6b26e1cf-006526b981-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:19.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:19.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:19.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:19.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:19.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:19.999 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:19.999 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.000 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.000 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.000 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:20.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.001 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E1kJzRxQPGulWefWYuXRl7IKP/w= 2023-10-11T16:09:20.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E1kJzRxQPGulWefWYuXRl7IKP/w=', 'Content-Length': '0'} 2023-10-11T16:09:20.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448293861f3d839a-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.006 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.006 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.007 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.007 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.007 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.008 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002972b829af69f87-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.013 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.013 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.014 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.014 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.014 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.015 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea627ff42816a92-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.016 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.020 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.021 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.021 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.021 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.021 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.022 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T16:09:20.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T16:09:20.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096cf862ac9a72faa-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.027 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.028 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.028 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.028 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.028 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.029 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T16:09:20.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T16:09:20.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f363a32dd7518116-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.034 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.034 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.035 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.035 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.035 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.036 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc= 2023-10-11T16:09:20.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc=', 'Content-Length': '0'} 2023-10-11T16:09:20.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024505b4a1df1e9ef-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.041 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.041 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.042 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.042 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.042 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.043 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc= 2023-10-11T16:09:20.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e5duarstPJSAZuT8TWQK75OMAxc=', 'Content-Length': '0'} 2023-10-11T16:09:20.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be18b40a17483ab7-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.044 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.049 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.049 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.050 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.050 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.050 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.051 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T16:09:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T16:09:20.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe87b5e90554a791-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.056 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.056 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.056 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.056 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.057 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.057 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY= 2023-10-11T16:09:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad4qy5G2cc8qiEGrLS6JdhgqyOY=', 'Content-Length': '0'} 2023-10-11T16:09:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000013f518bc30837c0-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.062 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.063 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.063 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.063 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.063 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.064 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bdc3777de0b20f9-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.069 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.069 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.070 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.070 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.070 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.071 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd2160d80037f9bf-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.075 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.076 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.076 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.076 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.077 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.077 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4= 2023-10-11T16:09:20.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4=', 'Content-Length': '0'} 2023-10-11T16:09:20.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc0ee751fcb1a86-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.081 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.082 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.082 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.082 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.083 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.083 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4= 2023-10-11T16:09:20.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T6ZM+sL5RRhKLY4th4xpunGvaN4=', 'Content-Length': '0'} 2023-10-11T16:09:20.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31e8265838efa78-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.084 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.088 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.088 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.089 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.089 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.089 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.090 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno= 2023-10-11T16:09:20.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno=', 'Content-Length': '0'} 2023-10-11T16:09:20.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71e961bf5aa70b4-006526b982-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.094 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.094 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.094 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.095 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.095 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.095 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno= 2023-10-11T16:09:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:aFRuKUlI42XHj2Qp0cCMqogLvno=', 'Content-Length': '0'} 2023-10-11T16:09:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77d8b7d28ae021f-006526b982-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.100 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.100 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.101 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.101 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.101 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.102 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa8f37ce16f747af-006526b982-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.107 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.107 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.107 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.108 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.108 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.109 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw= 2023-10-11T16:09:20.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+0caGueU7s6KEAHCm0g4GaUhWaw=', 'Content-Length': '0'} 2023-10-11T16:09:20.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa42a38d512e0a5-006526b982-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.110 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:20.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:20.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding": "None", 2023-10-11T16:09:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_index": { 2023-10-11T16:09:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "Normal", 2023-10-11T16:09:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "normal": { 2023-10-11T16:09:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 11, 2023-10-11T16:09:20.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:20.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "logs": [ 2023-10-11T16:09:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:20.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:20.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:20.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 11, 2023-10-11T16:09:20.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:20.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:20.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:20.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_id": "1#00000000001.16.6", 2023-10-11T16:09:20.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.11901769168167861916", 2023-10-11T16:09:20.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op": "write", 2023-10-11T16:09:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "object": "a", 2023-10-11T16:09:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance": "", 2023-10-11T16:09:20.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": "complete", 2023-10-11T16:09:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "index_ver": 1, 2023-10-11T16:09:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:03:52.775214532Z", 2023-10-11T16:09:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "ver": { 2023-10-11T16:09:20.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pool": 9, 2023-10-11T16:09:20.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 8 2023-10-11T16:09:20.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bilog_flags": 0, 2023-10-11T16:09:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "versioned": false, 2023-10-11T16:09:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner": "", 2023-10-11T16:09:20.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner_display_name": "", 2023-10-11T16:09:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones_trace": [ 2023-10-11T16:09:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T16:09:20.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_id": "5#00000000001.23.6", 2023-10-11T16:09:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.13555810181077397004", 2023-10-11T16:09:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op": "write", 2023-10-11T16:09:20.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "object": "d", 2023-10-11T16:09:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance": "", 2023-10-11T16:09:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": "complete", 2023-10-11T16:09:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "index_ver": 1, 2023-10-11T16:09:20.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:03:52.871975063Z", 2023-10-11T16:09:20.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "ver": { 2023-10-11T16:09:20.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pool": 9, 2023-10-11T16:09:20.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 3 2023-10-11T16:09:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bilog_flags": 0, 2023-10-11T16:09:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "versioned": false, 2023-10-11T16:09:20.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner": "", 2023-10-11T16:09:20.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner_display_name": "", 2023-10-11T16:09:20.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones_trace": [ 2023-10-11T16:09:20.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T16:09:20.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_id": "8#00000000001.55.6", 2023-10-11T16:09:20.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.12455247144329015868", 2023-10-11T16:09:20.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op": "write", 2023-10-11T16:09:20.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "object": "c", 2023-10-11T16:09:20.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance": "", 2023-10-11T16:09:20.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": "complete", 2023-10-11T16:09:20.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "index_ver": 1, 2023-10-11T16:09:20.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:03:52.853505798Z", 2023-10-11T16:09:20.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "ver": { 2023-10-11T16:09:20.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pool": 9, 2023-10-11T16:09:20.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 3 2023-10-11T16:09:20.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bilog_flags": 0, 2023-10-11T16:09:20.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "versioned": false, 2023-10-11T16:09:20.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner": "", 2023-10-11T16:09:20.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner_display_name": "", 2023-10-11T16:09:20.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones_trace": [ 2023-10-11T16:09:20.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T16:09:20.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_id": "10#00000000001.24.6", 2023-10-11T16:09:20.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op_tag": "daa316e9-869c-4074-a756-a8adfb4a3573.5749.15734065776330899090", 2023-10-11T16:09:20.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "op": "write", 2023-10-11T16:09:20.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "object": "b", 2023-10-11T16:09:20.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance": "", 2023-10-11T16:09:20.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": "complete", 2023-10-11T16:09:20.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "index_ver": 1, 2023-10-11T16:09:20.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:03:52.795448596Z", 2023-10-11T16:09:20.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "ver": { 2023-10-11T16:09:20.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pool": 9, 2023-10-11T16:09:20.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 10 2023-10-11T16:09:20.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bilog_flags": 0, 2023-10-11T16:09:20.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "versioned": false, 2023-10-11T16:09:20.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner": "", 2023-10-11T16:09:20.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "owner_display_name": "", 2023-10-11T16:09:20.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones_trace": [ 2023-10-11T16:09:20.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "entry": "daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20" 2023-10-11T16:09:20.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.150 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:20.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:20.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-55 2023-10-11T16:09:20.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 4 2023-10-11T16:09:20.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:20.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:20.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding": "None", 2023-10-11T16:09:20.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_index": { 2023-10-11T16:09:20.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:20.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "Normal", 2023-10-11T16:09:20.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "normal": { 2023-10-11T16:09:20.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:20.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:20.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "logs": [ 2023-10-11T16:09:20.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:20.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:20.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:20.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:20.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 11, 2023-10-11T16:09:20.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:20.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:20.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:20.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:20.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:20.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:20.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:20.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:20.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:20.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:20.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:20.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:20.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:20.166 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dghqvd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dghqvd-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [] 2023-10-11T16:09:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.171 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.171 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.172 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.172 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.172 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.173 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T16:09:20.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T16:09:20.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2d00e34c76c50fd-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:20.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.177 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:20.177 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:20.177 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:20.177 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.178 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:20.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.178 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xbmEBWtBj+ELuuP4GyZV5IJN1f4= 2023-10-11T16:09:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xbmEBWtBj+ELuuP4GyZV5IJN1f4='} 2023-10-11T16:09:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000015a3a919010b96-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.183 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.183 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.183 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.183 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.184 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.184 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T16:09:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T16:09:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000601058d522305759-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:20.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:20.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.189 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:20.189 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:20.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:20.189 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.190 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:20.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.190 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:e8D0ut42Gfm1sLQvmx8p5j8KYBc= 2023-10-11T16:09:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:e8D0ut42Gfm1sLQvmx8p5j8KYBc='} 2023-10-11T16:09:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005d107a9a5583b1c-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.195 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.195 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.195 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.196 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.196 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.196 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T16:09:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T16:09:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000017dd7c3d3e872-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:20.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.200 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:20.201 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:20.201 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:20.201 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.201 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:20.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.202 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:8sqQt1wgI8l7y78RG9YU+vtsQfM= 2023-10-11T16:09:20.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8sqQt1wgI8l7y78RG9YU+vtsQfM='} 2023-10-11T16:09:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa7e3df68236dfd4-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.206 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.206 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.207 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.207 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.207 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.208 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY= 2023-10-11T16:09:20.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mY1qaMzr5uyExkVF1MRTIf0/DKY=', 'Content-Length': '0'} 2023-10-11T16:09:20.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027b852bf4da0543f-006526b982-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:20.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:20.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:20.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.212 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:20.212 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:20.212 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:20.213 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:04:34 GMT 2023-10-11T16:09:20.213 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:20.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.213 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ERhjfAG1uYi+23MhwwF9wHDow1M= 2023-10-11T16:09:20.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:04:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ERhjfAG1uYi+23MhwwF9wHDow1M='} 2023-10-11T16:09:20.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ff44f7181481872-006526b982-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T16:09:20.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:20.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:20.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:20.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:35.049+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T16:09:20.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:40.050+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T16:09:20.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:45.050+0000 7fc78a5aa780 1 waiting to reach latest gen 1, on 0.. 2023-10-11T16:09:20.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:50.052+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.51.6, ] 2023-10-11T16:09:20.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:04:55.052+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.51.6, ] 2023-10-11T16:09:20.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:00.053+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.51.6, ] 2023-10-11T16:09:20.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:05.054+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.51.6, ] 2023-10-11T16:09:20.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:10.055+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:15.055+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:20.057+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:25.058+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:30.059+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:35.060+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:40.060+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:45.061+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:50.062+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:05:55.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:00.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:05.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:10.063+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:15.064+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:20.065+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:25.065+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:30.066+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:35.067+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:40.068+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:45.069+0000 7fc78a5aa780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, ] 2023-10-11T16:09:20.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:50.071+0000 7fc78a5aa780 1 bucket sync caught up with source: 2023-10-11T16:09:20.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [00000000003.20.6, 00000000002.51.6, 00000000002.29.6] 2023-10-11T16:09:20.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:50.071+0000 7fc78a5aa780 0 bucket checkpoint complete 2023-10-11T16:09:20.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T16:09:20.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:20.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:20.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:50.192+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T16:09:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:06:55.193+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T16:09:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:00.195+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T16:09:20.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:05.196+0000 7f41b709e780 1 waiting for incremental sync to catch up: 2023-10-11T16:09:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000001.270.6, , 00000000002.135.6] 2023-10-11T16:09:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:10.197+0000 7f41b709e780 1 bucket sync caught up with source: 2023-10-11T16:09:20.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:20.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:10.197+0000 7f41b709e780 0 bucket checkpoint complete 2023-10-11T16:09:20.246 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:20.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.250 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.251 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.251 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.251 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.251 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.252 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw= 2023-10-11T16:09:20.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw=', 'Content-Length': '0'} 2023-10-11T16:09:20.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000620e203592925cc9-006526ba1e-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:20.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:09:20.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.257 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.257 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.257 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.258 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.258 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.259 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw= 2023-10-11T16:09:20.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nV5CwZqMnq/P6y9QMt4Ua1U4WNw=', 'Content-Length': '0'} 2023-10-11T16:09:20.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2745bfb80b8039e-006526ba1e-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-11T16:09:20.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.264 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.264 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.264 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.265 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.265 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:20.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.266 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T16:09:20.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T16:09:20.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f72ea9fc54a10ef-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.267 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:20.267 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:20.267 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:20.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:20.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-11T16:09:20.268 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-11T16:09:20.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-11T16:09:20.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-11T16:09:20.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-11T16:09:20.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.274 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.274 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.274 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.274 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.275 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:20.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.275 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T16:09:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T16:09:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec86e5aae688a92e-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.277 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:20.277 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:20.277 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:20.277 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:20.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-11T16:09:20.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-11T16:09:20.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-11T16:09:20.279 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-11T16:09:20.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.283 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.283 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.283 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.284 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.284 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:20.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.285 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T16:09:20.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T16:09:20.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009141886205bdf2dd-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.290 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.290 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.290 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.291 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.291 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:20.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.292 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc= 2023-10-11T16:09:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Vl+vW0Te+XYNiRu9tjSexKp3Ulc=', 'Content-Length': '0'} 2023-10-11T16:09:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a0642635a071a3-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.293 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:20.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.297 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.297 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.298 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.298 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.298 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.299 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T16:09:20.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T16:09:20.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038973e0f407a9c53-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.304 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.304 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.304 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.304 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.305 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.305 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T16:09:20.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T16:09:20.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ff4681f2e1ae6c-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.310 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.310 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.311 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.311 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.311 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.312 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous= 2023-10-11T16:09:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous=', 'Content-Length': '0'} 2023-10-11T16:09:20.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9a83d87e221c0a6-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.317 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.318 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.318 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.319 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous= 2023-10-11T16:09:20.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:23x5gHSjt7Thuam0AUqDDLE5Ous=', 'Content-Length': '0'} 2023-10-11T16:09:20.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a3260567a023bc-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:20.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.324 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.324 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.325 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.325 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.325 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.326 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T16:09:20.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T16:09:20.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f3dac7e9745154-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:20.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.331 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.331 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.331 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.332 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.332 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:20.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.333 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s= 2023-10-11T16:09:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:GJP13mBAY02qsCHW52kun6Xbr+s=', 'Content-Length': '0'} 2023-10-11T16:09:20.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96be835112152a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:20.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.338 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.338 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.338 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.339 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.339 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.339 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e78adcd100a6c8-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.345 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.345 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.345 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.346 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.346 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.346 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9d78915839094a-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:20.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.352 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.352 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.352 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.353 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:20.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.353 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc= 2023-10-11T16:09:20.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc=', 'Content-Length': '0'} 2023-10-11T16:09:20.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aac62962c48427c-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:20.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.358 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.358 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.358 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.359 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.359 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:20.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.359 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc= 2023-10-11T16:09:20.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3eOD1C4eLnaQvbAFCekcs5SOICc=', 'Content-Length': '0'} 2023-10-11T16:09:20.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39f94ced54a5a65-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:20.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:20.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.364 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.364 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.365 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.365 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.365 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:20.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.366 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc= 2023-10-11T16:09:20.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc=', 'Content-Length': '0'} 2023-10-11T16:09:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4ac3eb4fe08005-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:20.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:20.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:20.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.370 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.371 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.371 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.371 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.372 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:20.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.372 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc= 2023-10-11T16:09:20.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5mtuMbXNB5OGayN8no1IxL64hhc=', 'Content-Length': '0'} 2023-10-11T16:09:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d96f8706ec71aac-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:20.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.377 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.377 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.378 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.378 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.378 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.379 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3953552236d14af-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:20.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:20.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.384 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.385 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.385 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.385 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.385 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.386 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008538207e3adc329c-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.387 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:20.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.392 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.392 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.392 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.392 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.393 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.393 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T16:09:20.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T16:09:20.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d130c8a6eecadc56-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.398 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.399 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.399 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.399 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.400 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.400 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T16:09:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T16:09:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021898d2614fde4ae-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.405 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.406 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.406 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.406 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.406 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.407 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw= 2023-10-11T16:09:20.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw=', 'Content-Length': '0'} 2023-10-11T16:09:20.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba458bc7eea48c11-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.412 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.412 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.412 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.413 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.413 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.414 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw= 2023-10-11T16:09:20.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:W+6YlRHwREWH9qcn7uFh/Mso1Qw=', 'Content-Length': '0'} 2023-10-11T16:09:20.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c0414a92fc113e2-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:20.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.419 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.419 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.419 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.420 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.420 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.421 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T16:09:20.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T16:09:20.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c8d5ef678ae0e4-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:20.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.426 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.426 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.426 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.427 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.427 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:20.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.428 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk= 2023-10-11T16:09:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sef67dlX/Q9oNn2GquRq8nMEqPk=', 'Content-Length': '0'} 2023-10-11T16:09:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab678f818a9922a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:20.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.433 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.433 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.433 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.433 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.434 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.434 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe3dba669478623-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:20.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.440 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.440 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.440 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.440 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.441 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.441 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f764250bc201757-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.446 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.446 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.447 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.447 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.447 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:20.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.448 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4= 2023-10-11T16:09:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4=', 'Content-Length': '0'} 2023-10-11T16:09:20.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf480253c40fd29-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.452 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.452 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.453 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.453 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.453 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.454 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4= 2023-10-11T16:09:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G+FvgUCakv/EJg/BTV52e9uvM/4=', 'Content-Length': '0'} 2023-10-11T16:09:20.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d480722d2a2e3d-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.455 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:20.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:20.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.458 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.459 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.459 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.459 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.460 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.460 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds= 2023-10-11T16:09:20.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds=', 'Content-Length': '0'} 2023-10-11T16:09:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee388329c794224-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:20.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:20.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.465 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.465 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.465 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.466 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:20.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.466 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds= 2023-10-11T16:09:20.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MMaFx8iS6iHZYR5df36bq2iLAds=', 'Content-Length': '0'} 2023-10-11T16:09:20.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000651844b7a42f737b-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.471 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.472 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.472 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.472 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.473 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.473 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a51297e1bb1b0231-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:20.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:20.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.478 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.479 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.479 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.479 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.480 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.480 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0605f412c9d8413-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.482 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.486 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.486 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.487 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.487 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.487 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.488 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T16:09:20.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T16:09:20.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084e06d1340b29454-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.493 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.493 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.493 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.494 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.494 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.494 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T16:09:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T16:09:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065786a95f35b41af-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.499 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.500 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.500 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.500 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.500 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.501 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc= 2023-10-11T16:09:20.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc=', 'Content-Length': '0'} 2023-10-11T16:09:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c15a5f7d23198cc-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.506 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.506 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.507 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.507 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.507 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.508 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc= 2023-10-11T16:09:20.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JsXWBp9dMVbrMCanY+oUkNA1Vmc=', 'Content-Length': '0'} 2023-10-11T16:09:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924ca3a581e75ff2-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.509 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.513 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.514 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.514 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.514 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.514 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.515 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T16:09:20.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T16:09:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bdd3fc54597e46a-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.520 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.520 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.520 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.521 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.521 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:20.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.522 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4= 2023-10-11T16:09:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:rC1vCDgXE+uOdh/jN4MjpXQhjh4=', 'Content-Length': '0'} 2023-10-11T16:09:20.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b675860ce1942f1a-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.527 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.527 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.527 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.528 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.528 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.528 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043688fa2fdbb39db-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.534 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.534 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.534 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.535 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.535 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.536 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddddd476a5dbc377-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:20.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.540 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.540 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.541 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.541 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.541 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.542 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI= 2023-10-11T16:09:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI=', 'Content-Length': '0'} 2023-10-11T16:09:20.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732ea1bb283f2f71-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.546 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.546 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.547 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.547 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.547 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.548 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI= 2023-10-11T16:09:20.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:1pc0/r9Bbsqe1hDrd/xMIy3hsHI=', 'Content-Length': '0'} 2023-10-11T16:09:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a00ab577bdffc2c-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.549 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:20.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.553 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.554 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.554 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:20.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.555 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA= 2023-10-11T16:09:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA=', 'Content-Length': '0'} 2023-10-11T16:09:20.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388d3f0ea019a4a4-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:20.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.559 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.559 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.560 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.560 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.560 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:20.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.561 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA= 2023-10-11T16:09:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HisRgS1zj7TOIm1Kd9DAsPh+xUA=', 'Content-Length': '0'} 2023-10-11T16:09:20.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009320e0315a3c8544-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.566 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.567 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.567 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.568 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c076036905df0d41-006526ba1e-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:20.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.573 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.573 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.574 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.574 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.574 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.575 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU= 2023-10-11T16:09:20.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:91Sx+spQ17jfrQjoVltQu7bl4QU=', 'Content-Length': '0'} 2023-10-11T16:09:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8dabf37df1a07d-006526ba1e-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.576 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.580 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.581 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.581 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.581 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.582 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.582 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI= 2023-10-11T16:09:20.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI=', 'Content-Length': '0'} 2023-10-11T16:09:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7506bc2a466b897-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.587 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.587 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.588 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.588 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.588 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.589 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI= 2023-10-11T16:09:20.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NDb4MAK7QCF6FVUkCK/ZQnJg4hI=', 'Content-Length': '0'} 2023-10-11T16:09:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e46b3f02e98972-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.594 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.594 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.594 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.595 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.595 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.596 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc= 2023-10-11T16:09:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc=', 'Content-Length': '0'} 2023-10-11T16:09:20.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004370426ef0730b35-006526ba1e-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:10 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.601 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.601 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.601 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.601 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:10 GMT 2023-10-11T16:09:20.602 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.602 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc= 2023-10-11T16:09:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:10 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:o70cRLQUlfV9Wu4T/6cEkQvIzRc=', 'Content-Length': '0'} 2023-10-11T16:09:20.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8507a65b60329f-006526ba1e-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.603 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.607 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.608 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.608 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.608 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.609 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.609 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g= 2023-10-11T16:09:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g=', 'Content-Length': '0'} 2023-10-11T16:09:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89c3aafef346523-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.614 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.615 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.615 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.615 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.616 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g= 2023-10-11T16:09:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xJflkgzIywGXUs5wVHAmPmjO5/g=', 'Content-Length': '0'} 2023-10-11T16:09:20.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000564af7774e4f0e27-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.620 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.621 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.621 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.621 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.622 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.622 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3bdd40e18998be-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.628 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.628 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.628 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.628 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.629 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.629 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032ab6d8bde9d05bd-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.634 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.634 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.634 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.635 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.635 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.635 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo= 2023-10-11T16:09:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo=', 'Content-Length': '0'} 2023-10-11T16:09:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f22b3f5223587b-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.640 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.640 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.640 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.640 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.641 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.641 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo= 2023-10-11T16:09:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:DVON71xIcQeATzMryIdfVJF6uHo=', 'Content-Length': '0'} 2023-10-11T16:09:20.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000814d492beccab93e-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.642 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.646 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.646 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.647 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.647 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.647 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.648 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8= 2023-10-11T16:09:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8=', 'Content-Length': '0'} 2023-10-11T16:09:20.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002809d7449c00f3aa-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:20.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.652 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.653 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.653 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.653 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.653 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.654 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8= 2023-10-11T16:09:20.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4KxVK5DKx9WKDpRQLL6Kz40/6/8=', 'Content-Length': '0'} 2023-10-11T16:09:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44ecd6411e4e4c1-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.659 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.659 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.659 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.660 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.660 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.661 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000373a70b15e9023c1-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.666 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.666 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.666 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.667 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.667 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.668 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0720d423dc4536f-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.669 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.673 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.673 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.674 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.674 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.674 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.675 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T16:09:20.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T16:09:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004772139006fda781-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.680 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.680 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.680 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.681 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.681 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.682 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T16:09:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T16:09:20.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa08fd7dc057966-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.687 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.688 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.688 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.688 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI= 2023-10-11T16:09:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e70d19d87e1d924-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.693 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.694 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.694 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.694 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.695 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.695 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI= 2023-10-11T16:09:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z/SBr5Gb3d1igUu2VBJB7e+huzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71f6d99042ec8d8-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.700 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.701 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.701 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.701 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.701 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.702 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T16:09:20.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T16:09:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000184470c3097cbbe8-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.707 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.707 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.707 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.708 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.708 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:20.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.709 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs= 2023-10-11T16:09:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:puDVyhiUnK+MQP2aNFQBEwrqWSs=', 'Content-Length': '0'} 2023-10-11T16:09:20.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794f95274a561898-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.714 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.714 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.714 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.715 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.715 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.716 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ba504872f3181a-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.721 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.721 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.721 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.722 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.722 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.723 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9441a808742c994-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.727 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.728 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.728 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.728 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.728 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.729 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8= 2023-10-11T16:09:20.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8=', 'Content-Length': '0'} 2023-10-11T16:09:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617934ea331eb5dd-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:20.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.734 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.734 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.734 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.734 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.735 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:20.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.735 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8= 2023-10-11T16:09:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:5yv1z61QvD1nq1ayO6P4eHZzmd8=', 'Content-Length': '0'} 2023-10-11T16:09:20.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfabac2a80bcf9b3-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.736 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.740 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.740 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.741 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.741 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.741 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.742 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE= 2023-10-11T16:09:20.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE=', 'Content-Length': '0'} 2023-10-11T16:09:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d484836f3ee6bc-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:20.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.746 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.747 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.747 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.747 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.747 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.748 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE= 2023-10-11T16:09:20.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:fQoP4xqYlaCszMF0cUZzcG4NzPE=', 'Content-Length': '0'} 2023-10-11T16:09:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd0679da743cd7bd-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.753 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.753 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.754 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.754 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.754 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.755 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f08a42464b668e-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:20.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.760 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.760 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.761 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.761 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.761 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.762 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3d87de62503cbbc-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.763 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:20.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.767 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.768 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.768 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.768 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.769 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.769 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T16:09:20.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T16:09:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48ef0ce168e7010-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.774 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.775 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.775 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.775 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.776 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.776 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T16:09:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T16:09:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8bc9c44ea7f45a0-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.781 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.782 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.782 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.782 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.783 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.783 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw= 2023-10-11T16:09:20.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw=', 'Content-Length': '0'} 2023-10-11T16:09:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2634d1ba169081-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.788 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.788 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.789 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.789 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.789 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.790 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw= 2023-10-11T16:09:20.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ScsEv13ai3HJhPTQZLQXUMFw3gw=', 'Content-Length': '0'} 2023-10-11T16:09:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326ce9ffcf3a5b05-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.795 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.795 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.796 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.796 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.796 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.797 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T16:09:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T16:09:20.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afca355f49c60d9c-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.802 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.802 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.802 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.803 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.803 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:20.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.804 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM= 2023-10-11T16:09:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ROUHjRUCTnMkJ4M8e/AOEfki7dM=', 'Content-Length': '0'} 2023-10-11T16:09:20.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e56fb0047973702-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:20.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.809 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.809 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.809 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.809 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.810 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.810 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f82b9d480e2dd7d-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:20.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.816 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.816 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.816 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.816 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.817 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.817 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004024ebb46f4a866b-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.822 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.822 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.823 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.823 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.823 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.824 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0= 2023-10-11T16:09:20.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0=', 'Content-Length': '0'} 2023-10-11T16:09:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db5a6556aa024ed-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.828 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.829 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.829 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.829 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.829 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.830 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0= 2023-10-11T16:09:20.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oyNrZypEkmEgqQZwrXVNxl2EjX0=', 'Content-Length': '0'} 2023-10-11T16:09:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006fb6def2def5ce5-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.831 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:20.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:20.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.835 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.835 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.835 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.836 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.836 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:20.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.837 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc= 2023-10-11T16:09:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc=', 'Content-Length': '0'} 2023-10-11T16:09:20.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1455bb81bfe5dcc-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:20.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.841 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.841 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.842 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.842 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.842 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.843 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc= 2023-10-11T16:09:20.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ud0gB6AUHoxBeIWJADraUVqu+Mc=', 'Content-Length': '0'} 2023-10-11T16:09:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d7ed31f0cc8206-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.848 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.848 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.848 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.849 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.849 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.850 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e12551572d8f5fc-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.855 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.855 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.855 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.856 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.856 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.857 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060e959a2ec53bc1e-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.858 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.862 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.862 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.863 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.863 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.863 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.864 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T16:09:20.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a86d089a7b7b92-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.869 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.870 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.870 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.870 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.870 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.871 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T16:09:20.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad095461caea772-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.876 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.876 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.877 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.877 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.878 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E= 2023-10-11T16:09:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E=', 'Content-Length': '0'} 2023-10-11T16:09:20.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a1b740f585439e1-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.882 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.883 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.883 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.883 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.883 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.884 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E= 2023-10-11T16:09:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pe3TWrrSvfcoft97jbK//uL5d7E=', 'Content-Length': '0'} 2023-10-11T16:09:20.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca57da46f21acd90-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.885 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.889 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.889 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.889 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.890 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.890 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.890 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T16:09:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32970ea7ec66ed7-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.895 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.895 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.896 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.896 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.896 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:20.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.897 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI= 2023-10-11T16:09:20.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xsH8d1c9uB1pWTd3qP6WlzrEIzI=', 'Content-Length': '0'} 2023-10-11T16:09:20.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009587881313d17369-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:20.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.901 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.901 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.902 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.902 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.902 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.903 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be4787f59b42ff7a-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:20.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.908 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.908 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.908 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.909 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.909 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.910 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34574061c7604ee-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:20.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.914 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.914 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.915 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.915 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.915 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:20.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.916 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY= 2023-10-11T16:09:20.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY=', 'Content-Length': '0'} 2023-10-11T16:09:20.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9c14a521b22e9b-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:20.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.920 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.920 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.921 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.921 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.921 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.922 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY= 2023-10-11T16:09:20.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BxetTq2VbkZGx9W6I70SbiuwWaY=', 'Content-Length': '0'} 2023-10-11T16:09:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4d83d516e0791a-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:20.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.926 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.927 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.927 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.927 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.928 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.928 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc= 2023-10-11T16:09:20.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc=', 'Content-Length': '0'} 2023-10-11T16:09:20.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005538ecc9c3a12e67-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:20.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:20.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.932 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.933 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.933 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.933 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.933 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.934 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc= 2023-10-11T16:09:20.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nYPegcwIBOthvHz3OGoPNGT5psc=', 'Content-Length': '0'} 2023-10-11T16:09:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e79b8191865ab60-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.939 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.939 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.940 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.940 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.940 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.941 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024482de1ec585031-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:20.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.946 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.946 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.947 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.947 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.947 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.948 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c370f2b74af5c51-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.949 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.953 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.953 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.953 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.954 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.954 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.955 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T16:09:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T16:09:20.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e20a1adc8317fa-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.960 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.960 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.960 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.960 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.961 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.961 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T16:09:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T16:09:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2403e99a69e0d05-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.966 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.967 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.967 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.967 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.967 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.968 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0= 2023-10-11T16:09:20.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0=', 'Content-Length': '0'} 2023-10-11T16:09:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb655b630913645-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.973 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:20.973 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.973 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.974 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.974 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.975 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0= 2023-10-11T16:09:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u66+rT2B8vVk8fa9AvHqivyzXr0=', 'Content-Length': '0'} 2023-10-11T16:09:20.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002039faac7c091e11-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.976 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.980 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.980 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.980 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.981 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.981 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.982 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T16:09:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T16:09:20.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb4a52df2c1bb25-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:20.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.986 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.987 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.987 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.987 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.988 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.988 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY= 2023-10-11T16:09:20.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ahc6nSLyrrHvCvAd0Oa9S0iJNYY=', 'Content-Length': '0'} 2023-10-11T16:09:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ee14ff8b90ef0e5-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:20.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:20.993 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:20.993 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.994 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:20.994 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:20.994 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:20.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:20.995 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:20.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2b38e8ce9dbce30-006526ba1f-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:20.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:20.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:20.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:20.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.000 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.000 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.000 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.000 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:21.001 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.001 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0= 2023-10-11T16:09:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PFZiwb8yV6JCGrQi675zmz7vrt0=', 'Content-Length': '0'} 2023-10-11T16:09:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1dda8607746d48-006526ba1f-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.006 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.006 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.007 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.007 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:21.007 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.008 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU= 2023-10-11T16:09:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU=', 'Content-Length': '0'} 2023-10-11T16:09:21.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000513e1a4843680c63-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.012 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.012 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.013 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.013 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:21.013 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.014 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU= 2023-10-11T16:09:21.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nS+kKtU6WiUZkwhQvLCJwwL84MU=', 'Content-Length': '0'} 2023-10-11T16:09:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fce73e1b6fa358b-006526ba1f-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.015 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:21.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.019 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.019 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.019 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.020 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:11 GMT 2023-10-11T16:09:21.020 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.021 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2oiqGvi95aVg2EGY3H9fqWgwbrQ= 2023-10-11T16:09:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:11 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2oiqGvi95aVg2EGY3H9fqWgwbrQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3655048d361a75e-006526ba1f-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.025 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.025 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.025 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.026 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:12 GMT 2023-10-11T16:09:21.026 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.027 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:u0Lu9KY4btwy9kZMFDRZ0lx7Jq8= 2023-10-11T16:09:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:u0Lu9KY4btwy9kZMFDRZ0lx7Jq8=', 'Content-Length': '0'} 2023-10-11T16:09:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c91c61e9ab139f-006526ba20-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.032 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.032 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.032 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.032 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:12 GMT 2023-10-11T16:09:21.033 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.033 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg= 2023-10-11T16:09:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg=', 'Content-Length': '0'} 2023-10-11T16:09:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c542aea0a4a5f6b8-006526ba20-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.039 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.039 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.039 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.040 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:12 GMT 2023-10-11T16:09:21.040 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.040 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg= 2023-10-11T16:09:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bc7OIsC8X5SZrbAqZtUwK5w0/jg=', 'Content-Length': '0'} 2023-10-11T16:09:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008109ecaa60fbb70a-006526ba20-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:21.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dghqvd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dghqvd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-55 2023-10-11T16:09:21.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 8 2023-10-11T16:09:21.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding": "None", 2023-10-11T16:09:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_index": { 2023-10-11T16:09:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "Normal", 2023-10-11T16:09:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "normal": { 2023-10-11T16:09:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "logs": [ 2023-10-11T16:09:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 0, 2023-10-11T16:09:21.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 11, 2023-10-11T16:09:21.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-55 2023-10-11T16:09:21.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:21.063 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:21.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:21.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:12.381+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T16:09:21.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:17.382+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T16:09:21.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:22.383+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T16:09:21.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:27.383+0000 7faa967ed780 1 waiting to reach latest gen 2, on 1.. 2023-10-11T16:09:21.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:32.384+0000 7faa967ed780 1 bucket sync caught up with empty source 2023-10-11T16:09:21.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:32.384+0000 7faa967ed780 0 bucket checkpoint complete 2023-10-11T16:09:21.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dghqvd-55 2023-10-11T16:09:21.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:21.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:32.504+0000 7f780e303780 1 bucket sync caught up with source: 2023-10-11T16:09:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, 00000000002.50.6, , , 00000000001.186.6, 00000000001.261.6, , , 00000000002.275.6, , 00000000002.135.6] 2023-10-11T16:09:21.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:07:32.504+0000 7f780e303780 0 bucket checkpoint complete 2023-10-11T16:09:21.070 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.073 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.074 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.074 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.074 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.075 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.075 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY= 2023-10-11T16:09:21.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY=', 'Content-Length': '0'} 2023-10-11T16:09:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bac79755a0258da3-006526ba34-1679-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/ 2023-10-11T16:09:21.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.079 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.080 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.080 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.080 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.081 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.081 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY= 2023-10-11T16:09:21.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EiaZ5TSLu2JtlVHLBMW9bZuejpY=', 'Content-Length': '0'} 2023-10-11T16:09:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd405c5c63e17bac-006526ba34-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.082 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-11T16:09:21.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.086 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.087 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.087 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.087 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.087 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.088 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T16:09:21.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T16:09:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4da632847e1ce07-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.089 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:21.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:21.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:21.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:21.090 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-11T16:09:21.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-11T16:09:21.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-11T16:09:21.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-11T16:09:21.092 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-11T16:09:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.096 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.096 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.096 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.097 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.097 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:21.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.097 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T16:09:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T16:09:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027fdc07668d6f212-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-10-11T16:09:21.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-10-11T16:09:21.099 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-10-11T16:09:21.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-10-11T16:09:21.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-10-11T16:09:21.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-10-11T16:09:21.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-10-11T16:09:21.101 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-10-11T16:09:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.105 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.105 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.105 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.106 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.106 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:21.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.106 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T16:09:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T16:09:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000778ed2e3a53564c9-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?versions&max-keys=999 2023-10-11T16:09:21.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.111 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.112 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.112 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.112 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.113 INFO:tasks.rgw_multisite_tests:/dghqvd-55/?versions 2023-10-11T16:09:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.113 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0= 2023-10-11T16:09:21.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WLzL7YG0nKYQ7aXv5jxnB8ByfH0=', 'Content-Length': '0'} 2023-10-11T16:09:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae421a07b2701c99-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55999falseanulltrue2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.114 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.118 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.119 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.119 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.119 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.119 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.120 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T16:09:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T16:09:21.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b687952624865766-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.125 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.125 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.125 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.126 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.126 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.127 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T16:09:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T16:09:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da776b2e4269ade-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.131 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.131 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.132 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.132 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.132 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.133 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c= 2023-10-11T16:09:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c=', 'Content-Length': '0'} 2023-10-11T16:09:21.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825f99769b2c5106-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.138 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.138 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.138 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.139 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.139 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.139 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c= 2023-10-11T16:09:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YmY9qHG7/dEZrhQsIv7Cp3wj91c=', 'Content-Length': '0'} 2023-10-11T16:09:21.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119f1f470500b33f-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.140 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.144 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.145 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.145 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.145 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.146 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.146 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T16:09:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T16:09:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd7adcc0a48513c-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a?versionId=null 2023-10-11T16:09:21.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.152 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.152 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.152 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.153 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.153 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a?versionId=null 2023-10-11T16:09:21.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.154 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w= 2023-10-11T16:09:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Sgl1K1oxBGjQTHWdy09xVvf2J7w=', 'Content-Length': '0'} 2023-10-11T16:09:21.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae931101b83fd86-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:21.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.158 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.159 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.159 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.159 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.160 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.160 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0153d9444e50aa-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:21.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.165 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.166 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.166 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.166 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.167 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.167 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf33dd2fef30e4e6-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.172 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.172 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.172 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.173 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.173 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:21.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.173 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY= 2023-10-11T16:09:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY=', 'Content-Length': '0'} 2023-10-11T16:09:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d55514d4b4446fe2-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.178 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.178 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.178 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.179 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.179 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:21.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.179 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY= 2023-10-11T16:09:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SQmjYc1t3lNYlj6aHCioaDM2ixY=', 'Content-Length': '0'} 2023-10-11T16:09:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22ef61269810abf-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.180 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-10-11T16:09:21.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:21.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.183 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.184 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.184 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.184 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.185 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:21.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.185 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE= 2023-10-11T16:09:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE=', 'Content-Length': '0'} 2023-10-11T16:09:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad00efa5fce9f6a1-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/a 2023-10-11T16:09:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/a 2023-10-11T16:09:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/a 2023-10-11T16:09:21.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.189 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.190 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.190 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.190 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.191 INFO:tasks.rgw_multisite_tests:/dghqvd-55/a 2023-10-11T16:09:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.191 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE= 2023-10-11T16:09:21.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9k3uZW45MZcsibB4rzO9+0HtNfE=', 'Content-Length': '0'} 2023-10-11T16:09:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f865d5edeb5cf2-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.196 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.196 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.197 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.197 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.197 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.198 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf3cafdb55bc777-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=a 2023-10-11T16:09:21.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=a 2023-10-11T16:09:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.203 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.203 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.204 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.204 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.204 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.205 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dedffd938b857de2-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55a1000falsea2023-10-11T15:03:52.775Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.206 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:21.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.210 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.210 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.211 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.211 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.211 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.212 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T16:09:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T16:09:21.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c1137552c4c86b8-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.217 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.217 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.217 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.217 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.218 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.218 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T16:09:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T16:09:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc1d0754db27ff6d-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.223 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.223 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.223 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.224 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.224 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.225 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4= 2023-10-11T16:09:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4=', 'Content-Length': '0'} 2023-10-11T16:09:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f140b49fd925dfd9-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.230 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.230 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.230 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.231 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.231 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.232 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4= 2023-10-11T16:09:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+xbI8oz6ORJer4CRzLJv8gU/Qe4=', 'Content-Length': '0'} 2023-10-11T16:09:21.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002385c78039c2d2bb-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.232 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.237 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.237 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.237 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.238 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.238 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.239 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T16:09:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T16:09:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1d9bf9e563dc15-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b?versionId=null 2023-10-11T16:09:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.243 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.243 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.244 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.244 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.244 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b?versionId=null 2023-10-11T16:09:21.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.245 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI= 2023-10-11T16:09:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ore16jINKbqw8h/+To8vmRSIrqI=', 'Content-Length': '0'} 2023-10-11T16:09:21.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbd4780b44dc07cd-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.250 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.250 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.250 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.251 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.251 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.252 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586f3fdffdd9780e-006526ba34-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.257 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.257 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.257 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.258 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.258 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.259 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae52c0fcbcafbe7c-006526ba34-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.263 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.263 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.264 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.264 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.264 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.265 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ= 2023-10-11T16:09:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8e89f3a3cbc6564-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.269 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.269 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.270 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.270 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.270 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.271 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ= 2023-10-11T16:09:21.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ARGXGCA8TDk2CIGi68dCll/WWnQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f8a73aef73dc23-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.272 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-10-11T16:09:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:21.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.276 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.276 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.276 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.276 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.277 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:21.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.277 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk= 2023-10-11T16:09:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk=', 'Content-Length': '0'} 2023-10-11T16:09:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e575e62928df405a-006526ba34-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/b 2023-10-11T16:09:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/b 2023-10-11T16:09:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/b 2023-10-11T16:09:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.283 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.283 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.284 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.284 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.284 INFO:tasks.rgw_multisite_tests:/dghqvd-55/b 2023-10-11T16:09:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.285 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk= 2023-10-11T16:09:21.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:YQM0QpaHOIpJiNARshbNjpIpPZk=', 'Content-Length': '0'} 2023-10-11T16:09:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009331d1da0acb998a-006526ba34-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:32 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.290 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.290 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.291 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.291 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:32 GMT 2023-10-11T16:09:21.291 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.292 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc= 2023-10-11T16:09:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:32 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eGL+BlZhMhXbLqMrPIucbk/IULc=', 'Content-Length': '0'} 2023-10-11T16:09:21.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001776325fc3f20977-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=b 2023-10-11T16:09:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=b 2023-10-11T16:09:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.297 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.297 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.297 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.298 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.298 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.299 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072f362b7c5e06f57-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55b1000falseb2023-10-11T15:03:52.795Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.300 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.304 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.304 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.304 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.305 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.305 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.306 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T16:09:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c5372c8c406e90-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.311 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.311 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.311 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.311 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.312 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.312 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T16:09:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000643dfefcb368998d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.317 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.318 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.318 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.318 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.319 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw= 2023-10-11T16:09:21.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw=', 'Content-Length': '0'} 2023-10-11T16:09:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1fc2eb05ecd48cf-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.324 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.324 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.324 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.325 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.325 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.326 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw= 2023-10-11T16:09:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:dqUkcAueR6I94CwC/9gUMJ1eISw=', 'Content-Length': '0'} 2023-10-11T16:09:21.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39acd1681b5879a-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.327 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.331 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.331 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.331 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.332 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.332 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.333 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T16:09:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d7d53b4c84e508-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c?versionId=null 2023-10-11T16:09:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.337 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.338 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.338 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.338 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.339 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c?versionId=null 2023-10-11T16:09:21.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.339 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ= 2023-10-11T16:09:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vNiGe2oCtjSNiIfPMyyLzrHQoaQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1fd65289e444d2-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.344 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.345 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.345 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.345 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.345 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.346 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4e69a7f8bcd874-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.351 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.351 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.352 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.352 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.352 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.353 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b131a851217edd5b-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:21.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.357 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.358 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.358 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.358 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.359 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.359 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk= 2023-10-11T16:09:21.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk=', 'Content-Length': '0'} 2023-10-11T16:09:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e78024897dc12f2-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:21.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.363 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.364 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.364 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.364 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.365 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:21.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.365 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk= 2023-10-11T16:09:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WfePrgHDxwAKVShUPirnhFgnOsk=', 'Content-Length': '0'} 2023-10-11T16:09:21.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d550c5edfea7fc9-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.366 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-10-11T16:09:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.370 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.370 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.371 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.371 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.371 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.372 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ= 2023-10-11T16:09:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afd17c6b055ccfc-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/c 2023-10-11T16:09:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/c 2023-10-11T16:09:21.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/c 2023-10-11T16:09:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.376 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.376 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.377 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.377 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.377 INFO:tasks.rgw_multisite_tests:/dghqvd-55/c 2023-10-11T16:09:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.378 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ= 2023-10-11T16:09:21.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VOi7/rPYpurUfbPYYsznLjhDpUQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64fe9f8b7d7273c-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.383 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.383 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.383 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.384 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.384 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.385 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6c10aea9c66f686-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=c 2023-10-11T16:09:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=c 2023-10-11T16:09:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.390 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.390 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.391 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.391 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.391 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.392 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000916f9069a1281bb7-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55c1000falsec2023-10-11T15:03:52.853Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.393 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.397 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.397 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.397 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.398 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.398 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.399 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T16:09:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T16:09:21.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a9b0450e40fc42-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.403 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.404 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.404 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.404 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.405 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.405 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T16:09:21.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T16:09:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bcada3eb68360c-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.410 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.411 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.411 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.411 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.411 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.412 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg= 2023-10-11T16:09:21.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg=', 'Content-Length': '0'} 2023-10-11T16:09:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff9fe4b6b64e93e-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.417 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.417 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.417 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.418 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.418 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.419 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg= 2023-10-11T16:09:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EmV/PvG3mPHQ5DGS4nPsQ6WSfeg=', 'Content-Length': '0'} 2023-10-11T16:09:21.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4838022b8d4e29-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.420 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.424 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.424 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.425 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.425 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.425 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.426 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T16:09:21.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T16:09:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0629f2e73fa97a6-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d?versionId=null 2023-10-11T16:09:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.431 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.431 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.432 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.432 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.432 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d?versionId=null 2023-10-11T16:09:21.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.433 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs= 2023-10-11T16:09:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HtpqCyeTyM6dMot4KU+RHIwLNqs=', 'Content-Length': '0'} 2023-10-11T16:09:21.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323a4b02593cad0a-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.438 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.438 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.438 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.439 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.439 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.439 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e126e637fd10ed45-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:21.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.444 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.445 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.445 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.445 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.446 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.446 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036b55dd41f7156b1-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.451 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.451 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.452 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.452 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.452 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:21.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.453 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE= 2023-10-11T16:09:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE=', 'Content-Length': '0'} 2023-10-11T16:09:21.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb68595b31ebe76-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.457 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.457 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.458 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.458 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.458 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.459 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE= 2023-10-11T16:09:21.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6p9J2Z0bkJzkrxXlInA56qHeJBE=', 'Content-Length': '0'} 2023-10-11T16:09:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d578b6d47a9c316-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.460 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-10-11T16:09:21.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:21.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.464 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.464 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.464 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.465 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.465 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:21.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.466 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4= 2023-10-11T16:09:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4=', 'Content-Length': '0'} 2023-10-11T16:09:21.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11e83a83033b904-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/d 2023-10-11T16:09:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/d 2023-10-11T16:09:21.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/d 2023-10-11T16:09:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.470 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.470 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.470 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.471 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.471 INFO:tasks.rgw_multisite_tests:/dghqvd-55/d 2023-10-11T16:09:21.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.472 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4= 2023-10-11T16:09:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2DK2d4JBQtdPMXf4zpA700AzQc4=', 'Content-Length': '0'} 2023-10-11T16:09:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:03:52 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7f2da3bb209ae9-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:21.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.476 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.477 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.477 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.477 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.478 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.478 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b93b74c4f87937-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=d 2023-10-11T16:09:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=d 2023-10-11T16:09:21.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.483 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.484 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.484 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.484 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.485 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.485 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc2a028083a888d-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55d1000falsed2023-10-11T15:03:52.871Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.486 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.492 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.492 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.493 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.493 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.493 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.494 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T16:09:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T16:09:21.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799195b9a3b53ab5-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.499 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.499 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.499 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.500 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.500 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.501 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T16:09:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T16:09:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fba8c42e1cc374-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.506 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.506 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.506 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.507 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.507 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA= 2023-10-11T16:09:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA=', 'Content-Length': '0'} 2023-10-11T16:09:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353dd576881c11a9-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.512 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.513 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.513 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.513 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.513 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.514 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA= 2023-10-11T16:09:21.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gEJBOfXFMqXahV2kf8X+AXOs0eA=', 'Content-Length': '0'} 2023-10-11T16:09:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e04e814f067444-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.515 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:21.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.519 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.520 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.520 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.520 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.521 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.521 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T16:09:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T16:09:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75e9dbd2c14feae-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e?versionId=null 2023-10-11T16:09:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.526 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.527 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.527 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.527 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.527 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e?versionId=null 2023-10-11T16:09:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.528 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg= 2023-10-11T16:09:21.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:7sK3QQgW12vwUx6mjGebDZmAzxg=', 'Content-Length': '0'} 2023-10-11T16:09:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c3049cb5be21f4-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.533 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.533 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.534 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.534 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.534 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.535 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a855b71b857cba4e-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.540 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.540 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.541 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.541 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.541 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.542 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000397c5ecb4278132-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.547 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.547 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.547 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.548 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.548 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:21.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.549 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M= 2023-10-11T16:09:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M=', 'Content-Length': '0'} 2023-10-11T16:09:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333eb761e7f8e030-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.553 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.554 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.554 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:21.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.555 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M= 2023-10-11T16:09:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:yD8UC7siijkUAl6Lq9LNDbEVs9M=', 'Content-Length': '0'} 2023-10-11T16:09:21.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1852090f0286031-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.556 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-10-11T16:09:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:21.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.559 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.560 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.560 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.560 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.561 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:21.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.561 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc= 2023-10-11T16:09:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc=', 'Content-Length': '0'} 2023-10-11T16:09:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006134feac37a54dab-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/e 2023-10-11T16:09:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/e 2023-10-11T16:09:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/e 2023-10-11T16:09:21.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.566 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.566 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.567 INFO:tasks.rgw_multisite_tests:/dghqvd-55/e 2023-10-11T16:09:21.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.567 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc= 2023-10-11T16:09:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MwjjlRLqw99ypSIOmPGD3YB8twc=', 'Content-Length': '0'} 2023-10-11T16:09:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca45b92f25edf48d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.572 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.573 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.573 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.573 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.573 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.574 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f4ac4102e1d497-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=e 2023-10-11T16:09:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=e 2023-10-11T16:09:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.579 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.580 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.580 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.580 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.580 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.581 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b403d9330144a794-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55e1000falsee2023-10-11T15:04:34.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.587 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.587 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.587 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.588 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.588 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.589 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T16:09:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T16:09:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448e0b3700cdafd0-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.593 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.594 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.594 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.594 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.594 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.595 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T16:09:21.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T16:09:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d497fd86baaec894-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.600 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.600 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.601 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.601 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.601 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.602 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM= 2023-10-11T16:09:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM=', 'Content-Length': '0'} 2023-10-11T16:09:21.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc1a0233aea80d2a-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.606 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.607 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.607 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.607 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.608 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.608 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM= 2023-10-11T16:09:21.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ia4qTp8zwdsmnQ7X0/tgn85KHSM=', 'Content-Length': '0'} 2023-10-11T16:09:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081c5e54f478e067d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.609 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:21.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.613 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.614 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.615 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.615 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T16:09:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T16:09:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000416400f393662a8e-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f?versionId=null 2023-10-11T16:09:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.620 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.621 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.621 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.621 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.621 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f?versionId=null 2023-10-11T16:09:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.622 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA= 2023-10-11T16:09:21.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jlkza+ihO4vXr7ulfchlDnDXJPA=', 'Content-Length': '0'} 2023-10-11T16:09:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027be72b6f1c179f6-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.627 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.627 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.628 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.628 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.628 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.629 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8231413ad6dda71-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.634 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.634 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.635 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.635 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.635 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.636 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec04010ac5b470c2-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.640 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.641 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.641 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.641 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.641 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.642 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs= 2023-10-11T16:09:21.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs=', 'Content-Length': '0'} 2023-10-11T16:09:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4974ee1411d4a2d-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:21.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.646 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.647 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.647 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.647 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.648 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:21.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.648 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs= 2023-10-11T16:09:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:B/Dk+0c2rekSpnYc3Pi8YmAsKhs=', 'Content-Length': '0'} 2023-10-11T16:09:21.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996ff6092546642d-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.649 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-10-11T16:09:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.653 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.653 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.653 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.654 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.654 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:21.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.655 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8= 2023-10-11T16:09:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8=', 'Content-Length': '0'} 2023-10-11T16:09:21.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e3b5247785ce4b-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/f 2023-10-11T16:09:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/f 2023-10-11T16:09:21.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/f 2023-10-11T16:09:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.659 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.659 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.660 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.660 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.660 INFO:tasks.rgw_multisite_tests:/dghqvd-55/f 2023-10-11T16:09:21.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.661 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8= 2023-10-11T16:09:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0xKX8Ld0siLbvoJBgOpQvlqxlu8=', 'Content-Length': '0'} 2023-10-11T16:09:21.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63794c43d34a876-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:21.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.665 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.666 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.666 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.666 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.667 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.667 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093dcaa6528fd5367-006526ba35-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=f 2023-10-11T16:09:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=f 2023-10-11T16:09:21.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.672 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.673 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.673 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.673 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.674 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.674 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0= 2023-10-11T16:09:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:AFTD8jyRfPjS6IzX1X+sMOhvyc0=', 'Content-Length': '0'} 2023-10-11T16:09:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdde13e60663426d-006526ba35-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55f1000falsef2023-10-11T15:04:34.831Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.676 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.680 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.680 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.681 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.681 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.681 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.682 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T16:09:21.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T16:09:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000219d68e5c42e689d-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.687 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.689 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.689 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.690 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T16:09:21.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T16:09:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d09cd8b690bddb-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.695 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.695 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.696 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.696 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.696 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.697 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY= 2023-10-11T16:09:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY=', 'Content-Length': '0'} 2023-10-11T16:09:21.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e13e56ae050c26-006526ba35-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.702 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.702 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.702 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.702 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.703 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.703 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY= 2023-10-11T16:09:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pWE5cKvMpZ2whMUgIpi7c9axvcY=', 'Content-Length': '0'} 2023-10-11T16:09:21.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37252c2ff690e55-006526ba35-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:33 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.704 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.709 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.709 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.709 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.710 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:33 GMT 2023-10-11T16:09:21.710 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.711 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs= 2023-10-11T16:09:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:33 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PB70TxAlfESb6DsNKaASkOhOuUs=', 'Content-Length': '0'} 2023-10-11T16:09:21.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bac85b6e84bfbfe-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g?versionId=null 2023-10-11T16:09:21.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.716 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.716 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.716 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.717 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.717 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g?versionId=null 2023-10-11T16:09:21.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.717 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pv1nZn9beWkF9tgTBJdVJ/1NGWA= 2023-10-11T16:09:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pv1nZn9beWkF9tgTBJdVJ/1NGWA=', 'Content-Length': '0'} 2023-10-11T16:09:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000150a4eb15b3feb0a-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:21.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.722 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.723 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.723 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.723 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.724 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.724 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef0784fd713df118-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.730 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.730 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.731 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.731 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.731 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.732 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000173cc7635199e6bf-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.736 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.736 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.737 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.737 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.737 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:21.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.738 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o= 2023-10-11T16:09:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o=', 'Content-Length': '0'} 2023-10-11T16:09:21.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1596d0366a5c526-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.742 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.742 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.743 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.743 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.743 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.744 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o= 2023-10-11T16:09:21.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BecmgQGXSYy/33Cv6n5vvcCTq3o=', 'Content-Length': '0'} 2023-10-11T16:09:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06e3d97296578ee-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.745 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-10-11T16:09:21.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.749 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.749 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.749 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.750 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.750 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:21.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.751 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ= 2023-10-11T16:09:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b28eea3cc31957-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/g 2023-10-11T16:09:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/g 2023-10-11T16:09:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/g 2023-10-11T16:09:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.755 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.755 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.756 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.756 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.756 INFO:tasks.rgw_multisite_tests:/dghqvd-55/g 2023-10-11T16:09:21.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.757 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ= 2023-10-11T16:09:21.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:X5J74MKYGJFfrsz5nY5iKkkpCCQ=', 'Content-Length': '0'} 2023-10-11T16:09:21.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032de5db1fe39e2ed-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.762 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.762 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.762 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.763 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.763 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.764 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008291425e12f8f01-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=g 2023-10-11T16:09:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=g 2023-10-11T16:09:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.769 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.769 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.770 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.770 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.770 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.771 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9cd23e42ec19d3-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55g1000falseg2023-10-11T15:04:34.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.772 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:21.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.776 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.777 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.777 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.777 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.778 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.778 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T16:09:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T16:09:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1eb0cfc959ab05b-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.783 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.783 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.784 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.784 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.784 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.785 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T16:09:21.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T16:09:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a169efc1e24191-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.790 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.790 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.791 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.791 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.791 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.792 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw= 2023-10-11T16:09:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw=', 'Content-Length': '0'} 2023-10-11T16:09:21.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f1a9d2e9c2ceb9-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.797 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.797 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.797 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.798 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.798 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.799 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw= 2023-10-11T16:09:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OsptMCcf6vHVwZlQyCBa8aAHFmw=', 'Content-Length': '0'} 2023-10-11T16:09:21.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af180b49eabce22e-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.799 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.804 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.804 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.804 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.805 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.805 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.805 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T16:09:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T16:09:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f137f25f5bfa1f-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h?versionId=null 2023-10-11T16:09:21.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.810 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.811 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.811 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.811 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.812 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h?versionId=null 2023-10-11T16:09:21.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.812 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs= 2023-10-11T16:09:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hXwmscusceekg56S7ATTsvzPtFs=', 'Content-Length': '0'} 2023-10-11T16:09:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233275cffee02586-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.817 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.817 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.818 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.818 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.818 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.819 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae2372aa79f6111-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.824 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.824 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.825 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.825 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.825 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.826 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb2f62c65b41f4aa-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.831 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.831 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.831 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.832 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.832 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.833 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A= 2023-10-11T16:09:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A=', 'Content-Length': '0'} 2023-10-11T16:09:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c822e375b84e81-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:21.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.837 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:21.837 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.838 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.838 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.838 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.839 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A= 2023-10-11T16:09:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZR88HzJeWULwaqr2PxWa66rPk8A=', 'Content-Length': '0'} 2023-10-11T16:09:21.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099524b4eee559aa5-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.840 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-10-11T16:09:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.843 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.844 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.844 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.844 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.844 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.845 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE= 2023-10-11T16:09:21.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE=', 'Content-Length': '0'} 2023-10-11T16:09:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-55:daa316e9-869c-4074-a756-a8adfb4a3573.5751.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ce58d63d1dd29b-006526ba36-1679-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/h 2023-10-11T16:09:21.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/h 2023-10-11T16:09:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/h 2023-10-11T16:09:21.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.849 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.850 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.850 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.850 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.851 INFO:tasks.rgw_multisite_tests:/dghqvd-55/h 2023-10-11T16:09:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.851 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE= 2023-10-11T16:09:21.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:D6gHHD4hJh0u/aecuFwzC3nCAuE=', 'Content-Length': '0'} 2023-10-11T16:09:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:04:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f492461f8fd83e-006526ba36-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.856 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.856 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.856 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.857 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.857 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.857 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499be8118ed35ba8-006526ba36-1679-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/ 2023-10-11T16:09:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/ 2023-10-11T16:09:21.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-55/?prefix=h 2023-10-11T16:09:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:21.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-55/?prefix=h 2023-10-11T16:09:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.863 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:21.863 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.863 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.864 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:07:34 GMT 2023-10-11T16:09:21.864 INFO:tasks.rgw_multisite_tests:/dghqvd-55/ 2023-10-11T16:09:21.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.865 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE= 2023-10-11T16:09:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:07:34 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lJ+xPi3UZKKhUR5oHb2E21LXoqE=', 'Content-Length': '0'} 2023-10-11T16:09:21.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088eacfde3c410827-006526ba36-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:07:34 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-55h1000falseh2023-10-11T15:04:34.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:21.866 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dghqvd-55 zones={a2, a1} 2023-10-11T16:09:21.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.868 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.417+0000 7fc432371700 -1 req 1802128087161529062 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-6:daa316e9-869c-4074-a756-a8adfb4a3573.4373.6,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.868 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.417+0000 7fc384a16700 -1 req 4206395784897013624 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-43:daa316e9-869c-4074-a756-a8adfb4a3573.5751.8,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.868 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.417+0000 7fc33717b700 -1 req 10502105560425516689 0.000999990s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-30:daa316e9-869c-4074-a756-a8adfb4a3573.4373.30,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.869 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.418+0000 7fc42e369700 -1 req 1278840046035576592 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-42:daa316e9-869c-4074-a756-a8adfb4a3573.5751.7,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.869 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.419+0000 7fc41c345700 -1 req 5401398298012547829 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-24:daa316e9-869c-4074-a756-a8adfb4a3573.4373.24,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.869 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.460+0000 7fc415b38700 -1 req 1360939669116435187 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-20:daa316e9-869c-4074-a756-a8adfb4a3573.4373.20,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.870 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.461+0000 7fc427b5c700 -1 req 2775820898864771056 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-4:daa316e9-869c-4074-a756-a8adfb4a3573.4373.4,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.870 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.465+0000 7fc413b34700 -1 req 12211526065910358041 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-21:daa316e9-869c-4074-a756-a8adfb4a3573.4373.21,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.870 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.502+0000 7fc422351700 -1 req 1475105592868418938 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-25:daa316e9-869c-4074-a756-a8adfb4a3573.4373.25,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.871 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.507+0000 7fc40f32b700 -1 req 501008692447420487 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-2:daa316e9-869c-4074-a756-a8adfb4a3573.4373.2,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding": "None", 2023-10-11T16:09:21.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_index": { 2023-10-11T16:09:21.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "Normal", 2023-10-11T16:09:21.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "normal": { 2023-10-11T16:09:21.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "logs": [ 2023-10-11T16:09:21.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.887 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.759+0000 7fc403313700 -1 req 4490894029272329692 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-51:daa316e9-869c-4074-a756-a8adfb4a3573.5751.16,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.887 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.765+0000 7fc413333700 -1 req 9408799144605805262 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-23:daa316e9-869c-4074-a756-a8adfb4a3573.4373.23,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.887 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.766+0000 7fc3b1a70700 -1 req 14024864955847415538 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-28:daa316e9-869c-4074-a756-a8adfb4a3573.4373.28,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.888 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.811+0000 7fc3a3a54700 -1 req 6312820520116418527 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-19:daa316e9-869c-4074-a756-a8adfb4a3573.4373.19,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.888 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.813+0000 7fc3e32d3700 -1 req 2451397968902671857 0.000999989s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-36:daa316e9-869c-4074-a756-a8adfb4a3573.5751.1,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.888 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:07:34.856+0000 7fc3ab263700 -1 req 5244464425991346776 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=daa316e9-869c-4074-a756-a8adfb4a3573,az=0},d={b=dghqvd-26:daa316e9-869c-4074-a756-a8adfb4a3573.4373.26,z=37b32c30-76c0-479d-b458-e4824d081f13,az=0}} returned ret=-2 2023-10-11T16:09:21.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dghqvd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:21.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dghqvd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding": "None", 2023-10-11T16:09:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_index": { 2023-10-11T16:09:21.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "Normal", 2023-10-11T16:09:21.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "normal": { 2023-10-11T16:09:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "logs": [ 2023-10-11T16:09:21.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 1, 2023-10-11T16:09:21.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "type": "InIndex", 2023-10-11T16:09:21.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "in_index": { 2023-10-11T16:09:21.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "gen": 2, 2023-10-11T16:09:21.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "layout": { 2023-10-11T16:09:21.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 3, 2023-10-11T16:09:21.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hash_type": "Mod" 2023-10-11T16:09:21.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.904 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-11T16:09:21.904 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.905 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-11T16:09:21.905 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-11T16:09:21.905 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:21.906 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-11T16:09:21.906 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-11T16:09:21.907 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-11T16:09:21.907 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-10-11T16:09:21.907 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-11T16:09:21.907 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-11T16:09:21.908 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-11T16:09:21.908 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-11T16:09:21.908 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-11T16:09:21.909 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-11T16:09:21.909 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-11T16:09:21.909 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-11T16:09:21.910 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-11T16:09:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:21.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:21.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:21.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:21.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:21.913 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:21.914 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.914 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.914 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:01 GMT 2023-10-11T16:09:21.915 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:21.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:21.915 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:98GGLTNRHLptXpJIMNE9FbzqtGk= 2023-10-11T16:09:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:98GGLTNRHLptXpJIMNE9FbzqtGk=', 'Content-Length': '0'} 2023-10-11T16:09:21.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000166d13f94cc0cb46-006526c159-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:21.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dghqvd-63 2023-10-11T16:09:21.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:21.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:21.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:21.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038681.046367_553.1", 2023-10-11T16:09:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:38:01.046367Z" 2023-10-11T16:09:21.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:21.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:21.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.925 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:21.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:01.226+0000 7f04d8e2b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:01.227+0000 7f04d8e2b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:21.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:21.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:21.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:21.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:21.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:21.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:21.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:21.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:21.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:21.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:21.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:21.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:01Z" 2023-10-11T16:09:21.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:21.947 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:21.947 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:21.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:21.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:21.948 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:21.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.949 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.949 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:21.949 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.950 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:21.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:21.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.951 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:21.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:21.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.952 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.953 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.953 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.953 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:21.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.954 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:21.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:21.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.955 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:21.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.956 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.956 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.956 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:21.957 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:21.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:21.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:21.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.959 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.960 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.960 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.960 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:21.960 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:21.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:21.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:21.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.963 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.963 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.963 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:21.964 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.964 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:21.964 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:21.965 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:21.965 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.965 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:01Z" 2023-10-11T16:09:21.966 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:21.966 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:21.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T16:09:21.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:21.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T16:09:21.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T16:09:21.967 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:21.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:06.345+0000 7f9f29d51780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:21.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:06.346+0000 7f9f29d51780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:21.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:21.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:21.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:21.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:21.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:21.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:21.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:21.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:21.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:21.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:21.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:21.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:21.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:21.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:21.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:21.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:21.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:21.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:21.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:21.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:21.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:21.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:21.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:21.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:21.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:21.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:06Z" 2023-10-11T16:09:21.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:21.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:21.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:21.990 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:21.991 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:21.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:21.991 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:21.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.992 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.992 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:21.992 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.992 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:21.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:21.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.994 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:21.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:21.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.995 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.995 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.996 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.996 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:21.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:21.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:21.997 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:21.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:21.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:21.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:21.998 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:21.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:21.998 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:21.999 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:21.999 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:21.999 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.000 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.002 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.002 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.002 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.003 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.003 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.005 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.005 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.006 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.006 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.006 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.007 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.007 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.007 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.007 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:06Z" 2023-10-11T16:09:22.008 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.008 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T16:09:22.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T16:09:22.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:11.464+0000 7f83719e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:11.465+0000 7f83719e6780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:22.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:22.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:22.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:22.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:22.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:11Z" 2023-10-11T16:09:22.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:22.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:22.034 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:22.035 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:22.035 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:22.035 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.036 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.036 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:22.036 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.037 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:22.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.038 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:22.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.039 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.040 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.040 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.040 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.040 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:22.041 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.041 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:22.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.043 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:22.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.043 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.044 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.044 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.044 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.045 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.047 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.047 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.048 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.048 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.048 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.051 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.051 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.052 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.052 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.052 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.053 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.053 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.053 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.053 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:11Z" 2023-10-11T16:09:22.054 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.054 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T16:09:22.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T16:09:22.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.056 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:16.584+0000 7ffa10843780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:16.585+0000 7ffa10843780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:22.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:22.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:22.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:22.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:22.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:22.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:22.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:16Z" 2023-10-11T16:09:22.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:22.078 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:22.079 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:22.079 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:22.079 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:22.080 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.080 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.081 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:22.081 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.081 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:22.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.082 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.082 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.083 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:22.083 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.083 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.084 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.084 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.085 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:22.085 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.086 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:22.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.087 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:22.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.088 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.088 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.088 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.089 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.089 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.089 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.091 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.092 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.092 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.092 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.095 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.096 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.096 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.096 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.096 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.097 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.097 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.097 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.098 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:16Z" 2023-10-11T16:09:22.098 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.098 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T16:09:22.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T16:09:22.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:21.706+0000 7f30bb2eb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:21.707+0000 7f30bb2eb780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:22.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:22.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:22.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:22.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:22.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:22.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:22.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:22.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:22.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:21Z" 2023-10-11T16:09:22.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.122 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:22.122 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:22.122 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:22.123 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:22.123 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:22.123 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.124 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.124 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:22.125 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.125 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:22.125 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.125 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.126 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.126 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:22.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.128 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.128 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.128 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.129 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:22.129 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.129 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:22.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.131 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:22.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.131 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.132 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.132 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.132 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.132 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.133 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.135 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.135 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.136 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.136 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.136 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038554.119201_549.1", 2023-10-11T16:09:22.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:35:54.119201Z", 2023-10-11T16:09:22.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.139 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.139 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.139 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.140 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.141 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.141 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.141 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:21Z" 2023-10-11T16:09:22.142 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.142 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038554.119201_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:35:54.119201Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038554.119201_549.1'} 2023-10-11T16:09:22.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697038681.046367_553.1 target=1_1697038554.119201_549.1 2023-10-11T16:09:22.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.144 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:26.830+0000 7f4c3ef26780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:26.831+0000 7f4c3ef26780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:22.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:22.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:22.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:22.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:22.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:22.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038681.046367_553.1", 2023-10-11T16:09:22.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:38:01.046367Z", 2023-10-11T16:09:22.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:22.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:22.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:22.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:38:26Z" 2023-10-11T16:09:22.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:22.166 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:22.166 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:22.166 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:22.167 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:22.167 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.168 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.168 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:22.168 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.169 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:22.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.169 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.170 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:22.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.171 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.172 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.172 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.172 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:22.173 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.173 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:22.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.174 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:22.175 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.175 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.175 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.176 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.176 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.176 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.176 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.177 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.177 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.177 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.178 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.178 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.178 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.179 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.179 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.179 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.180 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.180 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.180 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038681.046367_553.1", 2023-10-11T16:09:22.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:38:01.046367Z", 2023-10-11T16:09:22.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.182 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.183 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.183 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.183 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.184 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.184 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.184 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.185 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.185 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:38:26Z" 2023-10-11T16:09:22.185 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.185 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697038681.046367_553.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:38:01.046367Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697038681.046367_553.1'} 2023-10-11T16:09:22.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.190 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.191 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.191 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.191 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.192 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.192 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T16:09:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T16:09:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003600e0738f1cb693-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj11 2023-10-11T16:09:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj11 2023-10-11T16:09:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj11 2023-10-11T16:09:22.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.197 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.197 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.197 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.198 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.198 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj11 2023-10-11T16:09:22.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.199 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:IT9TJnDT9x0eVzK0ZAfrZiTh1iY= 2023-10-11T16:09:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:IT9TJnDT9x0eVzK0ZAfrZiTh1iY='} 2023-10-11T16:09:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af848d6b90b1a4cc-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.203 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.203 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.204 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.204 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.204 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.205 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T16:09:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T16:09:22.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e53ed03168c62ce-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj22 2023-10-11T16:09:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj22 2023-10-11T16:09:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj22 2023-10-11T16:09:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.209 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.209 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.209 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.210 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.210 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj22 2023-10-11T16:09:22.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.211 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UH1KqnfCxA9wrSNYKUBQZozqCus= 2023-10-11T16:09:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UH1KqnfCxA9wrSNYKUBQZozqCus='} 2023-10-11T16:09:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016bc7bb285c87211-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.215 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.215 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.215 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.216 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.216 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.217 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T16:09:22.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T16:09:22.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dea82871e9517bd4-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj33 2023-10-11T16:09:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj33 2023-10-11T16:09:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj33 2023-10-11T16:09:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.221 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.221 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.221 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.222 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.222 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj33 2023-10-11T16:09:22.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.223 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:G1y3yNF0ln7EbL4mB+r6+ZkGaVo= 2023-10-11T16:09:22.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:G1y3yNF0ln7EbL4mB+r6+ZkGaVo='} 2023-10-11T16:09:22.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3b8b7f1598833c9-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.227 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.227 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.228 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.228 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.228 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.229 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T16:09:22.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T16:09:22.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645075452b45e415-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj44 2023-10-11T16:09:22.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj44 2023-10-11T16:09:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj44 2023-10-11T16:09:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.233 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.234 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.234 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.234 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.235 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj44 2023-10-11T16:09:22.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.235 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jOwhl5UH1qsdbfVBmPTpUw3KIf8= 2023-10-11T16:09:22.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jOwhl5UH1qsdbfVBmPTpUw3KIf8='} 2023-10-11T16:09:22.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3ec626978b334bd-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.239 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.240 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.240 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.240 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.240 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.241 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8= 2023-10-11T16:09:22.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:RBc63s7lqXdpI7Jm9KQzuqeZCl8=', 'Content-Length': '0'} 2023-10-11T16:09:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbbc5c01a78a6c56-006526c172-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:26 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj55 2023-10-11T16:09:22.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj55 2023-10-11T16:09:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj55 2023-10-11T16:09:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.245 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.245 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.246 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.246 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:26 GMT 2023-10-11T16:09:22.246 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj55 2023-10-11T16:09:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.247 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ni7ca7b8t1OMiULdccTimySwv3A= 2023-10-11T16:09:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:26 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ni7ca7b8t1OMiULdccTimySwv3A='} 2023-10-11T16:09:22.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c59af43569dfbce-006526c172-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.248 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 5 2023-10-11T16:09:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.253 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.253 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.253 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.254 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.254 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.255 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d393f4f6df229de9-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj17 2023-10-11T16:09:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj17 2023-10-11T16:09:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj17 2023-10-11T16:09:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.259 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.259 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.259 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.260 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.260 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj17 2023-10-11T16:09:22.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.261 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:WWHXk42balAFDD4A1KLvpN/Dh5U= 2023-10-11T16:09:22.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:WWHXk42balAFDD4A1KLvpN/Dh5U='} 2023-10-11T16:09:22.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000107313ff247f973b-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.265 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.266 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.266 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.266 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.267 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069db2224bbd240df-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj34 2023-10-11T16:09:22.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj34 2023-10-11T16:09:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj34 2023-10-11T16:09:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.271 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.272 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.272 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.272 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.273 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj34 2023-10-11T16:09:22.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.273 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:8A5mIETNeYQ72zeX8Jm1ASkBJFQ= 2023-10-11T16:09:22.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8A5mIETNeYQ72zeX8Jm1ASkBJFQ='} 2023-10-11T16:09:22.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d664a0757739c79-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.277 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.278 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.278 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.278 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.279 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.279 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030addc0609d960f6-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj51 2023-10-11T16:09:22.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj51 2023-10-11T16:09:22.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj51 2023-10-11T16:09:22.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.283 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.284 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.284 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.284 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.285 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj51 2023-10-11T16:09:22.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.285 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:VUAUtMZm1CDdi51Jm6yd2AnbsPM= 2023-10-11T16:09:22.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VUAUtMZm1CDdi51Jm6yd2AnbsPM='} 2023-10-11T16:09:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e57b358b620758b3-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.289 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.289 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.290 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.290 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.290 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.291 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028703e5120562330-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj68 2023-10-11T16:09:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj68 2023-10-11T16:09:22.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj68 2023-10-11T16:09:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.295 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.295 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.296 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.296 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.296 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj68 2023-10-11T16:09:22.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.297 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:25JgIO3LgRnTaYuCfPeGVE/+6oo= 2023-10-11T16:09:22.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:25JgIO3LgRnTaYuCfPeGVE/+6oo='} 2023-10-11T16:09:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005969c0a876294c84-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.301 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.302 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.302 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.302 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.303 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.303 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7a1f571556c7ecc-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj85 2023-10-11T16:09:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj85 2023-10-11T16:09:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj85 2023-10-11T16:09:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.308 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.308 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.308 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.309 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.309 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj85 2023-10-11T16:09:22.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.310 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:beK/+Uj9ynvf8gAHkq/WgfxV514= 2023-10-11T16:09:22.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:beK/+Uj9ynvf8gAHkq/WgfxV514='} 2023-10-11T16:09:22.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003070663804cce92b-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.311 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 10 2023-10-11T16:09:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.316 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.316 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.317 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.317 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.318 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023a48f95830ba992-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj19 2023-10-11T16:09:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj19 2023-10-11T16:09:22.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj19 2023-10-11T16:09:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.322 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.322 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.323 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.323 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.323 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj19 2023-10-11T16:09:22.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.324 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:hvWIJ1XF54Ic/B6JtjfllyVHu8s= 2023-10-11T16:09:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:hvWIJ1XF54Ic/B6JtjfllyVHu8s='} 2023-10-11T16:09:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd7736b4657790eb-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.327 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.328 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.328 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.328 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.329 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.329 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3596654f737c850-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj38 2023-10-11T16:09:22.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj38 2023-10-11T16:09:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj38 2023-10-11T16:09:22.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.333 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.334 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.334 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.334 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.335 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj38 2023-10-11T16:09:22.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.335 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:VdczRJrbl7TB1Py8t23PIocqmtQ= 2023-10-11T16:09:22.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:VdczRJrbl7TB1Py8t23PIocqmtQ='} 2023-10-11T16:09:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000088e8e3c6bfe5d6a-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.339 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.340 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.340 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.340 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.341 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.341 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068bfe75efae03226-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj57 2023-10-11T16:09:22.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj57 2023-10-11T16:09:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj57 2023-10-11T16:09:22.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.345 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.346 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.346 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.346 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.347 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj57 2023-10-11T16:09:22.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.347 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:J4OK1NdR2vblypj+1486IbaesT4= 2023-10-11T16:09:22.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:J4OK1NdR2vblypj+1486IbaesT4='} 2023-10-11T16:09:22.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090ae412bfdbabaaa-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.352 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.352 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.352 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.352 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.353 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014503f59ef9db480-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj76 2023-10-11T16:09:22.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj76 2023-10-11T16:09:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj76 2023-10-11T16:09:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.357 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.358 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.358 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.358 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.359 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj76 2023-10-11T16:09:22.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.359 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vUmYAW3h6+/LG+v+78O71+vL21o= 2023-10-11T16:09:22.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vUmYAW3h6+/LG+v+78O71+vL21o='} 2023-10-11T16:09:22.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003dd4fbeddc7ce2d-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.363 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.363 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.364 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.364 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.364 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.365 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007abf1c50c6bd6890-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj95 2023-10-11T16:09:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj95 2023-10-11T16:09:22.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj95 2023-10-11T16:09:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.369 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.369 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.369 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.370 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.370 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj95 2023-10-11T16:09:22.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.371 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:So76MOqMB8NJCmtK4rm7ALVs5l0= 2023-10-11T16:09:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:So76MOqMB8NJCmtK4rm7ALVs5l0='} 2023-10-11T16:09:22.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001294bc477b5757e-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 15 2023-10-11T16:09:22.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.377 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.378 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.378 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.378 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.379 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.379 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d08e04b8f593068-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj23 2023-10-11T16:09:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj23 2023-10-11T16:09:22.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj23 2023-10-11T16:09:22.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.383 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.384 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.384 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.384 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.384 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj23 2023-10-11T16:09:22.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.385 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vztRQJ5chdFczeJYeVchMG/gCqQ= 2023-10-11T16:09:22.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vztRQJ5chdFczeJYeVchMG/gCqQ='} 2023-10-11T16:09:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a6ed00dd87faacf-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.389 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.390 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.390 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.390 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.391 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.391 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a279c8934b7f6a-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj46 2023-10-11T16:09:22.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj46 2023-10-11T16:09:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj46 2023-10-11T16:09:22.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.395 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.396 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.396 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.396 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.397 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj46 2023-10-11T16:09:22.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.397 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:I47fi8iWUxnZivO3wnM7/OdAqgw= 2023-10-11T16:09:22.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I47fi8iWUxnZivO3wnM7/OdAqgw='} 2023-10-11T16:09:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a50bbbe9b68f847a-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.401 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.401 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.402 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.402 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.402 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.403 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fff960f79ab8c863-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj69 2023-10-11T16:09:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj69 2023-10-11T16:09:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj69 2023-10-11T16:09:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.407 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.407 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.407 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.408 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.408 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj69 2023-10-11T16:09:22.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.408 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:bbRZiH2psJQO2H9csiVQMccRr3s= 2023-10-11T16:09:22.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:bbRZiH2psJQO2H9csiVQMccRr3s='} 2023-10-11T16:09:22.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f89fbb719b1b1477-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.412 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.413 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.413 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.413 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.414 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.414 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea2662a7d397fd3a-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj92 2023-10-11T16:09:22.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj92 2023-10-11T16:09:22.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj92 2023-10-11T16:09:22.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.419 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.419 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.419 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.419 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.420 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj92 2023-10-11T16:09:22.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.420 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:qtX+GtlKd88MrVGeIwg0FINuGyA= 2023-10-11T16:09:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qtX+GtlKd88MrVGeIwg0FINuGyA='} 2023-10-11T16:09:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a45807860d4e747f-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.425 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.425 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.425 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.425 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.426 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.426 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo= 2023-10-11T16:09:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:E6NZs/c9/L0VS0ByLD+AaDl4xbo=', 'Content-Length': '0'} 2023-10-11T16:09:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ece60d04333717d-006526c173-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj115 2023-10-11T16:09:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj115 2023-10-11T16:09:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj115 2023-10-11T16:09:22.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.431 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.431 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.431 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.432 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:27 GMT 2023-10-11T16:09:22.432 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj115 2023-10-11T16:09:22.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.433 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:qEetUc9cN7GMsamiwkL6ri8WB/A= 2023-10-11T16:09:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:27 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:qEetUc9cN7GMsamiwkL6ri8WB/A='} 2023-10-11T16:09:22.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea2177d4ce022337-006526c173-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:27 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 20 2023-10-11T16:09:22.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.439 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.440 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.440 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.440 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.441 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.441 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a94b01c45198897e-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj29 2023-10-11T16:09:22.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj29 2023-10-11T16:09:22.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj29 2023-10-11T16:09:22.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.445 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.446 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.446 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.446 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.447 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj29 2023-10-11T16:09:22.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.447 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:I7DbNYblV8CowsFBOu1OQy9eJ04= 2023-10-11T16:09:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:I7DbNYblV8CowsFBOu1OQy9eJ04='} 2023-10-11T16:09:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f154eca03afed3-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.452 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.452 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.452 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.453 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.453 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.453 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092c6d2b6ab4eae59-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj58 2023-10-11T16:09:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj58 2023-10-11T16:09:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj58 2023-10-11T16:09:22.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.458 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.458 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.458 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.459 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.459 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj58 2023-10-11T16:09:22.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.460 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/mL3gq6TkR6BYpF76hfMkjC1Nyg= 2023-10-11T16:09:22.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/mL3gq6TkR6BYpF76hfMkjC1Nyg='} 2023-10-11T16:09:22.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c27f40055698aa5-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.464 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.464 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.464 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.465 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.465 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.465 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0bf01ea5535565d-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj87 2023-10-11T16:09:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj87 2023-10-11T16:09:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj87 2023-10-11T16:09:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.470 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.470 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.470 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.471 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.471 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj87 2023-10-11T16:09:22.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.471 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LB2vzqNkyi8AGB0ybbPO4vUVp00= 2023-10-11T16:09:22.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LB2vzqNkyi8AGB0ybbPO4vUVp00='} 2023-10-11T16:09:22.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d216aff0569ca3d-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.475 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.476 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.476 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.476 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.477 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.477 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcfe9d56ec20d5ca-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj116 2023-10-11T16:09:22.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj116 2023-10-11T16:09:22.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj116 2023-10-11T16:09:22.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.482 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.482 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.482 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.482 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.483 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj116 2023-10-11T16:09:22.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.483 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:n0EX3NuwLwwhA96sgxk+IKgz0Ac= 2023-10-11T16:09:22.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n0EX3NuwLwwhA96sgxk+IKgz0Ac='} 2023-10-11T16:09:22.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092cd856f9ab302da-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.488 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.488 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.488 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.489 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.489 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.489 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afca4652d2778e7f-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj145 2023-10-11T16:09:22.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj145 2023-10-11T16:09:22.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj145 2023-10-11T16:09:22.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.494 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.494 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.494 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.495 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.495 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj145 2023-10-11T16:09:22.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.495 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:mo+li2p2QZMOaqHpQi1TyTcx+bM= 2023-10-11T16:09:22.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:mo+li2p2QZMOaqHpQi1TyTcx+bM='} 2023-10-11T16:09:22.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e37e930a07d85a60-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 25 2023-10-11T16:09:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.502 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.503 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.503 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.503 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.503 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.504 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b6d0eb35c0dd0d2-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj31 2023-10-11T16:09:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj31 2023-10-11T16:09:22.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj31 2023-10-11T16:09:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.508 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.508 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.509 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.509 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.509 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj31 2023-10-11T16:09:22.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.510 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:tXOAnCumLi/n+Y6sm3Ul2Ghsfyk= 2023-10-11T16:09:22.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tXOAnCumLi/n+Y6sm3Ul2Ghsfyk='} 2023-10-11T16:09:22.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b03f6e4767e0190-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.514 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.514 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.514 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.515 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.515 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.516 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002425b26e3d34862b-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj62 2023-10-11T16:09:22.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj62 2023-10-11T16:09:22.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj62 2023-10-11T16:09:22.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.520 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.520 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.521 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.521 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.521 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj62 2023-10-11T16:09:22.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.522 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ji51MCrWhhD3IqXBmMyTmus1Yac= 2023-10-11T16:09:22.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ji51MCrWhhD3IqXBmMyTmus1Yac='} 2023-10-11T16:09:22.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075884ac137a8613c-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.526 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.526 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.527 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.527 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.527 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.528 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009467bff4fd3a3f96-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj93 2023-10-11T16:09:22.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj93 2023-10-11T16:09:22.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj93 2023-10-11T16:09:22.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.532 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.533 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.533 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.534 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj93 2023-10-11T16:09:22.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.534 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:8dQNKzUFtNEget2vy5FhQOS/KsA= 2023-10-11T16:09:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8dQNKzUFtNEget2vy5FhQOS/KsA='} 2023-10-11T16:09:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053697fe018eff2e7-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.539 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.539 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.540 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.540 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.540 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.541 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000918ab5480cde9fa4-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj124 2023-10-11T16:09:22.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj124 2023-10-11T16:09:22.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj124 2023-10-11T16:09:22.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.545 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.545 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.546 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.546 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj124 2023-10-11T16:09:22.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.547 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:4jb/oWx7/bG/Gt+85JCwM5qbn/4= 2023-10-11T16:09:22.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4jb/oWx7/bG/Gt+85JCwM5qbn/4='} 2023-10-11T16:09:22.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c7505711b9f1356-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.551 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.551 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.552 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.552 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.552 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.553 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050867af917ce502b-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj155 2023-10-11T16:09:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj155 2023-10-11T16:09:22.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj155 2023-10-11T16:09:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.557 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.557 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.558 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.558 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.558 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj155 2023-10-11T16:09:22.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.559 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ep/OKTojQ810olyJ7nY+mp5Me+I= 2023-10-11T16:09:22.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ep/OKTojQ810olyJ7nY+mp5Me+I='} 2023-10-11T16:09:22.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000895b49986b0ea804-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-63 2023-10-11T16:09:22.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 30 2023-10-11T16:09:22.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.566 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.566 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.567 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.567 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.568 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a97f0036ff0d22-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj37 2023-10-11T16:09:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj37 2023-10-11T16:09:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj37 2023-10-11T16:09:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.572 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.572 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.572 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.573 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.573 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj37 2023-10-11T16:09:22.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.573 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LjrbKb7oNBywgKEedf3de5N2Kcw= 2023-10-11T16:09:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LjrbKb7oNBywgKEedf3de5N2Kcw='} 2023-10-11T16:09:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022314a689b53a213-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.577 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.578 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.578 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.578 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.579 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.579 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ddefc8c3db311f-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj74 2023-10-11T16:09:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj74 2023-10-11T16:09:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj74 2023-10-11T16:09:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.584 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.584 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.585 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.585 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.585 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj74 2023-10-11T16:09:22.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.586 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:q8cPyg1X605WmH0eXiHHiQpNN0Y= 2023-10-11T16:09:22.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:q8cPyg1X605WmH0eXiHHiQpNN0Y='} 2023-10-11T16:09:22.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c13bbe7aedc1159a-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.590 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.590 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.591 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.591 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.591 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.592 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ= 2023-10-11T16:09:22.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/yrLDTrzSQxb3b6HQjPHIsKBnyQ=', 'Content-Length': '0'} 2023-10-11T16:09:22.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0fd753c3c4449aa-006526c174-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj111 2023-10-11T16:09:22.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj111 2023-10-11T16:09:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj111 2023-10-11T16:09:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.596 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.597 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.597 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.597 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:28 GMT 2023-10-11T16:09:22.597 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj111 2023-10-11T16:09:22.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.598 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/Ao7a5MA4IUnOTk/y/Q4XnncI3A= 2023-10-11T16:09:22.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:28 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/Ao7a5MA4IUnOTk/y/Q4XnncI3A='} 2023-10-11T16:09:22.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfad7bc1ce7d7197-006526c174-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:28 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.602 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.603 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.603 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.603 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:29 GMT 2023-10-11T16:09:22.604 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.604 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU= 2023-10-11T16:09:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU=', 'Content-Length': '0'} 2023-10-11T16:09:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cc5ca6207bf12ad-006526c175-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj148 2023-10-11T16:09:22.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj148 2023-10-11T16:09:22.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj148 2023-10-11T16:09:22.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.609 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.609 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.609 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.609 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:29 GMT 2023-10-11T16:09:22.610 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj148 2023-10-11T16:09:22.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.610 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:tQnQwbiH4cC0Iv8QIjB25UhSWxo= 2023-10-11T16:09:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:tQnQwbiH4cC0Iv8QIjB25UhSWxo='} 2023-10-11T16:09:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d057ec3e37516a1d-006526c175-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/ 2023-10-11T16:09:22.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/ 2023-10-11T16:09:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/ 2023-10-11T16:09:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.614 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.615 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.615 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:29 GMT 2023-10-11T16:09:22.615 INFO:tasks.rgw_multisite_tests:/dghqvd-63/ 2023-10-11T16:09:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.616 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU= 2023-10-11T16:09:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LHygO0ZJnC5wJC1MhEFxUgP2XxU=', 'Content-Length': '0'} 2023-10-11T16:09:22.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095003e3331e8b9c4-006526c175-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-63/obj185 2023-10-11T16:09:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-63/obj185 2023-10-11T16:09:22.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-63/obj185 2023-10-11T16:09:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.620 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.620 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.621 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.621 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:38:29 GMT 2023-10-11T16:09:22.621 INFO:tasks.rgw_multisite_tests:/dghqvd-63/obj185 2023-10-11T16:09:22.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.622 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:a+P7XsISuZYa6fXgAoy/sImlwfY= 2023-10-11T16:09:22.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:38:29 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a+P7XsISuZYa6fXgAoy/sImlwfY='} 2023-10-11T16:09:22.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc1e45979e605836-006526c175-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:38:29 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-63 2023-10-11T16:09:22.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:29.132+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:34.132+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:39.133+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:44.133+0000 7fdfb357e780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:49.136+0000 7fdfb357e780 1 bucket sync caught up with source: 2023-10-11T16:09:22.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, , , , , , 00000000002.97.6, , , , , , 00000000002.64.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.108.6, , , , 00000000001.72.6] 2023-10-11T16:09:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, , , , , , 00000000002.97.6, , , , , , 00000000002.64.6, , , , , 00000000002.107.6, , , , , , , , , , , , , , , 00000000001.108.6, , , , 00000000001.72.6] 2023-10-11T16:09:22.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:49.136+0000 7fdfb357e780 0 bucket checkpoint complete 2023-10-11T16:09:22.627 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dghqvd-63 2023-10-11T16:09:22.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.628 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:22.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:49.250+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:54.251+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:38:59.252+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:04.253+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:09.253+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:14.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:19.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:24.255+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:29.256+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:34.257+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:39.257+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:44.258+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:49.258+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:54.259+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:39:59.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:04.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:09.260+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:14.261+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:19.261+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:24.262+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:29.263+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:34.263+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:39.264+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:44.265+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:49.266+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:54.267+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:40:59.267+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:04.268+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:09.268+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:14.269+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:19.270+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:24.270+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:29.271+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:34.272+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:39.271+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:44.272+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:49.273+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:54.274+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:41:59.274+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:04.275+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:09.276+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:14.276+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:19.277+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:24.278+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:29.279+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:34.280+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:39.280+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:44.281+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:49.282+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:54.282+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:42:59.283+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:04.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:09.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:14.284+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:19.285+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:24.285+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:29.286+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:34.286+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:39.287+0000 7f864fab3780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.288+0000 7f864fab3780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-10-11T16:09:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.288+0000 7f864fab3780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T16:09:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.288+0000 7f864fab3780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T16:09:22.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-11T16:09:22.649 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-11T16:09:22.649 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.649 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-11T16:09:22.650 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-11T16:09:22.650 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:22.651 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-11T16:09:22.651 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-11T16:09:22.651 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-11T16:09:22.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-10-11T16:09:22.652 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-10-11T16:09:22.652 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-10-11T16:09:22.653 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-10-11T16:09:22.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-10-11T16:09:22.653 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-10-11T16:09:22.653 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite.py", line 197, in admin 2023-10-11T16:09:22.654 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-10-11T16:09:22.654 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-10-11T16:09:22.654 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-11T16:09:22.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:09:22.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.658 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.659 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.659 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.659 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.660 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.660 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:55kHjM3BYzBk9RvwPPmXoJC74/4= 2023-10-11T16:09:22.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:55kHjM3BYzBk9RvwPPmXoJC74/4=', 'Content-Length': '0'} 2023-10-11T16:09:22.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a961adc8f22cd208-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dghqvd-64 2023-10-11T16:09:22.661 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.662 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:22.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:22.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039024.336393_729.1", 2023-10-11T16:09:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:43:44.336393Z" 2023-10-11T16:09:22.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T16:09:22.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:22.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:22.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.529+0000 7f969b7ff780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.530+0000 7f969b7ff780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:22.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:22.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:22.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:22.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:22.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:22.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:22.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:22.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:22.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:22.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039024.336393_729.1", 2023-10-11T16:09:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:22.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:43:44.336393Z", 2023-10-11T16:09:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:22.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:22.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:22.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:43:44Z" 2023-10-11T16:09:22.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:22.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:22.691 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:22.691 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:22.691 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:22.691 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:22.692 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:22.692 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.692 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.693 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:22.693 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.693 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:22.693 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.694 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:22.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.695 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:22.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:22.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.696 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.696 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.696 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:22.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.697 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:22.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.698 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:22.699 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.699 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.699 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.700 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.700 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:22.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.700 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.701 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:22.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.701 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:22.702 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.702 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.703 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.703 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:22.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:22.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:22.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:22.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039024.336393_729.1", 2023-10-11T16:09:22.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:22.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:22.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:22.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:43:44.336393Z", 2023-10-11T16:09:22.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:22.706 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.706 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:22.706 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:22.707 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.707 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:22.707 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:22.708 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:22.708 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:22.709 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:43:44Z" 2023-10-11T16:09:22.709 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:22.709 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039024.336393_729.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:43:44.336393Z', 'realm_epoch': 2}}] 2023-10-11T16:09:22.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T16:09:22.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039024.336393_729.1'} 2023-10-11T16:09:22.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.714 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.715 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.715 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.715 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.716 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.716 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T16:09:22.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T16:09:22.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a681a4ea78e06344-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11 2023-10-11T16:09:22.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.724 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.724 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.725 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.726 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11 2023-10-11T16:09:22.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.727 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ol4YF+Z5QFZsj6b0Dwvoo/Tz/2A= 2023-10-11T16:09:22.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ol4YF+Z5QFZsj6b0Dwvoo/Tz/2A='} 2023-10-11T16:09:22.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000273591553edb2d15-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.736 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.736 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.737 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.738 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.738 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.740 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T16:09:22.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T16:09:22.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000761f6cddb9ccd394-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:22.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22 2023-10-11T16:09:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.748 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.749 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.750 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.750 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22 2023-10-11T16:09:22.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.752 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:KZ3Ny9USaQuo4rmHN8YXVbdM5ec= 2023-10-11T16:09:22.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KZ3Ny9USaQuo4rmHN8YXVbdM5ec='} 2023-10-11T16:09:22.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009430e88add24bcba-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.760 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.760 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.761 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.761 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.762 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.763 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T16:09:22.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T16:09:22.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd5e6043c88e1336-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:22.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:22.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33 2023-10-11T16:09:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.770 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.771 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.771 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.772 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.773 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33 2023-10-11T16:09:22.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.773 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:JEl6kuCtYlNBj81mH7TQBIUwymU= 2023-10-11T16:09:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:JEl6kuCtYlNBj81mH7TQBIUwymU='} 2023-10-11T16:09:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf3985b3560ee56f-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.780 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.781 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.782 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.782 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.783 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.784 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T16:09:22.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T16:09:22.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072d154304e98e589-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:22.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:22.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44 2023-10-11T16:09:22.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.790 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.791 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.791 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.791 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.792 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44 2023-10-11T16:09:22.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.792 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:SaZwDoKWjmJD85iCHGLxKhJkFBY= 2023-10-11T16:09:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SaZwDoKWjmJD85iCHGLxKhJkFBY='} 2023-10-11T16:09:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044cfa94aa7a95c36-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.797 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.797 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.797 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.798 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.798 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.798 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4= 2023-10-11T16:09:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:lUr6pzf35c3bhqTPbkL77skzLE4=', 'Content-Length': '0'} 2023-10-11T16:09:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a388212bee61182-006526c2b0-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55 2023-10-11T16:09:22.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.803 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:22.803 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:22.803 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:22.803 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:44 GMT 2023-10-11T16:09:22.804 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55 2023-10-11T16:09:22.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.804 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:h4xRouq9jpYZSGoheUwtMSIPtB8= 2023-10-11T16:09:22.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:43:44 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:h4xRouq9jpYZSGoheUwtMSIPtB8='} 2023-10-11T16:09:22.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6738e0bdf19fbce-006526c2b0-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-64 2023-10-11T16:09:22.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:22.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:44.780+0000 7f6062859780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:49.782+0000 7f6062859780 1 bucket sync caught up with source: 2023-10-11T16:09:22.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, , , , , , , , 00000000001.111.6, 00000000002.102.6, 00000000002.118.6] 2023-10-11T16:09:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, , , , , , , , 00000000001.111.6, 00000000002.102.6, 00000000002.118.6] 2023-10-11T16:09:22.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:49.782+0000 7f6062859780 0 bucket checkpoint complete 2023-10-11T16:09:22.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dghqvd-64 2023-10-11T16:09:22.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:22.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:49.901+0000 7f57bca48780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:54.901+0000 7f57bca48780 1 waiting to reach incremental sync.. 2023-10-11T16:09:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:59.903+0000 7f57bca48780 1 bucket sync caught up with source: 2023-10-11T16:09:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: local status: [, , , , , , , , 00000000001.318.6, 00000000002.199.6, 00000000002.118.6] 2023-10-11T16:09:22.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: remote markers: [, , , , , , , , 00000000001.318.6, 00000000002.199.6, 00000000002.118.6] 2023-10-11T16:09:22.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:43:59.903+0000 7f57bca48780 0 bucket checkpoint complete 2023-10-11T16:09:22.812 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dghqvd-64 zones={a2, a1} 2023-10-11T16:09:22.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:22.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.816 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.817 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.817 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.817 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.818 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8= 2023-10-11T16:09:22.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8=', 'Content-Length': '0'} 2023-10-11T16:09:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e4b0abd88306396-006526c2bf-1fbc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.822 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.822 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.823 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.823 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.823 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.824 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8= 2023-10-11T16:09:22.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:LSZNSr6GNGVaihgVjeoN9s/Hru8=', 'Content-Length': '0'} 2023-10-11T16:09:22.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e66f49af8ee3d9bb-006526c2bf-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.825 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-10-11T16:09:22.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.829 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.829 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.830 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.830 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.830 INFO:tasks.rgw_multisite_tests:/dghqvd-64/?versions 2023-10-11T16:09:22.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.831 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T16:09:22.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T16:09:22.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c943f737f65372-006526c2bf-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.832 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-11T16:09:22.832 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-11T16:09:22.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-11T16:09:22.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-11T16:09:22.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-11T16:09:22.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-10-11T16:09:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.837 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.838 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.838 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.838 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.839 INFO:tasks.rgw_multisite_tests:/dghqvd-64/?versions 2023-10-11T16:09:22.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.839 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T16:09:22.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T16:09:22.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d205eef3bf1c852-006526c2bf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.840 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-10-11T16:09:22.841 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-10-11T16:09:22.841 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-10-11T16:09:22.841 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-10-11T16:09:22.842 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-10-11T16:09:22.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.846 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.846 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.847 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.847 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.847 INFO:tasks.rgw_multisite_tests:/dghqvd-64/?versions 2023-10-11T16:09:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.848 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T16:09:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T16:09:22.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa96647f64ea1e87-006526c2bf-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?versions&max-keys=999 2023-10-11T16:09:22.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.853 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.853 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.854 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.854 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.854 INFO:tasks.rgw_multisite_tests:/dghqvd-64/?versions 2023-10-11T16:09:22.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.855 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y= 2023-10-11T16:09:22.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nfJn8IFABpDoalsulX762iQtC1Y=', 'Content-Length': '0'} 2023-10-11T16:09:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000837bdaf262d9725c-006526c2bf-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:43:59 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64999falseobj11nulltrue2023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-11T16:09:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.861 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.861 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.861 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.861 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:43:59 GMT 2023-10-11T16:09:22.862 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.862 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:4Ekld866laImbjdWfLjt8OlUybM= 2023-10-11T16:09:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:43:59 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:4Ekld866laImbjdWfLjt8OlUybM=', 'Content-Length': '0'} 2023-10-11T16:09:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004e109e4911d1034-006526c2bf-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.867 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.868 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.868 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.868 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.869 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.869 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T16:09:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T16:09:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a8718138c60b60f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.874 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.875 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.875 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.875 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.876 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.876 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw= 2023-10-11T16:09:22.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw=', 'Content-Length': '0'} 2023-10-11T16:09:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000198660d72466a0a-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.881 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.882 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.882 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.882 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.883 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.883 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw= 2023-10-11T16:09:22.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:TYy5ciU4nMSVj/vQpSvu6BuEBUw=', 'Content-Length': '0'} 2023-10-11T16:09:22.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee28d6fe3dd5e5d8-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.884 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-11T16:09:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.888 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.889 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.889 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.889 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.890 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.890 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T16:09:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T16:09:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851a717181b43361-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.895 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.896 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.896 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.896 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.897 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11?versionId=null 2023-10-11T16:09:22.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.897 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc= 2023-10-11T16:09:22.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:cIlMEk0FEgtxtS1Lskhgbn4Zttc=', 'Content-Length': '0'} 2023-10-11T16:09:22.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000438e7d2d713f81e5-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.902 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.903 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.903 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.903 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.904 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.904 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f2c576ab3c8210-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.910 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.910 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.910 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.910 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.911 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.911 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:22.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:22.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc1e27914efab94d-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11 2023-10-11T16:09:22.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.916 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.916 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.916 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.917 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.917 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11 2023-10-11T16:09:22.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.918 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc= 2023-10-11T16:09:22.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc=', 'Content-Length': '0'} 2023-10-11T16:09:22.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087a1173d2377454b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11 2023-10-11T16:09:22.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.922 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.923 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.924 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11 2023-10-11T16:09:22.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.924 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc= 2023-10-11T16:09:22.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FrlvPhZ4DkdXYrKZWp1T3Eb0FCc=', 'Content-Length': '0'} 2023-10-11T16:09:22.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fad7491aacc3369-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-10-11T16:09:22.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11 2023-10-11T16:09:22.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.929 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.929 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.930 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.930 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.930 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11 2023-10-11T16:09:22.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.931 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8= 2023-10-11T16:09:22.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8=', 'Content-Length': '0'} 2023-10-11T16:09:22.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53123cf258aa197-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj11 2023-10-11T16:09:22.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj11 2023-10-11T16:09:22.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj11 2023-10-11T16:09:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.935 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.935 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.935 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.936 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.936 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj11 2023-10-11T16:09:22.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.937 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8= 2023-10-11T16:09:22.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:n/yQmmnwHlWQTWV0n5c2NfNCa+8=', 'Content-Length': '0'} 2023-10-11T16:09:22.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e023432cd59902c-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.942 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.942 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.942 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.942 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.943 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.943 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8f887088500722-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj11 2023-10-11T16:09:22.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.950 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.950 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.951 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.952 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.952 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:22.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.953 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:22.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:22.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000320d1c132d2a9a5-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj111000falseobj112023-10-11T15:43:44.576Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:22.955 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-11T16:09:22.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:22.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:22.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.964 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.964 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.965 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.966 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.966 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.968 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T16:09:22.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T16:09:22.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f512fe5c0faa681c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:22.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:22.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:22.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:22.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.977 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:22.978 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.979 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.979 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.980 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.981 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T16:09:22.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T16:09:22.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c4f58e769ab1d8-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:22.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:22.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:22.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:22.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:22.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:22.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:22.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:22.991 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:22.991 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.992 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:22.993 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:22.993 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:22.995 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM= 2023-10-11T16:09:22.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM=', 'Content-Length': '0'} 2023-10-11T16:09:22.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e149957ec6c8ab49-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:22.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:22.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:22.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:22.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.003 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.004 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.004 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.005 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.005 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.007 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM= 2023-10-11T16:09:23.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xN5y2ic0UAkrYlOo5Ajdrmf69tM=', 'Content-Length': '0'} 2023-10-11T16:09:23.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098318bf4d8f33b79-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.009 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-11T16:09:23.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.014 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.015 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.015 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.015 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.016 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.016 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T16:09:23.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T16:09:23.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4a4717efdda9db-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.021 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.021 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.022 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.022 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.022 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22?versionId=null 2023-10-11T16:09:23.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.023 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4= 2023-10-11T16:09:23.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0Z9gND/W+EyIAKYUPyBriYRsAh4=', 'Content-Length': '0'} 2023-10-11T16:09:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098e034a66bc2d3fb-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.028 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.028 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.029 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.029 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.029 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.030 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e30a722a79747ace-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.035 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.035 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.036 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.036 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.036 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.037 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002101cf78f7b1477b-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22 2023-10-11T16:09:23.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.042 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.042 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.042 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.043 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.043 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22 2023-10-11T16:09:23.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.043 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA= 2023-10-11T16:09:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA=', 'Content-Length': '0'} 2023-10-11T16:09:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2790ede29d211b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22 2023-10-11T16:09:23.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.047 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.048 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.048 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.048 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.049 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22 2023-10-11T16:09:23.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.049 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA= 2023-10-11T16:09:23.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:uRBjroIr0Ylep7F+ZuI/yv+t0pA=', 'Content-Length': '0'} 2023-10-11T16:09:23.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006074bb9a9dbff557-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.050 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-10-11T16:09:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22 2023-10-11T16:09:23.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.054 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.054 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.054 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.055 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.055 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22 2023-10-11T16:09:23.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.056 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4= 2023-10-11T16:09:23.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4=', 'Content-Length': '0'} 2023-10-11T16:09:23.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0cc94804b4b725-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj22 2023-10-11T16:09:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj22 2023-10-11T16:09:23.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj22 2023-10-11T16:09:23.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.060 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.060 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.060 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.061 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.061 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj22 2023-10-11T16:09:23.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.062 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4= 2023-10-11T16:09:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ih5LPPlPAczvx67wxFRRoS0SrZ4=', 'Content-Length': '0'} 2023-10-11T16:09:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4c3221e9b148f3-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.066 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.067 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.067 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.067 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.068 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.068 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be27341306fd96ba-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj22 2023-10-11T16:09:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.074 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.074 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.074 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.075 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.075 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.076 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d4c59906a018e5-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj221000falseobj222023-10-11T15:43:44.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.077 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-11T16:09:23.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.081 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.081 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.082 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.082 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.082 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.083 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T16:09:23.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T16:09:23.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fda38c380222dfc-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.088 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.088 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.089 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.089 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.089 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.090 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T16:09:23.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T16:09:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd82730528a7af0-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.095 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.095 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.095 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.096 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.096 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.097 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM= 2023-10-11T16:09:23.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM=', 'Content-Length': '0'} 2023-10-11T16:09:23.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644c8beb29bb620d-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.102 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.102 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.102 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.103 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.103 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.103 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM= 2023-10-11T16:09:23.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:NMaQrYYH+vnASG16iJOEz5nPWwM=', 'Content-Length': '0'} 2023-10-11T16:09:23.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6891f602eb4dff1-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.104 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-11T16:09:23.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.108 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.109 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.109 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.109 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.110 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.110 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T16:09:23.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T16:09:23.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674915dec152e000-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.115 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.116 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.116 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.116 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.116 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33?versionId=null 2023-10-11T16:09:23.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.117 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA= 2023-10-11T16:09:23.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:og4T/S24sUNl+/NB4Oye8eVzOfA=', 'Content-Length': '0'} 2023-10-11T16:09:23.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82a52bb6494a6cb-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.122 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.122 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.123 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.123 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.123 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.124 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043e4925f16fa38b2-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.129 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.129 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.130 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.130 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.130 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.131 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586ac7b6b311c130-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33 2023-10-11T16:09:23.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.135 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.136 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.136 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.136 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.136 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33 2023-10-11T16:09:23.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.137 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw= 2023-10-11T16:09:23.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw=', 'Content-Length': '0'} 2023-10-11T16:09:23.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb4c3a1585c58e34-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33 2023-10-11T16:09:23.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.141 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.141 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.142 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.142 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.142 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33 2023-10-11T16:09:23.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.143 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw= 2023-10-11T16:09:23.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PTj66FUwYA+hdNzxdI/B4fAlYcw=', 'Content-Length': '0'} 2023-10-11T16:09:23.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd4f1605714f74d-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.144 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-10-11T16:09:23.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33 2023-10-11T16:09:23.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.148 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.148 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.148 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.149 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.149 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33 2023-10-11T16:09:23.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.149 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE= 2023-10-11T16:09:23.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE=', 'Content-Length': '0'} 2023-10-11T16:09:23.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002607db9b03a6761c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj33 2023-10-11T16:09:23.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj33 2023-10-11T16:09:23.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj33 2023-10-11T16:09:23.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.154 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.154 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.154 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.155 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.155 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj33 2023-10-11T16:09:23.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.156 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE= 2023-10-11T16:09:23.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ZQtbfcvIDr3ITh5pR7LWLnqAzqE=', 'Content-Length': '0'} 2023-10-11T16:09:23.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090519a4e798f25c4-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.160 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.161 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.161 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.161 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.162 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.162 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73be8447f2ff09f-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj33 2023-10-11T16:09:23.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.168 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.168 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.168 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.168 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.169 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.169 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbef885706054fd2-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj331000falseobj332023-10-11T15:43:44.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.171 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-11T16:09:23.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.175 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.175 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.175 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.176 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.176 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.176 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T16:09:23.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T16:09:23.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007692888ec23e6834-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.181 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.182 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.182 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.182 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.183 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.183 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T16:09:23.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T16:09:23.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f882ace4b467a5f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.188 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.189 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.189 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.189 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.190 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.190 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM= 2023-10-11T16:09:23.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM=', 'Content-Length': '0'} 2023-10-11T16:09:23.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047413763f8337e97-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.195 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.195 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.196 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.196 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.196 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.197 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM= 2023-10-11T16:09:23.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jpkzBYHi4t5cUUq25YnHJ783CKM=', 'Content-Length': '0'} 2023-10-11T16:09:23.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ab39dfc21f7459-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.198 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-11T16:09:23.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.202 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.203 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.203 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.203 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.204 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.204 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T16:09:23.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T16:09:23.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a41f38c56b3b5e2-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.209 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.209 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.210 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.210 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.210 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44?versionId=null 2023-10-11T16:09:23.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.211 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM= 2023-10-11T16:09:23.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y9nXIjq2VYYyhE6/z3SHdhx6EXM=', 'Content-Length': '0'} 2023-10-11T16:09:23.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478b0c7c55965fd2-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.216 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.216 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.216 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.217 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.217 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.218 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d68a265eb5a797b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.223 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.223 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.223 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.224 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.224 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.225 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbac6441772653b8-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44 2023-10-11T16:09:23.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.229 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.230 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.230 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.230 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.230 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44 2023-10-11T16:09:23.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.231 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c= 2023-10-11T16:09:23.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c=', 'Content-Length': '0'} 2023-10-11T16:09:23.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb42e7113dbefa4-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44 2023-10-11T16:09:23.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.235 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.236 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.236 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.236 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.237 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44 2023-10-11T16:09:23.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.237 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c= 2023-10-11T16:09:23.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Ad+HsZqyRv1fE8XuzruMKi62H/c=', 'Content-Length': '0'} 2023-10-11T16:09:23.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae9029923af47a72-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.238 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-10-11T16:09:23.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44 2023-10-11T16:09:23.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.242 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.242 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.243 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.243 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.243 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44 2023-10-11T16:09:23.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.244 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0= 2023-10-11T16:09:23.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0=', 'Content-Length': '0'} 2023-10-11T16:09:23.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000372bfb5ff45985a3-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj44 2023-10-11T16:09:23.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj44 2023-10-11T16:09:23.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj44 2023-10-11T16:09:23.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.248 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.248 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.249 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.249 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.249 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj44 2023-10-11T16:09:23.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.250 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0= 2023-10-11T16:09:23.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:vJKkadS3W9Y47mOVIBQu3ghL6x0=', 'Content-Length': '0'} 2023-10-11T16:09:23.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae546aed97213128-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.255 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.255 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.256 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.256 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.256 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.257 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047692bbf75caf3b3-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj44 2023-10-11T16:09:23.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.262 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.262 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.262 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.263 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.263 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.264 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc4e7913d6d8677-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj441000falseobj442023-10-11T15:43:44.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.265 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-11T16:09:23.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.269 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.269 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.270 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.270 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.270 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.271 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T16:09:23.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T16:09:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0cfdf3ebb69a29-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.276 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.276 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.276 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.277 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.277 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.278 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T16:09:23.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T16:09:23.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c18207e979bd72f-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.283 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.283 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.283 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.283 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.284 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.284 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE= 2023-10-11T16:09:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE=', 'Content-Length': '0'} 2023-10-11T16:09:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178f2ae69256b3f1-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.289 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.289 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.290 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.290 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.290 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.291 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE= 2023-10-11T16:09:23.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6Tac1/Kv7qy3lAqRL9eKedrzzGE=', 'Content-Length': '0'} 2023-10-11T16:09:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66cbec0c10f63a2-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-11T16:09:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.296 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.296 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.297 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.297 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.297 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.298 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T16:09:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T16:09:23.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66579cb26ab1f9b-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.303 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.303 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.303 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.304 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.304 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55?versionId=null 2023-10-11T16:09:23.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.305 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4= 2023-10-11T16:09:23.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Pk2HfhdOIH4vrRAf+Ed8mcFnee4=', 'Content-Length': '0'} 2023-10-11T16:09:23.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f44bbabb5e7a571-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.310 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.310 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.310 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.310 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.311 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.311 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011df784e8529462c-006526c2c0-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.316 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.317 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.317 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.318 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.318 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU= 2023-10-11T16:09:23.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:comJIF9vv6Hf8JPTnHv2VXREAbU=', 'Content-Length': '0'} 2023-10-11T16:09:23.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ab6674b8f96e1f7-006526c2c0-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55 2023-10-11T16:09:23.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.325 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.325 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.326 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.326 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.326 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55 2023-10-11T16:09:23.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.327 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ= 2023-10-11T16:09:23.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ=', 'Content-Length': '0'} 2023-10-11T16:09:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d419868adf3d6f3a-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55 2023-10-11T16:09:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.331 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.331 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.332 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.332 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.332 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55 2023-10-11T16:09:23.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.333 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ= 2023-10-11T16:09:23.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:EYd3Gws/tWGUDCA8WTxVUwBoNeQ=', 'Content-Length': '0'} 2023-10-11T16:09:23.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef728494d0f659d1-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.334 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-10-11T16:09:23.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55 2023-10-11T16:09:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.337 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.337 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.338 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.338 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.338 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55 2023-10-11T16:09:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.339 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o= 2023-10-11T16:09:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o=', 'Content-Length': '0'} 2023-10-11T16:09:23.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-64:daa316e9-869c-4074-a756-a8adfb4a3573.5751.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f71bd7fbad963273-006526c2c0-1fbc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj55 2023-10-11T16:09:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj55 2023-10-11T16:09:23.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj55 2023-10-11T16:09:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.343 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.343 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.344 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.344 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:00 GMT 2023-10-11T16:09:23.344 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj55 2023-10-11T16:09:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.345 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o= 2023-10-11T16:09:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:00 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:x/hf0+bEZNmkqsJeevwRtXisx8o=', 'Content-Length': '0'} 2023-10-11T16:09:23.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:43:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3b229035ca5c17-006526c2c0-1675-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:44:00 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.350 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.350 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.350 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.351 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:01 GMT 2023-10-11T16:09:23.351 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.352 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0= 2023-10-11T16:09:23.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0=', 'Content-Length': '0'} 2023-10-11T16:09:23.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8e7b8c6a68c5b71-006526c2c1-1fbc-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:23.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/?prefix=obj55 2023-10-11T16:09:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.357 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:23.357 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.357 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.358 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:01 GMT 2023-10-11T16:09:23.358 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.359 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0= 2023-10-11T16:09:23.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:01 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:zcW8L2sJMX1+8lT+6cSlzL6ctN0=', 'Content-Length': '0'} 2023-10-11T16:09:23.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d20cd5794907780-006526c2c1-1675-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:44:01 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dghqvd-64obj551000falseobj552023-10-11T15:43:44.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-10-11T16:09:23.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dghqvd-64 zones={a2, a1} 2023-10-11T16:09:23.360 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-10-11T16:09:23.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-10-11T16:09:23.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:44:01.039+0000 7fa8713ba700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 177660) UID: 0 2023-10-11T16:09:23.361 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:44:01.039+0000 7fa8756237c0 -1 shutting down 2023-10-11T16:09:23.361 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T16:09:23.362 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-10-11T16:09:23.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.362 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 5 2023-10-11T16:09:23.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.368 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.368 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.369 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.369 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.370 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017b84873285865de-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj17 2023-10-11T16:09:23.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj17 2023-10-11T16:09:23.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj17 2023-10-11T16:09:23.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.374 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.374 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.375 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.375 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.375 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj17 2023-10-11T16:09:23.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.376 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xzcHUgxn9YdjxsxVPlx4271Amc8= 2023-10-11T16:09:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xzcHUgxn9YdjxsxVPlx4271Amc8='} 2023-10-11T16:09:23.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f43ca72fa332906-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.380 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.380 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.380 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.381 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.381 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.381 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d364f39943d96032-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj34 2023-10-11T16:09:23.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj34 2023-10-11T16:09:23.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj34 2023-10-11T16:09:23.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.386 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.386 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.386 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.387 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.387 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj34 2023-10-11T16:09:23.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.388 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0/I7ZaT18bLnkdRXl19NQkxJkgI= 2023-10-11T16:09:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0/I7ZaT18bLnkdRXl19NQkxJkgI='} 2023-10-11T16:09:23.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb9e3483309e4cbe-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.392 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.392 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.393 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.393 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.393 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.394 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d930c82ebe958e9-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj51 2023-10-11T16:09:23.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj51 2023-10-11T16:09:23.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj51 2023-10-11T16:09:23.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.398 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.398 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.399 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.399 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.399 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj51 2023-10-11T16:09:23.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.400 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UTccSc6+L5BfbQHyi8w7Cxkjr/U= 2023-10-11T16:09:23.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UTccSc6+L5BfbQHyi8w7Cxkjr/U='} 2023-10-11T16:09:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba6cdfbf99fab9ea-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.404 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.404 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.405 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.405 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.405 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.406 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bce69b7d1c3cf9bb-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj68 2023-10-11T16:09:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj68 2023-10-11T16:09:23.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj68 2023-10-11T16:09:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.410 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.411 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.411 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.411 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.411 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj68 2023-10-11T16:09:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.412 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/aT/IY0IA2IQqUYg50GTK/rGTFs= 2023-10-11T16:09:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/aT/IY0IA2IQqUYg50GTK/rGTFs='} 2023-10-11T16:09:23.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035fd752bce3a1d65-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.416 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.416 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.417 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.417 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.417 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.418 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000847bac2994db53ee-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj85 2023-10-11T16:09:23.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj85 2023-10-11T16:09:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj85 2023-10-11T16:09:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.422 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.423 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.423 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.423 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.424 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj85 2023-10-11T16:09:23.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.424 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:C4wqv8z2XqNweWzNH4ugGDGA5zc= 2023-10-11T16:09:23.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:C4wqv8z2XqNweWzNH4ugGDGA5zc='} 2023-10-11T16:09:23.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e32246fcb23f759a-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.426 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-10-11T16:09:23.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 10 2023-10-11T16:09:23.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.431 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.432 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.432 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.432 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.432 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.433 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcb23927bec7d2e6-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj19 2023-10-11T16:09:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj19 2023-10-11T16:09:23.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj19 2023-10-11T16:09:23.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.437 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.437 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.438 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.438 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.438 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj19 2023-10-11T16:09:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.439 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:xDY2irvpVjBpPJ/59QgB1wdaXNw= 2023-10-11T16:09:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:xDY2irvpVjBpPJ/59QgB1wdaXNw='} 2023-10-11T16:09:23.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf71e23bbdb33cf8-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.443 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.443 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.443 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.444 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.444 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.445 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a023bfdd95867ffa-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj38 2023-10-11T16:09:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj38 2023-10-11T16:09:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj38 2023-10-11T16:09:23.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.449 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.449 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.449 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.450 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.450 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj38 2023-10-11T16:09:23.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.451 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BiOSnukwVsxoOClFY031cIRxX/U= 2023-10-11T16:09:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BiOSnukwVsxoOClFY031cIRxX/U='} 2023-10-11T16:09:23.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad72778d364674db-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.455 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.455 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.455 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.455 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.456 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.456 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082c9fad529bfd244-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj57 2023-10-11T16:09:23.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj57 2023-10-11T16:09:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj57 2023-10-11T16:09:23.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.461 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.461 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.461 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.461 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.462 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj57 2023-10-11T16:09:23.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.462 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:SfpVt1qs+Qby8DJj77kBAi8GN5g= 2023-10-11T16:09:23.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:SfpVt1qs+Qby8DJj77kBAi8GN5g='} 2023-10-11T16:09:23.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6651f45f83fab1b-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.466 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.467 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.467 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.467 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.468 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.468 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000314e22a13c522f3-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj76 2023-10-11T16:09:23.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj76 2023-10-11T16:09:23.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj76 2023-10-11T16:09:23.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.472 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.473 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.473 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.473 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.473 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj76 2023-10-11T16:09:23.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.474 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:L8NPBDo51NLZZgT3+ETDgaEWX60= 2023-10-11T16:09:23.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:L8NPBDo51NLZZgT3+ETDgaEWX60='} 2023-10-11T16:09:23.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8f9222554808ecc-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.478 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.479 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.479 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.479 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.480 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.480 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004695ccec0f2b4f37-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj95 2023-10-11T16:09:23.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj95 2023-10-11T16:09:23.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj95 2023-10-11T16:09:23.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.484 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.484 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.485 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.485 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.485 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj95 2023-10-11T16:09:23.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.486 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:wJklscFLXwOh+1oltP6D2Mx96YE= 2023-10-11T16:09:23.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wJklscFLXwOh+1oltP6D2Mx96YE='} 2023-10-11T16:09:23.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0150af8aab51722-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 15 2023-10-11T16:09:23.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.492 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.492 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.493 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.493 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.493 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.494 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM= 2023-10-11T16:09:23.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:PnWOTg/LVfRYYQ9SHFm4L9TJPqM=', 'Content-Length': '0'} 2023-10-11T16:09:23.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adbd91bfba63df41-006526c2c7-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj23 2023-10-11T16:09:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj23 2023-10-11T16:09:23.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj23 2023-10-11T16:09:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.498 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.498 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.498 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.499 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:07 GMT 2023-10-11T16:09:23.499 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj23 2023-10-11T16:09:23.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.499 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jUbqLGqASHIF0pWZyooI9hylkVY= 2023-10-11T16:09:23.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jUbqLGqASHIF0pWZyooI9hylkVY='} 2023-10-11T16:09:23.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f613fa9f5b21fa1-006526c2c7-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.504 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.504 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.504 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.504 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.505 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.505 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021a5f09d24188a31-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj46 2023-10-11T16:09:23.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj46 2023-10-11T16:09:23.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj46 2023-10-11T16:09:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.510 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.510 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.510 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.510 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.511 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj46 2023-10-11T16:09:23.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.511 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ks9DRLNx/ZC8LKu93PYaHt7mYUc= 2023-10-11T16:09:23.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ks9DRLNx/ZC8LKu93PYaHt7mYUc='} 2023-10-11T16:09:23.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007997910d350b9b5e-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.515 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.516 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.516 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.516 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.516 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.517 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003501574f8211a525-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj69 2023-10-11T16:09:23.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj69 2023-10-11T16:09:23.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj69 2023-10-11T16:09:23.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.521 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.522 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.522 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.522 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.522 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj69 2023-10-11T16:09:23.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.523 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Rkyq4POxmIeR9ewPiAD/bmZGjDI= 2023-10-11T16:09:23.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Rkyq4POxmIeR9ewPiAD/bmZGjDI='} 2023-10-11T16:09:23.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023f19cf187279a79-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.528 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.528 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.528 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.528 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.529 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.529 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8e794c21413a4a4-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj92 2023-10-11T16:09:23.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj92 2023-10-11T16:09:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj92 2023-10-11T16:09:23.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.533 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.533 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.534 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.534 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.534 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj92 2023-10-11T16:09:23.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.535 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:sj/F5CrY4WFQjwS42isQxV9Lqxc= 2023-10-11T16:09:23.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:sj/F5CrY4WFQjwS42isQxV9Lqxc='} 2023-10-11T16:09:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028fffae5d762a430-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.539 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.539 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.539 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.540 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.540 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.541 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f85c5a070cc76be9-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj115 2023-10-11T16:09:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj115 2023-10-11T16:09:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj115 2023-10-11T16:09:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.545 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.545 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.545 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.546 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj115 2023-10-11T16:09:23.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.546 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:0RQ94yxt/NVAJhUPz17J0tjob/Y= 2023-10-11T16:09:23.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:0RQ94yxt/NVAJhUPz17J0tjob/Y='} 2023-10-11T16:09:23.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069d986a168c74400-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.547 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 20 2023-10-11T16:09:23.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.552 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.553 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.554 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.554 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7434fe44b5a8341-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj29 2023-10-11T16:09:23.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj29 2023-10-11T16:09:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj29 2023-10-11T16:09:23.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.558 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.559 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.559 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.559 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.559 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj29 2023-10-11T16:09:23.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.560 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Fd88ITtd5i26KvcmHdGc/uUsIwU= 2023-10-11T16:09:23.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Fd88ITtd5i26KvcmHdGc/uUsIwU='} 2023-10-11T16:09:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001be3a08a0af36425-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.564 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.564 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.565 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.565 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.565 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.566 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de765108d9c50d5-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj58 2023-10-11T16:09:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj58 2023-10-11T16:09:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj58 2023-10-11T16:09:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.570 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.570 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.571 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.571 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.571 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj58 2023-10-11T16:09:23.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.572 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9q7MUbxxTuVjeQkvLR5v3odc/wk= 2023-10-11T16:09:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9q7MUbxxTuVjeQkvLR5v3odc/wk='} 2023-10-11T16:09:23.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033f3ba0abca820c4-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.576 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.576 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.576 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.577 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.577 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.578 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008be039eb2c7659f2-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj87 2023-10-11T16:09:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj87 2023-10-11T16:09:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj87 2023-10-11T16:09:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.581 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.582 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.582 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.582 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.583 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj87 2023-10-11T16:09:23.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.583 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:KBvwYDkRmYXM76VoJq59VY5SKvs= 2023-10-11T16:09:23.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:KBvwYDkRmYXM76VoJq59VY5SKvs='} 2023-10-11T16:09:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000332dcbeeba98f7e1-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.587 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.588 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.588 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.588 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.588 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.589 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b301f4813eb1b6bd-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj116 2023-10-11T16:09:23.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj116 2023-10-11T16:09:23.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj116 2023-10-11T16:09:23.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.593 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.594 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.594 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.594 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.595 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj116 2023-10-11T16:09:23.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.595 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Y3ew7+SM57YQ8FB/vRVLCW2NcfM= 2023-10-11T16:09:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Y3ew7+SM57YQ8FB/vRVLCW2NcfM='} 2023-10-11T16:09:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013a25839401342e9-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.599 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.600 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.600 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.600 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.600 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.601 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f66c436421cb75ce-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj145 2023-10-11T16:09:23.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj145 2023-10-11T16:09:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj145 2023-10-11T16:09:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.605 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.606 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.606 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.606 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.606 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj145 2023-10-11T16:09:23.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.607 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:3/orUfb6W8J5qutfSmxPLfWM6Nk= 2023-10-11T16:09:23.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:3/orUfb6W8J5qutfSmxPLfWM6Nk='} 2023-10-11T16:09:23.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020a540907fcf400d-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 25 2023-10-11T16:09:23.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.613 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.614 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.614 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.615 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.615 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000988e9c5f1340387c-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj31 2023-10-11T16:09:23.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj31 2023-10-11T16:09:23.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj31 2023-10-11T16:09:23.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.619 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.619 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.620 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.620 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.620 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj31 2023-10-11T16:09:23.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.621 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:FQZh3MA03NdPY7ZNFpX3069JerQ= 2023-10-11T16:09:23.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:FQZh3MA03NdPY7ZNFpX3069JerQ='} 2023-10-11T16:09:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa2bb7852f902fd6-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.625 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.625 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.626 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.626 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.626 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.627 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee4e963c199be7eb-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj62 2023-10-11T16:09:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj62 2023-10-11T16:09:23.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj62 2023-10-11T16:09:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.631 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.631 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.631 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.632 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.632 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj62 2023-10-11T16:09:23.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.633 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2uHshzFVmfNcFv/N0B3Y4pfSEjo= 2023-10-11T16:09:23.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2uHshzFVmfNcFv/N0B3Y4pfSEjo='} 2023-10-11T16:09:23.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005aa0aa2ce9879b90-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.637 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.637 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.637 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.637 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.638 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.638 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51be8e2424051ff-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj93 2023-10-11T16:09:23.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj93 2023-10-11T16:09:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj93 2023-10-11T16:09:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.642 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.643 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.643 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.643 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.643 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj93 2023-10-11T16:09:23.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.644 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:nItTpqznpQ/xmkCXdEHy9ETyKjY= 2023-10-11T16:09:23.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:nItTpqznpQ/xmkCXdEHy9ETyKjY='} 2023-10-11T16:09:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072f462ca4933ec31-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.648 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.648 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.648 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.649 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.649 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.650 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b039be01fceeb487-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj124 2023-10-11T16:09:23.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj124 2023-10-11T16:09:23.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj124 2023-10-11T16:09:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.654 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.654 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.655 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.655 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.655 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj124 2023-10-11T16:09:23.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.656 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Kz56knOUqYm9Kn0Nv98WWwwvZNk= 2023-10-11T16:09:23.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Kz56knOUqYm9Kn0Nv98WWwwvZNk='} 2023-10-11T16:09:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b46c68dba1183ae-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.660 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.661 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.661 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.661 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.661 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.662 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0= 2023-10-11T16:09:23.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:pjHgUbL0EyCYWeuH+3DZzu0cGw0=', 'Content-Length': '0'} 2023-10-11T16:09:23.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004562046336953b61-006526c2c8-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj155 2023-10-11T16:09:23.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj155 2023-10-11T16:09:23.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj155 2023-10-11T16:09:23.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.666 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.667 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.667 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.667 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:08 GMT 2023-10-11T16:09:23.667 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj155 2023-10-11T16:09:23.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.668 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:BLKRE922d5bITrWXXnSzD7E9fZA= 2023-10-11T16:09:23.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:BLKRE922d5bITrWXXnSzD7E9fZA='} 2023-10-11T16:09:23.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aefea83d65b6daf7-006526c2c8-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dghqvd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dghqvd-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: tenant: 2023-10-11T16:09:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: bucket name: dghqvd-64 2023-10-11T16:09:23.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: total entries: 30 2023-10-11T16:09:23.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.675 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.675 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.676 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.676 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.676 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.677 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T16:09:23.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T16:09:23.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c2576370e5d957-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj37 2023-10-11T16:09:23.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj37 2023-10-11T16:09:23.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj37 2023-10-11T16:09:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.681 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.681 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.681 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.681 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.682 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj37 2023-10-11T16:09:23.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.682 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:iI8Mv8BrHRNR8zndcTT7EyHEcXU= 2023-10-11T16:09:23.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:iI8Mv8BrHRNR8zndcTT7EyHEcXU='} 2023-10-11T16:09:23.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009592dfd765025fc2-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.686 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.687 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.687 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.688 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.688 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T16:09:23.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T16:09:23.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef94b821db3a67b6-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj74 2023-10-11T16:09:23.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj74 2023-10-11T16:09:23.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj74 2023-10-11T16:09:23.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.692 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.693 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.693 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.693 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.694 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj74 2023-10-11T16:09:23.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.694 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:/nsZIBohCzBYRF3MAaqfSsEoMzs= 2023-10-11T16:09:23.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:/nsZIBohCzBYRF3MAaqfSsEoMzs='} 2023-10-11T16:09:23.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d33c200753c881d4-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.698 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.699 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.699 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.699 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.700 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.700 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T16:09:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T16:09:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eabe3be6b3a8070c-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj111 2023-10-11T16:09:23.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj111 2023-10-11T16:09:23.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj111 2023-10-11T16:09:23.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.705 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.705 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.705 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.706 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.706 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj111 2023-10-11T16:09:23.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.706 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:2xSlVctOxefxZEBs56CuB2kIM58= 2023-10-11T16:09:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:2xSlVctOxefxZEBs56CuB2kIM58='} 2023-10-11T16:09:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aae6300131353ab-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.711 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.711 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.711 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.712 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.712 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.712 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T16:09:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T16:09:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089f8d8bab8d86e79-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj148 2023-10-11T16:09:23.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj148 2023-10-11T16:09:23.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj148 2023-10-11T16:09:23.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.716 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.717 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.717 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.718 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj148 2023-10-11T16:09:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.718 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:gR0zFmjb60AwRkuBSM2qElU98HA= 2023-10-11T16:09:23.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gR0zFmjb60AwRkuBSM2qElU98HA='} 2023-10-11T16:09:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd3d43df4d747087-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/ 2023-10-11T16:09:23.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/ 2023-10-11T16:09:23.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/ 2023-10-11T16:09:23.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.722 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.723 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.723 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.723 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.724 INFO:tasks.rgw_multisite_tests:/dghqvd-64/ 2023-10-11T16:09:23.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.724 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s= 2023-10-11T16:09:23.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:UhROzEDkfhK19MeJXZrWhbzUM8s=', 'Content-Length': '0'} 2023-10-11T16:09:23.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d386a66b0c18456-006526c2c9-1675-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-64/obj185 2023-10-11T16:09:23.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-64/obj185 2023-10-11T16:09:23.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-64/obj185 2023-10-11T16:09:23.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-10-11T16:09:23.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.729 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.729 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-10-11T16:09:23.729 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.730 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:44:09 GMT 2023-10-11T16:09:23.730 INFO:tasks.rgw_multisite_tests:/dghqvd-64/obj185 2023-10-11T16:09:23.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.731 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MTSDNRnp7yz0nikjZBHwl4pVGmI= 2023-10-11T16:09:23.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:44:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MTSDNRnp7yz0nikjZBHwl4pVGmI='} 2023-10-11T16:09:23.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003daaa8562779af0e-006526c2c9-1675-a1'), ('Date', 'Wed, 11 Oct 2023 15:44:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:09.289+0000 7f0df07b6780 5 RGW-SYNC:data: waiting on data sync status lease 2023-10-11T16:09:23.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:09.290+0000 7f0df07b6780 5 RGW-SYNC:data: acquired data sync status lease 2023-10-11T16:09:23.734 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-10-11T16:09:23.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-10-11T16:09:23.734 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-10-11T16:09:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T16:09:23.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-11T16:09:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-10-11T16:09:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi158.front.sepia.ceph.com port 8001: Connection refused 2023-10-11T16:09:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > curl http://smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-11T16:09:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: Dload Upload Total Spent Left Speed 2023-10-11T16:09:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-11T16:09:23.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: anonymous 2023-10-11T16:09:23.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dghqvd-64 2023-10-11T16:09:23.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:23.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dghqvd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-10-11T16:09:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dghqvd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-10-11T16:09:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:10.501+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:15.502+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:20.502+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:25.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:30.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:35.503+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:40.504+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:45.504+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:50.505+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:44:55.506+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:00.506+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:05.507+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:10.508+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:15.509+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:20.509+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:25.510+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:30.510+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:35.511+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:40.511+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:45.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:50.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:45:55.512+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:00.513+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:05.513+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:10.514+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 0.. 2023-10-11T16:09:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:15.514+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:20.515+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:25.516+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:30.517+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:35.518+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:40.518+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:45.519+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:50.519+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:46:55.520+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:00.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:05.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:10.522+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:15.521+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:20.522+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:25.523+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:30.523+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:35.524+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:40.525+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:45.526+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:50.527+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:47:55.527+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:00.528+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:05.528+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:10.529+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:15.530+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:20.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:25.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:30.532+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:35.531+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:40.532+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:45.533+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:50.533+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:48:55.534+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:49:00.535+0000 7f6d581d5780 1 waiting to reach latest gen 6, on 4.. 2023-10-11T16:09:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:49:05.536+0000 7f6d581d5780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-10-11T16:09:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:49:05.536+0000 7f6d581d5780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T16:09:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:49:05.536+0000 7f6d581d5780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-10-11T16:09:23.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-10-11T16:09:23.758 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-11T16:09:23.758 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.759 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-11T16:09:23.759 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2023-10-11T16:09:23.759 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:23.760 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-11T16:09:23.760 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-11T16:09:23.761 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-11T16:09:23.761 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2180, in test_sync_flow_symmetrical_zonegroup_all 2023-10-11T16:09:23.761 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[1], content) 2023-10-11T16:09:23.762 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-11T16:09:23.762 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-11T16:09:23.762 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-11T16:09:23.763 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-11T16:09:23.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.763 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:23.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:23.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:23.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:51:01.459205Z" 2023-10-11T16:09:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:23.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:23.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:23.224+0000 7f7fba8e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:23.225+0000 7f7fba8e5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:23.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:23.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:23.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:23.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:23.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:23.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:23.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:23.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:23Z" 2023-10-11T16:09:23.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.793 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:23.793 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:23.793 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:23.794 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:23.794 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:23.794 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.795 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.795 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:23.795 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.796 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:23.796 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.796 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.797 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:23.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:23.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.799 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.799 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.799 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.799 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:23.800 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.800 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:23.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.801 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:23.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.802 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.802 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.803 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.803 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:23.803 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:23.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.806 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.806 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.806 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.807 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:23.807 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:23.808 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.808 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:23.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.809 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.810 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.810 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:23.810 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.811 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:23.811 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:23.811 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:23.811 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.812 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:23Z" 2023-10-11T16:09:23.812 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:23.812 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:23.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:23.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:23.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:23.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.814 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[]} 2023-10-11T16:09:23.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.816 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-10-11T16:09:23.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.818 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-10-11T16:09:23.819 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.820 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:09:23.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:23.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 15, 2023-10-11T16:09:23.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:09:23.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": [], 2023-10-11T16:09:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period_map": { 2023-10-11T16:09:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zonegroups": [ 2023-10-11T16:09:23.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "a", 2023-10-11T16:09:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a", 2023-10-11T16:09:23.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "api_name": "", 2023-10-11T16:09:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "is_master": true, 2023-10-11T16:09:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:09:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames": [], 2023-10-11T16:09:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames_s3website": [], 2023-10-11T16:09:23.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:23.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a2", 2023-10-11T16:09:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:09:23.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:23.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:23.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:23.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:23.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:23.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:23.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:23.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a1", 2023-10-11T16:09:23.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:09:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:23.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:23.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:23.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "placement_targets": [ 2023-10-11T16:09:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "default-placement", 2023-10-11T16:09:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [], 2023-10-11T16:09:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "storage_classes": [ 2023-10-11T16:09:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "STANDARD" 2023-10-11T16:09:23.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "default_placement": "default-placement", 2023-10-11T16:09:23.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:23.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_policy": { 2023-10-11T16:09:23.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "groups": [ 2023-10-11T16:09:23.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-group", 2023-10-11T16:09:23.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "data_flow": { 2023-10-11T16:09:23.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "symmetrical": [ 2023-10-11T16:09:23.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-flow1", 2023-10-11T16:09:23.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pipes": [ 2023-10-11T16:09:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-pipe", 2023-10-11T16:09:23.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "source": { 2023-10-11T16:09:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket": "*", 2023-10-11T16:09:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:23.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "dest": { 2023-10-11T16:09:23.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket": "*", 2023-10-11T16:09:23.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:23.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:23.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:23.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "params": { 2023-10-11T16:09:23.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "source": { 2023-10-11T16:09:23.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "filter": { 2023-10-11T16:09:23.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [] 2023-10-11T16:09:23.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "dest": {}, 2023-10-11T16:09:23.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "priority": 0, 2023-10-11T16:09:23.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "mode": "system", 2023-10-11T16:09:23.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user": "" 2023-10-11T16:09:23.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "enabled" 2023-10-11T16:09:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled_features": [ 2023-10-11T16:09:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "b", 2023-10-11T16:09:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b", 2023-10-11T16:09:23.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "api_name": "", 2023-10-11T16:09:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "is_master": false, 2023-10-11T16:09:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:09:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames": [], 2023-10-11T16:09:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames_s3website": [], 2023-10-11T16:09:23.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:23.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:23.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:23.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b1", 2023-10-11T16:09:23.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:09:23.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:23.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:23.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:23.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:23.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:23.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:23.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:23.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:09:23.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b2", 2023-10-11T16:09:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:09:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:23.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:23.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:23.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:23.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:23.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "placement_targets": [ 2023-10-11T16:09:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "default-placement", 2023-10-11T16:09:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [], 2023-10-11T16:09:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "storage_classes": [ 2023-10-11T16:09:23.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "STANDARD" 2023-10-11T16:09:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "default_placement": "default-placement", 2023-10-11T16:09:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_policy": { 2023-10-11T16:09:23.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "groups": [] 2023-10-11T16:09:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled_features": [ 2023-10-11T16:09:23.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:23.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "short_zone_ids": [ 2023-10-11T16:09:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:23.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 2025758522 2023-10-11T16:09:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:23.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 3270403816 2023-10-11T16:09:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:09:23.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 178712359 2023-10-11T16:09:23.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 2663201345 2023-10-11T16:09:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zonegroup": "a", 2023-10-11T16:09:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:23.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period_config": { 2023-10-11T16:09:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_quota": { 2023-10-11T16:09:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false, 2023-10-11T16:09:23.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "check_on_raw": false, 2023-10-11T16:09:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size": -1, 2023-10-11T16:09:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size_kb": 0, 2023-10-11T16:09:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_objects": -1 2023-10-11T16:09:23.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user_quota": { 2023-10-11T16:09:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false, 2023-10-11T16:09:23.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "check_on_raw": false, 2023-10-11T16:09:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size": -1, 2023-10-11T16:09:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size_kb": 0, 2023-10-11T16:09:23.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_objects": -1 2023-10-11T16:09:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user_ratelimit": { 2023-10-11T16:09:23.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:23.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:23.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_ratelimit": { 2023-10-11T16:09:23.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:23.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "anonymous_ratelimit": { 2023-10-11T16:09:23.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:23.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:23.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_name": "test-realm", 2023-10-11T16:09:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:23.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dghqvd-66 2023-10-11T16:09:23.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/ 2023-10-11T16:09:23.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/ 2023-10-11T16:09:23.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/ 2023-10-11T16:09:23.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:09:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.905 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.905 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.905 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.905 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.906 INFO:tasks.rgw_multisite_tests:/dghqvd-66/ 2023-10-11T16:09:23.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.906 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:MrlYqEGWcmlT/QSs0ZxsMEnHER4= 2023-10-11T16:09:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:MrlYqEGWcmlT/QSs0ZxsMEnHER4=', 'Content-Length': '0'} 2023-10-11T16:09:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009a8f8d9b22f7eb88-006526c47c-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/ 2023-10-11T16:09:23.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/ 2023-10-11T16:09:23.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/ 2023-10-11T16:09:23.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.910 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.911 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.911 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.911 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.912 INFO:tasks.rgw_multisite_tests:/dghqvd-66/ 2023-10-11T16:09:23.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.912 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:eJfKpF3FQUY94cCqkoJljnfeCm8= 2023-10-11T16:09:23.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:eJfKpF3FQUY94cCqkoJljnfeCm8=', 'Content-Length': '0'} 2023-10-11T16:09:23.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef1ef82a77f54332-006526c47c-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/obj1 2023-10-11T16:09:23.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/obj1 2023-10-11T16:09:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/obj1 2023-10-11T16:09:23.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:23.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.916 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.917 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:23.917 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.917 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.918 INFO:tasks.rgw_multisite_tests:/dghqvd-66/obj1 2023-10-11T16:09:23.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.918 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:gXgszBL+Y1ZxJ0JduLBZHkNEGsw= 2023-10-11T16:09:23.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:gXgszBL+Y1ZxJ0JduLBZHkNEGsw='} 2023-10-11T16:09:23.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000242bba2554e56617-006526c47c-2322-a1'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dghqvd-67 2023-10-11T16:09:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-67/ 2023-10-11T16:09:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-67/ 2023-10-11T16:09:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-67/ 2023-10-11T16:09:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:23.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.923 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.924 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.924 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.924 INFO:tasks.rgw_multisite_tests:/dghqvd-67/ 2023-10-11T16:09:23.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.925 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:ls2E1+6bFNKp5ApB/ghf95gOUUQ= 2023-10-11T16:09:23.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:ls2E1+6bFNKp5ApB/ghf95gOUUQ=', 'Content-Length': '0'} 2023-10-11T16:09:23.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000c1da0f351ed3dba6-006526c47c-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-67/ 2023-10-11T16:09:23.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-67/ 2023-10-11T16:09:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-67/ 2023-10-11T16:09:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.929 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:23.929 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.930 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.930 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.930 INFO:tasks.rgw_multisite_tests:/dghqvd-67/ 2023-10-11T16:09:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.931 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:oRUX9PH+5CruoElarAQJAiCUHE4= 2023-10-11T16:09:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:oRUX9PH+5CruoElarAQJAiCUHE4=', 'Content-Length': '0'} 2023-10-11T16:09:23.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083a5251a8fbb9bbc-006526c47c-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-67/obj2 2023-10-11T16:09:23.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-67/obj2 2023-10-11T16:09:23.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:23.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-67/obj2 2023-10-11T16:09:23.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:23.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:23.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:23.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:23.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:23.935 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:23.935 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:23.935 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:23.936 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:51:24 GMT 2023-10-11T16:09:23.936 INFO:tasks.rgw_multisite_tests:/dghqvd-67/obj2 2023-10-11T16:09:23.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:23.937 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:Gwwsl4LTc2dl8hwQ7xbqydUCpt4= 2023-10-11T16:09:23.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:51:24 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:Gwwsl4LTc2dl8hwQ7xbqydUCpt4='} 2023-10-11T16:09:23.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddc549f40fe2f225-006526c47c-232c-a2'), ('Date', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:23.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:23.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:23.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:23.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:23.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:23.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:51:24.295655Z" 2023-10-11T16:09:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:23.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:23.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.945 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:23.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:24.630+0000 7f9ae7a6e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:24.631+0000 7f9ae7a6e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:23.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:23.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:23.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:23.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:23.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:23.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:23.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:23.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:23.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:23.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:23.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:23.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:24Z" 2023-10-11T16:09:23.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:23.967 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:23.967 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:23.968 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:23.968 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:23.968 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.969 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.969 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:23.969 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.970 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:23.970 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.971 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:23.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:23.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.972 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.973 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.973 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.973 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:23.973 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.974 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:23.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:23.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.976 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.976 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.976 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.977 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:23.977 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:23.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:23.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.979 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.980 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.980 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:23.980 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:23.981 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:23.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:23.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:23.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:23.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:23.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:23.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:23.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:23.983 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.983 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:23.984 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:23.984 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:23.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:23.985 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:23.985 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:23.985 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:24Z" 2023-10-11T16:09:23.986 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:23.986 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:23.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:23.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:23.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:23.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:23.987 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:23.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:23.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:23.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:23.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:29.757+0000 7f3fa2e0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:29.758+0000 7f3fa2e0c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:23.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:23.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:23.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:23.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:23.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:23.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:23.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:23.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:23.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:23.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:23.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:23.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:23.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:23.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:29Z" 2023-10-11T16:09:24.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.009 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.009 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.010 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.010 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.010 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.011 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.011 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.012 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.012 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.012 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.013 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.015 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.015 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.015 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.016 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.016 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.016 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.017 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.017 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.018 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.018 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.019 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.019 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.019 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.022 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.022 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.023 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.023 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.023 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.025 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.026 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.026 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.026 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.027 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.028 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.028 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:29Z" 2023-10-11T16:09:24.029 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.029 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:34.881+0000 7f8033a04780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:34.882+0000 7f8033a04780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:34Z" 2023-10-11T16:09:24.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.052 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.053 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.053 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.053 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.054 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.054 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.055 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.055 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.055 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.056 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.057 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.058 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.058 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.058 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.059 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.059 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.059 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.060 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.060 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.061 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.062 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.062 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.062 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.065 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.066 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.066 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.066 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.067 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.068 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.069 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.069 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.069 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.070 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.070 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.070 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.070 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.071 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.071 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:34Z" 2023-10-11T16:09:24.072 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.072 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:40.004+0000 7f8f5e3f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:40.005+0000 7f8f5e3f0780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:40Z" 2023-10-11T16:09:24.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.095 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.095 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.096 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.096 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.096 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.097 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.097 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.098 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.098 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.098 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.099 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.099 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.100 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.101 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.101 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.101 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.102 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.102 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.102 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.103 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.104 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.105 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.105 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.105 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.105 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.108 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.108 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.108 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.109 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.111 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.112 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.112 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.112 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.113 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.113 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.113 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.113 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.114 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:40Z" 2023-10-11T16:09:24.114 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.114 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.116 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:45.129+0000 7f7d2f77a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:45.130+0000 7f7d2f77a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:45Z" 2023-10-11T16:09:24.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.138 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.138 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.138 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.139 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.139 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.139 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.140 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.140 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.140 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.141 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.141 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.141 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.142 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.142 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.142 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.143 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.143 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.144 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.144 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.144 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.145 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.146 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.147 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.147 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.147 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.148 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.148 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.148 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.150 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.151 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.151 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.151 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.154 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.154 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.154 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.155 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.155 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.155 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.156 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.156 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.156 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:45Z" 2023-10-11T16:09:24.156 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.157 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:50.256+0000 7efd7c11c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:50.257+0000 7efd7c11c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:50Z" 2023-10-11T16:09:24.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.180 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.180 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.180 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.181 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.181 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.182 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.182 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.182 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.183 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.184 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.185 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.185 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.185 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.186 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.186 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.186 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.187 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.188 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.188 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.189 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.189 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.189 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.190 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.192 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.192 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.193 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.193 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.193 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.194 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.196 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.196 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.196 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.197 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.197 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.197 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.197 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.198 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.198 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:50Z" 2023-10-11T16:09:24.198 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.199 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.201 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:55.376+0000 7fdf6f99a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:51:55.378+0000 7fdf6f99a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:51:55Z" 2023-10-11T16:09:24.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.223 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.223 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.223 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.224 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.224 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.225 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.225 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.225 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.226 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.226 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.226 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.227 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.228 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.229 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.229 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.229 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.230 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.230 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.231 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.232 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.232 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.233 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.233 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.235 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.235 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.235 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.236 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.236 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.236 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.237 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.237 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.239 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.240 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.240 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.240 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.240 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.241 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.241 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.241 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.242 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:51:55Z" 2023-10-11T16:09:24.242 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.242 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:00.504+0000 7fcd08439780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:00.505+0000 7fcd08439780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:00Z" 2023-10-11T16:09:24.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.266 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.267 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.267 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.267 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.267 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.268 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.268 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.269 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.269 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.269 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.270 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.271 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.272 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.272 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.272 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.273 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.273 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.274 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.275 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.275 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.276 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.276 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.276 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.279 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.279 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.279 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.280 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.282 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.283 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.283 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.283 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.283 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.284 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.284 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.284 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.285 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:00Z" 2023-10-11T16:09:24.285 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.285 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.287 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:05.628+0000 7f3235ac7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:05.629+0000 7f3235ac7780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:05Z" 2023-10-11T16:09:24.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.309 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.309 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.310 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.310 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.310 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.311 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.311 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.312 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.312 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.313 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.315 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.315 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.315 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.315 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.316 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.317 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.318 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.318 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.319 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.319 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.319 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.320 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.320 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.321 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.322 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.322 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.322 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.323 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.323 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.324 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.325 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.326 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.326 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.326 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.327 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.327 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.327 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.328 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.328 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:05Z" 2023-10-11T16:09:24.328 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.328 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.330 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:10.752+0000 7f30bd8d5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:10.753+0000 7f30bd8d5780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:10Z" 2023-10-11T16:09:24.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.355 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.355 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.355 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.356 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.356 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.357 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.357 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.357 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.358 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.358 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.359 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.360 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.361 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.361 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.361 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.361 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.362 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.363 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.364 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.364 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.365 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.365 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.365 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.368 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.368 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.368 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.371 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.372 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.372 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.372 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.373 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.373 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.374 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.374 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:10Z" 2023-10-11T16:09:24.374 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.374 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.376 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:15.877+0000 7f472a7ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:15.878+0000 7f472a7ee780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:15Z" 2023-10-11T16:09:24.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.398 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.399 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.399 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.399 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.400 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.400 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.400 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.401 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.401 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.401 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.402 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.403 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.404 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.404 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.404 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.406 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.407 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.408 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.408 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.408 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.408 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.409 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.411 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.412 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.412 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.412 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.413 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.413 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.413 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.413 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.414 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.415 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.415 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.415 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.416 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.416 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.416 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.417 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.417 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.417 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.417 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:15Z" 2023-10-11T16:09:24.418 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.418 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.419 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:21.001+0000 7f69aa35a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:21.002+0000 7f69aa35a780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:21Z" 2023-10-11T16:09:24.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.441 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.441 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.442 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.442 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.442 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.443 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.443 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.443 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.444 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.444 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.445 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.447 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.447 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.447 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.447 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.448 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.448 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.449 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.450 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.450 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.450 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.451 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.451 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.451 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.454 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.454 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.455 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.455 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.455 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.458 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.458 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.459 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.459 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.459 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.459 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.460 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.460 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.460 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:21Z" 2023-10-11T16:09:24.461 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.461 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.463 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:26.126+0000 7f5397e48780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:26.127+0000 7f5397e48780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:26Z" 2023-10-11T16:09:24.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.486 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.486 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.487 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.487 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.487 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.488 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.488 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.488 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.489 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.489 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.489 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.490 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.491 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.492 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.492 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.492 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.492 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.493 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.494 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.495 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.495 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.495 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.496 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.496 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.497 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.498 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.499 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.499 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.499 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.499 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.502 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.502 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.502 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.503 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.503 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.503 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.503 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.504 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.504 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:26Z" 2023-10-11T16:09:24.504 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.505 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:31.251+0000 7effad2e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:31.252+0000 7effad2e1780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:31Z" 2023-10-11T16:09:24.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.528 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.528 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.529 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.529 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.529 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.530 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.530 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.531 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.531 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.531 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.532 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.532 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.533 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.533 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.534 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.534 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.534 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.534 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.535 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.535 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.536 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.537 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.537 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.538 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.538 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.538 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.539 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.541 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.541 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.541 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.541 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.542 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.542 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.542 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.544 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.544 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.545 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.545 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.546 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.546 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.546 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.546 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:31Z" 2023-10-11T16:09:24.547 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.547 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:36.377+0000 7f95bce7e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:36.378+0000 7f95bce7e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:36Z" 2023-10-11T16:09:24.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.571 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.571 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.571 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.572 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.572 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.573 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.573 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.573 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.573 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.574 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.574 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.574 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.575 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.575 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.575 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.576 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.576 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.576 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.577 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.577 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.577 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.578 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.579 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.580 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.580 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.580 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.581 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.583 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.584 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.584 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.584 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.585 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.585 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.586 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.587 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.587 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.588 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.588 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.588 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.589 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.589 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.589 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.589 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:36Z" 2023-10-11T16:09:24.590 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.590 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.591 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:41.502+0000 7fbc6cfef780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:41.503+0000 7fbc6cfef780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:41Z" 2023-10-11T16:09:24.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.613 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.614 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.614 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.614 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.615 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.615 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.615 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.616 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.616 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.616 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.617 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.618 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.619 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.619 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.619 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.620 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.620 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.621 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.622 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.622 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.623 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.623 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.623 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.624 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.625 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.626 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.626 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.627 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.627 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.627 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.628 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.630 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.630 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.630 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.631 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.631 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.631 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.632 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.632 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.632 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:41Z" 2023-10-11T16:09:24.632 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.633 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:46.625+0000 7fc0d7f5c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:46.626+0000 7fc0d7f5c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:46Z" 2023-10-11T16:09:24.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.656 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.657 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.657 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.658 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.658 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.658 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.659 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.659 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.661 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.662 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.662 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.663 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.663 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.664 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.665 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.666 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.666 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.666 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.667 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.667 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.667 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.669 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.670 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.670 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.670 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.670 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.673 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.673 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.673 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.674 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.674 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.674 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.675 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.675 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.675 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:46Z" 2023-10-11T16:09:24.676 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.676 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:51.746+0000 7f4c6bc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:51.747+0000 7f4c6bc6c780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:51Z" 2023-10-11T16:09:24.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.699 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.700 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.700 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.700 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.700 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.701 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.701 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.701 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.702 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.702 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.703 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.703 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.704 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.705 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.705 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.705 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.706 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.706 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.708 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.708 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.709 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.709 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.709 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.709 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.710 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.710 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.710 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.711 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.711 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.711 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.712 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.712 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.712 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.712 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.713 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.713 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.713 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.714 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.715 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.716 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.716 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.716 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.717 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.717 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.717 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.718 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.718 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:51Z" 2023-10-11T16:09:24.718 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.719 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.720 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi158.stdout: INFO: 2023-10-11T15:52:55.282+0000 7fb229538700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-10-11T16:09:24.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:56.873+0000 7f8e89b4b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:52:56.874+0000 7f8e89b4b780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:52:56Z" 2023-10-11T16:09:24.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.742 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.742 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.743 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.743 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.743 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.743 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.744 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.744 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.744 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.744 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.745 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.745 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.745 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.746 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.747 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.747 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.747 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.748 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.748 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.748 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.749 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.750 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.750 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.751 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.751 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.751 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.752 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.753 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.753 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.753 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.753 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.754 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.754 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.754 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.755 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.755 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.755 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.757 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.758 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.758 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.758 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.759 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.759 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.759 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.759 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.760 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:52:56Z" 2023-10-11T16:09:24.760 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.760 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.761 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:02.014+0000 7f8c9cd0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:02.015+0000 7f8c9cd0e780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:02Z" 2023-10-11T16:09:24.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.784 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.784 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.784 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.785 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.785 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.785 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.786 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.786 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.786 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.786 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.787 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.787 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.787 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.788 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.789 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.789 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.790 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.790 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.790 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.791 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.792 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.793 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.793 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.793 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.793 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.795 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.796 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.796 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.797 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.797 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.798 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039461.459205_891.1", 2023-10-11T16:09:24.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:01.459205Z", 2023-10-11T16:09:24.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.800 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.800 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.800 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.801 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.801 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.801 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.802 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.802 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.802 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:02Z" 2023-10-11T16:09:24.803 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.803 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039461.459205_891.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:01.459205Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039461.459205_891.1'} 2023-10-11T16:09:24.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1697039484.295655_913.1 target=1_1697039461.459205_891.1 2023-10-11T16:09:24.804 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:07.137+0000 7f8819985780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:07.138+0000 7f8819985780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T16:09:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.826 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.826 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.826 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.826 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.827 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.827 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.827 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.828 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.828 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.829 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.831 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.831 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.831 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.832 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.832 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.833 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.834 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.834 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.834 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.835 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.835 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.837 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.838 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.838 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.838 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.839 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.839 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:24.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T16:09:24.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.841 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.841 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.842 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.842 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.843 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.843 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.843 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.844 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.844 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.844 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039484.295655_913.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.295655Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:24.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.846 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:24.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:24.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:24.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:24.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:09:24.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.853 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:24.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:24.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:24.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:24.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:24.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:24.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:24.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:24.873 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.873 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.873 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.874 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.874 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:24.874 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.875 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.875 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.875 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.875 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.876 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:24.876 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:24.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:24.878 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.878 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.878 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.879 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.879 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.879 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:24.879 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:24.880 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.880 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.881 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:24.881 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.881 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.881 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.882 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.882 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.883 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:24.883 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:24.883 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.883 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:24.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:24.884 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.885 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.885 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.885 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.886 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.886 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:24.886 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:24.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.887 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:24.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:24.888 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.888 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.888 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.889 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.889 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.889 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:24.889 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:24.890 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.890 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.890 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.891 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:24.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:24.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:24.892 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:09:24.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/ 2023-10-11T16:09:24.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/ 2023-10-11T16:09:24.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:24.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/ 2023-10-11T16:09:24.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:24.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.896 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:24.896 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.896 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.896 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.897 INFO:tasks.rgw_multisite_tests:/dghqvd-66/ 2023-10-11T16:09:24.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.897 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:8BjPqGGqepAMgr9Qdvhu+Ty6Btg= 2023-10-11T16:09:24.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:8BjPqGGqepAMgr9Qdvhu+Ty6Btg=', 'Content-Length': '0'} 2023-10-11T16:09:24.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7df2aba59629334-006526c4e3-232c-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/obj1 2023-10-11T16:09:24.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/obj1 2023-10-11T16:09:24.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:24.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/obj1 2023-10-11T16:09:24.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:24.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:24.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.901 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:24.901 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.902 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.902 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.902 INFO:tasks.rgw_multisite_tests:/dghqvd-66/obj1 2023-10-11T16:09:24.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.903 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:++2ygImC4Ugc4Gm130iMHETmHok= 2023-10-11T16:09:24.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:++2ygImC4Ugc4Gm130iMHETmHok=', 'Content-Length': '0'} 2023-10-11T16:09:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-66:daa316e9-869c-4074-a756-a8adfb4a3573.8996.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002acbc670a5739d36-006526c4e3-232c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-66/obj1 2023-10-11T16:09:24.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-66/obj1 2023-10-11T16:09:24.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:24.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-66/obj1 2023-10-11T16:09:24.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:24.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:24.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.907 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:24.908 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.908 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.908 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.908 INFO:tasks.rgw_multisite_tests:/dghqvd-66/obj1 2023-10-11T16:09:24.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.909 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:U6nUSieF2ZZjN7mFM0Ep++VefzU= 2023-10-11T16:09:24.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:U6nUSieF2ZZjN7mFM0Ep++VefzU=', 'Content-Length': '0'} 2023-10-11T16:09:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:51:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-66:daa316e9-869c-4074-a756-a8adfb4a3573.8996.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca19dff8a91a471f-006526c4e3-232c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-67/ 2023-10-11T16:09:24.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-67/ 2023-10-11T16:09:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-67/ 2023-10-11T16:09:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:24.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:24.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-10-11T16:09:24.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.914 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:24.914 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.914 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.915 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.915 INFO:tasks.rgw_multisite_tests:/dghqvd-67/ 2023-10-11T16:09:24.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.915 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:A+W7uV6JC2e4/T6fyh5Nq9pi498= 2023-10-11T16:09:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:A+W7uV6JC2e4/T6fyh5Nq9pi498=', 'Content-Length': '0'} 2023-10-11T16:09:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abbc392af5ac44c6-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-67/obj2 2023-10-11T16:09:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-67/obj2 2023-10-11T16:09:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:24.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-67/obj2 2023-10-11T16:09:24.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:24.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:24.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.920 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:24.920 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.920 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.921 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.921 INFO:tasks.rgw_multisite_tests:/dghqvd-67/obj2 2023-10-11T16:09:24.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.922 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9U9+zeOXd50UbJ2nzGY2Ypu0fug= 2023-10-11T16:09:24.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9U9+zeOXd50UbJ2nzGY2Ypu0fug=', 'Content-Length': '0'} 2023-10-11T16:09:24.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000005875d73ddf36534-006526c4e3-2322-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.922 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-11T16:09:24.923 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.923 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-10-11T16:09:24.924 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2023-10-11T16:09:24.924 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:24.925 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-10-11T16:09:24.925 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-10-11T16:09:24.926 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-10-11T16:09:24.926 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2432, in test_sync_single_bucket 2023-10-11T16:09:24.926 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2023-10-11T16:09:24.927 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-10-11T16:09:24.927 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-10-11T16:09:24.927 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-10-11T16:09:24.928 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-10-11T16:09:24.928 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-10-11T16:09:24.928 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-10-11T16:09:24.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.929 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:24.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:24.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:24.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:51:24.295655Z" 2023-10-11T16:09:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:24.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:07.614+0000 7f035d347780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:07.615+0000 7f035d347780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:24.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:24.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:24.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:24.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:24.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:24.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:24.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:24.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T16:09:24.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:24.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:24.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:24.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:24.958 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:24.958 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:24.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:24.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:24.959 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:24.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.960 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.960 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:24.960 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.960 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:24.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.962 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:24.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:24.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.963 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.963 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.964 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.964 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:24.964 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.965 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:24.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.966 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:24.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.967 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.967 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.967 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.968 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:24.968 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:24.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:24.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.970 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.971 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.971 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:24.971 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:24.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:24.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:24.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039484.295655_913.1", 2023-10-11T16:09:24.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:24.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:24.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:24.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.295655Z", 2023-10-11T16:09:24.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:24.974 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.974 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:24.974 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:24.975 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.975 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:24.975 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:24.976 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:24.976 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:24.976 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:07Z" 2023-10-11T16:09:24.976 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:24.977 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039484.295655_913.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.295655Z', 'realm_epoch': 2}}] 2023-10-11T16:09:24.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039484.295655_913.1'} 2023-10-11T16:09:24.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:24.978 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:24.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:24.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:24.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:24.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:24.982 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dghqvd-68 2023-10-11T16:09:24.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/ 2023-10-11T16:09:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/ 2023-10-11T16:09:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/ 2023-10-11T16:09:24.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:24.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:24.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.986 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:24.986 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.986 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.986 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.987 INFO:tasks.rgw_multisite_tests:/dghqvd-68/ 2023-10-11T16:09:24.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.987 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:HeImfQnPRmIaV/RebVFVUNHqiGk= 2023-10-11T16:09:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:HeImfQnPRmIaV/RebVFVUNHqiGk=', 'Content-Length': '0'} 2023-10-11T16:09:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006fa3fdfcb0615b3e-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dghqvd-69 2023-10-11T16:09:24.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-69/ 2023-10-11T16:09:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-69/ 2023-10-11T16:09:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:24.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-69/ 2023-10-11T16:09:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:24.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:24.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:24.992 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:24.992 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.992 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:24.993 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:07 GMT 2023-10-11T16:09:24.993 INFO:tasks.rgw_multisite_tests:/dghqvd-69/ 2023-10-11T16:09:24.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:24.993 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:jH02q8Ci+aAXbxTSBxBG64rd5EU= 2023-10-11T16:09:24.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:07 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:jH02q8Ci+aAXbxTSBxBG64rd5EU=', 'Content-Length': '0'} 2023-10-11T16:09:24.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ead90a13f2e8e8cf-006526c4e3-2322-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:07 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:24.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:24.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:24.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:24.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:24.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:24.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:24.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:24.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:24.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:53:07.869026Z" 2023-10-11T16:09:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:25.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:08.059+0000 7fe695668780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:08.060+0000 7fe695668780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:25.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T16:09:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:25.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:08Z" 2023-10-11T16:09:25.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:25.024 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.024 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.024 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.025 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.025 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.025 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.026 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.026 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.026 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:25.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.028 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:25.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:25.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.029 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.030 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.030 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.030 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.030 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.031 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:25.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.032 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:25.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.033 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.033 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.034 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.034 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.034 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:25.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.037 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.037 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.037 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.038 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.038 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T16:09:25.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.040 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.041 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.041 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.041 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.042 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.042 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:25.042 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:25.042 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.043 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:08Z" 2023-10-11T16:09:25.043 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.043 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039587.869026_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:07.869026Z', 'realm_epoch': 2}}] 2023-10-11T16:09:25.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:25.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:09:25.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:09:25.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dghqvd-68', '--dest-bucket=dghqvd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.048 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dghqvd-68', '--dest-bucket=dghqvd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=dghqvd-68 --dest-bucket=dghqvd-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-10-11T16:09:25.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.050 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:25.051 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.052 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dghqvd-68","zones":["a2","a1"]},"dest":{"bucket":"dghqvd-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-10-11T16:09:25.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "epoch": 16, 2023-10-11T16:09:25.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "predecessor_uuid": "3707f754-0ea4-480f-82c9-e0395881372c", 2023-10-11T16:09:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": [], 2023-10-11T16:09:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period_map": { 2023-10-11T16:09:25.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zonegroups": [ 2023-10-11T16:09:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "a", 2023-10-11T16:09:25.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a", 2023-10-11T16:09:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "api_name": "", 2023-10-11T16:09:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "is_master": true, 2023-10-11T16:09:25.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:09:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames": [], 2023-10-11T16:09:25.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames_s3website": [], 2023-10-11T16:09:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:25.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a2", 2023-10-11T16:09:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8001" 2023-10-11T16:09:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:25.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:25.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:25.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "a1", 2023-10-11T16:09:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi158.front.sepia.ceph.com:8000" 2023-10-11T16:09:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:25.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:25.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:25.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "placement_targets": [ 2023-10-11T16:09:25.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "default-placement", 2023-10-11T16:09:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [], 2023-10-11T16:09:25.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "storage_classes": [ 2023-10-11T16:09:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "STANDARD" 2023-10-11T16:09:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "default_placement": "default-placement", 2023-10-11T16:09:25.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_policy": { 2023-10-11T16:09:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "groups": [ 2023-10-11T16:09:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-group", 2023-10-11T16:09:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "data_flow": { 2023-10-11T16:09:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "symmetrical": [ 2023-10-11T16:09:25.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-flow1", 2023-10-11T16:09:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:25.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pipes": [ 2023-10-11T16:09:25.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "sync-pipe", 2023-10-11T16:09:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "source": { 2023-10-11T16:09:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket": "dghqvd-68", 2023-10-11T16:09:25.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "dest": { 2023-10-11T16:09:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket": "dghqvd-68", 2023-10-11T16:09:25.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "daa316e9-869c-4074-a756-a8adfb4a3573" 2023-10-11T16:09:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "params": { 2023-10-11T16:09:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "source": { 2023-10-11T16:09:25.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "filter": { 2023-10-11T16:09:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [] 2023-10-11T16:09:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "dest": {}, 2023-10-11T16:09:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "priority": 0, 2023-10-11T16:09:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "mode": "system", 2023-10-11T16:09:25.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user": "" 2023-10-11T16:09:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "enabled" 2023-10-11T16:09:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled_features": [ 2023-10-11T16:09:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "b", 2023-10-11T16:09:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b", 2023-10-11T16:09:25.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "api_name": "", 2023-10-11T16:09:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "is_master": false, 2023-10-11T16:09:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:09:25.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames": [], 2023-10-11T16:09:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "hostnames_s3website": [], 2023-10-11T16:09:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:25.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "zones": [ 2023-10-11T16:09:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:25.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b1", 2023-10-11T16:09:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8000" 2023-10-11T16:09:25.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:25.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:25.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:25.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "id": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:09:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "b2", 2023-10-11T16:09:25.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "endpoints": [ 2023-10-11T16:09:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "http://smithi172.front.sepia.ceph.com:8001" 2023-10-11T16:09:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_meta": false, 2023-10-11T16:09:25.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "log_data": true, 2023-10-11T16:09:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_index_max_shards": 11, 2023-10-11T16:09:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "read_only": false, 2023-10-11T16:09:25.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tier_type": "", 2023-10-11T16:09:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from_all": true, 2023-10-11T16:09:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_from": [], 2023-10-11T16:09:25.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "redirect_zone": "", 2023-10-11T16:09:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "supported_features": [ 2023-10-11T16:09:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "compress-encrypted", 2023-10-11T16:09:25.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "placement_targets": [ 2023-10-11T16:09:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "name": "default-placement", 2023-10-11T16:09:25.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "tags": [], 2023-10-11T16:09:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "storage_classes": [ 2023-10-11T16:09:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "STANDARD" 2023-10-11T16:09:25.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "default_placement": "default-placement", 2023-10-11T16:09:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_policy": { 2023-10-11T16:09:25.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "groups": [] 2023-10-11T16:09:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled_features": [ 2023-10-11T16:09:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "resharding" 2023-10-11T16:09:25.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ], 2023-10-11T16:09:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "short_zone_ids": [ 2023-10-11T16:09:25.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "02f9727d-0de9-432b-942a-b3bdfbfa6b9c", 2023-10-11T16:09:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 2025758522 2023-10-11T16:09:25.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "37b32c30-76c0-479d-b458-e4824d081f13", 2023-10-11T16:09:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 3270403816 2023-10-11T16:09:25.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "d3375114-0016-4301-a85b-ef71f68a1016", 2023-10-11T16:09:25.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 178712359 2023-10-11T16:09:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": 2663201345 2023-10-11T16:09:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zonegroup": "a", 2023-10-11T16:09:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "master_zone": "daa316e9-869c-4074-a756-a8adfb4a3573", 2023-10-11T16:09:25.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period_config": { 2023-10-11T16:09:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_quota": { 2023-10-11T16:09:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false, 2023-10-11T16:09:25.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "check_on_raw": false, 2023-10-11T16:09:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size": -1, 2023-10-11T16:09:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size_kb": 0, 2023-10-11T16:09:25.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_objects": -1 2023-10-11T16:09:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user_quota": { 2023-10-11T16:09:25.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false, 2023-10-11T16:09:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "check_on_raw": false, 2023-10-11T16:09:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size": -1, 2023-10-11T16:09:25.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_size_kb": 0, 2023-10-11T16:09:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_objects": -1 2023-10-11T16:09:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "user_ratelimit": { 2023-10-11T16:09:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:25.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "bucket_ratelimit": { 2023-10-11T16:09:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:25.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "anonymous_ratelimit": { 2023-10-11T16:09:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_ops": 0, 2023-10-11T16:09:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_ops": 0, 2023-10-11T16:09:25.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_read_bytes": 0, 2023-10-11T16:09:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "max_write_bytes": 0, 2023-10-11T16:09:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "enabled": false 2023-10-11T16:09:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_id": "17c10fd9-3252-456e-81e5-d40bac926339", 2023-10-11T16:09:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_name": "test-realm", 2023-10-11T16:09:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-10-11T16:09:25.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-10-11T16:09:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: {"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"dghqvd-68"},"dest":{"zone":"a1","bucket":"dghqvd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"dghqvd-68"},"dest":{"zone":"a2","bucket":"dghqvd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-10-11T16:09:25.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/ 2023-10-11T16:09:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/ 2023-10-11T16:09:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/ 2023-10-11T16:09:25.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.134 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:25.135 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.135 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.135 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:08 GMT 2023-10-11T16:09:25.135 INFO:tasks.rgw_multisite_tests:/dghqvd-68/ 2023-10-11T16:09:25.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.136 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:OyH5GR5cQtap/5hQ/+o24n+X7SA= 2023-10-11T16:09:25.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:OyH5GR5cQtap/5hQ/+o24n+X7SA=', 'Content-Length': '0'} 2023-10-11T16:09:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea7eb2c99e208d69-006526c4e4-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj1 2023-10-11T16:09:25.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj1 2023-10-11T16:09:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj1 2023-10-11T16:09:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.140 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:25.140 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:25.141 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:25.141 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:08 GMT 2023-10-11T16:09:25.141 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj1 2023-10-11T16:09:25.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.142 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:a0oSY9OW79IP8iBPaw05xnxj2kY= 2023-10-11T16:09:25.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:08 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a0oSY9OW79IP8iBPaw05xnxj2kY='} 2023-10-11T16:09:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056e9a796b04e1665-006526c4e4-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/ 2023-10-11T16:09:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/ 2023-10-11T16:09:25.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/ 2023-10-11T16:09:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.146 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:25.146 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.147 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.147 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.147 INFO:tasks.rgw_multisite_tests:/dghqvd-68/ 2023-10-11T16:09:25.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.148 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg= 2023-10-11T16:09:25.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg=', 'Content-Length': '0'} 2023-10-11T16:09:25.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abe56cdd17094c64-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj2 2023-10-11T16:09:25.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj2 2023-10-11T16:09:25.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:25.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj2 2023-10-11T16:09:25.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:25.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.152 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:25.153 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:25.153 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:25.153 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.153 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj2 2023-10-11T16:09:25.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.154 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:z+69En56i8KJ+/pFU3EX1DDNUCk= 2023-10-11T16:09:25.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:z+69En56i8KJ+/pFU3EX1DDNUCk='} 2023-10-11T16:09:25.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d488bed76932db49-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/ 2023-10-11T16:09:25.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/ 2023-10-11T16:09:25.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:25.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/ 2023-10-11T16:09:25.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:25.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.158 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:25.158 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.159 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.159 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.159 INFO:tasks.rgw_multisite_tests:/dghqvd-68/ 2023-10-11T16:09:25.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.161 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg= 2023-10-11T16:09:25.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:9YvbygBdrwU5GWpaQ+wpNvkFUxg=', 'Content-Length': '0'} 2023-10-11T16:09:25.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046a0beb3ac0e496d-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj3 2023-10-11T16:09:25.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj3 2023-10-11T16:09:25.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:25.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj3 2023-10-11T16:09:25.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:25.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.165 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:25.165 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:25.166 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:25.166 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.166 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj3 2023-10-11T16:09:25.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.167 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:a0RYF3WomOAgOOjDjBlPWwM863A= 2023-10-11T16:09:25.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:a0RYF3WomOAgOOjDjBlPWwM863A='} 2023-10-11T16:09:25.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000829fa6e6f49083a7-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-69/ 2023-10-11T16:09:25.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-69/ 2023-10-11T16:09:25.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:25.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-69/ 2023-10-11T16:09:25.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:25.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.171 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:25.171 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.172 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.172 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.172 INFO:tasks.rgw_multisite_tests:/dghqvd-69/ 2023-10-11T16:09:25.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.173 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:t491/9/tz6qHYMvBT/5avjxLi3U= 2023-10-11T16:09:25.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:t491/9/tz6qHYMvBT/5avjxLi3U=', 'Content-Length': '0'} 2023-10-11T16:09:25.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016d157d2b874b50c-006526c4e5-245b-a1'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T16:09:25.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T16:09:25.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-10-11T16:09:25.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T16:09:25.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:25.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-10-11T16:09:25.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8000 2023-10-11T16:09:25.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-10-11T16:09:25.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:25.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:25.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:25.177 INFO:tasks.rgw_multisite_tests:PUT 2023-10-11T16:09:25.178 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-10-11T16:09:25.178 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-10-11T16:09:25.178 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:53:09 GMT 2023-10-11T16:09:25.179 INFO:tasks.rgw_multisite_tests:/dghqvd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-10-11T16:09:25.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:25.179 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:smoOnp+pM6Z0eWZko8vxUCFll2o= 2023-10-11T16:09:25.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 11 Oct 2023 15:53:09 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:smoOnp+pM6Z0eWZko8vxUCFll2o='} 2023-10-11T16:09:25.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079fc455dda6cec8a-006526c4e5-245b-a1'), ('Date', 'Wed, 11 Oct 2023 15:53:09 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:25.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: No --period given, using current period=064106cf-4523-427d-9a69-628884ec70af 2023-10-11T16:09:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:25.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T14:30:56.410656Z" 2023-10-11T16:09:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "0.000000" 2023-10-11T16:09:25.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:27:44.387448Z" 2023-10-11T16:09:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:53:07.869026Z" 2023-10-11T16:09:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.187 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-10-11T16:09:25.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:09.437+0000 7febbcb49780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: 2023-10-11T15:53:09.438+0000 7febbcb49780 20 RGW-SYNC:meta: read sync status 2023-10-11T16:09:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 1, 2023-10-11T16:09:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:25.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "", 2023-10-11T16:09:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "0.000000", 2023-10-11T16:09:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "state": 1, 2023-10-11T16:09:25.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T16:09:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "realm_epoch": 2 2023-10-11T16:09:25.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 1, 2023-10-11T16:09:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 1 2023-10-11T16:09:25.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T16:09:25.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-10-11T16:09:25.210 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.210 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.210 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.211 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.211 INFO:tasks.rgw_multisite_tests: "period": "064106cf-4523-427d-9a69-628884ec70af", 2023-10-11T16:09:25.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.211 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.212 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.212 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.212 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1697034656.410656_41.1", 2023-10-11T16:09:25.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-10-11T16:09:25.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T14:30:56.410656Z", 2023-10-11T16:09:25.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.215 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.216 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.216 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.217 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-10-11T16:09:25.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.218 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.218 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-10-11T16:09:25.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.219 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.219 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.219 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.220 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.220 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.220 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1697038064.387448_379.1", 2023-10-11T16:09:25.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.221 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:27:44.387448Z", 2023-10-11T16:09:25.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.222 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.222 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.223 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.223 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-10-11T16:09:25.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1697039587.869026_925.1", 2023-10-11T16:09:25.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:53:07.869026Z", 2023-10-11T16:09:25.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-10-11T16:09:25.226 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.226 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.226 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.227 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.227 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.227 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-10-11T16:09:25.228 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-10-11T16:09:25.228 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.228 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T16:09:25.229 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.229 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1697034656.410656_41.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-10-11T14:30:56.410656Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1697038064.387448_379.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:27:44.387448Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1697039587.869026_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:07.869026Z', 'realm_epoch': 2}}] 2023-10-11T16:09:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1697034656.410656_41.1', 1: '', 2: '1_1697038064.387448_379.1', 3: '1_1697039587.869026_925.1'} 2023-10-11T16:09:25.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-10-11T16:09:25.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.231 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: [ 2023-10-11T16:09:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T16:09:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:53:09.033937Z" 2023-10-11T16:09:25.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T16:09:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:53:09.095029Z" 2023-10-11T16:09:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T16:09:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "last_update": "2023-10-11T15:53:09.128491Z" 2023-10-11T16:09:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:09:25.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T16:09:25.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.257 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.258 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.258 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.258 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.259 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.259 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.259 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.259 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.260 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.260 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.260 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.261 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.262 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.262 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.262 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.263 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.263 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.263 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.263 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.264 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.265 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.266 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.266 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.266 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.267 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.267 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.267 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.268 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.269 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.269 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.269 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.270 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.270 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.270 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.271 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.271 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.272 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.273 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.273 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.273 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.274 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.274 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.274 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.275 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.275 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:09Z" 2023-10-11T16:09:25.275 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.275 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:14Z" 2023-10-11T16:09:25.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.299 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.299 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.299 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.299 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.300 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.300 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.300 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.301 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.301 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.301 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.301 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.302 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.303 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.303 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.303 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.304 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.304 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.304 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.304 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.305 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.305 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.306 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.307 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.307 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.307 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.307 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.308 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.308 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.309 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.309 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.310 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.310 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.310 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.311 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.311 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.312 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.312 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.313 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.313 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.313 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.313 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.314 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.314 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.314 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.315 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:14Z" 2023-10-11T16:09:25.315 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.315 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:19Z" 2023-10-11T16:09:25.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.336 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.337 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.337 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.337 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.337 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.338 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.338 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.338 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.339 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.339 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.340 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.341 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.341 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.341 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.341 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.342 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.342 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.342 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.343 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.344 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.344 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.344 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.345 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.345 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.345 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.346 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.346 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.346 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.347 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.348 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.348 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.348 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.348 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.349 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.349 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.350 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.350 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.350 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.351 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.351 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.351 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.352 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.352 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.352 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.352 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.353 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:19Z" 2023-10-11T16:09:25.353 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.353 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:25Z" 2023-10-11T16:09:25.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.377 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.377 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.378 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.378 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.378 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.379 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.379 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.379 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.380 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.380 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.380 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.381 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.382 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.382 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.383 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.383 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.383 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.384 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.384 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.386 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.386 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.386 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.387 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.387 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.387 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.388 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.388 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.389 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.389 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.389 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.390 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.390 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.390 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.391 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.391 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.392 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.392 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.393 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.393 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.393 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.394 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.394 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.394 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.395 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.395 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:25Z" 2023-10-11T16:09:25.395 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.396 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:30Z" 2023-10-11T16:09:25.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.417 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.417 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.418 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.418 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.418 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.418 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.419 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.419 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.419 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.420 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.420 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.422 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.422 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.422 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.423 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.423 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.423 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.423 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.425 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.425 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.425 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.426 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.426 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.426 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.427 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.427 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.428 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.429 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.429 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.430 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.430 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.430 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.431 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.432 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.432 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.432 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.432 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.433 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.433 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.433 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.434 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.434 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:30Z" 2023-10-11T16:09:25.434 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.434 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:35Z" 2023-10-11T16:09:25.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.455 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.456 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.456 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.456 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.457 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.457 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.458 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.460 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.461 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.461 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.461 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.462 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.464 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.464 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.464 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.464 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.465 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.465 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.465 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.466 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.467 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.467 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.467 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.468 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.468 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.468 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.469 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.469 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.470 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.471 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.471 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.471 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.472 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.472 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.472 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.472 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.473 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:35Z" 2023-10-11T16:09:25.473 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.473 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.475 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.475 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:40Z" 2023-10-11T16:09:25.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.496 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.496 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.497 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.497 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.497 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.498 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.498 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.498 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.499 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.499 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.500 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.501 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.501 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.501 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.502 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.502 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.502 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.503 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.504 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.504 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.504 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.504 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.505 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.505 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.505 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.506 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.507 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.507 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.507 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.508 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.508 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.508 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.509 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.509 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.510 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.511 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.511 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.511 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.511 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.512 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.512 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.512 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.513 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:40Z" 2023-10-11T16:09:25.513 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.513 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:45Z" 2023-10-11T16:09:25.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.535 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.535 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.536 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.536 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.536 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.537 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.537 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.537 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.538 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.538 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.538 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.539 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.539 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.540 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.540 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.541 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.541 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.541 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.542 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.542 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.542 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.543 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.543 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.543 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.543 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.544 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.544 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.544 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.544 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.545 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.545 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.546 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.547 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.547 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.547 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.547 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.548 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.548 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.548 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.549 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.550 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.550 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.550 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.551 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.551 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.551 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.551 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.552 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.552 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.552 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:45Z" 2023-10-11T16:09:25.553 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.553 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.555 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:50Z" 2023-10-11T16:09:25.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.578 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.578 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.578 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.579 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.579 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.579 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.580 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.580 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.580 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.581 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.581 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.583 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.583 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.583 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.584 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.584 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.584 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.585 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.585 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.586 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.586 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.587 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.587 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.587 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.588 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.588 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.588 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.589 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.590 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.590 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.590 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.591 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.591 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.591 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.592 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.593 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.593 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.593 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.594 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.594 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.594 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.595 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.595 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.595 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:50Z" 2023-10-11T16:09:25.596 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.596 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.598 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:53:55Z" 2023-10-11T16:09:25.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.618 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.619 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.619 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.619 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.620 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.620 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.621 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.621 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.621 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.622 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.623 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.624 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.624 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.624 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.625 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.625 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.625 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.626 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.626 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.627 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.627 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.627 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.627 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.628 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.628 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.629 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.630 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.630 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.630 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.631 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.631 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.632 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.632 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.633 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.634 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.634 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.634 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.635 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.635 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.635 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.636 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.636 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:53:55Z" 2023-10-11T16:09:25.636 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.636 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:00Z" 2023-10-11T16:09:25.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.657 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.657 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.658 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.658 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.658 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.659 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.659 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.659 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.660 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.660 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.661 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.662 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.662 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.662 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.662 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.663 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.663 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.664 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.665 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.665 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.665 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.666 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.666 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.667 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.668 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.668 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.668 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.668 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.669 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.669 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.669 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.670 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.671 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.671 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.671 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.672 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.672 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.672 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.673 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.673 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.673 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:00Z" 2023-10-11T16:09:25.673 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.674 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.676 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:06Z" 2023-10-11T16:09:25.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.695 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.696 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.696 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.696 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.697 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.697 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.697 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.698 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.699 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.699 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.700 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.700 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.701 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.701 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.701 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.702 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.702 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.704 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.704 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.704 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.705 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.705 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.705 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.706 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.706 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.707 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.708 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.708 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.708 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.708 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.709 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.709 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.709 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.709 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.710 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.710 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.710 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.711 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.711 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.711 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.712 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.712 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.712 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.713 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.713 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:06Z" 2023-10-11T16:09:25.713 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.713 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:11Z" 2023-10-11T16:09:25.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.736 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.736 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.736 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.737 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.737 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.737 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.738 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.738 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.738 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.739 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.739 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.739 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.741 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.741 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.742 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.742 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.742 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.743 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.743 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.744 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.744 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.744 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.745 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.745 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.745 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.746 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.746 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.746 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.747 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.748 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.748 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.748 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.749 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.749 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.749 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.750 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.750 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.751 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.751 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.752 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.752 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.753 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.753 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.753 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.754 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:11Z" 2023-10-11T16:09:25.754 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.754 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.756 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:16Z" 2023-10-11T16:09:25.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.780 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.781 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.781 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.781 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.782 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.782 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.782 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.783 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.783 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.783 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.785 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.785 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.785 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.786 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.786 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.787 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.787 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.787 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.788 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.788 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.789 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.789 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.790 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.790 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.791 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.792 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.792 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.792 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.792 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.794 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.795 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.795 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.796 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.796 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.797 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.797 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.797 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.797 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:16Z" 2023-10-11T16:09:25.798 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.798 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.799 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.800 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.800 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:21Z" 2023-10-11T16:09:25.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.820 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.820 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.821 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.821 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.821 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.822 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.822 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.822 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.823 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.823 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.823 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.825 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.825 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.825 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.825 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.826 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.826 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.826 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.827 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.828 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.828 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.829 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.829 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.830 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.830 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.831 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.831 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.832 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.832 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.833 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.833 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.834 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.835 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.835 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.835 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.836 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.836 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.837 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.837 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.837 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:21Z" 2023-10-11T16:09:25.838 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.838 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.839 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.840 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:26Z" 2023-10-11T16:09:25.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.860 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.860 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.861 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.861 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.861 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.862 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.862 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.862 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.863 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.863 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.864 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.866 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.866 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.866 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.867 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.867 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.869 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.869 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.870 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.870 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.870 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.872 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.872 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.873 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.873 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.873 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.874 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.874 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.874 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.875 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.876 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.876 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.876 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.877 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.877 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.877 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.878 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.878 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:26Z" 2023-10-11T16:09:25.878 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.879 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.881 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:31Z" 2023-10-11T16:09:25.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.900 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.901 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.901 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.901 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.902 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.902 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.902 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.903 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.903 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.903 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.903 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.904 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.904 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.906 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.906 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.906 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.907 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.907 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.908 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.909 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.909 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.910 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.910 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.911 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.911 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.911 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.912 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.913 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.913 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.913 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.914 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.914 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.914 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.915 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.916 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.916 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.916 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.917 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.917 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.917 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.917 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.918 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.918 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:31Z" 2023-10-11T16:09:25.918 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.919 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:36Z" 2023-10-11T16:09:25.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.941 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.941 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.942 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.942 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.942 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.943 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.943 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.943 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.944 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.944 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.945 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.946 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.946 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.946 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.947 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.947 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.947 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.948 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.948 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.950 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.950 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.951 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.951 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.951 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.952 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.952 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.952 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.953 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.954 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.954 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.954 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.955 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.955 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.956 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.957 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.957 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.957 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.958 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.958 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.958 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:25.959 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:25.959 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.959 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:36Z" 2023-10-11T16:09:25.960 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:25.960 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:25.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:25.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:25.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:25.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:25.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:25.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:25.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:25.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:25.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:25.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:25.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:25.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:25.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:25.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:25.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:25.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:41Z" 2023-10-11T16:09:25.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:25.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:25.983 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:25.984 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:25.984 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:25.984 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:25.984 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:25.985 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.985 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:25.985 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.986 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:25.986 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.986 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.987 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:25.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:25.988 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.988 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.989 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.989 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:25.989 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.990 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.990 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:25.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:25.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:25.991 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.992 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.992 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.992 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:25.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.993 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.993 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:25.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.994 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:25.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:25.995 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.995 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.995 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:25.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:25.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:25.996 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:25.996 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:25.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:25.997 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:25.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:25.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:25.998 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.998 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:25.999 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:25.999 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:25.999 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:25.999 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.000 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.000 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.000 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:41Z" 2023-10-11T16:09:26.001 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.001 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.003 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:47Z" 2023-10-11T16:09:26.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.022 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.022 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.022 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.023 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.023 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.024 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.024 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.024 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.025 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.025 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.027 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.027 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.027 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.028 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.028 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.028 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.028 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.030 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.030 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.030 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.030 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.031 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.031 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.031 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.032 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.032 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.033 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.033 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.034 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.034 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.034 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.034 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.035 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.035 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.036 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.037 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.037 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.037 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.037 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.038 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.038 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.038 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.039 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:47Z" 2023-10-11T16:09:26.039 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.039 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:52Z" 2023-10-11T16:09:26.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.060 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.060 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.061 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.061 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.062 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.062 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.063 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.063 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.065 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.065 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.065 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.066 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.066 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.066 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.067 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.067 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.068 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.068 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.068 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.068 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.069 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.069 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.069 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.070 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.070 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.070 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.071 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.071 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.072 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.072 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.072 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.073 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.073 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.073 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.074 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.075 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.075 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.075 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.076 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.076 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.076 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.077 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.077 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:52Z" 2023-10-11T16:09:26.077 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.078 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:54:57Z" 2023-10-11T16:09:26.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.100 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.100 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.101 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.101 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.101 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.102 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.102 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.103 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.103 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.103 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.104 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.105 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.105 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.105 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.106 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.107 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.108 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.108 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.109 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.109 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.110 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.110 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.111 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.111 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.112 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.112 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.112 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.113 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.113 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.113 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.114 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.114 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.114 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.115 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.115 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.115 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.115 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.116 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.116 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.116 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.117 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.117 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:54:57Z" 2023-10-11T16:09:26.117 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.117 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.119 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:55:02Z" 2023-10-11T16:09:26.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.139 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.139 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.139 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.139 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.140 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.140 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.140 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.141 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.141 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.141 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.141 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.142 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.142 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.142 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.143 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.143 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.143 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.144 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.144 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.144 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.144 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.145 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.145 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.146 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.147 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.147 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.147 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.147 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.148 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.148 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.149 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.150 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.150 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.150 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.151 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.151 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.151 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.152 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.153 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.153 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.153 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.154 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.154 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.154 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.155 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.155 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.155 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:55:02Z" 2023-10-11T16:09:26.156 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.156 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.158 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:55:07Z" 2023-10-11T16:09:26.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.177 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.177 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.178 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.178 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.178 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.179 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.179 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.180 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.182 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.182 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045376", 2023-10-11T16:09:26.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.122137Z" 2023-10-11T16:09:26.184 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.184 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.184 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.185 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.185 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.186 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.187 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.188 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.188 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.188 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.189 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.189 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.189 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050970", 2023-10-11T16:09:26.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.190 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:51:24.245911Z" 2023-10-11T16:09:26.191 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.191 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.191 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.192 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.192 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.193 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044032", 2023-10-11T16:09:26.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.193 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.194 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:09.044541Z" 2023-10-11T16:09:26.194 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.195 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.195 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.195 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.195 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.196 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.196 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.196 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.197 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:55:07Z" 2023-10-11T16:09:26.197 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.197 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045376', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.122137Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050970', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:51:24.245911Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044032', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:44:09.044541Z'}}] 2023-10-11T16:09:26.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045376', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000050970', 3: '00000000000000000000:00000000000000044032'} 2023-10-11T16:09:26.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045517 target=00000000000000000000:00000000000000045376, shard=2 master=00000000000000000000:00000000000000051250 target=00000000000000000000:00000000000000050970, shard=3 master=00000000000000000000:00000000000000044174 target=00000000000000000000:00000000000000044032 2023-10-11T16:09:26.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.199 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-10-11T16:09:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-10-11T16:09:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setuser ceph since I am not root 2023-10-11T16:09:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-10-11T16:09:26.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "sync_status": { 2023-10-11T16:09:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "info": { 2023-10-11T16:09:26.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "sync", 2023-10-11T16:09:26.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "num_shards": 4, 2023-10-11T16:09:26.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "instance_id": 10532025945498841711 2023-10-11T16:09:26.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "markers": [ 2023-10-11T16:09:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 0, 2023-10-11T16:09:26.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T16:09:26.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T16:09:26.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 1, 2023-10-11T16:09:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 0, 2023-10-11T16:09:26.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 2, 2023-10-11T16:09:26.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T16:09:26.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 55, 2023-10-11T16:09:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T16:09:26.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: { 2023-10-11T16:09:26.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "key": 3, 2023-10-11T16:09:26.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "val": { 2023-10-11T16:09:26.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "status": "incremental-sync", 2023-10-11T16:09:26.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T16:09:26.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "next_step_marker": "", 2023-10-11T16:09:26.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total_entries": 9, 2023-10-11T16:09:26.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "pos": 0, 2023-10-11T16:09:26.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T16:09:26.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: ] 2023-10-11T16:09:26.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "full_sync": { 2023-10-11T16:09:26.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "total": 64, 2023-10-11T16:09:26.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "complete": 64 2023-10-11T16:09:26.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: }, 2023-10-11T16:09:26.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: "current_time": "2023-10-11T15:55:12Z" 2023-10-11T16:09:26.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi158.stdout: INFO: } 2023-10-11T16:09:26.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-10-11T16:09:26.219 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-10-11T16:09:26.220 INFO:tasks.rgw_multisite_tests: "info": { 2023-10-11T16:09:26.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-10-11T16:09:26.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-10-11T16:09:26.220 INFO:tasks.rgw_multisite_tests: "instance_id": 10532025945498841711 2023-10-11T16:09:26.221 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.221 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-10-11T16:09:26.221 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.222 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-10-11T16:09:26.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.222 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.223 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045517", 2023-10-11T16:09:26.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:53:09.033937Z" 2023-10-11T16:09:26.224 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.224 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.225 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.225 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-10-11T16:09:26.225 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.226 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.226 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032203", 2023-10-11T16:09:26.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-10-11T16:09:26.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:44:25.137463Z" 2023-10-11T16:09:26.228 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.228 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.228 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.229 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-10-11T16:09:26.229 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.229 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.229 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051250", 2023-10-11T16:09:26.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.230 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-10-11T16:09:26.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:53:09.095029Z" 2023-10-11T16:09:26.231 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.231 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.231 INFO:tasks.rgw_multisite_tests: { 2023-10-11T16:09:26.232 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-10-11T16:09:26.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-10-11T16:09:26.232 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-10-11T16:09:26.233 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044174", 2023-10-11T16:09:26.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-10-11T16:09:26.233 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-10-11T16:09:26.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-10-11T16:09:26.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-10-11T15:53:09.128491Z" 2023-10-11T16:09:26.234 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.234 INFO:tasks.rgw_multisite_tests: } 2023-10-11T16:09:26.235 INFO:tasks.rgw_multisite_tests: ] 2023-10-11T16:09:26.235 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.235 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-10-11T16:09:26.236 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-10-11T16:09:26.236 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-10-11T16:09:26.236 INFO:tasks.rgw_multisite_tests: }, 2023-10-11T16:09:26.237 INFO:tasks.rgw_multisite_tests: "current_time": "2023-10-11T15:55:12Z" 2023-10-11T16:09:26.237 INFO:tasks.rgw_multisite_tests:} 2023-10-11T16:09:26.237 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.033937Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032203', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-10-11T15:44:25.137463Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051250', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.095029Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044174', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-10-11T15:53:09.128491Z'}}] 2023-10-11T16:09:26.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045517', 1: '00000000000000000000:00000000000000032203', 2: '00000000000000000000:00000000000000051250', 3: '00000000000000000000:00000000000000044174'} 2023-10-11T16:09:26.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-10-11T16:09:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/ 2023-10-11T16:09:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/ 2023-10-11T16:09:26.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/ 2023-10-11T16:09:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:26.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:26.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-10-11T16:09:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:26.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:26.242 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:26.243 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.243 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.243 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:55:12 GMT 2023-10-11T16:09:26.244 INFO:tasks.rgw_multisite_tests:/dghqvd-68/ 2023-10-11T16:09:26.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:26.244 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:wMO2+Kun5BYCNS7h1/tbxdbspC0= 2023-10-11T16:09:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:wMO2+Kun5BYCNS7h1/tbxdbspC0=', 'Content-Length': '0'} 2023-10-11T16:09:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000055de4899916b366-006526c560-2451-a2'), ('Content-Length', '0'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:26.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj1 2023-10-11T16:09:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj1 2023-10-11T16:09:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:26.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj1 2023-10-11T16:09:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:26.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:26.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:26.249 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:26.249 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.249 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.249 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:55:12 GMT 2023-10-11T16:09:26.250 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj1 2023-10-11T16:09:26.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:26.250 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:T+KtNqibTadTT+JtBwfp6+CxnHk= 2023-10-11T16:09:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:T+KtNqibTadTT+JtBwfp6+CxnHk=', 'Content-Length': '0'} 2023-10-11T16:09:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-68:daa316e9-869c-4074-a756-a8adfb4a3573.8996.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c624b81c10e1ad0-006526c560-2451-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj1 2023-10-11T16:09:26.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj1 2023-10-11T16:09:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-10-11T16:09:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj1 2023-10-11T16:09:26.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:26.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:26.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:26.254 INFO:tasks.rgw_multisite_tests:GET 2023-10-11T16:09:26.255 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.255 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.255 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:55:12 GMT 2023-10-11T16:09:26.256 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj1 2023-10-11T16:09:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:26.256 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:+fhUKPIHQ6GkKTwksK49YyTRFbg= 2023-10-11T16:09:26.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:+fhUKPIHQ6GkKTwksK49YyTRFbg=', 'Content-Length': '0'} 2023-10-11T16:09:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 11 Oct 2023 15:53:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'daa316e9-869c-4074-a756-a8adfb4a3573:dghqvd-68:daa316e9-869c-4074-a756-a8adfb4a3573.8996.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc7378ec3049b81-006526c560-2451-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dghqvd-68/obj2 2023-10-11T16:09:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dghqvd-68/obj2 2023-10-11T16:09:26.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-10-11T16:09:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dghqvd-68/obj2 2023-10-11T16:09:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-10-11T16:09:26.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-10-11T16:09:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi158.front.sepia.ceph.com:8001 2023-10-11T16:09:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-10-11T16:09:26.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-10-11T16:09:26.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-10-11T16:09:26.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-10-11T16:09:26.260 INFO:tasks.rgw_multisite_tests:HEAD 2023-10-11T16:09:26.261 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.261 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.261 INFO:tasks.rgw_multisite_tests:Wed, 11 Oct 2023 15:55:12 GMT 2023-10-11T16:09:26.262 INFO:tasks.rgw_multisite_tests:/dghqvd-68/obj2 2023-10-11T16:09:26.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-10-11T16:09:26.262 INFO:tasks.rgw_multisite_tests:AWS MWUFUL61ESHN8BNC6ISZ:6iyxDcB8fFjs8V9QR8kIXYPgcW0= 2023-10-11T16:09:26.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Wed, 11 Oct 2023 15:55:12 GMT', 'Authorization': 'AWS MWUFUL61ESHN8BNC6ISZ:6iyxDcB8fFjs8V9QR8kIXYPgcW0=', 'Content-Length': '0'} 2023-10-11T16:09:26.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000074f64392d3c581b5-006526c560-2451-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 11 Oct 2023 15:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-10-11T16:09:26.263 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-10-11T16:09:26.264 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.264 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-10-11T16:09:26.265 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 5784.579s 2023-10-11T16:09:26.265 INFO:tasks.rgw_multisite_tests: 2023-10-11T16:09:26.265 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=5) 2023-10-11T16:09:26.268 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-11T16:09:26.354 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e17678c3f16471388aaa36f0710568f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_4305a5835ec2627a352020742c8983c32bac9341/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-10-11T16:09:26.356 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-10-11T16:09:26.367 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-10-11T16:09:26.407 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-11T16:09:26.418 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-10-11T16:09:26.418 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:26.434 INFO:tasks.rgw.c1.client.0.smithi158.stdout:2023-10-11T16:09:26.417+0000 7f4660afa700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 121094) UID: 0 2023-10-11T16:09:26.434 INFO:tasks.rgw.c1.client.0.smithi158.stdout:2023-10-11T16:09:26.417+0000 7f4664d637c0 -1 shutting down 2023-10-11T16:09:32.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-10-11T16:09:32.521 INFO:tasks.rgw.c1.client.0:Stopped 2023-10-11T16:09:32.522 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-10-11T16:09:32.546 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/vault-root-token 2023-10-11T16:09:32.621 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-10-11T16:09:32.621 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:32.662 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T16:09:32.660+0000 7fb235d51700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 182315) UID: 0 2023-10-11T16:09:32.663 INFO:tasks.rgw.c1.client.1.smithi158.stdout:2023-10-11T16:09:32.660+0000 7fb239fba7c0 -1 shutting down 2023-10-11T16:09:37.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-10-11T16:09:37.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-10-11T16:09:38.723 INFO:tasks.rgw.c1.client.1:Stopped 2023-10-11T16:09:38.724 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-10-11T16:09:38.747 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f /etc/ceph/vault-root-token 2023-10-11T16:09:38.819 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-10-11T16:09:38.820 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:38.821 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2023-10-11T16:09:38.820+0000 7f6459ecf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 121881) UID: 0 2023-10-11T16:09:38.821 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2023-10-11T16:09:38.820+0000 7f645e1387c0 -1 shutting down 2023-10-11T16:09:43.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-10-11T16:09:43.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-10-11T16:09:44.922 INFO:tasks.rgw.c2.client.0:Stopped 2023-10-11T16:09:44.923 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-10-11T16:09:45.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/vault-root-token 2023-10-11T16:09:45.128 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-10-11T16:09:45.128 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:45.129 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2023-10-11T16:09:45.128+0000 7fea10a3f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 123176) UID: 0 2023-10-11T16:09:45.130 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2023-10-11T16:09:45.128+0000 7fea14ca87c0 -1 shutting down 2023-10-11T16:09:49.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-10-11T16:09:49.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-10-11T16:09:51.231 INFO:tasks.rgw.c2.client.1:Stopped 2023-10-11T16:09:51.232 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-10-11T16:09:51.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/vault-root-token 2023-10-11T16:09:51.327 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-11T16:09:51.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-10-11T16:09:52.197 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-11T16:09:52.198 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-11T16:09:52.198 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-10-11T16:09:52.198 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:52.277 INFO:tasks.ceph.c2.osd.0:Stopped 2023-10-11T16:09:52.278 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-10-11T16:09:52.278 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:52.356 INFO:tasks.ceph.c2.osd.1:Stopped 2023-10-11T16:09:52.357 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-10-11T16:09:52.357 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:52.426 INFO:tasks.ceph.c2.osd.2:Stopped 2023-10-11T16:09:52.426 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-11T16:09:52.427 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-10-11T16:09:52.427 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:52.451 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-10-11T16:09:52.452 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-11T16:09:52.453 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-10-11T16:09:52.453 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:52.511 INFO:tasks.ceph.c2.mon.a:Stopped 2023-10-11T16:09:52.512 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-11T16:09:52.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-11T16:09:52.618 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T16:09:52.619 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-10-11T16:09:52.881 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T16:09:52.882 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-10-11T16:09:52.997 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T16:09:52.998 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-10-11T16:09:53.106 INFO:tasks.ceph:Archiving mon data... 2023-10-11T16:09:53.108 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/data/c2.mon.a.tgz 2023-10-11T16:09:53.108 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-11T16:09:53.133 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.NRqhpv2yPt 2023-10-11T16:09:53.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.NRqhpv2yPt 2023-10-11T16:09:53.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.NRqhpv2yPt 2023-10-11T16:09:53.390 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.NRqhpv2yPt is 344KB 2023-10-11T16:09:53.490 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.NRqhpv2yPt 2023-10-11T16:09:53.509 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-11T16:09:53.510 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T16:09:53.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T16:09:53.576 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-11T16:09:53.587 DEBUG:teuthology.orchestra.run.smithi158:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-10-11T16:09:53.934 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-11T16:09:53.935 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-11T16:09:53.936 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-10-11T16:09:53.936 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:53.981 INFO:tasks.ceph.c1.osd.0:Stopped 2023-10-11T16:09:53.981 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-10-11T16:09:53.982 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:54.036 INFO:tasks.ceph.c1.osd.1:Stopped 2023-10-11T16:09:54.037 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-10-11T16:09:54.038 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:54.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-10-11T16:09:54.081 INFO:tasks.ceph.c1.osd.2:Stopped 2023-10-11T16:09:54.082 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-11T16:09:54.082 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-10-11T16:09:54.082 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:54.106 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-10-11T16:09:54.107 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-11T16:09:54.108 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-10-11T16:09:54.108 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:09:54.166 INFO:tasks.ceph.c1.mon.a:Stopped 2023-10-11T16:09:54.166 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-11T16:09:54.167 DEBUG:teuthology.orchestra.run.smithi158:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2023-10-11T16:09:54.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T16:09:54.237 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-10-11T16:09:54.624 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T16:09:54.625 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-10-11T16:09:54.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T16:09:54.729 DEBUG:teuthology.orchestra.run.smithi158:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-10-11T16:09:54.806 INFO:tasks.ceph:Archiving mon data... 2023-10-11T16:09:54.808 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/data/c1.mon.a.tgz 2023-10-11T16:09:54.808 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2023-10-11T16:09:54.826 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.3lmbEgyGBn 2023-10-11T16:09:54.827 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.3lmbEgyGBn 2023-10-11T16:09:54.996 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.3lmbEgyGBn 2023-10-11T16:09:55.084 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.3lmbEgyGBn is 399KB 2023-10-11T16:09:55.180 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.3lmbEgyGBn 2023-10-11T16:09:55.197 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-11T16:09:55.198 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T16:09:55.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T16:09:55.273 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-11T16:09:55.273 DEBUG:teuthology.orchestra.run.smithi158:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-11T16:09:55.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-11T16:09:55.332 INFO:teuthology.orchestra.run.smithi158.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-11T16:09:55.345 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-11T16:09:55.351 INFO:tasks.ceph:Archiving crash dumps... 2023-10-11T16:09:55.353 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi158/crash 2023-10-11T16:09:55.354 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-11T16:09:55.405 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi172/crash 2023-10-11T16:09:55.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-11T16:09:55.437 INFO:tasks.ceph:Compressing logs... 2023-10-11T16:09:55.437 DEBUG:teuthology.orchestra.run.smithi158:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:09:55.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:22:20.576 INFO:tasks.ceph:Archiving logs... 2023-10-11T16:22:20.577 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi158/log 2023-10-11T16:22:20.579 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T16:22:36.361 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi172/log 2023-10-11T16:22:36.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T16:24:01.799 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-11T16:24:01.809 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-11T16:24:01.810 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T16:24:01.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T16:24:01.974 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-11T16:24:01.984 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-11T16:24:01.984 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:24:01.987 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:24:02.005 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-11T16:24:02.005 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2023-10-11T16:24:02.011 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:24:02.011 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2023-10-11T16:24:02.011 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 109 -1680us[-1672us] +/- 76ms 2023-10-11T16:24:02.012 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 7 377 104 -743us[ -735us] +/- 57ms 2023-10-11T16:24:02.012 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +2859us[+2859us] +/- 76ms 2023-10-11T16:24:02.012 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:24:02.043 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:24:02.044 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-11T16:24:02.044 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 111 -1754us[-1838us] +/- 76ms 2023-10-11T16:24:02.044 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 7 377 50 -825us[ -921us] +/- 58ms 2023-10-11T16:24:02.044 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +2771us[+2677us] +/- 76ms 2023-10-11T16:24:02.045 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:24:02.046 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-11T16:24:02.056 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-11T16:24:02.057 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-11T16:24:02.068 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T16:24:02.170 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T16:24:02.330 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:24:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:24:02.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:24:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:24:02.407 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-11T16:24:02.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-11T16:24:02.479 INFO:teuthology.task.internal:Duration was 7712.649564 seconds 2023-10-11T16:24:02.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-11T16:24:02.494 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-11T16:24:02.495 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T16:24:02.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T16:24:02.550 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:24:02.584 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:24:02.980 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-11T16:24:02.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2023-10-11T16:24:02.981 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T16:24:03.007 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-10-11T16:24:03.007 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T16:24:03.031 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-11T16:24:03.031 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:24:03.049 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:24:03.114 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-11T16:24:03.115 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T16:24:03.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T16:24:03.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-11T16:24:03.540 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-11T16:24:03.541 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T16:24:03.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T16:24:03.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-11T16:24:03.640 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:24:03.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:24:03.668 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2023-10-11T16:24:03.686 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-10-11T16:24:03.712 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:24:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:24:03.753 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:24:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:24:03.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-11T16:24:03.782 INFO:teuthology.task.internal:Transferring archived files... 2023-10-11T16:24:03.783 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi158 2023-10-11T16:24:03.784 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T16:24:03.832 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167/remote/smithi172 2023-10-11T16:24:03.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T16:24:03.871 INFO:teuthology.task.internal:Removing archive directory... 2023-10-11T16:24:03.871 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T16:24:03.874 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T16:24:03.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-11T16:24:03.936 INFO:teuthology.task.internal:Not uploading archives. 2023-10-11T16:24:03.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-11T16:24:03.953 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-11T16:24:03.953 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T16:24:03.956 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T16:24:03.971 INFO:teuthology.orchestra.run.smithi158.stdout: 526060 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 16:24 /home/ubuntu/cephtest 2023-10-11T16:24:03.980 INFO:teuthology.orchestra.run.smithi172.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 16:24 /home/ubuntu/cephtest 2023-10-11T16:24:03.982 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-11T16:24:03.993 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-11T16:24:04.108 INFO:teuthology.nuke:Checking targets against current locks 2023-10-11T16:24:04.132 DEBUG:teuthology.nuke:shortname: smithi158 2023-10-11T16:24:04.133 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:24:04.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 14:00:48.411449', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdZOUR/JCj79beNQ0wybs7M7i43qMxJmr70rOt4AEhBgiCaM4r0c1U41R5LwI8ihJ/8urmC0YXaW4daE5rbCfs='} 2023-10-11T16:24:04.159 DEBUG:teuthology.nuke:shortname: smithi172 2023-10-11T16:24:04.159 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:24:04.183 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-11_13:59:56-rgw-wip-yuri-testing-2023-10-10-0840-reef-distro-default-smithi/7421167', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 14:00:48.409643', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNBaymyTQnulDVyWSaS/bdKw8zsfFDB9kBVESWAo2vuV0yWDK8SWyG3kxrnaNRny+vL4X8wX5KsD37JgctSCCvU='} 2023-10-11T16:24:04.196 INFO:teuthology.orchestra.console.smithi158:Power off 2023-10-11T16:24:04.197 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T16:24:04.223 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2023-10-11T16:24:04.224 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:04.240 INFO:teuthology.orchestra.console.smithi172:Power off 2023-10-11T16:24:04.241 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T16:24:04.251 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-11T16:24:04.265 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-10-11T16:24:04.265 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:04.290 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-11T16:24:08.253 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:08.292 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:08.364 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-11T16:24:08.404 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-11T16:24:12.366 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:12.406 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:12.480 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2023-10-11T16:24:12.517 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-11T16:24:12.581 INFO:teuthology.orchestra.console.smithi158:Power off completed 2023-10-11T16:24:16.521 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:24:16.632 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-10-11T16:24:16.733 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-10-11T16:24:16.838 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 7712.649564027786 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e17678c3f16471388aaa36f0710568f status: fail success: false 2023-10-11T16:24:16.838 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T16:24:16.929 INFO:teuthology.run:FAIL